As always - Welcome to the /promptcollective!
This week I want to share a project with you that we have been working on at DRIVE beta for a while…
https://www.tiktok.com/@reactionsfromhell
What is going on here you might ask?
Well, we wanted to see how far we could already automate a content generation process using an AI pipeline, that we as always developed in-house. What I mean by that is, that we put together different generative AI tools that we access via their APIs via a coded backend. In this case, we are working with the German public service broadcaster ZDF on answering the question:
CAN AI BE CREATIVE (and maybe even funny)?
Or maybe a bit more concrete: How easy or difficult is it to create a fully automated workflow that generates video content and how good/bad is the content generated? For extremely interesting thoughts on what this type of content might eventually do to the internet have a listen of the podcast episode “Will AI break the internet? Or save it?” linked at the end of this post.
To do this we decided to use TikTok as a distribution platform, as the short videos and creative surroundings felt like the perfect arena for our experiment. And we decided to create the devil, as it is a character with high face- and name recognition :-).
And that is where the interesting challenges started! As so often with AI, it was not the things we thought would be difficult that ended up being it, but the seemingly simple. My personal highlight below…
HOW WE DO IT
We use D-ID via the API to animate our devil, but once we input our first images of the devil into the tool we saw, that the horns were not moving with the face. So we had to prompt a devil with really small horns that were in front of the forehead. This sounds like a simple task, but it ended up taking a really long time before my colleague Paulina cracked the code using DALL-E 3 image generator via ChatGPT Pro. (Give it a shot yourself if you want…)
Here is a selection of devils, that did not make the cut:
(here the horns were fine, but we just didn’t like his smug face)
And finally our devil of choice (sitting in a gaming chair, of course, courtesy to photoshop generative fill):
Once we had the devil image and knew it worked in D-ID, we started designing the rest of the workflow - it ended up looking like this:
As you can see, there really is only one input by the editorial team, this might be something like “banana bread”. For this input, we use an internal web-app. After input the automation takes over, generating the text, the voice, the b-roll footage as well as our avatar video and edits it all together applying zooms, cuts, and music to get a video like this:
Is it funny? I would say sometimes absolutely.
WHAT DID WE LEARN?
Is AI creative? That is such a difficult question! I have to say, that I am very impressed by the level of the output we are able to generate in a fully automated way. And looking at the curve of all the tools, from our language models to image generators and video creation, it will get shockingly good by the end of this year. But at the same time I wonder if the whole act of automating creative work is even a route I want to go down… We and I think a lot about the question not only of what AI is capable of doing, but what we actually want to use it for.
For now, I am really enjoying the experiment and hope we will be able to collect some more subscribers for our bad influencer from hell. He is a bit sad that not more people are following him…
For all german speaking readers: We have also set up a LinkedIn page together with the ZDF discussing all thing reactionsfromhell as well as generative AI in film production more in general. I would love for you to join our conversation!
You can find it here:
ONE THING I LOVED THIS WEEK
I have just listened to the 3 newest episodes of the Ezra Klein Show Podcast - they are about AI and I found each of them deeply interesting:
Thank you so much for reading along!
Who Are We?
The /promptcollective, by Jes Brandhøj and Hannes Jakobsen, is a hub for those fascinated by the intersection of AI and creativity. Got any ideas or questions? Reach out at join@promptcollective.xyz. And please subscribe.