Midjourney was my first actual text-to-image AI software expertise beginning again in 2022. I’ve watched it evolve because the trade swelled with competitors and plenty of different picture, animation and video instruments popping up nearly weekly since. By January 2023, the instruments began to evolve to a spot the place they made us sit up and take discover, as I outlined in my first AI Instruments article, AI Instruments Half 1: Why We Want Them.
However after years of progress and plenty of testing, Midjourney has now raised the bar but once more, with the introduction of their new video software, and I’m pleasantly shocked at what it may achieve this shortly and sensibly. Right here’s what I’ve explored to date…
Body from Noir model brief clip created in Midjourney Video
Midjourney Animate
Midjourney introduced the brand new video animation characteristic and the output is kind of spectacular!
I’ve been utilizing loads of completely different animation and video technology instruments the previous few years, as you might know in the event you’ve been following my AI Instruments collection right here on ProVideo. However that is probably the most seamless and quickest workflow I’ve but to have interaction with.
Most turbines require a beginning picture – like a keyframe if you’ll. I nearly at all times begin with a picture I’ve generated in Midjourney after which gone to a different software to animate it. (You possibly can see my final article, AI Instruments: Generative AI for Video & Animation Updates for extra examples of the workflow). However now in Midjourney, you’ll be able to both generate a brand new picture as your supply, or begin with your personal photograph.
First – the small print and specs…
Presently, everybody with an account can entry the Animate choice, however solely the Professional and Mega plans can use the Chill out mode, and movies devour 8x extra time to course of than photos, however it does offer you 4 variations to select from in every spherical.
Video Output Sizes & Codecs
Be aware that the utmost decision in the intervening time is 480p (832×464) and the sizes range relying on facet ratio after all.
You possibly can export your video in a compressed MP4 for social use, or a bigger RAW MP4 H.264 model (nonetheless compressed however much less) and animated GIF. You possibly can hyperlink to the finished video’s URL because it stays within the cloud in your account.
That is the Codec knowledge from a “RAW” information downloaded from Midjourney:
Midjourney Video Check Drive
In fact I needed to dive in and soak up all I might with this new characteristic and I spent a pair days working it via the paces.
Beginning off, I attempted utilizing some easy prompts for numerous information reporters for use as B-roll. (I’d use one thing like this in a pinch to placed on a display in a shot that simulated a newscast on TV, for example). The standard is nice sufficient for the size it gives (480p) however on no account meant for full-screen on this preliminary roll-out.
My first step was to get some figures to animate. I entered brief immediate descriptions for Midjourney to generate some examples. It’s humorous what AI thinks about ages at occasions. And among the outcomes are simply so improper they’re HILARIOUS!
After deciding on the topic I needed for every shot, I let Midjourney resolve on the movement with the Auto Animate choice. Every cross gives you with 4 completely different movies to select from so you’ll be able to prolong out many choices.
I created this video to indicate you the picks and outcomes for every topic.
I did the identical with these different examples from ChatGPT prompts and clarify the method for every instance.
(Be aware: the VO says 840p after I know rattling effectively it’s 480p! Linguistically dyslexic I assume!)
Photograph to video
Testing out the photograph to video characteristic in Midjourney, I used an outdated picture of my 80’s hair rock & roll days. A lot hair product again then!
I uploaded the photograph as the primary body and I let Midjourney do the work from there. I prolonged it simply as soon as extra to make about an 8 second clip. I exported it as an animated GIF (not JIF) since there isn’t any audio. If solely I used to be actually that cool on stage!
Utilizing Midjourney for Storyboarding and Previz
Presently, I see Midjourney as a software for creativity and serving to you deliver your concepts to life. Not essentially as an finish product, however to comprehend how the written phrase will be visualized on the display.
This might be a tremendous software for screenwriters attempting to promote a remedy, or for storyboarding scenes and photographs for places, units, lighting and blocking.
I created a brief scene utterly with AI instruments in just some steps – with two completely different variations to indicate how seamlessly Midjourney responds to prompts and extensions.
I began with ChatGPT asking for concepts for initiatives to do and this was one of many outcomes I adopted.
I used to be proud of the ensuing photos so I went with one I preferred and determined to construct a narrative round what the character ended up doing with my prolonged prompts.
I need to say, this was some of the satisfying and artistic initiatives in an intuitive workflow that I’ve achieved in years. And it actually solely took a number of hours from begin to end, as a result of I had no preconceived concept what it was going to be – and I used to be going to let the AI Instruments be my companions as my writers, actors, sound FX, staging and digital camera ops. I actually felt like a director of kinds.
Every render cross gives 4 completely different variations based mostly on the primary body (or continues from the final cross with every extension as much as 4x). It’s very subjective to resolve what take you wish to use, however that’s a part of the storytelling facet. In my case, I began with the ChatGPT unique immediate after which instructed the motion to the top of the immediate with directions.
Every cross I might add a brand new instruction or route. Largely the digital camera strikes and angles have been decided by Midjourney however these will be directed extra intently as effectively. Nevertheless, it doesn’t at all times observe directions for motion, however you’ll be able to typically idiot it by rewording directions. Typically although, the errors can really change the story and also you observe a unique rabbit down the outlet.
I’m together with a number of GIFs beneath displaying the order of the method and the following renders for every immediate instruction, and choice I produced from these render end result to proceed on constructing my scene.
Immediate (with chosen begin picture): movie noir model, trench coat detective below a streetlamp in heavy rain, black and white with refined shade tint, glistening cobblestone, intense distinction, Nineteen Fifties city alley setting, moody and mysterious, he’s holding a lit cigarette and appears round like he’s ready on somebody
Immediate change/addition: he begins to cross the road whereas the digital camera follows his motion and he flicks the cigarette down on the road. (He didn’t cross the road however I went with it)
Immediate change/addition: a lady seems from the shadows on the suitable and runs as much as him urgently. (extra like a sluggish saunter, however it works)
Immediate change/addition: the couple kisses and embrace.
You’ll have to observe the video beneath to see what Variation 2 ended like!
So I wanted so as to add sound to this brief scene and I wanted a voice over to relate in an acceptable tone and voice.
I began with ChatGPT once more and my AI script writing accomplice and I got here up with some good strains. (You’ll hear each variations within the video beneath).
I used the script textual content in ElevenLabs utilizing their new v3 Alpha mannequin for a extra pure speech supply and I discovered an important voice that basically match the time interval.
I additionally used ElevenLabs to provide my sound FX and music mattress.
All the pieces blended simply in Adobe Premiere Professional in simply minutes. And right here’s the end result(s)
For extra detailed information about Midjourney video choices and utilization directions, go to their web site.
