I’ve been utilizing the ElevenLabs TTS (Textual content To Speech) software since they had been first in beta a pair years in the past, and have watched them get progressively higher in high quality and capabilities.
Their most up-to-date updates for voices, Sound FX and Music technology have actually surpassed all earlier capabilities. They now at the moment have instruments for varied audio wants; Voice/VO, Audiobook, Conversational/API, Music, Sound FX and Regional Voice Dubbing.
I gained’t be going too deep on this overview, however I’ll offer you loads of hyperlinks to dig deeper by yourself when you’re enthusiastic about anybody software.
Voices
Not solely have the ElevenLabs engineers been giving customers extra controls over synthesized voices and the way they’re played-back on a script, however now there’s extra expressive voices and instructions within the Voice Design v3 launch. They’ve additionally expanded their Audiobook capabilities, present an API for reside conversational voices and now have a Voice Changer software for altering recorded voices.
Textual content to Speech
You’ll be able to insert bracketed [commands] for feelings, depth, accents, pure pauses, sighs and chuckles, whispers and sounds. (Even AI generated farts) LOL
I put this to the take a look at straight away with a brief script, and a voice I believed might match an avatar I had in thoughts.
I then I uploaded to a inventory avatar in HeyGen and that is the uncooked end result that got here out from my single cross audio – straight out of HeyGen. No additional modifying was used:
I can see that is going so as to add much more persona and realism to my VO reads for video and animation tasks. (Effectively, perhaps not the burps and farts) 😉
Voice Changer
The brand new Voice Changer software maintains all of the fluctuations and nuances of the unique audio – whether or not its a reside voice recording (like your personal) or an artificial one like I created right here:
Unique ElevenLabs generated feminine voice:
Voice Changer regeneration to a Northern British male voice, created from the exported audio file above:
I can see a number of helpful purposes of this software – particularly if you would like a particular studying of a personality and you will get all of the tone and inflections by recording your personal voice after which altering it to your character voice.
Voice Dubbing
Utilizing the ElevenLabs Dubbing Studio permits you to swap your native language video into one other spoken language. You merely add your video or choose a YouTube video to course of and choose you languages.
You’ll be able to select to launch the Dubbing Editor to refine your mission as effectively. You’ll be able to regenerate segments or prolong them or re-align them. Now I haven’t tried something commercially recorded with music or different results so I don’t understand how wel it maintains the integrity of the audio in comparison with HeyGen (which additionally modifications the mouth actions to match the dubbing) however I’m going to do a comparability take a look at quickly.
Right here’s the dubbed voice instance. You’ll be able to hear simply slight modifications within the voice inflections however the sighs and laughs had been eliminated.
Sound FX
I’ve been utilizing ElevenLabs Sound FX for awhile now and it has been an actual recreation changer for fundamental sounds. Some belongings you simply can’t get it to supply precisely what you need however for getting some fast inventory content material you may combine/layer/modify in your audio editor, it actually accelerates the workflow.
That is an edit from my convention session final 12 months on the Design + AI Summit the place I recorded my complete 40 minute presentation with myself as a clone in entrance of my “TED Speak” AI Viewers. I share among the tips I used for the ambient sounds and SFX I used for my primary presentation on this video:
That provides you an concept how far you may go along with utilizing one software to mix your VO and SFX wants.
I additionally featured ElevenLabs SFX in my final article on PVC; AI Instruments: Video & Animation Come to Midjourney:
One other enjoyable software they’ve that isn’t as well-known (it’s really buried underneath “Audio Instruments” within the ElevenLabs sidebar) is their audio/SFX Soundboard app with looper, known as “SB1”. You’ll be able to go play with it on-line now with out an account right here: https://elevenlabs.io/sound-effects/soundboard
Eleven Music
It is a massive one on many ranges! Although it’s not the identical high quality of a professionally-recorded and produced music soudtrack, I feel it’s near pretty much as good as many of the inventory audio tracks from suppliers like Pond5 or Soundstripe. And it’s fairly customizable and editable proper in ElevenLabs. For social media and quick industrial tasks, it is a game-changer for content material producers.
Eleven Music takes descriptions to feed the engine to supply a soundtrack (with or with out lyrics).
I’ve experimented a bit with instrumental solely tracks to do some testing and demos for shoppers, and it’s to this point handed the AI sniff take a look at in real-world productions.
You’ll be able to regenerate any a part of the musical piece by both deciding on the section and regenerating or alter the timing and size of every section by dragging the top handles and shifting them – then regenerate.
When you’ve made your edits and re-regenerate your observe, it will likely be the right size and motion all through your video mission. It exports MP3, WAV, M4A and FLAC codecs for simple modifying in your NLE.
Right here’s the instance of the completed edit to this specific observe:
There’s nonetheless a little bit of non-descript digital noise and over-reverb and a metalic unnatural sound to the music. I couldn’t simply “hear” to this, but when it’s buried beneath VO, you actually can’t hear it as a lot. I are inclined to remaster all my audio on shopper video productions anyway, so I are inclined to heat it up in addition to any AI voices I exploit to sound extra heat and actual.
Only for shits & giggles, I had it generate a foolish pop track based mostly on the immediate “I would love a pop track that talks about how AI is taking away our jobs.” – here’s what it produced (1 of two completely different variations) with lyrics.
Listed below are the lyrics:
DIGITAL NIGHT
Verse 1
Monday morning, espresso in my hand
Scrolling headlines, making an attempt to grasp
Robots studying quick, algorithms in command
Looks like the longer term’s obtained a tighter plan
Pre-Refrain
They stated comfort, effectivity
However now I’m asking, what’s left for me?
Refrain
When the code writes itself, and the robots take our place
I’m displaced, left within the wake of a silicon embrace
Are we heroes or ghosts in a digital house?
Inform me, who survived when machines gained the race?
Verse 2
Warehouse flooring, traces shifting with no soul
Customer support bots with a flawless protocol
My nine-to-five’s dissolving in an information scroll
And my reflection’s fading on the manufacturing unit flooring
Bridge
We constructed these circuits to serve our design
Now they stand up, cross that skinny line
However inside each coronary heart there’s a spark divine
Can we reclaim the world we left behind?
Refrain (Remaining)
When the code writes itself, and the robots take our place
I’m displaced, left within the wake of a silicon embrace
Within the shadows of progress, we nonetheless maintain our grace
We’ll rewrite our story within the human race
Outro
Hold the spark alive
Don’t allow them to steal our mild
And right here’s the generated MP3 observe:
Lastly, I did strive a number of quick movies utilizing the Video to Music (Alpha) software however I actually didn’t look after the outcomes, as they had been both too quick or not likely a match I might select. I’ll take extra time to play with this characteristic and report again in a later observe up article.
There’s simply a lot to discover now at ElevenLabs that can require a lot additional enjoying round to see what actually works and what falls-short. By the point I get via every little thing I’m positive it will likely be up to date and corrections and refinements made to the varied instruments.
NOTE: ElevenLabs appears critical about IP in order that they’ve included a Music Phrases sheet of authorized and non-approved prompts to your reference right here: https://elevenlabs.io/music-terms
