Voice cloning for creating new, voiced dialogues using the original actors' voices

[Vel (Tien-Yun) Wu]

Hi there,

This idea occurred to me recently and after doing some research, I would like to propose this and see how you folks feel about it. Please also vote in the poll which will close in 31 days. Is this something you'd like? If yes, then I'll start working on it and do my best to make it work for Warcraft 3.

Found an "application" which is written in Python 3 which happens to my most fluent programming language. If my attempt is successful, it will mean this project can implement completely new written dialogues that are voiced and we don't even need to find the original voice actors/actresses!

Two options to do this:

1. Corentin Jemine's Real-Time-Voice-Cloning solution:

- Demo:

- Source codes:
CorentinJ/Real-Time-Voice-Cloning

2. The Respeecher solution made by this fine company from Urkraine:
Voice Cloning Software for Content Creators | Respeecher

The 2nd option is likely to be more professional but it will obviously cost money. So I am prioritizing the 1st unless you folks know of some ways to afford the cost from using 2nd (I don't know how much it will cost but I will reach out to them)

I am also currently a graduate student at the University of Illinois at Urbana-Champaign in the United States, focusing on data science, so there is a non-zero chance I can propose this as a class project for Applied Machine Learning to the school, such that it earns me grades. This means that other than my own passion, I have an actual reason to make it work.

I am well aware that this proposal is very experimental, has a chance to fail, and is against the current design direction that Mr. Luca / @InsaneMonster has in mind. Therefore, I will be very grateful if more of you can join me in this discussion. Anything can be changed or discussed.

Take care, my fellow Warcraft 3 fans.

p.s.: If the poll does show that you folks are interested or if @InsaneMonster gives me the green light on this, I would be glad to have teammates.
If you know Python 3 and understand how these packages work, you are welcome to join me: pandas, numpy, pathlib, os, pytorch, sklearn

 

[Khazmo]

I really can't see the harm in trying it out, it might turn out that we cannot create new dialogue of sufficient quality but it might surprise us. It is definitely worth the experiment.

If you're going to experiment, I recommend starting with a character like Arthas using either his human or his undead campaign dialogue. If decent lines can be made from him, there is hope for others

 

[Ethenil]

I really can't see the harm in trying it out, it might turn out that we cannot create new dialogue of sufficient quality but it might surprise us. It is definitely worth the experiment.

If you're going to experiment, I recommend starting with a character like Arthas using either his human or his undead campaign dialogue. If decent lines can be made from him, there is hope for others

I fully agree!

 

[VictorZ]

I think it is a great idea,imagine that you can clone Arthas voice like orginal actor's voice, we will have more dialogue to improve the story.For instance,a small dialogue between Arthas and Anesterian when he destroy Silvermoon( Reforged just let Anesterian in the map and Arthas kill him like a creep).

 

[InsaneMonster]

The thread pool does seem a bit biased in its formulation! xD

Anyway, no harm in trying, but I will not devolve my own development time to this. And, like all additional dialogues, it's something we should only consider if really needed. I could see to use something like this to fix the italian translation of Kel'Thuzad, which is bugged in the game (basically, the dialogue voice is different from the response voice, and this is bad for boss battles!)

 

[Vel (Tien-Yun) Wu]

The thread pool does seem a bit biased in its formulation! xD

I know, right?? My thanks to you folks for expressing interests, though.

Please check attached ZIP for a bare-minimum example I just produced.

These are flac files, same as the audio extracted using the World Editor. You'll hear Jaina and Arthas's cloned voices saying these simple lines below:

"This is a cloned voice. There are many more like it, but this one example is for all of you. This was generated using <xxx> seconds worth of <Character Name>'s voice."

For Jaina, 59 seconds of her original voice was fed to the cloning program. For Arthas, it was 135 seconds.

Please have a listen and let me know what you think.

Please also comment if you'd like to hear a particular character say something, with the character's name and the dialogue you'll have him/her say (in text).

 

[Khazmo]

Okay, I had a quick listen to both examples.

I think its safe to say that both of them sound synthetic at the moment, and I found that it was much easier to identify Arthas' voice than Jainas. Though, I do think with a larger pool of dialogue there will be a lot more success with this process. I'm keen to hear how it sounds if you use more audio files.

If I remember rightly, Arthas' voice actor was only used in WC3, but Jaina's was used in WC3 and some of the early parts of WoW, so she may end up with a bigger pool to draw from. I do think you should focus on Arthas first though, to see how that goes.