Reforged - Lip sync on characters during cinematic

[Solcius123]

Hello everyone

Creating maps is still enjoyable and I feel like there is even more to do now despite some ridiculous bugs.

I would like to have my characters fake lip sync during cinematic, as they would in the past. As anyone tried to do it?

It is tougher that I thought, I have been analyzing some of Blizzard reforged maps to find where lip sync is called, if it is even there...

I have been searching from top to bottom in the Human 05 Interlude, being short and no gameplay, it is easier to look at.

• Cinematic - Play Dialogue to (All players) from Medivh 0002 <gen> with type Medivh. Play H05Medivh36 <gen>. Modify duration: Add 0.00 seconds and Don't wait

First I can see that dialogues are referenced elsewhere and not directly input in the triggers. It looks like the sound file links everything else that is needed for the dialogue. Unless the game is reading an external file.

Edit : I just realized that this "Cinematic - Play Dialogue From Speaker" function is new, they do not use "Cinematic - Transmission From Unit" anymore. They instead use this one where you cannot input text.

• Animation - Add the cinematic animation tag to Jaina 0004 <gen>

I noticed that they add this cinematic animation tag to every character, don't really know what it does...

Edit : It appears that this animation tag allows the characters to walk with the new Reforged animation. Not implemented in all of the units though. Been trying to add it to other units but in vain for now.

Then over here in the asset manager I notice a conversation.json which is basically keeping a structure for all the dialogues.

Bingo ! The json file is referencing the facial animation in animationSetFilePath, that should be where the lip sync on the characters is being called.

Every map has its own conversation.json file with the same name. We can see that it references a war3map.wts file where the dialogues are written.
However, I cannot find the English that it is reading from. As you can see it is missing from the asset manager, I can only find the other languages. Where is the English one ?
If the english one was there, I would have tried to build my own conversation.json and a war3map.wts and maybe hope that the game will reference it for me. But I doubt it.

So I am not entirely sure how to proceed, I haven't find any thread on this matter yet. Maybe someone has tried to do it.

Thank you!

 

[shinzaku]

I've definitely been interested in this as well! I didn't get as far as you - thanks for sharing!

 

[DrTema]

I don't understand. I take first human mission. Why there is facial animation file in import directory? Does it make different face animations? And this file is only for Arthas. Is this portrait animations or unit face animations? So Uther unit not talks?

 

[Solcius123]

I don't understand. I take first human mission. Why there is facial animation file in import directory? Does it make different face animations? And this file is only for Arthas. Is this portrait animations or unit face animations? So Uther unit not talks?

I just checked that, I do not know why they imported the animset directly when they are all referenced in the conversation.json where Uther has one too. Thank you for pointing that out ! But no, it's not only for Arthas in general, every unit has facial animation that are getting it from the conversation.json

 

[Solcius123]

I made some progress today. I have found the English strings ! All I had to do was to File > Export Strings...

I called it war3map.wts as I would think it should be named but it does not provide a name when I export it. So am I not sure of how the game is still referencing it.

I then tried to import the strings into a new map with import strings. But when I export back the file, here's what it shows.

It unfortunately only imported the first two strings, rest is blank, I believe that it is the state of the strings then.

But I tried to change the dialogue ID to 2 in the conversation.json that I have imported to see if it would read that string but no success, the text doesn't show up in game...

 

[Solcius123]

Still on that interlude 05 of the human campaign. I managed to change texts in game by editing the string .wts file and reimporting it back, the process is fairly simple : Edit the .wts > Importing it to the map, the name of the .wts does not matter > SAVING (or changes won't appear)

Then the text is indeed edited. But the sad part is that any modification on the map makes the facial animations / lip sync not work anymore. Even adding a tree to the map and saving, when I test back, the facial animations are gone...

*Sigh*

 

[Solcius123]

Hello.

I made some progress yesterday.

I inspected the human01 campaign which is the first human mission of ROC. As DrTema pointed out, we have the Arthas.animset directly imported in the asset manager :

Why ? No idea, but the results are interesting, because previously I would edit anything in the map, save it, run it and the lip sync / facial animations were completely gone. It looks like they "built" the maps using a special tool that would render the animations and any modification demands a rebuild. That is how I see it.

But this time, since the .animset is in the asset manager, guess what ! The lip sync of Arthas stays intact, throughout the cinematic ! Oh I was impressed by that.

I then ran some test, I would delete the cinematic tag, lip sync still there, it's really just to have the walk animation.
I would also replace the function Play Dialogue From Speaker by the good old Transmission From Unit and it works! So it is not reliant on that specific function either.

As long as I use the Arthas dialogue lines it works, so it is definitely dependent on that. Using a Uther voice line didn't work.
So I used the feature Replace Internal Sound in the Sound Editor, to replace one of Arthas's line by another sound file and it still works ! The file is still referenced and it doesn't matter what audio it is.

As I was curious I also deleted the .animset and as expected the lip sync is gone. But what if I reimport it ? It works again !

With that in mind I started to look for the Uther.animset in the game files using a CascViewer and I believe it is there.

I imported the .animset and changed the path to FacialAnimation/Uther.animset but that did not work unfortunately.

So then I thought, what if I import the Arthas.animset that proved itself to function into another campaign map ? Like the second one for example, I opened the human02 ROC campaign file, save it, run it, the facial animations are gone like expected. So I import the .animset with the proper file path for Arthas and... no... Did not work, the link is not made. This .animset working on the first map does not work on others... Sad.

So no way that it works on a new test map as well I'm afraid. It looks like there's a link already made with the .animset in the first mission. But if you import it on other maps, the game doesn't pick it up.

 

[InsaneMonster]

The way they implemented lip sync looks like a huge hack from what you report. I believe the code behind both the World Editor and the game is in a very bad state!

This is very sad since if we can not keep lip sync even in the same map when editing something else entirely we cannot even hope to reforge this... messforged game

 

[Solcius123]

InsaneMonster, sad indeed ! Looks like those features were rushed and not implemented properly with some hardcoded functions.

I have found something odd but intriguing :

When you input a condition as a bool comparison if you scroll all the way up you can find those 3 Sound - Set Facial Animation Label. I don't understand why it is a boolean comparison when it is setting something up and why 3 ? Darn, they really rushed this I think, having 3 exact function is very clumsy from them, and to me they are in the wrong place.

You can load a sound and input a string. I tried to mess around with it to see if it would attach what I write to a sound file so I can use the Cinematic - Play Dialog From Speaker but no success. Also tried to input the "animationLabel" from the conversation.json.

I haven't find any trace of that "condition" in the campaign files yet.

 

[InsaneMonster]

In Starcraft II Galaxy Editor, each unit portrait (or model, in the case of the cinematic HD ones) are compatible with a set of FaceFX files you can find in the Data Editor. These files are usually associated with the lip-sync of their cinematics.

I used that to make my transmission tool in Starcraft II (to have, like Warcraft III, a default talking animation). I checked some maps and I wasn't able to find the equivalent data in Warcraft III Reforged. It seems really they hardcoded something because I'm sure it works the exact same way.

PS: It could return a boolean True or False depending on whether or not the assigned facial animation is compatible (the right FaceFX file), but I'm just speculating...

 

[Solcius123]

Yes InsaneMonster, definitely hard coded. As I was dying to know, I opened a ticket on Blizzard's website and asked them if facial animation is a feature that map creators can work with.

I received this reply :

So that should be how far we can go for now I'm afraid.

 

[MasterBlaster]

Okay, but is there at least some generic/default "speaking" animation (with no lip sync) that people can use? Or are Reforged players stuck with non-animated portraits?

P.S. I'm sorry if you already explained it here, I'm just a bit confused

 

[InsaneMonster]

Yes InsaneMonster, definitely hard coded. As I was dying to know, I opened a ticket on Blizzard's website and asked them if facial animation is a feature that map creators can work with.

I received this reply :

So that should be how far we can go for now I'm afraid.

Terrible... I am really not sure it will ever be from that statement...

Okay, but is there at least some generic/default "speaking" animation (with no lip sync) that people can use? Or are Reforged players stuck with non-animated portraits?

P.S. I'm sorry if you already explained it here, I'm just a bit confused

If it is like Starcraft 2 not at all. A generic talking animation would be very very bad looking on realistic portraits, not to mention the amount of bones required to animate it properly.

This is soooo bad I actually bought reforged to make custom campaigns and now I'm starting to believe it's better to go back to Starcraft 2...

 

[MasterBlaster]

A generic talking animation would be very very bad looking on realistic portraits

I say it'd still be much better than having a static portrait - even if it was some of their already existing portrait speaking animations just playing in a loop.

 

[InsaneMonster]

It'd still be better than having a static portrait - even if it was one of their already existing lip sync animations playing in a loop.

There is a huge difference between a generic animation and a lip sync in a loop. If you loop it properly it can look really good.

As proof:

you can see the marine is using always the same lip sync, but arranged in a way it is not immediately noticeable, the same goes for the adjustant, which even talks a bit longer

Edit: formatting and grammar

 

[MasterBlaster]

There is a huge difference between a generic animation and a lip sync in a loop. If you loop it properly can look really good.

I'm not an animator, so I'm going to have to trust you on this one - all I'm saying is that if there's no option to have the portraits imitate talking in cinematics like they did in the original game then to me it sucks a lot.

 

[InsaneMonster]

Hey there, I believe to have solved the lip sync problem.

From my research, it appears there is a way to link in any map, for any character, a certain animset for that character (it has to be compatible with the model of course) with any sound or any text. I'll post a tutorial right before my next update of my campaign, which as of now can make fully use of lip synced characters

 

[DrTema]

Hey there, I believe to have solved the lip sync problem.

From my research, it appears there is a way to link in any map, for any character, a certain animset for that character (it has to be compatible with the model of course) with any sound or any text. I'll post a tutorial right before my next update of my campaign, which as of now can make fully use of lip synced characters

I would like not to miss your tutorial

 

[ShadiHD]

Hey there, I believe to have solved the lip sync problem.

From my research, it appears there is a way to link in any map, for any character, a certain animset for that character (it has to be compatible with the model of course) with any sound or any text. I'll post a tutorial right before my next update of my campaign, which as of now can make fully use of lip synced characters

@Tamplier777 read this

 

[Tamplier]

@InsaneMonster hello there. Any news about your solution?

 

[InsaneMonster]

@InsaneMonster hello there. Any news about your solution?

On Wednesday I'll post a both a written tutorial on Hive and a video tutorial on my YouTube channel. I'll also update my campaign with a completely new map (an interlude) and an updated version of the already released maps, all now featuring no hero glow and lip-sync. So... stay tuned in 2 days!

 

[Tamplier]

That is Awesome!

 

[InsaneMonster]

The tutorial about lip-sync is there:
[Misc] - How to: Warcraft 3 Reforged Animated Portraits

There is also a video version on YouTube:

Beside that, I've updated my campaign and now all the maps have lip-sync. You can find it at Warcraft 3 Re-Reforged Prologue Campaign