• Listen to a special audio message from Bill Roper to the Hive Workshop community (Bill is a former Vice President of Blizzard Entertainment, Producer, Designer, Musician, Voice Actor) 🔗Click here to hear his message!
  • Read Evilhog's interview with Gregory Alper, the original composer of the music for WarCraft: Orcs & Humans 🔗Click here to read the full interview.

version control for warcraft maps

Status
Not open for further replies.
Level 6
Joined
Jul 30, 2013
Messages
282
Hey

Does anybody done it?
working alone or as team?

how to .. ehm.. extract as much as possible all the test based assets from the game so they would be elegible for automated merge?

any ideas welcome, I am prepared to write some code to make it work!
 
You can pull it off for code by using //! import directives in the triggers instead of actually having any code there. You'd have all the code on GitHub. The //! import directives would import the code directly from your GitHub folder.

Good enough?

You'll need to manually create triggers, but it'll work nicely for modifying existing triggers~
 
Last edited:
Level 6
Joined
Jul 30, 2013
Messages
282
triggers we can figure out i think.. (!import can be used recursively afterall :) )

objects?
terrain?
spells?
models?
anything i might have missed?
 

Ralle

Owner
Level 79
Joined
Oct 6, 2004
Messages
10,183
I have often thought about this. I think this is the way I would do it:
Have a tool that every time you save, it exports all files from the map MPQ into another folder, allowing you to manage them in GIT. When you pull from a repository, it regenerates the map MPQ. It would require World Editor to be closed, but that should not be too much of a deal breaker.
 
Level 6
Joined
Jul 30, 2013
Messages
282
Well yeah, the basic concept is clear...

but then you run in to issues such as.. there are a ton of binary files in there and ideally you would want to store some representation that git can merge seemlessly aka text based..

and for most components i have very little if any idea of how they work, and i definitely wouldn't fancy writing everything from scrach..

i know there is storm.dll that i could leverage to extract the mpq..
but from there on is a whole bunch of custom binary files..

i'm wondering perhaps anybody has a script i could reuse..
or some insight on any of the file formats? (object data especially, that will definitely conflict!)
 
Level 6
Joined
Jul 30, 2013
Messages
282
hmm..
yes, json would be awesome!

(i don't have anything against xml.. but json tends to be just slightly easier to work with in my exp)

images wouldn't be very useful tho, git fails with binary formats..
but we can turn that in to some elegant json format too so it could work..

@Ralle Would you be so kind as to share those precious links? :)
if i can come up with some working setup ill be sure to share :)
 
Level 6
Joined
Jul 30, 2013
Messages
282
Subversion doesn't have a problem with images?

Could that possibly be because SVN never actually merges stuff? (i hear horror stories every time people attempt to actually do that..)

Cuz if you don't need to merge, then sure git has no issue either :p

But i DO need to merge.. i need to merge twice-daily..
and to do it manually would just be so painful... :'*(

this is why I would like to bring everything down to text-based formats, because they work well with automatic merging scripts and if there is an issue you can open it up and resolve the issue manually because json/xml/yaml/etc.. are human readable, unlike images or other binary formats.
 
Last edited:

Dr Super Good

Spell Reviewer
Level 64
Joined
Jan 18, 2005
Messages
27,259
Have you ever seen an image in a text format? Its like a huge great big mess. It also does not merge well since 2D changes do not reflect well on 1D text files.

I guess what could be done is to break maps into 32*32 tiles which are themselves separate images. That way unless there is a conflict (two people editing the same 32*32 areas) then it should be easy to merge work. 32*32 is chosen since that is the minimum dimension change allowed in WE?

This would need some intermediate tool that merges and splits 32*32 areas.
 
Level 6
Joined
Jul 30, 2013
Messages
282
You have some good points. But that would essentially be inventing our own custom (+ binary) data format + i would have to teach git all over how to do the merges, would it not?*

there is something that i am reallypuzzled about...
What has 1D and 2D have to do with this(or 3D or 4D or ND)?

A 1D data structure.. (JSON) please compare..

Code:
[1,2,3,4,5,6,7,8,9]
vs
Code:
[[1,2,3],[4,5,6],[7,8,9]]

some extra grouping perhaps.. but the data is essentially the same..
now lets break this down to different lines..
Code:
[
  [1,2,3],
  [4,5,6],
  [7,8,9]
]
and we have a fairly human readable format to hold arbitrary 2d data..
you can of course break it down even more..
Code:
[
  [
    1,
    2,
    3
  ],[
    4,
    5,
    6
  ],[
    7,
    8,
    9
  ]
]
something like this would be incredibly easy to merge with line based diffs (as the only conflicts would be if the same pixel is changed) tho it does look a bit big.
so we could go for some hybrid format..

hmm there IS a lot of data .. we don't want to put too much space under it..
you make a good point.
Code:
{
  width: <insert some int her for map width>/**/,
  height: <insert some int her for map height>,
  chunks:[
"base 64 encoded date for some number of pixels on the map, the position of the date can be restored from map size , and the index of the current chunk",
"base 64 encoded date for some number of pixels on the map, the position of the date can be restored from map size , and the index of the current chunk",
"base 64 encoded date for some number of pixels on the map, the position of the date can be restored from map size , and the index of the current chunk",
"base 64 encoded date for some number of pixels on the map, the position of the date can be restored from map size , and the index of the current chunk",
  ]
}
(base 64 strings having no indent is just premature optimization, because we will have lots of them)

this should be reasonable efficient in terms of space and still retain some good properties of text files.
(possibility of manually merging some sub-chunk features)

*Technically you could say this is a also a custom data format too.. but JSON parsers are abundant, and we don't have to write them, they are usually fairly stable and well tested.
thus writing any tools to manipulate them should be tons easier.

EDIT: how well is the terrain mesh documented, can we reliable break it down? how well?
 

Dr Super Good

Spell Reviewer
Level 64
Joined
Jan 18, 2005
Messages
27,259
What has 1D and 2D have to do with this(or 3D or 4D or ND)?
The way data is managed. A small square change of say 2*2 tiles in a 256*256 map will make two sequential alterations of data that are over 1000 bytes apart.

Now convert that to text and you have it that it is two changes over 1000 characters apart.

This is fine for a 2*2 change, but enlarge that to a 100*100 change. You will get 100 changes that are over 1000 bytes apart.

Seeing what has changed might as well be impossible since there is no way a human will understand or be able to vision the changes made. Sure it may be able to merge it to some degree as it is just text.

Let us not forget that a w3e file in text form will be several megabytes big, so by itself is pretty difficult to modify.

EDIT: how well is the terrain mesh documented, can we reliable break it down? how well?
Documentation is meh but sufficient. Check out what I did with water using a java interface I wrote.


GES.jpg

WC3ScrnShot_011813_031727_01.jpg

 
Level 6
Joined
Jul 30, 2013
Messages
282
well what we want to merge is all of the data in a map....
its just that the terrain seems to be like a big picture..

any better ways to merge it?

(and any tips on how to deal with other map data are still welcome of course! )
 
Level 29
Joined
Jul 29, 2007
Messages
5,174
Use non-compressed images (BMP/TGA/?), or even simple raw pixels without any header (if you know the size).
Then it's just like cooperative image drawing, except in github.

Using a text format (e.g. JSON) for binary data isn't a good solution, it will become very big very fast.
 
Level 11
Joined
Oct 13, 2005
Messages
233
Just in case anyone is still interested, I'll give my input on the matter.

When I used git, all I really did was use git for (v)JASS files, but if you're interested in what my repo looked like, here's a link: https://github.com/xalduin/survivetheruins

The only two files of interest are the import.j file and the includes.rb file, which automatically searched the src/ directory and wrote out the proper import statements.

I once worked on a project that used SVN to manage all aspects of a project and also supported importing external object data files which would each contain a single unit, ability, etc. I don't have the source code for that tool around, but there's a tool in the NewGen pack which should allow you to import and combine external object files.

So, if you separate out the code and the individual object data, you're not left with much in the central map file. If you have a lot of resources (models, textures, images, etc), you can just leave those sitting outside the map as well and there's likely another tool available to import them (if not, it should be trivial to write a program for that). Git doesn't excel at binary data and if you're worried, you could just manage the code with git and store/manage the other files elsewhere, but you'll probably be fine with them all in the same repository.
 
Last edited:
Level 13
Joined
Mar 19, 2010
Messages
870
sure, you need the "includes.rb"? I created a testmap and added "//! import "import.j" at the top of the Trigger editor ( Filname of the map )...

then i saved the map in a testfolder. In this folder i created another folder and in this one i added a init.j where i just display a text "Test". I put the import.j in the root of this testmap folder... That's it.

I saved the map, started and i saw the Text. So for what u need the "includes.rb"???
 
Status
Not open for further replies.
Top