1. Updated Resource Submission Rules: All model & skin resource submissions must now include an in-game screenshot. This is to help speed up the moderation process and to show how the model and/or texture looks like from the in-game camera.
    Dismiss Notice
  2. DID YOU KNOW - That you can unlock new rank icons by posting on the forums or winning contests? Click here to customize your rank or read our User Rank Policy to see a list of ranks that you can unlock. Have you won a contest and still havn't received your rank award? Then please contact the administration.
    Dismiss Notice
  3. The poll for Hive's 12th Concept Art Contest is up! Go cast your vote for your favourite genie!
    Dismiss Notice
  4. Travel to distant realms and encounter scenes unknown to the common folk. The Greatest of Adventures is upon us with the 8th Cinematic Contest. Join in on a fun ride.
    Dismiss Notice
  5. The 18th Icon Contest is ON! Choose any ingame unit and give him/her Hero abilities. Good luck to all.
    Dismiss Notice
  6. Contestants are to create a scene set in the Stone Age. Come and see what you can come up with. We wish you the best of luck!
    Dismiss Notice
  7. Colour outside the lines! Techtree Contest #13 is a go. The contest is optionally paired.
    Dismiss Notice
  8. Greetings cerebrates, our Swarm needs new spawners that will have numerous children. Join the HIVE's 31st Modeling Contest - Spawners and Spawned! The contest is optionally paired.
    Dismiss Notice
  9. Check out the Staff job openings thread.
    Dismiss Notice
Dismiss Notice
60,000 passwords have been reset on July 8, 2019. If you cannot login, read this.

Hosting a map on GitHub - is it even possible?

Discussion in 'The Lab' started by Cheezeman, Apr 8, 2018.

Tags:
  1. Cheezeman

    Cheezeman

    Joined:
    Aug 19, 2008
    Messages:
    437
    Resources:
    2
    Spells:
    1
    Tutorials:
    1
    Resources:
    2
    Has anyone attempted to host their map as a git repo on (example) GitHub? Some version control would be nice but I don't see how you would do it.

    .w3x is just a binary file so hosting that alone would be pointless.

    Extracting all files from .w3x using MPQMaster would let us view some of the files as plain text, notably war3map.j with all the compiled JASS code. But most other files are binary, and war3map.j does not contain any information from 3rd party extensions like vJass.

    Has anyone attempted this before? Does anyone have any ideas? Does anyone know how to view the other binaries from the .w3x MPQ archive?
     
  2. Chaosy

    Chaosy

    Joined:
    Jun 9, 2011
    Messages:
    10,608
    Resources:
    18
    Maps:
    1
    Spells:
    11
    Tutorials:
    6
    Resources:
    18
    I am somewhat confident it works.
    Since in the end the binary code is different each time you change something in a map.
     
  3. Cheezeman

    Cheezeman

    Joined:
    Aug 19, 2008
    Messages:
    437
    Resources:
    2
    Spells:
    1
    Tutorials:
    1
    Resources:
    2
    Yes but I don't know how to read them. I found a tutorial on the file format for some of these, like the .w3e file, but when I open the one from my map in a text or hex editor I only see nonsense hex values. How do I read these files?

    Edit: Apparently they're supposed to look like gibberish. The tutorial I linked shows how to understand the gibberish.
     
    Last edited: Apr 8, 2018
  4. mori

    mori

    Joined:
    Jun 13, 2016
    Messages:
    336
    Resources:
    2
    Spells:
    1
    Tutorials:
    1
    Resources:
    2
    I remember seeing some chinese guy posting a tool that converted a .w3x constitutent parts (e.g. w3e, w3o, etc. internal files) into text-based formats that played nicely with github. It sounds like a viable solution.

    Would multiplayer mapmaking be something that Blizzard will ever add?

    ^ There's stuff in this thread.

    There's also the option of externalizing triggers / object data as well. In one my personal projects, I just used vJASS imports for triggers and GMSI scripts for object data generation. It's a... rather quirky setup, and a lot of can (and will) go wrong, but it's good for version control. That just leaves terrain.

    Here's my repo: SamuelMoriarty/Yet-Another-RolePlay

    You can also use Wurst, since that combines externalized triggers with (somewhat still clumsy) object data generation.

    EDIT:

    Otherwise, I don't see much point in using Git for versioning map files, since you lose pretty much all of the benefits that Git gives you, since you can't really view changes, do merges, or any other stuff except for just slamming a commit message to a certain version of the map. Also, if you have large map sizes, git -will- slow down to a crawl eventually, and will take a long time to do pretty much anything. Overall, not worth it.
     
    Last edited: Apr 8, 2018
  5. Flux

    Flux

    Joined:
    Feb 6, 2014
    Messages:
    2,334
    Resources:
    28
    Maps:
    1
    Spells:
    19
    Tutorials:
    2
    JASS:
    6
    Resources:
    28
    You can code your project in Wurst so that each commit keeps track of all code changes you've made because you put the code in an external file unlike coding directly in World Editor/JNGP/WEX.

    If you want to do it in vJASS, you can also put the scripts in external file and explicitly put inside JNGP/WEX to include those files. Example for you to better understand: Protolord/BotD
     
  6. Cheezeman

    Cheezeman

    Joined:
    Aug 19, 2008
    Messages:
    437
    Resources:
    2
    Spells:
    1
    Tutorials:
    1
    Resources:
    2
    I made my own "rather quirky setup" in the 24 hours since my last post. Basically I extract all files from the .w3x using a MPQ archive manager. You'd think this wouldn't work because most files (except war3map.j) in there are detected as binary files by any text editor, but except for some null-chars it's all encoded in ASCII (except the .blp images). This means that telling git to treat them as text when calling 'git diff' by modifying the .gitattributes file will actually work. Check it out:

    [​IMG]

    You can see that I added a comment after the globals block declaration in the .wct file since my last commit. Pretty cool! That weird string you see before globals declaration is the .wct file header with some info for WE.

    Unfortunately, telling git to treat all files as text when calling 'git diff' only works for console commands. GitHub and any git web hosting service I know of (BitBucket, GitLabs) will ignore the .gitattributes and still treat them as binary.

    Thanks for sharing your GitHub repo. I'll have a look at it later to see if I can find a better solution. Here's my .gitattributes if you're interested:
    Code (Text):
    *.w3e diff
    *.shd diff
    *.tga diff
    *.doo diff
    *.w3i diff
    *.w3u diff
    *.wtg diff
    *.w3c diff
    *.w3r diff
    *.w3s diff
    *.wct diff
    *.imp diff
    *.wai diff
    Edit: I found that Chinese guy's workflow. Here's a Google Translate of their GitHub repo. Their solution would be a little bit more readable than mine in the end (especially for the .doo and .w3u files) but not much different in the end.
     
    Last edited: Apr 9, 2018
  7. GhostWolf

    GhostWolf

    Joined:
    Jul 29, 2007
    Messages:
    4,836
    Resources:
    2
    Tools:
    1
    Tutorials:
    1
    Resources:
    2
    "but except for some null-chars it's all encoded in ASCII (except the .blp images)"
    Binary data is just a chunk of memory, and text data is just a chunk of memory that typically doesn't have NULL or other unprintable characters in its bytes. You can open any file in existence in any text editor, in the same way you can open any file in existence in any hex editor.
    Indeed, the only reason opening a text file in an hex editor might look more intuitive is because hex editors tend to show both the byte values and the ASCII representation of them (which might have nothing to do with the actual data layout).
    I can't understand why you'd think it won't look like gibrish for you, did you think an arbitrary chunk of memory will have comments surrounding it an telling you what each byte is? these are file formats someone designed, and they have specifications you need to know and follow, if you want the data to mean anything to you.

    Regardless, this seems like a lot of work to get the same result as "Save As" in WE. What does this actually give you, given that the only files that are text files and thus will have a meaningful diff (meaningful for YOU) are the wct and j files, and even the j doesn't really help you.
     
  8. Waffle

    Waffle

    Joined:
    Jul 30, 2013
    Messages:
    271
    Resources:
    0
    Resources:
    0
    it would be really nice to have git be able to tell you what changed and when on a commit by commit basis, thus having the component files in scm sounds nice.
    also it raises the possibility that if 2 people work on a file then the work could be merged and you would not have to pick one and discard the other entirely.
     
  9. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,033
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    Do you modify .j, .w3a, .w3i etc. directly? Most do not or only via WE. But the .j for example is a compilation of .wtg and .wct together with some other information in WE terms. And this is what technologies like wurst like to do as well: separate the source from the output. If you understand the language of the source and can trust the compiler, you only need to consider the source and maybe it would not be required to upload the outcome at all if the target platform has an interpreter/JIT compiler or the final synthesis without the intermediary parts could be enough.

    Having said that, it is not like the existing middleware would be perfect or entire at this point in time and I find myself still often looking at the generated, low-level output, re-analyzing what I have fabricated or also maps of others. This is why, looking at my wc3libs library, it raised the question/idea of putting thin layers onto the wc3 formats like JSON or XML to have it as readable and easily-adjustable text.

    edit: Or better yet, a more sophisticated approach: Depending on the type of data, picking different representations may be preferable. I do not assume looking at a W3E tiles as a list of nodes with numbers for each attribute really makes sense as its semantics is a matrix and e.g. the tile height follows a metric -> you may want to identify the differences and compare it. It could be presented by multiple (loss-less) pictures as @Dr Super Good did before, or as one picture with multiple layers/channels, or as ASCII art. Aside from the last option, this would call for a custom difference-check tool of course but this should still be feasible. Maybe the header of W3E should be extracted completely, you can reorganize its tile matrix with .wpm, .shd etc. We just need to think and make a standard what would be an appropriate representation of a wc3 map.

    Now, wc3 maps may consist of a lot of files. You may think the ones listed in W3M and W3X Files Format were sufficient but those are mostly but to introduce the different formats. There are many more slks, each with their own semantics, you can overwrite scripts/common.j or the layout files (.fdf) (for the game, not in map though), .txt files like the profile in Units\ or the gameplay constants and interface stuff with basically arbitrary fields. The raw resources (overshadowed/new) without special meaning should go to a distinctive scope. But there is a lot of inbuilt interface media like you have custom UI bars. I found it always very inconvenient having to precisely know the (partly cryptic) paths or keys that must be overwritten for the desired effect, so I think collecting that intelligence and providing something better for them would be keen, too. Or maybe an UI designer tool with preview?

    The main problem would be to model dependencies. Like the .w3i (map and player information) is relatively autonomous but the .doo (object placements) files contain spatial (in reference to W3E coordinates) but also non-spatial widget data. Gathering everything in one file would kill readability and be wasteful, distributing it means the necessity of linkages and consistency contemplations. The UI stuff mentioned before is usually a singular, non-interweaved task. You won't dynamically change the day/night indicator based on the amount of your jass variables. Thus it is practical to have UI modifications as standalone. You may however want to alter your the map script leveraging object placements like a unit placement definition is converted to a 'call CreateUnit(...)' line. The WE does this as well. That specific scenario could perhaps be abstracted because it is a standard mechanism but custom and a number of the hacky ways we use in wc3 mapmaking like exporting selected object editor data to the map script would be different. Nah, we cannot account for everything, better if this intermediate representation is still fairly directly modeling the wc3 possibilities and explicit.

    The first step should always be to eliminate the pure boilerplate. You could also have multiple interstages, build yourself a delivery chain. The first repository has the source code, from that you translate to the first intermediate representation, from that to the second, ..., always descending on a lower level until you reach some sink(s). You can even fork it off, enabling different paths, adding an endpoint here or there to add/extract required information, recurrency. Go wc3 DevOps.
     
    Last edited: Apr 13, 2018
  10. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,470
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    Yes and no it is possible. Yes one can do it by breaking the map out of a MPQ archive into a file system. One can then modify the files with third party tools. GIT can still track changes like normal.

    No it cannot be done sensibly as GIT does not know how to track the changes of the binary files. Instead it will say the whole thing has changed each time.

    Outside of writing specialist tools to track the changes in the binary files (SVN supports this, not sure about GIT), one would have to write a binary to text translator. In the case of the pathing map and terrain mesh files one could translate them into a textual metadata file and a series of layer images which GIT could then track.
     
  11. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,033
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    Binary files are typically more rigidly structured, employing specialized encoding techniques, laid out in different patches, doing without delimiters, being sequential. Thus it is often impossible or more difficult to just eye and map meaning to some arbitrary middle parts, not just for the machine but also for human. It has to be read from the beginning or in its entirety. For a text file we simply say that for instance it's overall encoded in UTF-8, apply a global translator and thereby obtain a much bigger character set, which enables better distinction. I argue that the lack of such a baseline of rules (like the concept of lines is not given in binary files) makes it a more troublesome for a difference tool to identify insertions, deletions and modifications, which only deals in bytes. It would require more logic. But I guess for non-compressed wc3 formats the raw analysis could still be enough. When there is no compression that transforms the whole block subsequently, format specifications are seldom to do crazy stuff like jumping around or scattering closely-related properties.
     
    Last edited: Apr 14, 2018
  12. actboy168

    actboy168

    Joined:
    May 1, 2012
    Messages:
    95
    Resources:
    1
    Tools:
    1
    Resources:
    1
    Yes.

    Here is my map, actboy168/MoeHero. It uses sumneko/w3x2lni to do this.

    Usually I don't need to use WE to edit the map, just use vscode or other text editor.I like this way of working, because WE‘s save map is too slow.
     
  13. Waffle

    Waffle

    Joined:
    Jul 30, 2013
    Messages:
    271
    Resources:
    0
    Resources:
    0
    Sounds rly enticing :)
     
  14. Cheezeman

    Cheezeman

    Joined:
    Aug 19, 2008
    Messages:
    437
    Resources:
    2
    Spells:
    1
    Tutorials:
    1
    Resources:
    2
    Thank you, I found it in an earlier post you made :)

    Is it supposed to look like this?

    [​IMG]

    I can't find any English documentation, all the links in your repo is broken.

    A question too: can this convert back from Lni to w3x as well?

    Thank you for this lengthy post.

    No, I never edit the .j, .wct and .wtg files directly. I read them when using '$ git diff' to view version history, but when I want to write to them I import the files into my .w3x map and then use WE.

    I've thought a lot about making my own .w3x to text converter (the middleware you were talking about), and I think it's possible. Case in point Sumneko/W3x2lni. But I also want to convert it back from text to .w3x again, otherwise it would be pointless. But I don't want to go overboard with the individual file representation. The ideas you're suggesting are cool and all but coming up with a good data structure representation for .w3e and then implementing it is too much work. For my own converter I'll be focusing on getting the individual file headers readable and then just printing the rest of the data as-is.

    There are a lot of problems to solve for myself before doing it however. Most of it is just understanding how the different file formats work. Stuff like how MPQ archives are de-/compressed. How to scan for files in them when uncompressed. How I can be sure of a lossless conversion between .w3x and readable text. Right now I'm working on my map but I'm pretty excited to solve these problems :)
     
  15. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,033
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    Well there are mpq editors. I would at least remove that layer.
     
  16. Hotwer

    Hotwer

    Joined:
    Mar 10, 2013
    Messages:
    370
    Resources:
    0
    Resources:
    0
    So you just open the map to deal with with terrain? And what about the Object Editor?
     
  17. Cheezeman

    Cheezeman

    Joined:
    Aug 19, 2008
    Messages:
    437
    Resources:
    2
    Spells:
    1
    Tutorials:
    1
    Resources:
    2
    True, but it would be a nice one-button solution.

    Same way.
     
  18. Hotwer

    Hotwer

    Joined:
    Mar 10, 2013
    Messages:
    370
    Resources:
    0
    Resources:
    0
    Same way as what? Through editor? Or VSCode instead?
     
  19. Cheezeman

    Cheezeman

    Joined:
    Aug 19, 2008
    Messages:
    437
    Resources:
    2
    Spells:
    1
    Tutorials:
    1
    Resources:
    2
    Same way as you just said in your message that I was quoting. To deal with the object editor I open the map.
     
  20. Hotwer

    Hotwer

    Joined:
    Mar 10, 2013
    Messages:
    370
    Resources:
    0
    Resources:
    0
    Doens't TESH solves your issues with coding?