A new TESH Syntax Highlighter for Warcraft 3

[Gismo359]

Problems I encountered:

1. Increases map load time(in editor).
2. Increases object editor load time.
3. Causes JassHelper to freeze upon finding an error. This seems to happen even without a tesh on one of my computers, but on the other - only using either yours or the one in jngp 5d seems to cause it.
4. Sometimes JassHelper completely stops - it doesnt run on saving and no errors get shown, even if such are present. I have not encountered this bug without your tesh.
   Just had it on 5d. I think it has something to do with opening and closing a map in we, so it isnt a tesh problem.

 

[Deleted member 219079]

I suggest you make it so that Enter would input a linebreak even when there's a list active, tab could work as it is now.

 

[Wietlol]

For highlighting global variables or custom functions... I think you can do something like this:

Save all global variables that are made in this trigger + in the global variable list.
When a global variable is destroyed, you can remove it from the list.
Same goes for custom made functions.

I don't know if that is possible on the way I said or if it will be too performance effective.
Maybe I could test some stuff like it myself.
Would you mind?

 

[looking_for_help]

For highlighting global variables or custom functions... I think you can do something like this:

Save all global variables that are made in this trigger + in the global variable list.
When a global variable is destroyed, you can remove it from the list.
Same goes for custom made functions.

Its not that easy. As already said, first of all I need a method to access the text of non-selected trigger items.

Saving stuff in a list when added and removing them when deleted would be the very last (and by far the easiest) step. First I need access to the whole text. Then write a lexer that splits this text into tokens. Then write a parser that builds a syntax tree out of these tokens. Then write a semantic analyser that validates the tree. Then I can take the identifiers and add/remove them to a list.

So far I only have the lexer (which is also used to do the highlighting).

Maybe I could test some stuff like it myself.
Would you mind?

I wouldn't mind, but how are you going to do this?

 

[edo494]

hmm...I said it multiple times, I will say it again, but I dont know if you understood when I meantioned it the way I wanted to explain it.

Basically, when user loads map(this happens once per map obviously), you perform one-time IO to read the whole map script's file, maybe together with the trigger info file, to actually have idea what goes there, and voala you have yourself full map script , and then you only perform changes to it depending on user inputs through the trigger editor

 

[looking_for_help]

hmm...I said it multiple times, I will say it again, but I dont know if you understood when I meantioned it the way I wanted to explain it.

Basically, when user loads map(this happens once per map obviously), you perform one-time IO to read the whole map script's file, maybe together with the trigger info file, to actually have idea what goes there, and voala you have yourself full map script , and then you only perform changes to it depending on user inputs through the trigger editor

There are two problems with this approach: First, inputwar3map.j is only updated after saving, not when the map is opened. And second, its just one big file and has no mapping to an actual trigger treeview item. But that would be required so that I can add meta-data to all treeview items.

Or did I missunderstand something here?

 

[edo494]

well there are 3 ways how you can close/open map.

Change map in the opened map selection under Window, there is no possible difference in trigger data, so you dont need to read trigger-related data.

2. You actually close the map and open it
2.1 you saved before closing, in which case the data is overriden in the disk, so when you open you have correct data loaded from disk
2.2 you didnt save before closing, in which case you still load correct data from disk.


To the "separating code to triggers" problem:

With Xonok we found that the editor uses both .wtg and .wtc to build the Trigger data.
The editor will first pull the .wtg data, seeing the comments to put above the code blocks as well as the names(and therefore the order of folders as well as the trigger files themselves).

Then it will look into .wtc and split it appropriately.

The separator seems to be NUL byte[1] SOH NUL NUL in notepad++.

[1]: the byte seems random to me, as you see in the picture, but it may have some higher purpose we didnt find

notepad++ image

With this knowledge you should be able to actually simulate the world editor, and know exactly which piece of code maps to which trigger file.

The .wtg file seems to be a little bit more retarded when it comes to delimiters tho(dont know if you successfully parsed that before)

 

[looking_for_help]

That would be one possibility, but to do this I would need to read the map MPQs on map init. And unfortunatly I have no idea about MPQs and how to manipulate them

 

[edo494]

I have written C++ class to read and write files into mpq as well as some random stuff I considered worthy and it is pretty easy to use, you just create new object, open archive(one member function call), give it a listfile(one member function call)(currently doesnt support the in-built listfile, because there is possibility that it is not there, so I didnt implement that), load 2 files into std::strings(two times the same member function call).

It also has somewhat detailed error messages.

So if you are intrested, I can give it to you no problem

 

[looking_for_help]

I have written C++ class to read and write files into mpq as well as some random stuff I considered worthy and it is pretty easy to use, you just create new object, open archive(one member function call), give it a listfile(one member function call)(currently doesnt support the in-built listfile, because there is possibility that it is not there, so I didnt implement that), load 2 files into std::strings(two times the same member function call).

It also has somewhat detailed error messages.

So if you are intrested, I can give it to you no problem

Ok, that sounds great, I'm highly interested

 

[edo494]

http://www.hiveworkshop.com/forums/pastebin.php?id=2vw2y5

Here is a zip.

However, you will have to link against SFmpq.lib inside SFmpqapi folder, and I guess you know how static building works, but if you dont want it, remove the "SFMpqstatic.h" from includes in the mpqreader.h

The headers are also inside the SFmpqapi folder.

If you are using something with code highlighter the API should be clear enough, if not, let me know

 

[looking_for_help]

Ok, many thanks, I will take a look at this.

 

[Wietlol]

II wouldn't mind, but how are you going to do this?

I suppose I just change TESH and see what it does... Ofcourse I won't claim any rights and stuff etc etc etc.
But how do you make/edit TESH?
I never have made a tool for something similar to this.

btw is there anything in TESH with the combination of UHB?

 

[edo494]

well, you need the source code, and I dont know if LFH has made it public, but the old TESH has source code somewhere

 

[looking_for_help]

I suppose I just change TESH and see what it does... Ofcourse I won't claim any rights and stuff etc etc etc.
But how do you make/edit TESH?

That was basically my question, how you are supposed to edit it? Because the source code is not public.

You can of course try to edit the old TESH, which is open source. But you will need knowledge of the Delphi programming language.

btw is there anything in TESH with the combination of UHB?

What is UHB?

 

[Wietlol]

What is UHB?

Well I am not quite sure but I think that TESH crashes the world editor when you type some syntax.
Because I tried to make a system with the shortcut UHB but it crashes within 5 minutes.
No long code, just a new map and then boom once I wrote that.

But when I go to another map and edit that one, everything is fine.
And when I revert TESH back to 0.7 then the crash doesn't come.
(I doubt that the crash comes from a textfile where I saved my highlighting styles in.)

 

[looking_for_help]

Well I am not quite sure but I think that TESH crashes the world editor when you type some syntax.
Because I tried to make a system with the shortcut UHB but it crashes within 5 minutes.
No long code, just a new map and then boom once I wrote that.

Which syntax did you exactly type? And what is "the shortcut UHB"? I don't know such a shortcut.

(I doubt that the crash comes from a textfile where I saved my highlighting styles in.)

The new TESH doesn't perform any file IO, so yes, that cannot be the cause.

 

[looking_for_help]

http://www.hiveworkshop.com/forums/pastebin.php?id=2vw2y5

Here is a zip.

Alright, finally found the time to try this out. Took me just 30 min to get it working, and it works like a charm. Very nice done!

Thanks again.

 

[edo494]

no problem

 

[Cokemonkey11]

How's it going on this? Do you have a bug tracker somewhere?

 

[looking_for_help]

How's it going on this? Do you have a bug tracker somewhere?

Hi,

its going quite well. I decided to rewrite the code base a few weeks ago (first attempt always sucks^^) and there is still a lot to do to reach the state of the current available version (Therefore a bugtracker wouldn't make too much sense at the moment).

I also have setup a concurrent parser (at the moment only for functions and globals) which is already working fine and doesn't touch visible performance.

Im still having problems though to decipher the trigger seperators in the mpq files... It seems those are flags for the triggers state, but I didn't have the time yet to investigate this further...

 

[Cokemonkey11]

http://www.joelonsoftware.com/articles/fog0000000069.html

This is one of the most common causes of project failure.. I wish you luck

 

[Nestharus]

I rewrite code from scratch all the time : D. Usually my own code : p. This has resulted in breathtaking architectures. I rewrote the symbol table for xJASS over and over again for 3 months. It is now a work of art.

Rewrites are ok bro, and I'm excited to see the rewrite of this ;D

 

[Ruke]

open source, open source, cmon people, open source, open source

 

[Cokemonkey11]

You re-wrote the same thing over and over again for 3 months? That's hilarious.

Rewrites don't always cause failures - it's just worth being aware that they are a common source of problems.

 

[edo494]

is it possible for you to change the behaviour of the auto-suggestion box, so that it is more like the IntelliSense one?

By that I mean, when I type, it will show the box, but not "highlight" any option, until I hit up or down, because this would fix the problem Im having with this TESH(the problem sometimes happened with the old one too), where if I type nothing as function return type, I have to hit enter twice.

Also maybe option to disable the text highlightning when I just move the cursor?

 

[looking_for_help]

is it possible for you to change the behaviour of the auto-suggestion box, so that it is more like the IntelliSense one?

By that I mean, when I type, it will show the box, but not "highlight" any option, until I hit up or down, because this would fix the problem Im having with this TESH(the problem sometimes happened with the old one too), where if I type nothing as function return type, I have to hit enter twice.

Yes that should be possible although scintilla is unfortunatly quite unflexible when it comes to customization of the autocomplete window... It doesn't even provide API to get the position of that window

Also maybe option to disable the text highlightning when I just move the cursor?

You mean the shadow behind identifiers that are selected with the mouse? Yes, will be optional too

Rewrites don't always cause failures - it's just worth being aware that they are a common source of problems.

Thats true, but I basically had no other option
The article linked is interessting but most of the things don't apply here since its a one-man project... In a big professional project with multiple developers it would be a bad idea, thats true.

 

[edo494]

found new bug, after saving map, the ´ button doesnt respond(when on english keyboard it is = ), I would expect it to type = but nothing happens, and I have to alt-tab out of the window(lose focus), then regain back to make ´ work again

 

[PurgeandFire]

Any news on this? Just curious.

 

[looking_for_help]

Any news on this? Just curious.

Yes, there were no updates for quite some time now, unfortunatly.

I can give a little overview about the current state:

• The new lexer is basically finished and allows now URL highlighting in strings and comments as well as correct highlighting of nested blockcomments (which was quite hard to do this efficiently)
• The thread managment is finished and makes it possible to perform expensive operations in a background thread (like a possible parser in the future)
• Native keywords are now managed in an external file and can be easily extended/changed if neccessary.
• Communication with the editor is improved and more performant

However, there is still a lot to do. Until now no menus (Search/Replace, Function list, etc), no folding and no brace highlighting.

When those things are finished, I will upload the new version.

 

[pred1980]

 

[Flux]

If possible, add a feature where the custom function will have color coding (like SharpCraft natives) edited within JNGP without editing the file in the JNGP->Tesh directory. Just a suggestion.

 

[Bribe]

LFH, is it possible for TESH to pull from object editor data so that you can select the rawcode from a list, or identify where the rawcode points to?

 

[looking_for_help]

If possible, add a feature where the custom function will have color coding (like SharpCraft natives) edited within JNGP without editing the file in the JNGP->Tesh directory. Just a suggestion.

I'm working on this, but its not easy, unfortunatly. We'll see what the next version brings

LFH, is it possible for TESH to pull from object editor data so that you can select the rawcode from a list, or identify where the rawcode points to?

Well you can already select rawcodes from the autocomplete list by hitting CTRL-Space when writing inside a character (or string stuff when in a string).

But I guess you mean doing this dynamically with user defined raw codes? This is possible too using a mpq reader, I just didn't have the time until now to work this out... (currently I'm reworking the communication with the editor, because that was really slow in both the old tesh as well as in the new tesh. The new technique allows documents of almost arbitrary length without lagging).

 

[looking_for_help]

LFH, is it possible for TESH to pull from object editor data so that you can select the rawcode from a list, or identify where the rawcode points to?

I did some further tests on this because I really wanted to include user placed units to the autocomplete list.

Unfortunatly it seems that the WE only generates global variables like gg_unit_hfoo_0000 if unit was selected by a GUI trigger.

Even worse, even if just a comment like

// gg_unit_hfoo_0000

is included to the map file, it will make the map fail to compile, because no global variable is generated, but the asignment in CreateAllUnits is performed as if the global would have been declared. It seems the compiler just searches if the unit identifier exists in the map script (even within a comment) and uses it in the unit creation function but doesn't generate the corresponding global variable.

Even if I could access all unit identifiers (which I can't, because they seem to be only generated by GUI triggers) it wouldn't help to include them to the autocomplete list because they are not valid until used by a GUI trigger...

I haven't checked yet whether WE or vJass is the culprit but thats quite bad.

 

[edo494]

you could guess it, because all units that are pre-placed are in the auto-generated functions, and they always go by the gg_unit_name_identifier, you can get the name from rawcode(hardcode if you must, or read the map's mpq) and find how the identifier works(probably sequentially).

The comment is funny tho

 

[looking_for_help]

you could guess it, because all units that are pre-placed are in the auto-generated functions, and they always go by the gg_unit_name_identifier, you can get the name from rawcode(hardcode if you must, or read the map's mpq) and find how the identifier works(probably sequentially).

Won't help because the identifier is not valid until it was used from a GUI trigger. So even if I have a method to extract all unit names correctly the compiler doesn't know them.

Its because CreateUnitsForPlayerN looks like this:

globals
endglobals

function CreateUnitsForPlayer0 takes nothing returns nothing
    local player p= Player(0)
    local unit u

    set u=CreateUnit(p, 'earc', 0, 0, 0)
    set u=CreateUnit(p, 'hrif', 0, 0, 0)
    set u=CreateUnit(p, 'Hblm', 0, 0, 0)
endfunction

Only if I use a unit within a GUI trigger it gets expanded to

globals
    unit gg_unit_hrif_0001= null
endglobals

function CreateUnitsForPlayer0 takes nothing returns nothing
    local player p= Player(0)
    local unit u

    set u=CreateUnit(p, 'earc', 0, 0, 0)
    set gg_unit_hrif_0001=CreateUnit(p, 'hrif', 0, 0, 0)
    set u=CreateUnit(p, 'Hblm', 0, 0, 0)
endfunction

So the variable only exists in this version (and without the existance of the variable it wouldn't make sense to add it to the autocomplete list).

The worst case happens when the identifier is used without "registering" the variable by a GUI trigger. That evaluates to:

function CreateUnitsForPlayer0 takes nothing returns nothing
    local player p= Player(0)
    local unit u

    set u=CreateUnit(p, 'earc', 0, 0, 0)
    set gg_unit_hrif_0001=CreateUnit(p, 'hrif', 0, 0, 0) // unit gg_unit_hrif_0001 does not exist!
    set u=CreateUnit(p, 'Hblm', 0, 0, 0)
endfunction

which is obviously a compile error... Really annoying

 

[Bribe]

^ Just one more reason GUI isn't going away.

I've been "remastering" WarChasers (keeping all gameplay the same but optimizing most things and removing all memory leaks) and I intentionally keep it 95% GUI.

 

[looking_for_help]

I did some more tests and its definitly a WE problem, not related to vJass...

A good compiler could maybe fix that manually, but that would be a lot of work for little gain. Unfortunatly Blizzard wanted to save global variables but their mechanism is broken...

 

[edo494]

it is a little bit reasonable, imagine creating 1 global per destructable placed, with some maps having upwards to 10k trees, thats a lot of globals

 

[looking_for_help]

it is a little bit reasonable, imagine creating 1 global per destructable placed, with some maps having upwards to 10k trees, thats a lot of globals

Thats true but its also quite confusing. Because GUI just looks like a different representation of Jass and its not obvious that creating GUI triggers has side effects like that (and Jass doesn't).

I mean, their mechanism is basically just: "Check if the unit is referenced anywhere in GUI, if so create a global for that specific unit". There isn't really a reason to not perform exactly the same thing for Jass.

And they implemented it quite bad, because comments should never cause compile errors

 

[Bribe]

What creates the variable is when you use the "select unit" command from the GUI trigger tab. Once you've done that, the variable stays until you manually remove it from Object Manager IIRC.

 

[looking_for_help]

What creates the variable is when you use the "select unit" command from the GUI trigger tab. Once you've done that, the variable stays until you manually remove it from Object Manager IIRC.

Yes, thats correct.

But the asignment of the variable seems to be triggered by a simple string search in the map script. Thats why the variable name in a comment causes a compile error.

Not really much TESH could do about unfortunatly, so this feature wont be implemented.

 

[Wietlol]

I find it odd that

MAP_PLACEMENT_FIXED
MAP_PLACEMENT_USE_MAP_SETTINGS
MAP_PLACEMENT_TEAMS_TOGETHER

are not highlighted in my TESH.

Dunno really the version but I got it when I installed JNGP 2.0.
Haven't changed anything after that really.

 

[looking_for_help]

I find it odd that

MAP_PLACEMENT_FIXED
MAP_PLACEMENT_USE_MAP_SETTINGS
MAP_PLACEMENT_TEAMS_TOGETHER

are not highlighted in my TESH.

Then you are using some really old version of the first TESH. Even TESH 1 highlighted those keywords in its latest version.

Dunno really the version but I got it when I installed JNGP 2.0.
Haven't changed anything after that really.

Yes, JNGP 2.0 is still shipped with the old TESH. I would recommend you update it (just download, unzip and copy the files to your vJass directory) to the latest version of TESH 2.

 

[Wietlol]

Ah, now I remember why I didn't install TESH2...
Now I don't have my own coloring stuff.

Ow yes... and TESH2 converts a random triggers into "?" for no reason at all which makes me almost lose my mind until I found out that I still got that 4k line script stored under another map.

 

[looking_for_help]

Ah, now I remember why I didn't install TESH2...
Now I don't have my own coloring stuff.

You are right, that is unfortunatly still not finished. If you really want custom colors, you have to use the old TESH. But there is somewhere a version of the old TESH available that also highlights the keywords you said.

Ow yes... and TESH2 converts a random triggers into "?" for no reason at all which makes me almost lose my mind until I found out that I still got that 4k line script stored under another map.

Never experienced that and never heard that this happens... Could you explain what you did/what made this happen? Was the whole trigger converted or only comments/special charakters?

You wrote something similar to this before:

Well I am not quite sure but I think that TESH crashes the world editor when you type some syntax.
Because I tried to make a system with the shortcut UHB but it crashes within 5 minutes.
No long code, just a new map and then boom once I wrote that.

Which syntax did you exactly type? And what is "the shortcut UHB"? I don't know such a shortcut.

(I doubt that the crash comes from a textfile where I saved my highlighting styles in.)

The new TESH doesn't perform any file IO, so yes, that cannot be the cause.

but never responded anymore.

So it would be helpful if you could explain a bit detailed what you meant there.

 

[Wietlol]

It just removes an entire JASS "trigger" and places a single "?" in it when I save the map.
Then I copy and paste my original trigger into it (removing the "?" in the process) and save the map again. JASSHelper says "Unknown ascii sign "?"" (Or something similar to that.).

It just happens on a scripts, maybe the largest out of the map.

 

[looking_for_help]

It just removes an entire JASS "trigger" and places a single "?" in it when I save the map.
Then I copy and paste my original trigger into it (removing the "?" in the process) and save the map again. JASSHelper says "Unknown ascii sign "?"" (Or something similar to that.).

It just happens on a scripts, maybe the largest out of the map.

Ok, maybe its related to the trigger size then. Thanks, I will take a look at this.

 

[edo494]

I wonder, how is the online parsing and highlightning going?