A new TESH Syntax Highlighter for Warcraft 3

[looking_for_help]

I wonder, how is the online parsing and highlightning going?

At the moment I want to finish the rest of the recoding before even starting with that.

I just found a way to improve the communication with the trigger editor further (finally eliminating that annoying small lag when opening a new window like Search/Replace), but there is still quite a lot work to do...

 

[Ruke]

@looking_for_help, hi man, could you please add a few more keywords that i'm currently using for vrjass?. these are: cast, implements, break, while, endwhile, end

thanks!

 

[Chaosy]

I think break exists in zinc, don't quote me on that though.

 

[looking_for_help]

I think break exists in zinc, don't quote me on that though.

Maybe, but zinc isn't supported anyways.

@looking_for_help, hi man, could you please add a few more keywords that i'm currently using for vrjass?.

I can add those, though the new version will come with a database where all the keywords are stored (not hardcoded anymore), then its quite easy to add any new keywords.

I'm confident the new version will be ready within the next weeks. I'm still not sure how to handle all the different languages/dialects. The main question would be if they should be handled by the tesh or if the developers of those languages should just add the keywords to the tesh themselfes...

 

[LeP]

I think break exists in zinc, don't quote me on that though.

Dunno if this already highlights these but break is a keyword in cohadars jasshelper amongst for in to downto endfor while endwhile.

 

[looking_for_help]

Dunno if this already highlights these but break is a keyword in cohadars jasshelper amongst for in to downto endfor while endwhile.

No it doesn't (yet) because Cohadars Jasshelper is buggy, so I didn't include those. I might include them as well, but at the end that boils down to the same question posted above...

 

[Ruke]

also we might be able to do the autocompletation feature, let me know if you're interested on it.

 

[Zwiebelchen]

I'm working on this, but its not easy, unfortunatly. We'll see what the next version brings

Just a suggestion on custom function highlighting:

Would it be easier to implement if it only applies retroactively after saving?
So basicly, compiling the code will simply write a table with all custom defined functions and then on the next redrawing of TESH the function will be highlighted?
That would only add a minor inconvenience of having to compile your code every now and then if you want your newly written function to be highlighted, but you should do that anyway, so I guess that could also be called a feature...

 

[looking_for_help]

also we might be able to do the autocompletation feature, let me know if you're interested on it.

What do you mean? If I add the keywords to the highlighter they are automatically added to the autocompletion list too.

Just a suggestion on custom function highlighting:

Would it be easier to implement if it only applies retroactively after saving?
So basicly, compiling the code will simply write a table with all custom defined functions and then on the next redrawing of TESH the function will be highlighted?

I don't think so, unfortunatly. Well, it would, for something like a "10% solution" which would only look at the next token after all function tokens for example.

But that wouldn't work very well because all private functions would polute the global scope (with conflicting parameter lists and so on) just to name on example. To resolve visibility modifiers, you need to parse scopes correctly. To do that you need library reordering based on their dependencies. At the end doing everything thats needed results in a full parser.

The good news is: The new version will be prepared for that. It does already perform several tasks in background threads and most important it uses a lexer defined in the application itself, not in the Scinitlla dll. Doing so was considerably more work but that way one has finally access to the lexer tokens (Scintilla doesn't really tokenize the text, it performs lexing and styling at the same time but without generation of a token stream).

The challenges for such a funcionality are:

• Rewrite the whole TESH to make it "parser-friendly" (~95% done)
• Extract the code from the maps mpq at startup (~50% done)
• Write a vJass Parser and embedd it into the tesh (0% done)

So, progress is slow but hopefully once everything is setup and working the last point will be quite straight forward.

 

[Ruke]

i meant user defined function/struct/variables.

 

[Zwiebelchen]

I don't think so, unfortunatly. Well, it would, for something like a "10% solution" which would only look at the next token after all function tokens for example.

I don't get it... how would that be a "10% solution"? It's not like there are a million different ways to define a function. If it follows the "function" keyword, it is either a function declaration or input parameter for a "code" variable. Those are the only two ways the "function" keyword will ever appear in your code.
So to find the strings, isn't it literally just searching for "function" and "takes" and lexing what is inbetween the two?

Can't you just scan the active trigger/library/class window directly when performing the redraw command for that class, then append all the public functions to a file based on that? I don't get why you would need to parse the entire code.
In order to put a function in your map, you must view the class at least once (to do copy & paste or to manually type it out) unless you inject code directly via an MPQ editor or batch copy&paste multiple triggers.
So unless you do some real obscure stuff, it will be drawn at least once and thus properly added to the function lexer.
And if a function did not get properly added, you just need to view the class/trigger once to force a draw and write it in.

I don't really see a problem here or why you would need a parser for that? It works exactly like all other highlighting, just that the lexer for it updates continously whenever you view a trigger.

 

[edo494]

what about methods inside structs, you still need to be able to tell if it is inside the struct or not, or inside module, inside textmacro and then find uses of that textmacro. You also need to be able to tell if the function is or isnt where it shouldnt be(inside struct, inside module)

 

[looking_for_help]

i meant user defined function/struct/variables.

Ok I see... Might be difficult to integrate since the tesh is C++ and vrJass is Java or what did you have in mind?

I don't get it... how would that be a "10% solution"? It's not like there are a million different ways to define a function. If it follows the "function" keyword, it is either a function declaration or input parameter for a "code" variable.

As mentioned the problem is scoping:

scope s1
    private function foo takes nothing returns nothing
    endfunction
endscope

scope s2
    private function foo takes integer i returns nothing
    endfunction
endscope

scope foo
    private function bar takes nothing returns nothing
        local integer foo
        // what now here?
    endfunction
endscope

What calltip should I show there? 'nothing', 'integer i' or none? And how should foo be highlighted? There are a lot problems that will confound the user more than help him. Especially since many systems have a huge amount of private stuff that will conflict, it will work correct in a few cases. And to resolve private I need (almost full) language parser.

Of course I could just omit all private stuff entirely. But thats also quite a bad solution.

Those are the only two ways the "function" keyword will ever appear in your code.

You forgot the infamous function interface

But as edo mentioned, there are dozends of other things to consider: structs, member functions, textmacros, modules, library ordering, static if, debug, interfaces, extends (inheritance), array, operator overloading and so on and so on.

Only if all those things are included the highlighting will work "as people expect it". If I just look for function it will not work as desired in most of the cases, therefore a "10% solution" (more a "5% solution" acually).

 

[Ruke]

(fuck*ng god i lost my answer because of internet lost)

@looking_for_help i was thinking writing to a file all the informacion of the symbols. but since i don't know how autocompletation might work or which one is the best/easier approach i will first read a little bit about it (i will be checking how eclipse does it, or at least its api).

 

[Zwiebelchen]

what about methods inside structs, you still need to be able to tell if it is inside the struct or not, or inside module, inside textmacro and then find uses of that textmacro. You also need to be able to tell if the function is or isnt where it shouldnt be(inside struct, inside module)

Does it matter where the function is? Syntax highlighting works regardless of if the function can be seen or not.
It highlights the "local" keyword even if I am not allowed to declare a local at that point. Highlighting has nothing to do with Syntax checking.

What calltip should I show there? 'nothing', 'integer i' or none? And how should foo be highlighted? There are a lot problems that will confound the user more than help him.

Private functions are not meant to be seen outside of a scope anyway, so those don't even need to be lexed.
Public functions (as in with the public keyword, not just functions without a scope keyword) ARE a problem, but nobody uses them anyway because they look fugly.

Different structs sharing Method names? Hmm... okay, I see, that IS a big issue.

Only if all those things are included the highlighting will work "as people expect it". If I just look for function it will not work as desired in most of the cases, therefore a "10% solution" (more a "5% solution" acually).

It would already be a HUGE help if this only lexes unscoped functions (for example from snippets). You could just ignore methods or private/public functions in a first implementation.

 

[Wietlol]

(i will be checking how eclipse does it, or at least its api).

Imo, Netbeans has the most accurate and informative autocomplete option of all IDEs that Ive used.

To implement that, you could make a table of all the content in the script (created on save).
Inside the table you have content like variable, function, struct, library and scope names.
Structs, libraries and scopes are tables themselves. Each including their own functions/methods and variables.
Ofcourse, a struct inside a scope inside a library means you have to pass through 4 tables until you get to your function(method) names.

At least that is how I think it works...

 

[looking_for_help]

At least that is how I think it works...

I know how it works. I just didn't have the time because there are so many other things that have to be finished before that.

@looking_for_help i was thinking writing to a file all the informacion of the symbols. but since i don't know how autocompletation might work or which one is the best/easier approach i will first read a little bit about it (i will be checking how eclipse does it, or at least its api).

Eclipse for sure has a inbuild parser. Writing all that stuff to a file is also quite a lot of work, because you would basically have to serialize the whole symbol table and then read it in on the TESH part. Not sure if this is really easier than just writing a parser. Definitly it will be significantly slower than with a parser...

Does it matter where the function is? Syntax highlighting works regardless of if the function can be seen or not.
It highlights the "local" keyword even if I am not allowed to declare a local at that point. Highlighting has nothing to do with Syntax checking.

It matters because user defined keywords can be multily defined with having a different meaning. You can't declare a variable or a function named "local", so its ok if that one is always highlighted as inbuilt keyword. It will always stand for that keyword, nothing else. You can only have stuff like timer timer in structs (not for variables, because of the name mangeling of vJass) and thats already quite bad.

It would already be a HUGE help if this only lexes unscoped functions (for example from snippets). You could just ignore methods or private/public functions in a first implementation.

I could probably include an experimental option to do this. However, it will only work for the current selected trigger then (because the mpq stuff is also not ready yet).

 

[Kazeon]

Is this ready to use? I also wonder can you provide a feature to do syntax check without having to save the map? It helps a lot in making systems on a big map.

EDIT:
I tested this on JNGP 2.0.7 and it works! There's "Beta" writing thingy so I suppose it's not that ready to use yet. But I will use it anyway and will report if I find something.

I also agree with TriggerHappy's idea:

Nice!

Does it autocomplete user-defined variables or functions?

Tho I don't know how you will deal with GUI defined variables then.

Suggestion(s):
- Close the list when a certain term is completed. Like when I write "then", as soon as I completed the term -> close the list. So I don't have to press enter twice to end the line.

I will keep editing this post if I have other suggestions.

 

[Bribe]

Is this ready to use? I also wonder can you provide a feature to do syntax check without having to save the map? It helps a lot in making systems in a big map.

EDIT:
I tested this on JNGP 2.0.7 and it works! There's "Beta" writing thingy so I suppose it's not that ready to use yet. But I will use it anyway and will report if I find something. I also agree with TriggerHappy's idea:

pJass != tesh

 

[IcemanBo]

The images are not working.

 

[looking_for_help]

Is this ready to use?

Yes, it is ready to use.

I tested this on JNGP 2.0.7 and it works! There's "Beta" writing thingy so I suppose it's not that ready to use yet.

Right, its still in beta status.

The new version will provide more features when its done. Unfortunatly I encountered a problem with hotkey subclassing I wasn't able to solve until now (this problem does not exist in the current released version).

I also wonder can you provide a feature to do syntax check without having to save the map? It helps a lot in making systems on a big map.

In theory that could be done, but that would be very much work.

But I will use it anyway and will report if I find something.

Sure, you can use it and give feedback, thats the reason I uploaded it here

The images are not working.

Which images do you mean?

 

[IcemanBo]

I mean images in the first post. They seem to be not available.

 

[KILLCIDE]

After downloading this, one of the first things I noticed was a delay in the Trigger Editor. Whenever I click on a new GUI function or just open up the Actions menu, I have a slight delay, almost looks like a freeze, before the window actually pops up. This was an instaneous thing. Here's a comparison:

Original TESH

Your TESH

I hope the delay is noticable in the GIF I provided. I didn't think it would bother me that much, but after about 10 minutes of use, it started to irk me. Hoping you can find a solution and or provide some reasoning behind this!

 

[Flux]

Just curious, why use TESH if you're gonna use GUI anyway?

 

[KILLCIDE]

Just curious, why use TESH if you're gonna use GUI anyway?

Because I like using both

 

[looking_for_help]

I mean images in the first post. They seem to be not available.

Oh, you are right, I didn't notice that. Guess I have to make new ones then, seems that the images were removed.

After downloading this, one of the first things I noticed was a delay in the Trigger Editor. Whenever I click on a new GUI function or just open up the Actions menu, I have a slight delay, almost looks like a freeze, before the window actually pops up.
...
I didn't think it would bother me that much, but after about 10 minutes of use, it started to irk me. Hoping you can find a solution and or provide some reasoning behind this!

Yes, you are right about that. The reason is interprocess communication, I already found a way to solve this. Unfortunatly this solution comes with new problems I couldn't fix until now, but it will get fixed eventually.

 

[Glowackos]

Heya looking_for_help,

Moyack just released a new JNGP 2.0 version (2.0.9) and he's included your new TESH.

It's pretty sik!

Would be really cool if we could tinker with the theme though. I know you've had this on your todo list for a while. Just trying to give it a slight bump in priority. Even if to start with you just added a couple of presets rather than full customisation; would go a long way.

FWIW, here's the styling I used with the old TESH (in case you want to make presets haha):

 

[Glowackos]

Got some more feedback after using this a bit more.

- Probs related to the IPC stuff you mentioned above:

- Object editor is insanely slow to open dialog boxes such as model/icon selection if the trigger editor has been opened at any point while having WE open.
- ctrl+f in the trigger editor is also very slow to open, but not in all cases. Not sure if it depends on the size of the currenly open trigger, how long the editor has been in use, or something else.​

- I'm not a fan of inserting `\t`s. Would be nice to have an option to choose whether tab inserts `\t`s or spaces. Ofc could still make it feel tabby by making backspace remove a tab's amount of spaces instead of just one. Also, automatically adding an indentation level if you press enter after the keywords [`then`, `loop`, `returns <foo>`, etc] and removing one after [`endif`, `endloop`, `endmethod`, etc] would be awesome.

- What editor did you base shortcuts off? Like ctrl+d to duplicate line. I'd suggest basing off Atom or Sublime. Here are some nifty shortcuts from those editors that I'd love to see here:

- ctrl+shift+d duplicates the line (like what ctrl+d does currently)
- ctrl+d duplicates the cursor. So if I've highlighted the word `foo` and then press ctrl+d, it'll keep my current highlight but also highlight the next occurrence of `foo` (this can continue however many times I press ctrl+d). If I then start typing, it'll enter text simultaneously in all those places (ie a serach&replace of sorts) with a visible cursor at each location. Cursor movement actions like arrow keys or home key get applied to all of the cursors at the same time. Highlight actions (holding shift and moving cursor) work, and copypasting works on a per-cursor basis. So if i ctrl+d through a bunch of `foo`s, move cursors to the right a bit, highlight some text (different for each cursor) and ctrl+c it, if I then move the cursors elsewhere, say I press enter to create a new line at each location and then ctrl+v to paste, each cursor pastes what it copied (each one has its own clipboard, per se). Pressing esc or clicking away reverts to single cursor and pasting at this point would paste the copied text from all of the cursors as one contiguous blob. Hopefully I've explained that well. Try using it in Atom/Sublime to see what I mean. This is a super handy feature.
- ctrl+u undoes the most recent cursor duplication. ie unhighlights the most recent word - useful if you press ctrl+d too many times.
- ctrl+down moves the currently selected line down (swaps it with the line below it)
- ctrl+up does the same, but moves the line up.
- ctrl+shift+f searches all triggers.
​

All in all, it's still a much improved experience over the old tesh. Great work, hopefully there's more to come!

 

[Deleted member 219079]

- Object editor is insanely slow to open dialog boxes such as model/icon selection if the trigger editor has been opened at any point while having WE open.

Only thing preventing me from using this

 

[Chaosy]

....

I had no idea..

God knows how much time I've spent trying to find out why my GUI is slow AF. And the TESH was the issue?..

edit: Yup, changed the tesh and it works fine.
Sorry I will have to move away from this project

 

[Glowackos]

Bug report. I've hit this several times now.

I edit a trigger. ctrl+s to save. Map comes up with an error about some missing library. I navigate to a different trigger and then click back to the one I had been editing earlier and discover that everything is gone, replaced by this small snippet of corruption:



So the error about the missing library is because my several hundred or so lines of code in that trigger have disappeared. Worst is that this gets saved. Luckily I version control my project well, but this could be pretty bad for those who don't make backups often.

I haven't been able to pin down how to reproduce this reliably yet, but will provide details if/when I figure it out. It's happened twice to me today in the span of an hour, and a few more times in the past few weeks. The only additional info I can give at this stage is that if I try to ctrl+z on the trigger after this has happened, WE crashes.



e: K, so it has nothing to do with saving. Seems like in some cases, if i swap to another trigger and swap back, the whole trigger is corrupted. Something to do with saving/loading the text from memory. One time this happened, it pulled through some random map constants through. I should've taken a screenshot, but didn't. Will do next time I see it.

Once the trigger is corrupted, trying to paste the original code back in it doesn't stick - swapping to another trigger and back, reverts to corruptedness. So far the best solution I've found is to delete the trigger, create a new one and paste the old code there. So, I'm inclined to say it's something to do with the metadata you save for that trigger, be it clipboard history, scrollposition, or something else. Something is going awry. Hopefully this'll be enough to help pin it down.

 

[Deleted member 219079]

Sounds very fatal, I'd recommend you use TESH that ships with JNGP until this receives a patch (implying it will).

I version control

Are you referring to JassHelper's back-up feature, or?

 

[Glowackos]

Nah, more sophisticated

I wrote a GMSI script to extract all trigger data out of the map, which i then version with Git.

 

[Deleted member 219079]

Oh, I see.

I like global management systems incorporated too.

 

[Glowackos]

Haha,

I mean this thing: Release: GMSI - Wc3C.net

It's what makes YouTD possible.

 

[Deleted member 219079]

That's actually nice

Thanks for the link

 

[MindWorX]

Any update on the input latency brought up by KILLCIDE?

 

[looking_for_help]

Any update on the input latency brought up by KILLCIDE?

Yes, the problem is already fixed (however, not released yet). It is related to the inter-process communication of tesh and WE...

In the new version, both tesh and WE run in the same process which makes things a lot easier and faster.

I hope I will be able to release a new version during the upcoming public holidays...

 

[KILLCIDE]

That's great to hear looking_for_help! Any news on when you plan on letting users configure the highlighting?

 

[looking_for_help]

That's great to hear looking_for_help! Any news on when you plan on letting users configure the highlighting?

Currently working on this... I'm really not happy with the amount of time this is already taking, but fixing one problem often creates 5 new problems...

To give a quick insight on the current state what has been already done/fixed:

• Trigger editor delay is fixed (this one was hard)
  
• Trigger corruption is fixed
• New or custom keywords (like from ZinC and so on) can be added to a database conveniently
• Search/Replace, Function list and menu items already reworked and functional
• Lexer only scans text affected by modifications (this is a major performance improvement since Scintilla wasn't designed to be able to do this for context sensitive Code)
• Lots of small new features (will be listed on release)

Whats open is to make stuff like colors configurable, add calltips and some error handling for file I/O...

 

[KILLCIDE]

Awesome! I'm sure it's safe to say that people would rather wait a long time for a new, updated TESH than having an extremely buggy one released.

Also, the images you have linked in your first post are all broken.

 

[looking_for_help]

Awesome! I'm sure it's safe to say that people would rather wait a long time for a new, updated TESH than having an extremely buggy one released.

Exactly.

Also, the images you have linked in your first post are all broken.

Yes, I will make some new images.

 

[MindWorX]

Got an ETA on just the delay fix? I'm currently working on the JNGP replacement, and it appears TESH2 affects the entire editor. Even making new objects is slowed down. I'd love to include your updated version, but it is currently a pain to use.

 

[LeP]

Any thoughts on integrating jassdb?

 

[looking_for_help]

Got an ETA on just the delay fix? I'm currently working on the JNGP replacement, and it appears TESH2 affects the entire editor. Even making new objects is slowed down. I'd love to include your updated version, but it is currently a pain to use.

Can't really give an estimate, sorry... The thing is I had to rewrite (almost) everything to fix the problem, so I'm currently just re-implementing features which I can't just copy/paste due to different meachnisms.

I will use the upcoming public holidays but I can't promise that it will be finished very soon (though I will try).

Any thoughts on integrating jassdb?

Nice project, didn't know this before.

Function list descriptions will be configurable from outside (via a database), so you will be able to integrate this pretty easy (and then probably upload the modifed db here such that everyone can use it).

 

[looking_for_help]

Update to version 0.9

Yes, its finally done. It took much longer than initially planned, but I think it was worth it. This is a complete rewrite of the TESH 2.0, basically started from scratch. First lets start with a change list (probably not all changes are included here since there are so many):

• Fixed the input latency bug in the World Editor
• Fixed the trigger corruption bug.
• Added options dialog to allow user customization of styles and general options
• Almost double styles configurable compared to the old TESH
  
• Various options allow you to configure autocomplete behavior as well
• Different general options allow you for example to enable/disable warnings, use reduced folding or enable/disable auto-braces, vertical line, line numbers and many more
• Autocomplete window can now be configured to dynamically grow/shrink while typing and only listing matching items
  
• Escape characters are now highlighted within strings
• Int, Hex and Float numbers can now be styled differently
• Nested block-comments are now highlighted correctly depending on their nesting level
• Nesting levels can have a different style as well in block-comments (up to a nest level of 2)
• URL hyperlinks in strings and comments are now highlighted and are CTRL-Click sensitive
• Braces have now a primary and a secondary style for highlighting matches
  
• User includes (functions) can be added to the highlighter by customized text files (like in the old TESH)
• User includes can be reloaded and removed without the need to restart the editor
• Function list also adds user includes
• Function list now distinguishes between (normal) constants and events
• Trigger templates can now be configured (both the button and the menu entries)
• Users can build their own structure for trigger templates which is updated on the fly in the TESH menu
• Keywords (normal, string and character keywords) are now stored in a SQLite database and can therefore also be modified from outside (including calltips and function list descriptions)
• Search string for Trigger Editor window can now be configured by the .ini file "trigger_editor_name.ini"
  
• New about dialog with about 40 did-you-know messages
• Improved performance
  
• Fixed various bugs

In the attached screenshot you can see some of the new styling which is now possible with the new TESH:

Or some customized dark styling (just one example):

And attached some impressions from the new options dialog with some short explanations.

General options

Here you can setup some general options. For example you can enable/disable warnings here, configure if you want to see margins like line numbers and/or folding flags, if you want to have calltips, a vertical line and so on.

Autocomplete options

Here you can configure the behavior of the autocomplete list. You can for example specify a minimum number of characters before the list pops up, how many items should be listed, that enter always closes the list and adds a newline or that the list is closed on the first full match. Also case sensitivity or automatically added braces when accepting items from the autocomplete list can be turned on and off here.

Style options

Basically every style available in the TESH can be configured here. There are almost the double number of configurable styles compared to the old TESH, you can even customize things like calltip, indicator or AI function styles. Numbers are now seperated into three categories (integer, hex and float) and can be styled differently. Block comments are highlighted correctly even when nested and you can specify different colors for nesting levels up to level 2. URL Hyperlinks can be styled as well and are CTRL-Click sensitive, just as standard hotspots.

Additionally in the following screenshot, there are the new TESH menu items shown. Notice that the template menu items can be configured individually by just placing the templates in a .txt or a .j file in the tesh/templates folder (or subfolders therein).

New menu items

Here the new menu entries are shown. User includes are loaded from the files in tesh/includes folder. Trigger templates are loaded from tesh/templates folder. The "Load Template" button loads the content of the file from tesh/templates/standard.txt. By changing the content of this file, you can also change the template that is loaded when clicking on this button. If no such file exists, a default template is loaded.

I hope you like it.

 

[TriggerHappy]

Looks really great, I'll try it out soon.

 

[IcemanBo]

This is very impressive!

 

[DD_legionTN]

This going to make all of our mapping life much more easier then before since the latency bug was bugging us so hard. And I like the improvement of tesh functions too, gonna improve much for my scripting life in WE!

 

[moyackx]

It looks like pure sex, almost pornographic vJASS coding. I'll try it soon and, of course, it will be added to the JNGP