• 🏆 Texturing Contest #33 is OPEN! Contestants must re-texture a SD unit model found in-game (Warcraft 3 Classic), recreating the unit into a peaceful NPC version. 🔗Click here to enter!
  • 🏆 Hive's 6th HD Modeling Contest: Mechanical is now open! Design and model a mechanical creature, mechanized animal, a futuristic robotic being, or anything else your imagination can tinker with! 📅 Submissions close on June 30, 2024. Don't miss this opportunity to let your creativity shine! Enter now and show us your mechanical masterpiece! 🔗 Click here to enter!

Database for text managing?

Status
Not open for further replies.
Level 21
Joined
Mar 27, 2012
Messages
3,232
I'm interested in writing a program for managing information in a manner similar to the WC3 object editor with some notable differences:
Objects are inherited, not copied.
Should allow multiple inheritance from abstract objects. E.g - all abilities that are meant to be implemented have some values(rawcode, base ability).
New fields can be defined at runtime and on a per-object basis.
Files are human-readable. (this part is easy)
A search functionality that works through defining the limitation(s).(E.g, "Field X value must be Y" or "Must not have field X" or "Field name must contain X")

I mainly have 2 uses in mind for such a program, which are:
Organizing this list by grouping abilities with more specific tags.
Making an external object editor that could be read by an already-existing lua script, which implements object editor data.

How should I approach making something like this?
 

Dr Super Good

Spell Reviewer
Level 64
Joined
Jan 18, 2005
Messages
27,202
New fields can be defined at runtime and on a per-object basis.
This never works well for performance as it requires some form of non-static data structure. It is perfectly acceptable for seldom used data but I would not use the same system for data that is accessed thousands of times per second.

Files are human-readable. (this part is easy)
You should have a binary version of the file which is what the program uses. Being human readable and all is fine for editing but it is kind of lazy to parse such files again and again at run time.

You could try using something based on XML. SC2 uses such a system to support inheritance of objects. It also allows you to declare any old field for an object. It is also highly human readable. This makes it fulfil your requirements quite well.

For the search functionality you will probably need to implement it programmatically by searching through all objects and applying the filters to fine ones which are appropriate. This will work fine for a few thousand objects but should be avoided when performance is critical.

Since what you are doing involves data management and data sets, a data base management system might also work. I am not too sure how but there might be SQL ways to do what you are after. Since data base management systems are heavily engineered pieces of software, chances are it will perform superior to any implementation you make for performing queries. You might still want to declare the objects in XML and use a parser to feed them into a database for use.

One possible way to manage all the data inheritance is by chaining hash tables. Each hash table maps field names to data. An object then becomes a list of hashtable references with its own hashtable at the top and the oldest most parent at the bottom. Field queries are performed by checking each hashtable in the list from youngest (the object's) to oldest (root parent) returning when a result was found (or no field if it passes the bottom). This approach has the advantage that changing a parent field will reflect on all children that did not overwrite it automatically. It obviously is quite slow as queries not only have the hashtable complexity but also are linear with the hierarchy depth. I do not see a way for this to support multiple inheritance due to no clear collision handling.

If you do not want parent modifications to reflect on children during run time you can produce a flat hashtable of all fields. Start by copying all the field mappings from the root parent and work your way down to its own declared fields. At each step you overwrite field mappings that collide. This approach means that queries are performed just by a hashtable operation. This approach produces more memory bloat as the mappings of parents (not the data that was mapped to) are duplicated for each child.
 
Level 29
Joined
Jul 29, 2007
Messages
5,174
How do objects in the context of WC3 inherit multiple other objects exactly? that makes no sense.


This never works well for performance as it requires some form of non-static data structure. It is perfectly acceptable for seldom used data but I would not use the same system for data that is accessed thousands of times per second.


You should have a binary version of the file which is what the program uses. Being human readable and all is fine for editing but it is kind of lazy to parse such files again and again at run time.

You could try using something based on XML. SC2 uses such a system to support inheritance of objects. It also allows you to declare any old field for an object. It is also highly human readable. This makes it fulfil your requirements quite well.

For the search functionality you will probably need to implement it programmatically by searching through all objects and applying the filters to fine ones which are appropriate. This will work fine for a few thousand objects but should be avoided when performance is critical.

Since what you are doing involves data management and data sets, a data base management system might also work. I am not too sure how but there might be SQL ways to do what you are after. Since data base management systems are heavily engineered pieces of software, chances are it will perform superior to any implementation you make for performing queries. You might still want to declare the objects in XML and use a parser to feed them into a database for use.

One possible way to manage all the data inheritance is by chaining hash tables. Each hash table maps field names to data. An object then becomes a list of hashtable references with its own hashtable at the top and the oldest most parent at the bottom. Field queries are performed by checking each hashtable in the list from youngest (the object's) to oldest (root parent) returning when a result was found (or no field if it passes the bottom). This approach has the advantage that changing a parent field will reflect on all children that did not overwrite it automatically. It obviously is quite slow as queries not only have the hashtable complexity but also are linear with the hierarchy depth. I do not see a way for this to support multiple inheritance due to no clear collision handling.

If you do not want parent modifications to reflect on children during run time you can produce a flat hashtable of all fields. Start by copying all the field mappings from the root parent and work your way down to its own declared fields. At each step you overwrite field mappings that collide. This approach means that queries are performed just by a hashtable operation. This approach produces more memory bloat as the mappings of parents (not the data that was mapped to) are duplicated for each child.

Did you actually read the first post?
Not only is it perfectly fine to access a "non static data structure", whatever that means (for reference, you can check the most widely used language on this planet - JavaScript), but how is that even related to the specific use of this post.
Why would anything be accessed more than once in the context of this post.
And why would anything be parsed more than exactly once...

And why would anyone sane use XML for anything at this time and age, it's a piece of garbage.
If you're going for some standardized text format for some reason, at least choose something less crap, like JSON.

And why would you add a local SQL server for such a simple program....it's not like you're going to have 20 million entries.

All of that part of your post is simple premature optimization, detached from the actual reality of what WC3 is, living in a bubble.

As to the rest, you described prototypical inheritance, and mixins.
Nothing bad here, just give the proper terms, so that one could search for more information if required.
 
Level 21
Joined
Mar 27, 2012
Messages
3,232
This never works well for performance as it requires some form of non-static data structure. It is perfectly acceptable for seldom used data but I would not use the same system for data that is accessed thousands of times per second.

You should have a binary version of the file which is what the program uses. Being human readable and all is fine for editing but it is kind of lazy to parse such files again and again at run time.

You could try using something based on XML. SC2 uses such a system to support inheritance of objects. It also allows you to declare any old field for an object. It is also highly human readable. This makes it fulfil your requirements quite well.

For the search functionality you will probably need to implement it programmatically by searching through all objects and applying the filters to fine ones which are appropriate. This will work fine for a few thousand objects but should be avoided when performance is critical.

Since what you are doing involves data management and data sets, a data base management system might also work. I am not too sure how but there might be SQL ways to do what you are after. Since data base management systems are heavily engineered pieces of software, chances are it will perform superior to any implementation you make for performing queries. You might still want to declare the objects in XML and use a parser to feed them into a database for use.

One possible way to manage all the data inheritance is by chaining hash tables. Each hash table maps field names to data. An object then becomes a list of hashtable references with its own hashtable at the top and the oldest most parent at the bottom. Field queries are performed by checking each hashtable in the list from youngest (the object's) to oldest (root parent) returning when a result was found (or no field if it passes the bottom). This approach has the advantage that changing a parent field will reflect on all children that did not overwrite it automatically. It obviously is quite slow as queries not only have the hashtable complexity but also are linear with the hierarchy depth. I do not see a way for this to support multiple inheritance due to no clear collision handling.

If you do not want parent modifications to reflect on children during run time you can produce a flat hashtable of all fields. Start by copying all the field mappings from the root parent and work your way down to its own declared fields. At each step you overwrite field mappings that collide. This approach means that queries are performed just by a hashtable operation. This approach produces more memory bloat as the mappings of parents (not the data that was mapped to) are duplicated for each child.

I think this depends a lot on the framework and use case. Lua, for example, provides a single fairly efficient data structure that would probably work quite well for hashtables.

I've considered XML several times and can't deny that it has some benefits, although I do not like the < > syntax.

About the search algorithm, I assume I won't have to deal with data sets of extreme sizes(e.g, 8k objects). I'll have to do some stress tests to find out what specs I should work within.

About the hashtable implemenation, I guess I could do some kind of a mapping system where every inherited field internally links to the parent object.
Since I do want changes on the parent object to affect the child, I'll probably do something like you suggested with lua tables, although I'm not sure about many details.
For instance, imaginging a use case where a field gets removed from a parent object I'd have to somehow know which child objects it affects and if they may have inherited this same field from other objects too.(Noting that fields can be inherited from abstract objects without data)
I'd likely have to do some reference counting, it seems?
Or about multiple inheritance, how to manage a case where a single object inherits an identically named field, but with different data from more than one parent?

How do objects in the context of WC3 inherit multiple other objects exactly? that makes no sense.




Did you actually read the first post?
Not only is it perfectly fine to access a "non static data structure", whatever that means (for reference, you can check the most widely used language on this planet - JavaScript), but how is that even related to the specific use of this post.
Why would anything be accessed more than once in the context of this post.
And why would anything be parsed more than exactly once...

And why would anyone sane use XML for anything at this time and age, it's a piece of garbage.
If you're going for some standardized text format for some reason, at least choose something less crap, like JSON.

And why would you add a local SQL server for such a simple program....it's not like you're going to have 20 million entries.

All of that part of your post is simple premature optimization, detached from the actual reality of what WC3 is, living in a bubble.

As to the rest, you described prototypical inheritance, and mixins.
Nothing bad here, just give the proper terms, so that one could search for more information if required.
This is a bit more wide than the WC3 object editor, because with this same system it would be possible to handle more than just basic object data.
Thus, multiple inheritance would mean inheriting from abstract objects.

Part of the point of making something like this is that I could fiddle around with various features. E.g, I could make formulaic object data generation, with the results of the formulas being visible(like in excel).
Another reason for making this is work efficiency. WC3 object editor has only one level of control and it's very difficult to get past that directly. (SC2 shares this problem)

It could be said that I'm fixing the design issues of WC3 at least for my own uses.

One thing that I would like to do, but currently can't is providing a framework for triggered spells to be configured in a manner similar to the object editor. This would mean that code needs a way to access this kind of tables in some way, which I currently don't know how to do best.
 
Status
Not open for further replies.
Top