Help me fix my converter script! (Python)

[Gekigengar]

I prototyped a text based database convertor.

What this basically need to do is take the first 4 field from Re DB, and get the rest of the field from Pre DB with slight modification of string.

This is a tool I rapidly prototyped in Python for my Ragnarok Online Private Server,

its far from done, but If anyone could help me solve the error, that would be helpful!

This is the error I am getting after running the program

Traceback (most recent call last):
  File "C:\Users\Gekigengar\Desktop\kRO to jRO converter Project\kRO 2 jRO mobdb converter\kRO 2 jRO Mob Converter Functions v0.03 Alpha.py", line 232, in <module>
    MakeOutput(renewal_File, prerenewal_File, output_File, TEST_NUM)
  File "C:\Users\Gekigengar\Desktop\kRO to jRO converter Project\kRO 2 jRO mobdb converter\kRO 2 jRO Mob Converter Functions v0.03 Alpha.py", line 205, in MakeOutput
    wholeString = ProcessDBLine(renewal_DB,prerenewal_DB,x)
  File "C:\Users\Gekigengar\Desktop\kRO to jRO converter Project\kRO 2 jRO mobdb converter\kRO 2 jRO Mob Converter Functions v0.03 Alpha.py", line 133, in ProcessDBLine
    processed_String = processed_String + GetDBEntry(renewal_DB,num,z) + ','
  File "C:\Users\Gekigengar\Desktop\kRO to jRO converter Project\kRO 2 jRO mobdb converter\kRO 2 jRO Mob Converter Functions v0.03 Alpha.py", line 84, in GetDBEntry
    if wholeString[ch] == ',' or wholeString[ch:ch+EXIT_LENGTH] == EXIT_CODE:
IndexError: string index out of range

Here is the file
kRO to jRO DB converter

Thank you for reading!

 

[GhostWolf]

If you'd use logic and put a simple test print to see what the value of GetDBEntry(PRE,LineNum,z) on line 117 (which the interpreter gladly tells you), you'd see that the string literal "//1210,FILAMENTOUS,Filamentous,Filamentous,51,6088,0,1926,1353,1,425,525,35,10,1,35,30,5,83,40,10,12,1,4,23,0x3095,200,1500,500,1000,0,0,0,0,0,0,0,7008,4850,947,8000,943,3880,993,200,1451,40,757,18,509,1600,0,0,0,0,4045,1" is indeed not a base 10 integer.

As to why you get that on that specific line instead of whatever you were expecting to get, I can't answer, since your code is practically unreadable.

 

[Gekigengar]

If you'd use logic and put a simple test print to see what the value of GetDBEntry(PRE,LineNum,z) on line 117 (which the interpreter gladly tells you), you'd see that the string literal "//1210,FILAMENTOUS,Filamentous,Filamentous,51,6088,0,1926,1353,1,425,525,35,10,1,35,30,5,83,40,10,12,1,4,23,0x3095,200,1500,500,1000,0,0,0,0,0,0,0,7008,4850,947,8000,943,3880,993,200,1451,40,757,18,509,1600,0,0,0,0,4045,1" is indeed not a base 10 integer.

As to why you get that on that specific line instead of whatever you were expecting to get, I can't answer, since your code is practically unreadable.

I see, GetDBEntry() function is not supposed to return the whole line though,
but instead, a single part of the entry seperated by the commas.

I don't know what caused it, but I will try re-upload a more read-able
script. (Since it was just a rapid prototyping.)

Thank you for the reply, it gave me a new insight.

What do you suggest I should do to make it read-able?

Gonna try my best.

 

[GhostWolf]

I didn't understand what exactly you're trying to do, but open the files once and read them into buffers, use string split to split your tokens, and why on earth are you using an argument to say how many items the file has instead of simply reading all the items in the file?
You can write the parsing in a couple of lines using the proper built-in functions.

Your variable naming is also pretty bad, I have no idea what they mean without reading the code (which as mentioned is quite a hard task).

 

[Gekigengar]

I didn't understand what exactly you're trying to do, but open the files once and read them into buffers, use string split to split your tokens, and why on earth are you using an argument to say how many items the file has instead of simply reading all the items in the file?
You can write the parsing in a couple of lines using the proper built-in functions.

Your variable naming is also pretty bad, I have no idea what they mean without reading the code (which as mentioned is quite a hard task).

Updated the files with better variable names.

I fixed the previous error (It was due to the needless return on GetDBEntry from the comment checker),
and now another issue arise.

Traceback (most recent call last):
  File "C:\Users\Gekigengar\Desktop\kRO to jRO converter Project\kRO 2 jRO mobdb converter\kRO 2 jRO Mob Converter Functions v0.03 Alpha.py", line 232, in <module>
    MakeOutput(renewal_File, prerenewal_File, output_File, TEST_NUM)
  File "C:\Users\Gekigengar\Desktop\kRO to jRO converter Project\kRO 2 jRO mobdb converter\kRO 2 jRO Mob Converter Functions v0.03 Alpha.py", line 205, in MakeOutput
    wholeString = ProcessDBLine(renewal_DB,prerenewal_DB,x)
  File "C:\Users\Gekigengar\Desktop\kRO to jRO converter Project\kRO 2 jRO mobdb converter\kRO 2 jRO Mob Converter Functions v0.03 Alpha.py", line 133, in ProcessDBLine
    processed_String = processed_String + GetDBEntry(renewal_DB,num,z) + ','
  File "C:\Users\Gekigengar\Desktop\kRO to jRO converter Project\kRO 2 jRO mobdb converter\kRO 2 jRO Mob Converter Functions v0.03 Alpha.py", line 84, in GetDBEntry
    if wholeString[ch] == ',' or wholeString[ch:ch+EXIT_LENGTH] == EXIT_CODE:
IndexError: string index out of range

I don't really understand why it gave me an error, the Index never went out of range on previous DB entries...

Here is the file! : kRO to jRO DB converter

I hope its easier to read now!

 

[GhostWolf]

For crying out loud, stop re-opening/closing/reading the same files so many times, this is the reason your code is so awfully slow.

def read_file_lines(path):
    f = open(path)
    lines = f.readlines()
    f.close()
    return lines

To process each line, I'd either split it by comma, or use a regular expression, but I have no idea what you're trying to do, so explain that first.

def read_db(path):
    f = open(path)
    db = []
    for line in f:
        db.append(line.split(","))
    f.close()
    return db

 

[Gekigengar]

Databases

There is 2 Database,
The Renewal Database.
The Pre-Renewal Database.

Both are text-based database used for a game server to load
from.
Both database are similar, the only difference is the newer formula
which is used to calculate the fields, and how the Renewal is slightly
bigger (Because there are monsters that are available only in Renewal.)

Each row in both database defines different monsters.
Each collumn (Which is seperated by commas) defines the monster's fields.

GetDBEntry() to get the certain "Fields" as String

I am trying to convert specific "Fields" by taking it from the
"Pre-Renewal Database" to the "Renewal Database".
Because I will be using the old formula from "Pre-Renewal" into
the game.

Lets say I wanted to combine only the 4th and 5th field entry of the
"Pre-Renewal" into the "Renewal" databases.

GetDBEntry() to get the certain "Fields" as String

I would first check if both monster of the same ID and Name exist.
Then if it exist on both (If not, append signature of #MISSING ENTRY# on the monster.), I would need to identify "Where and What" is the 4th and 5th field in "Pre-Renewal", hence I need the "GetDBEntry()" function.

If entries does not exist on Pre-Renewal Database

Because the "Renewal" database is newer and larger,
I would first check the "Renewal" lines.
And compare it with the "Pre-Renewal" lines.

The program would then take different collumn entries from Renewal and Pre-renewal, to create a new database. (The output).

That is all the core points.

 

[GhostWolf]

In that case, again, use the language!
Read all your entries (which are not comments, empty lines, etc.) into dictionaries and work with them.
Granted, this will remove all your comments, but I don't see the point of having them in a database file in the first place.

def read_db(path):
    f = open(path)
    db = {} # Dictionary
    for line in f:
        if (line[0:2] != "//" and not re.match("^\s*$", line)):
            # If the entry isn't a comment or empty line or line with spaces, add it to the dictionary
            # with the entry ID being the key
            tokens = line.split(",")
            db[tokens[0]] = tokens
    f.close()
    return db

# For example:
db["1001"] = ["1001", "SCORPION", "Scorpion", "16", "153", "1", ...]

Once you have both dictionaries, you iterate over the keys of the larger one, and check if the current iterated key exists also in the other dictionary, and create a new entry in a third dictionary that will be the output.

Finally, when your output is full, simply write every entry to the output file.
Now since dictionary entries don't have any obvious ordering, you might want to handle some sorting if you want the entries to be sorted by ID in your files.
To do this, create an array of all your dictionary keys, sort the array, and then iterate over it and keep fetching entries from the dictionary using these sorted keys.
For example:

# The result dictionary of your code
output = ...
# Your keys are all number strings, so normal string sorting is all that's required
keys = sorted(db)
for key in keys:
    entry = output[key]
    # Write to file somehow
    ...

 

[Gekigengar]

In that case, again, use the language!
Read all your entries (which are not comments, empty lines, etc.) into dictionaries and work with them.
Granted, this will remove all your comments, but I don't see the point of having them in a database file in the first place.

def read_db(path):
    f = open(path)
    db = {} # Dictionary
    for line in f:
        if (line[0:2] != "//" and not re.match("^\s*$", line)):
            # If the entry isn't a comment or empty line or line with spaces, add it to the dictionary
            # with the entry ID being the key
            tokens = line.split(",")
            db[tokens[0]] = tokens
    f.close()
    return db

# For example:
db["1001"] = ["1001", "SCORPION", "Scorpion", "16", "153", "1", ...]

Once you have both dictionaries, you iterate over the keys of the larger one, and check if the current iterated key exists also in the other dictionary, and create a new entry in a third dictionary that will be the output.

Finally, when your output is full, simply write every entry to the output file.
Now since dictionary entries don't have any obvious ordering, you might want to handle some sorting if you want the entries to be sorted by ID in your files.
To do this, create an array of all your dictionary keys, sort the array, and then iterate over it and keep fetching entries from the dictionary using these sorted keys.
For example:

# The result dictionary of your code
output = ...
# Your keys are all number strings, so normal string sorting is all that's required
keys = sorted(db)
for key in keys:
    entry = output[key]
    # Write to file somehow
    ...

Tbh, we needed the comments and blank spaces too. (Since we uses GIT repo to update our databases).

(It will check where to position the new changes with reference to the lines nearby the changes, so sometimes it will refer to the commented lines!)

 

[GhostWolf]

Either way, use something like the first read_db I wrote.
Your current code is so inefficient and slow it makes me want to cry, not to mention it makes your other code a lot longer. Read your lines one time, and split them by commas.
This isn't a good way to design a database, but if you really need it ("it" being things that are not entries in the database, e.g. comments, empty lines), at least work with it properly.