bedeviled by missing ;'s? this may help

[cag]

This seems to be a problem of late and I may have a tool to help.

I'm writing a Python3 script to crawl a folder tree, including oxz files, and collect & convert shipdata (XML & OpenStep) into effectdata. The parser catches missing ;'s, unquoted keys containing dashes (this is turned off on line 22, as there are tons of them), keys w/o values and delimiter errors. (it cannot recover from a delimiter error, so it just moves on to the next file)

I've uploaded a crippled version here:
https://www.dropbox.com/s/hrppzl7i2kd4d ... a.zip?dl=0

on line 156, I've set COLLATE_FN = None, so it won't output its usual effectdata.plist, just a log file with any errors.

WARNING: this is a work in progress and if you undo line 156, it will output an effectdata.plist file!
Edit: it now outputs alleffectdata.plist and the line in question is now 170
It does rename any existing one but run it a few times and you can lose your current effectdata.plist file. So, if you want to play around with it, do so in a new folder!

It can also find duplicate definitions across files and can be used to reformat existing files (comments are preserved).

[cag]

Not many Python buffs out there, huh? Don't blame you! Anyway, I posted this in response to a flurry of posts by Fleurghber, who deserves kudos for his diligence in narrowing down the errant oxp's.

However, a couple he reported load w/o incident on my Win10 box, so I was wondering if the Mac version uses a different parser?

I downloaded all the oxz's on http://www.oolite.org/oxps/ (I had to anyways) and ran through all the shipdata.plist files; results listed below.
NB: there are 2 versions of 'Pods' on the site, same category, same author listed (see end of results)

I learned that Oolite's parser is more forgiving than the one I stole borrowed. I did find some stuff that should (eventually) be fixed but of no help to Fleurghber.

I will take a stab at the equipment.plist files, but no promises.

Code: Select all

file: H:\oolite.org-oxzs\Carriers_0.9.oxz
      ===================================
    Error! line 517, column 59 (char 22974): Expecting , delimiter
    ...  1.0, 0.0, 0.0, 0.0); position = ( -67.5.0, 19.0, -97 );},      { subentity_key =  ...
                                                ^ 

    Error! file failed to parse! 

file: H:\oolite.org-oxzs\CustomShields_0.83.oxz
      =========================================
    Error! line 1, column 0 (char 0): PListDecode, No plist object could be decoded (MatchFailError at " ").
    ...     {  "customshieldssparks" = {    ai_ ...
        ^ 

    Error! file failed to parse! 

file: H:\oolite.org-oxzs\Dark_Rainbow_1.2.oxz
      =======================================
    Error! line 77, column 11: not well-formed (invalid token)
    ...    <!--  ------------------------------------------ ...
                   ^ 

file: H:\oolite.org-oxzs\ETT_Homing_Beacon_1.02.oxz
      =============================================
    Error! line 81, column 24 (char 2207): (non-fatal) missing ";"
    ...   unpiloted = 1;    "weapon_energy" = 0.0    }} ...
                                                 ^ 

    Error! line 82, column 5 (char 2213): (non-fatal) missing ";"
    ... oted = 1;    "weapon_energy" = 0.0    }} ...
                                               ^ 

file: H:\oolite.org-oxzs\Executive_Spaceways_2.4.3.oxz
      ================================================
    Error! line 232, column 17 (char 6519): (non-fatal) missing ";"
    ... -83.203);        color = {          hue = 120.0        };        frequency = 1;        phase = 0 ...
                                                       ^ 

    Error! line 242, column 15 (char 6681): (non-fatal) missing ";"
    ... , -83.203);        color = {          hue = 0.0        };        frequency = 1;        phase = 0 ...
                                                       ^ 

    Error! line 252, column 17 (char 6844): (non-fatal) missing ";"
    ... -65.857);        color = {          hue = 180.0        };        frequency = 4;        phase = 0 ...
                                                       ^ 

    Error! line 304, column 17 (char 8144): (non-fatal) missing ";"
    ... -83.203);        color = {          hue = 120.0        };        frequency = 1;        phase = 0 ...
                                                       ^ 

    Error! line 314, column 15 (char 8306): (non-fatal) missing ";"
    ... , -83.203);        color = {          hue = 0.0        };        frequency = 1;        phase = 0 ...
                                                       ^ 

    Error! line 324, column 17 (char 8469): (non-fatal) missing ";"
    ... -65.857);        color = {          hue = 180.0        };        frequency = 4;        phase = 0 ...
                                                       ^ 

file: H:\oolite.org-oxzs\Extra_Thargoids.oxz
      ======================================
    Error! line 388, column 34 (char 11065): Expecting , delimiter
    ... ret";        position = (680.4,-216,-680.4.1);        orientation = (0,0,1,0);         ...
                                                  ^ 

    Error! file failed to parse! 

file: H:\oolite.org-oxzs\FuelTweaks.oxz
      =================================
    Error! line 113, column 15 (char 2550): Expecting ; or , delimiter
    ... };        frequency = 1.0;        phase = 0.5S;        size = 50.0;      },      // gree ...
                                                    ^ 

    Error! file failed to parse! 

file: H:\oolite.org-oxzs\InterstellarTweaks.oxz
      =========================================
    Error! line 196, column 50 (char 5143): (non-fatal) missing ";"
    ... info = {"randomshipnames" = "military";}  };  "IST_mil18" =  {    is_extern ...
                                                ^ 

    Error! line 1768, column 28 (char 43805): (non-fatal) missing ";"
    ...   fuel = 30;    roles = "IST_pirate(0.8)"  };  "IST_pirate22" =  {    is_ext ...
                                                 ^ 

file: H:\oolite.org-oxzs\Liners_v1.6.1.oxz
      ====================================
    Error! line 167, column 22 (char 4325): (non-fatal) missing ";"
    ... = (0, 1.0, -3.5);        color = { hue = 0 };        frequency = 1;        phase = 0; ...
                                                  ^ 

    Error! line 174, column 22 (char 4469): (non-fatal) missing ";"
    ... (2.0, 0.0, -4.0);        color = { hue = 0 };        frequency = 1;        phase = 0.3 ...
                                                  ^ 

    Error! line 181, column 22 (char 4617): (non-fatal) missing ";"
    ... -2.0, 0.0, -4.0);        color = { hue = 0 };        frequency = 1;        phase = 0.3 ...
                                                  ^ 

    Error! line 188, column 22 (char 4763): (non-fatal) missing ";"
    ...  (2.0, 0.0, 0.0);        color = { hue = 0 };        frequency = 1;        phase = 0.6 ...
                                                  ^ 

    Error! line 195, column 22 (char 4910): (non-fatal) missing ";"
    ... (-2.0, 0.0, 0.0);        color = { hue = 0 };        frequency = 1;        phase = 0.6 ...
                                                  ^ 

file: H:\oolite.org-oxzs\Oolite.oxp.CaptMurphy.BreakableTorusDrive.oxz
      ================================================================
    Error! line 11, column 32 (char 263): (non-fatal) missing ";"
    ... UTRAL";  script = "btd_dummy_entity.js"};} ...
                                               ^ 

file: H:\oolite.org-oxzs\Saleza_Aeronautics_2.3.3.oxz
      ===============================================
    Error! line 46, column 17 (char 1286): (non-fatal) missing ";"
    ... , -21.5);        color = {          hue = 120.0        };        frequency = 1;        phase = 0 ...
                                                       ^ 

    Error! line 56, column 15 (char 1442): (non-fatal) missing ";"
    ... 26, -21.5);        color = {          hue = 0.0        };        frequency = 1;        phase = 0 ...
                                                       ^ 

    Error! line 66, column 16 (char 1600): (non-fatal) missing ";"
    ... , -7.595);        color = {          hue = 60.0        };        frequency = 4;        phase = 0 ...
                                                       ^ 

    Error! line 132, column 17 (char 3428): (non-fatal) missing ";"
    ... -51.604);        color = {          hue = 120.0        };        frequency = 1;        phase = 0 ...
                                                       ^ 

    Error! line 142, column 15 (char 3590): (non-fatal) missing ";"
    ... , -51.604);        color = {          hue = 0.0        };        frequency = 1;        phase = 0 ...
                                                       ^ 

    Error! line 196, column 17 (char 4955): (non-fatal) missing ";"
    ... -51.604);        color = {          hue = 120.0        };        frequency = 1;        phase = 0 ...
                                                       ^ 

    Error! line 206, column 15 (char 5117): (non-fatal) missing ";"
    ... , -51.604);        color = {          hue = 0.0        };        frequency = 1;        phase = 0 ...
                                                       ^ 

    Error! line 257, column 17 (char 6529): (non-fatal) missing ";"
    ... , 15.75);        color = {          hue = 120.0        };        frequency = 1;        phase = 0 ...
                                                       ^ 

    Error! line 267, column 15 (char 6685): (non-fatal) missing ";"
    ... 02, 15.75);        color = {          hue = 0.0        };        frequency = 1;        phase = 0 ...
                                                       ^ 

file: H:\oolite.org-oxzs\YellOoCabs_v2.6.oxz
      ======================================
    Error! line 26, column 146 (char 1003): (non-fatal) missing ";"
    ... mal-specular.png"; extract_channel = "a"};                                      ...
                                                ^ 


File H:\oolite.org-oxzs\ClassicShips(Replace)_v1.4.oxz:
  - data also found in: H:\oolite.org-oxzs\Griff_normalmapped_ships_replace_1.1.4.oxz
  Warning!  this file contain entities with duplicate definitions for:
    adder, adder-player, anaconda, anaconda-pirate, anaconda-player, asp, asp-cloaked, asp-player, 
    boa, boa-mk2, boa-mk2-pirate, boa-mk2-player, boa-pirate, boa-player, cobra3-alternate, cobra3-pirate, 
    cobra3-player, cobra3-trader, cobramk1, cobramk1-alt, cobramk1-miner, cobramk1-player, constrictor, 
    ferdelance, ferdelance-player, gecko, krait, mamba, mamba-escort, moray, moray-player, morayMED, 
    morayMED-player, python, python-blackdog, python-player, python-trader, shuttle, sidewinder, sidewinder-escort, 
    tharglet, thargoid, transporter, transporter-miner, viper, viper-interceptor, viper-pursuit, worm, 
    worm-miner
  keys duplicated:
    like_ship

... edited for length


File H:\oolite.org-oxzs\Griff_normalmapped_ships_replace_1.1.4.oxz:
  - data also found in: H:\oolite.org-oxzs\Oolite.oxp.amah.noshaders_extra_stations_addon.oxz
  Warning!  this file contain an entity with duplicate definitions for:
    coriolis-station
  key duplicated:
    like_ship
  coriolis-station[ like_ship ]: "noshaders_coriolis_mainhull_diamonds"  -> "griff_coriolis_mainhull"

File H:\oolite.org-oxzs\Griff_normalmapped_ships_replace_1.1.4.oxz:
  - data also found in: H:\oolite.org-oxzs\Oolite.oxp.amah.noshaders_alternatestations.oxz
  Warning!  this file contain entities with duplicate definitions for:
    icosahedron-station, rock-hermit
  keys duplicated:
    like_ship
  icosahedron-station[ like_ship ]: "oolite_template_icosahedron-station"  -> "griff_ico_mainhull"
  rock-hermit[ like_ship ]: "oolite_template_rock-hermit"  -> "griff_rockhermit"

File H:\oolite.org-oxzs\HOopyCasino1.3.5.oxz:
  - data also found in: H:\oolite.org-oxzs\Trident_Down_2.5.2.oxz
  Warning!  this file contain an entity with duplicate definitions for:
    sidewinder-casinodefender
  key duplicated:
    model
  sidewinder-casinodefender[ model ]: "td-sidewinder_casdef.dat"  -> "hoopy_griff_sidewinder.dat"

File H:\oolite.org-oxzs\IllicitUnlock.oxz:
  - data also found in: H:\oolite.org-oxzs\HardShips_0.89.oxz
  Warning!  this file contain entities with duplicate definitions for:
    transporter-player, worm-player
  keys duplicated:
    like_ship
  transporter-player[ like_ship ]: "transporter-miner"  -> "transporter"
  worm-player[ like_ship ]: "worm-miner"  -> "worm"

File H:\oolite.org-oxzs\Pods.oxz:
  - data also found in: H:\oolite.org-oxzs\Pods_1.35.oxz
  Warning!  this file contain entities with duplicate definitions for:
    pods_UPS_breachBarrel, pods_UPS_emptyBarrel, pods_UPS_explodingBarrel, pods_UPS_jamBarrel, pods_UPS_piggybankBarrel, 
    pods_UPS_retryBarrel, pods_UPS_trumbleBarrel, pods_alloysBarrel, pods_bigBarrel_gems, pods_bigBarrel_kg, 
    pods_bigBarrel_t, pods_breachBarrel, pods_computersBarrel, pods_emptyBarrel, pods_explodingBarrel, 
    pods_firearmsBarrel, pods_foodBarrel, pods_fuelBarrel, pods_fursBarrel, pods_jamBarrel, pods_liquorWinesBarrel, 
    pods_luxuriesBarrel, pods_machineryBarrel, pods_mineralsBarrel, pods_missileBarrel, pods_narcoticsBarrel, 
    pods_piggyBankBarrel, pods_radioactivesBarrel, pods_retryBarrel, pods_slavesBarrel, pods_textileBarrel, 
    pods_trumbleBarrel
  keys duplicated:
    like_ship

... edited for length
    
    
Existing file was not found, new entities registered: 5498
  input: existing 0   shipdata 5,626,985  total 5,626,985
 output:   static 34,258  generated 3,884,170  total 3,918,428

[phkb]

Code: Select all

file: H:\oolite.org-oxzs\Dark_Rainbow_1.2.oxz

With this one, the issue appears to be that XML comments aren't being recognised as valid. Would it be because the comment prefix (<!--)is not in position 1 on the line, or just the presence of comments at all?

Code: Select all

file: H:\oolite.org-oxzs\CustomShields_0.83.oxz

With this one, I can't see anything in particular wrong with the file, except that the initial "{" is not on line 1, position 1. Is the position of that initial character that important?

Code: Select all

File H:\oolite.org-oxzs\IllicitUnlock.oxz:

With this one, it appears that HardShips and Illicit Unlock have some ship datakey names the same. It's certainly easy to change them, but I'm a little nervous to do so, because if a player has used Illicit Unlock to buy a ship, and the data key changes, their save file will become invalid and would require manual hacking to correct. Open to suggestions here.

[Smivs]

Code: Select all

File H:\oolite.org-oxzs\ClassicShips(Replace)_v1.4.oxz:
  - data also found in: H:\oolite.org-oxzs\Griff_normalmapped_ships_replace_1.1.4.oxz
  Warning!  this file contain entities with duplicate definitions for:
    adder, adder-player, anaconda, anaconda-pirate, anaconda-player, asp, asp-cloaked, asp-player, 
    boa, boa-mk2, boa-mk2-pirate, boa-mk2-player, boa-pirate, boa-player, cobra3-alternate, cobra3-pirate, 
    cobra3-player, cobra3-trader, cobramk1, cobramk1-alt, cobramk1-miner, cobramk1-player, constrictor, 
    ferdelance, ferdelance-player, gecko, krait, mamba, mamba-escort, moray, moray-player, morayMED, 
    morayMED-player, python, python-blackdog, python-player, python-trader, shuttle, sidewinder, sidewinder-escort, 
    tharglet, thargoid, transporter, transporter-miner, viper, viper-interceptor, viper-pursuit, worm, 
    worm-miner
  keys duplicated:
    like_ship

Something nuts here I think. Of course these all have a 'like_ship' key, because they all relate directly to the core game shipdata.plist. This is not an error or problem unless I'm missing something.

[cag]

@phkb

re: Dark_Rainbow_1.2.oxz

It's the Python library's XML parser that's squawking, not Oolite, so I think you can ignore it (no one's complained, right?).
It sees '--' and thinks you're trying to close the comment. Imho, that's crappy parsing but maybe there's an XML standard that doesn't allow '--' in comments.

To get it through, I replaced the minus' with underscores, but almost any character probably would've worked.

I can provide you with an OpenStep version if you like (a side effect of my script) but you'd have to transfer the comments by hand, as the Python XML parser doesn't pass comments through.

re: CustomShields_0.83.oxz

You're right, the plist parser I copied didn't like it, easily fixed. I meant to remove that from the log before posting - sorry.

re: IllicitUnlock.oxz:

I'm guessing these 2 oxp's haven't been used much together? It's my understanding that the plist data gets lumped together in the cache, so there's one db for shipdata, one for equipment, etc. (hence the need for unique names). I wouldn't advise changing anything now; if it ain't broke... Just keep it in mind on future projects.

If there is anything in a save game to differentiate 2 oxp's, keys can be (carefully) renamed across saves, but until you have a real need to do so, forget 'bout id.


@Smivs

It's only a problem if someone tries to use both oxp's, which in this case, is unlikely. What was edited for length was 51 lines like:

Code: Select all

adder[ like_ship ]: "griff_adder-NPC"  -> "oolite_template_adder"

meaning, if ClassicShips is loaded after Griff's, all his ships will look like Oolite's templates. Afaik, the loading order is indeterminate (read random).


@phkb
@Smivs

I hope I didn't cause much gnashing of teeth, pulling of hair. I probably should have just posted the errors. These warnings of duplicate keys is more my problem, in that I'm attempting to construct a comprehensive effectdata.plist.

The problem is addVisualEffect is unable to use ship data, so I change all the 'like_ship' keys to 'like_effect' and throw away keys not related to effects. The errors, warnings, XML conversion & output formatting are all side-effects of my script.

Btw, it spat out 148 errors regarding unquoted keys that contain minus characters. Good thing Oolite's parser is smarter/more fogiving than the one in Python's library!

[Smivs]

Ah, OK. No gnashing of teeth, just puzzlement, now allayed.

[cag]

Turns out, for parsing at least, equipment.plist files were rather easy:

Code: Select all

file: H:\oolite.org-oxzs\ExtraFuelTanks-v1.7.oxz
      ==========================================
    Error! line 45, column 67 (char 1470): PListDecode, No plist object could be decoded (MatchFailError at ",").
    ...                     "EQ_LQG_TANK_EMPTY",,                                       ...
                                                ^ 

    Error! file failed to parse! 

file: H:\oolite.org-oxzs\Oolite.oxp.QCS.QTHI_AuxEnergyGenerators_0.1.1.oxz
      ====================================================================
    Error! line 12, column 49 (char 615): (non-fatal) missing ";"
    ... cript" = "EQ_QTHI_AuxEnergyGenerator.js"    }  ),  (    5,    0,     ...
                                                ^ 

    Error! line 36, column 49 (char 1734): (non-fatal) missing ";"
    ... cript" = "EQ_QTHI_AuxEnergyGenerator.js"    }  ),  (    7,    0,     ...
                                                ^ 

    Error! line 60, column 49 (char 2878): (non-fatal) missing ";"
    ... cript" = "EQ_QTHI_AuxEnergyGenerator.js"    }  ),  (    7,    0,     ...
                                                ^ 

    Error! line 83, column 49 (char 4002): (non-fatal) missing ";"
    ... cript" = "EQ_QTHI_AuxEnergyGenerator.js"    }  ),  (    7,    5000,  ...
                                                ^ 

file: H:\oolite.org-oxzs\QTHI_AntiZap_0.1.oxz
      =======================================
    Error! line 10, column 38 (char 203): (non-fatal) missing ";"
    ... ;      "script" = "EQ_QTHI_AntiZap.js"    }  ),  (    1,    0,     ...
                                              ^ 

[cag]

last but not least, effectdata:

Code: Select all

file: H:\oolite.org-oxzs\Andromeda_1.5.oxz
      ====================================
    Error! line 212, column 3 (char 8424): (non-fatal) missing ";"
    ... "andromeda-CargoShip";    thrust = 20;  }} ...
                                                 ^