Compressed JSON parser

[phkb]

My work on Station Dock Control has led me to research ways of making JSON less bulky. The way JSON stores arrays of objects is to store the property names and values for every record. That's a lot of repeated text, and in the example of SDC it can be almost 2mb of data.

I came across this article http://stevehanov.ca/blog/index.php?id=104 which has a downloadable example of compressing a JSON string by deriving the schema and storing it independently of the data. The article describes the process better than I just did.

In my tests this halves the space usage of SDC (my 2mb save file was reduced to 1mb), and all the data appears to be present and usable the same as before. Here's a link (https://app.box.com/s/oq8clah7yf28eqe4xjtf7jhnebpvbm5b) to a tweaked version of Steve's code that can be dropped into an OXP project and activated by adding "compressed_json.js" to the worldscripts.plist file, and then changing any instances of "JSON.stringify" or "JSON.parse" to "CJSON.stringify" and "CJSON.parse".

So I have a couple of questions for those who know more about Javascript and the Oolite Javascript object model than I do:
1. Have I broken something somewhere that I'm not aware of? I'm attaching the CJSON objects to the "global" object, but maybe there's a better way. The original code had "window" (as it was designed for web usage), but I'm not sure what's a better alternative in the Oolite object model.

2. Is this of value? Does it matter if save game sizes reach 2mb or more? If it is of value, would it be worth making it a separate OXP? Can you see any pitfalls or gotcha's in the code?

3. The article and code was written 5 years ago, so I'm not sure if there's something better out there. Is there something else I should be looking at?

I don't know if this should be in Expansion Pack or not. Moderators, feel free to move it to a more logical spot.

[kanthoney]

I don't think it matters that much what the game file size is. If it does matter it'd be much more sensible to handle the compression in core rather than expect the OXP writers to do all the grunt work.

[Norby]

worth making it a separate OXP?

I think yes, if you add into the Miscellaneous category then oxpers can use it simply by a require in the manifest.plist.

[another_commander]

I have not looked at the script code yet, but I'll just drop a couple of thoughts based on the questions asked:
1. The global object seems like a sensible enough place to attach this to.
2. Whether it matters or not will probably depend on whether one can live with the extra two seconds that loading a game will take or not. Once the data is loaded, it does not really matter anymore. In my humble opinion though, in the days of terabyte storage and such, I would not really be too worried about a 2MB save file.
3. 5 years ago is more or less the same age as the version of Spidermonkey we are using; it should be fine.

[cim]

Things to consider:
1) Rather than using the native-code JSON parser, which should be quite fast, you're using something written in JS itself, which will be orders of magnitude slower. You might compensate for this by reading less data, of course, but reading from a local disk is fairly fast anyway. You might even hit the run-time limit on a worldscript call in playerWillSaveGame in extreme cases, which would be very bad indeed.
2) All the working memory of the CJSON parser has to be in Oolite's Javascript VM, which only has 32Mb available in total. Again, you might save memory because the memory it needs to produce the CJSON is less than the memory needed to temporarily store the larger JSON object, or you might end up using more memory temporarily because the parser allocates over 1Mb anyway.

Both of these could go either way, and probably different ways in different cases - I think the decision over whether to use CJSON or JSON would need to be tested carefully for speed and memory use in each case it was considered for.


Save file size I agree isn't generally a problem, though, at 2Mb. If we start getting OXPs (or more likely combinations of them) which lead to 20Mb save files, then we might start to consider compressing them. Even then, it might be just as easy to let the user compress their own.

[phkb]

Thanks cim, a_c, Norby and kanthoney. Great feedback. I think the potential for breaking something is too great to worry about save game file size. It made an interesting experiment, but I think in the end it's safer to stick with standard JSON calls and live with 2mb save files. I created a separate OXP with CJSON in it, so if anyone is interested I can publish it, but otherwise I'll just leave this aside for now.

[Norby]

stick with standard JSON calls and live with 2mb save files.

So better if any data which must be stored into a savefile is created as a simple numbered array and not an object with labeled fields.

For StationDockControl I can imagine an index array which store the field labels and used to determine the position of the field in a numbered data array:

Code: Select all

this._idx = [];
this._idx["system"] = 1;
this._idx["station"] = 2;
...

var data = [];
data[this._idx["system"]] = system.ID;
...
this._systemDockingData.push(data);

...
if (this._systemDockingData[i][this._idx["system"]] == system.ID
...

The fastest way if we use field numbers directly, just the code will be less obvious:

Code: Select all

var data = [];
data[1] = system.ID;
...
this._systemDockingData.push(data);
...
if (this._systemDockingData[i][1] == system.ID
...