Recommended tool for extensive editing of planetinfo.plist ?

[Redspear]

Planet info. Plist is a bit of a beast.
8 galaxies with 200+ planets each and not all of the data is writable by scripting.

So does anyone know of a suitable tool that could make the kind of changes that could be explained in a sentence without laboriously editing each individual planet?

For example: double sun distance in every rich industrial system. Only takes a few seconds to explain but potentially hours to edit.

Any known (and preferably free) software that could speed this up?

[MrFlibble]

Planet info. Plist is a bit of a beast.
8 galaxies with 200+ planets each and not all of the data is writable by scripting.

So does anyone know of a suitable tool that could make the kind of changes that could be explained in a sentence without laboriously editing each individual planet?

For example: double sun distance in every rich industrial system. Only takes a few seconds to explain but potentially hours to edit.

Any known (and preferably free) software that could speed this up?

sed
(edit, I'm having a twiddle at doing that in sed, but realistically perl or python would be able to do it in very few lines of code with regexes)

[Griff]

MrFlibble's mention of regexe there in their post reminded me of once using them in Notepad++ to fix something in a huge text file for work, can't remember what it was now but it did sort it, might be worth a look maybe if you haven't already
notepad++: https://notepad-plus-plus.org/

RegEx Cheat Sheet for Notepad++:
https://www.linguisticsweb.org/doku.php ... ex-notepad

[Wildeblood]

Hello, it is I, Mr Grumpy.

Did someone just ask for a user-friendly tool and receive replies including the word "regexes"?



In your heart, you know it's flat.

[Commander_X]

Taking regex out of the equation, awk could also be a good tool (mainly for the type of operation mentioned, i.e. involving just doing math on a value). It would also easily allow getting messy with regex.

[MrFlibble]

Did someone just ask for a user-friendly tool and receive replies including the word "regexes"?

Fair!

A user friendly tool can be built using command line tools. The command line tools themselves are often not immediately friendly.

Here's a crude tool to double sun distances on economy=0 planets. Works on Linuxmint 21.3, and probably almost any current-ish Linux with gnu-sed and bash. It works on a temporary file, which can be copied to the live installation once inspected.

Code: Select all

#!/bin/bash

thefile=~/GNUstep/Applications/Oolite/oolite.app/Resources/Config/planetinfo.plist
testfile=/tmp/testme.plist
cp $thefile $testfile

getgoodbits(){
 grep -Ee '^\s*"[0-9]+\s+[0-9]+"\s+=\s+{' -e sun_distance -e economy "$thefile" |\
  sed -E '
   s/^\s*//
   s/^"([0-9]+) ([0-9]+)\s*"\s*=\s*\{\s*$/\1,\2 /
   s/^economy\s*=\s*([0-9]+)\s*;\s*$/e\1 /
   s/^sun_distance\s*=\s*([0-9]+)\s*;\s*$/\1 !/
  ' |\
  tr -d '\n' | tr '!' '\n' | grep " e0 " |\
   while read x ; do
    echo "${x%% *} ${x##* }"
   done 
}

mashplist(){
 while read i ; do  p=${i% *} # planet index
  x=${p%,*} # p left
  y=${p#*,} # p right
  s=${i#* } # sun_distance
  d=$[s*2] # double sun distance
  echo -e "\rDoing planet $x $y with sun_distance old=$s new=$d   "
  # We have to dip in and out of double quotes to use bash strings in sed.
  sed -E '
   /^\s*"'"$x"'\s+'"$y"'"\s*=\s*\{/,/^\s*\}\s+$/{
    /^\s*sun_distance\s*=\s*[0-9]+\s*;\s*$/{
     s/[0-9]+/'"${d}"'/g
    }
   }
  ' $testfile > ${testfile}.tmp && mv ${testfile}.tmp $testfile ; done
 echo # reset prompt
}

getgoodbits | mashplist

Paste that into a file, modify the variables at the top to suit your setup, chmod +x the thing and profit.

This is brutal and slow as it parses the whole file for each item. It would be far nicer to do it in python or perl, sucking up the whole file first, and processing it in memory, however, I find bash a very quick way to prototype stuff.

...and it works

So, in summary. If there wasn't an easy tool to do that, there is now!

Edit: I've uploaded the edited plist file to testplanet.zip and a copy of the script as mashplist.zip. Those are temporary file shares which will conk out in a month.

[Commander_X]

If the comprehensibility of the regex is a challenge, then this awk code might be easier to grasp:

Code: Select all

{ 
  if ($1=="economy") 
  {
    if ($3=="0;")
      here=1;
    else
      here=0;
  }
  if (($1=="sun_distance") && here) 
  {
    split($3, val,";"); 
    print sprintf("\t\tsun_distance = %d;", val[1]*2); 
  }
  else
  {
    print $0;
  }
}

Saving this as a "double_sun_distance.awk" file, can be run as

Code: Select all

awk -f double_sun_distance.awk planetinfo.plist > new_planetinfo.plist

The new file, new_planetinfo.plist will be the one you can use as a replacement.
Funny fact, if you didn't want to waste disk space with a new file, the code can be passed as an one liner to awk

Code: Select all

awk -- '{if ($1=="economy") {if ($3=="0;") here=1; else here=0;} if (($1=="sun_distance")&&here) {split($3, val,";"); print sprintf("\t\tsun_distance = %d;", val[1]*2);} else {print $0;}}' planetinfo.plist > new_planetinfo.plist

with the same results.
This should work as-is in Linux and/or macOS.
For Windows, you'd have to either get a Win32 (i.e. native) version of awk, or install minimal mingw64, msys, cygwin solutions, or do a whole WSL altogether.

[hiran]

Is it that?

Someone asks for user friendly plist editing and we end up comparing regex, sed and awk?

I believe the real painpoint is the structure of the file. Forget a quote or a comma and you are gone. Neither of the tools under discussion help here.

But what does?
I never really investigated but bumped over this:
https://github.com/corpnewt/ProperTree

Would like to hear how that works or what other software in that category would exist.

[gilhad]

If I would like to make more such changes, I would probably start with extracting each planet to separate file.
Then filtering and processing some types of planets will be lot easier.
And finally I will assemble the planetinfo.plist together from those files.

[Wildeblood]

8 galaxies with 200+ planets each and not all of the data is writable by scripting.

Specifically, 8 x 256 = 2048, but are we sure not every property is writable by scripting? Could you give an example?

[Commander_X]

Is it that?
[...]
I never really investigated but bumped over this:
https://github.com/corpnewt/ProperTree

Would like to hear how that works or what other software in that category would exist.

The plist format was "inherited" by Apple from [NEXT|Open]Step. The Linux/Windows GNUStep was/is aligned with its initial "text" format.
Meanwhile, Apple deprecated the "text" format, for the XML and/or binary formats (also supported by GNUStep). Most recent (visual/user friendly?) plist tools would only deal with the XML and binary formats, including the ProperTree app you mentioned (using python plistlib library). I didn't try it, by right at the beginning of plist.py file (line 22) there is this:

Code: Select all

try:
    FMT_XML = plistlib.FMT_XML
    FMT_BINARY = plistlib.FMT_BINARY
except AttributeError:
    FMT_XML = "FMT_XML"
    FMT_BINARY = "FMT_BINARY"

The "text" format has the advantage that with a trivial text editor, one can make "quick" changes. Unfortunately, for multiple quick changes, a trivial text editor would not be enough, but a "programmable text editor" could help (Emacs? vim?).
I have the feeling that any user friendly plist editor (regardless of the file format) would have real trouble trying to do the type of changes described here

[MrFlibble]

This thread is turning up some goodies.

Good shout hiran, I'd forgotten about propertree... I used it a little while back. Great tool. Fine for editing, but not programmable to do what's been asked for here IIRC.

Awk.. I love the example above from Commander_X, and I'll bookmark it as a good example. I wish I'd had cause to fall further into the awk rabbit hole. I use pinches of it it in pipelines for simple things such as printing the fourth, second, then first word from each line when e.g. parsing csv's in shell scripts, but am not fluent in it. The example above is far cleaner than my bash-smashery, and certainly easier to port across platforms. I'll bet there's less Gnu vs BSD divergence than with sed and grep.

The awk script as-is relies on the keys in the xml being alphabetically sorted, so economy MUST be above sun_distance, which is happily true in this case.. I had set out aiming to make it work regardless the order of keys in a system block, but when I noticed the rigid ordering I took advantage So my script above is prone to the same potential failing upon adaption to other plists. It may have been simpler had I realised sooner.

I'd like to add to the heap of tools: plistutil, a tool to convert from/to binary/xml plist formats. Available on many distros via apt get plistutil, or from https://libimobiledevice.org. This is something propertree can do, but plistutil may be more useful as a command line tool when batch processing. A binary plist could be turned into xml, processed by whatever-script, then converted back, all without user interaction.

I agree that chunking it up into per-planet files would be a better approach to doing this sort of thing. Certainly so when using simple tools for batch edits like bash+sed/awk/grep. This would take some of the pain out of making a one-liner to test for a value and mangle a system accordingly, and thus simplify much of the potential line-ordering minefield.

So, here's a way to do that.

This works for me to split systems out to separate files from the planetinfo.plist. It adds zero padding to the system numbers to allow for easier sorting/searching/re-assembly.

Code: Select all

#!/bin/bash

# ! and % are absent in the plist, so I can use them as keys.
infile=~/GNUstep/Applications/Oolite/oolite.app/Resources/Config/planetinfo.plist
outdir=planets
mkdir -p "$outdir"

sed -E '
 /^\s*"[0-9]+\s+[0-9]+"\s*=\s*\{/,/^\s*\}\s+$/{
  s/$/!/g
 }
' "$infile" | grep -e '\!$' -e '\%$' | tr -d '\n' |\
 sed 's/\!\!/\!%/g' | \tr '%' '\n' |\
 while read p ; do
  s=$(
   echo "$p" | sed -E '
    s/^\s*"([0-9]+)\s+([0-9]+)"\s*=\s*\{.*$/\1-\2/
   '
  )
  x=${s%-*}
  y=$[ ${s#*-} +1000 ] ; y=${y#1} # add leading zeroes
  echo -en "\rSystem ${x}-${y} "
  dat=$(
   echo "$p" | tr '!' '\n' | tr -d '%' > "$outdir/p_${x}-${y}.frag"
  )
 done
echo #reset prompt

I manually chopped the top and tail from the xml, I'd suggest putting them in the output directory as a_top and z_tail. Then you can simply cat * > output.plist or whatever to re-assemble. The filenames should be just fine for wildcard jiggery-pokery.

Here's the top of the directory:

Code: Select all

$ ls planets/ | head -n 4
a_top
p_0-000.frag
p_0-001.frag
p_0-002.frag
p_0-003.frag

On my laptop it took 14 seconds to run. Considering how clunky the script is, I'll take that. Ripping the plist apart is not likely to be done very often, so it's not worth optimising.

Re-assembly is somewhat faster:

Code: Select all

$ time cat planets/* > newplanetinfo.plist

real	0m0.060s
user	0m0.024s
sys	0m0.035s

Here's a zip with the script and broken down planetinfo.plist as 2050 files in a directory chunky.zip. The link will expire in 30 days.

I know there are going to be more succinct ways to do this, but I like cobbling stuff together as one-liners in bash, then editing them into tools.

'd be happy to see a better, cross-platform method appear here.

[hiran]

So how about two 'simple and reusable' scripts then?
One to turn the planetinfo file into a CSV, another one to convert it back to plist (whichever format).

With that, a user can start editing in a spreadsheet application (Microsoft Office, OpenOffice or LibreOffice) or load it into some SQL database to do stuff. Editing gets easier, pivot tables, filters etc allow to have an overview and finally we can go all the way back to Oolite.

[MrFlibble]

pinfo2csv, csv2pinfo.. I guess this particular case does lend itself easily as long as one doesn't want to involve the top two blocks.

[Redspear]

This thread is turning up some goodies.

Thanks everyone. I didn't expect so much help so quickly
Thanks especially for the examples.

Right, I've got some experimenting to do...