Oolite interaction with Expansion Packs

hiran · Post by **hiran** » Tue Apr 13, 2021 9:03 pm

Hello there.

I am interested a bit more in how Oolite actually interacts with expansion packs. So far I understand the Expansion Manager can download a list of OXZs, the user chooses which to install and they get downloaded and dropped in the ManagedAddons folder. But then something needs to happen, right?

I guess Oolite open each of the zip files and reads the manifest.plist. But that just gives a summary of what might be expected in the OXP but is not the real content. So how do ships or other equipment get identified? What scripts from the OXP get executed, and when? How do the scripts interact with Oolite (there must be some API functions)?

So basically I am interested in the full lifecycle and events and whatever...

Hiran

Post by **another_commander** » Wed Apr 14, 2021 5:40 am

You are asking for an explanation of how everything works basically. That's a bit of a tall order, but there are a few pointers:

Check in the source for Core/MiniZip. That's what is used for decompression of the oxzs and it is a bare bones (de)packager based on the standard zlib.
Check out Core/ResourceManager.m. That's how resources are loaded, cached and used. It's a rabbit hole trying to follow what is going on from there onwards.
Check out the Core/Scripting folder. That is the implementation of the scripting engine. OOJavaScriptEngine.m is the heart of the engine, OOScriptTimer.m updates the timers for scripting, OOJSEntity.m is the basic Oolite javascript entity and most of the rest is the specific scripting entity implementations themselves. All (or almost all) JS API functions are listed in the wiki too for reference ( http://wiki.alioth.net/index.php/Catego ... _Reference ), but I find it easier to browse the source when I am looking for info on scripting entities. For quick reference, all scripts are executed continuously and every activity that happens in the game triggers an event handler (e.g. shipWillLaunchFromStation) that the scripts are looking for and act accordingly when the handler is fired. Additionally, every scripting entity in the game has certain properties that have links to the core code and can be read, written to or both. That's it in a nutshell.
Run Doxygen on the Doxyfile in the source's root folder. It will generate full documentation for the game's code and provide you with a map of the entire codebase. Very useful when questions like "how does it all fit in" are asked. I have uploaded the full documentation for whatever version we had back in 2018 here, but you will have to build your own for version 1.90. Warning: The full documentation weighs at around 700MB.

That's it for a start, hope it helps.

hiran · Post by **hiran** » Wed Apr 14, 2021 7:02 pm

Thank you for that response. I will have a look at the doxygen stuff first, but I doubt I would really have to read the source code.
If that is the case then it should be worth documenting the API so OXP developers know what they can rely on without needing to know the Oolite internals. And likewise, Oolite developers would know what they can change in the code without side effects on OXPs. It should be of mutual benefit.

It sounds feasible that there is an event handling system in place. This sounds like a publish/subscribe mechanism to me. How does that work? Would Oolite run a script from the OXP so it has a chance to register to events it is interested in? Or does Oolite always send all events and the OXP chooses which of them to ignore and which to process? And what do you mean by "all scripts are executed continuously"? Would that mean every OXP has a thread allocated to it?

Post by **another_commander** » Wed Apr 14, 2021 7:40 pm

hiran wrote: ↑Wed Apr 14, 2021 7:02 pm
T I will have a look at the doxygen stuff first, but I doubt I would really have to read the source code.
If that is the case then it should be worth documenting the API so OXP developers know what they can rely on without needing to know the Oolite internals. And likewise, Oolite developers would know what they can change in the code without side effects on OXPs. It should be of mutual benefit.

Reading the source for the specific case of JS classes is my personal preference. The API is already documented in the wiki well enough. I just prefer the more condensed list-like presentation of the properties and methods of each class, as it appears in the source. Others may be of different opinion and they can refer to the wiki instead. I don't think there is any action to be taken here, save for improvements on the already existing wiki documentation wherever might be needed, which are always welcome.

It sounds feasible that there is an event handling system in place. This sounds like a publish/subscribe mechanism to me. How does that work? Would Oolite run a script from the OXP so it has a chance to register to events it is interested in? Or does Oolite always send all events and the OXP chooses which of them to ignore and which to process? And what do you mean by "all scripts are executed continuously"? Would that mean every OXP has a thread allocated to it?

Oolite sends the events and each OXP having a corresponding handler will execute it when its turn comes up. By executing continuously, I mean executing at every frame.

hiran · Post by **hiran** » Wed Apr 14, 2021 7:50 pm

another_commander wrote: ↑Wed Apr 14, 2021 7:40 pm

It sounds feasible that there is an event handling system in place. This sounds like a publish/subscribe mechanism to me. How does that work? Would Oolite run a script from the OXP so it has a chance to register to events it is interested in? Or does Oolite always send all events and the OXP chooses which of them to ignore and which to process? And what do you mean by "all scripts are executed continuously"? Would that mean every OXP has a thread allocated to it?
Oolite sends the events and each OXP having a corresponding handler will execute it when its turn comes up. By executing continuously, I mean executing at every frame.

I think now we are getting closer. So the handlers are the entry points, and it is Oolite to decide when to call them.
That means I am highly interested how such handlers are declared. Will go for the wiki first.

hiran · Post by **hiran** » Wed Apr 14, 2021 7:59 pm

Is it safe to say "only an OXP that has an XXX event handler actually adds a XXX feature to the game?"
Or how could the contents of some OXP safely be categorized/understood?

Post by **another_commander** » Thu Apr 15, 2021 5:53 am

hiran wrote: ↑Wed Apr 14, 2021 7:59 pm
Is it safe to say "only an OXP that has an XXX event handler actually adds a XXX feature to the game?"
Or how could the contents of some OXP safely be categorized/understood?

No. An OXP can contain graphics, sounds, new models, new textures, planetary system configuration changes, characters, etc. and those can be features too. A new docking tunnel or new hyperspace effect are not reliant upon the shipWillLaunchFromStation or shipWillEnterHyperspace respectively, for example. There are many OXPs without any trace of a script inside them that add features. Understanding an OXP is a matter of opening its contents up with the appropriate software for each case and studying them.

hiran · Post by **hiran** » Thu Apr 15, 2021 6:29 am

another_commander wrote: ↑Thu Apr 15, 2021 5:53 am

hiran wrote: ↑Wed Apr 14, 2021 7:59 pm
Is it safe to say "only an OXP that has an XXX event handler actually adds a XXX feature to the game?"
Or how could the contents of some OXP safely be categorized/understood?
No. An OXP can contain graphics, sounds, new models, new textures, planetary system configuration changes, characters, etc. and those can be features too. A new docking tunnel or new hyperspace effect are not reliant upon the shipWillLaunchFromStation or shipWillEnterHyperspace respectively, for example. There are many OXPs without any trace of a script inside them that add features. Understanding an OXP is a matter of opening its contents up with the appropriate software for each case and studying them.

I do not yet understand.
So there are features that do not require handlers. But how do they get active?
I mean, is the sheer presence of files sufficient, or is there one file (like a manifest or deployment descriptor) that needs to register the features in Oolite? If the presence of a sound file is sufficient, it would surely have a predefined name, and every OXP would be limited to just one name. I do not believe this is the case.

So somehow Oolite needs to find out what's inside an OXP and thus decide what needs to be done, right? Well, I downloaded the documentation you mentioned but did not yet take a look.

Post by **another_commander** » Sat Apr 17, 2021 8:44 am

hiran wrote: ↑Thu Apr 15, 2021 6:29 am
I do not yet understand.
So there are features that do not require handlers. But how do they get active?
I mean, is the sheer presence of files sufficient, or is there one file (like a manifest or deployment descriptor) that needs to register the features in Oolite? If the presence of a sound file is sufficient, it would surely have a predefined name, and every OXP would be limited to just one name. I do not believe this is the case.

So somehow Oolite needs to find out what's inside an OXP and thus decide what needs to be done, right? Well, I downloaded the documentation you mentioned but did not yet take a look.

Scripts are only one part of the data Oolite reads in order to get its universe set up. The other main data that is used is property lists. These are the files with the weird syntax you have already encountered and written a parser for.

Plists contain data mainly in the form of arrays and dictionaries, which describe certain parts of the game each. So you have planetinfo.plist for the system generation, shipdata.plist for ships information, characters.plist for pikot characters information, customsounds.plist for custom sounds, hud.plist for the HUD descriptions, equipment.plist for equipment descriptions, descriptions.plist for all the descriptions and messages used in the game and many more. Look inside your oolite.app/Resources/Config folder for those.

OXPs can mimic the Resources folder structure to store their own plists, scripts and resource files depending on their functionality. When Oolite starts it scans all the OXP folders and detects files that have the same name and are located in the same named folder as its own resource files. It then goes through a process of either merging or overriding defintions and settings from those OXP files into its own ones. For example, all shipdata.plist files from all OXPs are combined into memory in what is referred to as Ship Registry, so Oolite now knows about all expansion pack ships, as well as its own. Equipment data from all equipment.plists of all OXPs are combined internally with the equipment.plist data that Oolite has in its own Resources folder. Worldscripts are initialized at startup and specific entity scripts are bound to their corresponding entities (mostly based on plist information) as well. Of course this can have problems when OXPs try to access the same information area, because the order in which they are read and merged or override eachother is OS specific and mostly undefined, which is why some OXPs conflict with others. Oolite does sanity checks and tries to avoid this as much as possible, but some things just can't work together. For example, if you have two OXPs with hud.plists and they are both trying to take control of the player's HUD, when they get merged during the data-read startup phase, one OXP's defintions will completely override the other's. Only one OXP will "win" at the end and which one that will be depends largely on the intricacies of the operating system.

The vast majority of all this data is cached for performance reasons and the cache can be found in oolite.app\GNUstep\Library\Caches\org.aegidian.oolite\Oolite-cache.plist on Windows. Search for Oolite-cache in Linux to locate it. The ResourceManager.m source file contains the specific details of how all the above gets bound together.

For the specific case of sounds you have mentioned, have a look at Resources/Config/customsounds.plsit. You can see it is just a dictionary of key-value pairs. This is the list of sounds you can change in the game and by changing the values in your own YourOXP/Config/customsounds.plist you can override the defaults. This of course does not mean you are limited to only those sounds. Why? Because you can control the sound via scripting as well. You can utilize event handlers to produce whatever sound you want whenever you need it with the help of the JS SoundSource class. You can play music if you want. For something truly impressive, check out Svengali's Orchestral Demo OXP. It adds situation-aware background music to the game. It is very cleverly scripted to know if you are on red alert, approaching a station etc. and play the right type of music at the right time.

I fully recommend you take the time to read as much of the generated documentation you have downloaded as you can. If you really, really want to know the details look at the source - start with ResourceManager and your doxygen class diagrams in the docs. Thankfully Jens threw in a lot of good comments in the code when writing it and that will hopefully be quite educational.

Commander_X · Post by **Commander_X** » Sat Apr 17, 2021 10:25 pm

The TL;DR version of another_commander's explanation is that the Resources _hierarchy_ used by the game 1) can be picked up from different places (e.g. ManagedAddOns, AddOns "folders") 2) is the main way to change parts of the game for extending.

To bring in a TL;DR for this comment, I'll come with an non-coding example.

The easiest "mod" you can do in Oolite is to create your own keyboard configuration _without_ changing the Resources folder:
- create your AddOns folder (check where that should be for your OS)
- create a Config folder inside AddOns
- copy the original keyconfig.plist from Resources/Config to your AddOns/Config
- change the AddOns/Config/keyconfig.plist to your needs

I called this a "mod" because as-is, is not an OXP and/or OXZ. In order to make it an OXP, you'd need to create a betterkeyboard.oxp folder under your AddOns folder, and move the new Config and its contents folder from AddOns to your AddOns/betterkeyboard.oxp folder.

hiran · Post by **hiran** » Sun Apr 18, 2021 12:09 pm

Thank you a_c and c_x.

This explains a lot.

So I created code that

checks the List of OXPs (I hope to have collected the same URL the Expansion Manager uses),

downloads them

and reads their manifest.

With that information I was able to create an overview page, but not a list of all ships or equipment and the such.
However now I know there are more plist files that I should be interested in.

I will dig into doxygen and also see what I can read from plists. Maybe that allows me to come up with an inventory for the OXPs that might be worth feeding into the wiki. Once OXPs also contain documentation that could be added - which is the documentation process I'd like to push.
See also https://bb.oolite.space/viewtopic.php?p=275351#p275351

Cholmondely · Post by **Cholmondely** » Sun Apr 18, 2021 2:16 pm

hiran wrote: ↑Sun Apr 18, 2021 12:09 pm
Thank you a_c and c_x.
This explains a lot.

So I created code that
checks the List of OXPs (I hope to have collected the same URL the Expansion Manager uses),

downloads them

and reads their manifest.
With that information I was able to create an overview page, but not a list of all ships or equipment and the such.
However now I know there are more plist files that I should be interested in.

I will dig into doxygen and also see what I can read from plists. Maybe that allows me to come up with an inventory for the OXPs that might be worth feeding into the wiki. Once OXPs also contain documentation that could be added - which is the documentation process I'd like to push.
See also https://bb.oolite.space/viewtopic.php?p=275351#p275351

Once you have them somewhere I can look at them, I can start to help editing them.

At a guess, perhaps 2/3 are already on the wiki, many under a different page heading.

hiran · Post by **hiran** » Sun Apr 18, 2021 6:43 pm

hiran · Post by **hiran** » Sun Apr 18, 2021 10:29 pm

another_commander wrote: ↑Sat Apr 17, 2021 8:44 am
Scripts are only one part of the data Oolite reads in order to get its universe set up. The other main data that is used is property lists. These are the files with the weird syntax you have already encountered and written a parser for.

Plists contain data mainly in the form of arrays and dictionaries, which describe certain parts of the game each. So you have planetinfo.plist for the system generation, shipdata.plist for ships information, characters.plist for pikot characters information, customsounds.plist for custom sounds, hud.plist for the HUD descriptions, equipment.plist for equipment descriptions, descriptions.plist for all the descriptions and messages used in the game and many more. Look inside your oolite.app/Resources/Config folder for those.
[...]
I fully recommend you take the time to read as much of the generated documentation you have downloaded as you can. If you really, really want to know the details look at the source - start with ResourceManager and your doxygen class diagrams in the docs. Thankfully Jens threw in a lot of good comments in the code when writing it and that will hopefully be quite educational.

I found https://github.com/OoliteProject/oolite ... eManager.m to start with.

hiran · Post by **hiran** » Mon Apr 26, 2021 10:21 pm

I'm not sure if I understand everything that ObjectiveC offers. So far I found out that these files are being loaded from an OXP:
// OXPMessages.plist
// nebulatextures.plist
// startextures.plist
// equipment.plist
// equipment-overrides.plist
// requires.plist or manifest.plist
// whitelist.plist
// logcontrol.plist
// role-categories.plist
// pirate-victim-roles.plist
// planetinfo.plist
// shader-uniform-bindings.plist
// world-scripts.plist
// script.plist
// material-defaults.plist

Did I miss something that would register ships?
Do these files have to reside on root level in the OXP or can they be stuffed in subfolders also?
Finally I believe for my purpose I do not need to access all of them. Equipment.plist and something for ships should be good enough for a start...

Oolite Bulletins

Oolite interaction with Expansion Packs

Oolite interaction with Expansion Packs

Re: Oolite interaction with Expansion Packs

Re: Oolite interaction with Expansion Packs

Re: Oolite interaction with Expansion Packs

Re: Oolite interaction with Expansion Packs

Re: Oolite interaction with Expansion Packs

Re: Oolite interaction with Expansion Packs

Re: Oolite interaction with Expansion Packs

Re: Oolite interaction with Expansion Packs

Re: Oolite interaction with Expansion Packs

Re: Oolite interaction with Expansion Packs

Re: Oolite interaction with Expansion Packs

Re: Oolite interaction with Expansion Packs

Re: Oolite interaction with Expansion Packs

Re: Oolite interaction with Expansion Packs