Impact of new builds

[hiran]

Now that we have the new builds (msix, appimage, flatpak) it is much easier to distribute Oolite again. In parallel the MacOS build was removed - it did not work anyway on today's Apple computers. Kudos for those who worked hard to get there!

At the moment when Oolite 1.92 was released and the 'latest' tag moved from 1.90, there was some impact that most of us missed to notice: The process to index all the Oolite resources failed. The result of this indexing is the Oolite Expansion catalog (source here, result here). It is this catalog that allows both the builtin Expansion Manager and the external OoliteStarter to offer OXPs for download and manage dependencies.

"Why did it fail?" you may ask. To have a complete index also the Oolite builtin resources get indexed - they are visible at Oolite 1.92. "But apparently it worked!" you should say now. Well, the builtin resources are zero ships, zero equipment and zero models.

Someone created a file called Oolite-1.92.zip with some content to keep the indexing going. But this file was created and added manually. This is a bad pattern since if can be forgotten again, or the content of the file may get wrongly packaged. Indeed it deviates from what we used to have. Therefore I'd like to see that wound spot get improved.

It is not easily possible to unpack resources from msix, appimage or flatpak builds. Also it is not possible to just read the content from Source.zip for two reasons:
- The manifest.plist is just created during build so it has the real version number inside
- Some resources are outsourced into the binary resources repository that also is not part of Source.zip.

Let's extend the build scripts to provide Oolite resources as zip file. It does not have to be a complete distribution, but it helps for further automation.

[mcarans]

Now that we have the new builds (msix, appimage, flatpak) it is much easier to distribute Oolite again. In parallel the MacOS build was removed - it did not work anyway on today's Apple computers. Kudos for those who worked hard to get there!

At the moment when Oolite 1.92 was released and the 'latest' tag moved from 1.90, there was some impact that most of us missed to notice: The process to index all the Oolite resources failed. The result of this indexing is the Oolite Expansion catalog (source here, result here). It is this catalog that allows both the builtin Expansion Manager and the external OoliteStarter to offer OXPs for download and manage dependencies.

"Why did it fail?" you may ask. To have a complete index also the Oolite builtin resources get indexed - they are visible at Oolite 1.92. "But apparently it worked!" you should say now. Well, the builtin resources are zero ships, zero equipment and zero models.

Someone created a file called Oolite-1.92.zip with some content to keep the indexing going. But this file was created and added manually. This is a bad pattern since if can be forgotten again, or the content of the file may get wrongly packaged. Indeed it deviates from what we used to have. Therefore I'd like to see that wound spot get improved.

It is not easily possible to unpack resources from msix, appimage or flatpak builds. Also it is not possible to just read the content from Source.zip for two reasons:
- The manifest.plist is just created during build so it has the real version number inside
- Some resources are outsourced into the binary resources repository that also is not part of Source.zip.

Let's extend the build scripts to provide Oolite resources as zip file. It does not have to be a complete distribution, but it helps for further automation.

Thanks for checking this. Lone_Wolf mentioned adding a source.zip that contains not just the oolite repo but all the deps as well for distro package creators. Would that help here?

I'm not clear how the expansion process works. Is the process this GitHub workflow: https://github.com/OoliteProject/oolite ... /Build.yml? Or one or more these workflows https://github.com/OoliteProject/Oolite ... er/actions? Or all or some of the above?

How do they interact with the main Oolite repository ie. what do they expect it to have and where do they expect it to be?

[hiran]

Thanks for checking this. Lone_Wolf mentioned adding a source.zip that contains not just the oolite repo but all the deps as well for distro package creators. Would that help here?

I'm not clear how the expansion process works. Is the process this GitHub workflow: https://github.com/OoliteProject/oolite ... /Build.yml? Or one or more these workflows https://github.com/OoliteProject/Oolite ... er/actions? Or all or some of the above?

How do they interact with the main Oolite repository ie. what do they expect it to have and where do they expect it to be?

You already identified the components.

OoliteAddonScanner is the tool that reads a list of URLs, downloads the stuff it finds and generates the summary. Output is the catalog for the expansion managers and a bunch of html files.

The second Github actions script you found compiles tOoliteAddonScanner's Java code, runs the tests and eventually provides a release.

The first workflow you identified uses the release of OoliteAddonScanner and executes it. It is triggered whenever the list of URLs change plus once a month (just in case the OXPs change although they remain on the same URL).

I created this toolchain when the old website was lost, and with it a mysql db and php frontend. The process relied on the Oolite zip distribution to read Oolite resources. Version 1.90 was official and available during all this time which has changed now. So now I need an alternative to access the ressources folder of an Oolite installation.

I saw and acknowledge Lone Wolf's comment about Sizrce.zip but I am not talking about recompiling Oolite again. If we just had a distribution without installer, or even just the resources folder it would be fine already.

There is little effort in getting such a distribution. When buildint Oolite the files accumulate on the build server anyway. Just zip them up before triggering msix/appimage/flatpak flows and publish it as assets.

One of the use cases for the zip is the indexer. Another would be to quickly inspect the content of a build if you do not want to analyze the packaging format.

Have a look at https://tooomm.github.io/github-release ... iteStarter
I am surprised about the download counts of the zip and tar balls although other packaging formats are offered.

The same analysis can be done for Oolite at https://tooomm.github.io/github-release ... ory=Oolite
Just scroll enough to see the Oolite 1.90 downloads.

[another_commander]

The Windows installers are lzma compressed files although they are executables at the same time. Can you treat any of them as an lzma (7zip opens them fine as an example) to get access to Resources?

[hiran]

The Windows installers are lzma compressed files although they are executables at the same time. Can you treat any of them as an lzma (7zip opens them fine as an example) to get access to Resources?

OoliteAddonScanner does not invoke 7zip. I can test if it can be read as zip file.

It's not me who wrote the above quote. hiran you may have mis-edited your response.

Indeed I misedited unintentionally. I'm sorry for that and hope to have restored the original post.

[hiran]

The Windows installers are lzma compressed files although they are executables at the same time. Can you treat any of them as an lzma (7zip opens them fine as an example) to get access to Resources?

I can do nothing about this file. Neither bare Java nor addon libraries will open the archive.

Code: Select all

$ file OoliteInstall-1.92-win-test.exe 
OoliteInstall-1.92-win-test.exe: PE32+ executable (GUI) x86-64 (stripped to external PDB), for MS Windows, 9 sections

$ lzmainfo OoliteInstall-1.92-win-test.exe 
lzmainfo: OoliteInstall-1.92-win-test.exe: Not a .lzma file

$ zip -l OoliteInstall-1.92-win-test.exe 
	zip warning: missing end signature--probably not a zip file (did you
	zip warning: remember to use binary mode when you transferred it?)
	zip warning: (if you are trying to read a damaged archive try -F)

zip error: Zip file structure invalid (OoliteInstall-1.92-win-test.exe)

What other options do we have?

[Lone_Wolf]

Is the expansion catalog also what is used for https://oolite.space/#oxp ?

[phkb]

Is the expansion catalog also what is used for https://oolite.space/#oxp ?

Yes

[Lone_Wolf]

What other options do we have?

If I understand correctly you need the contents of the resources folder as created upon first install ?

If so, would the Resources folder of a linux compile be useful ?

Maybe the workflow for the appimage or the flatpak could be adjusted to output their Resources folder and put it in a zip asset.

[hiran]

Is the expansion catalog also what is used for https://oolite.space/#oxp ?

That's the beauty: We have one source of truth.

Readable for expansion managers (https://addons.oolite.space/api/1.0/overview/)
Readable for users (https://oolite.space/#oxp)
Readable for the technically ambitioned (https://ooliteproject.github.io/oolite- ... n-catalog/)

Although only the third item will suffer from not reading the builtin resources.

[hiran]

Maybe the workflow for the appimage or the flatpak could be adjusted to output their Resources folder and put it in a zip asset.

That's it. Nothing else.

[mcarans]

Maybe the workflow for the appimage or the flatpak could be adjusted to output their Resources folder and put it in a zip asset.

That's it. Nothing else.

What files are read from the Resources folder? Can you point me to the specific code that reads the zip file as I couldn't find it? I tried searching org:OoliteProject /.*\.zip/ but none of the zips that appeared seemed to be relevant.

Do the files needed by this process change much each version or could they be read directly from the Oolite GitHub repository?

[hiran]

Maybe the workflow for the appimage or the flatpak could be adjusted to output their Resources folder and put it in a zip asset.

That's it. Nothing else.

What files are read from the Resources folder? Can you point me to the specific code that reads the zip file as I couldn't find it? I tried searching org:OoliteProject /.*\.zip/ but none of the zips that appeared seemed to be relevant.

Do the files needed by this process change much each version or could they be read directly from the Oolite GitHub repository?

Actually I am wondering why providing such files should be a problem. They have been there for years. Only with the removal of old builds in favour of the new ones the problem has come up.

But to answer your questions:

What files are read from the Resources folder?
It is manifest.plist and the equipment, ships, scripts, models. The resources folder is treated just like any other OXP that get indexed.
If you intend to deliver only the required files you would at the same time limit possible future extensions to the index. So why not simply the complete resources folder, as Lone Wolf already pointed out? We could as well offer a complete zipped distribution for manual installation, but then there would be the question of which OS or all. I would not go that far.

Can you point me to the specific code that reads the zip file as I couldn't find it?
https://github.com/OoliteProject/Oolite ... .java#L203

Do the files needed by this process change much each version or could they be read directly from the Oolite GitHub repository?

It is not easily possible to unpack resources from msix, appimage or flatpak builds. Also it is not possible to just read the content from Source.zip for two reasons:
- The manifest.plist is just created during build so it has the real version number inside
- Some resources are outsourced into the binary resources repository that also is not part of Source.zip.

Regardless of whether the files change much - it is more an issue to have the exact same files. After all the index will not just list the ships but also mention the OXP they came from. You will need that information to check if the index is correct, or to act if someone spots room for improvement.


One of the use cases for the index was to spot and complete a lot of missing wiki pages. Almost all of the equipment and ships on the expansion manager is backed by some wiki page today. See also https://wiki.alioth.net/index.php/Index_of_artefacts

[another_commander]

I am preparing a zipped 1.92 source code tarball and am going to add the manifest file (the one normally created during build) inside it. Resources will be included too. Hopefully this will help with the issue.

Edit: Uploaded source code tarball https://github.com/OoliteProject/oolite ... e-1.92.zip
Manifest file can be found inside the Resources folder.

[mcarans]

Actually I am wondering why providing such files should be a problem. They have been there for years. Only with the removal of old builds in favour of the new ones the problem has come up.

What files are read from the Resources folder?
It is manifest.plist and the equipment, ships, scripts, models. The resources folder is treated just like any other OXP that get indexed.
If you intend to deliver only the required files you would at the same time limit possible future extensions to the index. So why not simply the resources folder, as Lone Wolf already pointed out?
...
- Some resources are outsourced into the binary resources repository that also is not part of Source.zip.

I am thinking about how to simplify the build so that new developers can understand it easily. I'm also considering how to have the version come from the GitHub release tag into the build rather than having to be hard coded in different places. So I need to get an understanding of what is being used and how.

One thing I am interested in and have mentioned before is exploring dropping the Git submoduiles arrangement and putting all that stuff in the Oolite repository directly, then tools would only need to look in one repo to find things and source.zip would be complete.

Can you point me to the specific code that reads the zip file as I couldn't find it?
https://github.com/OoliteProject/Oolite ... .java#L203
...
- The manifest.plist is just created during build so it has the real version number inside

Given that the Java uses the latest tag to get the latest stable release, I think it might be possible to go from that to the tag 1.92 associated with the latest release and checkout the repo and submodules at that point. Then all the relevant Resource files and version information corresponding with that version would be available. I'll investigate further in due course.