Building Oolite 4 Linux

[hiran]

Looking at the commit history, I see this:
"I verified prereleases get created for branches but they do not modify the 'latest' tag, which will still point to the latest release."

The problem would most likely come not from the prereleases originating from branches, but from the actual release that would be published once a commit to master gets made. You need to verify that commits to master do not affect the 1.90 release tag.

Yes, that is actually the problem with the current windows builds. And my message is a bit incorrect as I am not talking about the tag with name 'latest' any more. But that is learning on the job...

Last year we hardcoded the release tag to be 'latest' (I'd say we did not know any better).
https://github.com/OoliteProject/oolite ... ws.yml#L44
Therefore, whenever the master branch is modified, that tag is moved to the current commit. Even more, the old release is replaced with the new one and we do not have a history.

With my new model, every release would get a unique identifier to be used both in the title and the release tag.
https://github.com/HiranChaudhuri/oolit ... l.yml#L146
With such distinct release tags Github can build a release history since we no longer move tags and replace releases, as can be seen here:
https://github.com/HiranChaudhuri/oolite/releases

Note that the release 1.91-15 currently has a green indicator 'Latest', but that is not the tag.
https://github.com/HiranChaudhuri/oolit ... ag/1.91-15

So if you want to have a permanent link always pointing to the latest release, Github provides it in means of

Code: Select all

curl https://api.github.com/repos/HiranChaudhuri/oolite/releases/latest

See also https://docs.github.com/de/rest/release ... st-release

[hiran]

Also, regarding incremental build numbers, we already have this in our nightlies versioning (see Makefile):
VER_GITREV := $(shell git rev-list --count HEAD)

This appends the github incremental commit number to the version, producing something like OoliteInstall-1.91.0.7404-230311-b459e81 (that 7404 is the incremental commit number). We are not using it as part of the filename just yet (only embedded in the installer version string and the watermark appearing on screen when the game is run), but it is there and could easily be used.

Nice! I will try that instead of the build number.

[hiran]

Also, regarding incremental build numbers, we already have this in our nightlies versioning (see Makefile):
VER_GITREV := $(shell git rev-list --count HEAD)

This appends the github incremental commit number to the version, producing something like OoliteInstall-1.91.0.7404-230311-b459e81 (that 7404 is the incremental commit number). We are not using it as part of the filename just yet (only embedded in the installer version string and the watermark appearing on screen when the game is run), but it is there and could easily be used.

Nice! I will try that instead of the build number.

Here is the first semantic version:
1.91-alternative2+7418

From https://semver.org:

Code: Select all

<valid semver> ::= <version core>
                 | <version core> "-" <pre-release>
                 | <version core> "+" <build>
                 | <version core> "-" <pre-release> "+" <build>
<version core> ::= <major> "." <minor> "." <patch>

<major> ::= <numeric identifier>

<minor> ::= <numeric identifier>

<patch> ::= <numeric identifier>

Need to check where we get the patch version from. It is not in the Oolite version number. Where should it come from?

[another_commander]

Yes, that is actually the problem with the current windows builds.

I would call it desired behaviour rather than a problem. Only one release, namely 1.90 at this time, should be tagged as latest and get a green tag. Everything else should be published as a prerelease.

Even more, the old release is replaced with the new one and we do not have a history.

Keep in mind that unlimited quotas on github are not a given, thus maybe not keeping a long history of builds is preferred. I do not know what will happen if we keep on adding builds to the releases list without restrictions. Maybe nothing will happen and I'm just being paranoid, but I doubt that github is allocating unlimited space on their systems for open source projects to put binaries on.

With my new model, every release would get a unique identifier to be used both in the title and the release tag.
https://github.com/HiranChaudhuri/oolit ... l.yml#L146
With such distinct release tags Github can build a release history since we no longer move tags and replace releases, as can be seen here:
https://github.com/HiranChaudhuri/oolite/releases

If it is confirmed that there are no issues with quotas for such history of binaries, fine. But I would prefer to check up on this one first just in case.

Note that the release 1.91-15 currently has a green indicator 'Latest', but that is not the tag.
https://github.com/HiranChaudhuri/oolit ... ag/1.91-15

So if you want to have a permanent link always pointing to the latest release, Github provides it in means of

Code: Select all

curl https://api.github.com/repos/HiranChaudhuri/oolite/releases/latest

See also https://docs.github.com/de/rest/release ... st-release

Github also shows the latest release on the project's front page like this:

I guess this will be affected if a commit to master generates a new latest release. This will no longer show 1.90, it will show whatever is in master at the time. Even if there is a direct github api link to the actual 1.90, the front page will no longer show that.

All I'm trying to say is that versioning and release/prerelease schemes should be given a bit more thought. My recommendation would be to:
- Make a PR just for the linux automated build, without changing anything else. Getting linux builds back is far more important IMO than anything right now and is the primary objective of this exercise. Such a PR would immediately get green lit.
- If you feel that the semantic versioning works better, test and submit a PR for that once fully happy that it is ready. I think that should be OK overall, we can test it out and see how it works.
- Plan the release scheme a bit more thoroughly, ensuring that we will not hit any walls with github itself if it is implemented as proposed. From my point of view, having nightly releases only for master would suffice. If anyone wants to have a particular feature in a branch tested, they can always generate a manual build. We've done this in the past; it works and it is not that often that it happens anyway. I would not mind having branch prereleases there though, as long as they are marked separately and do not interfere with the master commits.

[hiran]

All I'm trying to say is that versioning and release/prerelease schemes should be given a bit more thought. My recommendation would be to:
- Make a PR just for the linux automated build, without changing anything else. Getting linux builds back is far more important IMO than anything right now and is the primary objective of this exercise. Such a PR would immediately get green lit.
- If you feel that the semantic versioning works better, test and submit a PR for that once fully happy that it is ready. I think that should be OK overall, we can test it out and see how it works.
- Plan the release scheme a bit more thoroughly, ensuring that we will not hit any walls with github itself if it is implemented as proposed. From my point of view, having nightly releases only for master would suffice. If anyone wants to have a particular feature in a branch tested, they can always generate a manual build. We've done this in the past; it works and it is not that often that it happens anyway. I would not mind having branch prereleases there though, as long as they are marked separately and do not interfere with the master commits.

Good idea to separate issues. The discussion has drifted to version numbers, not on the linux release itself.
I will align the release handling to the current nightly situation and create a new PR.

[tsoj]

Unfortunately, I failed with adding all required libraries to the Linux nightly releases, but the only package a user had to install is probably just "gnustep-base", so I think that's reasonable.

[hiran]

Good idea to separate issues. The discussion has drifted to version numbers, not on the linux release itself.
I will align the release handling to the current nightly situation and create a new PR.

The new PR is out.

[hiran]

Unfortunately, I failed with adding all required libraries to the Linux nightly releases, but the only package a user had to install is probably just "gnustep-base", so I think that's reasonable.

How did you detect the required libraries? How did you test the result?

[another_commander]

The new PR is out.

The first release with both Windows and Linux builds is out on github. Well done!

[tsoj]

How did you detect the required libraries? How did you test the result?

So my original idea (simplified) was to run

Code: Select all

ldd ./oolite.app/oolite

to get all the required libraries that should go into the installer archive. However, even when I did this, on another machine (with Fedora) it complained that some other shared library that ldd didn't show (librt.so or something) wanted a glibc version that was wrong.

In the end when I gave up I just wanted to see which packages I had to install on the (mostly untouched) Fedora system to get the generated nightly to work. And after fixing the error "could not find gnustep-base.so" (or something like that) by running "sudo dnf install gnustep-base" it didn't throw any errors anymore.

[hiran]

The new PR is out.

The first release with both Windows and Linux builds is out on github. Well done!

Thank you for merging.

I shall bug you about feature-branch builds and version numbering soon...

[hiran]

How did you detect the required libraries? How did you test the result?

So my original idea (simplified) was to run

Code: Select all

ldd ./oolite.app/oolite

to get all the required libraries that should go into the installer archive. However, even when I did this, on another machine (with Fedora) it complained that some other shared library that ldd didn't show (librt.so or something) wanted a glibc version that was wrong.

In the end when I gave up I just wanted to see which packages I had to install on the (mostly untouched) Fedora system to get the generated nightly to work. And after fixing the error "could not find gnustep-base.so" (or something like that) by running "sudo dnf install gnustep-base" it didn't throw any errors anymore.

That sounds similar to what I tried at some point in time.

Be aware that the libraries we fetch may also have dependencies. So this is a recursive activity to get them all.
Then there is the task of ensuring that at runtime Oolite would receive from ld the libraries we supply, not the ones from the OS.

And finally some of the old libraries we distribute may have been compiled against a really old kernel and glibc, and that's where the whole approach breaks. I believe over many years there was some kind of backwards compatibility, but that may be fading out. I am a bit clueless here.

My attempt of compiling Oolite on Linux is stored in https://github.com/HiranChaudhuri/OoliteBuilder
In this while loop I tried to resolve all the dependencies and still did not get far. Today I am happy we have something running on Ubuntu 22.

Could it be that the only way forward is to update Oolite so it no longer requires these old shared libraries?

[Getafix]

...
While you are here, could you enlighten us a bit? In the past you were able to build so many versions of Oolite for so many operating systems.
We tried to recreate the builds, but so far we only have success on Ubuntu/Debian latest. How did you manage so many different OS?
...

To cut the long story short :

Environment Preparation
I used make-install to build from source the libraries that can be found at /deps/Linux-deps/{x86, x86_64}/lib/. These libraries (following to experimentations with 10,000 combinations that didn't work ) consist the minimum set that proved to be compatible with a multitude of distros for many years.
Note: Some of them libraries (the most low-level ones; I don't remember which ones, almost 15 years have passed since then ) were not built from source. They were taken as binaries from the Ubuntu 8.04LTS environment (the environment that was also used to build the Oolite releases).

Making Oolite Binary
When "making" the oolite executable, the linker references the libraries provided by the Linux O/S environment (remember that I did a make-install). The libraries in the /deps/Linux-deps/{x86, x86_64}/lib/ folder are just a copy of the ones that have been built from source with make-install. The libraries in that folder are not referenced during oolite binary making; keep reading.

Packaging Oolite Installer
The libraries in the /deps/Linux-deps/{x86, x86_64}/lib/ folder are only used during packaging the installer. They are just copied in the oolite installer package.

Oolite Installation
The installer extracts the packaged libraries either
in the ~/GNUstep/Applications/Oolite/oolite-deps/lib/ folder (home-directory installation type)
or
in the /opt/Oolite/oolite-deps/lib/ folder (system-wide installation type).

Oolite Execution
When you execute oolite, you actually execute a wrapper Linux script. This script, amongst other things and before calling the actual oolite binary, sets the following environment variables

Code: Select all

export LD_LIBRARY_PATH=${OOLITE_ROOT}/oolite-deps/lib
export ESPEAK_DATA_PATH=${OOLITE_ROOT}/oolite.app/Resources/

Then the wrapper script executes the oolite binary. The runtime linker first looks for dependency libraries in the LD_LIBRARY_PATH and only if a library is not found there, then it looks in the system folders.
In that way the oolite binary uses the libraries we deliver with the installer wherever it is installed.

Tadaaa!!!

[hiran]

To cut the long story short :

Environment Preparation
I used make-install to build from source the libraries that can be found at /deps/Linux-deps/{x86, x86_64}/lib/. These libraries (following to experimentations with 10,000 combinations that didn't work ) consist the minimum set that proved to be compatible with a multitude of distros for many years.
Note: Some of them libraries (the most low-level ones; I don't remember which ones, almost 15 years have passed since then ) were not built from source. They were taken as binaries from the Ubuntu 8.04LTS environment (the environment that was also used to build the Oolite releases).

Making Oolite Binary
When "making" the oolite executable, the linker references the libraries provided by the Linux O/S environment (remember that I did a make-install). The libraries in the /deps/Linux-deps/{x86, x86_64}/lib/ folder are just a copy of the ones that have been built from source with make-install. The libraries in that folder are not referenced during oolite binary making; keep reading.

Packaging Oolite Installer
The libraries in the /deps/Linux-deps/{x86, x86_64}/lib/ folder are only used during packaging the installer. They are just copied in the oolite installer package.

Oolite Installation
The installer extracts the packaged libraries either
in the ~/GNUstep/Applications/Oolite/oolite-deps/lib/ folder (home-directory installation type)
or
in the /opt/Oolite/oolite-deps/lib/ folder (system-wide installation type).

Oolite Execution
When you execute oolite, you actually execute a wrapper Linux script. This script, amongst other things and before calling the actual oolite binary, sets the following environment variables

Code: Select all

export LD_LIBRARY_PATH=${OOLITE_ROOT}/oolite-deps/lib
export ESPEAK_DATA_PATH=${OOLITE_ROOT}/oolite.app/Resources/

Then the wrapper script executes the oolite binary. The runtime linker first looks for dependency libraries in the LD_LIBRARY_PATH and only if a library is not found there, then it looks in the system folders.
In that way the oolite binary uses the libraries we deliver with the installer wherever it is installed.

Tadaaa!!!

That is a perfect summary, thank you so much!

And I believe that some of the libraries - as they are compiled against or bundled from a Ubuntu 8 installation have that old glibc version as a dependency. One of which is no longer compatible with today's kernels.

Do you think you could help me and tsoj to translate this into something working on newer distributions? I would then wrap it so it runs on GitHub without manual intervention.

[Getafix]

...
And I believe that some of the libraries - as they are compiled against or bundled from a Ubuntu 8 installation have that old glibc version as a dependency. One of which is no longer compatible with today's kernels.
...

Could you, please, elaborate on that?
Have we reached a distro where Oolite for Linux cannot be executed due to the old glibc version dependency?