Changing Oolite...

[hiran]

It seems to me the whole Oolite stuff is extremely volatile.

I tried now to run Oolite compiled on Ubuntu 20 on my Ubuntu 20. That seems to work.
I tried now to run Oolite compiled on Ubuntu 20 on Debian 10. That breaks and is no surprise.
I tried now to run Oolite compiled on Debian 10 on my Ubuntu 20. That segfaults if I include most libraries from Debian 10.
I tried now to run Oolite compiled on Debian 10 on my Ubuntu 20. That cannot find the gnustep library if I include just the Oolite's project's libs.
I tried now to run Oolite compiled on Ubuntu 18 on my Ubuntu 20. That cannot find the gnustep library if I include just the Oolite's project's libs.

I went into that trial phase after I read that for cross platform compatibility it is advisable to compile on the oldest still supported OS and thus increase chances the build would run on more up to date versions. Debian 10 is the base for Ubuntu 18. And I am not sure what libraries to package it with.

How has this been tackled in the past?

[another_commander]

It seems to me the whole Oolite stuff is extremely volatile.

I tried now to run Oolite compiled on Ubuntu 20 on my Ubuntu 20. That seems to work.
I tried now to run Oolite compiled on Ubuntu 20 on Debian 10. That breaks and is no surprise.
I tried now to run Oolite compiled on Debian 10 on my Ubuntu 20. That segfaults if I include most libraries from Debian 10.
I tried now to run Oolite compiled on Debian 10 on my Ubuntu 20. That cannot find the gnustep library if I include just the Oolite's project's libs.
I tried now to run Oolite compiled on Ubuntu 18 on my Ubuntu 20. That cannot find the gnustep library if I include just the Oolite's project's libs.

I went into that trial phase after I read that for cross platform compatibility it is advisable to compile on the oldest still supported OS and thus increase chances the build would run on more up to date versions. Debian 10 is the base for Ubuntu 18. And I am not sure what libraries to package it with.

How has this been tackled in the past?

This all happens because each distro has its own versions of libraries, so when you link the executable with the libraries on your build machine, there is no guarantee that the same versions will be found in other distros. It is very common for newer versions of libraries to implement new APIs or change existing ones drastically. So, you may build against e.g. libpng14, but the system you try to run the game on uses libpng16. Or maybe there is a change in the sound API from distro to distro. Now consider this for the tens of dependencies of Oolite This could explain why your Ubuntu 20-on-Ubuntu-20 combination works and everything else fails.

The only reasonable way to tackle this is what we decided and agreed to do early on in the project: Stick to very specific versions of libraries which are known to work without any problems, ensure the game is compiled against those specific dependencies and include the very same library versions used to compile and link against in the final game package that gets distributed. This is the reason why whenever there is a problem with self-compiles on Linux, the solution that works 99% of the time is to "download and use the pre-built binaries from the Oolite website". We know exactly how everything is compiled and we know that no matter what libraries exist in the user's system, the game will run using our own library versions (this is the main task of the wrapper script; to set up the closed ecosystem for the game to run in).

Compiling on the oldest supported OS is probably not as important as ensuring that you know exactly what dependencies you are building (and running) against right from the start.

[hiran]

It seems to me the whole Oolite stuff is extremely volatile.

I tried now to run Oolite compiled on Ubuntu 20 on my Ubuntu 20. That seems to work.
I tried now to run Oolite compiled on Ubuntu 20 on Debian 10. That breaks and is no surprise.
I tried now to run Oolite compiled on Debian 10 on my Ubuntu 20. That segfaults if I include most libraries from Debian 10.
I tried now to run Oolite compiled on Debian 10 on my Ubuntu 20. That cannot find the gnustep library if I include just the Oolite's project's libs.
I tried now to run Oolite compiled on Ubuntu 18 on my Ubuntu 20. That cannot find the gnustep library if I include just the Oolite's project's libs.

I went into that trial phase after I read that for cross platform compatibility it is advisable to compile on the oldest still supported OS and thus increase chances the build would run on more up to date versions. Debian 10 is the base for Ubuntu 18. And I am not sure what libraries to package it with.

How has this been tackled in the past?

This all happens because each distro has its own versions of libraries, so when you link the executable with the libraries on your build machine, there is no guarantee that the same versions will be found in other distros. It is very common for newer versions of libraries to implement new APIs or change existing ones drastically. So, you may build against e.g. libpng14, but the system you try to run the game on uses libpng16. Or maybe there is a change in the sound API from distro to distro. Now consider this for the tens of dependencies of Oolite This could explain why your Ubuntu 20-on-Ubuntu-20 combination works and everything else fails.

The only reasonable way to tackle this is what we decided and agreed to do early on in the project: Stick to very specific versions of libraries which are known to work without any problems, ensure the game is compiled against those specific dependencies and include the very same library versions used to compile and link against in the final game package that gets distributed. This is the reason why whenever there is a problem with self-compiles on Linux, the solution that works 99% of the time is to "download and use the pre-built binaries from the Oolite website". We know exactly how everything is compiled and we know that no matter what libraries exist in the user's system, the game will run using our own library versions (this is the main task of the wrapper script; to set up the closed ecosystem for the game to run in).

Compiling on the oldest supported OS is probably not as important as ensuring that you know exactly what dependencies you are building (and running) against right from the start.

What you wrote is exactly how I understand the problem. Yes.

That is why I compiled on Debian 10 and Ubuntu 18 (which is based on the latter) AND included all the libraries it depends on apart from glibc.
Why so?

Including all the libraries addresses exactly your second paragraph. That way I know the libraries the code was compiled against are available at runtime. glibc however has a strong dependendy on the running kernel, hence it makes little sense to include that as well.
As other sites state, usually newer versions of (kernel/glibc/libraries) add functionality and try to keep some API compatibility. Hence the chance to succeed by depending on the old but running on the new version is simply higher.

It is just unfortunate that I did not have the same luck as others. See my signature...

But what I had found out in a earlier post:
When compiling Oolite on Linux as it was suggested to me does not compile just with the libraries in the project. Other dependencies are also introduced. Should we now add them to the package? Should we simply create a .deb package that lists required libraries and their versions?
Should we create a plethora of builds so everybody can pick his poison?

I much better understand now why riding on top of a runtime (Python or Java) is so much smoother.

[another_commander]

I can only suggest that you study the installers of the Linux package and maybe use a VM of an older Ubuntu distro to generate the packages. Other than this very generic guidance, I'm afraid I cannot be of more help as far as Linux goes.

As for creating a plethora of builds for people to pick their poison: please no.

[Cholmondely]

builds for people to pick their poison: please no.

Are you and yours all right? Surviving the snow?

[another_commander]

Are you and yours all right? Surviving the snow?

Yes, thanks; it looks like we made it through this one.

[hiran]

I can only suggest that you study the installers of the Linux package and maybe use a VM of an older Ubuntu distro to generate the packages. Other than this very generic guidance, I'm afraid I cannot be of more help as far as Linux goes.

As for creating a plethora of builds for people to pick their poison: please no.

If only I had an entry point. There are numerous scripts available, and the command I have so far just calls the compiler but does not create a package.
For Windows builds we directly saw NSIS being called and ending up with the desired package.

[another_commander]

If only I had an entry point. There are numerous scripts available, and the command I have so far just calls the compiler but does not create a package.
For Windows builds we directly saw NSIS being called and ending up with the desired package.

Check out the Makefile in the source code root folder. It contains the commands executed for building both the Windows and Linux installers. You can see what options are available with make -fMakefile help . For the Linux builds, it appears that for e.g. a test release build (make -fMakefile pkg-posix-test) it executes

Code: Select all

installers/posix/make_installer.sh $(HOST_ARCH) $(VER) "release" ""

while for a nightly build you would apparently have to run make with the command line make -fMakefile pkg-posix-nightly, which in turn seems to execute

Code: Select all

installers/posix/make_installer.sh $(HOST_ARCH) $(VER) "release-snapshot" "nightly"

Again, I am not maintaining the Linux packages, these are just some pointers to hopefully help get an installer out. In addition to the above, I would recommend studying the actual oolite.org binary installers for Linux, see what exactly they install and where, what version of glibc is packaged (if any) etc.

[hiran]

If only I had an entry point. There are numerous scripts available, and the command I have so far just calls the compiler but does not create a package.
For Windows builds we directly saw NSIS being called and ending up with the desired package.

Check out the Makefile in the source code root folder. It contains the commands executed for building both the Windows and Linux installers. You can see what options are available with make -fMakefile help . For the Linux builds, it appears that for e.g. a test release build (make -fMakefile pkg-posix-test) it executes

Code: Select all

installers/posix/make_installer.sh $(HOST_ARCH) $(VER) "release" ""

while for a nightly build you would apparently have to run make with the command line make -fMakefile pkg-posix-nightly, which in turn seems to execute

Code: Select all

installers/posix/make_installer.sh $(HOST_ARCH) $(VER) "release-snapshot" "nightly"

Again, I am not maintaining the Linux packages, these are just some pointers to hopefully help get an installer out. In addition to the above, I would recommend studying the actual oolite.org binary installers for Linux, see what exactly they install and where, what version of glibc is packaged (if any) etc.

I tried to run

Code: Select all

make -fMakefile pkg-posix-test

With that I received an exit code 2 together with this on stderr (do not bother the many quotes - they have been added by ansible):

Code: Select all

            "/usr/share/GNUstep/Makefiles/objc.make:30: objc.make is deprecated.  Please use tool.make instead",
            "make[1]: *** No rule to make target 'deps-'.  Stop.",
            "make: *** [Makefile:218: pkg-posix-test] Error 2"

Similarly, I tried to run

Code: Select all

make -fMakefile pkg-posix-nightly

and again received exit code 2 with this on stderr:

Code: Select all

            "/usr/share/GNUstep/Makefiles/objc.make:30: objc.make is deprecated.  Please use tool.make instead",
            "make[1]: *** No rule to make target 'deps-nightly'.  Stop.",
            "make: *** [Makefile:224: pkg-posix-nightly] Error 2"

Somehow I believe we are in the deprecated area on each an every step. The language, the libraries, the build tools, ...

[another_commander]

Are you using gnustep-base 1.20 and its associated tools? That is the version we use for the builds. It looks like you are trying to use a newer gnustep.

Edit That 'deps-' should be 'deps-release'. It looks like a parsing failure somewhere.

[hiran]

Are you using gnustep-base 1.20 and its associated tools? That is the version we use for the builds. It looks like you are trying to use a newer gnustep.

Edit That 'deps-' should be 'deps-release'. It looks like a parsing failure somewhere.

I just checked the line in the makefile and it looks as you suggested. But then in stdout I have

Code: Select all

installers/posix/make_installer.sh  1.91.0.7444-220126-d52b6be \"release-snapshot\" \"nightly\"

so I guess the HOST_ARCH variable evaluated to nothing. Is that to be expected on a Linux system? I see it neither on Debian nor Ubuntu.
edit: answering my own question...
Both Makefile and GNUmakdefile contain

Code: Select all

HOST_ARCH                        := $(shell echo $(GNUSTEP_HOST_CPU) | sed -e s/i.86/x86/ -e s/amd64/x86_64/ )

and oolite.src contains

Code: Select all

HOST_ARCH=`uname -m | sed -e s/amd64/x86_64/`
if [ "$HOST_ARCH" = "x86_64" ]; then
   export GNUSTEP_HOST=x86_64-pc-linux-gnu
   export GNUSTEP_HOST_CPU=x86_64
else
   export GNUSTEP_HOST=i686-pc-linux-gnu
   export GNUSTEP_HOST_CPU=ix86
fi

The system comes with a different version of gnustep-base.

Code: Select all

$ find / -name "*libgnustep-base*"
/usr/lib/libgnustep-base.so.1.27.0
/usr/lib/libgnustep-base.a
/usr/lib/libgnustep-base.so
/usr/lib/libgnustep-base.so.1.27
/usr/share/lintian/overrides/libgnustep-base1.27
/usr/share/doc/libgnustep-base1.27
/usr/share/doc/libgnustep-base-dev
/var/cache/apt/archives/libgnustep-base-dev_1.27.0-3_amd64.deb
/var/cache/apt/archives/libgnustep-base1.27_1.27.0-3_amd64.deb
/var/lib/dpkg/info/libgnustep-base1.27.md5sums
/var/lib/dpkg/info/libgnustep-base1.27.triggers
/var/lib/dpkg/info/libgnustep-base1.27.shlibs
/var/lib/dpkg/info/libgnustep-base1.27.list
/var/lib/dpkg/info/libgnustep-base-dev.list
/var/lib/dpkg/info/libgnustep-base-dev.md5sums
/home/vagrant/oolite/deps/Linux-deps/x86/lib/libgnustep-base.so.1.20
/home/vagrant/oolite/deps/Linux-deps/x86_64/lib/libgnustep-base.so.1.20
$

[another_commander]

'deps-nightly' is also wrong. It should be 'deps-release-snapshot'. Clearly one argument in the command line fails to be recognized.

HOST_ARCH is defined in the Makefile as HOST_ARCH:= $(shell echo $(GNUSTEP_HOST_CPU) | sed -e s/i.86/x86/ -e s/amd64/x86_64/ )
What does this give you if you execute it in the shell? FYI, on Msys on Windows I get x86_64 as response (although I had to remove the parentheses from the env variable to get it to work).

Maybe gnustep-base 1.27 is not good for Oolite. It has certainly not been tested sufficiently, at least. 1.20 has.

[hiran]

'deps-nightly' is also wrong. It should be 'deps-release-snapshot'. Clearly one argument in the command line fails to be recognized.

HOST_ARCH is defined in the Makefile as HOST_ARCH:= $(shell echo $(GNUSTEP_HOST_CPU) | sed -e s/i.86/x86/ -e s/amd64/x86_64/ )
What does this give you if you execute it in the shell? FYI, on Msys on Windows I get x86_64 as response (although I had to remove the parentheses from the env variable to get it to work).

Maybe gnustep-base 1.27 is not good for Oolite. It has certainly not been tested sufficiently, at least. 1.20 has.

Code: Select all

$(shell echo $(GNUSTEP_HOST_CPU) | sed -e s/i.86/x86/ -e s/amd64/x86_64/ )

gives nothing as $(GNUSTEP_HOST_CPU) is undefined.
However the definition I found in oolite.src:

Code: Select all

$ uname -m | sed -e s/amd64/x86_64/
x86_64
$

What would I have to do to activate the 1.20 that resides in the oolite folder? Is it about setting the LD_LIBRARY_PATH before calling make?
My gut feeling is we might be better off to recreate the build system based on up to date libraries, then look into issues in the Oolite code itself.

[another_commander]

Code: Select all

$(shell echo $(GNUSTEP_HOST_CPU) | sed -e s/i.86/x86/ -e s/amd64/x86_64/ )

gives nothing as $(GNUSTEP_HOST_CPU) is undefined.

Whoa! This is not OK. GNUSTEP_HOST_CPU should be defined before the build starts, as well as a bunch of other gnustep environment variables. You are probably missing a step somewhere. Please check - on Windows we are running /mingw/../devlibs/gnustep1201/System/Library/Makefiles/GNUstep.sh inside the profile file. Make sure that this Makefiles/GNUstep.sh script runs before you start the build.

If you want to recreate the build system from scratch be my guest. But I cannot help you with that.

[hiran]

Code: Select all

$(shell echo $(GNUSTEP_HOST_CPU) | sed -e s/i.86/x86/ -e s/amd64/x86_64/ )

gives nothing as $(GNUSTEP_HOST_CPU) is undefined.

Whoa! This is not OK. GNUSTEP_HOST should be defined before the build starts, as well as a bunch of other gnustep environment variables. You are probably missing a step somewhere. Please check - on Windows we are running /mingw/../devlibs/gnustep1201/System/Library/Makefiles/GNUstep.sh inside the profile file. Make sure that this Makefiles/GNUstep.sh script runs before you start the build.

If you want to recreate the build system from scratch be my guest. But I cannot help you with that.

Well, I will not be able to recreate the build system alone.

The first variable is getting set in ./deps/Linux-deps/oolite.src
Is that one called automatically? Would I have to call it? I will add a call before running make...