dpkg-buildpackage and obsolete library files

[MrFlibble]

Having tried to use the binary installer on LinuxMint 22, I ended up manually copying libraries from an older distro into $HOME/GNUstep/Applications/Oolite/oolite-deps/lib (LinuxMint 21.3 was a suitable donor. Got it working, but came to the conclusion that the bundled libraries in the source are a bit of a mess. Some of them are obsolete and not used by Oolite, so the system versions of those libraries have to 'happen' to be the right version for it all to work. Ultimately, the distro vintage has to be 'just right' for the installer version to work without lots of tinkering.

IMO, it's clear that the libraries bundled with the Linux binary installer need an overhaul to get it to work on more recent distros. If binaries of libraries are to be bundled for compatibility, then should not all of them be bundled, and obsolete ones pruned from the installer? Bundling ALL relevant versions of libraries in the installer version will make it work on a wider range of distros, past, present, and probably future.

Given that hassle, and thinking of new users, I felt the urge to walk the path of building a deb package. If it works easily, then perhaps it might be worth churning out deb packages for a couple of current distros on the releases. A much more elegant solution (where applicable) than bundling a heap of libraries.

On the README.md at Oolite's github, the process of building a deb on Linux is described by one line:-

If you have the Debian package tools (installed by default with Debian and Ubuntu), use dpkg-buildpackage.

Hmm. I know it won't be THAT easy.

I grabbed the latest dev source tarball from the releases page, unpacked it, changed dir into the root of it, and straying from the one line clue, ran dpkg-checkbuilddeps to figure out the deps to install. It wanted to install 'python', not python3. I edited the debian/control file to change that, and did an "apt install" for the rest of the list.

Deps aboard. I then tried "dpkg-buildpackage -uc -us", (no point trying to sign it at this point).

Code: Select all

dpkg-buildpackage: error: cannot open file debian/changelog: No such file or directory

debian/changelog.in exists, but needs processing. Here's the top of the file.

Code: Select all

oolite (@@VERSION@@-@@REVISION@@) stable; urgency=low

  * Snapshot release of Oolite @@VERSION@@.
    See /usr/share/doc/oolite/changelog.gz for upstream changes.

 -- Darren Salt <[email protected]>  @@TIMESTAMP@@

So a brutal attempt at progress, knowing it's just for me, and not for publlication, I copy changelog.in to chagelog an try again.

Code: Select all

$ dpkg-buildpackage -uc -us
dpkg-buildpackage: warning:     debian/changelog(l1): version '@@VERSION@@-@@REVISION@@' is invalid: version number does not start with a digit
LINE: oolite (@@VERSION@@-@@REVISION@@) stable; urgency=low
dpkg-buildpackage: warning:     debian/changelog(l6): badly formatted trailer line
LINE:  -- Darren Salt <[email protected]>  @@TIMESTAMP@@
dpkg-buildpackage: warning:     debian/changelog(l8): found start of entry where expected more change data or trailer
LINE: oolite (1.73.3-1) stable; urgency=low
dpkg-buildpackage: warning:     debian/changelog(l8): found end of file where expected more change data or trailer
dpkg-buildpackage: info: source package oolite
dpkg-buildpackage: info: source version unknown
dpkg-buildpackage: error: version number does not start with a digit

Right. Let's just try and skip the issue.

Code: Select all

$ echo "Flibble woz 'ere" > debian/changelog
$ dpkg-buildpackage -uc -us
dpkg-buildpackage: warning:     debian/changelog(l1): badly formatted heading line
LINE: Flibble woz 'ere.
dpkg-buildpackage: warning:     debian/changelog(l1): found end of file where expected first heading
dpkg-buildpackage: error: fatal error occurred while parsing debian/changelog

So, I guess it's not a one-line job

I very rarely build deb packages, so the details are not resident in my head. I could dig deeper to find the solutions, but hope someone can give me a leg-up. Between us, perhaps we could fix the docs and source to make building a deb package trivial.

I'd hope there's merit in this, not just for individual users wanting to better use system resources, but also for distros to be able to include Oolite. Wouldn't it be good to be able to simply "apt install oolite", and/or attach a ppa to have bleeding edge versions on "apt upgrade".

[hiran]

Sounds great for me.

First figure out which libs are required/cleanup the linux-deps repo. Then think of additional packaging.

Deb is one choice. Fatpack, appimage and snaps could be more.

[MrFlibble]

Sounds great for me.

First figure out which libs are required/cleanup the linux-deps repo. Then think of additional packaging.

Deb is one choice. Fatpack, appimage and snaps could be more.

When calm and fed, I'll compare what's in my Mint 22 $HOME/GNUstep/Applications/Oolite/oolite-deps/lib with what's in the installer version as unpacked on Mint 21. That should at least give clues for those who want to move Oolite onto recent Ubuntu/Mint type setups from older ones.

I'll also have a crack at normal build on Mint 22. I've done it a few times on 21 and had no trouble.

F(l)atpack, appimage, and snaps are all horribly inefficient (on the target computer) ways to get it working on less supported or oddball distros with the benefit to the devs of only requiring one lot of effort. The benefit to non-tech users is questionable. Most Ubuntu/Mint/Similar users want to either use the Software Manager, or just "apt install thing", quite a lot don't mind adding a ppa to get there as long as the instructions are current and paste-able.

Another down-side compared to "package managed", is that the target system can end up riddled with different per-app versions of libraries etc. which never get updates or security patches. Smells like windoze!

The established Oolite installer method, albeit AFAIK abandoned upstream, is functional as long as the bundled libs get updated along the way.

I'd hoped it'd be quite easy to follow the README.md re: building a deb. If building a deb can be made functional again, surely the deps and therefore required libraries for a static 'lump' would be fairly easy to look up programmatically. Any github build/workflow could ideally spit out the debs, and then the legacy installer versions. I confess I've not scrutinised the oolite installer build scripts on github beyond quick attempts to plagiarise it for DC.

[hiran]

I think there is one more item to look at.

Despite we package some libraries through linux-deps and the installer Oolite tries to use the Ubuntu 22 (compile system) provided version. So at build time there is something to fix.

[MrFlibble]

I think there is one more item to look at.

Despite we package some libraries through linux-deps and the installer Oolite tries to use the Ubuntu 22 (compile system) provided version. So at build time there is something to fix.

Indeed. I've just done a clean install to determine my changes. libgnustep-base.so.1.28 wanted... apt install gets 1.29... bundled? 1.20. FAIL!

I expect quite a few of the bundled libs are obsolete offal... libplc, libplds ?

What I did in $HOME/GNUstep/Applications/Oolite/oolite-deps/lib to make Oolite work on LinuxMint22 (mostly trial and error at the time):

Add (copy from older system):

• libgnustep-base.so.1.28
• libicudata.so.70
• libicui18n.so.70
• libicuuc.so.70 libobjc.so.4

Optionally, remove: libgnustep-base.so.1.20 (why is that even here!)

That got it up and running. Running the installer again will replace the whole directory, so keep a copy if you're going to fix it this way!!

Running an strace -f to work out what's actually used shows loads of libs of course, including those used to run the shell. Many are provided by the system and 'happen' to work nicely with the old bundled libs. To make it fully static, or bundle all the possible libs required would be close to bundling a JEOS distro.

I put "strace -f" in $HOME/GNUstep/Applications/Oolite/oolite.app/oolite-wrapper, so just tracing the oolite executable (and child) calls to libraries. Ran oolite normally, launched, quit then ran this to get a report (below).

Code: Select all

cd ~/GNUstep/Applications/Oolite/oolite-deps/lib
UsedBundled=$(
	grep -E '^open(at)?\(.*\.so' /tmp/Oolite-strace.txt |\
	 grep -v "No such" | tr "," " " |\
	   awk '{print $2}' | sort  | uniq |\
	    grep GNUstep | sed 's@^.*/@@g' | sed 's/"//'
 )
UnusedBundled=$(
	ls | while read ; do
		echo "$UsedBundled" | grep -q $REPLY || echo $REPLY
	done | while read y ; do
		grep -q "$y" /tmp/Oolite-strace.txt || echo "$y" 
	done
)
FromSys=$(
	grep -E "^open(at)?\(.*/lib/.*\.so" /tmp/Oolite-strace.txt |\
	 grep -v ENOENT | sed 's@.*/@@;s@".*$@@' |\
	  sort | uniq | while read ; do
		[ -f $REPLY ] || echo "$REPLY"
	  done
)
echo -e "
# Used bundled libraries\n$UsedBundled\n
# Unused bundled libraries\n$UnusedBundled\n
# Used and not in bundle\n$FromSys\n"

..And here's the output.

# Used bundled libraries
libespeak.so.1
libgcrypt.so.20
libgmp.so.10
libgnustep-base.so.1.28
libgnutls.so.30
libgpg-error.so.0
libhogweed.so.4
libicudata.so.70
libicui18n.so.70
libicuuc.so.70
libnettle.so.6
libobjc.so.4
libogg.so.0
libopenal.so.1
libpng14.so.14
libportaudio.so.2
libSDL-1.2.so.0
libvorbisfile.so.3
libvorbis.so.0
libz.so.1

# Unused bundled libraries
libespeak.so.1.pulseaudio
libffi.so.4
libnsl.so.1
libnspr4.so.0d
libobjc.so.2
libplc4.so.0d
libplds4.so.0d

# Used and not in bundle
BIG5.so
CP1250.so
CP1251.so
CP1252.so
CP1253.so
CP1254.so
EUC-CN.so
EUC-JP.so
EUC-KR.so
ISO-2022-JP.so
ISO8859-10.so
ISO8859-11.so
ISO8859-13.so
ISO8859-14.so
ISO8859-15.so
ISO8859-1.so
ISO8859-2.so
ISO8859-3.so
ISO8859-4.so
ISO8859-5.so
ISO8859-6.so
ISO8859-7.so
ISO8859-8.so
ISO8859-9.so
KOI8-R.so
libasound.so.2
libavahi-client.so.3
libavahi-common.so.3
libbsd.so.0
libcap.so.2
libc.so.6
libdbus-1.so.3
libdl.so.2
libffi.so.8
libGB.so
libgcc_s.so.1
libGLdispatch.so.0
libGL.so.1
libGLU.so.1
libGLX.so.0
libicudata.so.74
libicuuc.so.74
libJIS.so
libKSC.so
liblz4.so.1
liblzma.so.5
libmd.so.0
libm.so.6
libnspr4.so
libOpenGL.so.0
libpthread.so.0
librt.so.1
libstdc++.so.6
libsystemd.so.0
libX11.so.6
libXau.so.6
libxcb.so.1
libXdmcp.so.6
libxml2.so.2
libxslt.so.1
libzstd.so.1
MACINTOSH.so
SJIS.so
UTF-16.so
UTF-32.so
UTF-7.so

I guess libespeak.so.1.pulseaudio needs to be there for non-pipewire systems. The rest of the unused bundled libs are a mystery to me.

This was done on a system where I'd already copied in the missing libraries. Also bear in mind also that I'm doing this in a VM without hardware graphics, so some other libraries may get called on a 'playable' setup, I'll try on a bare-metal install when I set one up, maybe not today.

(Edit: Tidied up and corrected the scripts, which evolved on the command line, and are not production quality.)
(Edit: Tweaked the "what I did" section.)