deploySite task in github workflow failing

[phkb]

I think I had this issue before when I was trying to package a ZIP of the Mac 1.90. The one that didn't work was zipped in Windows. The one that worked was zipped on the Mac itself. Apparently if you unzip with a different utility the app will work. Testing now.

[phkb]

Extracting with different archive utility didn't work. But I believe my premise is the answer: we will need to zip up the package on a Mac. Something is getting lost on the app when you zip on Windows or Linux.

[phkb]

It works!
Someone with a real Mac needs to confirm with these downloads:
Standard release: oolite-1.90a.zip
Test release: oolite-1.90a-Mac-TestRelease.zip

With many thanks to MrFlibble for coming up with the idea, and to Timer for adding the new URL.

[MrFlibble]

I'd simply dug into the zip using mc's VFS rather than extract and repack. Didn't think about zip version issues. Well caught phkb!!

I'll crack on with the Linux versions next. If I can just whip the libraries into good order as I did on my fork of 1.91.

[MrFlibble]

I've modded the Linux versions.

These downloads will self-destruct in 15 days.

32 bit:
oolite-1.90.linux-x86.tgz (103.77 MB)
oolite-1.90-test.linux-x86.tgz (103.99 MB)

64 bit:
oolite-1.90.linux-x86_64.tgz (104.2 MB)
oolite-1.90-test.linux-x86_64.tgz (104.43 MB)

To do this, I chopped the self-extract archive apart with sed into the script and tarball, extracted the tarball, tweaked the oolite executable, repacked the tarball, then used md5sum & cksum on that, altered the MD5, CRC, and filesizes in the self extract script before cat-ing them back together. Then of course tarballed the resulting self-extracting lump.

It looks like the bundled libraries are good. I tested the x86_64 on a fairly plain LinuxMint 22.1 and no complaints. The x86 version worked fine on a fresh 32 bit install of LMDE6. Should work on almost any distro from the last 5 or so years.

I suppose the source tarball ought to be adjusted too. If the automatic build actions gubbins on github can be held-back to avoid rebuilding 1.90 with newer a distro, (which would mess up the library bundle), would it make sense to fix it in the github source so that things like the flatpak build can inherit the corrected URL?

[hiran]

I suppose the source tarball ought to be adjusted too. If the automatic build actions gubbins on github can be held-back to avoid rebuilding 1.90 with newer a distro, (which would mess up the library bundle), would it make sense to fix it in the github source so that things like the flatpak build can inherit the corrected URL?

Not sure I am getting this right. The source code on Github already contains an updated URL for downloading expansions. You can see it here.

As Oolite builds on free Github services there are limitations in the build environment. However it is possible to add runners with custom operating systems so the build scripts could be changed to build on older OS.

I believe going for the right packaging format is the better alternative. There are flatpak, appimage and snap - maybe more.

[MrFlibble]

Not sure I am getting this right. The source code on Github already contains an updated URL for downloading expansions. You can see it here.

As Oolite builds on free Github services there are limitations in the build environment. However it is possible to add runners with custom operating systems so the build scripts could be changed to build on older OS.

I believe going for the right packaging format is the better alternative. There are flatpak, appimage and snap - maybe more.

Good points hiran. I know the current source has the current URL.

Here's my take on the situation. Sorry if it rambles a bit...

1.90

If downloading the source tarball from the 1.90 release page is the URL corrected? Same question re: the 1.90 git 'snapshot'.

There are many links to the 'stable' 1.90 release, and that's likely where noob user will land:-

Changing the 1.90 source on git cannot be the solution. That will cause an automatic rebuild, at a new version, which wouldn't be on the same runners as the original 1.90 build, breaking backward compatibility and Linux deps.

Rather than someone hosting custom runners for Mac and Linux (32/64) to do this one-off thing, and to retain backward compatibility, I've hex-edited the binaries. I couldn't figure out how to do that for the windows ones, perhaps a rebuild of those (in isolation) would not break backward compatibility, not something I know.

If my patched installers are OK, I'd be delighted to see them replacing the files on the releases page for 1.90. That same page could have the text changed to clue the new user up (in big flaming letters) that the windows version needs the 'user edit' (and/or Starter) to get the expansions working.

A perhaps more git-ish approach would be a 'source and windows only' 1.90-1 release which literally only changes that URL to the same as I've used, and manually put my tweaked Lin/Mac binaries in that release page. Remove 'broken' files from the 1.90 release page, and put a pointer (in big flaming letters) in the text there to the 1.90-1 re-release. Links elsewhere on t'web can then be tweaked over time to point at 1.90-1. Any links we don't find, or which are beyond our control, will still ultimately get the user to the correct place.

Using 1.90-1 as 'stable', and integrating Flatpak there could be a path to getting Oolite listed on flathub. No need to build the other Linux versions, as the tweaked existing bundles are already "just right".

Beyond 1.90-1 and toward 1.92

Right packaging, I agree. That's a subjective minefield though.

We're not doing a Linux 32 version for current releases. I don't know if there's any solid reason, other than that github doesn't have 32 bit runners now. Nobody seems to have grumbled, though people tend to "not come back" rather than grumble. Debian/LMDE and many other distros are still supporting 32 bit.

I've probably asked this before.. Can builds be done using an older/other distro via chroot/proot in a current github runner?

The current 'install to home' self extractor is pretty good. Familiar to existing users, user friendly-ish, largely dep free, reasonably universal. It just needs the library bundle fixed to be good for older and newer OS-es alike. Not difficult. I've updated the linux-deps lib bundle in my fork, and am about to re-focus on that for the new "oldest runner".

Appimage has great potential. It's tidy, and for the user it requires no learning beyond basic operating system stuff most users will already know. Download, chmod +x or make executable with GUI file manager, enjoy. Bonus... Using Appimage could be a way to automatically work out the linux-deps for the existing self-extract installer. Appimage promotes backward compatibility, though doing it optimally might involve someone hosting a runner (or chroot/proot!?). It's trivial to have a swathe of versions "installed" too.

Flatpak is convenient, albeit heavy-ish on users system resources. The current Flatpak test has the bonus of mostly not conflicting with non-Flatpak Oolite installations, by having it's own expansions dir. Bonus... if it get listed on flathub, it'll appear in the GUI software manager on some systems.

Not sure about snap, never needed to use it.

Finally, in an ideal world we'd also use native packaging (rpm,pkg,deb etc.) for all major platforms in their current supported versions, but of course, nobody has the time/inclination to cover them all. If somebody (else?) did the groundwork to conjure a deb package during build, and could convince Debian/Ubuntu based distro maintainers to add "stable" to their package managers, we might get a lot more casual Linux users aboard via their fluffy GUI Software Managers, and more 'proper' users, via "apt install oolite". Building a deb file for current and previous Ubuntu on the releases page would be a step toward having a PPA, which in turn is the least resource heavy for users' systems.

[hiran]

Here's my take on the situation. Sorry if it rambles a bit...

1.90

If downloading the source tarball from the 1.90 release page is the URL corrected? Same question re: the 1.90 git 'snapshot'.

I take it that the download links would all point towards the Github releases page. Or do they point to the download archives directly?

Changing the 1.90 source on git cannot be the solution. That will cause an automatic rebuild, at a new version, which wouldn't be on the same runners as the original 1.90 build, breaking backward compatibility and Linux deps.

We could branch off for 1.90 and add a fix, adding the correct tag for the new version. Maybe it even gets built successfully on Github with the known limitations (only current operating systems, no apple, ...). If that ends in build artifacts added to the releases page, we can delete them manually.
Also we can manually upload releases - that that's what we should do with your patched artifacts.

Thus do not fear the Github workflow, as we are still in control.

Rather than someone hosting custom runners for Mac and Linux (32/64) to do this one-off thing, and to retain backward compatibility, I've hex-edited the binaries. I couldn't figure out how to do that for the windows ones, perhaps a rebuild of those (in isolation) would not break backward compatibility, not something I know.

That is powerful magic you applied.

A perhaps more git-ish approach would be a 'source and windows only' 1.90-1 release which literally only changes that URL to the same as I've used, and manually put my tweaked Lin/Mac binaries in that release page. Remove 'broken' files from the 1.90 release page, and put a pointer (in big flaming letters) in the text there to the 1.90-1 re-release. Links elsewhere on t'web can then be tweaked over time to point at 1.90-1. Any links we don't find, or which are beyond our control, will still ultimately get the user to the correct place.

For a moment, take a step back and look why you patched the releases: We cannot build on Apple Mac at all, and the Linux builds only work on the platform Oolite was compiled on. Well, not even for the latest Ubuntu versions. Which means on Windows any user can directly download the 1.91 builds and get going.

Using 1.90-1 as 'stable', and integrating Flatpak there could be a path to getting Oolite listed on flathub. No need to build the other Linux versions, as the tweaked existing bundles are already "just right".

Yup, that's the idea behind these packaging formats.

We're not doing a Linux 32 version for current releases. I don't know if there's any solid reason, other than that github doesn't have 32 bit runners now. Nobody seems to have grumbled, though people tend to "not come back" rather than grumble. Debian/LMDE and many other distros are still supporting 32 bit.

I've probably asked this before.. Can builds be done using an older/other distro via chroot/proot in a current github runner?

If we do not want to set up custom runners yet want to deviate from the Github-supplied ones, I see only one option:
Build inside docker containers. That would allow to use different OS flavors (if not different Linux flavors) but I doubt we could vary the CPU architecture or kernel version. A stunt like that would require building inside Qemu. Not impossible but needs to be configured and maintained.

Appimage has great potential. It's tidy, and for the user it requires no learning beyond basic operating system stuff most users will already know. Download, chmod +x or make executable with GUI file manager, enjoy. Bonus... Using Appimage could be a way to automatically work out the linux-deps for the existing self-extract installer. Appimage promotes backward compatibility, though doing it optimally might involve someone hosting a runner (or chroot/proot!?). It's trivial to have a swathe of versions "installed" too.

I have seen quite many projects providing appimages, so yes I like it. Sometimes even more than Snap or Flatpak: Appimages are straightforward to understand for users.

Flatpak is convenient, albeit heavy-ish on users system resources. The current Flatpak test has the bonus of mostly not conflicting with non-Flatpak Oolite installations, by having it's own expansions dir. Bonus... if it get listed on flathub, it'll appear in the GUI software manager on some systems.

I am wondering about the 'need' to use their repository, plus the need to have the flatpak be built out of our control. See that other thread for more information.

Not sure about snap, never needed to use it.

Snap seems to be Ubuntu's way of "neutrally" packaging applications. Take it with a grain of salt.

[MrFlibble]

I take it that the download links would all point towards the Github releases page. Or do they point to the download archives directly?

I think a lot of the links out there are platform agnostic pointers to the 1.90 release page on github rather than individual files. Retaining the filenames/paths may be a good plan, just in case.

We could branch off for 1.90 and add a fix, adding the correct tag for the new version. Maybe it even gets built successfully on Github with the known limitations (only current operating systems, no apple, ...). If that ends in build artifacts added to the releases page, we can delete them manually.
Also we can manually upload releases - that that's what we should do with your patched artifacts.

Thus do not fear the Github workflow, as we are still in control.

This is somewhat akin to my 1.90-1 suggestion.

I only feared the automatic borkage of the main stable release. Fools rush in etc. Would a newly rebuilt Windows version (built with the current github windows runner) remain as backward compatible as the previous build do you think?

Rather than someone hosting custom runners for Mac and Linux (32/64) to do this one-off thing, and to retain backward compatibility, I've hex-edited the binaries.

That is powerful magic you applied.

I was surprised nobody else thought to try it.

A perhaps more git-ish approach would be a 'source and windows only' 1.90-1 release which literally only changes that URL to the same as I've used, and manually put my tweaked Lin/Mac binaries in that release page. Remove 'broken' files from the 1.90 release page, and put a pointer (in big flaming letters) in the text there to the 1.90-1 re-release. Links elsewhere on t'web can then be tweaked over time to point at 1.90-1. Any links we don't find, or which are beyond our control, will still ultimately get the user to the correct place.

For a moment, take a step back and look why you patched the releases: We cannot build on Apple Mac at all, and the Linux builds only work on the platform Oolite was compiled on. Well, not even for the latest Ubuntu versions. Which means on Windows any user can directly download the 1.91 builds and get going.

1.90 works fine on most current distros, because the libraries bundled with it were contemporary with the runner of the day, which is now older than anything it's likely to be wanted on. It works on nearly any distro 'since' that build.
1.91 is where the headaches are, because linux-deps hasn't had any love. Also, unless we build on the oldest available runner, it won't work on currently supported but 'previous' distros, like Linuxmint 21. I've worked around that on my fork, though need to redo it before I consider a PR.
While I revisit that, I may try and write a script to copy the libraries during the build. That makes more sense to me than having a separate project, and reduces manual involvement, to once per runner version change. The runner may be updated before build (?), so bundling the libraries by copy from runner will be more robust than using "last weeks" libraries.

I've probably asked this before.. Can builds be done using an older/other distro via chroot/proot in a current github runner?

If we do not want to set up custom runners yet want to deviate from the Github-supplied ones, I see only one option:
Build inside docker containers. That would allow to use different OS flavors (if not different Linux flavors) but I doubt we could vary the CPU architecture or kernel version. A stunt like that would require building inside Qemu. Not impossible but needs to be configured and maintained.

Hmm.. I hadn't thought of Docker TBH. Cool. If that'd work, I guess chroot/proot would too. Good to sort of know.
I wasn't looking to do different arch. That could conceivably be done more efficiently with cross-compiling anyway.

Flatpak is convenient, albeit heavy-ish on users system resources. The current Flatpak test has the bonus of mostly not conflicting with non-Flatpak Oolite installations, by having it's own expansions dir. Bonus... if it get listed on flathub, it'll appear in the GUI software manager on some systems.

I am wondering about the 'need' to use their repository, plus the need to have the flatpak be built out of our control. See that other thread for more information.

I've had a passing involvement in that thread.
If it's on Flathub, it might well show up in GUI software managers (like LinuxMint's one), for anyone searching "immersive space game" or similar on their desktop. Also it will allow automatic updates to work properly, rather than manual download.
Installing a flatpak manually from a file is not as easy as I'd hoped

Not sure about snap, never needed to use it.

Snap seems to be Ubuntu's way of "neutrally" packaging applications. Take it with a grain of salt.

Yes, I think that's why it was dropped as a default from LinuxMint.