Building with MSYS2 and Mingw64 (almost!)

[Mauiby de Fug]

Some weeks ago I was having some problems building Oolite on Arch Linux, and so I thought I'd look at the Windows build environment that I had that was working for some clues as to why. The downloadable Windows dev environment is great, in that it just works and everything is all set up. However, it annoys me a little in that it's a bit of a black box and it would be nice to be able to configure an environment from scratch.

The current setup uses the old MSYS. The lastest version is MSYS2, which is also available in GitHub Actions. So I thought I'd give it a go to try and build Oolite with the more modern tools and see how far I got. And learn a few things along the way!

After some days spent chasing one error after another, trying to build various dependencies, I finally managed to get Oolite to successfully compile. Note that I say compile, not run. Oolite then shouted at me that it couldn't find various DLLs, and as I tried to help it do so, I eventually got to an unhelpful message about an "entry point not found". So I went nuclear and copied all DLLs from the /bin/ folder into the Oolite.app folder, which, while not allowing the program to run, stopped those messages. I've since narrowed things down using lld and now only copy about half the folder's DLLs instead.

As I was falling asleep last night, it occurred to me to try running my build through gdb to see if I get any useful messages. And... Success! Ish. It runs! I can save and load games, and fly a Cobby! But only if I make a debug build, and only if I run that debug build with gdb. Running it without gdb gives me a split second of the splash screen and no more. What's more, there are messages in the latest.log:

Code: Select all

21:05:11.579 [dataCache.notFound]: No data cache found, starting from scratch.
21:05:11.580 [dataCache.write.buildPath.failed]: Could not create folder C:/msys64/home/Robert/oolite/oolite.app/GNUstep/Library/Caches.
...
21:05:36.221 [dataCache.write.buildPath.failed]: Could not create folder C:/msys64/home/Robert/oolite/oolite.app/GNUstep/Library/Caches.
21:05:36.221 [dataCache.write.failed]: Failed to write data cache.
21:05:56.187 [oxz.manager.error]: Could not create folder C:/msys64/home/Robert/oolite/oolite.app/GNUstep/Library/ApplicationSupport/Oolite/ManagedAddOns.
21:05:56.187 [oxz.manager.error]: Unable to create installation folder.

Current thoughts are that there's something I haven't configured with the GNUstep build, or that I need to go all the way and not use anything that's pre-built in the oolite-windows-dependencies repository. If the latter is the case, then I've potentially managed to make some further progress in that - using the mingw64 versions of libpng, openal and libvorbis. Not yet managed are espeak and the js library - I'm saving that pain for last!

It's been an educational little project. So far along my unexpected journey, I've re-learned about make and makefiles, learned how to use wget, tar, and patch, have sort-of learned more about compiler flags, and for a nice diversion, learned more about GitHub Actions and some of the different things one can do, and have duplicated my own computer's efforts online!

My current attempts can be found at https://github.com/Bzomak/oolite-msys2. It's a fairly empty repository - there's a shell script that replicates my efforts, and then everything else can be found in the various workflows for GitHub Actions, which I've been experimenting with. I feel like I've barely left the Shire, and already "here be dragons"!

If anyone has any ideas on how to get any further, I'm all ears!

[hiran]

Is it possible that you have the executable but no complete installation and now the resource files are missing?

Nevertheless I an now confused wheher you are building on arch, on Windows or using the Windows build environment in arch.

Please take a look om the developer's corner where we try to collect good information:
https://wiki.alioth.net/index.php/Developer%27s_Corner

It can also help to see how we build on Ubuntu:
https://github.com/OoliteProject/oolite ... ll.yml#L59

[another_commander]

This is a monumental effort taking place here and it warrants a long, detailed response. If it works out, it will be huge. Please give me some time and I will get back to you with a few thoughts on this.

[Mauiby de Fug]

Is it possible that you have the executable but no complete installation and now the resource files are missing?

There's no issue with the Oolite resource files - they appear without issue. What doesn't appear are the GNUstep folders, which ought to be created on a first run. That and the messages in the latest.log leads to my suspicion that something is wrong with my GNUstep build. Possible avenues of investigation that occurred to me after a good night's sleep include:

• Building with the patched GNUstep version 1.20.1, which I'd originally avoiding doing because I wanted to try building with up-to-date libraries that hopefully had our bugfixes/patches in already, and as it was early on in my attempts I hadn't worked out an easy way to script edits or use patches yet. Now I have that experience I could give it a go.
• Building libobjc2 and then building GNUstep with that, rather than using the provided gcc. When dabbling early on I couldn't get this to work so moved on...

Nevertheless I an now confused wheher you are building on arch, on Windows or using the Windows build environment in arch.

I'm building on Windows, using MSYS2 https://www.msys2.org/ and the MINGW64 environment. I've reproduced my successful steps so far with the shell script https://github.com/Bzomak/oolite-msys2/ ... mingw64.sh, which also has some comments reminding me what I'm trying to do and why. I've also managed to duplicate this to work with GitHub actions, both by simply calling the shell script, and also by splitting everything down into separate jobs, as seen at https://github.com/Bzomak/oolite-msys2/ ... -split.yml

Regarding my issue building on Arch Linux, it was a few months ago, so I can't guarantee precise details, but if I recall correctly, I tried building from the AUR https://aur.archlinux.org/packages/oolite, but ran into a build error for something that had already been fixed in the repositories. I looked to see if I could edit the PKGBUILD to build from the latest pre-release (off topic, but having a latest release released with a fixed name, i.e. latest, meant there was a predictable download link that scripts can use, which is not the case with the current situation when released with the build as the name) rather than the release of 1.90.4, but we don't as of yet provide the full source code in the download (again, I might be wrong, but I think that when I looked, the attached Source.zip only contains the source code for the Oolite repository, and not any of the submodules). I then moved to just checking out the Oolite repository and building myself, which worked, apart from the issues with libpng. These have been documented by others elsewhere, but it involves a mismatch between libpng14 and libpng16 (building with one and running with the other). As mentioned elsewhere on the boards and on the AUR, removing the png headers worked. However, in theory, the solution ought to have been to be able to make a deps-release, rather than a release, so that it builds and runs with the provided libpng14. But for some reason the Makefiles were not setting the HOST_ARCH variable and so it couldn't find the deps. So then I thought I'd have a look on Windows, since I already had the downloadable working build environment there, and then got distracted and went down this rabbit hole! At some point I'll go back and have a look and see if I can figure out what to do to fix it, now I've learnt/re-learnt a lot about building and libraries and Makefiles and such, and if I get anywhere, I'll file a bug report/fix at the appropriate location. In the meantime, I've got rather invested in this Windows build attempt!

Please take a look om the developer's corner where we try to collect good information:
https://wiki.alioth.net/index.php/Developer%27s_Corner

It can also help to see how we build on Ubuntu:
https://github.com/OoliteProject/oolite ... ll.yml#L59

I'm familiar with the current build systems. I've been studying them in great detail (as well as the ones for the various GNUstep libraries and all their respective HOW-TOs, READMEs and installation instructions). If anything actually comes of this project then the hope is that they'd eventually become obsolete.

The end goal here is to have an up-to-date and reproducible build environment, rather than using the current system https://github.com/OoliteProject/oolite ... -build-env which is essentially a mirror of an ancient system on a_c's computer that's been in use for so long that we're unable to set it up from scratch (or at least, not without considerable effort) and so must copy it. When you managed to set this up on GitHub Actions it was a massive step up from where we previously were - having Oolite automatically build and released for each new commit is absolutely fantastic. I'm attempting to build on your concept and hard work, but by taking out the pre-built stuff and the old environment, and by setting up each dependency as a clear and separate job, which would then hopefully make it easier to update and test those as well. If it's not present in the repositories, such as GNUstep, then as you can see the aim is to build it from source.

(For any kind of distribution, then even if a library is present in the repositories, we may need to be careful about versioning, as the MSYS2 repositories do update. I believe that it's possible to install older versions from the archives, but I haven't experimented with that yet. But that sort of thing is a long way in the future, and only worth looking into if this project manages to go anywhere.)

Hope that makes things a little clearer! Until yesterday, I didn't have anything tangible to show for my efforts. Since I now have a way of making Oolite run, albeit not fully and only under certain conditions, I thought I'd share my work and see if the experts who actually know what they are doing can offer any tips! It seems to have attracted a_c's interest; I'll sit back and patiently wait for him to find some time to see if he can make head or tail out of what I've done. And if it ultimately all comes to naught, then no worries, as I've definitely learnt a lot!

[hiran]

AC is right. You attempt another leap which might be more than 7 light years ahead.

Well I am not good with c and Makefiles but if I can help...

For the Linux version, as an alternative to debugging you could use strace. Something similar might exist in Windows.

And the GNUstep folder with ingredients - IIRC it gets created and populated by the installer, not the executable you compiled.

[another_commander]

And the GNUstep folder with ingredients - IIRC it gets created and populated by the installer, not the executable you compiled.

The GNUstep folder and it's subfolders get created by the game on its first run, not by the installer.

[hiran]

And the GNUstep folder with ingredients - IIRC it gets created and populated by the installer, not the executable you compiled.

The GNUstep folder and it's subfolders get created by the game on its first run, not by the installer.

I am somewhat surprised. In Ubuntu the whole oolite.app folder including the resources and the program binary are underneath $HOME/GNUStep. So how can this directory structure get created by the program binary?

But right now I have no proof.

[phkb]

I am somewhat surprised. In Ubuntu the whole oolite.app folder including the resources and the program binary are underneath $HOME/GNUStep. So how can this directory structure get created by the program binary?

On Windows, the GNUStep folder is a subfolder of the Oolite installation directory, and isn't created until you run the program once. For other OS's, not sure.

[another_commander]

@Mauiby: Now that I got some time to sit down and think a bit, here are some comments and advice on your endeavor.

First of all, thanks for doing this. Knowing what a colossal amount of work it is, I was never going to feel ready to take on the task myself for one more time. What you are doing now is something I did already thrice in the past (first time with GCC 3.4.2, then with GCC 4.3.2 and finally with GCC 4.7.1, which is the ancient version of GCC we use today) and I think it was enough. But before going further, let me lay down the basic philosophy behind the development of the development environment, so that you will have a solid view of what the main targets are. I will not be able to give technical advivce on your current effort, but maybe I will be able to give a few pointers for getting it to where I believe it has potential to reach.

When I first got involved in the project development on Windows was just starting to get to a stable result. It was shortly after the release of 1.65 and people were already hacking away on the Windows side of things back then. Test executables were being posted in the forun with each one bringing in new features and ideas. It was great, but it didn't take long before an apparent issue became evident: People compiling on Windows were using ad-hoc dev setups with no control or thought on what libraries were used for building and how they interacted with each other. The result was that we had executables being posted but very often those exevutables could not be run. Sometimes it was a conflict with the libraries distributed with the official exe, sometimes it was newer versions of those libraries being used for compiling and sometimes it was that older versions of the libraries were used for compiling. To be able to run a test executable, one had to replicate the libraries setup of the person providing the modified game executable in the exact same way, otherwise no test was possible. Imagine for a bit the Linux distros situation happening on the Windows builds. This was clearly not viable. We needed one dev setup, as automated as it could be possibly be, that anyone could easily use and not have to worry about the supporting libraries. Essentially an attempt at standardization was needed, so that we could all work from the same starting base. And that was the main idea behind the birth of the dev environment we use today.

So there are a few goals that must be achieved for the dev environment in order for it to be functional and achieve its purpose. The first goal is that all the support libraries for the game should ideally be built with that same environment. During Oolite's development we had to interfere with the support libaries code many times and fix bugs, backport features from later versions or simply tailor features to fit what we wanted to do with Oolite. Building all libraries from source enables us not only to do all that, but to also be able to fully debug the code, even in the case of very obscure and rare bugs. If you have binaries of the debug libraries built with debug symbols, you can basically obtain game executable's stack backtraces that dive into the actual code of those libraries and pinpoint exactly where things went wrong. You would not be able to do this with prebuilt libraries, as the stack backtrace stops containing useful information once prebuilt library code is encountered. In addition to all that, building all support libaries with the same dev environment ensures that we will avoid another major trap (which you may have found out already), exception models.

Exception models: Oolite makes heavy use of exceptions. This allows for graceful exit if something goes wrong, instead of a straight crash to desktop. In other cases exceptions allow the game to continue running, even in a non-optimal state, instead of terminating unexpectedly. But here is the catch: It is important that all program modules use the same exception model. GCC 4.7.1, used for building Oolite on Windows, uses the sjlj (short jump - long jump) exception model, therefore building all the libraries with it ensures that they all run under the same model. I guess the GCC you use for your effort is higher than version 10. GCC after version 4.8 switched its Windows exception handling to the much more modern and better SEH (Structured Exception Handling) model. The problem of course, is that the two models are not compatible. So using your newly built executables with the support libraries distributed with Oolite v1.90 is not something I would really recommend. This would result in mixing exception handling models and this usually results in runtime errors about __gxx_personality_sj0 or __gxx_personality_seh0 stuff. If you had any of those coming up during your efforts, that was the reason for them. I am actually quite surprised that the game manages to run with the mixing of libraries you did at some point. But regardless, the safest option is to build all libraries yourself and ensure that all the changes mentioned in the document ExternalLibrariesSourceCodeChanges.txt are inlcuded in your builds.

The second goal is that all external libraries must be tested and confirmed stable at one particular version and then have that version locked for the next few years. If you do not do this and continuously update your libraries, you will end up maintaining the libraries instead of the game. That's what most Linux maintainers of Oolite used to do for their distros and that's what we totally want to avoid on Windows. It is my understanding that Msys2 will try to pull the latest versions of libraries when using its package manager, but that may not be necessarily what we want. Keep this in mind, it will be best if you decide on a version for each library and stay on that. This is extremely important especially for the Spidermonkey library, which should NOT be updated to any later version than the one we currently use. Doing so may invalidate any number of OXPs due to potential changes in the scripting API and we cannot afford that. At this point I would like to add that Spidermonkey is a very tough nut to crack and, if you do manage to build it, I would totally recommend adding the JS_THREADSAFE build option for Windows, which we currently don't have. At the moment we do not have a need for JS_THREADSAFE in the current Oolite anyway so that's OK, but having it available could in theory help a lot with parallelization of AIs' execution, which would be a major potential future optimization for the game.

The third goal is to keep the entire setup as easy to use as possible. I don't have any major advice on that, as you seem to be on top of this and doing a good job from what I've seen. If it works out, we should be able to build the game with just a few commands, like we do now. So keep the focus on that part too.

The task is very difficult and very technical, as you have already obderved. But it is also EXTREMELY rewarding if it does get us somewhere at the end. Succeeding in this upgrade will mean that we can use more advanced and modern programming APIs, utilize newer technologies and renew interest for Oolite. It will be totally amazing and the point you have reached now is already much further than I thought anyone would be able to get to. Hopefully the little essay above will be useful and helpful in keeping the effort on track. I will add thoughts that might have been missed as they come up. Good luck!

[hiran]

I believe you two should gang up and collaborate. This will likely be a breakthrough.

Anyone alone on the task will more likely break before.

[Mauiby de Fug]

@Mauiby: Now that I got some time to sit down and think a bit, here are some comments and advice on your endeavor.

First of all, thanks for doing this.

You're most welcome! I tend to have far too many hobbies and interests to focus on all of them at once, but at the moment I'm back in a programming mindset. I'm finding this project interesting, educational, and most usefully: has a defined goal! Without anything definitive to aim for I have a tendency for things to fizzle out, so this gives me something to focus on. And having made far more progress than I too had initially expected, I hope to be in it for the long haul!

So there are a few goals that must be achieved for the dev environment in order for it to be functional and achieve its purpose. The first goal is that all the support libraries for the game should ideally be built with that same environment. During Oolite's development we had to interfere with the support libaries code many times and fix bugs, backport features from later versions or simply tailor features to fit what we wanted to do with Oolite. Building all libraries from source enables us not only to do all that, but to also be able to fully debug the code, even in the case of very obscure and rare bugs. If you have binaries of the debug libraries built with debug symbols, you can basically obtain game executable's stack backtraces that dive into the actual code of those libraries and pinpoint exactly where things went wrong. You would not be able to do this with prebuilt libraries, as the stack backtrace stops containing useful information once prebuilt library code is encountered.

At the moment I've been working on switching out the various deps for ones from MSYS2 where I can. Not yet done are espeak and Spidermonkey (which I'm not looking forward to much, given the comments you've made about its difficulty in multiple places on the forum in the past!).

Question: are all the dependencies currently used compiled with debug symbols? If so then I'll start working my way down the tree working out how to compile everything from source. Which I'm not averse to - we already have to for GNUstep and SDL anyway!

In addition to all that, building all support libaries with the same dev environment ensures that we will avoid another major trap (which you may have found out already), exception models.

Exception models: Oolite makes heavy use of exceptions. This allows for graceful exit if something goes wrong, instead of a straight crash to desktop. In other cases exceptions allow the game to continue running, even in a non-optimal state, instead of terminating unexpectedly. But here is the catch: It is important that all program modules use the same exception model. GCC 4.7.1, used for building Oolite on Windows, uses the sjlj (short jump - long jump) exception model, therefore building all the libraries with it ensures that they all run under the same model. I guess the GCC you use for your effort is higher than version 10. GCC after version 4.8 switched its Windows exception handling to the much more modern and better SEH (Structured Exception Handling) model. The problem of course, is that the two models are not compatible. So using your newly built executables with the support libraries distributed with Oolite v1.90 is not something I would really recommend. This would result in mixing exception handling models and this usually results in runtime errors about __gxx_personality_sj0 or __gxx_personality_seh0 stuff. If you had any of those coming up during your efforts, that was the reason for them. I am actually quite surprised that the game manages to run with the mixing of libraries you did at some point. But regardless, the safest option is to build all libraries yourself and ensure that all the changes mentioned in the document ExternalLibrariesSourceCodeChanges.txt are inlcuded in your builds.

Indeed, the default GCC version in MSYS2 at the moment is version 13.2, so just a little bit higher! I'll be honest, I'm a bit surprised that the game managed to run with the mixture too. Although obviously not properly and only in debug mode so far.

Question: the exception model used is obviously relevant when compiling - are there likely to be any related code changes needed in the source code or is it just about the compiler's output? (I appreciate that here you'd only be able to talk about Oolite and not any other libraries...)

The second goal is that all external libraries must be tested and confirmed stable at one particular version and then have that version locked for the next few years. If you do not do this and continuously update your libraries, you will end up maintaining the libraries instead of the game. That's what most Linux maintainers of Oolite used to do for their distros and that's what we totally want to avoid on Windows. It is my understanding that Msys2 will try to pull the latest versions of libraries when using its package manager, but that may not be necessarily what we want. Keep this in mind, it will be best if you decide on a version for each library and stay on that. This is extremely important especially for the Spidermonkey library, which should NOT be updated to any later version than the one we currently use. Doing so may invalidate any number of OXPs due to potential changes in the scripting API and we cannot afford that. At this point I would like to add that Spidermonkey is a very tough nut to crack and, if you do manage to build it, I would totally recommend adding the JS_THREADSAFE build option for Windows, which we currently don't have. At the moment we do not have a need for JS_THREADSAFE in the current Oolite anyway so that's OK, but having it available could in theory help a lot with parallelization of AIs' execution, which would be a major potential future optimization for the game.

If I say that my preferred Linux flavour is Arch, then you'll be able to infer that I like things to be the latest shiny version possible. As long as it works. If I ever get to a point where I have a viable working build, then we'd indeed need to lock down specific versions of all libraries. The hope is that with careful build system design it would be easier for devs to have the ability to test and upgrade dependencies with confidence. But at a time of their choosing - not being forced upon them by what's available on their distro.

It looks like it is possible to install specific older versions of packages on MSYS2 (https://www.msys2.org/docs/package-mana ... e-packages), albeit with more care required. On the other hand, it's possible that if we build everything from source from, say the original git repository, then this may not be needed.

The third goal is to keep the entire setup as easy to use as possible. I don't have any major advice on that, as you seem to be on top of this and doing a good job from what I've seen. If it works out, we should be able to build the game with just a few commands, like we do now. So keep the focus on that part too.

Thanks! As I think you can tell, that's one of my goals here too. The aim is to eventually get it so that for a dev to build on their own machine, all they would need to do is download the base MSYS2 environment from their website, and then download a shell script into it and run it, which would take care of all the dev tools, dependencies etc and then be able build Oolite. There would also be a comprehensive workflow setup on GitHub Actions, to provide the same functionality that Hiran has setup for us now but using these latest tools.

The task is very difficult and very technical, as you have already obderved. But it is also EXTREMELY rewarding if it does get us somewhere at the end. Succeeding in this upgrade will mean that we can use more advanced and modern programming APIs, utilize newer technologies and renew interest for Oolite. It will be totally amazing and the point you have reached now is already much further than I thought anyone would be able to get to. Hopefully the little essay above will be useful and helpful in keeping the effort on track. I will add thoughts that might have been missed as they come up. Good luck!

Again, thank you! It's reassuring that I'm not completely barking up the wrong tree and that it's a worthwhile endeavour. If I make any further progress I'll be sure to let you all know!

[hiran]

I configured Doxygen to also generate documentation for the Mac-specific code. The result is visible here:
https://ooliteproject.github.io/oolite/files.html

The interesting part might be the call graphs. If they are not completely off, it looks like some of the functions are never called. Could that mean the code is not required at all?

Thinking about that: Why should a posix build not work on MacOS? Yes, it may sound weird to develop in Objective-C and then run GNUStep on a Mac, but could that remove one big dependency?