Page 2 of 12
Re: Changing Oolite...
Posted: Fri Jan 07, 2022 7:13 am
by another_commander
hiran wrote: ↑Thu Jan 06, 2022 8:10 pm
I need help to refine them so they can actually perform successful builds.
I'm afraid I cannot be of much help with this. I can only speak about the Windows builds and all I can say is that I have serious doubts that a successful nightly can be generated. The dev environment used on Windows is a very specifically set up version of an older MinGW, aimed at giving you an executable Oolite installer at the end of the build process with just one command, when starting from a clean master branch checkout. Trying to build in any other environment, like a standard MinGW install without the exact necessary libraries for linking is not guaranteed to work as intended. To add to that, MSVC (accesible in various versions from within github actions) is right out of the question for building Oolite. You are more than welcome to try if you want, but I can already see blocking issues in the horizon. The reason it worked in the nightly generation system we had until recently is that a copy of the "correct" build environment was running on terrastorage, the nightly production server which was recently decommissioned, so a successful build was always guaranteed. If github could get access somehow to that specific build environment and its actions somehow be made to make use of it, then there could be a chance. Unfortunately, I haven't got the slightest idea how one would go about it.
Mac builds may also be another issue because the Xcode version targeted for generating them is an old one and I can see already the github actions in your PR are complaining about it. There is code somewhere in the Oolite repository for updating Xcode support to newer versions but we never used it because it would have broken compatibility with the Xcode we were using for making the Mac nightlies back then, which meant that the Mac nightlies would have stopped being generated much earlier than they finally did.
Linux, with its standard gcc support and not requiring a specific dev environment setup, might actually be the one that has the highest chance of working. It is also your OS of choice, so it should be theoretically easier to work with that anyway. And yes, DevOps is hard.
Re: Changing Oolite...
Posted: Fri Jan 07, 2022 8:05 am
by hiran
another_commander wrote: ↑Fri Jan 07, 2022 7:13 am
hiran wrote: ↑Thu Jan 06, 2022 8:10 pm
I need help to refine them so they can actually perform successful builds.
I'm afraid I cannot be of much help with this. I can only speak about the Windows builds
With that you know definitely more than I do. And I never mentioned it would be an easy ride. But together I think we can make it happen.
hiran wrote: ↑Thu Jan 06, 2022 8:10 pm
Mac builds may also be another issue because the Xcode version targeted for generating them is an old one and I can see already the github actions in your PR are complaining about it. There is code somewhere in the Oolite repository for updating Xcode support to newer versions but we never used it because it would have broken compatibility with the Xcode we were using for making the Mac nightlies back then, which meant that the Mac nightlies would have stopped being generated much earlier than they finally did.
This can be an even bigger issue since today I do not know anyone with programming knowledge who could help looking at the Mac builds.
hiran wrote: ↑Thu Jan 06, 2022 8:10 pm
Linux, with its standard gcc support and not requiring a specific dev environment setup, might actually be the one that has the highest chance of working. It is also your OS of choice, so it should be theoretically easier to work with that anyway. And yes, DevOps is hard.
While Linux is my chosen operating system it does not mean I develop on it. But I agree this may be the lowest hanging fruit of all three build environments.
The reason why I am looking at this is:
I feel the motivation to make/keep Oolite a really nice game has not died. But every single change to the codebase involves heavy lifting to get it built, let alone the insecurity of having broken something across the many platforms the project supports/used to support. With that, necessary maintenance will not happen, and with that the code will become both uncompilable and also not executable.
By not taking action we would not only give up all the enrgy we spent on this, but also all the energy that others have invested.
My idea is not to take over development. I cannot do this anyway. But I can help to cover the heavy lifting in case the codebase gets changed.
Thus I want to revive automatic builds. And I want to provide automatic tests.
With that, changing Oolite would get as easy as pushing the changes on a new branch, waiting for the builds and tests to complete and look at the reports to make a merge decision. It will not resolve problems in the software but restore maintainability. You probably know very well what I am talking about.
How about we create a branch and try to make Windows builds happen?
Re: Changing Oolite...
Posted: Fri Jan 07, 2022 8:18 am
by another_commander
hiran wrote: ↑Fri Jan 07, 2022 8:05 am
How about we create a branch and try to make Windows builds happen?
This would require more time than I can dedicate at this point, sorry. You already have a branch in your fork of the game though, where you could experiment with the Windows builds freely. What I could do from my side, is upload a copy of the full Windows development environment somewhere so that you can gain access to it. From that point on, you would need to find a way for github actions to "see" this dev environment and utilize it in the build actions.
Edit: I can also provide you with write access to the main Oolite repository if you think that this might help too.
Re: Changing Oolite...
Posted: Fri Jan 07, 2022 8:49 am
by hiran
another_commander wrote: ↑Fri Jan 07, 2022 8:18 am
hiran wrote: ↑Fri Jan 07, 2022 8:05 am
How about we create a branch and try to make Windows builds happen?
This would require more time than I can dedicate at this point, sorry. You already have a branch in your fork of the game though, where you could experiment with the Windows builds freely. What I could do from my side, is upload a copy of the full Windows development environment somewhere so that you can gain access to it. From that point on, you would need to find a way for github actions to "see" this dev environment and utilize it in the build actions.
Edit: I can also provide you with write access to the main Oolite repository if you think that this might help too.
My own repo is a branch - that is right. I am experimenting already.
Somehow I'd like to setup the whole build environment from scratch and automated so we have some documentation what is actually configured how.
And right now I am scratching my head how to get mingw installed without GUI interraction...
Re: Changing Oolite...
Posted: Fri Jan 07, 2022 8:54 am
by another_commander
Link to the full Oolite Windows Development Environment (contains everything, including the sources of all the dlls that accompany the executable of the game (yes, we build those specifically for Oolite too) and the necessary files and folders to generate installers - the complete deal):
https://we.tl/t-gFRocUi1dt
The link is valid for 7 days. To launch the environment, execute
[InstallFolder]\DevelopmentEnvironment\gcc\Msys_x2\1.0\msys.bat
This is the only environment guaranteed to build Oolite from source. Other MinGW versions will most likely not cut it. Newer MinGW versions may have additional problems with the exception model in use. Oolite 64-bit and all its dependencies are built with the older sjlj exception model, while new MinGWs use the (better) SEH exception model for 64-bit Windows. You either have to build
everything from scratch using a compiler following the SEH model or you have to use this environment, otherwise expect linking problems, or, if it finally links somehow, expect crashes where exceptions would normally occur.
Re: Changing Oolite...
Posted: Fri Jan 07, 2022 9:47 am
by hiran
another_commander wrote: ↑Fri Jan 07, 2022 8:54 am
Link to the full Oolite Windows Development Environment (contains everything, including the sources of all the dlls that accompany the executable of the game (yes, we build those specifically for Oolite too) and the necessary files and folders to generate installers - the complete deal):
https://we.tl/t-gFRocUi1dt
The link is valid for 7 days. To launch the environment, execute
[InstallFolder]\DevelopmentEnvironment\gcc\Msys_x2\1.0\msys.bat
This is the only environment guaranteed to build Oolite from source. Other MinGW versions will most likely not cut it. Newer MinGW versions may have additional problems with the exception model in use. Oolite 64-bit and all its dependencies are built with the older sjlj exception model, while new MinGWs use the (better) SEH exception model for 64-bit Windows. You either have to build
everything from scratch using a compiler following the SEH model or you have to use this environment, otherwise expect linking problems, or, if it finally links somehow, expect crashes where exceptions would normally occur.
Sounds like I hit an iceberg. But you warned me.
Download is running...
Re: Changing Oolite...
Posted: Fri Jan 07, 2022 11:37 am
by hiran
It seems to me that this is a huge chunk that just needs to be unzipped to some windows machine.
How can we achieve that on Github runners? we'd need some fileserver to host that deliverable.
Would it be feasible to create a git repository with all these files, and it has 'releases' that resemble the zip we have now?
Then some Oolite build could initialize the runner by downloading this build environment release...
Re: Changing Oolite...
Posted: Fri Jan 07, 2022 2:21 pm
by Commander_X
Some observations:
- Linux can compile quite nicely Windows binaries (mingw as a cross compiler si supported -- mostly -- out of the box in a large number of distributions). This might be an option to unify the two GNUstep/SDL builds.
- Mac builds would need someone more than programming knowledgeable, but also "dedicated" to the task. I tried the build in BigSur VM I mentioned (w/ the latest Xcode), and after some tweaks/hacks, I've got an Oolite.app created. Unfortunately it doesn't start, throwing an error about some NS argument thingy. Considering the very strict version targeting option of Xcode relative to macOS, the only option I see is to replicate in GitHub (if possible) the exact versions that were used for previous builds, and cross fingers that it will work.
Re: Changing Oolite...
Posted: Fri Jan 07, 2022 2:50 pm
by hiran
Commander_X wrote: ↑Fri Jan 07, 2022 2:21 pm
Some observations:
- Linux can compile quite nicely Windows binaries (mingw as a cross compiler si supported -- mostly -- out of the box in a large number of distributions). This might be an option to unify the two GNUstep/SDL builds.
- Mac builds would need someone more than programming knowledgeable, but also "dedicated" to the task. I tried the build in BigSur VM I mentioned (w/ the latest Xcode), and after some tweaks/hacks, I've got an Oolite.app created. Unfortunately it doesn't start, throwing an error about some NS argument thingy. Considering the very strict version targeting option of Xcode relative to macOS, the only option I see is to replicate in GitHub (if possible) the exact versions that were used for previous builds, and cross fingers that it will work.
So far I have not managed any single build - regardless of the operating system. Pointers are welcome. My attempts are visible my fork of Oolite on Github.
So if cross-compiling gives better results let's go for it.
And for the Mac - you certainly progressed further than I did.
After all I have no MacOS or Windows machine to try the build. I have to club together the Github Action file and push before I can see how it behaves. This makes development of that process quite blown-up.
Re: Changing Oolite...
Posted: Fri Jan 07, 2022 3:33 pm
by another_commander
hiran wrote: ↑Fri Jan 07, 2022 2:50 pm
So if cross-compiling gives better results let's go for it..
In our case it will not, for the same reasons mentioned earlier. You need to use the exact mingw version found in the dev environment you downloaded. Using different versions will not guarantee a successful build.
Re: Changing Oolite...
Posted: Fri Jan 07, 2022 7:59 pm
by hiran
another_commander wrote: ↑Fri Jan 07, 2022 3:33 pm
hiran wrote: ↑Fri Jan 07, 2022 2:50 pm
So if cross-compiling gives better results let's go for it..
In our case it will not, for the same reasons mentioned earlier. You need to use the exact mingw version found in the dev environment you downloaded. Using different versions will not guarantee a successful build.
Ok. We can play around or optimize later. I will stick to your instructions for now.
So right now I am uploading this environment into a github repository and convert it into a release. This release can then be downloaded/installed by the Oolite builds and we should be one step further.
Re: Changing Oolite...
Posted: Sat Jan 08, 2022 3:26 pm
by hiran
another_commander wrote: ↑Fri Jan 07, 2022 8:54 am
Link to the full Oolite Windows Development Environment (contains everything, including the sources of all the dlls that accompany the executable of the game (yes, we build those specifically for Oolite too) and the necessary files and folders to generate installers - the complete deal):
https://we.tl/t-gFRocUi1dt
The link is valid for 7 days. To launch the environment, execute
[InstallFolder]\DevelopmentEnvironment\gcc\Msys_x2\1.0\msys.bat
This is the only environment guaranteed to build Oolite from source. Other MinGW versions will most likely not cut it. Newer MinGW versions may have additional problems with the exception model in use. Oolite 64-bit and all its dependencies are built with the older sjlj exception model, while new MinGWs use the (better) SEH exception model for 64-bit Windows. You either have to build
everything from scratch using a compiler following the SEH model or you have to use this environment, otherwise expect linking problems, or, if it finally links somehow, expect crashes where exceptions would normally occur.
BTW, how does msys.bat find oolite? Parameters? Current working directory?
https://github.com/HiranChaudhuri/oolit ... focus=true
edit: it seems the script automatically found the cloned oolite repository:
https://github.com/HiranChaudhuri/oolit ... focus=true
The build failed (2 commands had problems) but I do not recognize any real error message.
Re: Changing Oolite...
Posted: Sat Jan 08, 2022 3:39 pm
by another_commander
hiran wrote: ↑Sat Jan 08, 2022 3:26 pm
It doesn't. Once the shell is launched, you just need to cd into oolite's root folder and then run
make -fMakefile pkg-win-snapshot to create a nightly test release installer. Setting the default working dir automatically (so that you won't need to execute a cd command) should also be possible by editing the file DevelopmentEnvironment\gcc\Msys_x2\1.0\etc\profile and changing the line
cd "$HOME" towards the end accordingly.
Also please check
https://github.com/HiranChaudhuri/oolit ... v/issues/1 if you haven't already. The dev environment I posted had two extra newer versions of gcc which are not needed to build, but weigh at around 1.7GB. Those can be removed unless rebuilding the full Oolite game with a newer gcc version on Windows is in your future plans.
Re: Changing Oolite...
Posted: Sat Jan 08, 2022 3:47 pm
by another_commander
Note that editing etc/profile allows you to not only change directory, but execute the Oolite build automatically. Just add the make command posted above at the end of the profile file and that should kick off the build as soon as Msys starts up.
Re: Changing Oolite...
Posted: Sat Jan 08, 2022 4:31 pm
by hiran
another_commander wrote: ↑Sat Jan 08, 2022 3:47 pm
Note that editing etc/profile allows you to not only change directory, but execute the Oolite build automatically. Just add the make command posted above at the end of the profile file and that should kick off the build as soon as Msys starts up.
The file
contains these lines:
Code: Select all
# Set up USER's home directory
if [ -z "$HOME" ]; then
HOME="/home/$LOGNAME"
fi
...
cd "$HOME"
So this means to me if I prepopulate the HOME environment variable I do not have to edit the script.
But where should it point to? Or if it is just a standard Unix home directory, where inside that should Oolite be checked out?