Windows MSYS2 MinGW64 environment development

[another_commander]

Would it be possible to include a UCRT clang in the dev environment and build the game using that?

I am thinking MingW64 with msvcrt and clang with ucrt as two distinct compilers for development, with the clang/ucrt one being the default for distribution on Windows. Not too savvy with Msys2 setup to attempt it, though. Thoughts?

[mcarans]

Would it be possible to include a UCRT clang in the dev environment and build the game using that?

I am thinking MingW64 with msvcrt and clang with ucrt as two distinct compilers for development, with the clang/ucrt one being the default for distribution on Windows. Not too savvy with Msys2 setup to attempt it, though. Thoughts?

It may be possible. If there are issues, I suspect they would be with building the JavaScript engine since espeak-ng, GNUStep and SDL already have MSYS2 UCRT packages. This would be another reason either to upgrade or change the JavaScript engine or to fully integrate the existing engine into the Oolite build process, modernising and removing all magic. I have started a discussion on this topic here. Your input would be welcome.

[another_commander]

Just try building js with clang/ucrt first (may need to specify clang as c++ compiler in the makefile) and see what happens. I don't think that there would be issues with it but if it fails, then try with mingw64/ucrt, I would expect that to work fine. Should be a relatively quick check.

[mcarans]

Just try building js with clang/ucrt first (may need to specify clang as c++ compiler in the makefile) and see what happens. I don't think that there would be issues with it but if it fails, then try with mingw64/ucrt, I would expect that to work fine. Should be a relatively quick check.

I can see the advantage of being able to have a gcc build in one environment and a Clang build in another. Either gcc or clang can be used in UCRT64 (the default environment) so probably best to try building the Spidermonkey js engine in that environment rather than the CLANG64 environment (where presumably Clang must be used). (As an aside, I noticed that QuickJS-ng switched their Windows release from MinGW64 to UCRT64, though they run tests with both)

I'll add trying the current Spidermonkey on UCRT64 to the list of things to look at but you're also welcome to try in the meantime by running the following in the folder where the PKGBUILD is located:

Code: Select all

makepkg -s -f --noconfirm

The fix for the bitfield issue : 1 has been released so I also need to update the libobjc2 PKGBUILD. I also need to remove espeak-ng and pcaudiolib from the build and install the standard packages instead.

[another_commander]

Here is a link to the new development environment I currently use featuring MinGW64/MSVCRT and Clang/UCRT. You can use it in a similar fashion to the legacy dev environment. Download, extract anywhere and run:

• For MinGW64/MSVCRT configuration ---> <FolderYouExtractedTo>\DevEnvMSVCRT\msys64\mingw64.exe
• For Clang/UCRT configuration ---> <FolderYouExtractedTo>\DevEnvUCRT\msys64\ucrt64.exe

You can create shortcuts to the two executables to make startup of the desired configuration easier if you want.

To build Oolite, make sure you have checked out the latest master revision. You can build just like before, no changes there. The build process can recognize the new environments and chose the right code paths to get you the latest and greatest game executable.

[Commander_X]

I tried a local mingw64 build using mcarans mingw64_environment.
Couple of things:
- I used an (older, but not that old -- 4.4.23(2) msys2 bash, 10.1.0 gcc) local mingw64 environment I'm using for other windows Linux-like tools I like to build.
- Being a Linux Slackware user, I didn't pacman anything, although I read the .PKGBUILD files if/when needed (mostly for either dependencies, or configuration hints)
- The environment is hosted on a (can call it) air-gaped VM, that is, a Windows 10 VM that has no clue or permission to know where the internet is, or how could it get there
I went for the gcc flavor.
The good news: most of the dependencies for the new build could be nicely built. This includes SDL, spidermonkey (JS, used as suggested the pre offered nspr), pcaudiolib, espeak-ng (including it's sonic dependency), gnustep-make (which is usually not a trouble maker).
The bad news: gnustep-base doesn't seem to want to cooperate.
I'm using lib-base-master_gcc.zip.
First thing, starting with line 6505 in its configure script, it seems

Code: Select all

LIBOBJC=`gnustep-config --objc-libs`

is returned as an empty string (error message:

Code: Select all

checking the Objective-C runtime... GNU
configure: error: The command 'gnustep-config --objc-libs' (provided by the gnustep-make package) returned no Objective-C library.  Unable to continue configuring without Objective-C support.

). Can bypass it, in interactive shell it is not empty, and a) commenting out the line above in configure script, and b) exporting the interactive shell LIBOBJC variable, does the job. Configure is done.
Unfortunately, when it tries to link it (I think) this is what comes out:

Code: Select all

 Compiling file GSFFIInvocation.m ...
/usr/bin/sh: -c: line 1: syntax error: unexpected end of file
make[4]: *** [z:/path/to/mingw/mingw64/share/GNUstep/Makefiles/Instance/library.make:301: obj/libgnustep-base.dll.a] Error 1
make[3]: *** [z:/path/to/mingw/mingw64/share/GNUstep/Makefiles/Instance/library.make:286: internal-library-all_] Error 2
make[2]: *** [z:/path/to/mingw/mingw64/share/GNUstep/Makefiles/Master/rules.make:297: libgnustep-base.all.library.variables] Error 2
make[1]: *** [z:/path/to/mingw/mingw64/share/GNUstep/Makefiles/Master/library.make:37: internal-all] Error 2
make: *** [z:/path/to/mingw/mingw64/share/GNUstep/Makefiles/Master/serial-subdirectories.make:53: internal-all] Error 2

GSFFIInvocation.m gets compiled.
I'm starting my MSYS2 bash / MINGW64 session using mintty console handler, and my best guess here is that the substitution in rules.make file (lasting from line 297 to line 352) somehow doesn't get through.
I've tried this with both libs-base-base-1_30.0.tar.gz and gnuestep-base-1.31.1.tar.gz source trees, with the same result.
Any hints would be more than welcome.

[mcarans]

I tried a local mingw64 build using mcarans mingw64_environment.
Couple of things:
- I used an (older, but not that old -- 4.4.23(2) msys2 bash, 10.1.0 gcc) local mingw64 environment I'm using for other windows Linux-like tools I like to build.
- Being a Linux Slackware user, I didn't pacman anything, although I read the .PKGBUILD files if/when needed (mostly for either dependencies, or configuration hints)
- The environment is hosted on a (can call it) air-gaped VM, that is, a Windows 10 VM that has no clue or permission to know where the internet is, or how could it get there
I went for the gcc flavor.
The good news: most of the dependencies for the new build could be nicely built. This includes SDL, spidermonkey (JS, used as suggested the pre offered nspr), pcaudiolib, espeak-ng (including it's sonic dependency), gnustep-make (which is usually not a trouble maker).
The bad news: gnustep-base doesn't seem to want to cooperate.
I'm using lib-base-master_gcc.zip.
First thing, starting with line 6505 in its configure script, it seems

Code: Select all

LIBOBJC=`gnustep-config --objc-libs`

is returned as an empty string (error message:

Code: Select all

checking the Objective-C runtime... GNU
configure: error: The command 'gnustep-config --objc-libs' (provided by the gnustep-make package) returned no Objective-C library.  Unable to continue configuring without Objective-C support.

). Can bypass it, in interactive shell it is not empty, and a) commenting out the line above in configure script, and b) exporting the interactive shell LIBOBJC variable, does the job. Configure is done.
Unfortunately, when it tries to link it (I think) this is what comes out:

Code: Select all

 Compiling file GSFFIInvocation.m ...
/usr/bin/sh: -c: line 1: syntax error: unexpected end of file
make[4]: *** [z:/path/to/mingw/mingw64/share/GNUstep/Makefiles/Instance/library.make:301: obj/libgnustep-base.dll.a] Error 1
make[3]: *** [z:/path/to/mingw/mingw64/share/GNUstep/Makefiles/Instance/library.make:286: internal-library-all_] Error 2
make[2]: *** [z:/path/to/mingw/mingw64/share/GNUstep/Makefiles/Master/rules.make:297: libgnustep-base.all.library.variables] Error 2
make[1]: *** [z:/path/to/mingw/mingw64/share/GNUstep/Makefiles/Master/library.make:37: internal-all] Error 2
make: *** [z:/path/to/mingw/mingw64/share/GNUstep/Makefiles/Master/serial-subdirectories.make:53: internal-all] Error 2

GSFFIInvocation.m gets compiled.
I'm starting my MSYS2 bash / MINGW64 session using mintty console handler, and my best guess here is that the substitution in rules.make file (lasting from line 297 to line 352) somehow doesn't get through.
I've tried this with both libs-base-base-1_30.0.tar.gz and gnuestep-base-1.31.1.tar.gz source trees, with the same result.
Any hints would be more than welcome.

Hmm, that isn't an error I've seen before. GNUstep make and base build from source for me without requiring modifying configure or make files. Since you have taken the flags to use from the PKGBUILD files, they should work although I've not tested on older MSYS2.

What happens if you run the lines below (assuming gnustep-make was configured to use /usr/local)?

Code: Select all

source /usr/local/share/GNUstep/Makefiles/GNUstep.sh
gnustep-config --objc-flags

Since you're using gcc, you should not see -fobjc-runtime=gnustep-2.2 which is for clang.

If you then type gnustep-config --objc-libs, what do you get?

AI gives a plausible reason for the failure consistent with your best guess:

"This is a classic error when compiling large projects (like GNUstep Base) on Windows/MinGW.

The cause is almost certainly the Windows Command Line Length Limit.
The Diagnosis

The error /usr/bin/sh: -c: line 1: syntax error: unexpected end of file means that the shell received a command that was incomplete or cut off in the middle.

The Context: You are at the linking stage (obj/libgnustep-base.dll.a).

The Reality: GNUstep Base consists of hundreds of source files. When make tries to link them into a single library, it expands a variable (like $(OBJ_FILES)) that lists every single compiled object file.

The Limit: Windows has a strict limit on the length of a command line string (usually 32,767 characters).

The Failure: The list of files generated by make exceeded this limit. The command passed to the shell was truncated (cut off) simply because it was too long.

Result: The shell likely received a command that looked like: gcc -shared -o lib.dll file1.o ... file500.o "some_opt (cutoff).

Because the command ended abruptly (perhaps inside a quote or before a required closing delimiter), sh threw a "syntax error: unexpected end of file".

How to Confirm

Run the make command again with verbose messaging enabled to see the actual command being executed. You will likely see it print a massive wall of text and then stop abruptly.
Bash

make messages=yes

How to Fix It
1. Enable "Response Files" (Best Fix)

Modern compilers (GCC/Clang) and Makefiles support "response files" to get around the Windows length limit. Instead of passing 500 filenames on the command line, make writes the filenames to a text file (e.g., files.txt) and passes @files.txt to the linker.

GNUstep should do this automatically on Windows, but your configuration might be missing it.

Check your config: Ensure you are using the MinGW-specific configuration of gnustep-make.

Update GNUstep Make: If you are using an older version of gnustep-make, update it.

2. Check for Spaces in Paths

You are building in z:/path/to/mingw/.... If any folder in that path contains a space (e.g., z:/path/to/my mingw/), it effectively doubles the complexity of the command line quoting.

Fix: Ensure your build directory and source directory paths contain no spaces.

3. Use a shorter path (Workaround)

Since the limit is based on total characters, deep directory structures eat up the buffer.

Current: z:/path/to/mingw/mingw64/share/GNUstep/...

Better: Move your project to something like C:/gs/src.

This is a temporary workaround, but it often reduces the character count enough to let the command squeeze under the 32k limit."

[Commander_X]

[...]
make messages=yes
[...]

This was what I was looking for (each build system, lately, has an obscure way of enable/disable verbosity of its works).
I knew that if I could capture the link command line I could make it work (I've done it so many times with cygwin, that it became sort of routine).
But I didn't need it. Funny enough, just passing the argument to make, overcome the issue.

I looked a bit closer to what "gnustep-config --objc-libs" returned, and I think I understand what the trouble might be. Along with all the nice linking options, it includes a path that's trying to point to my msys2 $HOME (should be as simple as -L~/GNUstep/Library/Libraries). Somewhere in the process, some script decided/decides(?) to use "cygpath -w" on $HOME, thus, what should be -Lz:/path/to/home/user/GNUstep/Library/Libraries, becomes -Lz:\path\to\home\user/GNUstep/Library/Libraries. Output of gnustep-config doesn't show any \ character, but I think this actually breaks both the configure flow, and the make command. The working make command (with verbose=yes) captures and shows the whole "-Lz:\path\to\home\user/GNUstep/Library/Libraries" parameter, in all its wrongness
Now proceeding with Oolite compilation.

[Commander_X]

[...]
Now proceeding with Oolite compilation.

Whew, quite the ride!
After couple of ins/outs with wingdi.h (in order to bring in the previous environment's DISPLAYCONFIG stuff), finally managed to get oolite.exe, of course, by using again "messages=yes" for linking
And, of (the same) course, it didn't work.
It turned out that some of the dependencies I installed to get the modern Oolite's dependencies rolling (e.g gnutls, openal, vorbis and their friends), were from a different (a bit more recent) version of mingw64 than the one I have installed. Lucky, though, I had a backup done prior to staring this adventure.
Some hours later, a brand new "oolite -hdr" rocked my boat!

[another_commander]

Here is a link to the new development environment I currently use featuring MinGW64/MSVCRT and Clang/UCRT. You can use it in a similar fashion to the legacy dev environment. Download, extract anywhere and run:

• For MinGW64/MSVCRT configuration ---> <FolderYouExtractedTo>\DevEnvMSVCRT\msys64\mingw64.exe
• For Clang/UCRT configuration ---> <FolderYouExtractedTo>\DevEnvUCRT\msys64\ucrt64.exe

You can create shortcuts to the two executables to make startup of the desired configuration easier if you want.

To build Oolite, make sure you have checked out the latest master revision. You can build just like before, no changes there. The build process can recognize the new environments and chose the right code paths to get you the latest and greatest game executable.

Dev Environment updated to contain the static libjs, which will soon be required to build the modern flavor. Also, Clang is now version 21.1.8. You can download from the same google drive link as above.

[another_commander]

Another update of the modern Dev Environment is up at the same google drive link. This one includes the debug JS libraries necessary for an Oolite debug build to succeed.