Building JS library from scratch

[mcarans]

Nice. Would it be possible to package the executable plus all dlls that are not inside System32 into a zip and upload it somewhere? I'd be interested in examining it.

Yes I can do that. In the meantime, I have set up a repo for the JS with simple bash scripts and build instructions here:

https://github.com/mcarans/oolite_mozjsnspr_mingw64

Also not sure if I mentioned it elsewhere, but I have the SDL repo with patches applied here:
https://github.com/mcarans/oolite_SDL-1 ... te_changes

The main branch is just a direct fork of the SDL1.2 repo, while the oolite_changes branch contains the patches. If needed, I can sync changes from the SDL repo into the main branch and then merge them into oolite_changes.

I build my Oolite with these flags: -DWINVER=0x0601 -D_WIN32_WINNT=0x0601 -DNTDDI_VERSION=0x0A00000F so that the appropriate code for HDR support from Windows system headers in MSYS2 is included without having to hack the original headers. I ensured that HDR was turned on in my Windows settings and launched Oolite with the -hdr switch and it ran ok.

[another_commander]

I build my Oolite with these flags: -DWINVER=0x0601 -D_WIN32_WINNT=0x0601 -DNTDDI_VERSION=0x0A00000F so that the appropriate code for HDR support from Windows system headers in MSYS2 is included without having to hack the original headers.

That's fine, but be aware that setting the NTDDI_VERSION to that value may raise the minimum Windows requirement to Win 11 24H1. This is actually one of the things I would like to test, as I have a spare system running Win10 here.

[mcarans]

I build my Oolite with these flags: -DWINVER=0x0601 -D_WIN32_WINNT=0x0601 -DNTDDI_VERSION=0x0A00000F so that the appropriate code for HDR support from Windows system headers in MSYS2 is included without having to hack the original headers.

That's fine, but be aware that setting the NTDDI_VERSION to that value may raise the minimum Windows requirement to Win 11 24H1. This is actually one of the things I would like to test, as I have a spare system running Win10 here.

Here is a link to the executable and dlls: https://www.filemail.com/d/mqixkqxakllbesy

On the flags we might be able to only set -DNTDDI_VERSION=0x0A00000F and put the other ones back to earlier Windows versions. I'm not sure. If it works on your Win 10 machine like this though, then no need to experiment with that.

[another_commander]

Thanks for the binaries. I ran a few tests here and the news are indeed good. Some points noted during testing, in no paricular order:

• Runs fine on Windows 10. I was a bit worried with the NTDDI version but it seems there is zero problems with that.
• Runs fine on Windows 11. I still need to properly test HDR and will do when I get back to the dev PC, but I see no reason why it would not work. On the non-HDR systems I tested on it behaved exactly as I would expect with warnings when using the -hdr switch and all.
• New SDL seems OK, switching from window to full screen and back works as expected, window resize as well. I found a small discrepancy with the standard binaries though: with the standard binaries when you go fullscreen, by design you are not allowed to use Win+Shift+left/right arrows to move the fullscreen window to a different monitor. This is not the case with the new binaries and the window can now be moved. It seems as if the WM_WINDOWPOSCHANGING event is not triggered. Not a show-stopper but it's worth investigating why there is this difference.
• Performance is the same as the original binaries, if not just marginally better. I run some stress tests using a benchmark scene and similar entities workload and monitored performance and frame pacing on multiple runs with Special K. In the benchmark scene the default binaries score on average around 147 fps, the new binaries manage approximately 151 fps. In normal gameplay there is no perceptible difference.
• Not sure if the new binary is at fault on this, but during testing with Special K injected I had a lockup and had to force-shutdown Oolite. Not sure what happened there but mentioning it because this is something I have never seen happening with the standard binaries and SK before. Note that this could easily be also a fault of SK, which updates itself quite frequently.
• Memory usage is the same as original binaries (good, that was another one of my worries), no leaks noted.
• OXP verifier works.
• Interaction with the debug console works.
• Still need to check that the expansion manager can download expansions but this will need to wait till I am back on a non-blocked connection.
• Some dlls in the package contain debug symbols (e.g. gnustep-base-1_31, nspr and possibly others too). It's fine for testing and debugging but the executable and all dlls should be stripped for actual distribution.
• It should be possible to link libgcc_s_seh-1.dll and libstdc++-6.dll statically to reduce the number of packaged dlls. Not critical though.

That's it for now, good job. If you can get espeak in there too, then I think we can declare 1:1 parity with the original binaries, after a period of testing of course to ensure that nothing unexpected pops up.

[another_commander]

Some further updates after getting back to the dev PC:

• HDR is confirmed functional.
• Unfortunately the Expansion Manager is unable to download expansions. When trying to download the OXZ list nothing happens and the following is generated in the log:

  19:15:19.773 [exception]: ***** Exception checking controls [pollDemoControls]: NSInternalInconsistencyException : Unable to find the gdnc tool.

  Attempting to download the list again results in Oolite hanging. This appears to be an issue with the menu controls most likely and not with connecting to the server. Requires investigation. Not sure what the gdnc tool is, never seen this message before.

[Lone_Wolf]

If MrFlibble has got the system espeak-ng working on Linux, maybe I can do the same in MSYS2.

An archlinux user figured out espeak-ng worked fine with oolite, I created a small patch for GNUmakefile.

MrFibble applied a similar fix to his fork.

It's a downstream patch for now as espeak is used by all oolite versions and all platforms.
Switching to a different txt-to-speech application for all platforms/targets supported by oolite requires a lot more then this simple patch.

[Commander_X]

[...] Not sure what the gdnc tool is, never seen this message before.

Can confirm also that the built worked both SDR and HDR here (Windows 10 VM).
About gdnc, I'd guess it's a tool provided with later versions of GNUstep. From the manual page:

Code: Select all

GDNC(1)                      GNUstep System Manual                     GDNC(1)

NAME
       gdnc - GNUstep Distributed Notification Center

SYNOPSIS
       gdnc

DESCRIPTION
       The  gdnc  daemon is used by GNUstep programs to send notifications and
       messages to one another.  By default it uses private inter-process com‐
       munications  accessible  only  to  the  curtrent user on the machine on
       which it is running (where the operating system supports them).

       Every user needs to have his own instance of gdnc running.  While  gdnc
       will  be  started  automatically  as soon as it is needed, it is recom‐
       mended to start gdnc in a  personal  login  script  like  ~/.bashrc  or
       ~/.cshrc.   Alternatively  (if you have no command-line tools which use
       distributed notifications) you can launch gdnc when your windowing sys‐
       tem  or the window manager is started. For example, on systems with X11
       you can launch gdnc from your .xinitrc script or alternatively - if you
       are  running  Window Maker - put it in Window Maker's autostart script.
       See the GNUstep Build Guide for a sample startup script.

This comes from libgnustep 1.28.0 version I manually installed in my Linux instance.

[Commander_X]

[...] for HDR support from Windows system headers in MSYS2 is included without having [...]

Please, please, stop using MSYS2 term for your build or toolchain. If you built it with/for MSYS2, you'll have a lot of msys_*.dll dependencies to provide, besides the msys-2.0.dll itself. Your build is mingw64 (i.e. native Windows program).
Hope I'm not offending and maybe I seem too pedantic, but the substitution is itching me the wrong way

[mcarans]

[...] for HDR support from Windows system headers in MSYS2 is included without having [...]

Please, please, stop using MSYS2 term for your build or toolchain. If you built it with/for MSYS2, you'll have a lot of msys_*.dll dependencies to provide, besides the msys-2.0.dll itself. Your build is mingw64 (i.e. native Windows program).
Hope I'm not offending and maybe I seem too pedantic, but the substitution is itching me the wrong way

Are you confusing MSYS2 with Cygwin? MSYS2 is simply a "software distribution and building platform for Windows" and "the main focus of MSYS2 is to provide a build environment for native Windows software and the Cygwin-using parts are kept at a minimum. MSYS2 provides up-to-date native builds for GCC, mingw-w64, CPython, CMake, Meson, OpenSSL, FFmpeg, Rust, Ruby, just to name a few." (https://www.msys2.org/)

MSYS2 provides a number of environments I had tried UCRT64, the default one, which is Windows native with newer universal Microsoft C++ runtime, but ran into issues building gnustep-base as I recall. Now I am using MSYS2's MINGW64 environment which also produces Windows native builds using the older msvcrt.

There are other MINGW64 pre=built toolchains, but they have far less pre-built packages than MSYS2 so a lot of dependencies would need to be compiled from scratch to use them. Hence, I chose MSYS2 for MINGW64 development of Windows native builds.

[mcarans]

[*] Unfortunately the Expansion Manager is unable to download expansions. When trying to download the OXZ list nothing happens and the following is generated in the log:

19:15:19.773 [exception]: ***** Exception checking controls [pollDemoControls]: NSInternalInconsistencyException : Unable to find the gdnc tool.

I was able to replicate the issue by not running from within an MSYS2 mingw64 terminal window. You need the gdnc.exe in the same folder as oolite.exe. Here it is: https://www.filemail.com/d/yfwsghfsqhrhfuu

For me the downloading worked, but then it failed to install the OXZ. I tried Bank of the Black Monk and it appeared to download 2.60Mb, but then said that it failed and to check latest.log. I found:

14:25:39.872 [oxz.manager.error]: Could not create folder C:/oolite/oolite.app/GNUstep/Library/ApplicationSupport/Oolite/ManagedAddOns.

When I manually created that folder and reran Oolite, the OXZ installed ok.

[*] Interaction with the debug console works.

I guess the debug console is something I don't have because I see:

14:16:24.806 [debugTCP.disconnect]: No connection to debug console: "Connection to debug console failed: 'No connection could be made because the target machine actively refused it.

' (outStream status: 1, inStream status: 7)."
14:16:24.806 [debugTCP.send.warning]: Error sending packet header, retrying.
14:16:24.831 [debugTCP.send.error]: The following packet could not be sent: {"Oolite version" = "1.91"; "packet type" = "Request Connection"; "protocol version" = 65792; }
14:16:24.832 [script.javaScript.willLoad]: About to load JavaScript ../AddOns/Basic-debug.oxp/Scripts/oolite-debug-console.js
14:16:24.834 [script.javaScript.load.success]: Loaded JavaScript: oolite-debug-console 1.91 -- Debug console script.
14:16:24.834 [debugTCP.disconnect]: No connection to debug console: "Connection to debug console failed: 'bad stream.' (outStream status: 0, inStream status: 0)."
14:16:24.834 [debugTCP.connect.failed]: Failed to connect to debug console at address 127.0.0.1:8563.

I just came across https://github.com/OoliteProject/oolite-debug-console which is new to me If I ran that, would that get rid of this error?

[Commander_X]

[...] Are you confusing MSYS2 with Cygwin? [...]

I had used Cygwin for almost 20 years (until like 5 years ago, since I switched to Linux 100% and msys2/mingw64 for my Windows works). While you hit the nail right on its head (i.e. they can be "confused" one with the other), my take is that you're confusing MSYS2 with mingw64.
Both MSYS2 and Cygwin can be used to build (mingw64) windows native target executables, but neither MSYS2 nor Cygwin are native. As you might have noticed, while there is no mingw64 bash.exe file, both MSYS2 and cygwin have it. That's because both MSYS2 and cygwin have their own POSIX emulation layer, and mingw64 doesn't.
While MSYS2 is often "packaged" as the supporting mingw64 build environment for native Windows apps (with the most notable "git windows layer"), you will always be able to get the same results using cygwin's mingw64 toolchain and noarch/mingw64 tools (of course, at slower build speeds, due to a better POSIX layer than MSYS2 has).
To lead this to an end, while both MSYS2 and cygwin have their own target builds (of msys and cygwin), which usually just copy/paste a unix/linux/bsd target, mingw64 is used for win32/win64 (native) builds. That is, all the exes/dlls in a msys2/cygwin build will end in a msys-2.0.dll/cygwin1.dll dependency, while in mingw64's case they'll end in Windows/System32 dlls. You can check this using either cygcheck or ldd on your /usr/bin/*.[exe|dll] and /mingw64/bin/*.[exe|dll] files.

[...] Hence, I chose MSYS2 for MINGW64 development of Windows native builds.

Here we agree: the builds you produced are mingw64 Windows native builds. Mentioning MSYS2 as a standalone environment could lead to confusion (and, in my case annoyance). E.g., the headers you mentioned (for HDR support) I'd bet are part of the /mingw64/x86_64-w64-mingw32/include folder, which is used by the mingw64 gcc compiler, not by MSYS2 gcc compiler (which would place its include files in /usr/include).

[mcarans]

I had used Cygwin for almost 20 years (until like 5 years ago, since I switched to Linux 100% and msys2/mingw64 for my Windows works). While you hit the nail right on its head (i.e. they can be "confused" one with the other), my take is that you're confusing MSYS2 with mingw64.
Both MSYS2 and Cygwin can be used to build (mingw64) windows native target executables, but neither MSYS2 nor Cygwin are native. As you might have noticed, while there is no mingw64 bash.exe file, both MSYS2 and cygwin have it. That's because both MSYS2 and cygwin have their own POSIX emulation layer, and mingw64 doesn't.
While MSYS2 is often "packaged" as the supporting mingw64 build environment for native Windows apps (with the most notable "git windows layer"), you will always be able to get the same results using cygwin's mingw64 toolchain and noarch/mingw64 tools (of course, at slower build speeds, due to a better POSIX layer than MSYS2 has).
To lead this to an end, while both MSYS2 and cygwin have their own target builds (of msys and cygwin), which usually just copy/paste a unix/linux/bsd target, mingw64 is used for win32/win64 (native) builds. That is, all the exes/dlls in a msys2/cygwin build will end in a msys-2.0.dll/cygwin1.dll dependency, while in mingw64's case they'll end in Windows/System32 dlls. You can check this using either cygcheck or ldd on your /usr/bin/*.[exe|dll] and /mingw64/bin/*.[exe|dll] files.

Here we agree: the builds you produced are mingw64 Windows native builds. Mentioning MSYS2 as a standalone environment could lead to confusion (and, in my case annoyance). E.g., the headers you mentioned (for HDR support) I'd bet are part of the /mingw64/x86_64-w64-mingw32/include folder, which is used by the mingw64 gcc compiler, not by MSYS2 gcc compiler (which would place its include files in /usr/include).

Noted.

[phkb]

If I ran that, would that get rid of this error?

It should, yes.

[mcarans]

I had used Cygwin for almost 20 years (until like 5 years ago, since I switched to Linux 100% and msys2/mingw64 for my Windows works).

Here's a task for which your experience especially of Windows programming would be helpful if you're interested: viewtopic.php?t=21958

[Commander_X]

[...]
Here's a task for which your experience especially of Windows programming would be helpful if you're interested: viewtopic.php?t=21958

I'd have to use some of my Linux experience on that one (as I didn't put in the effort to compile oolite.exe outside the current windows tree).
From what I understand, you managed to compile a DLL version of espeak-ng, but it cannot find its "espeak-ng data". I'm not sure how the current espeak dll is compiled, but from what I could read in the README.md of espeak-ng, it should be "compatible" with espeak. Thus you could use this "trick" to inform espeak-ng library where its data should be, like in Linux, in a command prompt session:

Code: Select all

set ESPEAK_DATA=<absolute_path_to|relative_path_to>\oolite.app\Resources
./oolite

If espeak-ng data is still not found, you can rename/copy the espeak-data in Resources to espeak-ng-data (they say they moved it to avoid conflicts).
Hope this helps.