Debugging Oolite startup issues on fresh mingw64

News and discussion of the PC port of Oolite.

Moderators: winston, another_commander

Post Reply
User avatar
mcarans
---- E L I T E ----
---- E L I T E ----
Posts: 392
Joined: Sun Jun 20, 2010 6:00 pm

Debugging Oolite startup issues on fresh mingw64

Post by mcarans »

I've been experimenting with building Oolite on a newly downloaded MSYS2 environment. I managed to get it to build and link with a load of effort. It begins to start up but then stops unfortunately.

In the logs, it gets as far as this:

09:44:16.628 [joystick.init]: Number of joysticks detected: 0
09:44:16.628 [rendering.opengl.version]: OpenGL renderer version: 4.6.0 ("4.6.0 NVIDIA 560.94"). Vendor: "NVIDIA Corporation". Renderer: "NVIDIA GeForce RTX 4060 Laptop GPU/PCIe/SSE2".
09:44:16.628 [rendering.opengl.extensions]: OpenGL extensions (404):


There are loads of OpenGL extensions listed after the above lines, but then nothing after that. What is the process for debugging failures like this? Is there a way to get much more detailed logging so I can track down where it is failing?
Last edited by mcarans on Tue Aug 26, 2025 3:48 am, edited 1 time in total.
Commander_X
---- E L I T E ----
---- E L I T E ----
Posts: 708
Joined: Sat Aug 09, 2014 4:16 pm

Re: Debugging Oolite startup issues

Post by Commander_X »

mcarans wrote: Mon Aug 25, 2025 9:53 pm
I've been experimenting with building Oolite on a newly downloaded MSYS2 environment. [...]
MSYS2? Why would you try that, and not, say, cygwin64? MSYS uses also some POSIX wrappers around windows calls part of their msys-2.0.dll.

I'm using another_commander's mingw64 build bundle, and it works as expected. Not broken, don't fix.
Didn't try to spawn a "personal" setup in my (almost clean) more complete mingw64 environment, as patching and installing the supporting libraries would take quite a while.

My .02 Cr on your issue would be the hybrid build you might've managed. First of all a "clean" MSYS2 Oolite would need to be compiled against Linux make targets. Or clone the Linux targets and ajust for MSYS2 layout (e.g. library names, if autotools/GNUstep don't pick them up correctly).
Second, you'd need your own compiled (or installed) "deps" libraries targeted for MSYS2 (and the big challenge here would definitely be Mozilla's JS; I don't think it would be trivial to get it as a MSYS2 dll).
If you didn't go the clean MSYS2 route, and tried to supplement your libraries with those provided as "deps" in the tree (deps/Windows-deps) that's a no-no. Similarly with Cygwin(64), MSYS(2) shouldn't mix "native" windows dlls with its "msys-" branded dlls -- that's a receipt for disaster.
User avatar
mcarans
---- E L I T E ----
---- E L I T E ----
Posts: 392
Joined: Sun Jun 20, 2010 6:00 pm

Re: Debugging Oolite startup issues on fresh mingw64

Post by mcarans »

Commander_X wrote: Tue Aug 26, 2025 2:37 am

MSYS2? Why would you try that, and not, say, cygwin64? MSYS uses also some POSIX wrappers around windows calls part of their msys-2.0.dll.

I'm using another_commander's mingw64 build bundle, and it works as expected. Not broken, don't fix.
Didn't try to spawn a "personal" setup in my (almost clean) more complete mingw64 environment, as patching and installing the supporting libraries would take quite a while.

My .02 Cr on your issue would be the hybrid build you might've managed. First of all a "clean" MSYS2 Oolite would need to be compiled against Linux make targets. Or clone the Linux targets and ajust for MSYS2 layout (e.g. library names, if autotools/GNUstep don't pick them up correctly).
Second, you'd need your own compiled (or installed) "deps" libraries targeted for MSYS2 (and the big challenge here would definitely be Mozilla's JS; I don't think it would be trivial to get it as a MSYS2 dll).
If you didn't go the clean MSYS2 route, and tried to supplement your libraries with those provided as "deps" in the tree (deps/Windows-deps) that's a no-no. Similarly with Cygwin(64), MSYS(2) shouldn't mix "native" windows dlls with its "msys-" branded dlls -- that's a receipt for disaster.
I am using MSYS2 mingw64 environment so should be similar although much more up to date than the Windows development build.

I have applied all the SDL changes to latest master of SDL1. 2: https://github.com/mcarans/SDL-1.2/tree/oolite_changes
Then I built it from source.

I also built gnustep 1.31.1 (using gcc). I might need to have a crack at building other libraries from source instead of using the ones in Windows deps. It would be great though to have some idea of which library is causing the failure so I can focus on the right library.

I think it would really help if there were some basic checks on Oolite startup to confirm important libraries like OpenGL, OpenAL, SDL, Javascript etc. can be called and hence rapidly narrow down the source of problems. Could such tests go in main.m before the application is started?

It gets much farther when running in gdb than when not, an idea I took from this post:
viewtopic.php?p=292025#p292025

Then I get these additional logs:

17:18:20.181 [rendering.opengl.shader.support]: Shaders are supported.
17:18:21.191 [searchPaths.dumpAll]: Resource paths:
Resources
../AddOns
../AddOns/Basic-debug.oxp
17:18:21.197 [MSAA.setup]: Multisample anti-aliasing not requested.
17:18:21.475 [shipData.load.begin]: Loading ship data.
17:18:24.540 [debugTCP.disconnect]: No connection to debug console: "Connection to debug console failed: 'No connection could be made because the target machine actively refused it.
' (outStream status: 7, inStream status: 7)."
17:18:24.540 [debugTCP.send.warning]: Error sending packet header, retrying.
17:18:24.566 [debugTCP.send.error]: The following packet could not be sent: {"Oolite version" = 1.91; "packet type" = "Request Connection"; "protocol version" = 65792; }
17:18:24.569 [debugTCP.disconnect]: No connection to debug console: "Connection to debug console failed: 'bad stream.' (outStream status: 0, inStream status: 0)."
17:18:24.569 [debugTCP.connect.failed]: Failed to connect to debug console at address 127.0.0.1:8563.
17:18:24.569 [display.initGL]: Requested a new surface of 1280 x 720, windowed.
17:18:24.612 [display.initGL]: Created a new surface of 1280 x 720, windowed.
17:18:24.694 [startup.complete]: ========== Loading complete in 6.60 seconds. ==========


Not sure if that provides any clues?
another_commander
Quite Grand Sub-Admiral
Quite Grand Sub-Admiral
Posts: 6945
Joined: Wed Feb 28, 2007 7:54 am

Re: Debugging Oolite startup issues on fresh mingw64

Post by another_commander »

mcarans wrote: Mon Aug 25, 2025 9:53 pm
What is the process for debugging failures like this? Is there a way to get much more detailed logging so I can track down where it is failing?
Check for the files stderr.txt and stdout.txt in oolite.app while the game is running. Do they contain any messages? If the game terminates normally these will be auto-deleted, be aware of that in case you can't find them.

You can also try editing logcontrol.plist. At the top of the file there is a commented out _override = yes; line. Remove the comment and activate it, it will force printing of all log message classes.

If all else fails you can always try the universally standard debug method: printfs in the code - or, in our case, NSLog(@"your message here") statements. They use the same format specifiers as printf. Sprinkle some of those after the OpenGL extensions have been printed out and see where NSLogs stop getting printed. With patience you can narrow it down to the exact line that fails. Sounds like torture? Sure! Been there, done that, got the t-shirt, used it as cleaning rag, threw it in the trash and then did it again. That's part of the price of admission if you want to set up a build environment for Oolite from scratch.

I have applied all the SDL changes to latest master of SDL1. 2: https://github.com/mcarans/SDL-1.2/tree/oolite_changes
Which version did you build? Oolite uses 1.2.13 and the patch with the changes may not be entirely applicable for later SDL 1.2 versions. Maybe it works, but there are no guarantees and I believe that a lot of the patch changes are already present in more recent SDL 1.2. You may need to ensure tha all changes made are indeed necessary and do not duplicate existing functionality if you have built SDL later than 1.2.13.
I also built gnustep 1.31.1 (using gcc).
Did you include TLS support when you built it? It is absolutely critical that you have TLS in gnustep, or 99% of OXPs cannot be downloaded due to living in https domains.
I think it would really help if there were some basic checks on Oolite startup to confirm important libraries like OpenGL, OpenAL, SDL, Javascript etc. can be called and hence rapidly narrow down the source of problems. Could such tests go in main.m before the application is started?
What do you mean exactly by "getting called"? Windows already has checks when loading dlls, that they contain all the functions the application asks for during its startup. If an application contains calls to a function that's not there you will get an error message along the lines of "Entry point for function whatever could not be located in whatsitsname.dll" and the app doesn't start. As long as all functions can be linked dynamically at run time, the app starts. Now, if a function in a dll is buggy or has strange interactions with the OS, that's something that we can only find out at the time of executing that function.

Oolite already has plenty of sanity checks when loading up libraries like OpenAL, JS, etc. Not sure about gnustep itself, but I am not sure about how you would add checks for gnustep's health either.
another_commander
Quite Grand Sub-Admiral
Quite Grand Sub-Admiral
Posts: 6945
Joined: Wed Feb 28, 2007 7:54 am

Re: Debugging Oolite startup issues on fresh mingw64

Post by another_commander »

mcarans wrote: Tue Aug 26, 2025 3:17 am
It gets much farther when running in gdb than when not, an idea I took from this post:
viewtopic.php?p=292025#p292025

Then I get these additional logs:

17:18:20.181 [rendering.opengl.shader.support]: Shaders are supported.
17:18:21.191 [searchPaths.dumpAll]: Resource paths:
Resources
../AddOns
../AddOns/Basic-debug.oxp
17:18:21.197 [MSAA.setup]: Multisample anti-aliasing not requested.
17:18:21.475 [shipData.load.begin]: Loading ship data.
17:18:24.540 [debugTCP.disconnect]: No connection to debug console: "Connection to debug console failed: 'No connection could be made because the target machine actively refused it.
' (outStream status: 7, inStream status: 7)."
17:18:24.540 [debugTCP.send.warning]: Error sending packet header, retrying.
17:18:24.566 [debugTCP.send.error]: The following packet could not be sent: {"Oolite version" = 1.91; "packet type" = "Request Connection"; "protocol version" = 65792; }
17:18:24.569 [debugTCP.disconnect]: No connection to debug console: "Connection to debug console failed: 'bad stream.' (outStream status: 0, inStream status: 0)."
17:18:24.569 [debugTCP.connect.failed]: Failed to connect to debug console at address 127.0.0.1:8563.
17:18:24.569 [display.initGL]: Requested a new surface of 1280 x 720, windowed.
17:18:24.612 [display.initGL]: Created a new surface of 1280 x 720, windowed.
17:18:24.694 [startup.complete]: ========== Loading complete in 6.60 seconds. ==========


Not sure if that provides any clues?
Running under gdb and failing when not under gdb usually relates to uninitialized variables. When a debugger is used everything gets initialized to either zero or some marker dummy value. The problem of course is a) confirming that indeed this is the reason and not something else and b) assuming that uninitialized objects is the reason, figuring out which object (and maybe in which library) is not initialized correctly.

Just to be sure, you do get the splash screen and the game window to open, right?
User avatar
mcarans
---- E L I T E ----
---- E L I T E ----
Posts: 392
Joined: Sun Jun 20, 2010 6:00 pm

Re: Debugging Oolite startup issues on fresh mingw64

Post by mcarans »

another_commander wrote: Tue Aug 26, 2025 5:52 am
Running under gdb and failing when not under gdb usually relates to uninitialized variables. When a debugger is used everything gets initialized to either zero or some marker dummy value. The problem of course is a) confirming that indeed this is the reason and not something else and b) assuming that uninitialized objects is the reason, figuring out which object (and maybe in which library) is not initialized correctly.

Just to be sure, you do get the splash screen and the game window to open, right?
Yes with gdb, I get the splash screen and game window, but not without gdb.

I applied the SDL changes manually by diffing the original 1.2.13 SDL code with a patched version then copying changes to the new master branch SDL version which required a bit of analysis of the code in some cases where the code was no longer the same.
Commander_X
---- E L I T E ----
---- E L I T E ----
Posts: 708
Joined: Sat Aug 09, 2014 4:16 pm

Re: Debugging Oolite startup issues on fresh mingw64

Post by Commander_X »

<still pursuing the possibility of a hybrid build>
Did you check that there is no msys-*.dll dependency for your Oollite.exe?
User avatar
mcarans
---- E L I T E ----
---- E L I T E ----
Posts: 392
Joined: Sun Jun 20, 2010 6:00 pm

Re: Debugging Oolite startup issues on fresh mingw64

Post by mcarans »

another_commander wrote: Tue Aug 26, 2025 5:45 am
Did you include TLS support when you built it? It is absolutely critical that you have TLS in gnustep, or 99% of OXPs cannot be downloaded due to living in https domains.
I checked the gnustep-base build and the config.log says:

configure:13680: checking gnutls support
configure:13990: result: yes
...
HAVE_GNUTLS='1'


I presume that means it built with TLS support.

BTW, I think that my patched master SDL could be copied into the Windows development environment and built in there. If it worked including HDR, force feedback etc., then at least the SDL would be the newest 1.2 version available. It is easy to update this SDL fork because all that is needed is to click the button in GitHub to update the fork when updates to the original repo occur (unless there is a merge conflict with patched code).
User avatar
mcarans
---- E L I T E ----
---- E L I T E ----
Posts: 392
Joined: Sun Jun 20, 2010 6:00 pm

Re: Debugging Oolite startup issues on fresh mingw64

Post by mcarans »

another_commander wrote: Tue Aug 26, 2025 5:45 am
Check for the files stderr.txt and stdout.txt in oolite.app while the game is running. Do they contain any messages? If the game terminates normally these will be auto-deleted, be aware of that in case you can't find them.

You can also try editing logcontrol.plist. At the top of the file there is a commented out _override = yes; line. Remove the comment and activate it, it will force printing of all log message classes.

If all else fails you can always try the universally standard debug method: printfs in the code - or, in our case, NSLog(@"your message here") statements. They use the same format specifiers as printf. Sprinkle some of those after the OpenGL extensions have been printed out and see where NSLogs stop getting printed. With patience you can narrow it down to the exact line that fails. Sounds like torture? Sure! Been there, done that, got the t-shirt, used it as cleaning rag, threw it in the trash and then did it again. That's part of the price of admission if you want to set up a build environment for Oolite from scratch.
Thanks for all your help so far. Running Oolite in GDB, I was able to start a new game and fly around. WIthout GDB, it stops before the splash screen. When it fails, stdout.txt and stderr.txt are 0 bytes. Editing logcontrol.plist did not add any new messages.

So I resorted to nslog. Here is where it failed: https://github.com/OoliteProject/oolite ... ger.m#L266



NSLog(@"Stage 1");
NSDictionary *gpuConfig = [self lookUpPerGPUSettingsWithVersionString:versionStr extensionsString:extensionsStr];
NSLog(@"Stage 2");


Stage 1 was printed, but not Stage 2. Do you have any idea what the problem is and why when running in GDB, that line presumably runs fine since I can fly around?
User avatar
phkb
Impressively Grand Sub-Admiral
Impressively Grand Sub-Admiral
Posts: 5393
Joined: Tue Jan 21, 2014 10:37 pm
Location: Writing more OXPs, because the world needs more OXPs.

Re: Debugging Oolite startup issues on fresh mingw64

Post by phkb »

mcarans wrote: Wed Aug 27, 2025 6:27 am
Stage 1 was printed, but not Stage 2.
Can you add some additional NSlog entries to the lookUpPerGPUSettingsWithVersionString function (lines 788-825 of the same file)? How far through the function does it get.
User avatar
mcarans
---- E L I T E ----
---- E L I T E ----
Posts: 392
Joined: Sun Jun 20, 2010 6:00 pm

Re: Debugging Oolite startup issues on fresh mingw64

Post by mcarans »

phkb wrote: Wed Aug 27, 2025 7:15 am
mcarans wrote: Wed Aug 27, 2025 6:27 am
Stage 1 was printed, but not Stage 2.
Can you add some additional NSlog entries to the lookUpPerGPUSettingsWithVersionString function (lines 788-825 of the same file)? How far through the function does it get.
It failed between stages 3 and 4: https://github.com/OoliteProject/oolite ... ger.m#L809


expr = [match objectForKey:@"vendor"];
NSLog(@"Stage 3");
if (!CheckRegExps(vendor, expr)) continue;
NSLog(@"Stage 4");
User avatar
phkb
Impressively Grand Sub-Admiral
Impressively Grand Sub-Admiral
Posts: 5393
Joined: Tue Jan 21, 2014 10:37 pm
Location: Writing more OXPs, because the world needs more OXPs.

Re: Debugging Oolite startup issues on fresh mingw64

Post by phkb »

mcarans wrote: Wed Aug 27, 2025 8:35 am
It failed between stages 3 and 4
...and in the CheckRegExps function (lines 741-767)? Maybe output the content of the parameters in that function as well (string and regexps).
User avatar
mcarans
---- E L I T E ----
---- E L I T E ----
Posts: 392
Joined: Sun Jun 20, 2010 6:00 pm

Re: Debugging Oolite startup issues on fresh mingw64

Post by mcarans »

phkb wrote: Wed Aug 27, 2025 8:41 am
mcarans wrote: Wed Aug 27, 2025 8:35 am
It failed between stages 3 and 4
...and in the CheckRegExps function (lines 741-767)? Maybe output the content of the parameters in that function as well (string and regexps).
I traced the call and eventually it reaches starting up the JavaScript engine. Perhaps unsurprisingly, this is where the problem occurs: https://github.com/OoliteProject/oolite ... ine.m#L283

Code: Select all

    NSLog(@"Stage 6");
    _runtime = JS_NewRuntime(jsRuntimeInMiB * 1024L * 1024L);
    NSLog(@"Stage 7");
Stage 6 is printed, not Stage 7.
another_commander
Quite Grand Sub-Admiral
Quite Grand Sub-Admiral
Posts: 6945
Joined: Wed Feb 28, 2007 7:54 am

Re: Debugging Oolite startup issues on fresh mingw64

Post by another_commander »

Oh, it looks like you may have to enter the hell of the JS library source to find the root cause...

Been there, not going back. Most likely you will need a JS lib built with debug symbols (actually best if you build everything (Oolite and any support library) with debug symbols) and set breakpoint at Stage 6, then try to get a backtrace and scan for any suspicious null pointers or anything that looks unusual.
Post Reply