Building JS library from scratch

[mcarans]

The threadsafe dll I generated is meant to be used in combination with the standard binaries distributed with the game. I would expect all sorts of issues if you tried to use it with binaries built on a different environment, as you have already experienced.

I have also tried the threadsafe newly built JS library with Oolite compiled with #define JS_THREADSAFE 1 and it now fails in the same place as the non-threadsafe one (SetVMFrameRegs) rather than much earlier so the flag was what I was missing before. Next step will be to compile debug versions of libraries to investigate SetVMFrameRegs.

Here are the versions of stuff that are being used when I run - most are standard MSYS2 libraries except for C:\oolite\oolite.app\mozjs.dll, gnustep-base 1.31.1 that needed to be rebuilt with gcc and SDL1.2 built from master branch (although I compiled without espeak for now):

Code: Select all

$ cygcheck ./oolite.exe
C:\oolite\oolite.app\oolite.exe
  C:\WINDOWS\system32\ADVAPI32.dll
    C:\WINDOWS\system32\msvcrt.dll
      C:\WINDOWS\system32\ntdll.dll
      C:\WINDOWS\system32\KERNELBASE.dll
    C:\WINDOWS\system32\SECHOST.dll
    C:\WINDOWS\system32\KERNEL32.dll
    C:\WINDOWS\system32\RPCRT4.dll
  C:\WINDOWS\system32\dwmapi.dll
    C:\WINDOWS\system32\win32u.dll
    C:\WINDOWS\system32\USER32.dll
      C:\WINDOWS\system32\GDI32.dll
  C:\msys64\mingw64\bin\libgcc_s_seh-1.dll
    C:\msys64\mingw64\bin\libwinpthread-1.dll
  C:\WINDOWS\system32\GLU32.dll
    C:\WINDOWS\system32\OPENGL32.dll
  C:\msys64\mingw64\bin\gnustep-base-1_31.dll
    C:\msys64\mingw64\bin\libffi-8.dll
    C:\msys64\mingw64\bin\libgnutls-30.dll
      C:\msys64\mingw64\bin\libbrotlidec.dll
        C:\msys64\mingw64\bin\libbrotlicommon.dll
      C:\msys64\mingw64\bin\libbrotlienc.dll
      C:\WINDOWS\system32\CRYPT32.dll
      C:\msys64\mingw64\bin\libgmp-10.dll
      C:\msys64\mingw64\bin\libhogweed-6.dll
        C:\msys64\mingw64\bin\libnettle-8.dll
      C:\msys64\mingw64\bin\libidn2-0.dll
        C:\msys64\mingw64\bin\libiconv-2.dll
        C:\msys64\mingw64\bin\libintl-8.dll
        C:\msys64\mingw64\bin\libunistring-5.dll
      C:\WINDOWS\system32\ncrypt.dll
      C:\msys64\mingw64\bin\libp11-kit-0.dll
        C:\WINDOWS\system32\SHELL32.dll
          C:\WINDOWS\system32\msvcp_win.dll
      C:\msys64\mingw64\bin\libtasn1-6.dll
      C:\WINDOWS\system32\WS2_32.dll
      C:\msys64\mingw64\bin\zlib1.dll
      C:\msys64\mingw64\bin\libzstd.dll
    C:\msys64\mingw64\bin\libicuin77.dll
      C:\msys64\mingw64\bin\libicuuc77.dll
        C:\msys64\mingw64\bin\libicudt77.dll
        C:\msys64\mingw64\bin\libstdc++-6.dll
    C:\WINDOWS\system32\NETAPI32.dll
    C:\msys64\mingw64\bin\libobjc-4.dll
    C:\msys64\mingw64\bin\libxml2-16.dll
      C:\WINDOWS\system32\bcrypt.dll
      C:\msys64\mingw64\bin\liblzma-5.dll
    C:\msys64\mingw64\bin\libxslt-1.dll
  C:\msys64\mingw64\bin\libopenal-1.dll
    C:\WINDOWS\system32\AVRT.dll
    C:\WINDOWS\system32\ole32.dll
      C:\WINDOWS\system32\combase.dll
    C:\WINDOWS\system32\WINMM.dll
  C:\msys64\mingw64\bin\libpng16-16.dll
  C:\msys64\mingw64\bin\SDL.dll
  C:\WINDOWS\system32\SHLWAPI.dll
  C:\msys64\mingw64\bin\libvorbisfile-3.dll
    C:\msys64\mingw64\bin\libvorbis-0.dll
      C:\msys64\mingw64\bin\libogg-0.dll
  C:\oolite\oolite.app\mozjs.dll
    C:\msys64\mingw64\bin\libnspr4.dll
      C:\WINDOWS\system32\MSWSOCK.dll

[Commander_X]

Not sure to what extent this could be an issue, but from your listing, I can see you're using a different flavor of gcc than the one that's used in a_c's dev environment. Your gcc seems to be the "seh" brand while the other one is "sjlj".

[another_commander]

Not sure to what extent this could be an issue, but from your listing, I can see you're using a different flavor of gcc than the one that's used in a_c's dev environment. Your gcc seems to be the "seh" brand while the other one is "sjlj".

This is correct. The mainstream Dev Environment uses the sjlj exception model oompiler back from 2008 or so and this new attempt uses the Structured Exception Handling exception model which is newer and the current standard native exception model for Windows. SEH is of course the better of the two since it is zero-overhead but mixing binaries from the two models is not really recommended. There is no telling how they could interact with each other. There are no current compilers using the sjlj exception model anymore though. From this standpoint, building all support libraries again with the new compiler is the correct way to move forward, but obviously some issues will need to be ironed out along the way.

[mcarans]

Not sure to what extent this could be an issue, but from your listing, I can see you're using a different flavor of gcc than the one that's used in a_c's dev environment. Your gcc seems to be the "seh" brand while the other one is "sjlj".

This is correct. The mainstream Dev Environment uses the sjlj exception model oompiler back from 2008 or so and this new attempt uses the Structured Exception Handling exception model which is newer and the current standard native exception model for Windows. SEH is of course the better of the two since it is zero-overhead but mixing binaries from the two models is not really recommended. There is no telling how they could interact with each other. There are no current compilers using the sjlj exception model anymore though. From this standpoint, building all support libraries again with the new compiler is the correct way to move forward, but obviously some issues will need to be ironed out along the way.

With a debug build of Oolite and mozjs, it started fine and I was able to fly the ship around without gdb so I think we must be pretty close!

Compiling a debug build of Oolite with release build of mozjs was not possible (linking errors).

With a release build of Oolite and debug build of mozjs, it crashed and in gdb showed this:

Code: Select all

Thread 1 received signal SIGSEGV, Segmentation fault.
0x00007ffac77b29c8 in JSContext::setCurrentRegs (this=0x0, regs=0x172961b8)
    at C:/oolite/deps/mozilla/js/src/jscntxt.h:1699
1699            this->regs = regs;

[another_commander]

With a debug build of Oolite and mozjs, it started fine and I was able to fly the ship around without gdb so I think we must be pretty close!

Compiling a debug build of Oolite with release build of mozjs was not possible (linking errors).

With a release build of Oolite and debug build of mozjs, it crashed and in gdb showed this:

Code: Select all

Thread 1 received signal SIGSEGV, Segmentation fault.
0x00007ffac77b29c8 in JSContext::setCurrentRegs (this=0x0, regs=0x172961b8)
    at C:/oolite/deps/mozilla/js/src/jscntxt.h:1699
1699            this->regs = regs;

The mozjs debug and release builds are very different, they are not just the same code built with different flags and debug symbols. They are truly different code. It is good that one of the two runs successfully, but need to ensure that the other one does as well and this may be an adventure of its own.

Your crash is a null pointer dereference. this (probably the context itself) becomes null for some reason and to investigate it you will need to get a backtrace from the crash and step into each frame of the trace and examine its variables to see where the switch to null occurred.

Also, it is not recommended to mix debug and release versions of Oolite and mozjs. I mean of course you can for investigative reasons, but the typical rule is debug Oolite runs together with debug mozjs and release Oolite runs together with release mozjs.

[mcarans]

With a debug build of Oolite and mozjs, it started fine and I was able to fly the ship around without gdb so I think we must be pretty close!

Compiling a debug build of Oolite with release build of mozjs was not possible (linking errors).

With a release build of Oolite and debug build of mozjs, it crashed and in gdb showed this:

Code: Select all

Thread 1 received signal SIGSEGV, Segmentation fault.
0x00007ffac77b29c8 in JSContext::setCurrentRegs (this=0x0, regs=0x172961b8)
    at C:/oolite/deps/mozilla/js/src/jscntxt.h:1699
1699            this->regs = regs;

The mozjs debug and release builds are very different, they are not just the same code built with different flags and debug symbols. They are truly different code. It is good that one of the two runs successfully, but need to ensure that the other one does as well and this may be an adventure of its own.

Your crash is a null pointer dereference. this (probably the context itself) becomes null for some reason and to investigate it you will need to get a backtrace from the crash and step into each frame of the trace and examine its variables to see where the switch to null occurred.

Also, it is not recommended to mix debug and release versions of Oolite and mozjs. I mean of course you can for investigative reasons, but the typical rule is debug Oolite runs together with debug mozjs and release Oolite runs together with release mozjs.

Thanks. If Oolite and libjs are in release mode, is it possible to step through frames and examine variables? If not, I suppose it will be back to log messages. Speaking of them, I'm not sure how useful this is but the log shows:

Code: Select all

16:11:19.723 [startup.complete]: ========== Loading complete in 9.84 seconds. ==========
16:11:20.780 [dataCache.write.buildPath.failed]: Could not create folder C:/oolite/oolite.app/GNUstep/Library/Caches.
16:11:20.780 [dataCache.write.failed]: Failed to write data cache.
16:11:29.254 [shipData.load.begin]: Loading ship data.
16:11:31.492 [script.javascript.init]: JavaScript reset successful.
16:11:31.502 [texture.load.png.warning]: ----- A PNG loading warning occurred for Resources/Textures/trumblekit.png: iCCP: profile 'ICC Profile': 'RGB ': RGB color space not permitted on grayscale PNG.
16:11:31.634 [script.javascript.init]: JavaScript reset successful.
16:11:31.916 [script.load.world.listAll]: Loaded 16 world scripts:
    Oolite Equipment Control 1.91
    Oolite Ship Library 1.91
    oolite-cloaking-device 1.91
    oolite-constrictor-hunt 1.91
    oolite-contracts-cargo 1.91
    oolite-contracts-helpers 1.91
    oolite-contracts-parcels 1.91
    oolite-contracts-passengers 1.91
    oolite-libPriorityAI 1.91
    oolite-nova 1.91
    oolite-populator 1.91
    oolite-primable-equipment-register 1.91
    oolite-registership 1.91
    oolite-thargoid-plans 1.91
    oolite-trumbles 1.91
    oolite-tutorial 1.91

[another_commander]

Thanks. If Oolite and libjs are in release mode, is it possible to step through frames and examine variables? If not, I suppose it will be back to log messages.

Of course it is, you just need to ensure that debug symbols are included in both the Oolite and the mozjs builds. You may have to tweak build flags to achieve that. Make sure -g (-g3 if you want the maximum possible debug info detail) is in there.

Speaking of them, I'm not sure how useful this is but the log shows:

Code: Select all

16:11:19.723 [startup.complete]: ========== Loading complete in 9.84 seconds. ==========
16:11:20.780 [dataCache.write.buildPath.failed]: Could not create folder C:/oolite/oolite.app/GNUstep/Library/Caches.
16:11:20.780 [dataCache.write.failed]: Failed to write data cache.
16:11:29.254 [shipData.load.begin]: Loading ship data.
16:11:31.492 [script.javascript.init]: JavaScript reset successful.
16:11:31.502 [texture.load.png.warning]: ----- A PNG loading warning occurred for Resources/Textures/trumblekit.png: iCCP: profile 'ICC Profile': 'RGB ': RGB color space not permitted on grayscale PNG.
16:11:31.634 [script.javascript.init]: JavaScript reset successful.
16:11:31.916 [script.load.world.listAll]: Loaded 16 world scripts:
    Oolite Equipment Control 1.91
    Oolite Ship Library 1.91
    oolite-cloaking-device 1.91
    oolite-constrictor-hunt 1.91
    oolite-contracts-cargo 1.91
    oolite-contracts-helpers 1.91
    oolite-contracts-parcels 1.91
    oolite-contracts-passengers 1.91
    oolite-libPriorityAI 1.91
    oolite-nova 1.91
    oolite-populator 1.91
    oolite-primable-equipment-register 1.91
    oolite-registership 1.91
    oolite-thargoid-plans 1.91
    oolite-trumbles 1.91
    oolite-tutorial 1.91

Not sure why it fails to create the cache. This is not a mozjs problem, but a gnustep-base related one. Is this a debug build output?

[mcarans]

Thanks. If Oolite and libjs are in release mode, is it possible to step through frames and examine variables? If not, I suppose it will be back to log messages.

Of course it is, you just need to ensure that debug symbols are included in both the Oolite and the mozjs builds. You may have to tweak build flags to achieve that. Make sure -g (-g3 if you want the maximum possible debug info detail) is in there.

Not sure why it fails to create the cache. This is not a mozjs problem, but a gnustep-base related one. Is this a debug build output?

That was a release build.

I added some flags based on your feedback and recompiled libjs and Oolite. I used -ggdb3 since I'm using gdb. For Oolite, I also used -O0. It was a bit strange but I needed to use -Og rather than -O0 for libjs otherwise the stack trace was incomplete.

I set up debugging in CLion which shows the threads and variables at the time of failure:

https://ibb.co/ks47S5qL
https://ibb.co/tMC7jn4v

The logs look like this (with full logging):

Code: Select all

...
17:07:10.043 [load.progress]: Starting JS engine
17:07:10.043 [script.javaScript.call]: Calling [oolite-constrictor-hunt].startUp()
17:07:10.043 [script.javaScript.call]: Calling [Oolite Equipment Control].startUp()
17:07:10.043 [script.javaScript.call]: Calling [oolite-contracts-passengers].startUp()
17:07:10.043 [script.javaScript.call]: Calling [oolite-registership].startUp()
17:07:10.043 [script.javaScript.call]: Calling [oolite-contracts-parcels].startUp()
17:07:10.043 [script.javaScript.call]: Calling [oolite-tutorial].startUp()
17:07:10.043 [script.javaScript.call]: Calling [oolite-contracts-cargo].startUp()
17:07:10.043 [script.javaScript.call]: Calling [oolite-nova].startUp()
17:07:10.043 [script.javaScript.call]: Calling [oolite-libPriorityAI].startUp()
17:07:10.043 [script.javaScript.call]: Calling [oolite-contracts-helpers].startUp()

Unable to create time zone for name: 'New Zealand Standard Time'
(source 'function: 'GetTimeZoneInformation()'').
...

I can see that the pc of regs is NULL, but It's unclear to me what's happening. Do the stack traces, variables and log tell you anything?

[another_commander]

Context cx also appears to be NULL inside the js code but when the call happens from Oolite it seems to have a valid looking value. Puzzling

Edit: Try stepping into the functions between the call in PlayerEntity and the point of crash and try to monitor the value of context cx to see where exactly it becomes NULL.

[mcarans]

Context cx also appears to be NULL inside the js code but when the call happens from Oolite it seems to have a valid looking value. Puzzling

Edit: Try stepping into the functions between the call in PlayerEntity and the point of crash and try to monitor the value of context cx to see where exactly it becomes NULL.

I found that the js it is failing to interpret is oolite-contracts-helpers.js (from logging).

GDB doesn't support Objective-C variables so I have to convert something to C to breakpoint on it. I could breakpoint on that script with:

Code: Select all

    const char *c_name = [[self name] UTF8String];
    OOLog(@"script.javaScript.call", @"Calling [%@].%@()", [self name], OOStringFromJSID(methodID));
    OOLogIndentIf(@"script.javaScript.call");

I could then set a conditional breakpoint based on the value of c_name:

Code: Select all

$_streq(c_name, "oolite-contracts-helpers")

Looking at the point where it fails, it is on:

Code: Select all

RELATIONAL_OP(<);

I guessed that it is doing a less than comparison.

The first less than comparison inoolite-contracts-helpers.js is:

Code: Select all

	if (missionVariables.oolite_contract_clientnames)
	{
		this.$clientNames = JSON.parse(missionVariables.oolite_contract_clientnames);
	}
	else
	{
		this.$clientNames = [];
		for (var i=0;i<20;i++)
		{
			// one-time initialisation
			this.$clientNames[i] = "";
		}
	}

My guess is that there aren't 20 mission_oolite_contract_clientnames, so it fails (I resumed GDB 17 times with a breakpoint on RELATIONAL_OP(<) before it failed).

I searched for oolite_contract_clientnames. Curiously it is defined in oolite-easy.oolite-save but not oolite-standard.oolite-save. I tried a new game choosing easy instead of standard and it went past that script (and failed on a later script oolite-populator.js).

Why is mission_oolite_contract_clientnames defined in oolite-easy.oolite-save but not oolite-standard.oolite-save or oolite-strict.oolite-save or oolite-tutorial.oolite-save?

Also, do you know how to get either the line number or the JavaScript line itself currently being processed to be printed for debugging purposes?

[phkb]

Weird. When I look at that JS code in isolation, I immediately think, "Ah! There's the bug", and change this:

Code: Select all

		this.$clientNames = [];

to this:

Code: Select all

		this.$clientNames = [20];

Which is a reasonable fix. What's weird is that the existing code works as advertised anyway. I confirmed this by checking the content of the $clientNames array immediately after it has been filled with blanks (after starting a new standard game), and the result is an array with 20 blanks in it. So it works with the current JS without the need for any fix.

Why is mission_oolite_contract_clientnames defined in oolite-easy.oolite-save but not oolite-standard.oolite-save or oolite-strict.oolite-save or oolite-tutorial.oolite-save?

Probably because the "easy" start was created by starting a standard game, jumping to Tionisla to set the location, and then editing the save file to create the new scenario. And because the initialisation of the $clientNames array happens on startup, so it would have been added to the save automatically.

[another_commander]

You can debug obj-c in gdb using the po and print() gdb commands. See https://stackoverflow.com/questions/817 ... s-with-gdb

[another_commander]

In case it helps, I have packaged my working folders for building js and nspr for Oolite on Windows here.

This assumes an installation of the Oolite build environment for Windows under C:\DevelopmentEnvironment and the libraries unzipped under the folder C:\DevelopmentEnvironment\gcc\Msys_x2\1.0\DevlibsBuildFiles64.

The already pre-configured makefiles, link batch files, gnu assembler commands and the actual source files with any modifications that I had to do when building the Spidermonkey library back in 2009/2010/2011 (who remembers anyway) are included. Inside the source folders look for files named howToBuild.txt and follow instructions there. You can try building either with the mainstream dev environment or with your own setup and see if they make any difference.

Anyway, this is what works for me and it is also the source code used to build the nspr and js32 libraries that went to master a few days ago.

Hope it proves useful.

[mcarans]

In case it helps, I have packaged my working folders for building js and nspr for Oolite on Windows here.

This assumes an installation of the Oolite build environment for Windows under C:\DevelopmentEnvironment and the libraries unzipped under the folder C:\DevelopmentEnvironment\gcc\Msys_x2\1.0\DevlibsBuildFiles64.

The already pre-configured makefiles, link batch files, gnu assembler commands and the actual source files with any modifications that I had to do when building the Spidermonkey library back in 2009/2010/2011 (who remembers anyway) are included. Inside the source folders look for files named howToBuild.txt and follow instructions there. You can try building either with the mainstream dev environment or with your own setup and see if they make any difference.

Anyway, this is what works for me and it is also the source code used to build the nspr and js32 libraries that went to master a few days ago.

Hope it proves useful.

Thanks, I have taken a copy of that in case I need to try building again in a different way.

I had to fix the oolite-contracts-helpers.js file like this to get past the failure:

Code: Select all

this.$clientNames = ["", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", ""];

I'm now looking at the next point of failure.

[another_commander]

I had to fix the oolite-contracts-helpers.js file like this to get past the failure:

Code: Select all

this.$clientNames = ["", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", ""];

I'm now looking at the next point of failure.

This is not the right approach though. As phkb already stated, the Oolite code works fine the way it is with the distributed js dll. You should not try to debug Oolite's code at this stage, but identify the root cause of the mozjs crash instead. With the above "fix" you are simply side-stepping the problem.