Compiling on recent (2017) distros - GCC 6.x issues

[zevans]

As discussed on the sticky thread - GCC 6.x has changed the defaults on a few things. The version of Oolite bundled with SpiderMonkey will not compile on a fresh Zesty install for instance.

There are at least two issues.

1. The easy one. GCC in Ubuntu's environment is now more paranoid and treats some warnings as errors, which prevents a compile. For now you can override it like this:

Code: Select all

make -f Makefile release OBJCFLAGS="-Wno-format-security"

The warnings all look pretty simple to fix but it's probably a slog to go through and find them all.

2. The tough one.

Code: Select all

$ cd oolite
$ git submodule update --init

to pulls in the Oolite version of SpiderMonkey correctly - but it will not compile.

Firstly it relies on autoconf v2.13 - Ubuntu now offers 2.13, 2.59, and 2.64 in separate packages.

Code: Select all

$ apt-get install autoconf2.13
$ cd deps/mozilla/js
$ autoconf2.13
$ ./configure
$ make
[ ... lots of typedef warnings...]
jsapi.cpp: In function ‘JSIdArray* JS_Enumerate(JSContext*, JSObject*)’:
jsapi.cpp:3980:16: error: cannot convert ‘bool’ to ‘JSIdArray*’ in return
         return false;
                ^~~~~
config/rules.mk:1475: recipe for target 'jsapi.o' failed
make[1]: *** [jsapi.o] Error 1
make[1]: Leaving directory '/scratch/src/oolite/deps/mozilla/js/src'
config/rules.mk:753: recipe for target 'default' failed
make: *** [default] Error 2

another_commander has already highlighted the problem - and the possible fix - to go through and patch Oolite's version of SpiderMonkey.

From this page: https://gcc.gnu.org/gcc-6/porting_to.html

Cannot convert 'bool' to 'T*'

The current C++ standard only allows integer literals to be used as null pointer constants, so other constants such as false and (1 - 1) cannot be used where a null pointer is desired. Code that fails to compile with this error should be changed to use nullptr, or 0, or NULL.

So - I'll see if I can make a list...

[another_commander]

So - I'll see if I can make a list...

Thinking a bit about it, I am sure I have seen this issue reported before. Can't remember if it was on this forum, on github, or in our internal devs mailing list, but I think someone had gone through it applying the proposed fix and there were just a few files with this kind of type conversion problems. Since I can't seem to be able to find that discussion anymore, bringing up all the points of interest would be very helpful. And I think it makes perfect sense to apply the fixes in our JS repository, be it that it's compiled with GCC 6.0 or not. Having a function that is expected to return a pointer returning 'false' just because 'false' happens to evaluate to 0 (which in GNU C++ is what NULL is defined as) is totally bonkers.

[zevans]

OK, here's the list. Considering how widely used this library was/is, there are some absolutely bonkers things going on with nullity. There's even helpfully a macro (!) JS_FALSE which is defined as a strongly typed zero, so you can helpfully return zero instead of nullptr for a whole new flavour of wrong. Brilliant.

I created this list by changing the offending occurrences of false (and/or JS_FALSE) to nullptr until the whole thing compiled. I had a quick look at the surrounding code in each case and I'm fairly convinced nullptr rather than zero is correct but I think someone who knows what they are doing needs to check.

I also had to set CXXFLAGS="-Wno-narrowing" to allow another implicit conversion through.

Code: Select all

jsfun.cpp:2055
jsfun.cpp:2661
jsiter.cpp:428
jsparse.cpp:3355
jsstr.cpp:1737
jsstr.cpp:3403 
jstypedarray.cpp:1671 [hilariously this is in a guard for unreached code...]
jstypedarray.cpp:1337
jsxml.cpp:285
jsxml.cpp:434 [this one might need to be NULL rather than nullptr]
./methodjit/InvokeHelpers.cpp:731

It still doesn't compile - I now have an undefined reference to deal with - but you should be able to commit a fix for this lot and regression test it meanwhile. Thanks! I'll probably be back for help with the next problem once I've done some basic troubleshooting...

[another_commander]

The type conversion fixes for the cases reported have been commited to our local Spidermonkey.

I used NULL on all cases instead of nullptr, in order to maintain compatibility with pre-C++11 compilers (nullptr is a C++11 thing).

[zevans]

Brill.

The problem now is that it builds ok, but there's a weird linking problem...

I am wondering if I need to install some build dependency for Yarr, or whether we can just not build the Yarr JIT?

Code: Select all

make[4]: Entering directory '/scratch/src/oolite/deps/mozilla/js/src/build-release/shell'
c++ -o js -Wno-narrowing  js.o jsworkers.o   -lpthread   -Wl,-rpath-link,/bin -Wl,-rpath-link,/usr/local/lib  -L../dist/bin -L../dist/lib -L/usr/lib/x86_64-linux-gnu -lplds4 -lplc4 -lnspr4 -lpthread -ldl ../editline/libeditline.a ../libjs_static.a -ldl
../libjs_static.a(jsclone.o): In function `JSStructuredCloneReader::startRead(js::Value*)':
jsclone.cpp:(.text+0x49bc): undefined reference to `JSC::Yarr::jitCompileRegex(JSC::ExecutableAllocator&, JSC::Yarr::RegexCodeBlock&, JSLinearString const&, unsigned int&, int&, bool&, bool, bool)'
jsclone.cpp:(.text+0x541f): undefined reference to `JSC::Yarr::jitCompileRegex(JSC::ExecutableAllocator&, JSC::Yarr::RegexCodeBlock&, JSLinearString const&, unsigned int&, int&, bool&, bool, bool)'
../libjs_static.a(jsregexp.o): In function `SwapRegExpInternals(JSContext*, JSObject*, js::Value*, JSString*, unsigned int)':
jsregexp.cpp:(.text+0x71a1): undefined reference to `JSC::Yarr::jitCompileRegex(JSC::ExecutableAllocator&, JSC::Yarr::RegexCodeBlock&, JSLinearString const&, unsigned int&, int&, bool&, bool, bool)'
jsregexp.cpp:(.text+0x72ab): undefined reference to `JSC::Yarr::jitCompileRegex(JSC::ExecutableAllocator&, JSC::Yarr::RegexCodeBlock&, JSLinearString const&, unsigned int&, int&, bool&, bool, bool)'
../libjs_static.a(jsregexp.o): In function `js_CloneRegExpObject(JSContext*, JSObject*, JSObject*)':
jsregexp.cpp:(.text+0x7db9): undefined reference to `JSC::Yarr::jitCompileRegex(JSC::ExecutableAllocator&, JSC::Yarr::RegexCodeBlock&, JSLinearString const&, unsigned int&, int&, bool&, bool, bool)'
../libjs_static.a(jsregexp.o):jsregexp.cpp:(.text+0x7ea3): more undefined references to `JSC::Yarr::jitCompileRegex(JSC::ExecutableAllocator&, JSC::Yarr::RegexCodeBlock&, JSLinearString const&, unsigned int&, int&, bool&, bool, bool)' follow
/usr/bin/ld: js: hidden symbol `_ZN3JSC4Yarr15jitCompileRegexERNS_19ExecutableAllocatorERNS0_14RegexCodeBlockERK14JSLinearStringRjRiRbbb' isn't defined
/usr/bin/ld: final link failed: Bad value

[another_commander]

The files under the yarr folder should be part of the project that gets compiled. The error you are getting indicates that the file js\src\yarr\yarr\RegexJIT.cpp, which contains the definition of jitCompileRegex, does not get built. Make sure that the makefile that is being used lists RegexJIT.cpp as one of the files to build.

Edit: It also looks like it's failing while trying to build the js shell executable, which means that the actual library that we need has been already built. In any case, once you include RegexJIT.cpp in the files that are linked to generate the shell executable, the problem should be theoretically gone.

[zevans]

The problem was src/yarr/Makefile needed -I.. to pick up some of the includes. The problem was it was silently failing due to a missing include.

Changed that. Back up a level, did a make distclean to force everything to regenerate.

It now seems to make it through to attempting to link libmozjs.so ...

Code: Select all

$ autoconf2.13
$ ./configure
$ make -f Makefile OBJCFLAGS="-Wno-format-security" CXXFLAGS="-Wno-narrowing -Wno-ignored-attributes -Wno-invalid-offsetof"
[...time passes...]
make[1]: Leaving directory '/scratch/src/oolite/deps/mozilla/js/src'
make libs
make[1]: Entering directory '/scratch/src/oolite/deps/mozilla/js/src'
rm -f libmozjs.so
c++ -Wno-narrowing -Wno-ignored-attributes -Wno-invalid-offsetof -fPIC -shared -Wl,-z,defs -Wl,-h,libmozjs.so -o libmozjs.so  jsanalyze.o jsapi.o jsarena.o jsarray.o jsatom.o jsbool.o jsclone.o jscntxt.o jscompartment.o jsdate.o jsdbgapi.o jsdhash.o jsdtoa.o jsemit.o jsexn.o jsfriendapi.o jsfun.o jsgc.o jsgcchunk.o jsgcstats.o jshash.o jsinterp.o jsinvoke.o jsiter.o jslock.o jslog2.o jsmath.o jsnativestack.o jsnum.o jsobj.o json.o jsopcode.o jsparse.o jsproxy.o jsprf.o jsprobes.o jspropertycache.o jspropertytree.o jsreflect.o jsregexp.o jsscan.o jsscope.o jsscript.o jsstr.o jstypedarray.o jsutil.o jswrapper.o jsxdrapi.o jsxml.o prmjtime.o sharkctl.o jstracer.o Assembler.o Allocator.o CodeAlloc.o Containers.o Fragmento.o LIR.o njconfig.o RegAlloc.o avmplus.o NativeX64.o jsbuiltins.o VMPI.o Writer.o MethodJIT.o StubCalls.o Compiler.o FrameState.o FastArithmetic.o FastOps.o StubCompiler.o MonoIC.o PolyIC.o ImmutableSync.o InvokeHelpers.o Retcon.o TrampolineCompiler.o checks.o conversions.o diy-fp.o v8-dtoa.o fast-dtoa.o platform.o utils.o Assertions.o ExecutableAllocatorPosix.o ExecutableAllocatorWin.o ExecutableAllocatorOS2.o ExecutableAllocator.o ARMAssembler.o Logging.o MacroAssemblerARM.o MacroAssemblerX86Common.o RegexCompiler.o RegexJIT.o pcre_compile.o pcre_exec.o pcre_tables.o pcre_xclass.o pcre_ucp_searchfuncs.o jsperf.o pm_linux.o     -lpthread   -Wl,-rpath-link,/bin -Wl,-rpath-link,/usr/local/lib   -ldl  -lm -ldl
jsapi.o: In function `js::RegExp::compileHelper(JSContext*, JSLinearString&)':
jsapi.cpp:(.text._ZN2js6RegExp13compileHelperEP9JSContextR14JSLinearString[_ZN2js6RegExp13compileHelperEP9JSContextR14JSLinearString]+0x7d): undefined reference to `JSC::Yarr::jitCompileRegex(JSC::ExecutableAllocator&, JSC::Yarr::RegexCodeBlock&, JSLinearString const&, unsigned int&, int&, bool&, bool, bool)'
collect2: error: ld returned 1 exit status
config/rules.mk:1291: recipe for target 'libmozjs.so' failed

[another_commander]

Are you sure that js/src/yarr/yarr/RegexJIT.cpp is getting built? That's where the missing definition of jitCompileRegex is.

Edit: Make sure that the macro ENABLE_ASSEMBLER is defined. If it's not, then the compileRegex code will not be included in the build.

[zevans]

Quick grep shows it's enabled.

Code: Select all

Makefile:CXXFLAGS += -DUSE_SYSTEM_MALLOC=1 -DENABLE_ASSEMBLER=1 -DENABLE_JIT=1

I know exactly what you mean, but those .cpp files are definitely now getting built and the earlier link now succeeds. It's just this final step that's failing. Any way I can use nm on the intermediate files to work out what's missing more precisely?

[Getafix]

This is just to confirm that the changes build fine with the old gcc v4.2.4 too.

@another_commander
You remember right.
https://github.com/OoliteProject/oolite/issues/200

[zevans]

https://github.com/OoliteProject/oolite/issues/200

Well that's interesting...

I was already wondering if it might be a problem with the different ways inline functions are handled in different standards, based on the discussion in this thread: http://clang.debian.net/status.php?vers ... =UNDEF_REF

I tried compiling with -std=gnu++98 in CXXFLAGS as suggested in the issue Getafix referred to, but

Code: Select all

./assembler/assembler/X86Assembler.h: In member function ‘void JSC::X86Assembler::push_r(JSC::X86Ass
embler::RegisterID)’:
./assembler/assembler/X86Assembler.h:380:9: error: ‘JaegerSpew’ is not a member of ‘js’
         js::JaegerSpew(js::JSpew_Insns,

[another_commander]

Would it be possible to post the full command line used when the file RegexJIT.cpp gets compiled? Can you also check the size of the object file generated from RegexJIT.cpp? Is it by any chance 1Kb or something really small?

[zevans]

OMG. What's causing that?

Code: Select all

$ ls -s yarr/RegexJIT.o
20 yarr/RegexJIT.o
$ ls -s build-release/RegexJIT.o
4 build-release/RegexJIT.o

Here's the output of a fresh build.

Code: Select all

$ make -f Makefile OBJCFLAGS="-Wno-format-security -Wno-narrowing" CXXFLAGS="-Wno-narrowing -Wno-ignored-attributes -Wno-invalid-offsetof -Wno-int-to-pointer-cast" | grep RegexJIT
RegexJIT.cpp
c++ -o RegexJIT.o -c  -I./dist/system_wrappers_js -include ./config/gcc_hidden.h -DOSTYPE=\"Linux4.12\" -DOSARCH=Linux -DEXPORT_JS_API -D__STDC_LIMIT_MACROS  -I. -I. -I./dist/include -I./dist/include/nsprpub     -I. -I./assembler -I./yarr  -fPIC -Wno-narrowing -Wno-ignored-attributes -Wno-invalid-offsetof -Wno-int-to-pointer-cast   -DMOZILLA_CLIENT -include ./js-confdefs.h -MD -MF .deps/RegexJIT.pp ./yarr/yarr/RegexJIT.cpp
ar cr libjs_static.a jsanalyze.o jsapi.o jsarena.o jsarray.o jsatom.o jsbool.o jsclone.o jscntxt.o jscompartment.o jsdate.o jsdbgapi.o jsdhash.o jsdtoa.o jsemit.o jsexn.o jsfriendapi.o jsfun.o jsgc.o jsgcchunk.o jsgcstats.o jshash.o jsinterp.o jsinvoke.o jsiter.o jslock.o jslog2.o jsmath.o jsnativestack.o jsnum.o jsobj.o json.o jsopcode.o jsparse.o jsproxy.o jsprf.o jsprobes.o jspropertycache.o jspropertytree.o jsreflect.o jsregexp.o jsscan.o jsscope.o jsscript.o jsstr.o jstypedarray.o jsutil.o jswrapper.o jsxdrapi.o jsxml.o prmjtime.o sharkctl.o jstracer.o Assembler.o Allocator.o CodeAlloc.o Containers.o Fragmento.o LIR.o njconfig.o RegAlloc.o avmplus.o NativeX64.o jsbuiltins.o VMPI.o Writer.o MethodJIT.o StubCalls.o Compiler.o FrameState.o FastArithmetic.o FastOps.o StubCompiler.o MonoIC.o PolyIC.o ImmutableSync.o InvokeHelpers.o Retcon.o TrampolineCompiler.o checks.o conversions.o diy-fp.o v8-dtoa.o fast-dtoa.o platform.o utils.o Assertions.o ExecutableAllocatorPosix.o ExecutableAllocatorWin.o ExecutableAllocatorOS2.o ExecutableAllocator.o ARMAssembler.o Logging.o MacroAssemblerARM.o MacroAssemblerX86Common.o RegexCompiler.o RegexJIT.o pcre_compile.o pcre_exec.o pcre_tables.o pcre_xclass.o pcre_ucp_searchfuncs.o jsperf.o pm_linux.o
c++ -Wno-narrowing -Wno-ignored-attributes -Wno-invalid-offsetof -Wno-int-to-pointer-cast -fPIC -shared -Wl,-z,defs -Wl,-h,libmozjs.so -o libmozjs.so  jsanalyze.o jsapi.o jsarena.o jsarray.o jsatom.o jsbool.o jsclone.o jscntxt.o jscompartment.o jsdate.o jsdbgapi.o jsdhash.o jsdtoa.o jsemit.o jsexn.o jsfriendapi.o jsfun.o jsgc.o jsgcchunk.o jsgcstats.o jshash.o jsinterp.o jsinvoke.o jsiter.o jslock.o jslog2.o jsmath.o jsnativestack.o jsnum.o jsobj.o json.o jsopcode.o jsparse.o jsproxy.o jsprf.o jsprobes.o jspropertycache.o jspropertytree.o jsreflect.o jsregexp.o jsscan.o jsscope.o jsscript.o jsstr.o jstypedarray.o jsutil.o jswrapper.o jsxdrapi.o jsxml.o prmjtime.o sharkctl.o jstracer.o Assembler.o Allocator.o CodeAlloc.o Containers.o Fragmento.o LIR.o njconfig.o RegAlloc.o avmplus.o NativeX64.o jsbuiltins.o VMPI.o Writer.o MethodJIT.o StubCalls.o Compiler.o FrameState.o FastArithmetic.o FastOps.o StubCompiler.o MonoIC.o PolyIC.o ImmutableSync.o InvokeHelpers.o Retcon.o TrampolineCompiler.o checks.o conversions.o diy-fp.o v8-dtoa.o fast-dtoa.o platform.o utils.o Assertions.o ExecutableAllocatorPosix.o ExecutableAllocatorWin.o ExecutableAllocatorOS2.o ExecutableAllocator.o ARMAssembler.o Logging.o MacroAssemblerARM.o MacroAssemblerX86Common.o RegexCompiler.o RegexJIT.o pcre_compile.o pcre_exec.o pcre_tables.o pcre_xclass.o pcre_ucp_searchfuncs.o jsperf.o pm_linux.o     -lpthread   -Wl,-rpath-link,/bin -Wl,-rpath-link,/usr/local/lib   -ldl  -lm -ldl
jsapi.o: In function `js::RegExp::compileHelper(JSContext*, JSLinearString&)':
jsapi.cpp:(.text._ZN2js6RegExp13compileHelperEP9JSContextR14JSLinearString[_ZN2js6RegExp13compileHelperEP9JSContextR14JSLinearString]+0x7d): undefined reference to `JSC::Yarr::jitCompileRegex(JSC::ExecutableAllocator&, JSC::Yarr::RegexCodeBlock&, JSLinearString const&, unsigned int&, int&, bool&, bool, bool)'
collect2: error: ld returned 1 exit status

[another_commander]

OK, here is the problem:

Code: Select all

c++ -o RegexJIT.o -c  -I./dist/system_wrappers_js -include ./config/gcc_hidden.h -DOSTYPE=\"Linux4.12\" -DOSARCH=Linux -DEXPORT_JS_API -D__STDC_LIMIT_MACROS 
-I. -I. -I./dist/include -I./dist/include/nsprpub     -I. -I./assembler -I./yarr  -fPIC -Wno-narrowing -Wno-ignored-attributes -Wno-invalid-offsetof
-Wno-int-to-pointer-cast   -DMOZILLA_CLIENT -include ./js-confdefs.h -MD -MF .deps/RegexJIT.pp ./yarr/yarr/RegexJIT.cpp

The macro ENABLE_ASSEMBLER is not defined when compiling RegexJIT.cpp. The size of RegexiJIT.o means that only the object headers are in that file.

Edit: Try building using CXXFLAGS+="-Wno-narrowing -Wno-ignored-attributes -Wno-invalid-offsetof -Wno-int-to-pointer-cast" (+= rather than = in the make command).

[zevans]

+= did not fix it. (I think the makefiles do that anyway.)

However, adding -DENABLE_ASSEMBLER to CXXFLAGS does! So there must be some piece of autoconf or Makefile logic broken somewhere...

Brilliant, thanks. SpiderMonkey now builds.

Except now the final link for oolite itself fails.

Code: Select all

Linking objc_program oolite ...
./obj.spk.dbg/oolite.obj/OODebugMonitor.m.o: In function `OBJECT_TO_JSVAL':
/scratch/src/oolite/deps/mozilla/js/src/build-debug/dist/include/jsapi.h:221: undefined reference to `JSVAL_NULL'
./obj.spk.dbg/oolite.obj/OODebugMonitor.m.o: In function `OOJSValueFromNativeObject':
/scratch/src/oolite/src/Core/Scripting/OOJavaScriptEngine.h:241: undefined reference to `JSVAL_NULL'
./obj.spk.dbg/oolite.obj/OODebugMonitor.m.o: In function `-[OODebugMonitor(Private) oo:jsValueInContext:]':
/scratch/src/oolite/src/Core/Debug/OODebugMonitor.m:986: undefined reference to `JSVAL_NULL'
./obj.spk.dbg/oolite.obj/OOJSConsole.m.o: In function `OBJECT_TO_JSVAL':
/scratch/src/oolite/deps/mozilla/js/src/build-debug/dist/include/jsapi.h:221: undefined reference to `JSVAL_NULL'

Probably a similar deal which I can hopefully track down tomorrow.

Edit: It'll be a missing header file so those #defines don't get #defined and the linker assumes they're symbols instead.