Using gdb to help us

[another_commander]

This post is here for those of you who build Oolite from source and wish to help us resolve some nasty crasher bugs quicker. If you don't build from source, either because you think it's too hard (it isn't) or just because you can't be bothered, please move on. Nothing to see here.

For those who are still here with us, this is the situation: Many times it has happened that someone submits a bug report for a crash, but there is almost nothing that we can do about it because we cannot reproduce it. This may create the impression that we don't care fixing it, but it is not true. Few things are more frustrating than trying to get the game to crash in a particular way because someone said that it does and failing to reproduce the problem. Sometimes it can be an intermittent situation depending on the OXPs installed or a particular combination of hardware, services running in the background etc. Sometimes it can be just bad luck. The point is, if we could have a way to "drive" the game to crash in a particular way, the chances of fixing the bug would be very high.

So, it seems like we will need some help from those of you who report crashes and build the game from source. For those building under Windows using the pre-packaged development environment, I am pleased to let you know that the tool you need for debugging is already there in the package you downloaded. It's called gdb (GNU Project Debugger). I am also pleased to let you know that using it is very easy for what we want to do. The only thing that you need to do for helping us cure the crashes is run the game normally, but under gdb instead of on its own. So this is how it works:

1. Start up the development environment, just as you do when you are about to compile the source.
Note: Make sure that the game has been built using make debug=yes. This ensures that maximum debugging information is included in the executable.
2. Navigate to the oolite.app folder of the executable you want to debug. Let's assume that you are building the game in a folder called D:\myOolite. You will need to do:

Code: Select all

cd /d/myOolite/oolite.app

3. Tell gdb that you are starting a debugging session on Oolite:

Code: Select all

gdb oolite.dbg.exe

4. Start up Oolite under gdb:

Code: Select all

run

5. Play as you normally do. Be aware that performance will take a hit when using the debugger, don't be alarmed. If (when) it crashes, the program will freeze and the development environment's window will be displaying a message like the one below (actual message may vary slightly, but if you see this it means that execution has stopped and in a "real world" scenario the game would have already crashed to desktop/caused computer to melt/whatever by now):

Code: Select all

Program received signal SIGSEGV, Segmentation fault.
0x6784840a in objc_msg_lookup () from D:\oosvn\trunk\oolite.app\objc-1.dll
(gdb)

6. The last line in brackets above is the debugger's prompt, awaiting your input. Without doing anything else, type the command:

Code: Select all

bt

which stands for backtrace. Once you execute it, you will get something like this as feedback from gdb:

Code: Select all

#0 0x6784840a in objc_msg_lookup () from D:\oosvn\trunk\oolite.app\objc-1.dll
#1 0x00544500 in -[HeadUpDisplay drawCompass:] (self=0xcd487b8,
_cmd=0x1ce6348, info=0x1f7dcf8) at src/Core/HeadUpDisplay.m:1134
#2 0x00540ac6 in -[HeadUpDisplay drawHUDItem:] (self=0xcd487b8,
_cmd=0x61bad8, info=0x1f7dcf8) at src/Core/HeadUpDisplay.m:635
#3 0x0053ec83 in -[HeadUpDisplay drawDials] (self=0xcd487b8, _cmd=0x61bac0)
at src/Core/HeadUpDisplay.m:501
#4 0x005409be in -[HeadUpDisplay renderHUD] (self=0xcd487b8, _cmd=0x63e2d8)
at src/Core/HeadUpDisplay.m:476
#5 0x00596ab4 in -[Universe drawUniverse] (self=0x1f8b430, _cmd=0x630a90)
at src/Core/Universe.m:3414
#6 0x0056dc5f in -[MyOpenGLView updateScreenWithVideoMode:] (self=0x2030010,
_cmd=0x630a80, v_mode=1 '\001') at src/SDL/MyOpenGLView.m:489
#7 0x0056c5fc in -[MyOpenGLView updateScreen] (self=0x2030010, _cmd=0x630a38)
at src/SDL/MyOpenGLView.m:460
#8 0x00569c68 in -[GameController doPerformGameTick] (self=0x1d3db30,
_cmd=0x62dcc8) at src/Core/GameController.m:358
#9 0x00568e53 in -[GameController performGameTick:] (self=0x1d3db30,
_cmd=0x62dce8, sender=0xe3454f8) at src/Core/GameController.m:321
#10 0x64922590 in gnustep-base-1_20!GSUnregisterCurrentThread ()
from D:\oosvn\trunk\oolite.app\gnustep-base-1_20.dll
#11 0x648fd8af in gnustep-base-1_20!NSRangeFromString ()
from D:\oosvn\trunk\oolite.app\gnustep-base-1_20.dll
#12 0x648f9d1f in gnustep-base-1_20!NSRangeFromString ()
from D:\oosvn\trunk\oolite.app\gnustep-base-1_20.dll
#13 0x648fa8a3 in gnustep-base-1_20!NSRangeFromString ()
from D:\oosvn\trunk\oolite.app\gnustep-base-1_20.dll
#14 0x648f9682 in gnustep-base-1_20!NSRangeFromString ()
from D:\oosvn\trunk\oolite.app\gnustep-base-1_20.dll
#15 0x0056999e in -[GameController applicationDidFinishLaunching:] (
self=0x1d3db30, _cmd=0x62f248, notification=0x0)
at src/Core/GameController.m:267
#16 0x0056be99 in SDL_main (argc=1, argv=0x1ce6ae0) at src/SDL/main.m:119
#17 0x005a49e7 in console_main (argc=1, argv=0x1ce6ae0)
at ./src/main/win32/SDL_win32_main.c:246
#18 0x005a4bc5 in WinMain (hInst=0x400000, hPrev=0x0, szCmdLine=0x241f05 "",
sw=10) at ./src/main/win32/SDL_win32_main.c:382
#19 0x005a43c4 in main ()
(gdb)

7. That's it. Doesn't matter whether you understand this output or not, just attach it to your crash report. Even if we are unable to reproduce the issue, with an output like the above, there is a good chance we'll be able to see what may be going wrong and we will most likely be able to see exactly where program execution stopped and take it from there.
8. To quit gdb, just type q at its prompt, after the backtrace has been generated.

Please don't forget to always mention the SVN revision your bug report refers to. As development happens, the contents of the source files change continuously and we need to be able to correlate the line numbers reported by gdb to the actual files that contain them and for that we need to know which revision is being debugged.

Thanks in advance to anyone who is willing to take the time to assist with this. It may be really helpful for those cases where the bug resolution ends up being just "Works For Me".

[Gimi]

Hmm...
Interesting

I build source using the easy method on Windows. Can I just compile that setup with debug=yes and then go on as you have explained, or do I need the bona fide development environment.

[another_commander]

Hmm...
Interesting

I build source using the easy method on Windows. Can I just compile that setup with debug=yes and then go on as you have explained, or do I need the bona fide development environment.

The easy method on Windows uses the complete dev environment which contains the debugger and which we use for development on Windows (minus the installer, which is not necessary for just making a build of the game). You should be able to compile with debug=yes without problems, but I would recommend doing a make -fMakefile clean first, so that we are sure you start the build from scratch and no chance of previously created files intefering exists.

[Capt. Murphy]

Thanks for that another_commander. Fortunately the only time Oolite has ever crashed on me was that fixed HUD re-drawing issue that came to light with sniper-scope.

However would the gdb give useful info about the cause of Oolite hanging (i.e. not a crash, but frozen at 100% CPU). It's an issue I used to have fairly regularly when using Full Shaders and now resolved by using Simple Shaders, but if it could help improve the code I'll happily play a debug build with Full Shaders .

[another_commander]

Capt. Murphy: GDB can give a lot more information than discussed above, but it requires some programming experience to be able to use it to its full potential. Catching crashes and showing you why the program crashed is just one of the things you can do. You can change the code while it runs, monitor variables' and expressions' values, set the program to stop execution where you tell it to so that you can look up what is happening at a specific point and continue when you want and a lot more others, which are all explained in the documentation.

The problem of 100% CPU usage is probably best examined by profiling the code. I don't think that gdb can do much to help here. Profiling is a whole different beast, but doing it will show you which parts of the code take longer to execute and pinpoint any bottlenecks that will require optimization. A Google search for "profiling with gcc" yields plenty of useful information how to do it. If I can find some time - very difficult these days I'm afraid, I could maybe upload a test profiling executable that you can run and hopefully see what is that requires 100% of the CPU.

[Cholmondely]

Ahem! Is this still relevant?

And just for Windows? Or also AppleMac/Linux/Raspberry/ESP32/Arduino etc?

[another_commander]

Ahem! Is this still relevant?

Yes.

And just for Windows? Or also AppleMac/Linux/Raspberry/ESP32/Arduino etc?

The instructions here are for Windows, but the principle is the same for any environment using gcc. As far as Macs are concerened: no idea, because they use a different compiler.