Version: Revision 2033 (although I experienced same issue in earlier version)
Problem: When I try to switch the full screen resolution while playing, one of two things happen (The native resolution is 1680x1050):
1. Try to reduce the resolution and it won't let me.
2. Increase the resolution and the game crashes with this output:
X Error of failed request: BadValue (integer parameter out of range for operation)
Major opcode of failed request: 134 (XFree86-VidModeExtension)
Minor opcode of failed request: 10 (XF86VidModeSwitchToMode)
Value in failed request: 0x73
Serial number of failed request: 133
Current serial number in output stream: 135
To recover I must manually edit GNUstep/Defaults/.GNUstepDefaults
There are known issues when trying to go fullscreen during gameplay and there have been there since at least 1.65, with a variety of symptoms from one version to another. The ones you describe seem quite extreme and could be in part due to the graphics card (I get shivers with Intel cards and I have one myself). Until this is fully sorted out, I would recommend switching to full screen before first launch and avoiding changing it again until the end of your gaming session.
I think I tracked this down to xrandr reporting unsupported resolutions for my monitor. It reports 1600x1200 as the next resolution down from my native resolution and when I switch down in Oolite, I assume X crashes as the monitor can't do 1600x1200.
I have submitted a bug report about xrandr to debian.