Occasional random memory gobbling

Post by **winston** » Sun Oct 30, 2005 3:09 pm

I'm occasionally seeing explanation-less and fairly random memory gobbling episodes. After several hours of uneventful play, occasionally Oolite will start allocating ridiculous amounts of memory. I just watched it going from smooth, 100fps lovelyness to causing the machine to hopelessly thrash until the kernel's oom killer killed the game. Once it started this memory usage increase, it just kept going up and up very rapidly.

I don't think it's specific to the Linux version - I suspect it's one of the post 1.55 changes (since it never used to happen). Unfortunately, every time things like this happens, it's always when I'm just trying to play a game rather than testing in the debugger!

oO · Post by oO » Sun Oct 30, 2005 6:48 pm

I saw an almost-OOM in rev 171, but i thought it was normal because i saw you added a few NSAutoreleasePool in 172.

The game went totally unresponsive and top reported 639Mb VIRT.
Then it became playable again for a while, then again unresponsive for a few times. Until it was frozen for a long time, top said 1.09Gb VIRT and remained there for some time until i got bored and killed it. (I got 0.5Gb ram + 1Gb swap).

This happened in the middle of a battle against a lot of thargoid warships, maybe it's because they release a lot of robofighters and they don't release memory or something when they die?

Post by **winston** » Sun Oct 30, 2005 7:09 pm

No, the Thargoids have been a feature for a long time, and it's never done that (I think the bulk of oolite's allocated memory is cached textures - don't forget those PNGs will sit as *uncompressed* bitmaps in memory). Most texture-mapped games like allocating lots of memory.

This is different - my hunch is that it's not a memory leak in the traditional sense (i.e. whoops we forgot to free what we just made) but something triggering something that starts allocating memory in small chunks at a high rate. When it happened to me, the memory useage steadily climbed in megabytes per second from about 50% memory usage until the oom killer viciously knifed it in the back with signal 9. I have a hunch it may be related to the new wormhole stuff, but that's merely a hunch - it happens so infrequently that it's difficult to tell. The two times it's happened so far (in about 4 hours of play testing) has always been right after a wormhole event showed up in the console log. As I said it's merely a completely unscientific hunch. Wormholes seem to get created all the time without all my memory vanishing down them.

JensAyton · Post by **JensAyton** » Sun Oct 30, 2005 7:24 pm

I’ve been playing with wormholes for a while and not seeing this. The Mac build did run up to 3.5 gigs or so, but that was when I had FoundationKit’s zombie tracking on (in which objects are never released) and played for a couple of hours…

aegidian · Post by **aegidian** » Sun Oct 30, 2005 10:33 pm

An out-of-control memory allocating loop or race-condition. That would explain a number of crashes on Macs I've been seeing - the NSScanner crashes in scanTokensFromString and some NSDictionary allocation crashes, I think that the Mac could be handling OOM problems rather quietly.

like this:

Code: Select all

Thread 0 Crashed:
0   <<00000000>> 	0xfffeff18 objc_msgSend_rtp + 24
1   com.apple.CoreFoundation       	0x9073d308 __CFDictionaryFindBuckets1b + 140
2   com.apple.CoreFoundation       	0x9073d24c CFDictionaryGetValue + 212
3   org.aegidian.oolite            	0x0001f400 -[TextureStore getTextureNameFor:] + 64 (TextureStore.m:86)
4   org.aegidian.oolite            	0x00022864 -[Entity initialiseTextures] + 104 (Entity.m:730)
5   org.aegidian.oolite            	0x0002251c -[Entity drawEntity::] + 652 (Entity.m:626)
6   org.aegidian.oolite            	0x00047a04 -[ShipEntity drawEntity::] + 156 (ShipEntity.m:2186)
7   org.aegidian.oolite            	0x0002270c -[Entity drawSubEntity::] + 192 (Entity.m:681)
8   org.aegidian.oolite            	0x00047a88 -[ShipEntity drawEntity::] + 288 (ShipEntity.m:2195)
9   org.aegidian.oolite            	0x00013b0c -[Universe drawFromEntity:] + 1572 (Universe.m:3333)

or this

Code: Select all

Thread 0 Crashed:
0   libSystem.B.dylib              	0x90003b24 szone_malloc + 200
1   libSystem.B.dylib              	0x900039fc malloc_zone_malloc + 100
2   com.apple.CoreFoundation       	0x90729c88 CFAllocatorAllocate + 296
3   com.apple.CoreFoundation       	0x90729a7c _CFRuntimeCreateInstance + 224
4   com.apple.CoreFoundation       	0x90778208 CFCharacterSetCreateMutable + 44
5   com.apple.CoreFoundation       	0x9077de24 __CFCharacterSetCreateCopy + 232
6   com.apple.CoreFoundation       	0x907931f0 CFCharacterSetCreateInvertedSet + 156
7   com.apple.Foundation           	0x928bf128 -[NSCFCharacterSet invertedSet] + 32
8   org.aegidian.oolite            	0x000219dc +[Entity scanTokensFromString:] + 148 (Entity.m:182)
9   org.aegidian.oolite            	0x000122b4 -[Universe getShipWithRole:] + 224 (Universe.m:2704)
10  org.aegidian.oolite            	0x00012d74 -[Universe getContainersOfScarceGoods:] + 276 (Universe.m:2938)
11  org.aegidian.oolite            	0x0004a05c -[ShipEntity becomeExplosion] + 1592 (ShipEntity.m:3220)
12  org.aegidian.oolite            	0x00050120 -[ShipEntity manageCollisions] + 196 (ShipEntity.m:5253)
13  org.aegidian.oolite            	0x00045d2c -[ShipEntity update:] + 236 (ShipEntity.m:1368)
14  org.aegidian.oolite            	0x0001884c -[Universe update:] + 1124 (Universe.m:4881)

I'm not sure what to do to investigate this properly.

JensAyton · Post by **JensAyton** » Sun Oct 30, 2005 11:04 pm

The szone_malloc things are not out-of-memory. I’ve had them and subsequently checked peak swap file usage; it’s always been less than free disk space and less than 4 gigs (the 32-bit process limit).

Post by **winston** » Tue Nov 08, 2005 10:04 pm

Bump.
This is still occasaionally happening. There is no particular thing I can identify that triggers it, but it happened with the most recent SVN head on Linux tonight. It is perhaps happening once every 6 hours of play which makes it very difficult to tie to a particular event.

kaypy · Post by **kaypy** » Sat Dec 10, 2005 9:16 am

Hi

Got another few "oolite starts to sieze up" events today. What I wanted to add to the knowledge pool is:

Remembering what people said in one of the related threads about it possibly being the AI, I tried hyperspacing out when my system started to intermittently freeze. Once I got to a new system, the juddering stopped and I was able to get my ship home and save the game.

May be somewhat useful to debuggers and more useful to folks wanting to rescue their ships from the memory glitch of doom...

kp

kaypy · Post by **kaypy** » Sat Dec 10, 2005 9:57 am

And another one. This time I was taking dumps of "ps aux | grep oolite" periodically:

just started a new game
kaypy 6264 46.8 22.9 139196 118700 pts/1 SLl+ 17:19 0:35 ./oolite

still pretty normal
kaypy 6264 52.7 25.0 149748 129284 pts/1 SLl+ 17:19 2:47 ./oolite

about here would be the first inklings of trouble
kaypy 6264 57.6 34.0 196056 176008 pts/1 SLl+ 17:19 12:35 ./oolite

this would be juddering a fair bit... its time to bail out
kaypy 6264 57.4 37.1 212496 191620 pts/1 SLl+ 17:19 12:51 ./oolite

just before the warp hits. freezing badly now
kaypy 6264 57.6 65.7 366824kp 339684 pts/1 SLl+ 17:19 13:25 ./oolite

now in a new system. memory still up at 58%, but must all be no longer used and hence swapped out or something, because the freezing has stopped
kaypy 6264 57.3 58.1 327332 300244 pts/1 SLl+ 17:19 13:36 ./oolite

a bit later in the new system
kaypy 6264 58.0 57.0 321776 294696 pts/1 SLl+ 17:19 17:09 ./oolite

The freeze ups seem to happen when theres lots of fighting and looting. Ive had about 4 freeze ups today, all of which were "forget the salvage- the universe is about to die!" events...

kp

TedJ · Post by **TedJ** » Fri Jan 20, 2006 1:51 am

I've also been spotting this one occasionally (linux 1.62rc1&2), I've had several in the last 2-3 days... Usually happens to me after a large battle when I'm scooping cargo.

Post by **winston** » Fri Jan 20, 2006 9:41 am

Damn. I've not seen a recurrence since building 1.61. Can you take a look at 'top' next time it happens and report the game's size? (I've had separate performance issues with lave.oxp, but only with that one OXP, and it didn't involve excessive memory usage).

Murgh · Post by **Murgh** » Fri Jan 20, 2006 10:57 am

winston wrote:
I've had separate performance issues with lave.oxp, but only with that one OXP, and it didn't involve excessive memory usage

what can be the fault with it? all the crazy AI? too many AI and spawnings? the added moon?

Post by **winston** » Fri Jan 20, 2006 1:25 pm

I dunno yet, I really need to flush out my AddOns directory and get all the latest versions of the OXPs so I can make sure out of date or conflicting add-ons aren't causing the problems.

TedJ · Post by **TedJ** » Sun Jan 22, 2006 11:29 pm

winston wrote:
Damn. I've not seen a recurrence since building 1.61. Can you take a look at 'top' next time it happens and report the game's size? (I've had separate performance issues with lave.oxp, but only with that one OXP, and it didn't involve excessive memory usage).

Happened to me again last night, top reported vmem usage had shot to nearly 500MB (I only have 512MB swap), rss size was around 180 MB. I zapped X shortly thereafter, as the swapping was really thrashing my disk and we were having a very warm night (got to 43C yesterday).

EDIT: More details. Once again I was scooping cargo after tackling a Cobra 3 and an Asp, so I doubt it's an issue with an OXP, unless it's a problem with the nubarrel model in aegidian-special.

TedJ · Post by **TedJ** » Mon Jan 23, 2006 11:39 pm

It just gets weirder... Oolite decided to dine heavily on my swap space again last night, only this time I was docked.

On Winston's advice, I've taken to keeping top running in a background terminal while playing, just to keep an eye on things. I have to admit being a little surprised how much CPU Oolite uses while paused (around 70-90% on my PIII 866).