Virtual memory exhausted

[Svengali]

Could it be that subentities (flashers are subentities) leak the most?

Wasn't there something about a discovery that flashers would need 30 MBs for some reason unknown?

I don't think that FLASHERs are leaking. Even with the Constitution the mem is correctly deallocated (on my machine). When I've reported it (see OXPShipRegulator and Ahrumans answer) I've only said that every FLASHER needs ~300KB. I've used the original Constitution (made by Charly), fixed the problems and tested it - so this is maybe different to the RS Constitution. And this only affects the mem-usage, not leaking.

BTW: L, really, I think you should do some tests on your own. The devs will need every single bit of infos to nail the leaks that are happening with RS/OSE installed. I don't think that RS/OSE is the only oxp that causes/triggers this leaking, but the effect is probably very feelable with your oxp. So please, start digging.

[Lestradae]

Hi Svengali!

... I've only said that every FLASHER needs ~300KB. I've used the original Constitution (made by Charly), fixed the problems and tested it - so this is maybe different to the RS Constitution. And this only affects the mem-usage, not leaking ... I don't think that RS/OSE is the only oxp that causes/triggers this leaking, but the effect is probably very feelable with your oxp. So please, start digging.

1.) How did you fix the Constitution for your tests?

2.) I did dig ... but atm I am more than a bit clueless. I just don't have any idea where to look for the leakage, except that according to Screet, it must have something to do with a common denominator of the four big ship classes, and there especially the Constitution(s) and perhaps the Condor. That's why I was thinking flashers - perhaps subentity count?



L

[Solas]

where can one download the RS/OSE OXP for testing ?

[Svengali]

1.) How did you fix the Constitution for your tests?

I don't remember every step with Charlies Constitution, but for testing it's always the same. First comes the deep dumb blind work:
- Delete unnecessary files for the test (readmes, other cad-files, pdfs, bmps, gifs, demoships, illegal_goods, ...)
- Open every remaining plist
- Delete every unnecessary entry (comments, blank lines, trailing spaces, trailing tabs)
- Replace leading spaces with tabs
- Sort everything alphabetically (first entities, then keys)
- While sorting you will already find some things (bounty as string declared,...)
- Then take a look if every subent has a corresponding entry, same for messages
- For XML - Check if every opening tag has a closing tag. For OpenStep - Check braces, commatas and semikolons.
- Use the verifier (Repeat it)
- Very often there are some unneccessary declarations (mostly weapon_positions, view_positions, max_missiles) for NPCs - e.g. a ship with only a front weapon doesn't need all weapon_positions declared. Delete unneccessary stuff.
- Use the verifier (Repeat it)
- Sort the rest out (mostly fuzzy booleans)

Then comes the interesting part:
- Check the roles, scanClasses and AI declarations - change them if necessary (I gave the Constitution a testrole+AI for the test)
- Check the AIs (syntax)
- Use the verifier (Repeat it)

With Charlies oxps there are still problems after these steps, so:
- Take a deeper look in his subents. The Constitution uses some Dummies. Delete these entries.
- Then fire up Oolite and check the log. If there are still problems - fix them.
- Use the verifier (Repeat it)

And after all these things you can start testing. First with the Flasher declarations (a few times) then delete the Flashers and test it again (a few times). It is a lot of work, but hey - that's oxp'ing.

2.) I did dig ... but atm I am more than a bit clueless. I just don't have any idea where to look for the leakage, except that according to Screet, it must have something to do with a common denominator of the four big ship classes, and there especially the Constitution(s) and perhaps the Condor. That's why I was thinking flashers - perhaps subentity count?

We don't know what it is - so testing is necessary. Try a few things, move entities in and out, maybe delete even hundreds of them to see what happens or delete only subents, single keys. OXPing is not only fun, mainly it is hard work and testing needs a lot more time than scripting. But to get a good result...

[Screet]

Flashers *ARGH*

I just made a very simple test:

I disabled all oxps except neolite, then I did copy the constitutions flashers to the player cobra.

Sudden rise in memory usage by over 200MiB.

I then did copy the same flashers to the two NPC cobras.

The memory usage did rise to 800-1700MiB instantly!

This looks to me as if *every* flasher is consuming a hell lot of memory, while I understood it up to now that only one models flashers would consume memory and multiple instances of one ship would then use the same model, thus not increasing the memory usage that extremely.

The result is, that models like the constitution could cause "out of memory" simply by randomly having too many of them fly around in a single system!

As to leaking memory, this wasn't that conclusive. After a few reloads the game crashed out of mem, but that might well be attributed to randomly having too many cobras fly around. Mem usage minimum was almost 900MiB though. That goes well in accordance with the observation that a single set of constitutions flashers consume around 200 MiB.

Screet

[Solas]

Awesome Screet !

Assuming the problem is not in OXP flasher definition strings ..
maybe Ahruman or one of the other devs could look into this as flashers are defined in the source code.
Constitutions have 250 flashers which would mean that each one consumes approx 1 MB ( and 1 MB each seems excessive ).
if there's a way to determine how many flashers were active when Oolite crashes it should indicate whether there's a problem with the associated memory not being released.

[Lestradae]

OK, that looks as if someone really should have a look into flasher memory consumption, doesn't it

[Svengali]

Sudden rise in memory usage by over 200MiB.

Does it mean that Oolite under Vista uses three times more mem than under XP? Really? Then RS/OSE wouldn't run at all - Oolite wouldn't start at all (>1.2GB == dead man). Can you repeat it, please?

when I've tested this I took (for the first tests) Oolites internal Adder (without any oxp installed). After starting Oolite launch from the station and wait till the mem-usage is nearly stable (important). Then spawn this ship and take a look how much memory it takes. Repeat it a few times (shutting down Oolite, starting again,...). Then add 1, 10 and finally 50 Flashers to the Oolite shipdata and repeat all tests a few times.

On XP (32Bit) it took ~300KB for every Flasher (on my machine) in v1.72.2, windowed mode, full shader, details full. The mem is deallocated when the ship (or the player) leaves and does not leak!

[Screet]

Does it mean that Oolite under Vista uses three times more mem than under XP? Really? Then RS/OSE wouldn't run at all - Oolite wouldn't start at all (>1.2GB == dead man). Can you repeat it, please?

Yes, that's true. When I was busy doing missions, I had to uninstall RS/OSE in order to make more than a very few jumps followed by a "out of memory" crash.

With the Constitution and Condor (flasher ships) removed, it's now pretty good with OSE in.

I did observe a massively higher mem load in general under Vista 32 compared to XP32/64 few weeks ago already. Sometimes the game was allocating almost twice as much memory initially!

I don't know if that's a driver issue or where it comes from.

Screet

[Svengali]

I did observe a massively higher mem load in general under Vista 32 compared to XP32/64 few weeks ago already. Sometimes the game was allocating almost twice as much memory initially!

I don't know if that's a driver issue or where it comes from.

Is it the machine with the Nvidia card? I've updated my drivers a week ago and after that the CPU!!! load was higher than before. So I stepped back to my older drivers and everything is ok now. Hmmm.

[Screet]

I did observe a massively higher mem load in general under Vista 32 compared to XP32/64 few weeks ago already. Sometimes the game was allocating almost twice as much memory initially!

I don't know if that's a driver issue or where it comes from.

Is it the machine with the Nvidia card? I've updated my drivers a week ago and after that the CPU!!! load was higher than before. So I stepped back to my older drivers and everything is ok now. Hmmm.

Yes, the high memory usage with flashers was observed with Vista32 and nVidia card.

However, during my ATI tests, that machine had an ATI 4870x2 installed and used roughly the double amount of memory compared to the XP 32/64 machine with another ATI card.

Screet

[Svengali]

Pffft. Screet. Take a look - http://www.anandtech.com/
Edit: Sorry, direkt linking to this doc is not working. The docs are named A Messy Transition (3 parts)

I've read a lot about the SUPERFETCH in Vista. Maybe I'm a oldschool user, but my system can't know what I want to use in the next minutes. Possiby Windows 7 has a better approach - but I'd bet that it will be a more aggressive system, disallowing users to configure it, collecting more data about users and their behaviours and such a #!$%.

BTW: Start thinking about the day XP is no more supported is the last day for a M$ OS on my machines. And this after more than 20 years. Even my professional apps (for my work) are already available for Linux/Mac systems. The more I think about it... I'm getting angry.

If these guys are professionals than I'd bet they are spending too much time with drinking beer and playing golf.

[Lestradae]

Just submitted the following on BERLIOS:

[ Bug #15476 ] *FLASHER* consumes hundreds of MBs of memory
Date:
2009-Apr-03 10:36
Submitted By:
lestradae Assigned To:
none
Category:
Game Engine Priority:
5
Bug Group:
Crash Resolution:
None
Summary:
*FLASHER* consumes hundreds of MBs of memory
Original Submission:
I have checked time and again: The game-internal *FLASHER* - subentity uses up enormous amounts of memory.

As far as I know, a subentity of a given type should only use its memory slot once?

OK, so flashers can be adjusted into different settings and so might be used more than once. But. I read that one should use up 300kB at most.

As you all know, RS/OSE has a lot of flashers in there. I made a simple experiment and renamed all *FLASHER* subentities into *FLASHER-OFF*.

In other words, ran the game without any and all RS/OSE flashers. Didn't change anything else. Guess what: There was 500MBs (!!) less RAM memory usage at the same configuration - same oxps, same savegame, same everything!

Repeated this three times to be really sure.

This is a serious problem for RAM memory as is obvious to be seen, and I request that someone look into this who can. Thanks in advance, L

[Screet]

I've read a lot about the SUPERFETCH in Vista. Maybe I'm a oldschool user, but my system can't know what I want to use in the next minutes. Possiby Windows 7 has a better approach - but I'd bet that it will be a more aggressive system, disallowing users to configure it, collecting more data about users and their behaviours and such a #!$%.

Finally I've got the system so that I can test it with Vista 64 and more RAM.

What I found is really amazing: The dedicated RAM amount for the graphics cards under Vista appears to be TWICE the amount of RAM installed on the graphics card!

With Vista 32 I can only use 3GiB RAM. Vista reduces it's demand slightly, reserving over 2 GiB RAM for the graphics, so I bet it's no wonder that Oolite can crash "out of memory" at 800 MiB RAM usage. I only wonder how I could get other games to use around 1.4 GiB.

With Vista 64 it's using the 8 GiB RAM and thus reserves the full desired amount of RAM for the graphics card, way over 3GiB.

That appears stupid to me! Trying to reserve the double amount of graphics RAM in normal RAM - WHY??? This practically makes Vista32 unusable when the system has graphics cards with 512 MiB or more RAM installed...no wonder that my XP X64 machine with 2 GiB RAM had less such problems!

Screet