Expansion testing

[hiran]

Since Oolite supports a powerful expansion system most of the development is focusing on expansions. We have lots of good ones, and with a growing numbers the possible combinations explode.

This means we need lots of testing, and it means not only testing single expansions but combinations. This is definitely nothing one can do manually in a reliable way.

The debug console allows to read and write internals so I am thinking of running macros there. These could setup a scene, emulate some user input and check for expected results.

Together with OoliteStarter's expansion sets this could be seen as test harness. And over time we could gather a number of performance, quality and compatibility tests.

Is this vision somewhat achievable? Would it be useful? Would any of you want to use it or even contribute?

[phkb]

It's an interesting idea, and worth batting around a bit.

Some expansions are going to be easier to test that others. Smaller OXPs will have a greater chance of success in this regard.

The difficulty is in determining if something has worked. Most test systems that I've ever seen, when it comes to randomly generated content, have to seed things in a way to ensure only 1 possibility will ever play out. Otherwise you end up with a lot of data that is different with each test, and no clear way to determine what was right and what wasn't.

So, that means overriding all the spawn routines, forcing a particular configuration on everything (where police are, where the traders are, where the pirates are, etc, and what ship is used for each). With AI's that have random node selection (that is, 50% chance of doing one thing or another), those chances would have to be fixed.

None of this is impossible, but it's a lot of work. Limiting the scope to non-visual OXP's (ie, things that only make changes to data), might be possible. For instance, testing parts of the Bounty system would be relatively easy to test, as you could throw a bunch of "demerits" at the player through code, jump around a bit, and then interrogate the bounty system data to see if everything is the way it should be.

[hiran]

Random events definitely are a challenge. How does randomness happen in Oolite? Is there a chance to seed?

Even if not the tests would have to be meaningful. E.g. test all 8000+ ships if they can dock using ILS information. And it would be up to the test to check for relevant items.

Setting a seed would be so much easier than overriding all the populate methods (which would evade the testing then). So this seems the main issue.

Also I am wondering if Oolite could spin up on a headless machine and be stopped through the debug console.

[hiran]

I grepped the Oolite source code for 'random' and found about three locations that we need to keep an eye on:
- Oolite core itself
- The javascript engine
- plist files (and wherever they get interpreted, somehow I believe it is the core)

Hopefully all of them would boil down to src/Core/legacy_random. And if we can override the seed using a command line parameter we'd be in good shape already.
A downside may be planetinfo.plist and some code in Universe that makes heavy use of randomly generated stuff - based on the correct rand seed.

This was a short analysis and I must confess the details are definitely over my top.