Oolite Bulletins

I've been thinking about some of the changes in 1.80 compared with 1.77 (or 1.76) and whether they were actually the right thing to do - I'm not actually sure any more, and would like your opinions.

Summary: how much OXP breaking is acceptable between stable versions? "None" is a possible answer, but isn't - despite the consistent API - what the current policy provides.

Current policy

The current "stable version" policy is basically that any OXP API which was in a previous 1.xx stable version should also be in all future 1.xx stable versions. I've been thinking about this and the 1.80 release, while trying to put together a list of features to work on for 1.82.

With very few exceptions, any OXP which worked in 1.65 still has the APIs available eight years later to work in 1.80. It's quite unlikely, however, that many OXPs not updated since 1.65 would still fit in with the rest of the game and OXP set.

While the APIs might stay the same, the overall context does change - 1.80 made changes to how the core works which mean that for best performance (or in some cases even functional compatibility with new core features) the following categories of OXPs need updating:

• ships
• ship sets
• stations
• HUDs
• sound sets

(and quite a few other OXPs which include those things)

This is potentially quite a mess, I think. On the other hand, the specific restriction on API changes means that introducing certain new features - even if they'd only break the API used by a couple of OXPs - is either impossible or involves having two different ways to do things available and OXPs very slowly migrating to the better one. Some of those alternative ways take up quite a bit of space in the code, are easily broken, or make the game code inefficient.

Options

So ... there are a few options for how to go forward here, and it makes quite a big difference which is chosen for what sorts of features are planned for 1.82 (and what preparation needs to be in 1.82 to put in frameworks that might become features in 1.84, for that matter)

1. Prioritise OXP compatibility. No changes to what's offered to OXPs. Someone playing without OXPs won't notice much difference between 1.80 and 1.82
2. Planned removal of deprecated features. Some older interfaces will be removed, and OXPs using them will no longer work. So if we do this for 1.82, you'd be notified now (at the start of the journey to 1.82) which ones are going, and have the entire 1.81 process to update and test OXPs to use the newer interfaces - which can be done without using a nightly build. Only those which have a more modern equivalent will be removed. For APIs which are used by a very large number of OXPs we might even start announcing possible 1.84 removals now.
3. Planned change. As well as removal of deprecated features, interfaces will occasionally change to allow new features to be implemented. This would mean that even some OXPs written using the latest 1.80 features will need rewriting for 1.82 - again, this will be announced now and the new features will be put into nightlies as soon as possible, so you'll have, on current form, 12-24 months to (adopt and) rewrite any OXPs which should be kept going, though you'll need to use the nightly builds to test the features.

At the moment, we're currently on "1" for the actual APIs, and approximately on "3" for the wider context OXPs run in. This, I think, is getting us the worst of both worlds: OXPs don't quite die, so there's no urgency to maintain them, but they don't necessarily work either. It makes sense, I think, to use the same option for both - and I'd like some direction on what to do here.

Implications

If we go for option 1, you get a very stable OXPing platform for the foreseeable future - but most new features will need to be added by OXP, with sometimes some help in the core game to add optional hooks for those features to hang from.

If we go for option 3, you can get a lot of new core game features, and some big new OXP capabilities - I have a long list of major features which people have requested at various times on the forum, most of which are impractical to do properly without a few breaks here and there to OXP interfaces - but the catch, of course, is that the community has to do a lot more work to keep OXPs from previous Oolite versions running, and there might have to be some hard choices made over which ones we have time to keep alive.

Option 2 is in-between. Most of the possible features are implementable with option 2 - but rather than being implemented in a single release, they might take two releases (anything that had to take three - unless the intermediate steps were worthwhile in their own right - would probably not happen at all). The slower implementation gives quite a bit more time to update OXPs, though.

Note that this is about the maximum amount of incompatibility the community considers acceptable. If you want option 3, that doesn't make it the default for every change or even most of them - it just makes it a possibility. Whether to go with that fast approach, or try to introduce features in a multi-stage way (option 2) or even to only introduce those parts of the feature which fit with existing OXPs (option 1) still needs to be decided on a feature-by-feature basis as part of discussing the details.

Examples

Here's an example of a few ideas from the big list of previous requests, and how - if it was selected - they might happen under each option:

Requested feature list wrote:

Improve the commodities market so that all of the goods are more useful, trading is more interesting, secondary station markets are easier to balance, new commodities can be defined easily, and commodities.plist is easier to read.

1. Mostly not possible. Changes to the market prices, quantities, etc. will affect existing OXPs which expect the previous commodities data. It should be possible to add a new format for commodities.plist keys for secondary station pricing, though.
2. 1.82 introduces a new property on stations to set their market prices and quantities in the new way, which overrides the 'market' property. New trade-goods.plist introduced but not used for core game prices, to keep older OXPs working as before.
   1.84 removes commodities.plist and market property for stations, sets new prices in core game trade-goods.plist. Ability to add new commodities enabled. OXPers have between 1.82 and 1.84 to update stations.
3. 1.82 introduces new trade-goods.plist with new market data and new shipdata property for configuring station markets. Ignore commodities.plist and old market properties on stations, using some sensible defaults for existing stations. OXPers should be able to update OXPs to work for both 1.80 and 1.82 before 1.82 is released.

This is one where even if option 3 was considered acceptable, we might go for the option 2 approach anyway to reduce disruption - it's just slower to get the same result -on the other hand, we might choose option 3 because there are relatively few market-affecting OXPs to update and it improves the core game faster. That's something we'd discuss when working out exactly how this change might work.

Here's an example of one which isn't really practical except with option 3.

Requested feature list wrote:

Have multiple witchpoints, one for each connected system.

1. Clearly not possible to do this if all existing OXP behaviour is to be preserved.
2. Also not practical - the version which adds the new witchpoints is going to break OXPs which expect the player to arrive at a specific location, and it's not possible to have a version which both has one witchpoint and multiple witchpoints in the same system. There might be some multi-stage process which could cope with it, but I'm not sure what. All the solutions I can think of break something.
3. Implement this, somehow. I haven't given much thought to the details, but it would probably make sense to position the sun at the origin rather than one of the witchpoints, and then rethink a little how spacelanes are defined. It might still take a couple of versions even then - one to add a set of witchpoints not at the origin, and let OXPs adapt to that, and then one to move the first witchpoint away from the origin so the sun can go there.

They're just examples - don't worry at this stage about whether you think the specific examples are actually good ideas.

What next?

So, which option do you think? If you pick 2 or 3, it might also be worth mentioning if you have any personal thresholds for it - e.g. "option 3, but only if fewer than ten OXPs use the feature being changed" or "option 2, but only for Javascript, not plist keys".

(Feel free to suggest options which aren't listed above, too)

cim wrote:

(Feel free to suggest options which aren't listed above, too)

Longshot here but might there be a possibility for some sort of 'oxp converter' for some of the simpler code changes?
I'm imagining this as a seperate programme, not something within oolite itself. I appreciate that (even if it were practical) it may only help with converting them rather than be able to complete the task itself.

Not so much an option but perhaps a consideration...
The tinkerer's workshop thread could be a place for temporary fixes or alterations to be made available (or popular ones requested) by the community, especially for unmaintained oxps.

In terms of which option, personally I think that option 3 might be acceptable as a maximum (but then I don't have a billion oxps to convert...) although wherever there is likely to be a particularly awkward change, I'd probably favour 2.

If option one were my preference then I would at least have the option to keep playing with 1.80 (or earlier) when,

cim wrote:

Someone playing without OXPs won't notice much difference between 1.80 and 1.82

Maybe those with a tonne of oxps to maintain will have more useful input...

Personally I'm inclined to think 2 is the way to go, I was going to suggest to try to maintain compatibility at least 2 release versions back.. eg 1.82 should maintain compatibility back to the 1.77.1 api, 1.84 to the 1.80 api.... That said I can also understand why you would want to do things faster.

cim wrote:

Here's an example of one which isn't really practical except with option 3.

Requested feature list wrote:

Have multiple witchpoints, one for each connected system.

1. Clearly not possible to do this if all existing OXP behaviour is to be preserved.
2. Also not practical - the version which adds the new witchpoints is going to break OXPs which expect the player to arrive at a specific location, and it's not possible to have a version which both has one witchpoint and multiple witchpoints in the same system. There might be some multi-stage process which could cope with it, but I'm not sure what. All the solutions I can think of break something.

Perhaps it would help to help to put some of the features you think would require method 3 to the community for ideas on how they might be implemented in a method 2 staged process. Outline the idea, and the biggest hurdles in the way. We have a lot of creative minds in this community - maybe it would help to put them to work.

- Neelix

Certainly lots to ponder there, and I agree that a better idea of what changes were being considered would help in deciding.
As an aside, I wouldn't want multiple witchpoints at all.

Being a dumb pilot, that's mostly beyond my ken. I have no OXPs to maintain/update - and I play with relatively few OXPs anyway, preferring to sail pretty close to the core game - so I'd go for whatever accelerates the progression of the core game and its potential new features - which would be option 3, yes?

I'd ask for more examples so that I could decide between 2 & 3.

Neelix wrote:

Perhaps it would help to help to put some of the features you think would require method 3 to the community for ideas on how they might be implemented in a method 2 staged process. Outline the idea, and the biggest hurdles in the way. We have a lot of creative minds in this community - maybe it would help to put them to work.

Certainly the details of the transition plans - how quick to move, how to minimise option 3 usage - would be part of the discussions about a new feature concept/implementation.

It actually took quite a while to find a suggestion that couldn't be implemented under option 2 - option 3's advantages are usually speed of implementation and not having to deal with weird cases which arise in the "middle" when OXP 1 is using the old method and OXP 2 is using the new method, rather than being able to do it at all.

Smivs wrote:

As an aside, I wouldn't want multiple witchpoints at all.

Nor me, I think. There are fun things it allows - deliberately misjumping "sideways" and then jumping in to the target system so you arrive at an "uncharted" entry point, for instance - but it feels like too much extra complexity for relatively little gain. I was just struggling to find something which had been suggested which actually required option 3 to do any of it at all.

Smivs wrote:

a better idea of what changes were being considered would help in deciding.

"Considered" probably implies a later stage of the process, I think. I have a list of things which have been suggested on the forum and sounded like an idea with potential to develop further (either as a core feature or as something which would make a good OXP if there was a suitable core framework to put it on) - but had really obvious compatibility issues. So they went on the list in case I had some inspiration later for how to beat the compatibility issues, or (as with New Cargoes) thought of a way to OXP around the problem.

Some other examples from that list: removing unused/barely-used legacy APIs, cleanup of plist formats, adjustments for better cross-OXP compatibility, system scales and consequences, multiple "main" stations, OXPable systems/galaxies, NPC shields and consequences, weapons adjustments, options for balancing OXP equipment, NPC equipment use, ship customisation, upgraded legal system, more interesting trade economy, OXP-defined trade goods, importance of credits to a non-Jameson, player as assassin/escort/pirate, more impact of differences between systems, ways for OXPs to define regional/organisational powers into some sort of framework, procedural mission generation, persistent "famous" NPCs, more dynamic universe, even some sorts of graphic upgrades.

Zireael wrote:

I'd ask for more examples so that I could decide between 2 & 3.

Pick one or two off the list above that you think sound interesting and I'll talk them through.

cim wrote:

Some other examples from that list: removing unused/barely-used legacy APIs

As much as I loathe javascript, I can see no reason why the legacy scripting system wasn't completely removed back in 1.76.

Wildeblood wrote:

As much as I loathe javascript, I can see no reason why the legacy scripting system wasn't completely removed back in 1.76.

It wasn't until 1.77 that there was a JS replacement for every component of it. (And from an OXP community perspective, completely removing it would require significant abandon-or-rewrite decisions for around 10 large OXPs, as well as updates to a bunch of others, which is certainly not a decision to take quickly)

Neelix wrote:

Personally I'm inclined to think 2 is the way to go, I was going to suggest to try to maintain compatibility at least 2 release versions back.. eg 1.82 should maintain compatibility back to the 1.77.1 api, 1.84 to the 1.80 api....

I agree. If we want too much in one step then the release date of 1.82 will be later, maybe near when the 1.84 will be released using the 2. method.

If we're talking about a 12 to 18 months release cycle, then 3 is an entirely appropriate option IMHO. Maintaining backwards compatibility at all cost is a can of worms that you really want to avoid. I'd even go as far as intentionally breaking old OXPs in order to make sure that stuff is updated for a new Oolite version in a mindful way rather then having an OXP that is technically compatible but fails in subtle semantic ways (see the discussion about playability due to a now misbehaving OXP as an example).

maik wrote:

...Maintaining backwards compatibility at all cost is a can of worms that you really want to avoid...

Examples from people more influential than Oolite: 1) the bloated pile of crap that is microsoft windows, 2) the six year delay in the introduction of digital TV in Australia.

cim wrote:

option 3's advantages are usually speed of implementation and not having to deal with weird cases which arise in the "middle" when OXP 1 is using the old method and OXP 2 is using the new method

Not being one of those with a large stable of OXPs to maintain, I'd be inclined to go with option 3. Having seen far too much of it in my last job, I'm not in favour of having to half-do a job twice over (generally, -in the case of my employers- this was a case of do a quick, poor job the first time, then fix it properly later at considerable extra effort and expense).

I much prefer the approach of "do it right, do it once".

Norby wrote:

If we want too much in one step then the release date of 1.82 will be later, maybe near when the 1.84 will be released using the 2. method.

Actually, I think it would save time, not take more time. As cim pointed out, you get faster development, because you're not trying to tippy-toe around the legacy stuff, attempting to change things without breaking the old stuff. To paraphrase what I posted above, it takes less time and effort to do things right the first time, than it does to do a half-arsed job and then fix it later.

OXPable systems/galaxies,

OXP-defined trade goods,

weapons adjustments,

Please tell me how 2 & 3 would differ for those

Diziet Sma wrote:

Not being one of those with a large stable of OXPs to maintain, I'd be inclined to go with option 3. Having seen far too much of it in my last job, I'm not in favour of having to half-do a job twice over (generally, -in the case of my employers- this was a case of do a quick, poor job the first time, then fix it properly later at considerable extra effort and expense).

I much prefer the approach of "do it right, do it once".

Likewise from me, too. The thing is, OXPs can get "broken" in all sorts of ways, anyway - for example, the upgraded NPC AI (which is, I think, indisputably a Good Thing) suddenly turned a whole lot of tough-but-dumb OXP opponents into hugely overpowered, lethal enemies.

I don't, of course, use a lot of OXPs which significantly alter gameplay, and nor do I write them. I can appreciate that this is a much more important topic for those who have crafted lots of large, complex OXPs. But I think this is also an opportunity to give OXP authors a much larger array of tools, buttons, levers, pulleys and so forth to play with in the future.