Oolite Bulletins

phkb wrote: ↑Sat Nov 04, 2023 3:28 am

hiran wrote: ↑Fri Nov 03, 2023 9:45 pm

and now knowing where the field might stem from we may take a look at these build scripts (I haven't seen any so far).

I think this was a feature of the previous OXP admin page, where, once you had filled in all the text entry fields and saved the form, it would give you the exact content of the manifest file at the bottom of the page, including things like upload_date and download_url. We don't have this any more with the current system.

Ah, yet one more advantage for the current process.

But to ease OXZ creation we might put a template into the wiki. I can take care of that.

Switeck wrote: ↑Sat Nov 04, 2023 3:13 am

In my case, I am not even sure where/when/how my OXP causes broken behavior in the game because I may not even be looking/studying that.
So I often only find out when someone who tests my OXP broke something.
And then I have to determine which version they are using and go back through my 20+ intermediate version saves to see if earlier versions also have that problem.
Multi-OXP conflicts that arrive from unexpected directions make that really fun, as I may not even be able to reproduce broken behavior until I get an exact OXP list+setup of whoever told me about the problem/s.

Before I know it, I'm troubleshooting other OXPs I at best barely understand.
(Code Documentation! Does anybody do it?!)
...and I may never fix/eliminate the broken behavior spotted.
I usually have to wait till someone else comes along and fixes it.

It's perhaps useful / important that we all have access to at least some earlier versions of various OXPs/OXZs that have seen major changes.
I am concerned we may lose access to the long list of OXPs completely.
I've already been asked if I have a backup of an OXP or 2 that may have only briefly been released before 2012.

More OXP maintainers are needed, even if for 1-off fixes for old, broken OXPs.

Now you are describing issues I am trying to address:

The exact list of OXP/OXZs can be exported with the click of a button when using OoliteStarter.

If we had better control over the release process we could enforce good version numbers, and with version control in the background you would see the code being used within seconds.

That does not solve bugs and it does not provide commented or fixed code. But it would allow you to focus on OXP development rather than (some of) the stuff you described. There is a price to pay - learning to handle these 'layers of complexity'.

The choice is yours - I am willing to help.

phkb wrote: ↑Sat Nov 04, 2023 3:28 am

I think this was a feature of the previous OXP admin page, where, once you had filled in all the text entry fields and saved the form, it would give you the exact content of the manifest file at the bottom of the page, including things like upload_date and download_url. We don't have this any more with the current system.

I thought used the build scripts cim created, ironically hosted on github, but I don't see anything in that code that would do that, so you may be correct, although it is possible that just I copied a manifest.plist from an oxp I had extracted from an oxz someone else had packaged. The documentation was not as comprehensive back then.
cim's build scripts

These are my scripts for easily packaging and testing OXPs/OXZs. They're available in case anyone else wants to use or adapt them, as they've made things much easier for me. Advantages:

No need to shift-restart during development, so you can get quicker from-cache startup the rest of the time.

Update the version in one place, have it propagate everywhere else. This is less useful now that the version in manifest.plist can automatically fill in the version in JS files, but it's still useful.

Provides a common JS script header.

Automatically builds well-packaged OXPs and OXZs.

Automatically runs the OXP verifier (with a little bit of work)

They work with the Linux install locations; they will need some modification to work on Windows or Mac.

These scripts really make it easy for me to package an oxp into the zipped format required for the manager.

hiran wrote: ↑Fri Nov 03, 2023 9:45 pm

Compiling Oolite is far more spectacular than 'compiling' OXPs. I am putting the second compiling into quotes since all a build script would do is to zip up the repository contents - I would not call that compiling. And it is the exact same step you'd have to do in your home installation anyway. There is not even a reason why it should work on one side and not on the other. If you like we can have a look together.

I'm open to having a look, but as I've said, I am very comfortable with the tools I am using. I know I haven't updated or released anything for a long time but I do actually work on a few things when I've got time.

hiran wrote: ↑Sat Nov 04, 2023 5:56 am

If we had better control over the release process we could enforce good version numbers,

What, exactly do you mean by this? What are good version numbers?
As far as I can tell the developers have complete control over how OXZs are linked to the manager. At this point I don't even know how to link a new expansion to it or the proper procedure to upload a new version of any of my existing OXPs and there doesn't seem to be any documentation available that explains how to do that.

hiran wrote: ↑Sat Nov 04, 2023 5:56 am

The exact list of OXP/OXZs can be exported with the click of a button when using OoliteStarter.

If we had better control over the release process we could enforce good version numbers, and with version control in the background you would see the code being used within seconds.

That does not solve bugs and it does not provide commented or fixed code. But it would allow you to focus on OXP development rather than (some of) the stuff you described.

The exact OXP/OXZ list complete with file sizes and/or hash values, to verify those are all unmodified?

Enforce good version numbers?
I am not sure what you mean.
Does this mean OXPs/OXZs will be rejected from the auto-downloader if they don't meet certain criteria?
Or at least not "recommended"?

At the least, having the version in the OXZ's filename would be nice...especially in a way that alphabetically sorts. ( #.10 is less than #.9 alphabetically but not numerically)

"version control in the background you would see the code being used within seconds."

That...sounds like a bit of a performance hit, and some of the bugs I run across only exist under heavy loads.

"That does not solve bugs and it does not provide commented or fixed code."

Much of the problems I have isn't that my code isn't documented.
I'm usually trying to determine other OXP authors' intentions.

Stormrider wrote: ↑Sat Nov 04, 2023 7:37 pm

I thought used the build scripts cim created, ironically hosted on github, but I don't see anything in that code that would do that, so you may be correct, although it is possible that just I copied a manifest.plist from an oxp I had extracted from an oxz someone else had packaged. The documentation was not as comprehensive back then.
cim's build scripts

These scripts really make it easy for me to package an oxp into the zipped format required for the manager.

Thank you for the URL. I will study that resource - maybe I can adapt the scripts as cim suggested such that some nice OXZs actually come out.
Maybe even invoked from OoliteStarter so you can easily jumpstart a new expansion.

Stormrider wrote: ↑Sat Nov 04, 2023 7:37 pm

hiran wrote: ↑Sat Nov 04, 2023 5:56 am

If we had better control over the release process we could enforce good version numbers,

What, exactly do you mean by this? What are good version numbers?

When you release an OXP or OXZ you likely put some version number, let's say 1. Or 1.0 if you prefer. Whatever.
When you release that OXP again, do you really always remember to increase that number? What ensures you always have something that your users can refer to and you know what was inside? Can you from looking at two such numbers easily decide which of them is more recent?
If these numbers really work as they are intended to work, that's what I call 'good version numbers'.

As a good example, take
-1.0
- 1.0.1
- 1.3
- 2.0
You can easily tell these numbers are in ascending order.

As a bad example, take commit hashes like
- 5971f8cfc000582e8bf6ceefc978fde9a84aa47b
- de5ca370429148f522d67d6b296b812176cb55be
- 37870ee21320f51e678dd5095495ecc721948f9b
- 39059aa72559ad099a6506abda4a89984273efcb

Although these version numbers are unique, a human being would not be able to remember one, let alone tell which sequence they are in.

Stormrider wrote: ↑Sat Nov 04, 2023 7:37 pm

As far as I can tell the developers have complete control over how OXZs are linked to the manager. At this point I don't even know how to link a new expansion to it or the proper procedure to upload a new version of any of my existing OXPs and there doesn't seem to be any documentation available that explains how to do that.

True, you spotted a gap. When we discussed repeatedly how the website and expansion publishing should work someone asked if, instead of describing the concept could just showcase it I created the new mechanism, and it seems well active already. But we missed to document it.

Where would you search for such documentation? Then I know where to put answers...
If you like I can also put the old and the current process side by side so you can see the difference.

Probably in this thread makes sense: https://bb.oolite.space/viewtopic.php?f=4&t=16688
However, that thread is long, and we would really only add to the bottom, so it might be time to create a new thread with up to date instructions.

As for it being well active recently, I *think* that’s all me! I don’t think anyone else has been updating OXPs recently.

My thoughts on the new method: it’s super simple to publish a new OXP - just paste the download URL into the text file. Updating is easy too - find the previous URL in the file, and replace it with the new one.

But there are a few downsides: there is no user restrictions or safety rails in place. Anyone who can edit the file can change *anything*, including other users OXPs. There is nothing to prevent accidents, which are probably easily fixed because of versioning, but that then requires someone who knows what they’re doing restoring a previous version. I mean, if someone was wanting to be malicious, they could make life difficult for us!

To be honest, the system feels a little raw. I feel like I’m updating the db at the backend, rather than doing something official.

phkb wrote: ↑Sat Nov 04, 2023 9:07 pm

Probably in this thread makes sense: https://bb.oolite.space/viewtopic.php?f=4&t=16688
However, that thread is long, and we would really only add to the bottom, so it might be time to create a new thread with up to date instructions.

Somehow I believe a thread is not the right form of documentation. It can get diluted by too many comments, and there is no chance of cleaning up.
How about a wiki page that we can keep simple and up to date? We can for sure post the URL into the thread in case someone really reads till the end.

phkb wrote: ↑Sat Nov 04, 2023 9:07 pm

As for it being well active recently, I *think* that’s all me! I don’t think anyone else has been updating OXPs recently.

My thoughts on the new method: it’s super simple to publish a new OXP - just paste the download URL into the text file. Updating is easy too - find the previous URL in the file, and replace it with the new one.

Your impression is correct. Apart from me setting up that repository, all the relevant edits are on your account.
And what you describe is the whole process I had in mind.

phkb wrote: ↑Sat Nov 04, 2023 9:07 pm

But there are a few downsides: there is no user restrictions or safety rails in place. Anyone who can edit the file can change *anything*, including other users OXPs. There is nothing to prevent accidents, which are probably easily fixed because of versioning, but that then requires someone who knows what they’re doing restoring a previous version. I mean, if someone was wanting to be malicious, they could make life difficult for us!

To be honest, the system feels a little raw. I feel like I’m updating the db at the backend, rather than doing something official.

I believe that is also what @timer was concerned about. Now that we have seen my idea in action, let's mull over the missing bits and how to merge them in. But: Has any OXZ author so far tried to publish something? AFAIK there was no pull request, no issue opened, no forum post or anything. So the fact that only phkb was active may not be related to repository privileges. Please someone correct me...

Ultimately we might return to
* https://bb.oolite.space/viewtopic.php?p=291440#p291440
* https://bb.oolite.space/viewtopic.php?p=291570#p291570
* https://bb.oolite.space/viewtopic.php?p=291618#p291618

hiran wrote: ↑Sat Nov 04, 2023 8:27 pm

As a good example, take

Seems like most expansions follow this standard reasonably well, I can't think of any that use hashes or anything so cryptic. I see no problems with something like that, but how do you achieve this on git? Every version of Oolite-trunk I've ever used was hashed this way.

hiran wrote: ↑Sat Nov 04, 2023 8:27 pm

Where would you search for such documentation?

I would say either on this wiki page or on it's own that is linked to it, probably link it to the expansion manager and manifest.plist wiki pages as well. I agree with phkb that a thread on this forum that is dedicated to it's use could be helpful as well.

Stormrider wrote: ↑Sat Nov 04, 2023 10:49 pm

hiran wrote: ↑Sat Nov 04, 2023 8:27 pm

As a good example, take

Seems like most expansions follow this standard reasonably well, I can't think of any that use hashes or anything so cryptic. I see no problems with something like that, but how do you achieve this on git? Every version of Oolite-trunk I've ever used was hashed this way.

What we perceive from OXZs is that they follow such a pattern. But are we sure that for every published OXZ the version number has been increased?
If you are focused on the expansion content and come up with a fix to a difficult to spot problem, do you really remember to first update the manifest.plist before uploading the fixed version? Or is it possible that several versions being published under the same version number, thus overwriting the previous ones and not being recognized by expansion managers or users?

On git many automations just count the number of commits - something you can get via 'git log'.
Alternatively you can just use Gitversion. It is available as Github Action as well.

Stormrider wrote: ↑Sat Nov 04, 2023 10:49 pm

hiran wrote: ↑Sat Nov 04, 2023 8:27 pm

Where would you search for such documentation?

I would say either on this wiki page or on it's own that is linked to it, probably link it to the expansion manager and manifest.plist wiki pages as well. I agree with phkb that a thread on this forum that is dedicated to it's use could be helpful as well.

I agree a forum thread to discuss the wiki page is helpful. Some place where questions get answered and discussions take place.
Looking at the wiki page you are referring to, section Packaging and distribution seems to match best.

Just let me have a moment before I write an update as it seems we are going to discuss the current process again.

hiran wrote: ↑Sat Nov 04, 2023 11:46 pm

Just let me have a moment before I write an update as it seems we are going to discuss the current process again.

Thank you,

Switeck wrote: ↑Sat Nov 04, 2023 8:23 pm

hiran wrote: ↑Sat Nov 04, 2023 5:56 am

If we had better control over the release process we could enforce good version numbers, and with version control in the background you would see the code being used within seconds.

The exact OXP/OXZ list complete with file sizes and/or hash values, to verify those are all unmodified?

So far the list contains OXZ identifier and version number. There is no checksum or guarantee that they are unmodified.

Even more a problem may be the OXPs. First of all they do not have to contain a manifest.plist, and second users are encouraged to play around with them. And they can create side effects just like other OXZs. So what to do?

My solution will not solve or change the problem but make it visible: An OXP can be uniquely identified by it's directory path. Version number is only available if read from a manifest.plist - but handle with care as the files are modified. So if you as expansion developer see what a user has installed you have a chance to react better than without.

To be honest I thought of a checksum, possibly secured with a digital signature to prove that an expansion is untampered. Some indication for users and (OXZ) developers what to look for - I would not ban any of the files. But same as with this thread such mechanisms should be welcome by the majority otherwise I'd have to swim upstream.

Switeck wrote: ↑Sat Nov 04, 2023 8:23 pm

Enforce good version numbers?
I am not sure what you mean.
Does this mean OXPs/OXZs will be rejected from the auto-downloader if they don't meet certain criteria?
Or at least not "recommended"?

At the least, having the version in the OXZ's filename would be nice...especially in a way that alphabetically sorts. ( #.10 is less than #.9 alphabetically but not numerically)

I would try to ensure the OXZ developer does not forget to increase the version number. Which means improve on the build process, similar as cim's scrips already do.

So I believe we should take his scrips, update them if necessary and advertize them wherever suitable.

On the receiving end - when it comes to adding an OXZ to the expansion catalog - there should be some checks. We have not yet discussed what to look for and how to react.

Switeck wrote: ↑Sat Nov 04, 2023 8:23 pm

"version control in the background you would see the code being used within seconds."

That...sounds like a bit of a performance hit, and some of the bugs I run across only exist under heavy loads.

Does it impact the performance of an OXP at runtime if you as OXZ developer can get to the relevant lines of code faster? How so?

Switeck wrote: ↑Sat Nov 04, 2023 8:23 pm

"That does not solve bugs and it does not provide commented or fixed code."

Much of the problems I have isn't that my code isn't documented.
I'm usually trying to determine other OXP authors' intentions.

Truly a problem. Even bigger if the original author dropped support for the expansion.
So what do we do? Remove such expansions from the manager? Discourage the use of them? Ask one person to continue the support? Take over as community?

Switeck wrote: ↑Sat Nov 04, 2023 8:23 pm

I'm usually trying to determine other OXP authors' intentions.

hiran wrote: ↑Sun Nov 05, 2023 6:57 am

Truly a problem. Even bigger if the original author dropped support for the expansion.
So what do we do? Remove such expansions from the manager? Discourage the use of them? Ask one person to continue the support? Take over as community?

OXP interactions are always going to be a problem, and something that the Expansion Manager has limited ways of fixing. By far the easiest way to approach known unwanted interactions, and the *only* way the Expansion Manager can help, is just to flag the OXP in the "conflict_oxps" section of manifest.plist.

The problem is, some of these OXPs are really good, and we want to keep them in our game. Which means we have to wrestle with the code to work out (a) what the author was intending, and then (b) how to keep it working and get it to work *with* our OXP.

This isn't something that the Expansion can really assist with. It really is just hard work on the developers side. Obviously, the community can help, in offering suggestions and workarounds. But the Expansion Manager isn't the place to solve these issues.

As for what to do with legacy OXP's, this has been batted around before, but I think, if the OXP author isn't around anymore, and if it's proven that the OXP is incompatible with the latest version of Oolite, we just update the manifest so that is has a maximum Oolite version inside. If the incompatibilities are OXP related, again, change the manifest and add ID's to the "conflict_oxps". Anything else is beyond the scope of the Expansion Manager.

phkb wrote: ↑Sun Nov 05, 2023 7:39 am

hiran wrote: ↑Sun Nov 05, 2023 6:57 am

Truly a problem. Even bigger if the original author dropped support for the expansion.
So what do we do? Remove such expansions from the manager? Discourage the use of them? Ask one person to continue the support? Take over as community?

OXP interactions are always going to be a problem, and something that the Expansion Manager has limited ways of fixing. By far the easiest way to approach known unwanted interactions, and the *only* way the Expansion Manager can help, is just to flag the OXP in the "conflict_oxps" section of manifest.plist.

The problem is, some of these OXPs are really good, and we want to keep them in our game. Which means we have to wrestle with the code to work out (a) what the author was intending, and then (b) how to keep it working and get it to work *with* our OXP.

This isn't something that the Expansion can really assist with. It really is just hard work on the developers side. Obviously, the community can help, in offering suggestions and workarounds. But the Expansion Manager isn't the place to solve these issues.

As for what to do with legacy OXP's, this has been batted around before, but I think, if the OXP author isn't around anymore, and if it's proven that the OXP is incompatible with the latest version of Oolite, we just update the manifest so that is has a maximum Oolite version inside. If the incompatibilities are OXP related, again, change the manifest and add ID's to the "conflict_oxps". Anything else is beyond the scope of the Expansion Manager.

Good point - we have one more option: Ensure these two expansions are not installed in parallel.

And the expansion manager cannot resolve all problems. First of all incompatibilities need to be spotted. I think that's what our OXP authors are struggling with most. Here clear version numbers and knowledge of the expansions installed by a user helps a lot. Once the root cause is found we can decide how to resolve them. If it is related to interaction of another OXP, some incompatibility can be flagged. But I doubt every problem is due to other expansions, and I doubt all collisions can only be resolved on 'the other side'.

My questions should then read:

So what do we do with unsupported expansions?
- Remove them from the manager?
- Discourage their use?
- Ask one person to continue the support?
- Take over support as community?
- Whenever we spot conflicts with some unsupported expansion, mark them as conflicting in the manifest.plist?

I am not going to address all the issues/questions directly but hopefully this clarifies my position.

phkb wrote:
"We want to attract the widest group of users into this development environment. With only simple tools, and a fairly basic understanding of what’s going on with each file, you can launch into creating your own mod."

...without any consideration of whether your own mod conflicts with other OXPs, or even the core game!

So there probably needs to be a couple levels of separation between a newly-made OXP and thoroughly tested OXZs, for other mod-makers and players benefit.

I am quite happy with posting unfinished or testing-only mods I make as OXPs, even going so far as making them only seen if you look closely at my User contributions page.
This is a low-effort bar for others who want to "barely" get into making/modifying OXPs/OXZs.

By the time an OXZ is available to everyone, it should have passed a couple tests beyond whether it works by itself.

One way to catch unknown problems is maybe something as simple as "New Releases or Updates" category?

There should also be an approximate measure of how much the OXZ changes the game as well as difficulty increase or decrease.

Difficulty changes can be due to combat, economic, or even distances between star, planet, and stations! (Our play time is limited, and sadly our lifetime too...)
Difficulty changes are also difficult to define -- a mission/campaign OXZ may add rare ships that are overwhelmingly helpful to a clean player and murderous to a fugitive player, but while doing the mission/campaign the player is hard-pressed to survive.

Ever had an "Ambience" or "Station" OXZ kill you because of an easy-to-trigger bug? (I have!) How do you define that difficulty?

It would be helpful if OXZs were defined by multiple categories, if applicable, with the strongest ones noted.
I've had to argue the case about what category my own OXZs belong in, because I could only pick one despite them being more complex than that.

And on the subject of categories, "Depreciated/Broken/Hopelessly Buggy" needs to be at least one, and not show by default.
"Switeck's Shipping" belongs there!

Might also be worth adding an "Incomplete/Unfinished" category too!

...Ultimately, I don't have clear answers for much of this!

Levels of OXZs...

I believe as soon as we have the expansion manager promote an expansion and allow easy installation that expansion should have some quality and support.

Which does not mean we have to remove all others immediately.
Instead we could create different lists of OXZs that are stable/compatible, another one that is less stable/unfinished and a third called experimental
Then users can switch on/off their preferences and know what to expect.

The recommendation could be to add only one experimental to a set of stables for constructive testing. And if we can collect metrics like amount of bugs reported vs installations/flight hours we could promote OXPs into ghe next list.