Expansion Development

[hiran]

ADCK Imperial Trader V 2.10 is now available:
[...]

Hello Montana05,

you have worked/improved on uncountable expansions and done a great job.

What I'd like to understand a bit is: how/where did you get the code from? Where did you put it afterwards? I there some communication with other authors, or did it all just happen on your desk? Any version control etc behind that process?

[hiran]

Since neither Montana nor anyone else responded to this thread, I'll keep muttering to myself.

What I have on my mind is a Github repository per expansion. It's content is what we all know well of, so we have version control and a way to collaborate it's development. Again via Github Actions it could automatically be checked, transferred into an OXZ and with that be available for download. Plus - depending on our trust - the newly built version could automatically be registered for the expansion manager.

Now such a setup would have to be setup for each an every expansion. It might be error-prone to develop that pattern over an over again.
So my thought is to create a template. One 'Expansion' respository that can be cloned by expansion developers and hosted i their own Github accounts.
Once we have such a repository we obviously should then create guides how to use it so expansion authors would less wonder how to do the infrastructure and get to focus on the expansion's content right away.

Now what do you guy think? Would that be useful?

[Cholmondely]

My worry is that things continually change in the wonderful world of the silicon chip.

Oolite used to run wonderfully on the AppleMac. But the AppleMac changed.

The various images stored on once-free Photobucket now come out like this:



and ditto for the other old once-free photo storage sites.

Box.com turned itself into Box.app and only a few days ago fixed the links.

Et cetera.

What happens when Github starts playing similar games?

[Cody]

What happens when Github starts playing similar games?

Quite! Owned by microslurp, of course!

[hiran]

What happens when Github starts playing similar games?

Good question.

Currently we rely completely on free services. These services are sponsored by different organizations and persons. What do we do if one of those takes a decision with the feared impact?

Please do not argue that the wiki, the forum and the website are sponsored by community members. It has happened that one of them decided to cancel the contract, and there was nothing we could do against it. So even that is not the solution.

In my eyes there is only one thing you can do against it: Setup your own infrastructure. Technically I am able to do so, but I do not scale. Financially I am not able to do it. But only if you setup a contract with a provider you can be sure the contractual conditions will be met - or enforce it via the judiciary system. That is what all companies or organizations do.

In the meantime I'd say: Ask for your money back...

[montana05]

ADCK Imperial Trader V 2.10 is now available:
[...]

Hello Montana05,

you have worked/improved on uncountable expansions and done a great job.

What I'd like to understand a bit is: how/where did you get the code from? Where did you put it afterwards? I there some communication with other authors, or did it all just happen on your desk? Any version control etc behind that process?

Sorry for the late reply Hiran, I am not much at Oolite anymore.

The code came mainly from Cholmondely's links, which he is tireless, publishing on the wiki. Since most of the original authors are long gone, all modifications happen on my desk and there is no communication. After various test runs, the new version will be published on wiki and on the expansion manager.

[hiran]

Hello Montana05,

you have worked/improved on uncountable expansions and done a great job.

What I'd like to understand a bit is: how/where did you get the code from? Where did you put it afterwards? I there some communication with other authors, or did it all just happen on your desk? Any version control etc behind that process?

Sorry for the late reply Hiran, I am not much at Oolite anymore.

The code came mainly from Cholmondely's links, which he is tireless, publishing on the wiki. Since most of the original authors are long gone, all modifications happen on my desk and there is no communication. After various test runs, the new version will be published on wiki and on the expansion manager.

Thank you for coming back on me here. Actually what you said is what I suspected. And the most important part is:
The original authors are long gone. Maybe even intermediate authors are gone. Yet the product shall survive.

Hence my idea to move the code to a location where everyone can see it, modify it and download it with state of the art collaboration tools.
Would you mind if I try to move some expansions according to my thoughts above?

I continue experiments with nginx settings and (my copy-past mistake) sent to addons.oolite.org - http request header Host=oolite.org, fixed it to addons.oolite.org value.

While Montana05 mentioned publishing expansions on the Expansion Manager, and the manifest is served by addons.oolite.space:
How does the publishing process work currently? Or, if we just serve a static file: Where is that file hosted? How can we maintain it?

[timer]

While Montana05 mentioned publishing expansions on the Expansion Manager, and the manifest is served by addons.oolite.space:
How does the publishing process work currently? Or, if we just serve a static file: Where is that file hosted? How can we maintain it?

Well...
I think, the original had two options:
1) PHP on the fly generated a file from the mysql database
2) The file is static and was generated when add/update any manifest via web site
How exactly - we can look at the PHP sources in the git and understand, but I don't think it's so important.

As it is now: initially, I just set up two nginx virtual hosts (my VPS), the first - addons.oolite.space and it forwards to addons.oolite.org, the second oolite.space forwards to oolite.org. These virtual hosts simply do forward requests. On the VPS, the hosts file contains the binding of oolite.org and the IP of the original server, so nginx knows - where to forward. After one day access from my server to the original server was lost (cause unknown) - I placed the plist file on my server, and my server began to give it a local copy - this allowed the in-game OXP manager to work, although the original site was not available. After a couple of days, access to the original server was restored (cause unknown). But since then the update file is still served by my server. I set up a cron task to periodically update the file from the original server, so if someone updates their manifest now, the OXP manager will see this update with some time lag (no more than 15 min).

As I see the new option: on CF Worker we set up the API for editing the OXP database (CF D1/KV), after each update of the manifest, the Worker makes a "build" and push a new static file to the GitHub or stay as static on CF. The API is used by our static site hosted on GitHub. We will get an option essentially no different from what it was.

But, if we switch to the option of generating an update file on the fly, the possibilities are almost endless even without changing the in-game manager (IMHO, I probably don't see many pitfalls yet).

For example, if add to the url (local config) of the update file just a parameter like uniq_pilot_id=18912847128474987 (which is generated when you first enter the site) - this will already allow the site to give back just what was configured and selected on the site.

IMHO the development of the OXP management functionality on the site is very promising - the browser downloads full data about the extensions, and then you can perform any manipulations with them - search, selection, mark as tracked, favorites, create your own meta-packages, etc. We can add various links to extensions: to the wiki, to the forum, to the author's page, etc. I think that viewing, selecting and creating "presets" is easier to do on the side of the website - it's easy to develop and quickly change, unlike the in-game manager.

The problem is that working with the database on the fly requires much more resources than with static - and it is likely that we can go beyond the limitations of the Free plans.

I believe that sooner or later - all free plans will disappear or be curtailed (become unusable for real cases), business is developing not for the sake of charity.

Now, for oolite.org forwarding, I bought a server in the Netherlands for about $15 per YEAR, and it's not that weak - 20GB disk, 8GB RAM... of course, having your own server, you can do anything and not build infrastructure from a bunch of free plans on different services: GitHub, CF, Supabase, HelioHost... But in this case, history may repeat itself... On the other hand, if you keep the documentation and code in the GitHub, then the loss of the server will only require the deployment of a new one - the main thing is not to lose control over the domain.

Hiran, I am ready to clarify any points if something is not clear.

[hiran]

Thank you timer.

What I understood is that currently we serve a static file that gets synced regularly from a source we likely cannot update.

And I agree we should change that, either a) with a dynamic web application (PHP, NodeJS, Java, ...) or b) with something coming through version control.

a) has the advantage of full self service and the danger of being hacked and abused. If someone sneaks in a bad OXP that would get transported and executed on other machines. Remote code execution is the biggest risk in today's world - we are just lucky to run a hopelessly outdated scripting engine.
But also think of the necessary backups of the database and maintenance of that infrastructure.

b) would mean the gitops way. We have a list of trusted URLs in version control and automatically and regularly compile it into the real manifest - to be deployed with the full website. Adding/changing the URL would require a pull request to be approved by some human being. In this process we can add some automatic and some manual approval steps and increase quality and security, while the website remains a simple and static site.

Somehow I prefer b).
What do others think?

[timer]

What I understood is that currently we serve a static file that gets synced regularly from a source we likely cannot update.

It works well.

b) would mean the gitops way. We have a list of trusted URLs in version control and automatically and regularly compile it into the real manifest - to be deployed with the full website. Adding/changing the URL would require a pull request to be approved by some human being. In this process we can add some automatic and some manual approval steps and increase quality and security, while the website remains a simple and static site.

Somehow I prefer b).
What do others think?

um... I don't quite understand how to make it technically easy for developers?...

Everyone who wants to publish their extension will first need to create his own repo in GitHub, then SOMEONE from the "verified / trusted" members of the main repo "somehow adds" a new url (like a git submodule?) to the main repo and then automatic building(gitops?) of the main manifest file?

[hiran]

What I understood is that currently we serve a static file that gets synced regularly from a source we likely cannot update.

It works well.

I agree it works for the time being. But how would we get new expansion in? How would we update/correct existing entries, or remove some if needed?

b) would mean the gitops way. We have a list of trusted URLs in version control and automatically and regularly compile it into the real manifest - to be deployed with the full website. Adding/changing the URL would require a pull request to be approved by some human being. In this process we can add some automatic and some manual approval steps and increase quality and security, while the website remains a simple and static site.

Somehow I prefer b).
What do others think?

um... I don't quite understand how to make it technically easy for developers?...

Everyone who wants to publish their extension will first need to create his own repo in GitHub, then SOMEONE from the "verified / trusted" members of the main repo "somehow adds" a new url (like a git submodule?) to the main repo and then automatic building(gitops?) of the main manifest file?

You've got a point there. Not everyone may be familiar or happy with cloning the repo just to add one line.
Should we allow as an alternative that OXP developers raise issues requesting their URL to be added?

[timer]

What I understood is that currently we serve a static file that gets synced regularly from a source we likely cannot update.

Sorry, this may be misunderstood, but the old site's OXP manifest manager is being used and "It works well".

I agree it works for the time being. But how would we get new expansion in? How would we update/correct existing entries, or remove some if needed?

For now, we can continue to use it http://oolite.space/admin/

I am coding a replacement for this. If phkb gives me mysql DB dump, we can save all current accounts. First, let's just create an analogue of the current /admin part of the site. After this we can switch oolite.space to new site version.

You've got a point there. Not everyone may be familiar or happy with cloning the repo just to add one line.

to change someone else's OXP?

Should we allow as an alternative that OXP developers raise issues requesting their URL to be added?

URL to oxz file?

Do you think that all OXP that will be included in the main manifest file should store their code in GitHub?
Plz, can you describe in more detail?

[phkb]

If phkb gives me mysql DB dump

Check your PMs.

[hiran]

What I understood is that currently we serve a static file that gets synced regularly from a source we likely cannot update.

Sorry, this may be misunderstood, but the old site's OXP manifest manager is being used and "It works well".

I agree it works for the time being. But how would we get new expansion in? How would we update/correct existing entries, or remove some if needed?

For now, we can continue to use it http://oolite.space/admin/

I am coding a replacement for this. If phkb gives me mysql DB dump, we can save all current accounts. First, let's just create an analogue of the current /admin part of the site. After this we can switch oolite.space to new site version.

You've got a point there. Not everyone may be familiar or happy with cloning the repo just to add one line.

to change someone else's OXP?

Should we allow as an alternative that OXP developers raise issues requesting their URL to be added?

URL to oxz file?

Do you think that all OXP that will be included in the main manifest file should store their code in GitHub?
Plz, can you describe in more detail?

Quite some valid points that you are bringing up.
Do you think it would be ok to have an online session, with maybe some more people interested in discussing the topic?

For all: It is sufficient to be interested in the topic to discuss with us. This is not about programming algorithms but a way that would suit us all how to manage our expansions. So please give a shout if you believe to have an opinion.

@timer: might be good to have an up to date status of the current landscape for the session.

[timer]

@timer: might be good to have an up to date status of the current landscape for the session.

Hello, @hiran!
I'm not sure I understood your question )

I can only describe the architecture that I implementing as a base/starting option now. CloudFlare D1 is used as a data store for OXP manifests, CloudFlare Worker implements an API available at api.oolite.space for authentication and updating the database, this API used by our site. When the data changes, the Worker generates a plist "on the fly" and uploads the new file via the GitHub API to the repository to the only-static branch, and then it will be available at the same address http://addons.oolite.space/api/1.0/overview.

Some automation options can be offered, for example, the developer can place the manifest file in the public access (GitHub or another place), and by one click on the our site - Worker can fetch the manifest file, parse it, and update the data in the database. IMHO, it is possible to fully automate the update process through the mechanics of GitHub hooks, which will react to updating the manifest file in the repo and send signal to Worker.

Of course, no one cancels the need for a convenient manifest editor (on the site), which can be used to initially create a file (at least).