Oolite Bulletins

timer wrote: ↑Wed Oct 11, 2023 4:36 am

hiran wrote: ↑Tue Oct 10, 2023 3:27 pm

timer wrote: ↑Tue Oct 10, 2023 3:14 pm

ooh, I just took a closer look at "Oolite Known Universe"... it seems to me that there is too much "raw" information there...
I'm not sure that we should dump this information on site visitors in such a form and in such a volume...

I need a space where users can browse the files directly. I did not ask to place them in the middle of the homepage...

https://artifacts.oolite.space

...I built a house for you. It is over there. But I won't tell you how to get in...

Thank you for that URL. But where do I have to send the files?

hiran wrote: ↑Wed Oct 11, 2023 5:32 am

timer wrote: ↑Wed Oct 11, 2023 4:36 am

hiran wrote: ↑Tue Oct 10, 2023 3:27 pm

I need a space where users can browse the files directly. I did not ask to place them in the middle of the homepage...

https://artifacts.oolite.space

...I built a house for you. It is over there. But I won't tell you how to get in...

Thank you for that URL. But where do I have to send the files?

at the moment, I just copied the files to my VPS server. BUT I know you won't like this solution

although I can give access to the VPS and you can do a lot of things on it - this serer is for tasks related to Oolite.

I propose the following solution - on an existing (or you can create a new) repo in GitHub - enable GitHub Pages and bind the subdomain name to this instance. I believe that moving artifacts into a separate resource is more correct than trying to put it into the site - in this case, we don’t have to worry about not matching site styles, and develop this resource separately. We can place a link to this subdomain in the Community section first.

timer wrote: ↑Wed Oct 11, 2023 5:52 am

hiran wrote: ↑Wed Oct 11, 2023 5:32 am

timer wrote: ↑Wed Oct 11, 2023 4:36 am

https://artifacts.oolite.space

Thank you for that URL. But where do I have to send the files?

at the moment, I just copied the files to my VPS server. BUT I know you won't like this solution

If it is a manual or one-time update, I've alrady had the Wiki page. Noone likes to do that on a regular base.
So you are guessing correctly.

timer wrote: ↑Wed Oct 11, 2023 5:52 am

I propose the following solution - on an existing (or you can create a new) repo in GitHub - enable GitHub Pages and bind the subdomain name to this instance. I believe that moving artifacts into a separate resource is more correct than trying to put it into the site - in this case, we don’t have to worry about not matching site styles, and develop this resource separately. We can place a link to this subdomain in the Community section first.

Sounds usable. Could you configure the access from the domain? I believe you are much faster than I am.

@phkb: We need yet another repo.

hiran wrote: ↑Wed Oct 11, 2023 6:06 am

Sounds usable. Could you configure the access from the domain? I believe you are much faster than I am.

No problem
I will be happy to make a setup, but in reality, yes - you can do it too, I just want to point out that such actions are not possible only for me (the system does not depend on just one person).

So I updated Manual Witchspace Alignment today. New version is 2.7, and I included the information url in the manifest. The expansion list now has 2.7 listed (and the link to the wiki is working successfully), but the release date is 1/1/1970. Do we need to put the upload_date in the manifest as well?

phkb wrote: ↑Wed Oct 11, 2023 6:46 am

So I updated Manual Witchspace Alignment today. New version is 2.7, and I included the information url in the manifest. The expansion list now has 2.7 listed (and the link to the wiki is working successfully), but the release date is 1/1/1970. Do we need to put the upload_date in the manifest as well?

We do not. The list in expansionUrls.txt needs to be updated.
Many URLs point to outdated locations, and the servers respond with a "301 Moved permanently". Many clients just silently follow the redirect - even OoliteAddonScanner does when downloading the expansions. But that actually means we always need two (or more) lookups to find one expansion - hence we do not follow redirects when getting the upload date.

Here you can see whereever we updated the URL the upload_date would become valid.
https://github.com/OoliteProject/oolite ... 13980c9793

hiran wrote: ↑Wed Oct 11, 2023 6:56 am

Here you can see whereever we updated the URL the upload_date would become valid.

So, I would have to change the download URL each time in order to get the system to change the upload date? For reference, for my OXP's I keep the same URL no matter what version is being released. I find that keeps it simple from a management point of view.

With this setup, would I need to put the upload date in? Or do I have to change my methodolgy and have a new download URL with every release?

hiran wrote: ↑Wed Oct 11, 2023 6:56 am

Here you can see whereever we updated the URL the upload_date would become valid.

Wouldn't a change in the version number be a better trigger for a change in the upload date?

phkb wrote: ↑Wed Oct 11, 2023 7:07 am

hiran wrote: ↑Wed Oct 11, 2023 6:56 am

Here you can see whereever we updated the URL the upload_date would become valid.

Wouldn't a change in the version number be a better trigger for a change in the upload date?

The upload_date is not connected to the date when you update the url in the list.
However a URL lookup returns a Last-Modified header - which resembles the file date just like in a filesystem.

The reason why we need to update the url is that the "301 Moved permanently" does not contain any Last-Modified header. While the code could again silently follow the redirect and get the Last-Modified timestamp from there there is an unnecessary lookup that I want to remove. Yes, I am pedantic here.


Just for completeness: Let's look at two scenarios where an expansion gets updated:
1) Someone creates a new version of the expansion. The filename contains the version number. When publishing this new version on the web it has a URL that was not there before. We will have to add this URL to the oolite-expansions-catalog, from then on the scanner will access the expansion. Metadata is extracted and put into the catalog, and the Last-Modified header is taken as the upload_date.
I think this is what we intend to have.

2) Someone creates a new version of the expansion. The filename however does not contain the version number. When publishing this new version on the web it overwrites the previously existing file. The URL to the expansion remains the same - yet the content changed. As the URL is already in oolite-expansions-catalog no change is required. But as the scanner is not just triggered on repo changes but also regularly, sooner or later the scanner will access the expansion. Metadata is extracted and put into the catalog, and the Last-Modified header is taken as the upload_date.
This covers the case where an expansion is changed without URL changes.

hiran wrote: ↑Wed Oct 11, 2023 7:11 am

The reason why we need to update the url is that the "301 Moved permanently" does not contain any Last-Modified header. While the code could again silently follow the redirect and get the Last-Modified timestamp from there there is an unnecessary lookup that I want to remove. Yes, I am pedantic here.

Sooooo... I have to change my methodology to change the url each time?

phkb wrote: ↑Wed Oct 11, 2023 7:14 am

hiran wrote: ↑Wed Oct 11, 2023 7:11 am

The reason why we need to update the url is that the "301 Moved permanently" does not contain any Last-Modified header. While the code could again silently follow the redirect and get the Last-Modified timestamp from there there is an unnecessary lookup that I want to remove. Yes, I am pedantic here.

Sooooo... I have to change my methodology to change the url each time?

You don't - please look at my edit in previous post.

I'm sure everything will become easier when we manage the list of URLs through the new admin panel - it will be possible to force a signal to rescan.

@hiran, Can GitHub Action execute perl scripts? )

timer wrote: ↑Wed Oct 11, 2023 7:27 am

I'm sure everything will become easier when we manage the list of URLs through the new admin panel - it will be possible to force a signal to rescan.

@hiran, Can GitHub Action execute perl scripts? )

We are using Github hosted runners - that is the servers executing the actions. Here you can see their 'supported software':
https://docs.github.com/en/actions/usin ... d-software

Assuming we use Ubuntu 22 LTS, there seems to be Perl 5.34.0 installed.
https://github.com/actions/runner-image ... -Readme.md

However I do not believe it will become easier with the admin panel. It seems to me PHKB updates expansions by overwriting the online files.
In that case there is not even the need to go to an admin panel to enter the new version number.

hiran wrote: ↑Wed Oct 11, 2023 7:32 am

We are using Github hosted runners - that is the servers executing the actions. Here you can see their 'supported software':
https://docs.github.com/en/actions/usin ... d-software

thanks! there is a starting point to explore...

hiran wrote: ↑Wed Oct 11, 2023 7:32 am

Assuming we use Ubuntu 22 LTS, there seems to be Perl 5.34.0 installed.
https://github.com/actions/runner-image ... -Readme.md

hiran wrote: ↑Wed Oct 11, 2023 7:32 am

However I do not believe it will become easier with the admin panel. It seems to me PHKB updates expansions by overwriting the online files.
In that case there is not even the need to go to an admin panel to enter the new version number.

I agree, but only if rescanning happens quite often and it is fast.

I want to clarify about the admin panel - I believe that its main purpose is to allow everyone to add their own extensions.
1) register on the site
2) add your URL
3) manage your URLs

IMHO this is difficult to do only through GitHub - we cannot give everyone write rights.