Oolite Bulletins

hiran wrote: ↑Tue Oct 03, 2023 7:04 pm

It has happened. The expansions catalog only contains three expansions.
Now it is time to reconstruct the full list.

Edit: I noticed something is weird on the grammar. Some fields are missing. Working on it...
Edit2: Problem fixed. We now just need to add more expansion URLs. If someone has a quick way to get the full list - it is welcome.
Edit3: I added all the existing URLs that I knew of.

We can go into normal maintenance now - which means that OXP authors may keep the file up to date.

Did I break anything with my fixes?

UPD: I think I was in a hurry...

UPD2: I stopped understanding what was happening
why it removed all?
https://github.com/OoliteProject/oolite ... c66bebc71d

timer wrote: ↑Wed Oct 04, 2023 10:46 am

hiran wrote: ↑Tue Oct 03, 2023 7:04 pm

It has happened. The expansions catalog only contains three expansions.
Now it is time to reconstruct the full list.

Edit: I noticed something is weird on the grammar. Some fields are missing. Working on it...
Edit2: Problem fixed. We now just need to add more expansion URLs. If someone has a quick way to get the full list - it is welcome.
Edit3: I added all the existing URLs that I knew of.

We can go into normal maintenance now - which means that OXP authors may keep the file up to date.

Did I break anything with my fixes?

UPD: I think I was in a hurry...

UPD2: I stopped understanding what was happening
why it removed all?
https://github.com/OoliteProject/oolite ... c66bebc71d

Depends on when you touched something. We can check this on the oolite-web repo.

Right now I believe that
- your update mechanism is sabotaged (I did it)
- my update mechanism is active, but...
after I tested successfully with a couple of OXPs I assumed it is mature and added all. The last log I viewed shows there is some special condition that I need to treat better.

So this special situation (which I still need to drill down on) caused the generator to create a file but deliver no content and report success. This empty file was then pushed as the new catalog.

To followup what/why read from here:
https://bb.oolite.space/viewtopic.php?p=291389#p291389

If you have questions just ask.

I improved the generator in OoliteAddonScanner and removed two urls:
- one old one still pointing to oolite.org -> removed
- one showing download problems - commented out

Now the index creation works. And if it does not work, the Github Action will stop, rather than overwriting an otherwise fine expansions index.

hiran wrote: ↑Wed Oct 04, 2023 4:02 pm

Right now I believe that
- your update mechanism is sabotaged (I did it)

How exactly was my mechanism stopped? Changes in oolite-web repo (oo-acc-cf only read rights)?

hiran wrote: ↑Wed Oct 04, 2023 4:02 pm

To followup what/why read from here:
https://bb.oolite.space/viewtopic.php?p=291389#p291389

I read the first post (from phkb) that the old site was down and immediately went to make fixes...
But I was a little hasty with my fixes - first I stopped the cron script on the VPS, then uploaded the index.html file, and ONLY THEN carefully read all posts. By the time UPD2 was written, I already understood how index.html is now generated (in general terms).

My fix would have been completely unnecessary if the index.html file in the repo had not been empty, but on the other hand - if it had not been empty - I probably would have studied it and realized that there was no need to change it.
So, we can probably say that I didn’t do anything wrong with my hasty actions?

hiran wrote: ↑Wed Oct 04, 2023 4:02 pm

If you have questions just ask.

If I understand correctly, your Java code scans (GitHub Action) all manifests in the archives and generates a new “general manifest”? How often does this happen? Archives can be very large - does GitHub really download all this to extract a small manifest?

And one more - we need to create synchronization for oxp.json, otherwise the site will have outdated information on the OXP page. For now I can try to configure it through my VPS.

timer wrote: ↑Thu Oct 05, 2023 8:06 am

hiran wrote: ↑Wed Oct 04, 2023 4:02 pm

Right now I believe that
- your update mechanism is sabotaged (I did it)

How exactly was my mechanism stopped? Changes in oolite-web repo (oo-acc-cf only read rights)?

Yes, that's how I did it.

timer wrote: ↑Thu Oct 05, 2023 8:06 am

hiran wrote: ↑Wed Oct 04, 2023 4:02 pm

To followup what/why read from here:
https://bb.oolite.space/viewtopic.php?p=291389#p291389

I read the first post (from phkb) that the old site was down and immediately went to make fixes...
But I was a little hasty with my fixes - first I stopped the cron script on the VPS, then uploaded the index.html file, and ONLY THEN carefully read all posts. By the time UPD2 was written, I already understood how index.html is now generated (in general terms).

My fix would have been completely unnecessary if the index.html file in the repo had not been empty, but on the other hand - if it had not been empty - I probably would have studied it and realized that there was no need to change it.
So, we can probably say that I didn’t do anything wrong with my hasty actions?

I do not think your action had caused any damage. What happened:
The old webserver was reconfigured to only produce an error page (likely by the provider)
Your script downloaded the index, noticed a change and forwarded the data into the repository. It was working as designed, but from this moment the expansion index was corrupted.
I sabotaged your mechanism and created an alternative. This alternative seemed to work well for a few expansions.
Then I fed in the full list, but some situation made my code run havoc. It just created the new index file, failed to write content and reported success. As a consequence, the empty file got pushed into the repository.

So it was me who emptied the index.

I did not know you were working on the same problem in parallel.

timer wrote: ↑Thu Oct 05, 2023 8:06 am

hiran wrote: ↑Wed Oct 04, 2023 4:02 pm

If you have questions just ask.

If I understand correctly, your Java code scans (GitHub Action) all manifests in the archives and generates a new “general manifest”? How often does this happen? Archives can be very large - does GitHub really download all this to extract a small manifest?

Yes, the expansions are downloaded to the build server. This happens upon every push to the repo and once per month.
To speed up subsequent runs I configured a cache. Currently the expansions sum up to 5 GB. The cache is allowed to go up to 10 GB and last for two weeks.

timer wrote: ↑Thu Oct 05, 2023 8:06 am

And one more - we need to create synchronization for oxp.json, otherwise the site will have outdated information on the OXP page. For now I can try to configure it through my VPS.

The generator can not only emit plist, it can also write json. Where in the oolite-web do you want that file?

hiran wrote: ↑Thu Oct 05, 2023 8:38 am

Yes, the expansions are downloaded to the build server. This happens upon every push to the repo and once per month.
To speed up subsequent runs I configured a cache. Currently the expansions sum up to 5 GB. The cache is allowed to go up to 10 GB and last for two weeks.

oh, when will I get serious about exploring the capabilities of GitHab Action...

hiran wrote: ↑Thu Oct 05, 2023 8:38 am

The generator can not only emit plist, it can also write json. Where in the oolite-web do you want that file?

I have mentioned this file more than once
branch only-media /data/oxp.json

dash before id is not mistake?

timer wrote: ↑Thu Oct 05, 2023 8:53 am

hiran wrote: ↑Thu Oct 05, 2023 8:38 am

The generator can not only emit plist, it can also write json. Where in the oolite-web do you want that file?

I have mentioned this file more than once
branch only-media /data/oxp.json

Sorry, I may have missed that information before.
I will check that location.

timer wrote: ↑Thu Oct 05, 2023 9:07 am

dash before id is not mistake?

It is intentional. Thanks the intutivity of the yaml format.
Every step requires a dash. The name attribute is optional, but for one of the steps I needed the id attribute.

Nice. While the expansion catalog stuff works overall I am fixing details. The latest version should restore dependencies. Here we may introduce even advanced features.

Another thing that is still missing is the upload date.

I'm not sure how Oolite itself performs, but OoliteStarter has a very pedantic catalog parser. It is detecting detail after detail of things just not right.
Having fixed upload dates and dependencies I am now working on a better implementation for the tags.

Wondering why noone else noticed that malfunction. At least noone reported it...

when I parsed and converted the database dump, I also saw various incorrect entries (mostly a mess of dependency names and tags)

timer wrote: ↑Sat Oct 07, 2023 4:30 am

when I parsed and converted the database dump, I also saw various incorrect entries (mostly a mess of dependency names and tags)

I believe the DB was worse than the expansions themselves (syntax wise) but better maintained. So now we have more stable data but need to look into maintenance.

The oxp.json is getting updated meanwhile. Please have a look.

There is a bunch of other files that the OoliteAddonScanner can generate. Since we are doing the heavy lifting of collecting/reading all expansions anyway I'd like to make them more easily accessible.
Currently they end up in releases like
https://github.com/HiranChaudhuri/Oolit ... x-20231006

Do you think we can simply feed them into the site as well?

hiran wrote: ↑Sat Oct 07, 2023 6:47 am

The oxp.json is getting updated meanwhile. Please have a look.

maybe hide the Updated column for now?

hiran wrote: ↑Sat Oct 07, 2023 6:47 am

There is a bunch of other files that the OoliteAddonScanner can generate. Since we are doing the heavy lifting of collecting/reading all expansions anyway I'd like to make them more easily accessible.
Currently they end up in releases like
https://github.com/HiranChaudhuri/Oolit ... x-20231006

Do you think we can simply feed them into the site as well?

um... post a link to the source code on the Download page?

timer wrote: ↑Sat Oct 07, 2023 7:16 am

hiran wrote: ↑Sat Oct 07, 2023 6:47 am

The oxp.json is getting updated meanwhile. Please have a look.

maybe hide the Updated column for now?

Why would you like to hide it? Is the data in the json file not good?

timer wrote: ↑Sat Oct 07, 2023 7:16 am

hiran wrote: ↑Sat Oct 07, 2023 6:47 am

There is a bunch of other files that the OoliteAddonScanner can generate. Since we are doing the heavy lifting of collecting/reading all expansions anyway I'd like to make them more easily accessible.
Currently they end up in releases like
https://github.com/HiranChaudhuri/Oolit ... x-20231006

Do you think we can simply feed them into the site as well?

um... post a link to the source code on the Download page?

The release is an archive containing a lot of html files together with an index.html. Rather than having users download and extract them I'd like to make them available for browsing directly.