Oolite Bulletins

Posted: **Sat Oct 07, 2023 10:19 am**

hiran wrote: ↑Sat Oct 07, 2023 9:21 am
Why would you like to hide it? Is the data in the json file not good?

"uploadDate":null

Posted: **Sat Oct 07, 2023 10:42 am**

hiran wrote: ↑Sat Oct 07, 2023 9:21 am
The release is an archive containing a lot of html files together with an index.html. Rather than having users download and extract them I'd like to make them available for browsing directly.

maybe I didn't search well, but I didn't find the HTML files in the archive :(

Posted: **Sat Oct 07, 2023 11:31 am**

timer wrote: ↑Sat Oct 07, 2023 10:42 am

hiran wrote: ↑Sat Oct 07, 2023 9:21 am
The release is an archive containing a lot of html files together with an index.html. Rather than having users download and extract them I'd like to make them available for browsing directly.
maybe I didn't search well, but I didn't find the HTML files in the archive

Oh, I am sorry. You are right. The release I mentioned does indeed not carry the artifact.
Have a look at https://github.com/HiranChaudhuri/Oolit ... x-20230901.

I assume these many additional files are based on my changes both to the OoliteAddonScanner and the catalog file, and some of them are invalid. I will remove them.

Posted: **Sat Oct 07, 2023 3:00 pm**

phkb wrote: ↑Sat Sep 30, 2023 1:14 pm

hiran wrote: ↑Sat Sep 30, 2023 12:54 pm
It seems the old server was deactivated,
The DB is still active and working. I just logged into it to check.

@phkb, I have a question: are the database IP and web server IP the same?
Do I understand correctly that now the database is NOT in use and its state can be considered as frozen?
And one more thing - can I ask you to send me the latest current database dump?

Posted: **Sat Oct 07, 2023 3:29 pm**

@hiran, what about this?
https://bb.oolite.space/viewtopic.php?p=291501#p291501

Posted: **Sat Oct 07, 2023 3:52 pm**

timer wrote: ↑Sat Oct 07, 2023 10:19 am
hiran wrote: ↑Sat Oct 07, 2023 9:21 am
Why would you like to hide it? Is the data in the json file not good?
NaN-NaN-NaN
https://oolite.space/#oxp
Code: Select all
"uploadDate":null

Oh. On some expansions the upload time is unknown. In such cases you see the null value. But that is absolutely valid JSON. AFAIK it should not be null every time.

So where does the NaN come from?

Edit: maybe I am getting it now. Look for
"upload_date" = 1397505898;
The field has a different name in my generated output...
Without being an expert here: is the frontend code not using the correct field anyway?
https://github.com/OoliteProject/oolite ... te.js#L450

Edit2: I believe the presence of an upload date depends on the presence of a last-modified HTTP header on the download URL. Many of the URLs should be updated as the server responds with
301 Moved permanently
So I intend to update the URLs, which has happened to only 74 OXZs so far. Help is welcome.

Posted: **Sun Oct 08, 2023 3:09 am**

timer wrote: ↑Sat Oct 07, 2023 3:00 pm
are the database IP and web server IP the same?

As far as I can tell, yes. I'm using the same web address to access it, although I've never looked at the underlying IP to see if that's changed.

timer wrote: ↑Sat Oct 07, 2023 3:00 pm
Do I understand correctly that now the database is NOT in use and its state can be considered as frozen?

Based on my observations, yes. During the period when I could do updates, I wasn't seeing the changes I made in the DB reflected on the web page, and vice-versa.

timer wrote: ↑Sat Oct 07, 2023 3:00 pm
can I ask you to send me the latest current database dump?

Look out for a PM shortly.

Posted: **Sun Oct 08, 2023 7:30 am**

hiran wrote: ↑Sat Oct 07, 2023 3:52 pm
timer wrote: ↑Sat Oct 07, 2023 10:19 am
hiran wrote: ↑Sat Oct 07, 2023 9:21 am
Why would you like to hide it? Is the data in the json file not good?
NaN-NaN-NaN
https://oolite.space/#oxp
Code: Select all
"uploadDate":null
Oh. On some expansions the upload time is unknown. In such cases you see the null value. But that is absolutely valid JSON. AFAIK it should not be null every time.

So where does the NaN come from?

Edit: maybe I am getting it now. Look for
"upload_date" = 1397505898;
The field has a different name in my generated output...
Without being an expert here: is the frontend code not using the correct field anyway?
https://github.com/OoliteProject/oolite ... te.js#L450

can u turn off convert of field names to camelCase?
I believe that field names should not change (all compound names have been changed and (for example) oxp download icon fail too).

hiran wrote: ↑Sat Oct 07, 2023 3:52 pm
Edit2: I believe the presence of an upload date depends on the presence of a last-modified HTTP header on the download URL.

interesting thought... wget -S --spider can get information about a file (without downloading)

hiran wrote: ↑Sat Oct 07, 2023 3:52 pm
Many of the URLs should be updated as the server responds with
301 Moved permanently
So I intend to update the URLs, which has happened to only 74 OXZs so far. Help is welcome.

first of all, replace all http to https for all wiki urls

UPD but changing urls in generated file imho bad idea...

Posted: **Sun Oct 08, 2023 7:57 am**

phkb wrote: ↑Sun Oct 08, 2023 3:09 am

timer wrote: ↑Sat Oct 07, 2023 3:00 pm
are the database IP and web server IP the same?
As far as I can tell, yes. I'm using the same web address to access it, although I've never looked at the underlying IP to see if that's changed.

hmm... interesting... hoster intercept requests to port 80, but let them through to 3306

Posted: **Sun Oct 08, 2023 10:07 am**

timer wrote: ↑Sun Oct 08, 2023 7:30 am

hiran wrote: ↑Sat Oct 07, 2023 3:52 pm
Edit: maybe I am getting it now. Look for
"upload_date" = 1397505898;
The field has a different name in my generated output...
Without being an expert here: is the frontend code not using the correct field anyway?
https://github.com/OoliteProject/oolite ... te.js#L450
can u turn off convert of field names to camelCase?

To be honest, I do not bother about the field names in JSON. The OoliteAddonScanner reads all the manifest into internal classes and it generates a plist that sticks to the officially supported format. JSON was never specified as an official format, not to talk about it's structure and field names. So generating JSON is currently a side-product since serializing the internal structure is just too easy. But without extra tweaking the field names stick to the internal variables of OoliteAddonScanner.

So we have two ways to go: Standardize the JSON format to what you expect it to be, or trust in the existing names.

timer wrote: ↑Sun Oct 08, 2023 7:30 am
I believe that field names should not change (all compound names have been changed and (for example) oxp download icon fail too).

Yes, they should not change too often as we would break dependencies otherwise.

timer wrote: ↑Sun Oct 08, 2023 7:30 am

hiran wrote: ↑Sat Oct 07, 2023 3:52 pm
Edit2: I believe the presence of an upload date depends on the presence of a last-modified HTTP header on the download URL.
interesting thought... wget -S --spider can get information about a file (without downloading)

I believe that works since wget follows redirects. I decided to reduce traffic by updating outdated URLs.

timer wrote: ↑Sun Oct 08, 2023 7:30 am

hiran wrote: ↑Sat Oct 07, 2023 3:52 pm
Many of the URLs should be updated as the server responds with
301 Moved permanently
So I intend to update the URLs, which has happened to only 74 OXZs so far. Help is welcome.
first of all, replace all http to https for all wiki urls

UPD but changing urls in generated file imho bad idea...

Feel free to perform that change at the source level. The file is https://github.com/OoliteProject/oolite ... onUrls.txt

Posted: **Sun Oct 08, 2023 6:14 pm**

hiran wrote: ↑Sun Oct 08, 2023 10:07 am
To be honest, I do not bother about the field names in JSON. The OoliteAddonScanner reads all the manifest into internal classes and it generates a plist that sticks to the officially supported format. JSON was never specified as an official format, not to talk about it's structure and field names. So generating JSON is currently a side-product since serializing the internal structure is just too easy. But without extra tweaking the field names stick to the internal variables of OoliteAddonScanner.

The format of the original names uses the underscore; when changing the data format, there is no need to change the naming scheme.
If your code cannot save the original names - please stop rewriting the JSON file, I implement synchronization by other tools.

Posted: **Sun Oct 08, 2023 6:17 pm**

@timer: I checked the java code and a customized json is not difficult. Even more your frontend will not require all the data anyway. Would you like to disclose which fields are really used?

Posted: **Sun Oct 08, 2023 7:15 pm**

hiran wrote: ↑Sun Oct 08, 2023 6:17 pm
@timer: I checked the java code and a customized json is not difficult. Even more your frontend will not require all the data anyway. Would you like to disclose which fields are really used?

If we are talking about fields that contain an underscore, then this
information_url, upload_date, download_url.
But this does not change my deep conviction that when converting a format, the data (field names) should not change.

Posted: **Sun Oct 08, 2023 7:53 pm**

timer wrote: ↑Sun Oct 08, 2023 6:14 pm

hiran wrote: ↑Sun Oct 08, 2023 10:07 am
To be honest, I do not bother about the field names in JSON. The OoliteAddonScanner reads all the manifest into internal classes and it generates a plist that sticks to the officially supported format. JSON was never specified as an official format, not to talk about it's structure and field names. So generating JSON is currently a side-product since serializing the internal structure is just too easy. But without extra tweaking the field names stick to the internal variables of OoliteAddonScanner.
The format of the original names uses the underscore; when changing the data format, there is no need to change the naming scheme.
If your code cannot save the original names - please stop rewriting the JSON file, I implement synchronization by other tools.

I recreated the structure. And with that the website looks nice again.

Posted: **Sun Oct 08, 2023 8:10 pm**

hiran wrote: ↑Sun Oct 08, 2023 7:53 pm
I recreated the structure. And with that the website looks nice again.

Amazing!

I need to think about how to improve management (without breaking the current mechanism) - editing a text file in Git is probably still not the most convenient way (IMHO).
I assumed that the primary source of data would be the database (CF D1), and we would receive information from it using the created API. But adding information into the database is possible either by loading the extension manifest.plist, or (the lazy way) by adding the download URL (I think this method is acceptable for relatively small archive sizes). This will allow us to validate the manifest when adding it.

Oolite Bulletins

Oolite Website Domain & Fixing the Expansions Manager

Re: Oolite Website Domain & Fixing the Expansions Manager

Re: Oolite Website Domain & Fixing the Expansions Manager

Re: Oolite Website Domain & Fixing the Expansions Manager

Re: Oolite Website Domain & Fixing the Expansions Manager

Re: Oolite Website Domain & Fixing the Expansions Manager

Re: Oolite Website Domain & Fixing the Expansions Manager

Re: Oolite Website Domain & Fixing the Expansions Manager

Re: Oolite Website Domain & Fixing the Expansions Manager

Re: Oolite Website Domain & Fixing the Expansions Manager

Re: Oolite Website Domain & Fixing the Expansions Manager

Re: Oolite Website Domain & Fixing the Expansions Manager

Re: Oolite Website Domain & Fixing the Expansions Manager

Re: Oolite Website Domain & Fixing the Expansions Manager

Re: Oolite Website Domain & Fixing the Expansions Manager

Re: Oolite Website Domain & Fixing the Expansions Manager