Oolite Website Domain & Fixing the Expansions Manager

[timer]

Why would you like to hide it? Is the data in the json file not good?

NaN-NaN-NaN
https://oolite.space/#oxp

Code: Select all

"uploadDate":null

[timer]

The release is an archive containing a lot of html files together with an index.html. Rather than having users download and extract them I'd like to make them available for browsing directly.

maybe I didn't search well, but I didn't find the HTML files in the archive :(

[hiran]

The release is an archive containing a lot of html files together with an index.html. Rather than having users download and extract them I'd like to make them available for browsing directly.

maybe I didn't search well, but I didn't find the HTML files in the archive

Oh, I am sorry. You are right. The release I mentioned does indeed not carry the artifact.
Have a look at https://github.com/HiranChaudhuri/Oolit ... x-20230901.

I assume these many additional files are based on my changes both to the OoliteAddonScanner and the catalog file, and some of them are invalid. I will remove them.

[timer]

It seems the old server was deactivated,

The DB is still active and working. I just logged into it to check.

@phkb, I have a question: are the database IP and web server IP the same?
Do I understand correctly that now the database is NOT in use and its state can be considered as frozen?
And one more thing - can I ask you to send me the latest current database dump?

[timer]

@hiran, what about this?
https://bb.oolite.space/viewtopic.php?p=291501#p291501

[hiran]

Why would you like to hide it? Is the data in the json file not good?

NaN-NaN-NaN
https://oolite.space/#oxp

Code: Select all

"uploadDate":null

Oh. On some expansions the upload time is unknown. In such cases you see the null value. But that is absolutely valid JSON. AFAIK it should not be null every time.

So where does the NaN come from?

Edit: maybe I am getting it now. Look for
"upload_date" = 1397505898;
The field has a different name in my generated output...
Without being an expert here: is the frontend code not using the correct field anyway?
https://github.com/OoliteProject/oolite ... te.js#L450

Edit2: I believe the presence of an upload date depends on the presence of a last-modified HTTP header on the download URL. Many of the URLs should be updated as the server responds with
301 Moved permanently
So I intend to update the URLs, which has happened to only 74 OXZs so far. Help is welcome.

[phkb]

are the database IP and web server IP the same?

As far as I can tell, yes. I'm using the same web address to access it, although I've never looked at the underlying IP to see if that's changed.

Do I understand correctly that now the database is NOT in use and its state can be considered as frozen?

Based on my observations, yes. During the period when I could do updates, I wasn't seeing the changes I made in the DB reflected on the web page, and vice-versa.

can I ask you to send me the latest current database dump?

Look out for a PM shortly.

[timer]

Why would you like to hide it? Is the data in the json file not good?

NaN-NaN-NaN
https://oolite.space/#oxp

Code: Select all

"uploadDate":null

Oh. On some expansions the upload time is unknown. In such cases you see the null value. But that is absolutely valid JSON. AFAIK it should not be null every time.

So where does the NaN come from?

Edit: maybe I am getting it now. Look for
"upload_date" = 1397505898;
The field has a different name in my generated output...
Without being an expert here: is the frontend code not using the correct field anyway?
https://github.com/OoliteProject/oolite ... te.js#L450

can u turn off convert of field names to camelCase?
I believe that field names should not change (all compound names have been changed and (for example) oxp download icon fail too).

Edit2: I believe the presence of an upload date depends on the presence of a last-modified HTTP header on the download URL.

interesting thought... wget -S --spider can get information about a file (without downloading)

Many of the URLs should be updated as the server responds with
301 Moved permanently
So I intend to update the URLs, which has happened to only 74 OXZs so far. Help is welcome.

first of all, replace all http to https for all wiki urls

UPD but changing urls in generated file imho bad idea...

[timer]

are the database IP and web server IP the same?

As far as I can tell, yes. I'm using the same web address to access it, although I've never looked at the underlying IP to see if that's changed.

hmm... interesting... hoster intercept requests to port 80, but let them through to 3306

[hiran]

Edit: maybe I am getting it now. Look for
"upload_date" = 1397505898;
The field has a different name in my generated output...
Without being an expert here: is the frontend code not using the correct field anyway?
https://github.com/OoliteProject/oolite ... te.js#L450

can u turn off convert of field names to camelCase?

To be honest, I do not bother about the field names in JSON. The OoliteAddonScanner reads all the manifest into internal classes and it generates a plist that sticks to the officially supported format. JSON was never specified as an official format, not to talk about it's structure and field names. So generating JSON is currently a side-product since serializing the internal structure is just too easy. But without extra tweaking the field names stick to the internal variables of OoliteAddonScanner.

So we have two ways to go: Standardize the JSON format to what you expect it to be, or trust in the existing names.

I believe that field names should not change (all compound names have been changed and (for example) oxp download icon fail too).

Yes, they should not change too often as we would break dependencies otherwise.

Edit2: I believe the presence of an upload date depends on the presence of a last-modified HTTP header on the download URL.

interesting thought... wget -S --spider can get information about a file (without downloading)

I believe that works since wget follows redirects. I decided to reduce traffic by updating outdated URLs.

Many of the URLs should be updated as the server responds with
301 Moved permanently
So I intend to update the URLs, which has happened to only 74 OXZs so far. Help is welcome.

first of all, replace all http to https for all wiki urls

UPD but changing urls in generated file imho bad idea...

Feel free to perform that change at the source level. The file is https://github.com/OoliteProject/oolite ... onUrls.txt

[timer]

To be honest, I do not bother about the field names in JSON. The OoliteAddonScanner reads all the manifest into internal classes and it generates a plist that sticks to the officially supported format. JSON was never specified as an official format, not to talk about it's structure and field names. So generating JSON is currently a side-product since serializing the internal structure is just too easy. But without extra tweaking the field names stick to the internal variables of OoliteAddonScanner.

The format of the original names uses the underscore; when changing the data format, there is no need to change the naming scheme.
If your code cannot save the original names - please stop rewriting the JSON file, I implement synchronization by other tools.

[hiran]

@timer: I checked the java code and a customized json is not difficult. Even more your frontend will not require all the data anyway. Would you like to disclose which fields are really used?

[timer]

@timer: I checked the java code and a customized json is not difficult. Even more your frontend will not require all the data anyway. Would you like to disclose which fields are really used?

If we are talking about fields that contain an underscore, then this
information_url, upload_date, download_url.
But this does not change my deep conviction that when converting a format, the data (field names) should not change.

[hiran]

To be honest, I do not bother about the field names in JSON. The OoliteAddonScanner reads all the manifest into internal classes and it generates a plist that sticks to the officially supported format. JSON was never specified as an official format, not to talk about it's structure and field names. So generating JSON is currently a side-product since serializing the internal structure is just too easy. But without extra tweaking the field names stick to the internal variables of OoliteAddonScanner.

The format of the original names uses the underscore; when changing the data format, there is no need to change the naming scheme.
If your code cannot save the original names - please stop rewriting the JSON file, I implement synchronization by other tools.

I recreated the structure. And with that the website looks nice again.

[timer]

I recreated the structure. And with that the website looks nice again.

Amazing!

I need to think about how to improve management (without breaking the current mechanism) - editing a text file in Git is probably still not the most convenient way (IMHO).
I assumed that the primary source of data would be the database (CF D1), and we would receive information from it using the created API. But adding information into the database is possible either by loading the extension manifest.plist, or (the lazy way) by adding the download URL (I think this method is acceptable for relatively small archive sizes). This will allow us to validate the manifest when adding it.