Databasing/Indexing Oolite

[hiran]

This is visible here (and may need an update, despite the fact that the rough numbers are still correct):

We have 688 expansions, 2313 equipment (way more than your guess of 500), 7255 ships.

The strange thing from my POV is that if some equipment is a missile we would put it under 'offensive', but if the same OXP comes with a HUD also we would put that into 'Astrogatory'.

If categories can only be applied to OXPs we would simply list both. But going back to the single equipment to find the correct category would no longer be possible.
If instead we place categories on the OXP content, like ships and equipment then it would be easy to calculate the OXP's category from it's content.

This scenario is already reality with the Vimana HUD and the Vimana Beam both coming from the same OXP.

Notes on the above!
1) 2313 equipment is not true. It is 2313 mentions of "equipment" (presumably for listing on the F3 screen):

eg:
Secondary Gravity Scanner
Repair 2nd Gravity Scanner
Refund 2nd Gravity Scanner
Repair 2nd Gravity Scanner with second-hand parts
Gravity Scanner
Repair Gravity Scanner
Refund Gravity Scanner
Repair Gravity Scanner with second-hand parts
are 8 of the database equipment entries: but to me they are only 2 pieces of equipment.
[...]
Hence the perceived need to do this manually...

You've got a point there. For a user these many entries seem to come from some common equipment.
Yet interestingly each of these items has it's own unique ID in Oolite (it is mentioned on the detail page for the respective entry). As these entries do have their unique IDs they can potentially be treated differently. And the way they are used looks a bit strange to me either.

There is no way to identify this automatically so I agree this is a humanoid task.

2) Dual listings

So the OXP pages are categorised according to the type of OXP. I hope to categorise the equipment redirect pages according to the type of equipment.

This makes sense absolutely.

So far I've come across two difficulties with dual listings.
i) Some pieces of Equipment (especially those from the vanilla game) have their own content-filled page: so these will also be tagged with the Equipment category.
ii) Some OXP pages include a piece of equipment with the same name as the page: eg Sniper Gun, which is both the name of the OXP and of one of the 6 pieces of equipment described on that page: so that page gets tagged both for the OXP category and for the Laser category - while the other 5 redirect pages get tagged accordingly ( laser / equipment (defensive) / equipment (offensive) ) depending on their content.
I intend to put up a link for the Category:Laser page on the Category:Equipment (Offensive) page.

O August and Illustrious Compiler of Databases - does this make any sense, or am I being a twit?

I did not check this very example, however it seems to be possible per design that OXPs overwrite ships or equipment from the vanilla game. They can even overwrite each other's content, that is why the load sequence is very important. Whenever my indexing automation spots such a replacement it logs a global warning, which can be seen here: http://wiki.alioth.net/index.php/Index_ ... cts#Global

So far I have no idea how to resolve that dilemma. After all in the Wiki we have one page only, and we'd have to define whether this page would be for the vanilla game or for the OXP variant. In case of yet more OXPs overwriting the same stuff we'd be back to square one.

Would it make sense to identify pages uniquely by putting the OXP's identifier as a prefix?
So if ShipX exists in the vanilla game, it could own the page ShipX. If ShipX is also presented by OXP35 then that one could refer to OXP35_ShipX.
Now if OXP36 also were to overwrite ShipX, it would refer to a page called OXP36_ShipX.

[hiran]

I loaded the database now up

https://onedrive.live.com/?id=495C3C97A ... 97A6639589

Just respond, what you think about it and if you find any errors

Hello Josef,

I tried to download that file but it seems a Microsoft Account is required to authenticate.

Hiran

[Cholmondely]

I loaded the database now up

https://onedrive.live.com/?id=495C3C97A ... 97A6639589

Just respond, what you think about it and if you find any errors

Hello Josef,

I tried to download that file but it seems a Microsoft Account is required to authenticate.

Hiran

Umm... same problem as Hiran, I'm afraid...

[Cholmondely]

Hiran: just to say "thank you" for the latest re-jig of your ingenious index: the extra bells and whistles really help! I would never have gotten as far as this without you!

Cholmondeley

[hiran]

Hiran: just to say "thank you" for the latest re-jig of your ingenious index: the extra bells and whistles really help! I would never have gotten as far as this without you!

Cholmondeley

Nice to hear. In the beginning I was not sure whether I was pressing too much - but now I think it's good we had those intense discussions.

[Cholmondely]

Hiran: just to say "thank you" for the latest re-jig of your ingenious index: the extra bells and whistles really help! I would never have gotten as far as this without you!

Cholmondeley

Nice to hear. In the beginning I was not sure whether I was pressing too much - but now I think it's good we had those intense discussions.

Almost finished the basic equipment list. I've done the Equipment OXPs and the Weapon OXPs. I probably missed one or two, but I'm pretty sure that it is only one or two. Currently working my way down the OXP List on the wiki looking for ancient treasure! (... and finding it too ...)! And there may be one or two things from the F4 page which are not equipment artefacts, but which we would consider equipment...

http://wiki.alioth.net/index.php/Category:Equipment: 286 so far!

[Cholmondely]

... is it possible to add a picture here?

Josef,

Just curious about how your gaming is going.

You clearly have a much better idea than most others do about the various OXPs now.

Can I ask which ones you yourself use in-game?

And what sort of game you play (trading/mining/bounty hunter/pirate/contract driven/mission driven...)?

And what you like about Oolite? And what you dislike?

[Cholmondely]

I would guess that a new OutputDir might be an idea - lots of new oxp's: IronAss vols 1-4, Wyverns new additions, Montana's steady stream, etc. Can we include any of the oxp's which are not oxz's?

That index gets rebuilt automatically every month. Just download it from the releases page on Github:
https://github.com/HiranChaudhuri/Oolit ... r/releases

The indexer will cover everything that is distributed via the addons manager. I'd be surprised if OXPs were contained in that list, so someone would have to add that functionality.

Ahah! What would we need to do with our oxp's to enable this? Could we make this feasible for your rebuilding programme? Would we need the oxp's to be stored on the wiki, for example?

[hiran]

I would guess that a new OutputDir might be an idea - lots of new oxp's: IronAss vols 1-4, Wyverns new additions, Montana's steady stream, etc. Can we include any of the oxp's which are not oxz's?

That index gets rebuilt automatically every month. Just download it from the releases page on Github:
https://github.com/HiranChaudhuri/Oolit ... r/releases

The indexer will cover everything that is distributed via the addons manager. I'd be surprised if OXPs were contained in that list, so someone would have to add that functionality.

Ahah! What would we need to do with our oxp's to enable this? Could we make this feasible for your rebuilding programme? Would we need the oxp's to be stored on the wiki, for example?

How would you upload an OXP to the wiki? I guess from a technical standpoint it becomes an OXZ at the moment you zip it up. Then all that is left is to add it to the expansion manager's list of OXZs, and the next indexing run will cover it automatically.

If you do not want to add the OXP to the expansion manager, then how should the indexer find it? How should users find the OXP and get to install it?

[Cholmondely]

That index gets rebuilt automatically every month. Just download it from the releases page on Github:
https://github.com/HiranChaudhuri/Oolit ... r/releases

The indexer will cover everything that is distributed via the addons manager. I'd be surprised if OXPs were contained in that list, so someone would have to add that functionality.

Ahah! What would we need to do with our oxp's to enable this? Could we make this feasible for your rebuilding programme? Would we need the oxp's to be stored on the wiki, for example?

How would you upload an OXP to the wiki? I guess from a technical standpoint it becomes an OXZ at the moment you zip it up. Then all that is left is to add it to the expansion manager's list of OXZs, and the next indexing run will cover it automatically.

If you do not want to add the OXP to the expansion manager, then how should the indexer find it? How should users find the OXP and get to install it?

So OXP's are far less bother than OXZ's. These are better for experimental versions, and of course OXZ's did not exist before 2013.

There are two main lists:
OXP List (mostly up to 2012/3): the last column shows if the oxp exists as an oxz, and has not been updated (apart from a to c) since 2013 or so.
Guide to Unlisted OXP's (stuff I've found here and there since Dec 2020 - a few on the OXP List, many in Box.com collections, but none - I think - on the OXZ list). This also has older versions of oxz's/oxp's for older versions of Oolite. And other stuff - a link to Ahruman's attempt at a 3D version of the F6 chart, for example.

There will be a fair amount of overlap between the 3 lists. There are maybe another 700 oxp's on the other two lists, excluding the ones which are broken and the duplicates.

[hiran]

So OXP's are far less bother than OXZ's. These are better for experimental versions, and of course OXZ's did not exist before 2013.

True. But OXPs are just directories in the filesystems. Would you upload each of these files into a wiki or first zip them up? While zipping, just ensure the file extension changes from OXP to OXZ and you should be done.
And unless you mean to run a local indexing (which could be possible) OXPs would be meaningless.

There are two main lists:
OXP List (mostly up to 2012/3): the last column shows if the oxp exists as an oxz, and has not been updated (apart from a to c) since 2013 or so.
Guide to Unlisted OXP's (stuff I've found here and there since Dec 2020 - a few on the OXP List, many in Box.com collections, but none - I think - on the OXZ list). This also has older versions of oxz's/oxp's for older versions of Oolite. And other stuff - a link to Ahruman's attempt at a 3D version of the F6 chart, for example.

There will be a fair amount of overlap between the 3 lists. There are maybe another 700 oxp's on the other two lists, excluding the ones which are broken and the duplicates.

So this sounds like you might have three groups of OXZs:
- those that are released for easy installation on the expansions manager
- those that are not released/supported but available
- those that are purely unfinished, unstable or merely unfinished.

How about doing the same concept as with the first list, which is the expansions manager? It means one only would have to collect the OXZs in some online location, then add them to one of the three lists. While the expansions manager would automatically feed only from the first list, an indexer could make use of all three and also mark up on which of the list some expansion was found.

Such a concept is followed by a lot of Linux distributions. Check out the meaning of Main/Universe/Restricted/Multiverse in https://help.ubuntu.com/community/Repositories/Ubuntu

[Cholmondely]

Hiran,

just looking at the list of "Errors" in your July '23 edition of the "Index of artefacts",

I find the entry for my Addons for Beginners (Vital Statistics) states this:


1) Description mismatch between OXP Manifest and Expansion Manager at character position 0001 (LATIN CAPITAL LETTER D vs LATIN CAPITAL LETTER V)
2) Identifier mismatch between OXP Manifest and Expansion Manager at character position 0022 (LATIN SMALL LETTER Y vs LATIN SMALL LETTER E)
3) Author mismatch between OXP Manifest and Expansion Manager at character position 0011 (LATIN SMALL LETTER Y vs LATIN SMALL LETTER E)
4) Required Expansions mismatch between OXP Manifest and Expansion Manager at character position 0242 (LATIN CAPITAL LETTER T vs SPACE)
5) Information URL mismatch between OXP Manifest and Expansion Manager at character position 0034 (LATIN CAPITAL LETTER A vs LATIN CAPITAL LETTER V)
6) No version in dependency reference to oolite.oxp.cim.ships-library:null
7) No version in dependency reference to oolite.oxp.spara.TechnicalReferenceLibrary:null
8) No version in dependency reference to oolite.oxp.phkb.GalacticRegistry:null
9) No version in dependency reference to oolite.oxp.z.phkb.XenonUI:null
10) No version in dependency reference to oolite.oxp.spara.market_observer:null
11) No version in dependency reference to oolite.oxp.spara.market_ads:null
12) No version in dependency reference to oolite.oxp.Gnievmir.VimanaHUD:null
13) No version in dependency reference to oolite.oxp.Norby.HUDSelector:null


I just want the EM to download the newest version of each oxp, and not to need to update my own each time one of the others is updated.

For me, this seems not to be an error. Is there something here I should be worried about?

[hiran]

I believe the way we run the website (and with that the Expansion Manager manifest) should be improved.

When you create an OXZ you need to fill in a file manifest.plist. Then you upload the expansion to some location on the internet (often it is the wiki but other locations exist, too).
Finally you would logo to oolite.org and register ,our expansion. During this step you would essentially provide data from manifest.plist plus the download url.

So there are two sources of information avout your OXZ: the manifest.plist and the copy you provided when publishing the OXZ. Guess what happens if you update one but not the other:

The expansion manager would offer you an OXZ via it's catalog, but after downloading you would get something different. These differences are just cosmetic but it could be worse, and the OoliteAddonScanner is the only tool to actually compare the data.

I would like to change the handling but need a common understanding and help for that.

To answer your question:
You do not have to download all the other OXZs to get it right. Just ensure the data within your OXZ and what is on the registey is in sync.