Enable Oolite development (2)

[Cholmondely]

Before elevating some artefact to a standard I think we should check how common it is already. So far it seems we have two OXPs, and know nothing about OXZs.

Still I'd not recommend creating a parallel standard since the real solution to this dilemma would be to simply collect information from Index.plist and requires.plist into manifest.plist, which is part of the OXP specification.

I've not taken the time to properly check (it would be a considerable amount of time for this dumb pilot), but I looked at a few more oxps and they had it too. I checked another two oxzs (written as oxzs - not converted into oxzs from oxps) and they did not. When I created my handful of oxzs I never came across any mention of a need for a info.plist

(Why index.plist? None of the ones I found are called that...)

I would guess that the info.plist was probably never used but included by most oxp authors. And when cim created the Expansions Manager he needed more information and so created the Manifest.plist (which has replaced it). I do wonder if the oxp authors bothered to update the version number.

OXPs
Without:
Blackjack's Bullion (Rustybolts)
Dictators (Ramirez) - this is now an oxz, but started off life as an oxp
Factions (m4r35n357)
Galactic Hyperdrive (Wildeblood)
Hotrods (Murgh & Arexack Heretic)
ILS (Norby) - this is now an oxz, but started off life as an oxp
Interstellar Rescue (McLane) - this is now an oxz, but started off life as an oxp
Military Targetting System (CSOTB)
Neolite retextures (3 oxps - SimonB)
Nuit Station (Killer Wolf)
Povray Planets (Submersible)
Orbits (Ebi)
Xeptatl's Sword (Smivs)

With:
Commies (Dr Nil) - this is now an oxz, but started off life as an oxp
Dark Ray (Stranger) - this is now an oxz, but started off life as an oxp
Galactic Navy (Matt634 & Nemoricus)

OXZs
Without:
Anarchies (McLane) - this is now an oxz, but started off life as an oxp
Tori (Murgh)- this is now an oxz, but started off life as an oxp
All of my OXZs

With:
Dark Ray (Stranger) - as mentioned above

[hiran]

Do I understand correctly we have only four OXPs that have some kind of index.plist and a bigger amount without?

Then let's touch those four and migrate them to OXPs with manifest.plist and the issue is over?

[hiran]

To whom it may concern - or: For everyone using OoliteStarter to juggle Oolite versions and expansions...

The latest version OoliteStarter 0.1.17 is out. It guides users through configuration, savegames and OXP mismatches visually and thus allows experienced and inexperienced users to understand what is going on and take action. With that I consider it feature complete. While there are still lots that can be implemented I usually prefer to have a distinct job for a tool and create another one rather than the 1000 parts swiss army knife that fits into no pocket.

Please have a look, enjoy OoliteStarter and let me know if it misbehaves so we can still further improve it.

[arquebus]

To whom it may concern - or: For everyone using OoliteStarter to juggle Oolite versions and expansions...

The latest version OoliteStarter 0.1.17 is out. It guides users through configuration, savegames and OXP mismatches visually and thus allows experienced and inexperienced users to understand what is going on and take action. With that I consider it feature complete. While there are still lots that can be implemented I usually prefer to have a distinct job for a tool and create another one rather than the 1000 parts swiss army knife that fits into no pocket.

Please have a look, enjoy OoliteStarter and let me know if it misbehaves so we can still further improve it.

Couple things with this one I noticed:

First, the regex search field is SLOOOOW. Typing is effectively non-responsive. Also, because it's regex, it's very confusing to use. Most users will expect to be able to type in a portion of text and have the program search for that text found even as a fragment within an entry.

Second, the conflicts don't seem quite right. That comes in 3 parts. Part 1 is the flags. For example, one of my OXPs shows the following flags:

Online, Enabled, Current, Conflict (red), Conflict (brown), Conflicting (less brown)

It's very unclear what those last three mean distinctly.

Part 2 is that some of the conflicts don't quite make sense. For example, I have the following OXZ:

oolite.oxp.UK_Eliter.Ferdelance_3G.oxz

It apparently conflicts with:

oolite.oxp.Ferdelance_3G

But that one is not only not installed, it also doesn't exist. I can't find it in the list.

Lastly, the text box for conflicts does not really behave the way most users would expect. I think only the conflicts that currently exist (i.e., between OXZs that are installed) should be listed. Right now it looks like it just pulls in the list of potential conflicts from the data file and shows the whole thing. The list also includes the expansion itself (in the example above), which is unexpected.

Cosmetic issue: the alphabetization is machine-readable but not quite human-readable. Users won't expect capital letters to sort above lowercase.

The expansion set I'm working with is here:

https://www.dropbox.com/s/9jpfnc1bivsnc ... t.zip?dl=0

[Cholmondely]

Lastly, the text box for conflicts does not really behave the way most users would expect. I think only the conflicts that currently exist (i.e., between OXZs that are installed) should be listed. Right now it looks like it just pulls in the list of potential conflicts from the data file and shows the whole thing. The list also includes the expansion itself (in the example above), which is unexpected.

Ahah! This might explain why I'm seeing conflicts there which don't conflict!

[hiran]

Couple things with this one [OoliteStarter 0.1.17] I noticed:

First, the regex search field is SLOOOOW. Typing is effectively non-responsive. Also, because it's regex, it's very confusing to use. Most users will expect to be able to type in a portion of text and have the program search for that text found even as a fragment within an entry.

This speed depends on your machine and the number of OXPs and - yes - also the regexp you are typing. But I will reduce the number of lookups.
They will no longer occur after every keypress but after you stopped typing. That should allow you to do more efficient text input.

Give it a try and fill the field with just a portion of text and see the program find it even as fragment. Beware though the search is case sensitive. I use it to my advantage but indeed it needs to be known.

Second, the conflicts don't seem quite right. That comes in 3 parts. Part 1 is the flags. For example, one of my OXPs shows the following flags:

Online, Enabled, Current, Conflict (red), Conflict (brown), Conflicting (less brown)

It's very unclear what those last three mean distinctly.

Yes, that is also for me confusing. And I believe it came through the history of development - which does not mean it is good to use. But let's shed some light on this. It would be good to have a screenshot and to know which OXP you are looking at.
Conflict (Red): The expansion conflicts with some other installed expansion(s), regardless whether it is installed or not. Use this to decide if it is
safe to add.
Conflict (Brown): The expansion is installed and conflicting with something. This is already a situation that needs to be resolved.
Conflict (Orange): Indeed the same meaning as Conflict (brown), but I added it when I wanted one color to draw attention throughout OoliteStarter.

If you can read code, this is where the confusion is happening: https://github.com/HiranChaudhuri/Oolit ... .java#L122

So let's reduce. Probably we just need a warning of possible conflict since the expansion is not enabled and an error of actual conflict.

Part 2 is that some of the conflicts don't quite make sense. For example, I have the following OXZ:

oolite.oxp.UK_Eliter.Ferdelance_3G.oxz

It apparently conflicts with:

oolite.oxp.Ferdelance_3G

But that one is not only not installed, it also doesn't exist. I can't find it in the list.

Lastly, the text box for conflicts does not really behave the way most users would expect. I think only the conflicts that currently exist (i.e., between OXZs that are installed) should be listed. Right now it looks like it just pulls in the list of potential conflicts from the data file and shows the whole thing. The list also includes the expansion itself (in the example above), which is unexpected.

I checked the list and indeed found the OXZ you mentioned. And it declares to conflict with oolite.oxp.Ferdelance_3G. Now what should OoliteStarter do? To me it shows the conflict potential in the text box but no tag about actual conflict is displayed for me, and therefore the text box is also not marked up in color.
So your guess is absolutely right: It shows all the declared conflicts but emphasizes those that apply. Here are two screenshots that hopefully clarify:

Cosmetic issue: the alphabetization is machine-readable but not quite human-readable. Users won't expect capital letters to sort above lowercase.

I hope I do not have to touch that subject. Currently I am using the standard Java sorting. In database systems (Oracle, MariaDB, ...) sorting is a huge topic as many users across countries expect different behaviour.

So a collation defines how sorting should be done. It can be changed in Java, see https://docs.oracle.com/javase/tutorial ... /rule.html.
But when you just look at the number of collations supported by MariaDB you may understand that I'd like to ignore the whole topic:
https://mariadb.com/kb/en/supported-cha ... collations

However I am open for help. If you find a maintainable strategy to add these collations and pick the right one for the respective user, then let's go!

The expansion set I'm working with is here:
https://www.dropbox.com/s/9jpfnc1bivsnc ... t.zip?dl=0

Thank you! Reading all the above I thought 'I wish I had this list', and you have it.
Copy taken, for eventual analysis.

[arquebus]

I checked the list and indeed found the OXZ you mentioned. And it declares to conflict with oolite.oxp.Ferdelance_3G. Now what should OoliteStarter do? To me it shows the conflict potential in the text box but no tag about actual conflict is displayed for me, and therefore the text box is also not marked up in color.

Right, but that doesn't address the underlying problem: I only have the one OXZ installed (the UK_Eliter one). I don't have both. So there should not be any conflict between those two, as the second one (oolite.oxp.Ferdelance_3G) isn't installed. Oolite Starter should presumably not be flagging potential conflicts when the conflict itself is not possible in the currently installed set. (BUT SEE BELOW)

Related but not specific to the above issue:

If I see an OXP/OXP in the "problematic" filter, I would expect that at least one of the things it's conflicting with should also appear in the filter. However, given the piecemeal nature of OXP development, I get that some (earlier) OXPs won't be aware of (later) OXPs that conflict with them, and so their data files won't show a mutual conflict. (That is, the test of the earlier OXP does not come back with a conflict because that conflict isn't specified in the data file.) But, if possible, I think that if there is a conflict between installed OXPs in at least one direction, *all* involved OXPs should show up in the "problematic" filter, but perhaps with a notation on the OXPs that are unaware of the conflict.

This would make it possible to make quick decisions about which OXP to retain in a conflict situation, without having to remove the problematic filter and go searching for the unaware OXP to turn it off (if that's what you want to do). Because the unaware OXP will never self-identify as conflicting, there's no easy way to filter for it without using the aware OXP as a "flag" of sorts.

ADDED:
I wonder if perhaps I got thwarted by the capitalization sort, and didn't find oolite.oxp.Ferdelance_3G installed because I was looking in the lowercase f area. (I am not at the right computer at the moment to check this.) And if that's the case, and there really is a conflict, then the "related but not specific to the above issue" section above is more relevant. Having the conflicting oolite.oxp.Ferdelance_3G show up in the "problematic" filter list would have allowed me to avoid the mistake entirely. (As would ignoring capitalization in the sort.)

[arquebus]

I hope I do not have to touch that subject. Currently I am using the standard Java sorting. In database systems (Oracle, MariaDB, ...) sorting is a huge topic as many users across countries expect different behaviour.

So a collation defines how sorting should be done. It can be changed in Java, see https://docs.oracle.com/javase/tutorial ... /rule.html.
But when you just look at the number of collations supported by MariaDB you may understand that I'd like to ignore the whole topic:
https://mariadb.com/kb/en/supported-cha ... collations

However I am open for help. If you find a maintainable strategy to add these collations and pick the right one for the respective user, then let's go!

I'd wager that non-programmers have certain common default assumptions about how things get alphabetized, regardless of country/culture (certainly it's consistent among English speakers). One of those is ignoring capitalization. And sorting with capital letters actually increases the work on the part of the user, because the user has to be aware that they may need to check two positions in the list to find what they're looking for. As UI/UX goes, it's sub-optimal.

Whatever else is going on in the sorting method, I think it should treat capital letters as identical to lowercase. In the (likely nonexistent) edge case of two text strings being identical except for a single letter (one capital, one lowercase), use consisting sorting of one form or another (cap priority or lowercase priority) - but users won't care which, as long as it's consistent.

[hiran]

I checked the list and indeed found the OXZ you mentioned. And it declares to conflict with oolite.oxp.Ferdelance_3G. Now what should OoliteStarter do? To me it shows the conflict potential in the text box but no tag about actual conflict is displayed for me, and therefore the text box is also not marked up in color.

Right, but that doesn't address the underlying problem: I only have the one OXZ installed (the UK_Eliter one). I don't have both. So there should not be any conflict between those two, as the second one (oolite.oxp.Ferdelance_3G) isn't installed. Oolite Starter should presumably not be flagging potential conflicts when the conflict itself is not possible in the currently installed set. (BUT SEE BELOW)

Related but not specific to the above issue:

If I see an OXP/OXP in the "problematic" filter, I would expect that at least one of the things it's conflicting with should also appear in the filter. However, given the piecemeal nature of OXP development, I get that some (earlier) OXPs won't be aware of (later) OXPs that conflict with them, and so their data files won't show a mutual conflict. (That is, the test of the earlier OXP does not come back with a conflict because that conflict isn't specified in the data file.) But, if possible, I think that if there is a conflict between installed OXPs in at least one direction, *all* involved OXPs should show up in the "problematic" filter, but perhaps with a notation on the OXPs that are unaware of the conflict.

This would make it possible to make quick decisions about which OXP to retain in a conflict situation, without having to remove the problematic filter and go searching for the unaware OXP to turn it off (if that's what you want to do). Because the unaware OXP will never self-identify as conflicting, there's no easy way to filter for it without using the aware OXP as a "flag" of sorts.

ADDED:
I wonder if perhaps I got thwarted by the capitalization sort, and didn't find oolite.oxp.Ferdelance_3G installed because I was looking in the lowercase f area. (I am not at the right computer at the moment to check this.) And if that's the case, and there really is a conflict, then the "related but not specific to the above issue" section above is more relevant. Having the conflicting oolite.oxp.Ferdelance_3G show up in the "problematic" filter list would have allowed me to avoid the mistake entirely. (As would ignoring capitalization in the sort.)

I'd like to understand and support your point here. But it needs to be broken down to some algorithm to detect conflicts.
Currently the way I can think of is to use the information from manifest.plist, where an OXP author declares (in)compatibility with other OXPs. Yes, older ones that are no longer maintained will not declare (in)compatibility with younger ones.

How exactly would you do that detection?
Can we create a scenario (expansion set) where conflicts should be marked, and maybe one where conflicts should not be marked?

[hiran]

I hope I do not have to touch that subject. Currently I am using the standard Java sorting. In database systems (Oracle, MariaDB, ...) sorting is a huge topic as many users across countries expect different behaviour.

So a collation defines how sorting should be done. It can be changed in Java, see https://docs.oracle.com/javase/tutorial ... /rule.html.
But when you just look at the number of collations supported by MariaDB you may understand that I'd like to ignore the whole topic:
https://mariadb.com/kb/en/supported-cha ... collations

However I am open for help. If you find a maintainable strategy to add these collations and pick the right one for the respective user, then let's go!

I'd wager that non-programmers have certain common default assumptions about how things get alphabetized, regardless of country/culture (certainly it's consistent among English speakers). One of those is ignoring capitalization. And sorting with capital letters actually increases the work on the part of the user, because the user has to be aware that they may need to check two positions in the list to find what they're looking for. As UI/UX goes, it's sub-optimal.

Whatever else is going on in the sorting method, I think it should treat capital letters as identical to lowercase. In the (likely nonexistent) edge case of two text strings being identical except for a single letter (one capital, one lowercase), use consisting sorting of one form or another (cap priority or lowercase priority) - but users won't care which, as long as it's consistent.

I just tried to play around with Java Collator and see how far I would get with the collations defined inside the platform. No surprise, the sorting is different from normal String sorting. But when i tried to apply this technique to OoliteStarter I did not see the need for it. It already sorts the way you describe it should sort. See this screenshot.



So if you think this is different on your end, could you indicate where exactly?

[arquebus]

---delete me---

[arquebus]

I'd like to understand and support your point here. But it needs to be broken down to some algorithm to detect conflicts.
Currently the way I can think of is to use the information from manifest.plist, where an OXP author declares (in)compatibility with other OXPs. Yes, older ones that are no longer maintained will not declare (in)compatibility with younger ones.

How exactly would you do that detection?
Can we create a scenario (expansion set) where conflicts should be marked, and maybe one where conflicts should not be marked?

So, my understanding of how your conflict detection works is that it checks against all installed OXPs. That is, it reads the manifest of OXP A, which shows an incompatibility with OXP B, and if you have OXP B installed it indicates the conflict, but if you don't have OXP B it does not indicate the conflict (but see the caveat below).

So you already have the results of the test. Would you not be able to then flag OXP B somehow in the installed list? Not necessarily with the same color, but some indicator that *another* OXP (OXP A in this case) says they can't be used together.

For example, I currently have

oolite.oxp.Norby.cag.Telescope
oolite.oxp.Norby.HUDSelector

installed. The Telescope OXP flags a conflict with the HUDSelector OXP, but the reverse is not true. So when I filter by "problematic" *only* the Telescope OXP shows up. I understand why that happens (because HUDSelector itself doesn't know about Telescope), but that's unexpected behavior for a naive/novice player. As soon as Oolite Starter detects that Telescope lists HUDSelector as a conflict, and then sees that HUDSelector is in the installed list, it should mark HUDSelector in some way so that it appears in the "problematic" list alongside Telescope.

Now, to the caveat. At least one OXP has an "invalid" list of conflicting OXPs. The OXP oolite.oxp.UK_Eliter.Ferdelance_3G has two conflicts listed in the conflict box:

oolite.oxp.Ferdelance_3G
oolite.oxp.UK_Eliter.Ferdelance_3G

At first I assumed Oolite Starter's behavior was to include the OXP itself in the conflict list, but having checked other conflicts, I discovered that you're not the one that put the second OXP there in that list! The creator of UK_Eliter Ferdelance put his own OXP in the conflict list - basically flagging it as conflicting with itself. Perhaps Oolite Starter should do a quick check to make sure the OXP itself is not listed as self-conflicting. (This may only be an error in this one OXP, though; I haven't checked others.)

[arquebus]

A totally new question for you!

I see that Oolite Starter has a filter list for "updatable." What exactly is that tracking? Within Oolite, the only expansions that get flagged as "updated pack available for download" are expansions that are currently installed. If it's not installed, it doesn't need to be updated.

However, Oolite Starter lists a bunch of updatable expansions that I don't have installed. What information is Oolite Starter tracking for these?

EDIT:

Ok, I think I know what might be happening here. I ran a list update in Oolite, then updated the light blue entries inside the game, and then when I came back to Oolite Starter my list of updateable expansions had gotten a lot smaller. The ones that remain are edge cases. Here is my "updatable" list:

oolite.oxp.Storm.TionislaChronicleArray:1.04 (downloaded, not enabled)
oolite.oxp.Storm.TionislaChronicleArray:1.05 (enabled)
oolite.oxp.Svengali.BGS:1.10.8 (not downloaded)
oolite.oxp.Svengali.BGS:1.10.9 (not downloaded)
oolite.oxp.Svengali.BGS:2.5.1 (enabled)
oolite.oxp.spara.market_observer:2.3.2 (not downloaded)
oolite.oxp.spara.market_observer:3.7 (enabled)

Note that inside Oolite, if I use the u (installed and updatable) filter, the list is empty, because I've just updated everything inside the game.

So it looks like what is happening is this: if the Oolite expansion list includes more than one version of an expansion, for whatever reason, Oolite Starter will trigger the "updatable" flag on it because the older version exists in the list. Or something. Because each of these has weird listings in Oolite itself:

In my installed list, TionislaChronicleArray shows an installed version of "-" and an available version of 1.05
In the full list, BGS shows up multiple times (BGS 2.5.1; BackGroundSet 1.10.9; BackGroundSet 1.10.
In the full list, Market Observer shows up twice (Market Observer 3.7, Market Observer 2.3.2)

These might just have to be handled as exceptional cases in Oolite Starter. Something is goofy with the expansion list.

[arquebus]

And some more! (Sorry.)

If I use the Validate button on my set, the resulting list includes these two:

oolite.oxp.Lone_Wolf.ShieldCycler
oolite.oxp.Lone_Wolf.ShieldCyclerNext

I only have ShieldCyclerNext installed. ShieldCyclerNext lists the following:

Requires: oolite.oxp.Svengali.Library:1.7.1
Conflicts: oolite.oxp.Lone_Wolf.ShieldCycler

Since I don't have ShieldCycler installed, only ShieldCyclerNext, that can't be why the validation flagged it. But I also have Svengali Library installed, at the right version, so that also can't be why the validation flagged ShieldCyclerNext.

What caused those two expansions to show up in the validation results list?

[arquebus]

I just tried to play around with Java Collator and see how far I would get with the collations defined inside the platform. No surprise, the sorting is different from normal String sorting. But when i tried to apply this technique to OoliteStarter I did not see the need for it. It already sorts the way you describe it should sort. See this screenshot.



So if you think this is different on your end, could you indicate where exactly?

In my "all" list, oolite.oxp.ZygoUgo.shadyAsteroids is sorted just above oolite.oxp.ace_56.Cruzer when I first open Oolite Starter. It appears that this is happening because the underlying list is sorted with capitals before lowercase, and when Oolite Starter brings in the list it doesn't do any of its own sorting until you click on one of the columns. But as soon as you click on the Identifier column, you get the carat and then the list is properly sorted. (You can see in your screenshot that you have set it to ascending sort by Identifier - you had to click the column header to make that happen, it doesn't happen automatically when you open Oolite Starter.)

So that's actually the issue - Oolite Starter is sorting things correctly, but it's not sorting them until you specify a sort parameter. But because the unsorted list is actually sorted, but weirdly, it looks like the sort in Oolite Starter is weird.

Suggestion: enforce a sort by Identifier as soon as Oolite Starter opens.