Are OXZ's an improvement ...

[spara]

This must be a stupid question, but I'll ask it anyway. Is there a problem in some systems with the OXZ files. In my system, Debian with xfce, thunar and xarchiver, oxz-files are automatically recognized as archive files. If oxz type is a problem in some systems, needing some extra work to be recognized as an archive, then it's better to just keep it as zip.

Never under-estimate the stupidity of Windows. Under the flashy graphical transitions is the same pile of crud that has always been there. To Windows only .cab and .zip are archives. OXZ files are just random noise until you rename them to .zip.

Ok. Then I must apologize my ignorance as this makes handling OXZ files harder than zip-files in some systems. And that's not a good thing at all .

[Neelix]

While we are putting the world to rights, can anything be done about the location of the expansions (of any flavour). As it stands we now have AddOns and the managed expansions folder in two different locations, and where your AddOns lives seems to be different between O/Ss and even installation location (home/system wide for example). This has been a source of much confusion (particularly for newcomers) on countless occassions in the past.
There might be technical reasons for this, but as a 'layman' in terms of these things it just seems to me that if AddOns (which might also include managed expansions in the future?) should be in the same location as the main Oolite files. Surely having everything together is desirable.

Seconded.

Personally I think this, and the already planned further development of the expansion pack manager is most of what really needs to be done to fix things... People seem to forget that the Manager is in it's infancy, and is not yet full featured, so a lot of the difficulties are simply due to the fact that the features that would obviate the problem have yet to be developed.

Perhaps two features that probably aren't yet planned for the manager should be added to todo list though:
* Set OXP to be managed manually - moves OXZ file from ManagedAddons to Addons and adds the version number to the file name
* Convert to uncompressed OXP format - as before but actually decompresses it into an OXP folder as well.

I think allowing compressed OXPs is a definite step forward... One of the things that struck me when I first encountered Oolite (version 1.77) was that it seemed rather a waste of space to require OXPs to be uncompressed, so I don't think the capability for that should be removed. but certainly ease of tweakability could and should be improved.

- Neelix

[Lone_Wolf]

This must be a stupid question, but I'll ask it anyway. Is there a problem in some systems with the OXZ files. In my system, Debian with xfce, thunar and xarchiver, oxz-files are automatically recognized as archive files. If oxz type is a problem in some systems, needing some extra work to be recognized as an archive, then it's better to just keep it as zip.

On linuxes 'magic bytes' are used to determine file type, and that means extensions are mainly used for readability.
OS X is a unix and very likely works the same.
Afaik Windows has no such method and relies solely on extension for linking files to determine what application needs to handle it.

While we are putting the world to rights, can anything be done about the location of the expansions (of any flavour). As it stands we now have AddOns and the managed expansions folder in two different locations, and where your AddOns lives seems to be different between O/Ss and even installation location (home/system wide for example). This has been a source of much confusion (particularly for newcomers) on countless occassions in the past.
There might be technical reasons for this, but as a 'layman' in terms of these things it just seems to me that if AddOns (which might also include managed expansions in the future?) should be in the same location as the main Oolite files. Surely having everything together is desirable.

Given that Oolite supports linux, OS X and windows, that's going to be very hard.
each of those OSes have their own unique setup where to store programs & user data.

On linux users have write access to /home/username folder, but other locations (besides temporary files) are read-only / not accessible at all.
programs installed in user's home have the same access rights.
A linux systemwide installation of Oolite will store the program under /usr or /opt , which are read-only for non-root users.
user-specific data is always stored in users home folders.

Linux systemwide installs need to use distro-specific package manager which takes care of EVERYTHING related to installing & updating systemwide.
The closest Windows has to a package manager is windows update.
compared to linux package manager windows update is a bloated monster with many limits a decent package manager doesn't need to have.
(the lack of a proper package management system is the main reason why so many programs on windows have their own update mechanism.)

[Smivs]

Hmmm, technical reasons!
If it has to be that way, then we'll have to live with it. Not a big issue.

[Diziet Sma]

I also still personally prefer to develop in OXP format, then convert to OXZ for addition to the manager.

Give cim's build scripts a go.. you may be pleasantly surprised..

Do these actually work on Mac or PC? I haven't had chance to look at them yet (I'm on holiday at the mo), but I note from the comment at the top of their thread that they may need some work to get them to as they were designed and set up on Linux.

I can't speak for Mac at the moment (my macbook is over 1000km from here for the next 2 weeks or so) but it theoretically should work without needing much alteration.. I plan testing on PC in the next day or two.

As I said before, I'm with Smivs about preferring to develop under OXP, as accurately working on files within an archive is usually not fun on a PC (or at least on my PC with my software load-out).

That's the beauty of the scripts.. you're not working on files in an archive.. you're working on uncompressed files in a standard OXP-style folder tree. And they don't even need to be in the AddOns folder. You work on the files wherever you want (mine are in diziet/Projects/Oolite/Dizzy's_OXPs/) and when you're ready to test, run cim's build script from within the project folder. The script will plunk a freshly built OXZ into AddOns, whereupon you run Oolite and test. You then exit Oolite and make the necessary changes to the files in your project folder. When done, run the script again, and a new OXZ containing your new changes, will overwrite the old OXZ, meaning it has a new timestamp as well, so there is no need to force Oolite to rebuild the cache, as Oolite will see the change and do so automatically. Test your changes, and rinse and repeat as required. It really is as easy as falling off a log.. and you only need to change the version number in one place to have the script automatically propagate it for you into every file than needs updating.

(Oh yeah, and every time the script builds a new OXZ, it also builds a new OXP (versioned) for you as well, though it doesn't auto-plunk it into AddOns for you, but puts it at the root level of your project instead, along with another copy of the OXZ version.)

[aegidian]

I think allowing compressed OXPs is a definite step forward... One of the things that struck me when I first encountered Oolite (version 1.77) was that it seemed rather a waste of space to require OXPs to be uncompressed, so I don't think the capability for that should be removed. but certainly ease of tweakability could and should be improved.

No. Space is not an issue, textures are usually already compressed and compressing a few text files saves very little space, we're not using BBC micros here!

AFAICS the only benefit of compression is a single-file download, there's no useful purpose in keeping OXPs compressed beyond that.

[Smivs]

The planet Rear is scourged by evil well-intentioned OXZs

<chuckles>

[cim]

Okay ... the reason that the Managed AddOns folder is not the same place as the AddOns folder, is because I'm not confident in my ability to write an OXP manager which can cope with all the weird things people do in an AddOns folder without raising questions to which the answer is "I didn't think of that case, but it should be fixed in 1.82. In the meantime, do you have any backups?". Especially when so many people use their AddOns folder as the place they develop OXPs, I think that would be a worse sort of problem to have than "I'm not sure how to open this OXZ to customise it".

It's not hidden (it's considerably easier to find from Latest.log than the similarly obscure folder where Oolite keeps its cache file, and if you're familiar with the Cocoa/GNUStep file hierarchy it's even in a logical place): you just shouldn't be doing anything to the contents directly unless you already know what you're doing and why. There are cases where you might want to create a subfolder of it called something.oxp and shift about half the OXZs into it - I did that once while testing something late in 1.79 - but you really shouldn't be doing that unless you have a very specific need and know exactly what it'll break as a result.

Similarly we don't go around widely advertising the location of Oolite's core resources folder, because you're not generally supposed to change it; you're supposed to put a file in AddOns to override it.

The reason I've been saying "don't touch the files in there - download them separately to AddOns if you want to play with them" isn't out of some belief that OXZs should only be opened by trained specialists - it's because I know that if I say "yeah, go ahead, stick a few OXPs in there as well and it should work fine" to someone who could safely do it, the same advice will be given a few weeks or months later by someone well-meaning to a new user who really shouldn't be doing that, who will trigger one of the many things that will go wrong if you change the contents of that folder away from Oolite's fairly strict expectations, and end up with a broken installation that takes a fair bit of looking through Latest.log files to fix.

I do not have the skill or experience in this area to make an expansion manager that would cope with the oddities of the existing OXP format and AddOns folder usages (nested OXPs, same thing installed twice, OXPs modified in place keeping the same manifest ID and version, etc.) with full safety and reliability. For that, it has to be OXZ format (or at least, not OXP format, but a single file format is considerably easier to manage), and it has to have its own working space which we strongly recommend people don't touch.

...

So, short-term - the OXZ manager and the OXZ list on the Oolite website already point out that you can download them without using the manager. [wiki]OXP_howto[/wiki] is linked from the OXZ list on the Oolite website, and needs updating to include information on how to unpack an OXZ to poke around at it. [wiki]OXP[/wiki] also needs updating.

Also short-term, if people want to customise OXZs that they've installed through the manager, the safe way to do this is to uninstall it through the manager, download it and install it to the usual AddOns folder using a web browser, then play around with it however they like in there - either directly if their OS/tools support editing inside a ZIP file, or by uncompressing it to an OXP if not.

Longer-term (i.e. 1.82, so probably a bit over a year away on previous form), a way to give people easier access to the insides of OXZs they download through the manager would be to offer an 'extract' command through the manager for any installed OXZ, which:
- creates a folder in AddOns with an .oxp extension that isn't the name of any existing folder (identifier.version.oxp should usually be safe: possibly it should just refuse to extract if that exists)
- unpacks the OXZ into it
- deletes the OXZ if that all worked (or disables it, if I get the "enable/disable without uninstalling" feature working)
- tells them where to find it
(Note that this expansion is now no longer managed, and won't be upgradable through the manager until they delete the OXP folder, which the manager won't do)

[Wildeblood]

I think you all annoyed him. Shame on youse ungrateful sods.

I like the OXZ format; as I said above, the format is not the problem. As Smivs correctly identifies, the unseemly land rush is the problem.

[aegidian]

I'm sorry cim, and Wildeblood I have to disagree, the OXZ format is a problem. It obfuscates files that users should be able to change, it hides how easy it is to makes ones' own mod, and it has been far, far too rapidly adopted because the expansion manager is so good that everybody wants their mod to work with it, whether it's ready for it or not.

I've given a couple of interviews about Oolite in the past few weeks, and one thing became particularly plain to me. People liked Oolite because its data-driven design made it very easy to mod. I have had to reluctantly conclude that hiding any of that data in an OXZ, doesn't fit with that paradigm - there simply aren't supposed to be mods that users aren't encouraged to tinker with.

The expansion manager, however is very, very good. Its success is why some of the problems with the OXZ format, particularly the one of freezing errors into place, have emerged.

I would like the OXZ format of expansions to go, but I would like the expansion manager to remain. I think it's possible, it will be more difficult to write than the current manager, but as has been pointed out - that's a work in progress.


Short term, I would prefer the current emphasis on producing expansions as OXZs to be scaled back dramatically. Having an expansion that's not intended to be tinkered with is a dangerous conceit. And yes, it will always be possible for a tinkerer to break the game. That's the point, you are allowed to make mistakes, and mess around and learn from doing that. That's how some of the most innovative and outrageous expansions got to be created. Being fussy about OXZs loses that opportunity, and I think it's too important to lose.

Long term, OXPs managed and otherwise should be in one place. And it's perfectly possible to check that managed OXP files are unaltered and in their correct places ready to be updated or uninstalled. Keeping a list of such files in Oolite's application support/preferences area isn't an insurmountable task.


I know cim and others have put a lot of work into supporting the OXZ format and creating the expansion manager. I think it was an experiment that was worth pursuing, but I also think that wrapping OXPs in zip files and moving them out of sight of the user has become a mistake. That's okay, I think experimenting with different ways of doing things is worth making mistakes with, it allows the game to improve rapidly. I really appreciate the work done with this and, as I've said, the expansion manager is a BRILLIANT idea that's come from the same hard work. Let's keep the good stuff, and let the poorer stuff go without feeling anything but pride in having made Oolite better.

[Switeck]

Firstly, something that has barely been mentioned.
I go to a lot of trouble making a readme file for my OXPs/OXZs.
Seriously, I spend far more time-and-effort on that than I do in script coding all of my simpler OXPs!

With OXPs, few will read them.
With OXZs, nobody will read them.

This is a problem!

A link to the OXZ's forum message thread or Wiki page for the OXZ in question is insufficient.
People could be playing the game with limited internet access, having downloaded the game and add-ons some time before. For them, getting online to read a readme seems silly.

"View readme for the OXZ for more details" seems the best solution. Manifest.plist can already include what the readme filename is.

Also short-term, if people want to customise OXZs that they've installed through the manager, the safe way to do this is to uninstall it through the manager, download it and install it to the usual AddOns folder using a web browser, then play around with it however they like in there - either directly if their OS/tools support editing inside a ZIP file, or by uncompressing it to an OXP if not.

Considering many OXZs are already over 20 MB, double-downloading them just to edit them might not be the best solution for people with slow internet connections.

Windows install of Oolite already has a link to the AddOns folder, why not add one for the OXZs folder?
If we're going to manually manage things in that way, no point trying to automate it further.
...Just maybe have a warning that it's a bad idea to uncompress OXZs in their folder.

[Norby]

the OXZ format is a problem.

If I read right the problem is not the filename extension of the one-file packages, it is indifferent if called to .oxp.zip or .oxz (and the second is better for the wiki) so I think you do not want to change this back to .oxp.zip, but you mean the core should unpack the .oxz files into .oxp folders right after download?

[cim]

To Windows only .cab and .zip are archives. OXZ files are just random noise until you rename them to .zip.

Or tell it differently. Any Windows users want to experiment with running

Code: Select all

cmd /c assoc .oxz=CompressedFolder

as administrator?

It did about what you'd expect it to do when I tried it. Makes it much easier to see what's inside them and maybe even edit. (I don't know how you'd reverse it if it turns out to have horrible side-effects though, so don't be too hasty to try it)

Would that resolve the issue with opacity of OXZs? (On Windows, anyway. I assume similar things can be done on Mac OS?)

[Thargoid]

Works beautifully, at least as far as Windows Explorer is concerned.

One thought I did have on this whole discussion (which I'm loathed to get involved in, as there are enough viewpoints and emotions involved already) is it may be worth stepping back and considering what the average player of the game may be after, compared to what the average user who posts on this forum may be after. I don't think that the two are the same, as here we get the more "advanced user" who does want to get more involved, tinker, modify, write add-ons and even get their hands dirty on the raw trunk code.

What Aegidian said before about Oolite being good as it's so modifiable is of course core to this, but I'd draw the distinction between having the game that's modifyable with the vast range of OXPs provided and the step further for those who either don't find OXPs that they want or have ideas for them, and who want to write or tinker (with the former being a further step on from the latter).

For most players who just want to be given blocks to build their own custom version of the game then the OXZ format is perfect. For the latter of course it is less so, and here I'd venture that using a .zip format rather than renaming it .oxz might make things a little clearer (although there are downsides such as storage on the wiki and the format that the decompressed files need to be put into).

I'd be fairly sure that the players who just want "building blocks" will significantly outnumber those who want to dig deeper. This comes from my OXP download figures compared to the forum registration count. Here we have a bit under 3000 registered users (and that includes duplicates, spammers and dead accounts) and my most downloaded OXP has a counter sitting at slightly under 11,000 downloads. So even allowing for multiple versions being downloaded (it's a total download count), we have much more people downloading than are voicing opinions here. Whilst those voiced are definitely valid (and there is no argument from me that those who may want to progress from downloaders to tinkerers and modders should be given all the tools and encouragement to do so) I would just like to remind people that the silent majority do exist and should not be forgotten in all of this discussion.

[Ranthe]

Never under-estimate the stupidity of Windows. Under the flashy graphical transitions is the same pile of crud that has always been there. To Windows only .cab and .zip are archives. OXZ files are just random noise until you rename them to .zip.

Or (especially if you're using something like 7-Zip) associate them with the zip/unzip application and class them as "Oolite Expansion Pack".