Page 1 of 5

Hosting site for oxz files

Posted: Sat Jan 18, 2014 9:44 am
by Thargoid
As Cim mentioned here, the 1.79 version of the game will use the new .oxz compressed/single file format for expansion packs. These need to be hosted somewhere with direct URL access, so Box.com won't be a suitable site (at least the free accounts) as they don't do direct access.

So I was wondering if people have a recommendation for a suitable replacement site? I'm looking for somewhere which has such direct access, is free, has at least minimal file download logging (for example a download count or similar) and doesn't require too much by way of maintenance (for example doesn't delete the file if not accessed for a period of time, or have too low a bandwidth limit).

Inputs and suggestions welcome, as I know I'm far from the only one currently using Box, but in the near future we may need something of a migration (which is going to be "fun" for me, given my back catalogue :evil: ).

Re: Hosting site for oxz files

Posted: Sat Jan 18, 2014 2:25 pm
by Norby
OXZ manager is a great improvement, thank you cim!

I suggest to make support for http://goo.gl/ redirects to get a download counter which is the main problem if we want to leave box.com. Maybe not so hard to parse the real url from the received data.

There is a comparsion table of hosting services, each has its own weaknesses. I tried the best ones but not found any with counted direct links, please tell if you found one.

Alternatively there was some offer by members in the forum earlier:
- Keeper has unlimited space and bandwidth on keeper1st.com/Oolite.
- Selezen has 5GB of webspace with unlimited bandwidth.
- Cheyd host Griff's oxps on deephorizonindustries.com.

In this case we need to make an upload page (in php) to avoid asking the owner for updates. The Centralised OXP hosting topic contain good ideas, maybe a bit simpler solution is enough.

Moreover Diziet Sma own the oxpcentral.net. If one of the offers will be selected for use then can be set this domain to point there. I suggest to use this name in manifest.plists which help fix hosting problems without changing plists.

Re: Hosting site for oxz files

Posted: Sat Jan 18, 2014 2:50 pm
by Diziet Sma
Norby wrote:
Alternatively there was some offer by members in the forum earlier:
- Keeper has unlimited space and bandwidth on keeper1st.com/Oolite.
- Selezen has 5GB of webspace with unlimited bandwidth.
- Cheyd host Griff's oxps on deephorizonindustries.com.
The only problems with such an approach are that it creates a huge workload for one person to maintain everything, or it requires setting up dozens of FTP accounts if authors are to be capable of maintaining their own OXZs.. and that it leaves things vulnerable to unexpected events happening to the owner of the domain. If hosting payments are not maintained for whatever reason, up to and including the sudden demise of the owner, the entire hosting setup could go permanently offline without warning.
Norby wrote:
Moreover Diziet Sma own the oxpcentral.net. If one of the offers will be selected for use then can be set this domain to point there. I suggest to use this name in manifest.plists which help fix hosting problems without changing plists.
As mentioned before, I'm quite happy to turn ownership of this domain over to Aegidian or the dev team should it be deemed a useful domain for such a project.

Re: Hosting site for oxz files

Posted: Sun Jan 19, 2014 12:11 pm
by maik
Norby wrote:
OXZ manager is a great improvement, thank you cim!

I suggest to make support for http://goo.gl/ redirects to get a download counter which is the main problem if we want to leave box.com. Maybe not so hard to parse the real url from the received data.

There is a comparsion table of hosting services, each has its own weaknesses. I tried the best ones but not found any with counted direct links, please tell if you found one.

Alternatively there was some offer by members in the forum earlier:
- Keeper has unlimited space and bandwidth on keeper1st.com/Oolite.
- Selezen has 5GB of webspace with unlimited bandwidth.
- Cheyd host Griff's oxps on deephorizonindustries.com.

In this case we need to make an upload page (in php) to avoid asking the owner for updates. The Centralised OXP hosting topic contain good ideas, maybe a bit simpler solution is enough.

Moreover Diziet Sma own the oxpcentral.net. If one of the offers will be selected for use then can be set this domain to point there. I suggest to use this name in manifest.plists which help fix hosting problems without changing plists.
I also have about 40GB left on my server with unlimited bandwidth and am happy to offer those.

One other option could be to use the wiki to upload and maintain OXZ files. At the moment, arbitrary file uploading is disabled, but I could allow .oxz files. That way authors can upload their own files (they just need a wiki account), we would have versioning in place, and the nightly wiki backup would mirror them as well. Dylan's server (which runs the wiki) has 30GB left, which should be plenty. If this solution seems workable I can contact him for his approval.

Re: Hosting site for oxz files

Posted: Sun Jan 19, 2014 2:16 pm
by Smivs
Yes, a central repository of some sort would make a lot of sense, and the wiki at first glance looks like a logical candidate.
Personally I can host OXZs on my own site as I currently do for OXPs, but the different system means that I would not be able to get download numbers (stats) easily, and there is currently an issue with making a .oxz file downloadable although there are ways round it.
I for one would be more than happy to have a reliable central repository available, and use it, particularly if stats were also available.

Re: Hosting site for oxz files

Posted: Sun Jan 19, 2014 4:10 pm
by Norby
maik wrote:
I could allow .oxz files
This is the best imho. If we can get a File: page as for images then there is an access counter at the bottom of page which is (almost) as good as a download counter in this case. The oxz manager built in the game can increase it also if download this small page before the real oxz.

Another benefit if we use the wiki as engine is that a new hosting offer is equivalent with a new wiki mirror: more is safer and automatically updated.

I suggest to use oxpcentral.net in plists which can point to the actual wiki server and can be changed by the dev team to another wiki if the main is permanently lost. Moreover can be an oxzhosts.plist bundled into the game with hostnames of all wiki mirrors and retry the download from another if one is not responding.

Current OXPs need about 2GB except planetary textures, but a base oxz always can be in the wiki to count downloads and the large textures in other places. So maik, please ask Dylan. :)

Re: Hosting site for oxz files

Posted: Sun Jan 19, 2014 8:58 pm
by Commander McLane
The Elite Wiki itself can host download files, however there's a file size limit of 2MB, which makes it impractical for OXZs containing more than a few small texture files.

Re: Hosting site for oxz files

Posted: Sun Jan 19, 2014 9:14 pm
by maik
Commander McLane wrote:
The Elite Wiki itself can host download files, however there's a file size limit of 2MB, which makes it impractical for OXZs containing more than a few small texture files.
This is a configurable setting, thanks for reminding me though.

Re: Hosting site for oxz files

Posted: Sun Jan 19, 2014 9:19 pm
by maik
Norby wrote:
Current OXPs need about 2GB except planetary textures, but a base oxz always can be in the wiki to count downloads and the large textures in other places. So maik, please ask Dylan. :)
Just did, including the request to change the size limits for uploading files (thanks again Commander McLane). Do 50MB suffice? Is anyone aware of a bigger OXP except planet textures which would need to be added manually?

Re: Hosting site for oxz files

Posted: Sun Jan 19, 2014 9:23 pm
by Cody
maik wrote:
Is anyone aware of a bigger OXP except planet textures which would need to be added manually?
Griff's shipset (Resources ~120MB), but that is safely hosted by Cheyd (also Solo's shipset, if that was to return).

Re: Hosting site for oxz files

Posted: Sun Jan 19, 2014 9:27 pm
by Thargoid
50MB would easily cover all of mine (probably in total :wink: ).

Max file size of the few larger ones is a little over 10MB and most are much smaller.

Re: Hosting site for oxz files

Posted: Sun Jan 19, 2014 9:35 pm
by Norby
I think no other over 50MB than these due to I downloaded once everything from [wiki]OXP_List[/wiki]:

126M Griff_Shipset_Resources_v1.2.25.oxp
112M RandomHits1.7.oxp
94M griff_station_bundle_fullsize_tex_v1.1.oxp
73M Griff_No_Shaders_ShipSet0.95.oxp
60M Staer9ShipsetShadedv1.1.oxp

Re: Hosting site for oxz files

Posted: Mon Jan 20, 2014 5:51 am
by maik
Email from Dylan regarding using the wiki server for OXZ storage:
Dylan wrote:
Hi

Sounds good to me. Go for it.

Cheers.

Re: Hosting site for oxz files

Posted: Mon Jan 20, 2014 7:42 pm
by Norby
Cody wrote:
Griff's shipset (Resources ~120MB), but that is safely hosted by Cheyd
I think this and RandomHits are a very common OXPs and a mistake if these are left out from the download manager or forced to leave without counter, so I suggest to set the limit to 150MB.
I do not see any danger in this due to only manually approved wiki members can upload files and everything is logged so no place for abuse but a great help for players.

Re: Hosting site for oxz files

Posted: Mon Jan 20, 2014 8:20 pm
by cim
maik wrote:
One other option could be to use the wiki to upload and maintain OXZ files. At the moment, arbitrary file uploading is disabled, but I could allow .oxz files. That way authors can upload their own files (they just need a wiki account), we would have versioning in place, and the nightly wiki backup would mirror them as well. Dylan's server (which runs the wiki) has 30GB left, which should be plenty. If this solution seems workable I can contact him for his approval.
A minor point: it's not strictly required, but it's better to change the download_url for every version of the OXZ file if you're using the automatic downloader: the reason is that if you don't, someone can download version 1.7 but Oolite (with an old index) thinks it's downloading 1.6, so the download will complete and then say "failed". You'd want to make sure you were always using the permanent URL for that file version as the download_url if possible.