Oolite Bulletins

Selezen wrote:

It's certainly a step in the right direction, but again it's still not "open" enough to ensure quick resolution of issues.

...

Although I do understand your position, maik, I believe that the restrictions you put on the access would potentially cripple the ability to make changes/fixes quickly IF the need arose.

Why is it not fast enough? Assume that there already is a backup even while I am there. Once I am not there anymore, a new one can be searched for (or even before) and handed one of the stock accounts. There is no delay. Just a hurdle to qualify, that's all. cim proved that the hurdle is not too high. Winston passes it. I assume Aegidian would, assuming he is running this BB on Linux. There are probably others in the Oolite on Linux crowd.

aegidian wrote:

The aim of securing access to Oolite, oxps, and the bulletin board is a noble one.

However, since the threats to these mostly arise out of the fact that the main sites are maintained by fallible individuals, putting all these sites under one roof would crystallise those threats into a single point of possible failure.

However I'm not sure what to suggest in terms of decentralisation that could help with ensuring access. There are many better brains here - let's open the search for solutions to the whole community and see how many ideas we can genereate.

For your info, I PM'ed Aegidian to ask his opinion on creating a live backup slave for the BB (which is read-only while the master BB is available. In case it is running on MySQL, it seems easy enough to setup and is even made for only occasionally connected slaves. That way, when the BB becomes unavailable we will have its up-to-date content in the backup BB, allow read-write to the backup BB, and redirect people to use this one until the master becomes available again. Then we will have to reverse roles, get the main site up-to-date again, and reverse roles again to keep updating the backup as before.

Edit to add: I also suggested that I would test that first with other servers by installing the BB to gain experience with DB replication in this particular case.

maik wrote:

In case it is running on MySQL, it seems easy enough to setup and is even made for only occasionally connected slaves.

The theory is as you say, but my experience with MySQL replication has been fairly unpleasant, for what it's worth. It just about works most of the time, but it can take a while to synchronise changes and it's very easy for the secondary to fall behind when the primary is doing a lot of writes and then stay behind for some time. (And this is on a fast intranet with constant connectivity; occasional internet-speed connections could be much worse)

Good to know. What does it mean when you say "when the primary is doing a lot of writes"? How would you assess this BB in contrast?

maik wrote:

Good to know. What does it mean when you say "when the primary is doing a lot of writes"? How would you assess this BB in contrast?

I would expect that a single BB would have relatively few insert/update/delete operations on the scale of things, and those that there are should be relatively small - this was for a fairly heavily-used server where it got increasingly far behind the live server while trying to recover its synchronisation. There may have been other bottlenecks involved, possibly, of course.

As long as there's reasonably constant connectivity it should be okay on the BB scale: I think if it's only going to be occasional connectivity then mysqldump might be a more practical method of keeping the backup updated.

It'll probably mostly work most of the time, but I wouldn't expect it to be particularly easy to set up (and the MySQL versions really should be identical, which may or may not be practical). Both MySQL servers would also need to listen on the network, rather than just localhost, which increases security concerns. Again, mysqldump would be better from that angle (it could be done automatically and fairly securely with a restricted-purpose SSH key)

WTF?

This is getting out of hand. The BB goes down on average once every two years. One day out of 700. Does that justify the amount of time and effort it would take to implement this slave backup system, and the amount of mysql and php knowledge it would need to be able to administer it? The longer this conversation goes on, the more it seems that anyone administering the Oolite community will need a computing science degree...

The original scope, I will say again, was to implement something that would allow the oolite.org site to be updated, simplify the process of adding photos to the gallery and hopefully make administration of the back end systems possible. At most it needs a CMS (of which there are two demonstrated options), a wiki to which access has been granted to the server side hosting (which Winston has now provided) and access to someone's web hosting (which maik and I and others have offered at various levels of "trust"). As far as notification and awareness of downtime, we have a twitter account, a facebook account, two linkedin accounts and a .info site. On top of the three domains that we currently use. Isn't that enough?

The whole purpose of making the admin available to selected members of the community was to ensure that someone would be available to make the necessary changes or updates to the back end if necessary and to resolve problems quickly rather than having to wait on the single point of contact for a system to be available (based on the recent unavailability of Winston). Not to become a sysop for what is turning into the most complex concept this side of Windows 8.

Keep It Simple, Stupid (no offence meant). The more complex this gets then the more difficult it will be to administer and the longer it will take to fix problems if and when they arise.

- If we use a cms for oolite.org then it should be a simple one.
- IMO if we make a dedicated oolite wiki then it should be used as the cms
- leave the bb alone. it's fine. at most, if Giles is OK with it, then someone should have access to the back end (I think Ahruman has this, actually)
- try and consolidate file hosting somewhere.
- ensure that at any point in time, more than one person active on the bb has access to the back end hosting systems in order to fix problems, and that their contact details are known to the community and that they check the BB, facebook, twitter and linked in groups for news.

That's all we need.

As for Linux knowledge, I don't really see how that would be needed. I have no idea how to administer a linux-based computer system (other than installing and using the OS) but I maintain a linux-based web hosting service for my Butterflies ADHD website. i access it all through secure FTP and MySQL Administrator. My own personal hosting is Windows IIS based with almost full PHP support (except clean URL support), both through control panels. I've maintained commercial LAMP and WinIIS systems for the NHS and maintain a WAMP system to host Bugzilla in my current job. I haven't once had to delve into SSH because I carefully choose the systems and applications used to ensure that they are easy to maintain and update both directly and remotely. FTP has never caused any security problems on ANY commercial or private system I've used. Security and safety will be maintained by the logins and passwords.

At the risk of sounding confrontational again, and apologies if that's the case, maik, if you are not happy allowing people access to your web hosting then in my opinion you shouldn't offer it. If you restrict the access and require hoops to be jumped through in order to grant permissions then it will be a possible bottleneck should someone need access quickly. There's no guarantee that your level of involvement with the forum will remain at the supremely attentive level you currently enjoy.

As an example (perhaps of my naivety) and at the risk of repetition, if I were to offer hosting to the community then I would do this with no restrictions. I would release the root password to the system (allowing creation of subdomains, email addresses, user and FTP accounts etc) to Oolite Management and create lower level access passwords as required for administrators to be distributed as necessary. Further, I would ensure that the hosting would be able to be administered if i was incommunicado or something unfortunate happened to me. I think that if someone offers a community use of webspace then that's the only way it can be handled.

Selezen,

Aegidian was asking for ideas to improveway access to servers, not only to oolite.org. I find your style of discussion quite destructive. It more and more sounds like you are taking it personally that my ideas of sharing access to a server are more restrictive than yours. You are wrong in that I should not offer help or resources. If they are not wanted on my terms then nobody has to take them. I made an effort to take your goals for access into account. You only seem to be happy if not only your goals are met but it is also by your way.

You also seem to take complexity personally and become destructive, reiterating that you want things simple and that the original attempt was updating oolite.org. IIRC, there was a BB outage that resulted in message loss. My proposal reduces this risk to a minimum. It is a proposal. I heard your wish for simplicity but unfortunately do not know a simple solution for everything. Maybe you could come forward with a solution of your own instead of aggressively going against mine. Then we can discuss this.

I'm looking forward to your constructive comments. Please do not share further frustration as long as it is only destructive.

cim wrote:

maik wrote:

Good to know. What does it mean when you say "when the primary is doing a lot of writes"? How would you assess this BB in contrast?

I would expect that a single BB would have relatively few insert/update/delete operations on the scale of things, and those that there are should be relatively small - this was for a fairly heavily-used server where it got increasingly far behind the live server while trying to recover its synchronisation. There may have been other bottlenecks involved, possibly, of course.

As long as there's reasonably constant connectivity it should be okay on the BB scale: I think if it's only going to be occasional connectivity then mysqldump might be a more practical method of keeping the backup updated.

It'll probably mostly work most of the time, but I wouldn't expect it to be particularly easy to set up (and the MySQL versions really should be identical, which may or may not be practical). Both MySQL servers would also need to listen on the network, rather than just localhost, which increases security concerns. Again, mysqldump would be better from that angle (it could be done automatically and fairly securely with a restricted-purpose SSH key)

I also think the BB is not a heavy use scenario. Keeping MySQL versions identical is a reasonable requirement. The security aspect can be mitigated by using a SSH tunnel between the two servers.

About using mysqldump: would incremental backups be possible? My concern is that a full backup every day or so is too much given the volume of messages.

maik wrote:

About using mysqldump: would incremental backups be possible? My concern is that a full backup every day or so is too much given the volume of messages.

Not with mysqldump, though with sufficient access to the servers one could copy across binary logs instead. That - like the live synchronisation - is more fragile, though.

On the other hand, it's a pretty fast program, the output gzips well for network transfer or storage, and the BB database can't be much more than a Gb in uncompressed size even now. I do daily backups of larger databases than that with mysqldump, and restoring from the backups only takes 10-15 minutes.

Not suitable for failover - continuous synchronisation is the only thing suited to that, and MySQL does not have great options for it - but sufficient for disaster recovery.

I remember reading something about DB locks with mysqldump. Do you know if that means that during the dump writing is disabled?

mysqldump can be an alternative if it is acceptable to loose the data between the last backup and the time of a failure, and if we want to wait for the BB to be setup from the last dump again. Personally, I don't feel at ease with data loss, the restore time seems acceptable.

For the wiki mysqldump definitely makes sense. Data is changed much less frequently here and even if there were modifications between the last dump and time of failure there is still the RSS feed of recent changes to reconstruct the modifications from.

Question.

It seems to me that the disagreements here stem from, not the presented solutions, but because there is a difference in opinion on what the requirements are.
Would it be an idea to look at what requirements we have for the different sites.

Ultimately, with limited resources and with Oolite being an open source indy project based on volunteers, there will always be single points of failure.
But we can of course minimize them.

An example could be (would read better in a table format though):

Oolite bb
Duplication and mirroring: Not required
Off line backup: Required.
Standby backup site: Not required
Multiple administrators: Required
Superusers: Required (Moderators)
Community access: Requires read/write

Oolite.org
Duplication and mirroring: Required
Off line backup: Required.
Standby backup site: Not required
Multiple administrators: Required
Superusers: Preferable (Update site with latest information)
Community access: Requires read only

There are probably a lot I have not covered here, bit it may well be easier to agree upon this first.
Or am I totally over the top here?

This is a good start, Gimi. I would go for requirements more from a user perspective though, e.g.

• What kind of data loss is acceptable for BB, Wiki, web site? None? One day? One week? One month?
• What kind of restoration time is acceptable? None? One day? One week?

maik wrote:

I'm looking forward to your constructive comments. Please do not share further frustration as long as it is only destructive.

maik, please stop it. You are trying to help by providing stuff. Thank you from all of us. But Selezen is trying to help too and he did not attack you personally, merely queried the solution.

Gimi wrote:

It seems to me that the disagreements here stem from, not the presented solutions, but because there is a difference in opinion on what the requirements are.

A classic cause of project failure: implementing the solution before defining the problem. The team have arguments, loads of scope creep occurs and nobody is happy with the final result.

Gimi wrote:

Would it be an idea to look at what requirements we have for the different sites.

Yes - and once there is agreement on that, only then a solution can be determined.

maik wrote:

This is a good start, Gimi. I would go for requirements more from a user perspective though, e.g.

• What kind of data loss is acceptable for BB, Wiki, web site? None? One day? One week? One month?
• What kind of restoration time is acceptable? None? One day? One week?

If we were to do this, a few polls might be in order, but I would push this by current administrators and owners of sites first. Also, up-time and data loss are different requirements. I can easily accept a certain amount of down time, but would be more reluctant towards accepting data loss.

maik wrote:

This is a good start, Gimi. I would go for requirements more from a user perspective though, e.g.

• What kind of data loss is acceptable for BB, Wiki, web site? None? One day? One week? One month?

I would have thought 24 hours' data loss would be tolerable for the wiki. People can remember what they wrote and re-produce it because it was considered writing that describes something they have produced such as a ship or OXP.

For the forum, re-creating what was posted is almost impossible and any active threads would not continue if any messages were lost. But how much risk of data loss is there? There wasn't any lost last time, except a part-written message that had not been posted.

maik wrote:

[*]What kind of restoration time is acceptable? None? One day? One week?[/list]

Ideally, no more than 12 hours but I don't believe aiming for less than 2 hours is realistic. About 8 people had appeared in the IRC channel querying what the problem was within the first two hours, and a dozen popped in after that with the same questions. When we get to 50 concurrent logged-in users daily, with advertising generating income revenue, then I would suggest those times need to be shorter.