Oolite.org updated media?

[Selezen]

The point is moot, we do not start from scratch but want to carry over what we have.

Not saying we should, just saying that there ARE technologies out there that provide good applications without needing to use databases

Yes, it is more simple. It just seems that gpEasy requires a few more hoops to jump through to get a copy of the current oolite.org design. Something like textpattern does not place restrictions on the design.

Granted, there are a few more hoops in the setup stages, but the ultimate goal of this exercise is ease of maintainability in the long term.

It's not "if I get my way". The current forum runs on phpBB3 which needs a database. I thought one idea was to centralize web site + wiki + forum to have fewer required maintainers (as well as making the website more accessible).

Apologies, that may have read more harshly than it was intended. I honestly don't think that the forum should be moved, as it is the hub of Oolite activity and has been for years. It has also been published in several magazines and articles. Unless, of course, Giles thinks otherwise...

True. The risk of a DB falling over is relatively small though.

I disagree. Every database I have ever used has had "unexplained" outages both small and large. I have lost data (sometimes entire databases) to unforeseen circumstances. And this is both privately and commercially and with different service providers. No ISP will guarantee 100% uptime or 100% reliability because such a thing doesn't exist.

The DB on my site is only accessible from the same machine, not over the internet. The more frequently used web applications are all reasonably secured against SQL injection attacks and malicious file uploading and/or respond to security issues relatively quickly. Add to this that only people from the community can actually upload files. Anyways, I feel relatively good about my server but of course would be happy to learn about holes in its security.

I don't understand that first statement. By definition, if a database is being used by a website then it HAS to be accessible from the internet. How would anyone other than you be able to administer it? Whether or not it's remote access, myPhpAdmin (shudder) or some other MySQL client, the database must be accessible in some way, and thus by definition vulnerable. I couldn't comment on HOW vulnerable, as I'm not a hacker and don't know about the tools of the trade...

Edit to add: go ahead and make database-less copies of oolite.org, the wiki, and the bb. You will have my undying appreciation if you manage this without losing anything and keeping the same design and I will rest my case.

It can be done relatively simply but will take time to do that I don't have alone. That was, I understand, your initial issue: the amount of time and effort it would take to move content. Oolite.org can be moved in an hour, and I can modify certain themes to remove headers and allow the use of the fancy header from the site. The wiki could be ported to pmWiki relatively easily but would again take time to export then re-import and move files across. Again, time that I don't have on my own. The forum I wouldn't want to move, to be honest. As I said, aegidian.org has been the game's home since its inception and I don't think it would be right to move it. My focus for this topic was consolidation of the wiki and main site.

I think the main issue here is we are looking at the situation from different perspectives. I think the best option is taking more time in the migration to make the hosting strategy easier to maintain in the future, whilst you are taking the angle of getting it done quickly. Yes, from that perspective your options are better, but IF something bad happens in the future (missing admins, disaster, hacking, blah blah) then more time and effort could be needed to resolve those problems due to the multi-tier approach, and I worry that there are more entry points for problems.

[maik]

It's not "if I get my way". The current forum runs on phpBB3 which needs a database. I thought one idea was to centralize web site + wiki + forum to have fewer required maintainers (as well as making the website more accessible).

Apologies, that may have read more harshly than it was intended. I honestly don't think that the forum should be moved, as it is the hub of Oolite activity and has been for years. It has also been published in several magazines and articles. Unless, of course, Giles thinks otherwise...

Oops, my bad. For some reason I thought that it is run by Winston as well.

True. The risk of a DB falling over is relatively small though.

I disagree. Every database I have ever used has had "unexplained" outages both small and large. I have lost data (sometimes entire databases) to unforeseen circumstances. And this is both privately and commercially and with different service providers. No ISP will guarantee 100% uptime or 100% reliability because such a thing doesn't exist.

I guess I was lucky so far then. However, we are talking anecdotal evidence which isn't helpful. I agree DBs can fail in unlucky circumstances.

The DB on my site is only accessible from the same machine, not over the internet. The more frequently used web applications are all reasonably secured against SQL injection attacks and malicious file uploading and/or respond to security issues relatively quickly. Add to this that only people from the community can actually upload files. Anyways, I feel relatively good about my server but of course would be happy to learn about holes in its security.

I don't understand that first statement. By definition, if a database is being used by a website then it HAS to be accessible from the internet. How would anyone other than you be able to administer it? Whether or not it's remote access, myPhpAdmin (shudder) or some other MySQL client, the database must be accessible in some way, and thus by definition vulnerable. I couldn't comment on HOW vulnerable, as I'm not a hacker and don't know about the tools of the trade...

*scratches head* The only application that is accessible over the internet is Apache. Apache runs PHP, which accesses the DB. There is no phpMyAdmin or other admin interface and there is no way to connect with a remote MySQL client. You have to ssh into my machine and use the command line client of MySQL to connect to it. Or run a DB dump script if you want to export its data and move it elsewhere.

Edit to add: go ahead and make database-less copies of oolite.org, the wiki, and the bb. You will have my undying appreciation if you manage this without losing anything and keeping the same design and I will rest my case.

It can be done relatively simply but will take time to do that I don't have alone. That was, I understand, your initial issue: the amount of time and effort it would take to move content. Oolite.org can be moved in an hour, and I can modify certain themes to remove headers and allow the use of the fancy header from the site.

Probably true.

The wiki could be ported to pmWiki relatively easily but would again take time to export then re-import and move files across.

This I don't believe. You could probably move the pure content but all formatting would be lost (think templates), all history would be lost, etc. The time you spent with scripts just to make the OXP List visible did not result in anything pretty. I can't see that this is wisely invested time.

I think the main issue here is we are looking at the situation from different perspectives. I think the best option is taking more time in the migration to make the hosting strategy easier to maintain in the future, whilst you are taking the angle of getting it done quickly. Yes, from that perspective your options are better, but IF something bad happens in the future (missing admins, disaster, hacking, blah blah) then more time and effort could be needed to resolve those problems due to the multi-tier approach, and I worry that there are more entry points for problems.

There is that, but also the "don't just talk but do" aspect. We can go in circles until the dawn of time arguing back and forth about the merits of file based vs. database based, but this won't solve anything. If you spend the one hour of effort to get oolite.org into your CMS then I'm happy to host it instead of textpattern. Or you host it. In any case, we'll be one step further.

[Selezen]

*scratches head* The only application that is accessible over the internet is Apache. Apache runs PHP, which accesses the DB. There is no phpMyAdmin or other admin interface and there is no way to connect with a remote MySQL client. You have to ssh into my machine and use the command line client of MySQL to connect to it. Or run a DB dump script if you want to export its data and move it elsewhere.

Web browser contacts server, invoking apache, invoking php, invoking mysql. Any SQL commands can be run through that route (depending on protection against injection et al). Any web page on your server that uses the database connection is a point of vulnerability. Is the MySQL database hosted through an ISP or is it a computer in your house? I seem to read evidence to support both scenarios!! If you are using SSH on a computer in your house, then how are remote users meant to administer it? If you are giving them logins to access the database on the server through SSH then that indicates that the database is administrable remotely. If it can't be, then your setup is the same as Winston's, and if you back out of the community then we'd be in the same boat as we are now.

This I don't believe. You could probably move the pure content but all formatting would be lost (think templates), all history would be lost, etc. The time you spent with scripts just to make the OXP List visible did not result in anything pretty. I can't see that this is wisely invested time.

I didn't spend a lot of time with scripts at all. I exported then imported the content (10 mins) then installed a plugin that allowed the formatting to be updated (5 mins to locate, download, install) then did some manual faffing about to change the formatting to see how hard it was to do it via replace in Notepad++. It was research, not actively carrying out the upgrade, and it took less than an hour to do what I did with it. During that time I found that a regex or similar would easily allow the correction of formatting errors. Yes, the process would be a LOT longer than simply using another copy of MediaWiki and there would be some functional differences, but as I keep saying my goal was to investigate ALTERNATIVES that would streamline the admin processes for the future. That, to me, is worth some effort in the migration phase. History would be lost, but ultimately that isn't much of a loss, since we're happy on the whole with the current content, and there's no need to roll any of it back as far as I'm aware. pmWiki's repository has scripts and mods for templating but again I didn't look into it much. I'll be honest, I was/am prepared to seriously look into it, but I'm not going to waste time with it for no reason, and you came along at that point and did it all yourself using MediaWiki.

There is that, but also the "don't just talk but do" aspect. We can go in circles until the dawn of time arguing back and forth about the merits of file based vs. database based, but this won't solve anything. If you spend the one hour of effort to get oolite.org into your CMS then I'm happy to host it instead of textpattern. Or you host it. In any case, we'll be one step further.

"Don't talk but do" is a concept that ends up causing long-term problems as the solutions haven't been adequately thought through. I've been a commercial software developer for over a decade and through all that time the mantra has been "plan, consult, refactor, execute". Simply barrelling ahead and implementing the first solution that looks good, especially without consultation, is often a recipe for disaster. In this case the risk is that there could be similar problems up ahead related to hosting and admin access using your solution.

As for the "arguing" part of the above paragraph, we obviously have two separate approaches here that would both be workable. Maybe a third party needs to look at the options and voice an opinion that's unbiased... I'm being objective about this, but the one thing I can't get past is that using textpattern introduces a database tier for the website, increasing maintenance overheads and complexity. Other than that fact, textpattern looks awesome and has indeed provided everything that's needed, and kudos to you for getting it up and running so quickly. My free time at work to work on or talk about Oolite stuff has been taken up more with adding to this discussion than being able to do anything constructive about the site, to be honest, or the script would have been ported by now. Again, though, I would rather plan and make sure that the consensus of the community is for the suggestion before taking the time to action.

For the record, in case I sound anti-MediaWiki, I don't have a problem with MediaWiki as such, and honestly I don't really care which wiki technology is being used. My push for non-databased content management was primarily for the website (the original purpose of the conversation). As long as the wiki's front and back ends are EASILY accessible and maintainable by a responsible Oolite Forum Member (with appropriate logins and access) then it's definitely worth moving it rather than porting it to another technology.

[maik]

*scratches head* The only application that is accessible over the internet is Apache. Apache runs PHP, which accesses the DB. There is no phpMyAdmin or other admin interface and there is no way to connect with a remote MySQL client. You have to ssh into my machine and use the command line client of MySQL to connect to it. Or run a DB dump script if you want to export its data and move it elsewhere.

Web browser contacts server, invoking apache, invoking php, invoking mysql. Any SQL commands can be run through that route (depending on protection against injection et al). Any web page on your server that uses the database connection is a point of vulnerability. Is the MySQL database hosted through an ISP or is it a computer in your house? I seem to read evidence to support both scenarios!! If you are using SSH on a computer in your house, then how are remote users meant to administer it? If you are giving them logins to access the database on the server through SSH then that indicates that the database is administrable remotely. If it can't be, then your setup is the same as Winston's, and if you back out of the community then we'd be in the same boat as we are now.

There is one server, being hosted. This one server I have full root access to. I installed Apache with PHP5 and MySQL on this server. As I mentioned earlier, cim has full access to everything that is needed as far as I can tell: Apache, MySQL, and the directory that contains the web applications (textpattern and MediaWiki at the moment). This access is exclusively by ssh. When you ssh into that server, you arrive at a bash shell. From there you invoke the mysql client. So yes, the whole machine can be administered using ssh remotely, but the MySQL client only runs locally. MySQL is configured such that it only accepts connections from localhost, not from anywhere else. You would have to tunnel via ssh if you wanted a client that runs on your PC. Sorry for being a bit verbose, but I feel like we are not understanding each other.

Regarding SQL injection: Only web pages that process user input and access the DB based on this user input are potentially vulnerable to SQL injection. Otherwise you cannot inject anything. On the web pages that we have, only the search forms on MediaWiki and phpBB come to mind which are accessible to people who are not logged in. Every other possibility for user input requires that you are logged in and thus a member of this forum or someone with a wiki account. And since these two are probably the most widely used, it is not very likely that they have an unknown SQL injection vulnerability on their standard search forms.

I've been a commercial software developer for over a decade and through all that time the mantra has been "plan, consult, refactor, execute". Simply barrelling ahead and implementing the first solution that looks good, especially without consultation, is often a recipe for disaster.

This is the recipe to follow if you work in a project environment: someone gives you a goal, a contract, a budget, and a timeframe, and you provide the best possible solution and consult the client for steering the project where needed. Here we don't have a client, no contract, no budget, no timeframe, and no decision structure. Just a set of not very clearly stated and not widely agreed goals (make oolite.org content management easier; centralize web site, wiki, and bb to make server maintenance easier; have more than one person with access in each case). So it is all down to people who eventually find some time to do something. In this environment without clear decision structures you don't arrive anywhere if you don't show what you can do.

Having said this, I would like to stop this discussion here until there is a sound alternative not only on paper. I happy to contribute my thoughts to any plan for a solution or to review any implementation of alternatives that you do.

[Selezen]

MySQL is configured such that it only accepts connections from localhost, not from anywhere else. You would have to tunnel via ssh if you wanted a client that runs on your PC. Sorry for being a bit verbose, but I feel like we are not understanding each other.

Ok, if it only accepts connections from localhost that means that it will accept input from any website being hosted on your server, right? That means that any web application hosted on your server will be able to run SQL commands.

Regarding SQL injection: Only web pages that process user input and access the DB based on this user input are potentially vulnerable to SQL injection. Otherwise you cannot inject anything. On the web pages that we have, only the search forms on MediaWiki and phpBB come to mind which are accessible to people who are not logged in. Every other possibility for user input requires that you are logged in and thus a member of this forum or someone with a wiki account. And since these two are probably the most widely used, it is not very likely that they have an unknown SQL injection vulnerability on their standard search forms.

Unlikely, granted. User creation forms (phpbb only in our case) are vulnerable too and are the usual way that hackers get in. I'm not saying that it WILL happen, or that it's likely to happen. I'm just highlighting ONE reason why databases add another layer of complexity and/or risk to a hosting strategy in relation to using a db-based CMS for the website.

This is the recipe to follow if you work in a project environment: someone gives you a goal, a contract, a budget, and a timeframe, and you provide the best possible solution and consult the client for steering the project where needed. Here we don't have a client, no contract, no budget, no timeframe, and no decision structure. Just a set of not very clearly stated and not widely agreed goals (make oolite.org content management easier; centralize web site, wiki, and bb to make server maintenance easier; have more than one person with access in each case). So it is all down to people who eventually find some time to do something. In this environment without clear decision structures you don't arrive anywhere if you don't show what you can do.

Again, although I agree with your definition of a project I don't agree that what we've been discussing is not a project. It's just a smaller one. It doesn't need reams of documentation or budgeting spreadsheets, but it DOES warrant some thought and input from the client base (the forum users) to futureproof the concept. Leaving talk of what technology to use to one side, is it logical to implement a technology that is more complex than what currently exists? If so, how much more complex would be acceptable?

I've just been trying to make sure that the process for repairing/maintaining the WEB site is easy for ANYONE who wants to get involved and has the time and resources to do it. For textpattern the set of skills needed includes knowledge of ssh, mysql, php and apache. ssh, mysql and apache are all very much Linux-based concepts which immediately dissuades anyone who has a Mac or PC and thus restricts the amount of people on the forum who COULD do it.

Having said this, I would like to stop this discussion here until there is a sound alternative not only on paper. I happy to contribute my thoughts to any plan for a solution or to review any implementation of alternatives that you do.

I don't want to STOP the discussion, but I would like to open it up to the rest of the forum denizens for opinions and maybe options. At the moment it seems very much like Maik vs Selezen from opposite sides of a concept. Ultimately, if the community is happy with things the way they are, then the whole conversation is moot anyway! Mountains out of molehills and all that...

[Cody]

I know zilch about the technicalities or security aspects of all this, so have little useful input. Preservation of the wiki resource seems the most important thing to me. If it was to go down (as has happened) and Winston was not available/interested (as it seems he isn't), it would be a major problem. I like the idea of a distinct Oolite wiki, that the community's chosen few can administer when needed. I'd happily switch to using only the Oolite wiki, but I can see the problems for many users of having two wikis to update etc. As I said, little useful input.

[maik]

I've just been trying to make sure that the process for repairing/maintaining the WEB site is easy for ANYONE who wants to get involved and has the time and resources to do it. For textpattern the set of skills needed includes knowledge of ssh, mysql, php and apache. ssh, mysql and apache are all very much Linux-based concepts which immediately dissuades anyone who has a Mac or PC and thus restricts the amount of people on the forum who COULD do it.

There we come from different angles, too. To me, this sounds like a recipe for disaster. Not the Linux aspect, but the anyone with time and resources aspect. Most people are already overwhelmed by maintaining their own PC. Letting them loose on a web server is something that I would never encourage. And certainly not support if its a web server that I am ultimately responsible for. NB: while I run OS X on my desktop and before that ran Windows for ages, Linux on the server does not dissuade me. But maybe that's just me.

Having said this, I would like to stop this discussion here until there is a sound alternative not only on paper. I happy to contribute my thoughts to any plan for a solution or to review any implementation of alternatives that you do.

I don't want to STOP the discussion, but I would like to open it up to the rest of the forum denizens for opinions and maybe options. At the moment it seems very much like Maik vs Selezen from opposite sides of a concept. Ultimately, if the community is happy with things the way they are, then the whole conversation is moot anyway! Mountains out of molehills and all that...

Prediction: The discussion will continue to go in circles until there are alternatives that you can actually touch and see...

[maik]

Ok, I looked at a few more light-weight CMS that fit snuggly into the existing server setup and seem to not have a very steep learning curve and was most happy with textpattern. Its user community seems very active and friendly (good match for us) and they also produce a lot of plugins. It runs on http://maikschulz.de now and I already copied the oolite.org contents into it.

This was only a rough exercise so far, except for the gallery page I did not add any flexibility with placeholders for articles, so the only advantage over oolite.org at the moment is that you can edit the pages directly in the CMS front end and don't have to have server access.

The gallery looks quite horrible at the moment, but this is due to my limited HTML/CSS knowledge and to textpattern not seeing libGD which would be responsible for automatically creating thumbnails. So thumbnails have to be created manually at the moment until I get this fixed.

If someone would like to play around with it PM me for a user.

Edit: Fixed the thumbnail generation

Fixed the look of the gallery as well. The thumbnails have slightly different heights which looks odd but otherwise it works.

[cim]

Okay - radical suggestion time - does Oolite need a separate wiki and website at all? Yes, obviously now, if the wiki is on alioth.net and not an exclusive Oolite wiki - but if the new Oolite wiki comes in, does there need to be a non-wiki website, or can it just be done through modifications to the Oolite Wiki?

Currently on the Oolite website:
- fancy sparkly stars
- A picture of a Cobra III and an excerpt from Status Quo
- A brief description of Oolite, not far different to that on the Wiki front page
- A few excerpts from reviews of Oolite.
- Some download links that point off to Berlios. Also already on the Wiki, though should be more prominent there as it takes a few clicks from the main page to reach them
- Some links, most of which go to the Wiki anyway (followed by the forum and Berlios)
- A small image gallery, and three links to YouTube videos

A little bit of reorganisation of the Wiki main page, a little bit of extra content, and that could probably all be handled by the wiki. That's one less application to deal with, means we don't need to manage separate CMS and wiki apps and accounts, and uses technology we're already familiar with as a community.

(I'd say heavily tweak the "navigation" box on the wiki as part of this - the standard 'Wikipedia' links are mostly useless for the Oolite wiki or could be moved to the "toolbox" section, and it could better hold things like "download", "manual", "OXP list", and other such key links)

[maik]

Quite radical indeed. You named the advantages. The disadvantage is heavy restriction on design. Oolite's home page is the first thing new users see. It should be designed perfectly in my opinion with little to no restrictions on the designer (other than to stick to HTML/CSS/JavaScript). A wiki should look like a wiki, that's also what users are familiar with. But that's just my opinion.

[cim]

Quite radical indeed. You named the advantages. The disadvantage is heavy restriction on design. Oolite's home page is the first thing new users see. It should be designed perfectly in my opinion with little to no restrictions on the designer (other than to stick to HTML/CSS/JavaScript). A wiki should look like a wiki, that's also what users are familiar with. But that's just my opinion.

True... though I don't know about heavy restrictions, necessarily. I'll see if I can mock something up that both looks nice and works as a general wiki template: there seems to be a fair amount of flexibility available in Mediawiki skins.

(To pick out "Oolite's home page is the first thing new users see.", actually the Oolite page on the alioth Wiki was the first thing I found. Took me a while to figure out where to actually get the game from there )

[maik]

Ok, I looked at a few more light-weight CMS that fit snuggly into the existing server setup and seem to not have a very steep learning curve and was most happy with textpattern. Its user community seems very active and friendly (good match for us) and they also produce a lot of plugins. It runs on http://maikschulz.de now and I already copied the oolite.org contents into it.

This was only a rough exercise so far, except for the gallery page I did not add any flexibility with placeholders for articles, so the only advantage over oolite.org at the moment is that you can edit the pages directly in the CMS front end and don't have to have server access.

The gallery looks quite horrible at the moment, but this is due to my limited HTML/CSS knowledge and to textpattern not seeing libGD which would be responsible for automatically creating thumbnails. So thumbnails have to be created manually at the moment until I get this fixed.

If someone would like to play around with it PM me for a user.

Edit: Fixed the thumbnail generation

Fixed the look of the gallery as well. The thumbnails have slightly different heights which looks odd but otherwise it works.

Finally played a bit more with making parts of the site dynamic: The press blurbs on the About page can now be updated as well as the links and link sections on the Links page. In addition, I made header and footer parts which cover the navigation and berlios contents for all pages respectively.

[cim]

True... though I don't know about heavy restrictions, necessarily. I'll see if I can mock something up that both looks nice and works as a general wiki template: there seems to be a fair amount of flexibility available in Mediawiki skins.

Okay - here's an (unfinished) attempt: http://maikschulz.de/wiki/User:Cim/Altmain combined with the custom stylesheet at http://maikschulz.de/wiki/User:Cim/monobook.css
(I reorganised the web page content to fit all the important things on one page, though again it's just illustrative and I expect people to disagree with my personal assessment of "important")

Results:

Full-size: http://compsoc.dur.ac.uk/~cim/oolite/ss/wikihome.png
Of course the Oolite logo needs to be changed to transparent background, and the background starfield needs a proper screenshot with decent vertical wrapping, and there are other design issues too, but hopefully it shows we could in theory have a nice-looking site and a Wiki at the same time. (I can code HTML/CSS very well, but coming up with nice aesthetics isn't my strength...)

One of the nice features of Mediawiki is that it's straightforward to have page-specific styles, so the home page really could look even more different than that - take out all the wiki furniture entirely for the main page, perhaps. I've taken out the toolbox and editing tabs to demonstrate.

[maik]

Great start, cim!

[cim]

Great start, cim!

Thanks. Okay, so, do we go for the "wiki for the website" idea, then? If home page design was the only downside to it then I think we can work around that through customising the stylesheet.
If so:
- what, if any, wiki furniture remains on the home page? Or do we just expand out the content area? Navigation (with the links modified) and search?. The other bits can probably go: people wanting to edit the home page can set a different skin for editing purposes and we could put most of the home page content in templates elsewhere on the site to make it editable.
- what content do we want on the home page? There's probably space to have a few more links than in the sample (but which links?) A rotation of a few "featured images" showing off Oolite would be easy enough to set up.

Whether we do or not:
- for the content area on the general wiki pages... white text on black or black text on white? I find the usual black on white far more readable for long pages, so I'd say that - but getting it to work with a dark page background is trickier, and I like having an Oolite-y background for the site. And the thing about space is that it's black... Maybe there's some sort of "data pad" aesthetic that might work, though?
- what links should go in Navigation? "Main page" and "Recent Changes" should probably stay, but then what? There's probably room for all or most of the major links in the table on the current wiki main page to go in there, though I'm not sure all of them should, and rearranging the top-level pages a bit might be a good idea.