Added "high availability" to NeoOffice websites

[pluby]

Background

As many of you may already know, for some time now we have been running all of the various NeoOffice websites on three dedicated FreeBSD machines and in the past we have protected our users' data by doing the following:

1. Security. Strictly limiting access to our servers to only us and only other an encrypted connection. Even our webhosting provider cannot access our machines unless they physically walk into the data center and manually reboot the machine. To enhance security is why we added HTTPS support to our websites and to our own mail server: to make sure that passwords and other personal data are sent over an encrypted connection to prevent eavesdropping when using insecure networks.

2. Frequent hot backups. All of our webpage changes are stored in a code repository at a different location than our webservers. Also, we take snapshots of our databases (Trinity, Bugzilla, NeoWiki, EAP, etc.) every 12 hours and copy that snapshot to one of the other servers. By doing these procedures, we will be able to rebuild any of our websites and databases on a different server within a very short period of time.

Issues that needed to be solved

So far, the security protections and backup procedures have served us well. Nevertheless, our three little FreeBSD servers handle a tremendous amount of traffic 24 hours a day, 7 days a week. While we can rebuild a website fairly quickly if one (or maybe even two) of our servers suffer a catastrophic hardware failure, our current procedures leave us with two potentail issues:

1. We could lose up to 12 hours of user data. While we can recover EAP data from PayPal quickly and easily, with Trinity, Bugzilla, and NeoWiki we would have to use the last dataabase backup.

2. One or more of our websites would sites would be inaccessible while we are rebuilding the websites and databases on another server. This isn't catastrophic, but it is highly inconvenient for users who are looking for the answer to a problem and all links point to the website that is on a dead machine. This would be particularly annoying for users of our upcoming NeoOffice Mobile service as they would be not be able to access their data until we got the website and database rebuilt.

Changes we have made

To solve these issues, over the last few weeks we have been adding "high availability" services to our websites and databases. Specifically, we put the following new services in production last night for all of our websites and databases:

1. Database replication - In addition to the database backups every 12 hours, each of our databases now has a matching read-only "replicated" database on another server. The replicated database is updated immediately after every update to the production database. Should there be a catastrophic failure on the production database's machine, we have a complete working backup database already running. At worst, we will only lose changes made immediately before the failure.

2. Website failover - Now that we have a complete working backup database, we have modified our webpages to use the read-only backup database if the production database is down. While you won't be able to update or add anything to Trinity, Bugzilla, or NeoWiki if this happens, the sites will be fully viewable while we rebuild the production website.

3. Webserver failover - If a webserver goes down (or the machine it is on goes down), we have each of our website's webpages staged on the same machine as its matching backup database. The webserver is alwasy running so to activate the failover site, we only need to login to our Network Solutions account and update the DNS information for the server.

Hope that gives everyone a good sense of what Ed and I manage the NeoOffice.org infrastructure.

Patrick

[sardisson]

Wow

That's a serious professional IT staff you have hiding in your back pockets

Smokey