Maintenance and you: A primer

With the relaunch of the site, and the first software upgrades already lined up to go in (likely on Sunday morning), I figured it was time to write up a quick note on how maintenance downtime will go.

Basic Info

• Maintenance downtime will only happen for significant upgrades to the server backends, or for most upgrades to the software the site itself runs. Minor upgrades will just happen in the background when I have the time.
  • There may rarely be short-notice maintenance for things like security fixes or downtime in case of any actual problems. If this were to occur, there will often be about 10-15 minutes notice.
• Regular maintenance will happen as much on 'off hours' as I can make it (weekend mornings, for instance), and will last for as short a time as I can make it (almost always less than 30 minutes, probably less than 15; this does exclude major software upgrades, as those may take a few hours).
  • Generally, the longer the planned maintenance time, the more notice will be given beforehand.

Maintenance Procedure

1. I'll post a short note to the front page, usually about 12 hours in advance of a 'normal' maintenance period. This may be extended to a few days prior in the case of longer expected downtime.
2. 15 minutes before maintenance time, a warning will be posted prominently on all pages. When you see this warning, please do not start doing anything new. Finish up what you're in the middle of, but don't start writing a new post, for example, as the site may go down before you can finish.
3. The site will be placed into 'maintenance mode' (site disabled, and tells you it's undergoing maintenance), and, dependent on what upgrades are being done, may be disabled entirely for the duration (site disabled, pages don't load or load with 500 or 503 errors).
4. Magic behind the scenes happens.
5. Site will come back up, usually within 15-30 minutes unless previously noted.

FAQ-ish

1. Why is maintenance necessary?
   • The server runs a set of software that's updated on what's known as a rolling release schedule. This means smaller upgrades more often, rather than huge upgrades all at once. This is both good and bad -- personally, I prefer it as it generally leaves less chance of severely screwing up anything (or worse, screwing up more than one thing at a time).
2. Site XYZ doesn't have maintenance downtime, why do you?
   • This is pretty simple -- the technical side here is pretty much a one-man show, and I'm self-trained (slowly over about 10 years, so I'm not too bad at it). I take down the site for because I don't have a good way to test major upgrades to the backends of the server while the site's still live, and these are the sort of tests that are needed to make sure things keep running smoothly.
3. How often will maintenance happen?
   • Barring short-notice or emergency maintenance for a specific purpose, no more often than once a week -- probably more like once every 2-3 weeks. I will be trying to find out a good time to do this regularly, as I settle into the routine. (Perhaps Saturday or Sunday morning.)
4. What if something goes wrong?
   • If something happens that goes wrong, don't worry. If it's a problem I can't solve within a short period of time, I'll often just revert things to how they were before the issue occurred and investigate the problem over the next few days, before scheduling another maintenance period. I always make a quick backup of any data I could be affecting by doing an upgrade in case something happens to the live data, and in an absolute worst-case scenario I would need to restore from a backup made the previous evening (meaning the worst data loss possible is about 24 hours worth -- but this is extremely unlikely).

Above all, keep in mind that this (a) keeps the site running well, (b) is less downtime than WoW has, and (c) won't happen too often or for too long.