The lateness of this post is likely a little misleading. We finally got the server problems sorted out last week, and I took a 5 day mini-vacation in Montreal (actually 4 days, and I kind of screwed around on Tuesday). Apparently my MySQL-foo is pretty spot-on, and I was able to restore 10 jigabytes of data to to a backup server from our backup server. We're now rebuilding the failed DB server, which looks more and more as if its LSI Logic RAID card simply reset at some point, causing a massive RAID failure. Luckily, we do nightly backups.

However, the version of mysqldump had a bug in it, whereby it exported the system database, information_schema, along with our own db. This system database can't be imported, so the whole restore process kept failing. The only solution was to edit the db backup (which was, as I said, 10 jigabytes, making it impossible to open in a conventional editor), so we used a line editor to remove the first x lines of the backup. This worked, but it then took about 8-10 hours to re-import the data and only after upgrading the MySQL server and other required packages.

So, the server was offline for a total of 3 business days, and two of those days were spent trying to recover the server without destroying the data onboard. Turns out our time was better spent finding a new (if temporary) home for the data.

Montreal was fun.

Mailing List Signup

What? There's a mailing list? Sign up for the Mailing List

About Me

I suppose if you've been reading my site for any length of time, you're probably curious to know who I am, and why I think people will read my blog. Online, I'm known as Zuckervati, mostly because it's easy...
» More ...

Follow Me on Twitter

Recent Entries

Tag Cloud

politik / film / video / skeptic / cewl / techno / humour / haxors / nosh / can-con / gaming / religion / weird / sex / funny / eco / music / stupid / photos / cocktails / George Bush / blogosphere / flickr / travel / evolution / politics / creationism / creationist / mobile / awful / photography / Star Wars / awesome / geek / bartending / coffee / drinking / mixology / alcohol / liquor / bartender / cocktail recipe / cats / food / Savoy Cocktail Book / science / Lovecraft / parody / books / Cthulhu / articles / TV / pets / John McCain / promos / Movable Type / cartoons / sci-fi / Canada / Radio Zuckervati / cool / Roger Ebert / system / atheism / comics / reviews / technology / zombies / anime / anonymous / MMO / Star Trek / television / writing / Alinea / animation / Batman / Futurama / Halloween / horror

Twitter Stream

D H McKee's bookshelf: to-read

Sunset and Sawdust
tagged: to-read
The Thicket
tagged: to-read
tagged: to-read