So I generally don't write about my day job, but today we had a somewhat... eventful day in the office. An ESX machine failed, our backup procedures had not been adhered to properly, and for a while it seemed like we'd lost all the data. Miraculously, after the most frustrating and stressful day I've had in recent memory, it all started working again. But that's just not the real story I'm writing about today.
Before the miracle, we started rebuilding one of the machines, where I learnt a very valuable lesson. NEVER EVER make a mistake in your /etc/network/interfaces file in Ubuntu! You WILL NOT be able to boot your server.
"Well, once you know the problem, that's easy to fix! Just use a live CD and fix the file!" you might say. Not so fast. Ubuntu uses an LVM set up for the root partition these days. The only live CD I had on hand didn't have the LVM tools on it. Fortunately, this blog came to the rescue, and I've learnt some important lessons today, even despite the "failed" hard drives suddenly coming good after all.