I woke up rather on the early side but ended up not actually getting out of bed for a while.  I did my morning weigh in and was up just a smidgen but that was to be expected after yesterday’s huge drop all at once.  Then it was down to work with intermittent trips upstairs to see the family.

We took some pictures with dad and Liesl this morning then dad headed out the door sometime after ten twenty to get onto the road back home to Peoria.  Just after dad left I went back down to work and attempted to sign on to my email and discovered, to my horror, that the email system, as well as the instant messaging system, were down.  Quickly I discovered that the host server that handles both of them had suffered catastrophic drive failure that included one or more of its drives physically being pulled from their connectors.  I don’t know how this happened and I have never seen it in a server before.

I spent the first few hours trying to get the drives reconnected by talking someone through the procedure over the phone which is never fun.  Eventually we had that portion of the problem figured out and got the Smart Array able to recognize all of the drives.  The Smart Array was happy to tell us that everything was “okay” but, of course, it was not really “okay”.

I thought for sure that the RAID array had completely failed and that there was going to be nothing at all.  Things were not quite that bleak.  It turned out that the array survived somewhat intact but that the filesystem on the drives was seriously hosed.  I have no idea what trauma that server went through to cause so much damage but this was really something.  I have never seen so much filesystem damage.

I spent much of the afternoon in a panic attempting to get a rescue disc mounted and booted on the server.  The data center didn’t answer ten of my twelve phone calls and eventually I resorted to email from my Yahoo account.  I am guessing that something bad was happening at the datacenter.  They always stop taking calls when they have broken a bunch of stuff.

Eventually we got a maintenance CD mounted but then spent a few hours trying in vain to get that to work.  It turns out, we believe, that the CD was bad.  It took forever to get another CD mounted but eventually we did and we were able to get to the console and begin working on the box.

The first thing that we learned was that the filesystem really was hosed and that there was almost no chance of salvaging anything.  Now that is depressing.  Pretty much the only thing to do was to run a complete file system check and hope for the best.  The filesystem was in heavy use when the world was yanked out from under it so the damage is potentially pretty significant.

In the process of looking for the status of backups after the server failed I also quickly discovered that around the same time this morning that the email and instant messaging systems died that the backup server – a completely separate machine with its own redundanct drive systems – had also completely and utterly disappeared!  Now this is a bad day in the making.  Key server gone along with the backups.

I ended up doing nothing today but work and work on these servers.  It was exhausting.  Exhausting and depressing.

The filesystem check came back with the most depressing results of the day – everything was gone.  Everything.  Gone.  Nothing left.  Nope.  Nadda.

Around nine o’clock this evening we made the call that there was just nothing that could be done with the lost filesystem and any continued work on it was a wasted effort that could be better spent elsewhere and there was little to no chance of being able to repair the lost SunFire server remotely as it has been restarted and no one at the data center was able to determine anything about its status.  So that left me with nothing to do but to hop into the Mazda PR5 and hit the road for Scranton.

It was just after ten when I actually made it out of the door and onto the road.  I arrived at the Scranton Data Center just a few minutes after midnight.  Luckily the crew was standing around outside smoking so I was able to find everyone that I needed instantly and get right in, derack the two lost servers, load them into the car, swing into Turkey Hill to pick up a pack of cashews (I haven’t eaten since breakfast) and an energy drink and to get back onto the road heading to Peekskill.

I arrived back at the house in Peekskill just minutes after two in the morning.  Four hours for a round trip from Peekskill to Scranton with two servers being deracked is pretty impressive if I do say so myself.  No time wasted anywhere.

My first order of business was restoring the backup server itself.  The handy thing about tonight’s move was that that server was always supposed to be deracked and moved to the house in Peekskill.  Originally we were not planning on making that move until after everything else had left Scranton but this ended up working out reasonably well as it was from that perspective.

I got the backup server working and determined that there was an elusive backup available from September which was our “best care scenario” once we had seen how catastrophic things were.  So getting a copy of that backup was of primary interest although these backups are so large that just moving a copy from one machine to another is rather difficult.

While I was working on getting the backups moved around to places more useful (and to make additional copies of them for safety as they are now the master copies) I got back to work on building the Zimbra server itself that is going to take over for the failed machine.  Again, another multi-hour long process.

The startup of the first run of the Zimbra server ended up taking a very long time as did the file copies.  In the end I resorted to just going to bed and leaving the Zimbra server to come up on its own, completing the first, small file copy and kicking off a massive compression job on the one server in the hopes of reducing the amount of data that has to be moved around.

It was six in the morning when I finally managed to call it a night.  Not nearly as much progress as I was hoping to have made but I think that there was enough done that it is likely that the new server will be completed significantly enough so that the machine can be packed and shipped to Toronto tomorrow afternoon.  Now I just have to get up and set things up with the datacenter in the morning so that they are ready to receive the new box!!

What a day.

My personal experience with Zimbra, this far, is running the 4.5.x series on CentOS 4 (RHEL 4).  Using CentOS instead of actual Red Hat Enterprise Linux presents a few extra issues with the installer but have no fear, the process does work.

While this document is based on the Red Hat Enterprise Linux version of Zimbra, I expect that non-RPM based systems will behave similarly.

To upgrade an existing installation of Zimbra, first do a complete backup. I cannot state the importance of having a complete and completely up-to-date backup of your entire system.  Zimbra is a massive package that is highly complex.  You will want to be absolutely sure that you are backed up and prepared for disaster.  If you use the open source version of Zimbra, as I do, that means taking Zimbra offline so that a backup can be performed.  I won’t go into backup details here but LVM or virtual instances of your server will likely be your best friend for regular backups.  Email systems can get very large very quickly.

Go to the Zimbra website and download the latest package for your platform.  If you use CentOS, get your matching RHEL package.  It will work fine for you.  I find that the easiest way to move the package to your Zimbra server is with wget.  Downloading to /tmp is fine as long as you have enough space.

Unpack your fresh Zimbra package.  Zimbra downloads as a tarball (gzip’ed tar package) but contains little more than a handy installation script that automates RPM deployments.  It is actually a very nice package.

tar -xzvf zimbra-package.tar.gz

You can cd into your newly unpacked directory and inside you will find that there is a script, install.  Yes, the installation process is really that simple.  If you are on most platforms you may simple run the install script.  If you are on CentOS, rather than RHEL, you will need one extra parameter: –platform-override.

./install.sh –platform-override

Be prepared for this process to run for quite some time, by which, I mean easily an hour or more.  Depending on the version of the platform that you are upgrading from and to you may find that this process can run for quite some time.  Also, depending on the size of your mail store, that may impact the speediness of the process as well.

The installation script will fire off checking for currently installed instances of Zimbra, checking your platform for compatibility (be sure to check this manually if using the override option but CentOS users can rest assured that RHEL packages work perfectly for them), performing an integrity check on your database and checking prerequisite packages.  Chances are that you will need to do something in order to prepare your system for the upgrade.

In my case, upgrading from 4.5.9 to 5.0.9, I needed to install the libtool-libs package.

yum install libtool-libs

While there are processes here that can certainly go wrong, the Zimbra upgrade process is very simple and straightforward.  As long as you have good backups (make sure not to start Zimbra and receive new mail after having made you last backup) you should not be afraid to upgrade your Zimbra Open Source system.

You can also purchase a support contract from Yahoo/Zimbra so that you can move to the Network version of Zimbra and Zimbra support staff are happy to walk you through the process.  Having someone there to make sure everything is okay is always nice.

References:

Linux Zimbra Upgrade HowTo from GeekZine

Boy were we all tired this morning.  Dominica got up at six and called into work.  There was no way that she, being over 28 1/2 weeks pregnant, was going to be able to go into work on that little sleep – even having napped in the car a little on the way home.  Oreo was pretty exhausted as well and was quite glad to have some time to sleep in late.  I slept in until close to eight myself.

I got up and logged into the office before eight.  I thought that there would be a huge backlog of work for me to work on but there was hardly any.  I had about five thousand email messages to sort through but that was actually pretty light considering how long I was away.  (Yes, that is the actual number that were in my inbox.)

Dominica slept in until a bit after ten.  She was feeling pretty tired still and considered going back to bed again but gave up and got ready for work instead.  We thought about having Oreo head into daycare but by the time that we thought of doing that it was too late and it was already “nap time” at daycare and new dogs aren’t allowed to enter after that point.

InfoWorld has an early look at Microsoft’s new Internet Explorer 8 browser and notices that it consumes more memory than the entire Windows XP operating system!  In other web browser news, Google announced their own Chrome browser today based on the WebKit web rendering engineer.  Chrome is based on the same base technology as Konqueror and Safari.  If nothing else this will add some weight to the third place browser family.  It will be interesting to see the impact of Chrome on the browser market.

I downloaded Chrome and gave it a try today.  So far I am pretty impressed.  It won’t be replacing Firefox anytime soon for me – primarily because it does not work with Zimbra which is my biggest web application, but for normal browsing it is very nice.

Since Dominica did not have Oreo today we decided that she would stop at CheeseBurger in Paradise near her office and bring home something different for dinner.  It is so seldom that we get any variety.  So she got cheddar BBQ veggie burgers and fried pickles for dinner.

My Process Management class at RIT began yesterday but I did not sign in until today.  As of yesterday, my cousin Sara and I are both attending RIT at the same time.  Pretty weird.  Sara is also a graduate of Monroe Community College where I did my Associates degree.  I ordered my books for RIT this evening.  Thank goodness for Amazon Prime or I wouldn’t have them for a week.  I am hopeful that the book will arrive right away because there is only one left in stock!  Oops.

Most of the evening was just spent relaxing.  We needed it.  The past week has been very tiring.  Tomorrow, being Wednesday, is my early day so I need to get some sleep tonight.  My trip to the Georgian restaurant had to be postponed, though, because tomorrow there is a Lunch & Learn with Red Hat at the office.  There were not enough seats for me to get in, however, so I have to buy my own lunch and eat at my desk while listening to the presentation on the phone.  Not quite the same.