Last night I topped 2,000 Tweets on Twitter.  That is a lot of microblogging.  Originally I didn’t think that Twitter and microblogging in general would be all that interesting but I have found it to be quite cool.

Through Twitter I have come across the new blog, Trains and Bricks, which is written by Linda Smith and is chronicling her adventures in starting in the model railroading hobby.  Linda, like Dominica and I, is into N Scale model railroads (they are the coolest, as you know.)  Linda received a Kato starter set on Valentine’s Day from her husband and seems quite excited.  She must as she has a blog about it.  Linda is also very much into Legos!  It is neat to have Linda getting into N Scale model railroading at almost exactly the same time that Dominica and I are getting into the hobby as well.

The hobby is not new for me and not quite new for Dominica but, as of yet, Dominica has never had a running train but has done a bit of actual modeling that is extremely advanced for a beginner (foam benchwork, plaster-cloth on newspaper terrain, ballasted flextrack on cork, weathered buildings, etc.)

The weather turned very cold today and there is a solid dusting of snow on the ground that is determined to not be going anywhere.  Oreo took only the shortest of walks today to minimize the time spent outside.

I was not really all that busy today at the office, not for a Friday, but somehow I ended up being quite busy around lunch time and missed taking a break for lunch at all.

The new server got hooked up early this afternoon in Toronto, Ontario, Canada.  We are all extremely excited about this.  We have had so many problems with the network stability and power stability of the datacenter in Scranton that we are really looking forward to seeing what we get out of Toronto.  I am saving a bit of money by going this way as well.

There was a little bit of configuration work that needed to be done to get the email system actually online and working but it took very little and things started working like a charm almost immediately.  Many of the issues that we had seen earlier this week were, as we had guessed, caused by the temporary hosting on a consumer network and vanished as soon as we were on the commercial backbone network.

Dominica did get a chance to do another segment of her class today.  This second course in her four course series has several fewer objectives or segments than the last one did (this one has ten compared to nineteen, I believe) but each one is much more involved and difficult so instead of doing two or three at a time she is pushing had to be able to finish one in the tiny amount of time that she is able to get without Liesl causing too much of an interruption.  Any progress is good progress though.

Not long after lunch Liesl actually fell asleep and Dominica, being exhausted from yesterday, fell asleep as well in the recliner in the living room.  They both managed to get really good, several hour long naps in and did not get up until around seven in the evening when I was wrapping up my work for the office.

For dinner Dominica cooked a “Mama Mia Casserole” which is a baked spaghetti dish.  We watched some Murder She Wrote during which I had to make constant trips to the basement to check in on the office.  I had some amount of work that continued until about half past midnight.  Nothing major but I had to stay in contact.  No rest on a Friday night for me.

I decided tonight, because they were both sitting in just the right place to make this easy, to pop in the two dead DVDs that we have just to be absolutely sure that they were unusable.  The one DVD, which we bought many years ago used from Hollywood Video, was Magnolia. The DVD never once played, to any degree, in any device that we owned.  We’ve kept it more out of principle than anything else.  We bought it so we figured that we should keep it.  So tonight, when I popped it into the HP external DVD drive that I have on my Windows workstation and it read the entire disc without an error I was completely taken by surprise.  We now have a three hour long movie that we did not have yesterday.  Very cool.

I then took the Build Design Dream DVD that Model Railroader sent to us in the mail as an advertisement – that showed up as being blank on every DVD player that I put it into – into the DVD drive on Dominica’s laptop and it too was able to be read without a single error.  It’s completely crazy.

So now we have two more things to watch that we already owned without spending a dime.  Nice.

We did get some time to relax until Liesl dropped off and was happily asleep around ten or so.  Then the real work began – shampooing the living room carpet.  We moved all of the furniture from the south side of the living room (by the deck doors) to the dining room area so that we had roughly half of the room completely clear.

I did a hard-core vacuuming job with the new Kirby and got all of the loose stuff up and off of the carpet.  This will be our first time doing a full shampoo.  We (I) bought the Kirby because of its shampoo capacity but it is such an undertaking, especially with a baby and a dog, to relocate all of the furniture and to have the carpets be wet that we have not done it yet.

It took two full loads of Kirby shampoo liquid to do that portion of the living room.  It took us a while to figure out how to do the shampooing since it was our first time.  We had to spend some time with the manual and only I ever saw someone use it in this way before so it was all new to us.

Overall, I think, the shampooing went pretty smoothly.  It was a little bit of a learning curve but not too bad.  It was almost one in the morning when we wrapped up.

Tomorrow I start work at seven in the morning (less than six hours from when I managed to go to bed) so I am going to be a bit tired.  I have deployments starting and seven and disaster recovery testing starting at eight.

I also need to get up tomorrow and vacuum the carpet that we shampooed tonight after it is completely dry.  Then, once that is done, all of the furniture that is in the dining room area has to be squeezed into the living room area on the south side so that we can shampoo that portion of the carpeting.  Most of that section was not touched during the Kirby demonstration and is still loaded with stains from the previous owners of the house.

I woke up rather on the early side but ended up not actually getting out of bed for a while.  I did my morning weigh in and was up just a smidgen but that was to be expected after yesterday’s huge drop all at once.  Then it was down to work with intermittent trips upstairs to see the family.

We took some pictures with dad and Liesl this morning then dad headed out the door sometime after ten twenty to get onto the road back home to Peoria.  Just after dad left I went back down to work and attempted to sign on to my email and discovered, to my horror, that the email system, as well as the instant messaging system, were down.  Quickly I discovered that the host server that handles both of them had suffered catastrophic drive failure that included one or more of its drives physically being pulled from their connectors.  I don’t know how this happened and I have never seen it in a server before.

I spent the first few hours trying to get the drives reconnected by talking someone through the procedure over the phone which is never fun.  Eventually we had that portion of the problem figured out and got the Smart Array able to recognize all of the drives.  The Smart Array was happy to tell us that everything was “okay” but, of course, it was not really “okay”.

I thought for sure that the RAID array had completely failed and that there was going to be nothing at all.  Things were not quite that bleak.  It turned out that the array survived somewhat intact but that the filesystem on the drives was seriously hosed.  I have no idea what trauma that server went through to cause so much damage but this was really something.  I have never seen so much filesystem damage.

I spent much of the afternoon in a panic attempting to get a rescue disc mounted and booted on the server.  The data center didn’t answer ten of my twelve phone calls and eventually I resorted to email from my Yahoo account.  I am guessing that something bad was happening at the datacenter.  They always stop taking calls when they have broken a bunch of stuff.

Eventually we got a maintenance CD mounted but then spent a few hours trying in vain to get that to work.  It turns out, we believe, that the CD was bad.  It took forever to get another CD mounted but eventually we did and we were able to get to the console and begin working on the box.

The first thing that we learned was that the filesystem really was hosed and that there was almost no chance of salvaging anything.  Now that is depressing.  Pretty much the only thing to do was to run a complete file system check and hope for the best.  The filesystem was in heavy use when the world was yanked out from under it so the damage is potentially pretty significant.

In the process of looking for the status of backups after the server failed I also quickly discovered that around the same time this morning that the email and instant messaging systems died that the backup server – a completely separate machine with its own redundanct drive systems – had also completely and utterly disappeared!  Now this is a bad day in the making.  Key server gone along with the backups.

I ended up doing nothing today but work and work on these servers.  It was exhausting.  Exhausting and depressing.

The filesystem check came back with the most depressing results of the day – everything was gone.  Everything.  Gone.  Nothing left.  Nope.  Nadda.

Around nine o’clock this evening we made the call that there was just nothing that could be done with the lost filesystem and any continued work on it was a wasted effort that could be better spent elsewhere and there was little to no chance of being able to repair the lost SunFire server remotely as it has been restarted and no one at the data center was able to determine anything about its status.  So that left me with nothing to do but to hop into the Mazda PR5 and hit the road for Scranton.

It was just after ten when I actually made it out of the door and onto the road.  I arrived at the Scranton Data Center just a few minutes after midnight.  Luckily the crew was standing around outside smoking so I was able to find everyone that I needed instantly and get right in, derack the two lost servers, load them into the car, swing into Turkey Hill to pick up a pack of cashews (I haven’t eaten since breakfast) and an energy drink and to get back onto the road heading to Peekskill.

I arrived back at the house in Peekskill just minutes after two in the morning.  Four hours for a round trip from Peekskill to Scranton with two servers being deracked is pretty impressive if I do say so myself.  No time wasted anywhere.

My first order of business was restoring the backup server itself.  The handy thing about tonight’s move was that that server was always supposed to be deracked and moved to the house in Peekskill.  Originally we were not planning on making that move until after everything else had left Scranton but this ended up working out reasonably well as it was from that perspective.

I got the backup server working and determined that there was an elusive backup available from September which was our “best care scenario” once we had seen how catastrophic things were.  So getting a copy of that backup was of primary interest although these backups are so large that just moving a copy from one machine to another is rather difficult.

While I was working on getting the backups moved around to places more useful (and to make additional copies of them for safety as they are now the master copies) I got back to work on building the Zimbra server itself that is going to take over for the failed machine.  Again, another multi-hour long process.

The startup of the first run of the Zimbra server ended up taking a very long time as did the file copies.  In the end I resorted to just going to bed and leaving the Zimbra server to come up on its own, completing the first, small file copy and kicking off a massive compression job on the one server in the hopes of reducing the amount of data that has to be moved around.

It was six in the morning when I finally managed to call it a night.  Not nearly as much progress as I was hoping to have made but I think that there was enough done that it is likely that the new server will be completed significantly enough so that the machine can be packed and shipped to Toronto tomorrow afternoon.  Now I just have to get up and set things up with the datacenter in the morning so that they are ready to receive the new box!!

What a day.

My personal experience with Zimbra, this far, is running the 4.5.x series on CentOS 4 (RHEL 4).  Using CentOS instead of actual Red Hat Enterprise Linux presents a few extra issues with the installer but have no fear, the process does work.

While this document is based on the Red Hat Enterprise Linux version of Zimbra, I expect that non-RPM based systems will behave similarly.

To upgrade an existing installation of Zimbra, first do a complete backup. I cannot state the importance of having a complete and completely up-to-date backup of your entire system.  Zimbra is a massive package that is highly complex.  You will want to be absolutely sure that you are backed up and prepared for disaster.  If you use the open source version of Zimbra, as I do, that means taking Zimbra offline so that a backup can be performed.  I won’t go into backup details here but LVM or virtual instances of your server will likely be your best friend for regular backups.  Email systems can get very large very quickly.

Go to the Zimbra website and download the latest package for your platform.  If you use CentOS, get your matching RHEL package.  It will work fine for you.  I find that the easiest way to move the package to your Zimbra server is with wget.  Downloading to /tmp is fine as long as you have enough space.

Unpack your fresh Zimbra package.  Zimbra downloads as a tarball (gzip’ed tar package) but contains little more than a handy installation script that automates RPM deployments.  It is actually a very nice package.

tar -xzvf zimbra-package.tar.gz

You can cd into your newly unpacked directory and inside you will find that there is a script, install.  Yes, the installation process is really that simple.  If you are on most platforms you may simple run the install script.  If you are on CentOS, rather than RHEL, you will need one extra parameter: –platform-override.

./install.sh –platform-override

Be prepared for this process to run for quite some time, by which, I mean easily an hour or more.  Depending on the version of the platform that you are upgrading from and to you may find that this process can run for quite some time.  Also, depending on the size of your mail store, that may impact the speediness of the process as well.

The installation script will fire off checking for currently installed instances of Zimbra, checking your platform for compatibility (be sure to check this manually if using the override option but CentOS users can rest assured that RHEL packages work perfectly for them), performing an integrity check on your database and checking prerequisite packages.  Chances are that you will need to do something in order to prepare your system for the upgrade.

In my case, upgrading from 4.5.9 to 5.0.9, I needed to install the libtool-libs package.

yum install libtool-libs

While there are processes here that can certainly go wrong, the Zimbra upgrade process is very simple and straightforward.  As long as you have good backups (make sure not to start Zimbra and receive new mail after having made you last backup) you should not be afraid to upgrade your Zimbra Open Source system.

You can also purchase a support contract from Yahoo/Zimbra so that you can move to the Network version of Zimbra and Zimbra support staff are happy to walk you through the process.  Having someone there to make sure everything is okay is always nice.

References:

Linux Zimbra Upgrade HowTo from GeekZine

I got some maintenance work done on SGL today. The biggest change is that I am tracking the site using Google Analytics now. So I am quite excited to see how that works. I think that we will find that my traffic profile is a lot better than Word Press Stats suggest. I am also quite interested to see more historical data and geographic data.

Today I finished “reading” “Lost Discoveries : The Ancient Roots of Modern Science–from the Babylonians to the Maya” by Dick Teresi. It is a fascinating book and covers a lot of interesting ground. A great read although, like most things that I enjoy, rather dry.

I began “reading” Simon Winchester’s “Outposts: Journeys to the Surviving Relics of the British Empire” which is an interesting look at the remains of Britain’s once vast empire now reduced to mostly tiny islands scattered to the four corners of the world. Winchester is one of my favourite authors and always has great insight no matter what he is working on. Unfortunately “Outposts” had to be abridged for the audio version and the parts on Hong Kong and the Falkland Islands were left “on the cutting room floor” so to speak. Although Hong Kong is no longer a relic of the empire it was at the time of the original writing.

Today was busy and I was stuck in the office a little late again. But not too bad. I had my now usual falafel lunch from the little Halal truck one block west of here and got through the day.

On the way home I swung into Borders on Broadway and found a book on Ruby that I was looking for: “Practical Ruby for System Administration” by Andre Ben Hamou and APress.  I prefer Ruby to Perl for system administration tasks (and most everything else) and was interested to see what this book might have to offer.  It had a warm reception in some online reviews that I read so I decided that as a full time system administrator who uses Ruby it just seemed appropriate that I should have this book if for no other purposes than knowing whether or not to recommend it to others.

I did some work on my Ruby script for reading in Netgear firewall logs via IMAP and parsing them into a MySQL Database.  My script is working pretty well now.  I updated it so that it now logs to the system event log which is very handy for trouble shooting.  I also set it to run every hour on the hour so that my email mailbox stays clean.  Now I don’t have to worry about manually running it all of the time.

Tonight I started the project of taking all of my old, archived Netgear firewall logs that were downloaded to Thunderbird and saved as an offline folder and put onto my home SAN – my Netgear SC101.  I remounted the offline folder to Thunderbird and began the process of reloading the data onto the email server for processing.  There are scores of thousands of emails to be uploaded.  This is going to be quite a project that will definitely take a few days at the least.  I moved as many as I could tonight before going to bed.

Dominica was in the mood for makizushi sushi and so decided to have some delivered.  I wasn’t very hungry and sushi didn’t really do it for me.  The food was good but I really don’t enjoy sushi all that much.  We don’t get sashimi very often.  It is the seaweed and rice rolls (makizushi) that Dominica really enjoys.  I don’t mind it but it doesn’t get me very excited and I just wasn’t in the mood for it tonight.

We watched a few episodes of The Fresh Prince of Bel Air and tried to call it an early night.  We haven’t been getting enough sleep and are starting to get tired.