August 24, 2008: Long Awful Day

89 Days to Baby Day! (27 Weeks and Two Days Pregnant)

Amazingly I actually got some sleep last night.  It was approaching two in the morning before I had a chance to actually lay down to get some sleep and it was probably getting close to two thirty before I actually fell asleep.  Once I did fall asleep I got some pretty good rest, I think.

I slept in until nine or nine thirty this morning and Dominica slept in a little bit longer than that.  Once I was up the long day began.  And what a day it would be.

I checked in just in case some miracle happened during the night and the server had come back up on its own, but that was not to be.  I sent out some email and did some other light work while waiting for Dominica to get up.  We got packed right away, had a quick breakfast with the Toccos and were on the road pretty early.  This is possibly the earliest that we have ever left Frankfort after going there for the weekend.

The first order of business was to drive to Scranton to the data center.  We ended up taking a seriously back roads route, similar to the route suggested by Google Maps, as suggested by our Garmin GPS unit – “Bossy Bettie”.  The route went some crazy places that even Dominica has never been having grown up right there.  The route was gorgeous and the weather was perfect.  The drive itself was awesome.  What a beautiful day for exploring central New York.  The route actually got us through to Scranton very quickly as well.  The route that we took involved using CR 18 on which I have never been before today.

We did so much driving today that my left arm was completely sunburnt by the end of the day!

We arrived in Scranton at the data center at two in the afternoon.  We would have been there half an hour earlier but we got trapped in two construction zones with traffic coming to a standstill in both.  Interstate 81 is unavoidable going in and out of Scranton, but it is just so awful.  The “construction” that they have going on there has traffic completely able to move anytime of the day or night.

I worked at the data center for about an hour and a half.  Things did not go nearly as well as I had hoped. I did manage to verify my suspicions from last night that the ethernet cable on one of my servers had been pulled out by the data center tech while he was working on another machine.  That was the first thing that I was able to fix.

It took one of the data center guys and I almost half an hour to get the hard drives out of the server that had died.  It turned out that the chassis of the server was smashed in on the one corner and the hard drive sled (also known as a caddy) was trapped inside the chassis.  It took a lot of work to get it to release and the server will never be rack mountable again after what we did to it.

My hope had been that by placing the drives into another machine of the same type (another HP DL360 G1) that they it would just fire up and everything would be fine.  We tried that first.  Nothing.  The machine would fire up and never do anything after its POST process.

The next step was to take the drives with me and to return to Newark where I had another spare server and the resources to actually work on the system.  The drive took another two hours and we arrived at Eleven80 in Newark at approximately six in the evening.  By this point I was already pretty tired and feeling quite run down.

I set up the server that we had here and popped in the drives.  Nope.  This time the reaction was slightly different, though.  This time I was able to just barely see the Windows 2003 splash screen on the console before the console would suddenly go dark and the system would restart.  After doing some research I am pretty sure that the issue is that the system is missing the RAID array driver for the Compaq Smart Array 5i that this unit has that the old one did not.

Unfortunately, adding a driver for a missing mass storage device, especially one needed for the system to book, is not a simple process.  The first thing that I needed was access to the recovery console and that was especially hard as the CD that had that was at dad’s house.  So we set about attempting to get a working copy of that disc so that at least some testing could begin.

It took three hours to download the CD ISO image, not including the time that it took to find the initial CD, make an ISO image of it, compress it and then to burn a good copy of it after the transfer was complete.  It was a quarter until midnight when I finally had the CD ready for its very first trial to see if we even had a good CD let alone a solution or even were going down the right path.

Dominica went into the bedroom with Oreo around eight to relax and attempt to go to bed.  She did manage to relax but did not fall asleep as she was quite hungry.  She made herself some dinner at nine thirty and then went to bed and actually fell asleep shortly thereafter.  I skipped dinner myself as eating very much food would surely contribute to my drowsiness and that was not really a viable option tonight.

While I was waiting for the CD image to download I took the opportunity to perform a complete shutdown, update and backup of my instant messaging server.  I also did a WildFire 3.2 to OpenFire 3.5 migration.  (OpenFire is the new name for WildFire from Ignite RealTime, the former makers of Spark, but the name change made the update process more complicated.)  I had been putting this update work off for quite a long time and was glad to have it finally done.  It went very smoothly.  No issues whatsoever.  Pheww.

I did some system updates to my email server as well but didn’t do a full blown email system update.  I was tempted to do so, but my energies were running low after having updated OpenFire and having done everything else.

It was midnight by the time that I finally got the server to boot to the recovery console.  That, in and of itself, was a major achievement today.  We are far closer than we have been since the beginning.

After much testing and trying I decided that the only course of action, and Microsoft’s Knowledge Base seemed to agree with me on this, was to perform an in-place upgrade.  So I kicked that off at a quarter after one in the morning.  What a night.

Since it is now technically Monday morning, I am going to go ahead and post the update for today.  I’m still awake and still have no idea what this night is going to bring.

I decided that a full backup of the email system had to be done tonight one way or another so at one thirty I shut that down and kicked off the backup.  This seems to be the slowest part of the night and the least likely to really impact anyone.

Leave a comment