February 11, 2009: Starting to Recover

Not much sleep for me last night.  Got to bed a bit after six and did not fall asleep all that quickly.  Then a somewhat restless night and up again at nine thirty and straight down to the office to work.  I did weigh in this morning and I have lost 3.3lbs since yesterday morning.  Stress is a great diet.

My sinuses are killing me today.  I was in a hurry to get to bed last night and did not refill my CPAP with water so it ran mostly dry which always causes problems.  I should know better but when you are that tired you are often not thinking about the water levels in your nighttime breathing apparatus.

Today was crazy as I spent the day attempting to build a new Zimbra server.  Building Zimbra is never as easy as one would hope.  It is an exceptionally involved process.  There are always a load of errors and tons of things that you have to work around.  It took most of the day but by the late afternoon we appeared to have a pretty smoothly functioning email server – no email being sent or received yet.  That won’t be for a few days at a minimum.  At least progress is being made.

I also got most of the instant messaging system rebuilt today.  I got a start on it, anyway.  It has been a very busy day.

This evening I finally got to a point where I was able to fire up Zimbra and connect to the admin interface and start reconstructing things.  I spent the rest of the evening putting information back into the email system.  That took several hours.

After an exhausting day we went to bed around midnight.  Sorry that the updates are so lean but I have been awfully busy today.  Keep in mind anyone trying to email me that you need to reach me via my Yahoo or GMail accounts since my usual is gone.

I was originally going to drive in to Warren tomorrow for drinks but that was cancelled and/or postponed until next week.  So I will not be driving anywhere tomorrow.

February 10, 2009: Disaster Day

I didn’t realize it when I woke up this morning but today was destined to be one of those horribly long and awful days that you just can’t imagine how stressful they can be until they happen.

I woke up rather on the early side but ended up not actually getting out of bed for a while.  I did my morning weigh in and was up just a smidgen but that was to be expected after yesterday’s huge drop all at once.  Then it was down to work with intermittent trips upstairs to see the family.

We took some pictures with dad and Liesl this morning then dad headed out the door sometime after ten twenty to get onto the road back home to Peoria.  Just after dad left I went back down to work and attempted to sign on to my email and discovered, to my horror, that the email system, as well as the instant messaging system, were down.  Quickly I discovered that the host server that handles both of them had suffered catastrophic drive failure that included one or more of its drives physically being pulled from their connectors.  I don’t know how this happened and I have never seen it in a server before.

I spent the first few hours trying to get the drives reconnected by talking someone through the procedure over the phone which is never fun.  Eventually we had that portion of the problem figured out and got the Smart Array able to recognize all of the drives.  The Smart Array was happy to tell us that everything was “okay” but, of course, it was not really “okay”.

I thought for sure that the RAID array had completely failed and that there was going to be nothing at all.  Things were not quite that bleak.  It turned out that the array survived somewhat intact but that the filesystem on the drives was seriously hosed.  I have no idea what trauma that server went through to cause so much damage but this was really something.  I have never seen so much filesystem damage.

I spent much of the afternoon in a panic attempting to get a rescue disc mounted and booted on the server.  The data center didn’t answer ten of my twelve phone calls and eventually I resorted to email from my Yahoo account.  I am guessing that something bad was happening at the datacenter.  They always stop taking calls when they have broken a bunch of stuff.

Eventually we got a maintenance CD mounted but then spent a few hours trying in vain to get that to work.  It turns out, we believe, that the CD was bad.  It took forever to get another CD mounted but eventually we did and we were able to get to the console and begin working on the box.

The first thing that we learned was that the filesystem really was hosed and that there was almost no chance of salvaging anything.  Now that is depressing.  Pretty much the only thing to do was to run a complete file system check and hope for the best.  The filesystem was in heavy use when the world was yanked out from under it so the damage is potentially pretty significant.

In the process of looking for the status of backups after the server failed I also quickly discovered that around the same time this morning that the email and instant messaging systems died that the backup server – a completely separate machine with its own redundanct drive systems – had also completely and utterly disappeared!  Now this is a bad day in the making.  Key server gone along with the backups.

I ended up doing nothing today but work and work on these servers.  It was exhausting.  Exhausting and depressing.

The filesystem check came back with the most depressing results of the day – everything was gone.  Everything.  Gone.  Nothing left.  Nope.  Nadda.

Around nine o’clock this evening we made the call that there was just nothing that could be done with the lost filesystem and any continued work on it was a wasted effort that could be better spent elsewhere and there was little to no chance of being able to repair the lost SunFire server remotely as it has been restarted and no one at the data center was able to determine anything about its status.  So that left me with nothing to do but to hop into the Mazda PR5 and hit the road for Scranton.

It was just after ten when I actually made it out of the door and onto the road.  I arrived at the Scranton Data Center just a few minutes after midnight.  Luckily the crew was standing around outside smoking so I was able to find everyone that I needed instantly and get right in, derack the two lost servers, load them into the car, swing into Turkey Hill to pick up a pack of cashews (I haven’t eaten since breakfast) and an energy drink and to get back onto the road heading to Peekskill.

I arrived back at the house in Peekskill just minutes after two in the morning.  Four hours for a round trip from Peekskill to Scranton with two servers being deracked is pretty impressive if I do say so myself.  No time wasted anywhere.

My first order of business was restoring the backup server itself.  The handy thing about tonight’s move was that that server was always supposed to be deracked and moved to the house in Peekskill.  Originally we were not planning on making that move until after everything else had left Scranton but this ended up working out reasonably well as it was from that perspective.

I got the backup server working and determined that there was an elusive backup available from September which was our “best care scenario” once we had seen how catastrophic things were.  So getting a copy of that backup was of primary interest although these backups are so large that just moving a copy from one machine to another is rather difficult.

While I was working on getting the backups moved around to places more useful (and to make additional copies of them for safety as they are now the master copies) I got back to work on building the Zimbra server itself that is going to take over for the failed machine.  Again, another multi-hour long process.

The startup of the first run of the Zimbra server ended up taking a very long time as did the file copies.  In the end I resorted to just going to bed and leaving the Zimbra server to come up on its own, completing the first, small file copy and kicking off a massive compression job on the one server in the hopes of reducing the amount of data that has to be moved around.

It was six in the morning when I finally managed to call it a night.  Not nearly as much progress as I was hoping to have made but I think that there was enough done that it is likely that the new server will be completed significantly enough so that the machine can be packed and shipped to Toronto tomorrow afternoon.  Now I just have to get up and set things up with the datacenter in the morning so that they are ready to receive the new box!!

What a day.

February 9, 2009: Dad Gets to Stay Another Day

I worked from home today so I was able to emerge from the basement periodically throughout the day to visit with dad, Dominica, Liesl and Oreo although, for the most part, I was just stuck hiding down in the basement all by myself.  It can be a lonely existance sometimes.

Dad had been thinking that he was going to have to head back home a day early and leave today rather than tomorrow because of the weather but when he checked the weather forecast this morning things were looking pretty good so he decided to visit with Liesl and the Millers for an extra day and to go back home tomorrow as he had originally planned.

I did not get very much time to visit with dad today, but I think that it was a good day for him to get a lot of time just hanging out with Liesl and it gave him a chance to see a fairly normal Liesl day in action.  It was good that dad was able to stay today.  All of that driving for just Saturday evening and Sunday during the day is not very worth it.

After work was over this evening we spent most of the night just visiting but then, just before going off to bed, we watched the first few episodes (except for the pilot which is not available via NetFlix) of The Vicar of Dibley which dad has never seen.  It is strange that Netflix is missing the first episode of the first season.  It is the episode that explains the show and introduces the characters.  I wouldn’t want to skip it.

A lot of the things that we watch on Hulu, we have noticed, are missing really popular episodes pulled out of the middle of seasons.  It is as if they wanted to make sure that the online show experience was not going to replace the DVD experience for collectors but only to satisfy the need to sample shows and to watch casual reruns.

February 8, 2009: The LCD Is Mounted

I got paged out during the night but it was nothing that I really had to respond to.

I found a great picture of an early Genesee & Wyoming steam engine.  I have never seen a steam engine on the GNWR so seeing a picture of one is a big odd.

Liesl got me up this morning.  Dominica was very tired so I got up with Liesl and walked around with her for a while.  We even went downstairs to the living room and visited with dad for a little bit before Liesl decided that she was hungry.  So then I woke up Dominica who fed Liesl and then they both fell back asleep.

I ended up needing to work on and off throughout the day today.  We had some issues at the office and I kept getting called about them.  So I spent a few hours in the basement office working.

Dad and I unloaded the car today.  Three boxes of books went down to the basement which I immediately unpacked and put onto the shelves.  Dad also brought our three drawer “filing cabinet” unit that matches our office desks which will be very nice as it will provide additional, out-of-the-way storage which we desperately need.  Once the rest of Dominica’s desks arrive it will be hidden underneath them.  For now it is just sitting by her desk.

Dad also brough our cold air humidifier (the plastic cow) and our good coffee maker that we got as a wedding present and which I have really been missing.  Dominica likes the one cup Keurig coffee style but I much prefer the grind and brew style.  Dominica likes flavoured coffees, but I just prefer really high quality normal coffee.  Dad also brought our good leather “moon chair” that matches Dominica good reclining swivel chair that came some time ago.  We are not sure where we will put it, though.

Dad was also able to bring, at the last minute, about twenty of our DVDs from our collection that we have been storing at his house.  This will give me additional material to convert to h.264.  He is starting to make random grabs from the collection whenever he comes down and then I covert them and send them back on the next trip.  In this way we always have something “new” to watch and the converted collection continues to grow and slowly, very slowly, the collection will get converted.

I found a 1.5TB external USB Seagate hard drive on Amazon today that is pretty inexpensive that I think I will get to hook to Dominica’s laptop.  That will give it the storage that it needs to be able to really just keep working at the conversions all of the time.  Disk space and I/O are our biggest limitations currently.

Dad and I attempted to mount the new Samsung LCD in the living room but even before removing the old Westinghouse LCD from the wall dad had figured out that the mouting bracket that we currently are using is too small for the Samsung’s custom mounting configuration.  So once Liesl fell asleep dad and Dominica ran out to go shopping at Best Buy to see if they would be able to find a suitable mounting bracket.  I was unable to go out shopping because of work.  So I stayed home taking care of Liesl and Oreo.

I am trying to figure out how I can put a model railroad into our small townhouse.  There really is no space for one.  I am thinking that putting one into the attic might be a doable solution although it would be such a pain to go see, use or work upon not to mention how difficult it would be for anyone else to ever see it.  I am considering the possibility of having one next to my desk in the basement, kind of behind me when I work.  It would be very small but I might be able to do as much as five feet in length and as deep as two feet or maybe even a little more.  Small, but not overly ambitious.  I don’t want something overwhelming.  Just something on which I can work when the mood strikes me.  Having it so close to my desk would make it very easy to connect to the computer which is how I would want to run it anyway.

Dominica finally figured out today where we had put the wireless dongle and memory card for the Nintendo GameCube.  They were in the Wii’s travel case which goes in the hall closet.  There was an extra pocket in there about which I had forgotten and it had just never occurred to us to look in that case which, of couse, is where they should have been.  The case is so so seldom that it was an out of sight, out of mind problem.

As some of you know, I collect Genesee & Wyoming Railroad Model Train pariphenalia.  Today I was bidding on eBay on an Atlas Genesee & Wyoming MP-15DC RN 45 in N scale.  I do not have one of the MP15DC switchers so I was anxious to win one and getting one in great used condition is a lot cheaper than getting one new.  They are about $90.  This particular switcher is, in real life, one that I see often either in Retsof or down near the new salt mine.  You can often see it moving cars around when you pass by on i390, especially if you are north bound.  I ended up winning the engine for just $36.

It took a lot of doing but by eight this evening the Samsung LCD was mounted on the wall above the fireplace.  Mounting the 70lbs behemoth was no simple task.  It looks great, though, and makes watching shows and playing video games in the living room much more enjoyable.  It also cleans up the space in front of the fireplace and makes the living room feel much larger again.

At nine or so we settled down and watched The Dark Knight, the second Batman movie in the new series with Christian Bale who is by far the best Batman ever, on BluRay.  It was an awesome movie.  Christian Bale has been one of Hollywood’s brightest stars since Empire of the Sun many, many years ago and he never fails to impress.  I was not as impressed with Heath Ledger’s performance which, I feel, everyone only raves about because he died before the film released which always makes people appreciate the performance more than they should.  He didn’t do a bad job and admittedly the character of the Joker is a ridiculous character and there is very little to do well with it   The replacement of pathetic Katie Holmes as Rachel Dawes with Maggie Gyllenhaal was an excellent choice and really served to fix cast shortcomings between the films.

After the movie was over it was time for bed.  Oreo needed to be walked twice as he sometimes does.  Dad is thinking about heading home tomorrow as the weather is possibly going to be pretty bad on Tuesday with the potential for freezing rain.  He is going to play it by ear and will make a decision tomorrow morning.

February 7, 2009: Dad Drives to Peeksill

I was extremely exhausted when I pulled myself out of bed at seven thirty this morning.  Oreo made me take him for another walk after I signed off and had started heading to bed last night which always makes it hard to fall asleep.

I am very glad, though, that Oreo made me walk him so late at night.  Because it was so late we took a shorter than usual path that lead us down below and in front of the Mazda parked in our parking lot.  As we strolled by I was low enough that I was able to see that the interior light of the car was on and had been on, I would assume, for about twenty-eight hours!  Oops.  I have no idea how or why it was turned on but it had to have been me when I drove home from Warren on Thursday night.

So, to be safe, I took Oreo inside then came back out with the keys and fired up the car and let it run for a little while.  Fortunately the battery was not dead at all and the car started right up.  I took the opportunity to take out the trash since I was stuck waiting for the car to run for a while anyway.

I started work this morning at eight as I normally do on Saturdays.  The upside to always having work scheduled so early is that I never, ever sleep away my weekends.  I get a nice, long weekend and often my work is done around the time that the rest of the family gets out of bed so I don’t miss any time with them.

Dad was able to leave Peoria just before nine to get on the road.  His first stop is McDonald’s for breakfast and then on to Peekskill.  He is hoping to arrive around three this afternoon.

Work for the office went pretty well this morning.  My scheduled work wrapped up around ten.  Then there was my unscheduled maintenance work to do.

House preparations went well this morning.  Some redding up was done and the kitchen was cleaned by ten.

I managed to get my database server set up finally.  That took some work and, of course, each time that I do it I forget that there is a problem with SELinux and MySQL when you move the default database location so I had to fix that too.  Seems to be working well now.

I kicked off a ton of Handbrake compression jobs this morning.  I am attempting to move as much stuff to h.264 as possible before dad leaves so that he can take the source media and put it into storage and also to clean off Dominica’s hard drive which has been holding a lot of stuff in preparation for compression for quite a while.  He laptop doesn’t really have spare drive space for things like that.

We are planning to get a large server for network storage sometime soon but, in the meantime, I think that we need to have a temporary solution as there is so much stuff that we need to store and move around right now.  No one believes the amount of capacity that we need on a regular basis.  We can eat through terabyte disks like they are nothing.

Dad made some good progress on the road today.  He called from just 110 miles away before it was even one in the afternoon.  Around one o’clock Liesl fell asleep and Dominica took the opportunity to run out to Stop and Shop to stock up on groceries.  Our cupboards are pretty bare.

While Dominica was out shopping I watched over the kids and continued to do some cleaning and redding.  I got a lot of good, manual carpet scrubbing done that I have been wanting to do.  The carpet in the dining room area right as you enter the house gets so much traffic and in the winter that traffic is always dirty and salty both from shoes and from Oreo’s feet plus it gets all of the dirt that sticks to your socks when you are on the kitchen linoleum.  There is no avoiding the linoleum because there is a patch that separates the dining room carpeting from the stairs going upstairs.  So you are stuck dragging the dirt onto the carpeting no matter what you do.  I also got a second round of dishes cleaned.  Lots of dish cleaning done today.

Dominica got back just a little bit before dad arrived.  Dad managed to get to Peekskill before three in the afternoon.  His drive went pretty well.  Good weather with a little cloud cover so that it was not too bright.

We spent most of the afternoon visiting and catching up.  It has been over a month since Dad has seen little Liesl and they had some catching up of their own to do.  I had lingering work that I was not able to complete until the early evening.

We had most of the evening free and we able to just hang out in the living room.  We saved the unloading of dad’s car for tomorrow.  We were all very tired tonight.