February 12, 2009: And the week continues…

My new Atlas N Scale Genesee & Wyoming switcher, road number 45, arrived today.  I got it on eBay and got a great deal on it.  I collect G&W model railroad items.  Hopefully someday (soon) I will be able to have a small model railroad in the basement on which I can run them.

First thing this morning I discovered that the power had gone out at dad’s house.  They are having fifty-five mile an hour winds today and the temperature has really dropped from yesterday.  The power was out all morning which did nothing to improve my nerves.  Instead of being able to spend the morning dealing with getting the email server back up and running again I spent the morning worrying that the three servers that went down with the power outage (we exhausted the available battery backed capacity.)

Work was very, very slow today.  I was very surprised how slow it was.  I am quite relived that I did not go in to Warren today as I would have been more upset about the downed servers.  I had done a manual backup just a few hours before the systems had gone down but with as badly as everything has been going this past week I was not well prepared for their to be any additional problems of this magnitude.

The power came back on in the early afternoon and I was able to begin checking things out.  It turns out that one server, a virtualized machine, completely disappeared.  Luckily it was a machine that I had just built and had not begun using yet so nothing was lost except for a few hours of initial setup.  Not a big deal.  All of the other machines and virtual machines were still okay.

I got an additional surprise this afternoon when the bank called and informed us that the escrow account was set up improperly and that it was not going to cover the school taxes which are already due.  So now we are forced to come up with the school taxes without any warning at all (they are already slightly past due when we found out that they even existed) even after we have been paying into the escrow for several months now.  I thought that the purpose of the escrow was to protect against tax surprises rather than cause them!  Instead of buffering against tax issues the escrow actually created problems this time by causing us to “pay the taxes” each month and then have to pay the actual tax, in full, out of pocket even after paying into the escrow account.  So we have now paid this tax twice.  Nothing like making things easy.

So we paid the taxes and that was that.  At least we were able to pay the taxes.  It could have been quite bad.  It did nothing to make me more calm after a very long and hard week, though.

Dominica put in an hour or more this morning working on getting our medical insurance dealt with.  We have been having all kinds of problems and this morning our payment for Dominica and Liesl’s hospital stay was denied.  The whole problem has arisen because our insurance was switched on us after having gone to the hospital.  Then we got in trouble for not pre-authorizing the trip to the hospital even though we were not put on that particular insurance until after the visit was over.  The whole process is insane.  We also found out that since Dominica’s old job keeps switching the insurance plan that we are on (it is not us deciding these things, they keep getting forced upon us) that we have to pay the deductable amount three times!  So we are not nearly as insured as it would appear that we are.

I came upstairs for dinner and took over watching Liesl for a while.  Liesl has been in a crazy mood today.  I wonder if it might be a result of all of the stress that we have been under this week.  Maybe she senses it.  She has avoided napping today almost totally.  She slept less than two hours all day today.  She almost dropped off many times but would always leap back to being completely awake.  She also took several bottles today.  She just kept eating and eating.

Liesl finally seemed to have fallen asleep at almost ten thirty tonight while Dominica and I were watched the first season ofGrace & Favour which is more well known in America as Are You Being Served? Again.

We ended up staying up pretty late tonight.  It was after one in the morning when we finally headed off to bed to get some sleep.  Chris and I have been trying to get an FTP transfer working between us which has been a bit of a pain.  We finally got something working tonight and I kicked off a transfer of the email data so that he can work on the email restore portion while I am working on the email primary portion.  The estimated transfer time for the FTP transfer of the highly compressed backup file is a measly twenty-four hours.  Ha ha.  Somehow I imagine the connection getting severed at some point during that transfer process.  That is a pretty long time for two computers to remain connected over the Internet without any major problems.  I think that we will likely have to attempt this transfer multiple times before it actually does any good.

February 11, 2009: Starting to Recover

Not much sleep for me last night.  Got to bed a bit after six and did not fall asleep all that quickly.  Then a somewhat restless night and up again at nine thirty and straight down to the office to work.  I did weigh in this morning and I have lost 3.3lbs since yesterday morning.  Stress is a great diet.

My sinuses are killing me today.  I was in a hurry to get to bed last night and did not refill my CPAP with water so it ran mostly dry which always causes problems.  I should know better but when you are that tired you are often not thinking about the water levels in your nighttime breathing apparatus.

Today was crazy as I spent the day attempting to build a new Zimbra server.  Building Zimbra is never as easy as one would hope.  It is an exceptionally involved process.  There are always a load of errors and tons of things that you have to work around.  It took most of the day but by the late afternoon we appeared to have a pretty smoothly functioning email server – no email being sent or received yet.  That won’t be for a few days at a minimum.  At least progress is being made.

I also got most of the instant messaging system rebuilt today.  I got a start on it, anyway.  It has been a very busy day.

This evening I finally got to a point where I was able to fire up Zimbra and connect to the admin interface and start reconstructing things.  I spent the rest of the evening putting information back into the email system.  That took several hours.

After an exhausting day we went to bed around midnight.  Sorry that the updates are so lean but I have been awfully busy today.  Keep in mind anyone trying to email me that you need to reach me via my Yahoo or GMail accounts since my usual is gone.

I was originally going to drive in to Warren tomorrow for drinks but that was cancelled and/or postponed until next week.  So I will not be driving anywhere tomorrow.

February 10, 2009: Disaster Day

I didn’t realize it when I woke up this morning but today was destined to be one of those horribly long and awful days that you just can’t imagine how stressful they can be until they happen.

I woke up rather on the early side but ended up not actually getting out of bed for a while.  I did my morning weigh in and was up just a smidgen but that was to be expected after yesterday’s huge drop all at once.  Then it was down to work with intermittent trips upstairs to see the family.

We took some pictures with dad and Liesl this morning then dad headed out the door sometime after ten twenty to get onto the road back home to Peoria.  Just after dad left I went back down to work and attempted to sign on to my email and discovered, to my horror, that the email system, as well as the instant messaging system, were down.  Quickly I discovered that the host server that handles both of them had suffered catastrophic drive failure that included one or more of its drives physically being pulled from their connectors.  I don’t know how this happened and I have never seen it in a server before.

I spent the first few hours trying to get the drives reconnected by talking someone through the procedure over the phone which is never fun.  Eventually we had that portion of the problem figured out and got the Smart Array able to recognize all of the drives.  The Smart Array was happy to tell us that everything was “okay” but, of course, it was not really “okay”.

I thought for sure that the RAID array had completely failed and that there was going to be nothing at all.  Things were not quite that bleak.  It turned out that the array survived somewhat intact but that the filesystem on the drives was seriously hosed.  I have no idea what trauma that server went through to cause so much damage but this was really something.  I have never seen so much filesystem damage.

I spent much of the afternoon in a panic attempting to get a rescue disc mounted and booted on the server.  The data center didn’t answer ten of my twelve phone calls and eventually I resorted to email from my Yahoo account.  I am guessing that something bad was happening at the datacenter.  They always stop taking calls when they have broken a bunch of stuff.

Eventually we got a maintenance CD mounted but then spent a few hours trying in vain to get that to work.  It turns out, we believe, that the CD was bad.  It took forever to get another CD mounted but eventually we did and we were able to get to the console and begin working on the box.

The first thing that we learned was that the filesystem really was hosed and that there was almost no chance of salvaging anything.  Now that is depressing.  Pretty much the only thing to do was to run a complete file system check and hope for the best.  The filesystem was in heavy use when the world was yanked out from under it so the damage is potentially pretty significant.

In the process of looking for the status of backups after the server failed I also quickly discovered that around the same time this morning that the email and instant messaging systems died that the backup server – a completely separate machine with its own redundanct drive systems – had also completely and utterly disappeared!  Now this is a bad day in the making.  Key server gone along with the backups.

I ended up doing nothing today but work and work on these servers.  It was exhausting.  Exhausting and depressing.

The filesystem check came back with the most depressing results of the day – everything was gone.  Everything.  Gone.  Nothing left.  Nope.  Nadda.

Around nine o’clock this evening we made the call that there was just nothing that could be done with the lost filesystem and any continued work on it was a wasted effort that could be better spent elsewhere and there was little to no chance of being able to repair the lost SunFire server remotely as it has been restarted and no one at the data center was able to determine anything about its status.  So that left me with nothing to do but to hop into the Mazda PR5 and hit the road for Scranton.

It was just after ten when I actually made it out of the door and onto the road.  I arrived at the Scranton Data Center just a few minutes after midnight.  Luckily the crew was standing around outside smoking so I was able to find everyone that I needed instantly and get right in, derack the two lost servers, load them into the car, swing into Turkey Hill to pick up a pack of cashews (I haven’t eaten since breakfast) and an energy drink and to get back onto the road heading to Peekskill.

I arrived back at the house in Peekskill just minutes after two in the morning.  Four hours for a round trip from Peekskill to Scranton with two servers being deracked is pretty impressive if I do say so myself.  No time wasted anywhere.

My first order of business was restoring the backup server itself.  The handy thing about tonight’s move was that that server was always supposed to be deracked and moved to the house in Peekskill.  Originally we were not planning on making that move until after everything else had left Scranton but this ended up working out reasonably well as it was from that perspective.

I got the backup server working and determined that there was an elusive backup available from September which was our “best care scenario” once we had seen how catastrophic things were.  So getting a copy of that backup was of primary interest although these backups are so large that just moving a copy from one machine to another is rather difficult.

While I was working on getting the backups moved around to places more useful (and to make additional copies of them for safety as they are now the master copies) I got back to work on building the Zimbra server itself that is going to take over for the failed machine.  Again, another multi-hour long process.

The startup of the first run of the Zimbra server ended up taking a very long time as did the file copies.  In the end I resorted to just going to bed and leaving the Zimbra server to come up on its own, completing the first, small file copy and kicking off a massive compression job on the one server in the hopes of reducing the amount of data that has to be moved around.

It was six in the morning when I finally managed to call it a night.  Not nearly as much progress as I was hoping to have made but I think that there was enough done that it is likely that the new server will be completed significantly enough so that the machine can be packed and shipped to Toronto tomorrow afternoon.  Now I just have to get up and set things up with the datacenter in the morning so that they are ready to receive the new box!!

What a day.

February 9, 2009: Dad Gets to Stay Another Day

I worked from home today so I was able to emerge from the basement periodically throughout the day to visit with dad, Dominica, Liesl and Oreo although, for the most part, I was just stuck hiding down in the basement all by myself.  It can be a lonely existance sometimes.

Dad had been thinking that he was going to have to head back home a day early and leave today rather than tomorrow because of the weather but when he checked the weather forecast this morning things were looking pretty good so he decided to visit with Liesl and the Millers for an extra day and to go back home tomorrow as he had originally planned.

I did not get very much time to visit with dad today, but I think that it was a good day for him to get a lot of time just hanging out with Liesl and it gave him a chance to see a fairly normal Liesl day in action.  It was good that dad was able to stay today.  All of that driving for just Saturday evening and Sunday during the day is not very worth it.

After work was over this evening we spent most of the night just visiting but then, just before going off to bed, we watched the first few episodes (except for the pilot which is not available via NetFlix) of The Vicar of Dibley which dad has never seen.  It is strange that Netflix is missing the first episode of the first season.  It is the episode that explains the show and introduces the characters.  I wouldn’t want to skip it.

A lot of the things that we watch on Hulu, we have noticed, are missing really popular episodes pulled out of the middle of seasons.  It is as if they wanted to make sure that the online show experience was not going to replace the DVD experience for collectors but only to satisfy the need to sample shows and to watch casual reruns.

February 8, 2009: The LCD Is Mounted

I got paged out during the night but it was nothing that I really had to respond to.

I found a great picture of an early Genesee & Wyoming steam engine.  I have never seen a steam engine on the GNWR so seeing a picture of one is a big odd.

Liesl got me up this morning.  Dominica was very tired so I got up with Liesl and walked around with her for a while.  We even went downstairs to the living room and visited with dad for a little bit before Liesl decided that she was hungry.  So then I woke up Dominica who fed Liesl and then they both fell back asleep.

I ended up needing to work on and off throughout the day today.  We had some issues at the office and I kept getting called about them.  So I spent a few hours in the basement office working.

Dad and I unloaded the car today.  Three boxes of books went down to the basement which I immediately unpacked and put onto the shelves.  Dad also brought our three drawer “filing cabinet” unit that matches our office desks which will be very nice as it will provide additional, out-of-the-way storage which we desperately need.  Once the rest of Dominica’s desks arrive it will be hidden underneath them.  For now it is just sitting by her desk.

Dad also brough our cold air humidifier (the plastic cow) and our good coffee maker that we got as a wedding present and which I have really been missing.  Dominica likes the one cup Keurig coffee style but I much prefer the grind and brew style.  Dominica likes flavoured coffees, but I just prefer really high quality normal coffee.  Dad also brought our good leather “moon chair” that matches Dominica good reclining swivel chair that came some time ago.  We are not sure where we will put it, though.

Dad was also able to bring, at the last minute, about twenty of our DVDs from our collection that we have been storing at his house.  This will give me additional material to convert to h.264.  He is starting to make random grabs from the collection whenever he comes down and then I covert them and send them back on the next trip.  In this way we always have something “new” to watch and the converted collection continues to grow and slowly, very slowly, the collection will get converted.

I found a 1.5TB external USB Seagate hard drive on Amazon today that is pretty inexpensive that I think I will get to hook to Dominica’s laptop.  That will give it the storage that it needs to be able to really just keep working at the conversions all of the time.  Disk space and I/O are our biggest limitations currently.

Dad and I attempted to mount the new Samsung LCD in the living room but even before removing the old Westinghouse LCD from the wall dad had figured out that the mouting bracket that we currently are using is too small for the Samsung’s custom mounting configuration.  So once Liesl fell asleep dad and Dominica ran out to go shopping at Best Buy to see if they would be able to find a suitable mounting bracket.  I was unable to go out shopping because of work.  So I stayed home taking care of Liesl and Oreo.

I am trying to figure out how I can put a model railroad into our small townhouse.  There really is no space for one.  I am thinking that putting one into the attic might be a doable solution although it would be such a pain to go see, use or work upon not to mention how difficult it would be for anyone else to ever see it.  I am considering the possibility of having one next to my desk in the basement, kind of behind me when I work.  It would be very small but I might be able to do as much as five feet in length and as deep as two feet or maybe even a little more.  Small, but not overly ambitious.  I don’t want something overwhelming.  Just something on which I can work when the mood strikes me.  Having it so close to my desk would make it very easy to connect to the computer which is how I would want to run it anyway.

Dominica finally figured out today where we had put the wireless dongle and memory card for the Nintendo GameCube.  They were in the Wii’s travel case which goes in the hall closet.  There was an extra pocket in there about which I had forgotten and it had just never occurred to us to look in that case which, of couse, is where they should have been.  The case is so so seldom that it was an out of sight, out of mind problem.

As some of you know, I collect Genesee & Wyoming Railroad Model Train pariphenalia.  Today I was bidding on eBay on an Atlas Genesee & Wyoming MP-15DC RN 45 in N scale.  I do not have one of the MP15DC switchers so I was anxious to win one and getting one in great used condition is a lot cheaper than getting one new.  They are about $90.  This particular switcher is, in real life, one that I see often either in Retsof or down near the new salt mine.  You can often see it moving cars around when you pass by on i390, especially if you are north bound.  I ended up winning the engine for just $36.

It took a lot of doing but by eight this evening the Samsung LCD was mounted on the wall above the fireplace.  Mounting the 70lbs behemoth was no simple task.  It looks great, though, and makes watching shows and playing video games in the living room much more enjoyable.  It also cleans up the space in front of the fireplace and makes the living room feel much larger again.

At nine or so we settled down and watched The Dark Knight, the second Batman movie in the new series with Christian Bale who is by far the best Batman ever, on BluRay.  It was an awesome movie.  Christian Bale has been one of Hollywood’s brightest stars since Empire of the Sun many, many years ago and he never fails to impress.  I was not as impressed with Heath Ledger’s performance which, I feel, everyone only raves about because he died before the film released which always makes people appreciate the performance more than they should.  He didn’t do a bad job and admittedly the character of the Joker is a ridiculous character and there is very little to do well with it   The replacement of pathetic Katie Holmes as Rachel Dawes with Maggie Gyllenhaal was an excellent choice and really served to fix cast shortcomings between the films.

After the movie was over it was time for bed.  Oreo needed to be walked twice as he sometimes does.  Dad is thinking about heading home tomorrow as the weather is possibly going to be pretty bad on Tuesday with the potential for freezing rain.  He is going to play it by ear and will make a decision tomorrow morning.