I was thinking that today was going to be a nice, relaxing day other than the drive home tonight.  No such luck.  I logged into the office just to make sure that everything was okay since it is a Sunday and very few people work on Sunday so things can easily get missed if no one checks in and I discovered that we had quite a few machines down.  Oh boy.

So I started working at noon.  It took a bit of testing before we realized that the commonality between all of the machines that were not working were that they all ran SAN and that they were all involved in receiving one particular patch from the engineering department that updated the kernel.  So, from that, we were able to rather that the chances were that, like they often do, the engineers failed to take into consideration the SAN software stack and had broken it again.  This is a never ending problem that we had with them.

The work ended up just going for hours and hours.  First we had to do a ton of troubleshooting.  That took a few hours because it was not very apparently what had happened.  By the end of the day there were five us working on the problem.

I worked on that issue until after six (yes, for over six hours) and then, after just a few minutes break, was called out to deal with another issue for another team.  I ended up being “at work” from noon until seven.  What an exhausting day.  Things were not even completely done when I left and some work just had to be saved until tomorrow.  Dominica and I really could not wait any longer to get onto the road.  Our plan had been to leave Frankfort between three and four in the afternoon and we completely missed that window.

So it was around seven thirty in the evening when we finally got into the car and drove out of Frankfort.  We are taking the opportunity to go to Scranton and pick up two servers while we are making this trip.  That meant that we got to take the really nice drive south out of Frankfort and skip the highway pretty much altogether.  I love the southerly drive out of Frankfort.  That part of New York is so gorgeous.  It really reminds me of rural England.

We got down to Scranton around a quarter after ten.  Dominica waited in the car with Liesl and Oreo while I went into the datacenter and helped as we unracked two servers, a DL145 G2 and a DL360 G1.  The DL360 G1 is old and going to be going to Castile Christian Academy.  The DL145 G2 is going to be repurposed as a dedicated backup server running Red Hat Linux and the Amanda backup server.

It was a really good thing that we went to Scranton today, even though it was a bit of a pain to do one more thing today.  One of the two machines that is staying in Scranton for a few more weeks had a failed hard drive that I was able to swap out while I was there.  Only took a minute to fix but would have been quite the disaster later on.

Our plan is to be completely done with Scranton in just a few weeks.  Then I will make one last special trip up there to pick up what remains and that will be that.  I will be very happy when that is all over and done.

It takes about two hours to get back to Peekskill from Scranton.  Less than, but almost, two hours.  We had to make one doggy pit stop along the way.  He tried to be fast.

So it was just around one in the morning when we rolled into Peekskill.  We were very glad to be home.  Dominica tried to go right to bed but Liesl’s schedule was all messed up and she was having problems falling asleep.  I went down and checked on the DVD conversion processes running in the basement.  They were still running and had a lot of work left to do.  I guess that I did a good job loading them up before we left.  My main desktop will still be working for another few days before it completes the work that I left for it!

I did some calculations and my desktop would need to run continuously for about four years in order to convert everything that we have!  Four years of non-stop work.  And that is if I was able to keep it running all of that time and able to keep it completely loaded up with prepped work.  This is no small project!  I can’t wait until we have some local server capacity to help offload some of this work.

We tried going to bed at two but Liesl woke right up and was not about to go back to sleep.  Dominica was really frustrated and unhappy so I got up and stayed up with Liesl until almost three when she finally fell asleep.  This weekend has been way too exhausting for me.

It appears that the Arizona Game and Fish Department has conspired to exterminate the last of the American Jaguar population from the country.

The long day continues.  The in-place Windows Server 2003 Web Edition installation on the Compaq Proliant DL360 G2 continued until one forty in the morning.

It was just a little after one thirty when it occurred to me that a lot of the work that I was doing here in the apartment in Newark could be done remotely.  I have become accustomed to the equipment in Scranton not having the benefit of remote optical media but this server does (it has a full ILO rather than the older RILOE card that we had to add to the earlier generation servers.)  And since the installation media that I am using was generated from an ISO image already – we could have just used that remotely to do everything that we are doing right now.  It is not like I am doing anything with a local keyboard, mouse or monitor – even sitting right next to the server I am doing everything through the ILO!  Argh.

I hate it when I get so tired and so close to a problem that I can no longer think of the obvious.  Like yesterday at the data center – I had Andy do a ton of “remote hands” work on a server that I was standing directly in front of because I had no means to access it.  But I had forgotten about the laptop that was right out in the car!  Argh, again.

This was my first even in-place Windows Server 2003 installation.  I never did that on Windows 2000 Server either.  I have not done it, unless I am forgetting something, since my Windows NT 4 days!

It was truly amazing to see the Windows 2003 splash screen come up finally on the remote console of the server.  What a day, what a day.  Still a lot more work left to be done.  But at least that bit is done.  The first huddle has been cleared.

Around two in the morning I had the “new” machine online.  I ran in to tell Dominica (who was very fast asleep as was Oreo) that it was working and that I would shortly be out the door for Scranton.  The valets, knowing that I might need to run out at any minute, had the BMW ready for me on the side of the building.  So I emailed the datacenter to let them know about when I would arrive and I was out the door with my server (DL360 G2), rails, two cans of Monster Juiced energy drinks, my BlackBerry, GPS and my iPod.

Dominica woke up when I was leaving so she helped me get ready to head out the door.  We discussed the issue of me making it back in time for her to leave for work.  It was going to be close.  It is roughly two hours, each direction, to and from Scranton from Newark.  Since it was already a quarter after two, that meant that best case scenario was that I would be back at roughly a quarter after six.  She has to leave for work at seven.  That is a pretty tight window considering that I had to cover two hundred and twenty miles, a good portion of which would be done during early rush hour, as well as get gas and stop at the datacenter, rack and test the server!

I made great time driving out to Scranton.  It was just after four when I arrived.  There was no traffic whatsoever and the construction zones didn’t affect me in the least.  The drive was very nice, all things considered.

Once at the datacenter I got straight in and first did a quick memory swap so that the new server would have more memory than the machine that it was replacing rather than less.  Then into the rack it went.  The racking went very smoothly indeed.  No issues at all.  Since I was pulling two other servers out of the rack there were plenty of cables available for me to use.

The server fired right up without any problems at all.  I plugged in a KVM cable just so that I could verify that the machine had come up on the monitor that they had there in the row.  Once I saw the Windows Server 2003 splash screen come up I knew that it was time to go.

I pulled two servers (HP DL360 G1) from the rack and loaded them into the trunk and headed back for home.  I ran across the street to the Turkey Hill gas station and topped up the tank so that neither I nor Dominica would have to worry about it today.  I text messaged Dominica from the gas station so that she would know that the car would be available for her to go to work and then I was off for the trip home.

The drive back didn’t go quite as quickly as the drive westward but it was not bad.  I hit heavy traffic around Tannersville, Pennsylvania and ended up driving all of the way from there to Newark, New Jersey is light, early morning Manhattan rush hour traffic.

I arrived at Eleven80 at six twenty in the morning.  Boy was I ever tired.  Dominica was not quite up yet when I came in the door.  She had been hoping, I think, that I would not make it back in time and that she would be stuck calling in to work, but I was back with plenty of time to spare so she was able to get to work.

I did some wrap up and verification work while she was getting ready.  I ran a little late, but only a little, to the office today.  The walk to the train and from the WTC to the end of Wall Street were good for me as they helped to keep me awake.

My morning was not too busy.  The United Kingdom and the Kingdom of Bahrain both have a holdiay so the people who generally produce the most work for me are all out today.

For lunch, which is my first proper meal since eating dinner at the Franklin Hotel in Rome on Saturday evening, Katie and I decided to hit the Waterstone Grill between Pearl and Stone.  We are always on the prowl for new “foodie” places to eat and especially ones with outdoor seating.  It is supposed to rain this afternoon and I am looking to sit out in the rain and enjoy some serious seafood.

Lunch was truly amazing.  This was the best meal, in my opinion, that Katie and I have discovered yet and I definitely beat her in picking the best menu item today.  I went for the pan crispy striped bass which was truly amazing.  Glorious, in fact.  Just what I needed.  Katie had the lemongrass encrusted Chilean sea bass which was also good but not truly comparable to the pan crispy striped bass.  I can’t recommend it enough.

I also had a lobster and onion tart as well as a bowl of lobster bisque with a shot of sherry.  Keep in mind that this was my first real meal since Saturday night!  All of the food was really excellent.

The afternoon was not as busy, which was very good because by the time that I had food I was getting incredily sleepy.  I kind of just drifted along through the afternoon until five thirty when I headed for Newark.

I arrived at Eleven80 and Ramona, who was meeting us for dinner, we though, around seven, was already there and waiting for me in the lobby.  Dominica had run into traffic because of an accident on NJ21 and was running late.

We just ordered in dinner from Ninos’ since it was Monday and places like 27Mix were likely to be closed.  We ate and I had to head off to bed around nine o’clock.  I just could stay up and visit any longer.  Oreo was very excited to be able to come to bed early with me and was very snuggly.  I have no idea how long Dominica and Ramona stayed up visiting.

It was an extra short night with a pretty big interruption to walk Oreo in the middle of me trying to sleep.  The one thing that worked out well is that because I was up in the middle of the night with Oreo I was able to get the luggage cart so that it was in the hallway and very easy to load up first thing this morning.

We packed the car as best as we could. We have a decent load of stuff going up to dad’s place this time. This is our first trip back there since Christmas if you can believe that! We packed up all of the CDs and DVDs in the apartment, two humidifiers, the space heater and more. It made a huge dent in our clutter here (and will be barely noticeable in dad’s already packaged garage.)

It’s a pain having to pack the car in a panic in the morning though. It’s times like this that I really wish that I had a garage.  What a difference that would make.

Dominica dropped me off at the Broad Street Train station on her way to take Oreo to daycare. There I caught the train to Summit and the shuttle to Warren like I normally do when working out in New Jersey.

Today is an early market close day so a lot of my usually late night Friday work was able to be done in the middle of the afternoon which was really nice. I was in good shape when Dominica arrived to pick me up a little after six. She had used the GPS to get her from Totowa to Wallington to Warren which worked really well as it routed her around some stuff and took her a different route than she would have taken normally and it saved a bit of time.

It was around six thirty when we left Warren. Traffic wasn’t bad in New Jersey and we only hit a small delay right at the Delaware Water Gap when people were going through the toll booths there.

Normally we avoid the i81 corridor through eastern Pennsylvania as PennDOT has completely destroyed the entire corridor with poor labeling and extended single lane zones around the Scranton Metro which cause massive delays but today it couldn’t be helped. We are delivering two computers up to Apalachin, New York and the i81 corridor is the only easy way from New Jersey to the Binghamton area.

We let the GPS guide us on a faster route through New Jersey. It took us off of i78 and up NJ206 to i80. Not a route that we have taken in a very long time. We stopped along the way to eat at Panera Bread and to pick up some bagels for the morning.

The trip through Scranton ended up adding two to three additional hours to the trip in a construction zone that has been a problem for almost nine months now. It is completely ridiculous. It would have been better to drive back to Newark from Warren then go up to the Hudson Valley and catch NY17 west to Binghamton. Tons of extra miles but it would have saved an hour or more. We will be completely and utterly avoiding eastern Pennsylvania from now on. You just can’t trust PennDOT to do reasonable traffic patterns or warnings. They also don’t provide traffic updates over FM radio for the GPS units so everyone gets stuck in the same traffic jams that could easily have been avoided.

Instead of arriving at a little after nine at John Stephen’s apartment like we had planned it was well after midnight! We were pretty tired and unhappy by the time that we made it through all of that construction traffic. Our long night was turning into a really, really long night.

We only stayed at John’s for a tiny bit. Just enough time to unload the car, load up a computer that he had for me and get a quick tour of his new place. He has only been there for a month or two. We got to meet John’s girlfriend. They are hoping to come down to New Jersey to visit us around mid-July. This is the last summer for folks looking to hang out with us while we live so close to Manhattan.

The rest of the trip from John’s place to dad’s went pretty smoothly and quickly. It was awfully late by the time that we pulled into dad’s driveway. We were quite tired. But at least we were able to deliver the computers tonight and they are not hanging over our heads until Monday evening. That would not be fun. We are going to want to be able to drive right back to Newark on Monday afternoon.