Sheep Guarding Llama

Medifast Status: Day 120, Down ~51.5lbs

I am a little behind on the news but just found out that GameStop, the bigger US video game retailer, has been opening products that they sell as “new” and taking stuff out of the packaging – and doing so officially.  Talk about completely illegal and unethical.  And they were doing it to customers who were pre-ordering new games!  Not just stuff off of the shelf that people could inspect before purchasing.  This is like allowing someone to custom order a new car then letting the employees of the car dealer drive it on their vacation for a week before letting the customer take delivery of it – even though they already bought and paid for it as new!  In the past I have always supported GameStop because I like to support my local video game store even though they tend to cost more than Amazon and are less convenient to use.  But no more.  I immediately threw out my GameStop card and will never shop there again.  I don’t need to deal with people like that.

Work was very slow this morning.  At nine John and I drove over to La Cima to escape for a while and to enjoy some good coffee.  We just missed breakfast which we need to try to hit sometime.  I always forget about getting free breakfast over there and since I am paying for it already we should really take advantage of it.  A guest is included in the breakfast deal so John can get breakfast there for free.  I can’t eat breakfast in any meaningful way on the MediFast plan so I’m pretty much out of luck.  But at least the coffee is good.

Today was my first time getting to see the club since the new reception desk and waterfall were installed.  They look way better than they did before.  I also noticed that two of the most annoying televisions have been removed as well.  So now there are a total of four televisions in the lounge area rather than six.  Far from good but much better than it was before.

Dominica and the girls came down and picked me up for lunch so I was at the club twice today.  It was a tomato and olive sauce on tilapia which was quite good.  Liesl loved the new waterfall and spent a bit of time just standing in front of it watching it up close.  Dominica thought that the removal of two of the televisions helped the look of the place a bit too.

It was a slow afternoon after getting back from lunch.  This week is particularly slow, people think, because of the weather in New York affecting the traders.

I got home and managed to actually have a pretty good evening with the family.  I got a lot of time in with the girls.  Liesl was in a particularly good mood tonight and never got into any trouble, which is amazing as her attitude lately has not been the best.  She was a little angel tonight.

Luciana also had a good day.  We’ve been trying to add rice cereal to her bottles but she gets really frustrated that she cannot then get the formula out of the bottle.  So Dominica tried actually spoon feeding her and she actually loved it.  She opens her mouth and loves eating food.  Totally unlike Liesl who hated to eat and ended up going directly from bottle to solid food and even now gets around half of her total nutrition via Pedisure because she prefers to drink her food than to eat it.  I suppose that she gets that from me.  That is why the Medifast works out for me – I prefer convenience to pretty much everything else.

So this is really awesome.  Luciana has been so fussy for days.  We think that this will fix it.  And she will likely get back to sleeping again.  She has barely slept for a week.

Medifast Status: Day 119, Down ~51.5lbs

I had to get up at five this morning so that I could get ready quickly and get into the office.  Because of the storm in New York I am on the early, early shift today – starting at six – so that I can help with any issues that may have arisen over the weekend.  This morning is also Dominica’s first day of trying out running in her new “C25K” or Couch to 5K program.  So she got up at five as well and while I was showering and getting ready for work she went out jogging around the neighbourhood.

Dominica’s run went okay but she ended up being chased by a pit bull and had to come back home a different way which caused her to be much later than she had meant to be so instead of getting back at five forty so that I could jump in the car and jet off to the office it was actually six when she got back to the house so that I could leave.  So I was already late to work.  Fortunately it only took me eleven minutes to get to the office as there was no traffic whatsoever.

Like many sites I noticed this morning, SpiceWorks is down.  It is days like today, with much of the northeast without power, that we see how many companies rely on single sites to keep their web and email online without power redundancy, regional failover, etc.  One datacenter goes down and tons of companies go offline completely.  The upside of SpiceWorks being offline is that it is that much easier for me to catch up with everything else now.  Normally that sucks up tons of my time.

Things were pretty quiet in the office this morning.  I thought that there would be a lot of early morning people but by eight thirty there was still really just the usual ones.  No one extra was in today.

We are totally rethinking our European vacation yet again.  In our continuous attempt to mitigate the insane cost of the trip and to lower the travel time giving us more time on the ground to actually get around and enjoy Europe rather than spending all of our time moving from one location to another we are considering dropping Austria.  Austria is just too far from where we will be and it is so incredibly expensive to do anything there.  Vienna has always been a dream to go see but realistically it is adding a pretty large burden to our trip when we could be relaxing in Italy, Germany or France for a fraction of the cost.  I would be very sad not to see Austria but this just does not seem like it should be the trip for that.  Zurich and Vienna are not, in any way whatsoever, close to each other.

I went home at lunch and decided to just work from home this afternoon.  I was in early today and so had put in nearly a full day even before leaving for lunch.  I didn’t want to drive back to the office just to be there for an hour.

It was probably a good day to be home for a while today as Luciana was extremely fussy and Dominica was worn out by the time that I got home.  Luciana did not really nap all day but was clearly tired.  She probably has a tummy ache or something like that.

Mostly we relaxed this evening.  Or tried to at least.  Both kids were out of control today so relaxing was not actually very relaxing.  We watched some Star Trek: Voyager when we could and I caught up on the last two episodes of Wizards of Waverly Place that Dominica had seen but I had not.  I had some time to watch them while Dominica was out shopping and I was home alone watching both girls.

Dad emailed me tonight to let me know that his aunt Naomi, his father’s oldest sibling, had passed away at the age of 102.  Aunt Naomi is the only member of the Miller branch of the family (other than my father) that I have ever met and that was just a couple of years ago in Ohio during Monica’s graduation party.  Here is the obituary entry from the Canton Repository:

“Naomi Yoder age 102 years and 9 months, of Louisville, Ohio died at Aultman Hospital on Aug. 26, 2011 with her family by her side, after a brief illness. She was born Naomi Miller in Grantsville, Maryland on Dec. 7, 1908 to the late Joseph and Lydia (Nissley) Miller, former minister of Beachy Mennonite Church in Hartville, Ohio. She was the oldest of ten children. At age 18 she was married to Jacob J. Yoder. They had three children, Raymond, Wilbur and Norma Jean. They lived in Hartville, then in Canton, Ohio. After her husband passed away, she lived with her son and daughter-in- law, Wilbur and Lydia Yoder of Louisville, OH for 20 years.

She was preceded in death by her husband, Jacob in 1991 and her son, Raymond in 2001; five brothers, Alvin Miller, Mahlon Miller, Amos Miller, Claude Miller and Melvin Miller; three sisters, Verna Miller (Schmucker), Orpha Miller (Heilman) and Delema Miller (Beachy). Survivors include her son and daughter-in- law, Wilbur and Lydia (Miller) Yoder of Louisville, Ohio; daughter, Norma Jean Doutlick of Middlebury, Ind.; brother, Corvin Miller of North Canton, Ohio; five grandchildren, eight great grandchildren, three great great grandchildren and numerous nieces and nephews.

She was a dedicated homemaker and had a career as a housekeeper for many people in the Canton area. She retired from housekeeping at the age of 80. She was an avid children’s Sunday School Teacher for over 50 years, combined time between Canton Mennonite and First Church of the Nazarene in Canton, where she received a distinguished service award. She was involved in Women’s Home Bible Studies for 10 years and the Canton Christian Women’s Organization for 25 years. Naomi was also a great cook and was well known for her delicious pies and cookies.

No calling hours are planned. The Family will have a private graveside service. She will be buried beside her husband and son at Sunset Hills Burial Park. The Reed Funeral Home Canton Chapel is entrusted with the arrangements.”

I got to bed nice and early tonight.  I was very tired after only getting a few hours last night and having to be in the office so early today.  Dominica is planning on bringing the girls down to Irving and joining me for lunch at La Cima tomorrow.

After having worked in IT for decades you tend to accumulate anecdotes.  This is one of my favourites.

I once consulted for a major “new Ivy League” university on a project involving their database server cluster used to backend their core line of business application.  The university’s library completely relied on this system and it had to have near perfect uptimes and the data stored in it was critical.  Because of these needs the university had spared little expensive buying high end SAN, clustering the database, buying great hardware, hiring consulting engineers (like myself) and having a full time system administrator dedicated to this singular system.  The resources being thrown at this system were truly impressive which I found very surprising as I could not figure out how this system paid for itself – but that was a business decision whose factors I was not privy to so perhaps it was well justified.

The point here, however, is that the enormity of the resources thrown at the single system were staggering.  The SAN itself had to be over a half-million dollar investment alone.  Perhaps much more.

While I was consulting there one day the system administrator went into the other room to make some minor change at her console for the SAN and accidentally clicked on the wrong LUN.  One click.  One mistake.  Because it was a SAN this was a system that was regularly managed, but a “set and forget” storage system that would go years without human intervention.  This was a normal, every day change.  Nothing big.  No reason to not touch it during business hours.  No cause for concern.  No need for special safeties.  Except that clicking on the wrong line in the display was the difference between a temporarily LUN for testing and the production LUN on which the entire operation depended.

In the blink of an eye everything was gone.  One wrong click and all of the expensive, all of the redundancy, all of the planning went right out the window.  It was gone.  The entire database was just… gone.

So now we are operating in an “outage” state.  What is the plan?  What do we do from here?  We have no idea because instantly, upon realizing what had happened, the staff system administrator basically went into shock.  The stress of not only having the system go offline on her watch but to have done so due to an error that she made was too much and she was, quite instantly, useless.  She could not talk, stand or do anything.  She was so useless, in fact, that rather than dealing with the outage the remaining IT staff had to deal with getting her breathing normally and moved to a cafeteria or some other place out of the way so that she could recover and we could get to work repairing the damage.

So in this case, the human component not only caused the error (to err is human and all that) but upon having erred went into a “failure state.”  This was not good at all.

Fortunately a SAN administrator was available and able to start working on getting me available storage and I was able to start working on rebuilding the cluster.  After a few hours the systems were back online and working.  Data loss was relatively significant since the mid-priced storage and high levels of redundancy had lured them into believing that the system would not fail so the backups were not nearly as recent as one would have hoped but they were recent enough to bring the system back online eventually.

Had this event not happened on exactly the day that I was there consulting and immediately after having been walked through their cluster setup the order of magnitude of the disaster would have been much higher.  This could have been an outage of days or weeks.  There was little protection, either technically or via process, to protect against catastrophic human error and no redundancy for the most fragile piece of the system – the human.

I learned many lessons from this event that I have carried with me through my career.  These are great lessons and ones that everyone should know.  There are so many assumptions in IT that we often forget to step back and evaluate the big picture.  There are several things that could have prevented or mitigated this disaster including a better backup process, physical separations for environments and process controls to keep production changes limited to non-production hours of which there was plenty for this system since it only ran about 70-80 hours per week.  Plenty of time to have been doing work during off-hours.

Some of the things that I learned include that human error is our biggest fear – computers rarely fail as spectacularly as humans.  I learned that the presence of redundancy, even at every level, does not protect against intra-system failure – in this case the redundancy replicated the error to all points in the system instantly.  Humans need redundancy often more than computers do.  Humans need redundancy every day when they eat, sleep, travel, vacation, get sick, etc.  Computers need it only when something bad happens.  Processes are very important – had someone written down that development and production were on the same interface in a manual they probably would have realized that that was a bad idea.  And finally that having unbelievably reliable technology not only can cost more than it can possibly save but also that it makes it very tempting to trust the technology to protect against every scenario and fail to plan accordingly.

In this particular case this multi-million dollar system could have been reduced to a single, fifteen thousand dollar server and run faster and more reliably.  The complexity of the system ended up contributing to its downfall.

Medifast Status: Day 118, Down ~51.5lbs

Another one pound down today!  It would appear that my weight loss has kicked back in again.  Yay!  I’m remaining pretty close to having lost half a pound per day for four months.  That’s pretty awesome.

Today is another quiet day at home.  Nothing planned and little to do.  Did a little more cleaning today and some work in the office.  I have been working my way through some C# and MVC3 tutorials, slowly, as I get time.  I had set it up and started on one yesterday.  I’ve always been working on testing PBX in a Flash builds this weekend which I have not used in many, many generations of the product.

I stayed home with Liesl for a while today while Dominica and Luciana went out to do some quick grocery shopping.  We were running low on supplies.

Dominica did some work on my Database Administrator class from University of Illinois.  She has been putting that off for months.  I am hoping that she can wrap it up quickly.  My classes at RIT start this week.  We are attempting to figure out if there is any way for me to get student loans to help with the cost but so far things do not look very promising.

I was logged in to the office most of the day checking in regularly to see what was going on with the storm up in New York.  The New York area got hit with the storm this morning and there is heavy rain and high winds.  We are looking like we will weather it not so badly but there is going to be a lot of flooding one way or another.  It is turning out to not be nearly as bad as everyone had feared.  We have heard nothing about our own house in Peekskill yet. I suppose that no news is good news, for the most part.  Or it means that there is no power and the house was swept away in the storm.

We started watching Star Trek: Voyager today.  This is one of the “new” Star Trek series.  “New” meaning that it is newer than when I have watched Star Trek.  Through my childhood I watched Star Trek in reruns and the Star Trek cartoon in reruns and watched, new, all of TNG.  But TNG went off of the air – its final season being my senior year of high school.  Star Trek: Deep Space Nine had started a year earlier and I had seen the first season but it was not very good at all compared to TNG and so I lost interest.  It was fine to watch when nothing else was on.  But it was hardly an event to get to watch an episode.

Once I went to college I did not have access to television for quite some time.  I remember when Voyager was announced and everyone thought that it would be a big deal.  But somehow I missed seeing it, at all, until today.  Sixteen years later and I now have an entire Star Trek series that I have never seen in any way whatsoever.  I have no idea of the plot, the characters or the setting.  So it is pretty exciting to get to watch it now.

Voyager retains much of the look of TNG while carrying that generic 1990s sci-fi show look that you also see in Babylon 5, Stargate SG-1 and others during the trendy sci-fi era that was occurring them.  I missed all of these shows when they were new.  I missed practically all television after 1994.  It is really amazing how little I have seen television from that era.  Most everything that I know from then I have seen more recently because I saw it on DVD or even more recently on Netflix and Hulu.

Medifast Status: Day 117, Down ~50.5lbs

It was a very big relief when I weighed in this morning and discovered that I had passed the fifty pound mark on my Medifast diet!  Fifty pounds!  That sure sounds good to say.  I still have a very long way to go.  But this is a very big milestone.  This will re-energize me, I’m sure.

We did a bit of cleaning today and the house is looking pretty decent.  It was really nice to have a basically empty weekend in which to stay home and just deal with normal things.   I need some downtime and some time just hanging out with my girls.

I was logged in to work all day.  We are watching to see what is going to happen because New York might really get slammed by the hurricane.  We might have a lot of work to do over the weekend depending on how the storm goes.

The heat is just continuing and continuing.  It is about 104 today and we are projected for 107 for the next two days!  There is some hope of some rain in about  a week which will make a huge difference.  We can’t wait for there to be rain again.

Overall today was decently relaxing.  Dominica hung out in the office with me some today and did more research into our trip to Europe.  We are thrashing on our planning, I think.  Just going from one idea to another.

The current plan now is completely different than anything that we had talked about previously.  We are now looking at skipping Ireland entirely and flying directly from New York (or possibly New Jersey) to Warsaw, Poland and skipping our time spent far out west because that makes the whole trip that much harder trying to add in Ireland.  That would be like going to both NYC and LA in the same trip trying to hit Belfast and Warsaw.  So cutting off Belfast makes everything so much easier.

Once we decided to do that we were very relived as the trip is now far, far simpler.  This will save tons of money and time.  It will be far less stressful.  NYC to Warsaw is eight hours on a plane.  Our original plan was NYC to Ireland, Ireland to Krakow and Krakow to Warsaw for a total of twelve to sixteen hours not including layovers.

We are now planning to eliminate Croatia and most of Italy.  Instead, since we have the girls who are not going to be happy traveling all over the place, we are going to attempt to focus on Austria, Switzerland and the Alsace-Lorraine and minimize how much traveling we have to do.  The upside is it saves tons of times, effort and travel costs.  The downside is that we see far less of Europe and the places where we primarily spend our time are the most expensive ones.  So hundreds of dollars a night for a hotel instead of forty.  That is going to be rough.  We are attempting to find apartments in Austria and Switzerland that we can rent to lower the cost.  We might spend a night in Budapest to make that one night cheaper too.

I got a lot of time with my daughters today.  They are growing up so fast.  Liesl is doing great on her potty training.   That is going to save a lot of money in diapers.  And it means that she will be ready for the trip to Europe too!

We managed to relax this evening.  We watched some of Star Trek: The Next Generation which Dominica and I had started watching back in Geneseo and never got around to continuing with.  We are still partway through season two.

It is funny watching the old TNG episodes.  In my memory they were so incredibly modern and the acting and people still feel that way.  But when you actually watch it, especially as it was filmed on tape rather than film, much of it actually feels like watching the original Battlestar Galactica.  The models, the landscapes, the film quality – it really reminds you of how close in time those two shows actually were to each other.  To me, being a child at the time, the distance between them was significant.  One was from my early, murky memories and the other was a show that I looked forward to as I already knew the original Star Trek well and looked forward to a new series.  But now, looking back, TNG was much longer ago than I feel that it must have been and definitely looks like a product of its time period.  It is very enjoyable to watch still, though.