Sheep Guarding Llama

Medifast Status: Day 121, Down ~51.5lbs

Today was Dominica’s second time out running in her C25K program.  Today went better with no dogs chasing her and she did not accidentally double the distance that she was supposed to have gone.  I need to get out and do some exercise to keep my diet on track.  I’m waiting for the heat wave to break before I try to get out walking.  We are supposed to cool down on Monday.  I’m hopeful.  Getting out of the 100s will be awesome.  I won’t have to spend my days monitoring the status of the air conditioning system and we can uncover the windows and get light into the house again too.

I liked the look of the lunch menu at La Cima today so Dominica and the girls picked me up from work and we headed over there.  Yesterday it was tilapia and eggplant.  Today we had salmon and zucchini con queso which was really good.  We found out today that our membership options are about to change at the club and very much in our favour.  The new options will cost less and give us more dining!  Hard to beat that.  I don’t know how the golf benefits change; I need to find out about that yet.  But costing less per month is awesome and moving from two free meals to half priced dining is great.  Some of the other clubs in the area already had that and we were jealous.  So we are very happy to hear that we might be able to switch over as early as next month… which starts tomorrow.

I really like doing the club for lunch.  Because I don’t have to drive home and back I get quite a bit more time with the girls when we go to the club.  It is a bit of work for Dominica so I don’t want to do it all of the time but Liesl and Luciana like getting to see me and Liesl just loves being at the club.  It is so fun driving there with her.  She sees it from quite far away and points and says “I see it, I see it!”

Lunch was nice and we got time to relax for a while.  This evening we managed to get quite a bit of family time in together.  It was a good evening.

Liesl has a new Dora toilet seat cover for her bathroom as she is starting to move from the training potty to the real deal.  She is doing really well.

Liesl spent a good twenty minutes hanging out with me tonight.  She made me sit at the far end of the house while she performed song and dance numbers for me.  She would make up a song and do a cross between ballet and interpretive dance and when she was done she would take a deep bow and expect some applause.  All very funny.  She did this over and over.  She would the “Daddy Song” and sing about her daddy or the “Luciana Song” and she about her sister.  She sang a song about everyone and some of her toys as well.  Even Oreo got a turn.

This week, like last, is just flying by.  I cannot believe that it is already Wednesday.  Maybe it is going into the office earlier than makes me feel like time is moving past more quickly.

Medifast Status: Day 120, Down ~51.5lbs

I am a little behind on the news but just found out that GameStop, the bigger US video game retailer, has been opening products that they sell as “new” and taking stuff out of the packaging – and doing so officially.  Talk about completely illegal and unethical.  And they were doing it to customers who were pre-ordering new games!  Not just stuff off of the shelf that people could inspect before purchasing.  This is like allowing someone to custom order a new car then letting the employees of the car dealer drive it on their vacation for a week before letting the customer take delivery of it – even though they already bought and paid for it as new!  In the past I have always supported GameStop because I like to support my local video game store even though they tend to cost more than Amazon and are less convenient to use.  But no more.  I immediately threw out my GameStop card and will never shop there again.  I don’t need to deal with people like that.

Work was very slow this morning.  At nine John and I drove over to La Cima to escape for a while and to enjoy some good coffee.  We just missed breakfast which we need to try to hit sometime.  I always forget about getting free breakfast over there and since I am paying for it already we should really take advantage of it.  A guest is included in the breakfast deal so John can get breakfast there for free.  I can’t eat breakfast in any meaningful way on the MediFast plan so I’m pretty much out of luck.  But at least the coffee is good.

Today was my first time getting to see the club since the new reception desk and waterfall were installed.  They look way better than they did before.  I also noticed that two of the most annoying televisions have been removed as well.  So now there are a total of four televisions in the lounge area rather than six.  Far from good but much better than it was before.

Dominica and the girls came down and picked me up for lunch so I was at the club twice today.  It was a tomato and olive sauce on tilapia which was quite good.  Liesl loved the new waterfall and spent a bit of time just standing in front of it watching it up close.  Dominica thought that the removal of two of the televisions helped the look of the place a bit too.

It was a slow afternoon after getting back from lunch.  This week is particularly slow, people think, because of the weather in New York affecting the traders.

I got home and managed to actually have a pretty good evening with the family.  I got a lot of time in with the girls.  Liesl was in a particularly good mood tonight and never got into any trouble, which is amazing as her attitude lately has not been the best.  She was a little angel tonight.

Luciana also had a good day.  We’ve been trying to add rice cereal to her bottles but she gets really frustrated that she cannot then get the formula out of the bottle.  So Dominica tried actually spoon feeding her and she actually loved it.  She opens her mouth and loves eating food.  Totally unlike Liesl who hated to eat and ended up going directly from bottle to solid food and even now gets around half of her total nutrition via Pedisure because she prefers to drink her food than to eat it.  I suppose that she gets that from me.  That is why the Medifast works out for me – I prefer convenience to pretty much everything else.

So this is really awesome.  Luciana has been so fussy for days.  We think that this will fix it.  And she will likely get back to sleeping again.  She has barely slept for a week.

Medifast Status: Day 119, Down ~51.5lbs

I had to get up at five this morning so that I could get ready quickly and get into the office.  Because of the storm in New York I am on the early, early shift today – starting at six – so that I can help with any issues that may have arisen over the weekend.  This morning is also Dominica’s first day of trying out running in her new “C25K” or Couch to 5K program.  So she got up at five as well and while I was showering and getting ready for work she went out jogging around the neighbourhood.

Dominica’s run went okay but she ended up being chased by a pit bull and had to come back home a different way which caused her to be much later than she had meant to be so instead of getting back at five forty so that I could jump in the car and jet off to the office it was actually six when she got back to the house so that I could leave.  So I was already late to work.  Fortunately it only took me eleven minutes to get to the office as there was no traffic whatsoever.

Like many sites I noticed this morning, SpiceWorks is down.  It is days like today, with much of the northeast without power, that we see how many companies rely on single sites to keep their web and email online without power redundancy, regional failover, etc.  One datacenter goes down and tons of companies go offline completely.  The upside of SpiceWorks being offline is that it is that much easier for me to catch up with everything else now.  Normally that sucks up tons of my time.

Things were pretty quiet in the office this morning.  I thought that there would be a lot of early morning people but by eight thirty there was still really just the usual ones.  No one extra was in today.

We are totally rethinking our European vacation yet again.  In our continuous attempt to mitigate the insane cost of the trip and to lower the travel time giving us more time on the ground to actually get around and enjoy Europe rather than spending all of our time moving from one location to another we are considering dropping Austria.  Austria is just too far from where we will be and it is so incredibly expensive to do anything there.  Vienna has always been a dream to go see but realistically it is adding a pretty large burden to our trip when we could be relaxing in Italy, Germany or France for a fraction of the cost.  I would be very sad not to see Austria but this just does not seem like it should be the trip for that.  Zurich and Vienna are not, in any way whatsoever, close to each other.

I went home at lunch and decided to just work from home this afternoon.  I was in early today and so had put in nearly a full day even before leaving for lunch.  I didn’t want to drive back to the office just to be there for an hour.

It was probably a good day to be home for a while today as Luciana was extremely fussy and Dominica was worn out by the time that I got home.  Luciana did not really nap all day but was clearly tired.  She probably has a tummy ache or something like that.

Mostly we relaxed this evening.  Or tried to at least.  Both kids were out of control today so relaxing was not actually very relaxing.  We watched some Star Trek: Voyager when we could and I caught up on the last two episodes of Wizards of Waverly Place that Dominica had seen but I had not.  I had some time to watch them while Dominica was out shopping and I was home alone watching both girls.

Dad emailed me tonight to let me know that his aunt Naomi, his father’s oldest sibling, had passed away at the age of 102.  Aunt Naomi is the only member of the Miller branch of the family (other than my father) that I have ever met and that was just a couple of years ago in Ohio during Monica’s graduation party.  Here is the obituary entry from the Canton Repository:

“Naomi Yoder age 102 years and 9 months, of Louisville, Ohio died at Aultman Hospital on Aug. 26, 2011 with her family by her side, after a brief illness. She was born Naomi Miller in Grantsville, Maryland on Dec. 7, 1908 to the late Joseph and Lydia (Nissley) Miller, former minister of Beachy Mennonite Church in Hartville, Ohio. She was the oldest of ten children. At age 18 she was married to Jacob J. Yoder. They had three children, Raymond, Wilbur and Norma Jean. They lived in Hartville, then in Canton, Ohio. After her husband passed away, she lived with her son and daughter-in- law, Wilbur and Lydia Yoder of Louisville, OH for 20 years.

She was preceded in death by her husband, Jacob in 1991 and her son, Raymond in 2001; five brothers, Alvin Miller, Mahlon Miller, Amos Miller, Claude Miller and Melvin Miller; three sisters, Verna Miller (Schmucker), Orpha Miller (Heilman) and Delema Miller (Beachy). Survivors include her son and daughter-in- law, Wilbur and Lydia (Miller) Yoder of Louisville, Ohio; daughter, Norma Jean Doutlick of Middlebury, Ind.; brother, Corvin Miller of North Canton, Ohio; five grandchildren, eight great grandchildren, three great great grandchildren and numerous nieces and nephews.

She was a dedicated homemaker and had a career as a housekeeper for many people in the Canton area. She retired from housekeeping at the age of 80. She was an avid children’s Sunday School Teacher for over 50 years, combined time between Canton Mennonite and First Church of the Nazarene in Canton, where she received a distinguished service award. She was involved in Women’s Home Bible Studies for 10 years and the Canton Christian Women’s Organization for 25 years. Naomi was also a great cook and was well known for her delicious pies and cookies.

No calling hours are planned. The Family will have a private graveside service. She will be buried beside her husband and son at Sunset Hills Burial Park. The Reed Funeral Home Canton Chapel is entrusted with the arrangements.”

I got to bed nice and early tonight.  I was very tired after only getting a few hours last night and having to be in the office so early today.  Dominica is planning on bringing the girls down to Irving and joining me for lunch at La Cima tomorrow.

After having worked in IT for decades you tend to accumulate anecdotes.  This is one of my favourites.

I once consulted for a major “new Ivy League” university on a project involving their database server cluster used to backend their core line of business application.  The university’s library completely relied on this system and it had to have near perfect uptimes and the data stored in it was critical.  Because of these needs the university had spared little expensive buying high end SAN, clustering the database, buying great hardware, hiring consulting engineers (like myself) and having a full time system administrator dedicated to this singular system.  The resources being thrown at this system were truly impressive which I found very surprising as I could not figure out how this system paid for itself – but that was a business decision whose factors I was not privy to so perhaps it was well justified.

The point here, however, is that the enormity of the resources thrown at the single system were staggering.  The SAN itself had to be over a half-million dollar investment alone.  Perhaps much more.

While I was consulting there one day the system administrator went into the other room to make some minor change at her console for the SAN and accidentally clicked on the wrong LUN.  One click.  One mistake.  Because it was a SAN this was a system that was regularly managed, but a “set and forget” storage system that would go years without human intervention.  This was a normal, every day change.  Nothing big.  No reason to not touch it during business hours.  No cause for concern.  No need for special safeties.  Except that clicking on the wrong line in the display was the difference between a temporarily LUN for testing and the production LUN on which the entire operation depended.

In the blink of an eye everything was gone.  One wrong click and all of the expensive, all of the redundancy, all of the planning went right out the window.  It was gone.  The entire database was just… gone.

So now we are operating in an “outage” state.  What is the plan?  What do we do from here?  We have no idea because instantly, upon realizing what had happened, the staff system administrator basically went into shock.  The stress of not only having the system go offline on her watch but to have done so due to an error that she made was too much and she was, quite instantly, useless.  She could not talk, stand or do anything.  She was so useless, in fact, that rather than dealing with the outage the remaining IT staff had to deal with getting her breathing normally and moved to a cafeteria or some other place out of the way so that she could recover and we could get to work repairing the damage.

So in this case, the human component not only caused the error (to err is human and all that) but upon having erred went into a “failure state.”  This was not good at all.

Fortunately a SAN administrator was available and able to start working on getting me available storage and I was able to start working on rebuilding the cluster.  After a few hours the systems were back online and working.  Data loss was relatively significant since the mid-priced storage and high levels of redundancy had lured them into believing that the system would not fail so the backups were not nearly as recent as one would have hoped but they were recent enough to bring the system back online eventually.

Had this event not happened on exactly the day that I was there consulting and immediately after having been walked through their cluster setup the order of magnitude of the disaster would have been much higher.  This could have been an outage of days or weeks.  There was little protection, either technically or via process, to protect against catastrophic human error and no redundancy for the most fragile piece of the system – the human.

I learned many lessons from this event that I have carried with me through my career.  These are great lessons and ones that everyone should know.  There are so many assumptions in IT that we often forget to step back and evaluate the big picture.  There are several things that could have prevented or mitigated this disaster including a better backup process, physical separations for environments and process controls to keep production changes limited to non-production hours of which there was plenty for this system since it only ran about 70-80 hours per week.  Plenty of time to have been doing work during off-hours.

Some of the things that I learned include that human error is our biggest fear – computers rarely fail as spectacularly as humans.  I learned that the presence of redundancy, even at every level, does not protect against intra-system failure – in this case the redundancy replicated the error to all points in the system instantly.  Humans need redundancy often more than computers do.  Humans need redundancy every day when they eat, sleep, travel, vacation, get sick, etc.  Computers need it only when something bad happens.  Processes are very important – had someone written down that development and production were on the same interface in a manual they probably would have realized that that was a bad idea.  And finally that having unbelievably reliable technology not only can cost more than it can possibly save but also that it makes it very tempting to trust the technology to protect against every scenario and fail to plan accordingly.

In this particular case this multi-million dollar system could have been reduced to a single, fifteen thousand dollar server and run faster and more reliably.  The complexity of the system ended up contributing to its downfall.

Medifast Status: Day 118, Down ~51.5lbs

Another one pound down today!  It would appear that my weight loss has kicked back in again.  Yay!  I’m remaining pretty close to having lost half a pound per day for four months.  That’s pretty awesome.

Today is another quiet day at home.  Nothing planned and little to do.  Did a little more cleaning today and some work in the office.  I have been working my way through some C# and MVC3 tutorials, slowly, as I get time.  I had set it up and started on one yesterday.  I’ve always been working on testing PBX in a Flash builds this weekend which I have not used in many, many generations of the product.

I stayed home with Liesl for a while today while Dominica and Luciana went out to do some quick grocery shopping.  We were running low on supplies.

Dominica did some work on my Database Administrator class from University of Illinois.  She has been putting that off for months.  I am hoping that she can wrap it up quickly.  My classes at RIT start this week.  We are attempting to figure out if there is any way for me to get student loans to help with the cost but so far things do not look very promising.

I was logged in to the office most of the day checking in regularly to see what was going on with the storm up in New York.  The New York area got hit with the storm this morning and there is heavy rain and high winds.  We are looking like we will weather it not so badly but there is going to be a lot of flooding one way or another.  It is turning out to not be nearly as bad as everyone had feared.  We have heard nothing about our own house in Peekskill yet. I suppose that no news is good news, for the most part.  Or it means that there is no power and the house was swept away in the storm.

We started watching Star Trek: Voyager today.  This is one of the “new” Star Trek series.  “New” meaning that it is newer than when I have watched Star Trek.  Through my childhood I watched Star Trek in reruns and the Star Trek cartoon in reruns and watched, new, all of TNG.  But TNG went off of the air – its final season being my senior year of high school.  Star Trek: Deep Space Nine had started a year earlier and I had seen the first season but it was not very good at all compared to TNG and so I lost interest.  It was fine to watch when nothing else was on.  But it was hardly an event to get to watch an episode.

Once I went to college I did not have access to television for quite some time.  I remember when Voyager was announced and everyone thought that it would be a big deal.  But somehow I missed seeing it, at all, until today.  Sixteen years later and I now have an entire Star Trek series that I have never seen in any way whatsoever.  I have no idea of the plot, the characters or the setting.  So it is pretty exciting to get to watch it now.

Voyager retains much of the look of TNG while carrying that generic 1990s sci-fi show look that you also see in Babylon 5, Stargate SG-1 and others during the trendy sci-fi era that was occurring them.  I missed all of these shows when they were new.  I missed practically all television after 1994.  It is really amazing how little I have seen television from that era.  Most everything that I know from then I have seen more recently because I saw it on DVD or even more recently on Netflix and Hulu.