Sheep Guarding Llama

Medifast Status: Day 122, Down ~51.5lbs

I am completely exhausted today.  Really dragging.  Even though I am so tired I woke up twenty minutes before my alarm this morning.  I couldn’t make myself get up extra early, though, so I went back to sleep until the alarm went off.  Not normal for me.  Normally if I am up I just get up and get started.

I came home for lunch today.  La Cima keeps forgetting to email out the lunch menu in the mornings and when we don’t receive it we really don’t even consider going over there since we have no idea if there will be something that we want to eat or not.

Work was not too busy today.  The market volatility, nature disasters and everything else have caused us to really have very little to do as we just go from one freeze to another.

This evening Dominica decided that she wanted to read and not watch anything – which is good because it makes us a lot more production.  Once the television is on we are pretty much trapped in the play room and it is very hard to do anything.  That is the biggest disadvantage of the playroom television situation.  It keeps the television from taking  over the main parts of the house but it also causes us to be isolated when using it.  If the television was in the living room than walking around and doing whatever is relatively easy as the television can be seen from just about everywhere at the same time.  In the playroom there are just the two seats and even walking over to the bar to get coffee makes you go behind the television.  So the bar and kitchen not having line of sight to the television is bad for everyday use.  No different than using the one in our bedroom, though.

I had my two hour Thursday evening call tonight.  While I was on Dominica brought me dinner.  She made her cauliflower pizza which has now gone through another iteration of development and is continuing to improve.  It is really tasty and actually begins to satisfy the craving for pizza.  That is one of the foods that I miss the most.  This is not the same but it is far, far closer to the real thing than you might imagine.  We even use vegetarian pepperoni substitute.

Dominica spent the evening reading travel information and I spent the evening working in the office.  Luciana went to bed so early that she was in bed while I was still on my call.  Liesl painted and watched Wonder Pets most of the evening.

I was on FaceBook tonight looking to talk to Roni from high school, she graduated with me, because she lives in Tuscany and I needed some travel advice.  It turns out that she moved to Dallas just last week.  How bizarre is that?  Having grown up walking distance from each other, being about nine houses away, we now live thousands of miles away and are just twenty minutes separated.  Pretty weird.  So I talked to her for a bit and we are having lunch on Wednesday.

Roni also told me that Peggy, also who was in our class, moved to Dallas as well.  So there are at least three of us from my graduating high school class of only sixty four people now living in Dallas.  That means that 5% of my class is here.  Now that is not something that I would have ever suspected.

Medifast Status: Day 121, Down ~51.5lbs

Today was Dominica’s second time out running in her C25K program.  Today went better with no dogs chasing her and she did not accidentally double the distance that she was supposed to have gone.  I need to get out and do some exercise to keep my diet on track.  I’m waiting for the heat wave to break before I try to get out walking.  We are supposed to cool down on Monday.  I’m hopeful.  Getting out of the 100s will be awesome.  I won’t have to spend my days monitoring the status of the air conditioning system and we can uncover the windows and get light into the house again too.

I liked the look of the lunch menu at La Cima today so Dominica and the girls picked me up from work and we headed over there.  Yesterday it was tilapia and eggplant.  Today we had salmon and zucchini con queso which was really good.  We found out today that our membership options are about to change at the club and very much in our favour.  The new options will cost less and give us more dining!  Hard to beat that.  I don’t know how the golf benefits change; I need to find out about that yet.  But costing less per month is awesome and moving from two free meals to half priced dining is great.  Some of the other clubs in the area already had that and we were jealous.  So we are very happy to hear that we might be able to switch over as early as next month… which starts tomorrow.

I really like doing the club for lunch.  Because I don’t have to drive home and back I get quite a bit more time with the girls when we go to the club.  It is a bit of work for Dominica so I don’t want to do it all of the time but Liesl and Luciana like getting to see me and Liesl just loves being at the club.  It is so fun driving there with her.  She sees it from quite far away and points and says “I see it, I see it!”

Lunch was nice and we got time to relax for a while.  This evening we managed to get quite a bit of family time in together.  It was a good evening.

Liesl has a new Dora toilet seat cover for her bathroom as she is starting to move from the training potty to the real deal.  She is doing really well.

Liesl spent a good twenty minutes hanging out with me tonight.  She made me sit at the far end of the house while she performed song and dance numbers for me.  She would make up a song and do a cross between ballet and interpretive dance and when she was done she would take a deep bow and expect some applause.  All very funny.  She did this over and over.  She would the “Daddy Song” and sing about her daddy or the “Luciana Song” and she about her sister.  She sang a song about everyone and some of her toys as well.  Even Oreo got a turn.

This week, like last, is just flying by.  I cannot believe that it is already Wednesday.  Maybe it is going into the office earlier than makes me feel like time is moving past more quickly.

Medifast Status: Day 120, Down ~51.5lbs

I am a little behind on the news but just found out that GameStop, the bigger US video game retailer, has been opening products that they sell as “new” and taking stuff out of the packaging – and doing so officially.  Talk about completely illegal and unethical.  And they were doing it to customers who were pre-ordering new games!  Not just stuff off of the shelf that people could inspect before purchasing.  This is like allowing someone to custom order a new car then letting the employees of the car dealer drive it on their vacation for a week before letting the customer take delivery of it – even though they already bought and paid for it as new!  In the past I have always supported GameStop because I like to support my local video game store even though they tend to cost more than Amazon and are less convenient to use.  But no more.  I immediately threw out my GameStop card and will never shop there again.  I don’t need to deal with people like that.

Work was very slow this morning.  At nine John and I drove over to La Cima to escape for a while and to enjoy some good coffee.  We just missed breakfast which we need to try to hit sometime.  I always forget about getting free breakfast over there and since I am paying for it already we should really take advantage of it.  A guest is included in the breakfast deal so John can get breakfast there for free.  I can’t eat breakfast in any meaningful way on the MediFast plan so I’m pretty much out of luck.  But at least the coffee is good.

Today was my first time getting to see the club since the new reception desk and waterfall were installed.  They look way better than they did before.  I also noticed that two of the most annoying televisions have been removed as well.  So now there are a total of four televisions in the lounge area rather than six.  Far from good but much better than it was before.

Dominica and the girls came down and picked me up for lunch so I was at the club twice today.  It was a tomato and olive sauce on tilapia which was quite good.  Liesl loved the new waterfall and spent a bit of time just standing in front of it watching it up close.  Dominica thought that the removal of two of the televisions helped the look of the place a bit too.

It was a slow afternoon after getting back from lunch.  This week is particularly slow, people think, because of the weather in New York affecting the traders.

I got home and managed to actually have a pretty good evening with the family.  I got a lot of time in with the girls.  Liesl was in a particularly good mood tonight and never got into any trouble, which is amazing as her attitude lately has not been the best.  She was a little angel tonight.

Luciana also had a good day.  We’ve been trying to add rice cereal to her bottles but she gets really frustrated that she cannot then get the formula out of the bottle.  So Dominica tried actually spoon feeding her and she actually loved it.  She opens her mouth and loves eating food.  Totally unlike Liesl who hated to eat and ended up going directly from bottle to solid food and even now gets around half of her total nutrition via Pedisure because she prefers to drink her food than to eat it.  I suppose that she gets that from me.  That is why the Medifast works out for me – I prefer convenience to pretty much everything else.

So this is really awesome.  Luciana has been so fussy for days.  We think that this will fix it.  And she will likely get back to sleeping again.  She has barely slept for a week.

Medifast Status: Day 119, Down ~51.5lbs

I had to get up at five this morning so that I could get ready quickly and get into the office.  Because of the storm in New York I am on the early, early shift today – starting at six – so that I can help with any issues that may have arisen over the weekend.  This morning is also Dominica’s first day of trying out running in her new “C25K” or Couch to 5K program.  So she got up at five as well and while I was showering and getting ready for work she went out jogging around the neighbourhood.

Dominica’s run went okay but she ended up being chased by a pit bull and had to come back home a different way which caused her to be much later than she had meant to be so instead of getting back at five forty so that I could jump in the car and jet off to the office it was actually six when she got back to the house so that I could leave.  So I was already late to work.  Fortunately it only took me eleven minutes to get to the office as there was no traffic whatsoever.

Like many sites I noticed this morning, SpiceWorks is down.  It is days like today, with much of the northeast without power, that we see how many companies rely on single sites to keep their web and email online without power redundancy, regional failover, etc.  One datacenter goes down and tons of companies go offline completely.  The upside of SpiceWorks being offline is that it is that much easier for me to catch up with everything else now.  Normally that sucks up tons of my time.

Things were pretty quiet in the office this morning.  I thought that there would be a lot of early morning people but by eight thirty there was still really just the usual ones.  No one extra was in today.

We are totally rethinking our European vacation yet again.  In our continuous attempt to mitigate the insane cost of the trip and to lower the travel time giving us more time on the ground to actually get around and enjoy Europe rather than spending all of our time moving from one location to another we are considering dropping Austria.  Austria is just too far from where we will be and it is so incredibly expensive to do anything there.  Vienna has always been a dream to go see but realistically it is adding a pretty large burden to our trip when we could be relaxing in Italy, Germany or France for a fraction of the cost.  I would be very sad not to see Austria but this just does not seem like it should be the trip for that.  Zurich and Vienna are not, in any way whatsoever, close to each other.

I went home at lunch and decided to just work from home this afternoon.  I was in early today and so had put in nearly a full day even before leaving for lunch.  I didn’t want to drive back to the office just to be there for an hour.

It was probably a good day to be home for a while today as Luciana was extremely fussy and Dominica was worn out by the time that I got home.  Luciana did not really nap all day but was clearly tired.  She probably has a tummy ache or something like that.

Mostly we relaxed this evening.  Or tried to at least.  Both kids were out of control today so relaxing was not actually very relaxing.  We watched some Star Trek: Voyager when we could and I caught up on the last two episodes of Wizards of Waverly Place that Dominica had seen but I had not.  I had some time to watch them while Dominica was out shopping and I was home alone watching both girls.

Dad emailed me tonight to let me know that his aunt Naomi, his father’s oldest sibling, had passed away at the age of 102.  Aunt Naomi is the only member of the Miller branch of the family (other than my father) that I have ever met and that was just a couple of years ago in Ohio during Monica’s graduation party.  Here is the obituary entry from the Canton Repository:

“Naomi Yoder age 102 years and 9 months, of Louisville, Ohio died at Aultman Hospital on Aug. 26, 2011 with her family by her side, after a brief illness. She was born Naomi Miller in Grantsville, Maryland on Dec. 7, 1908 to the late Joseph and Lydia (Nissley) Miller, former minister of Beachy Mennonite Church in Hartville, Ohio. She was the oldest of ten children. At age 18 she was married to Jacob J. Yoder. They had three children, Raymond, Wilbur and Norma Jean. They lived in Hartville, then in Canton, Ohio. After her husband passed away, she lived with her son and daughter-in- law, Wilbur and Lydia Yoder of Louisville, OH for 20 years.

She was preceded in death by her husband, Jacob in 1991 and her son, Raymond in 2001; five brothers, Alvin Miller, Mahlon Miller, Amos Miller, Claude Miller and Melvin Miller; three sisters, Verna Miller (Schmucker), Orpha Miller (Heilman) and Delema Miller (Beachy). Survivors include her son and daughter-in- law, Wilbur and Lydia (Miller) Yoder of Louisville, Ohio; daughter, Norma Jean Doutlick of Middlebury, Ind.; brother, Corvin Miller of North Canton, Ohio; five grandchildren, eight great grandchildren, three great great grandchildren and numerous nieces and nephews.

She was a dedicated homemaker and had a career as a housekeeper for many people in the Canton area. She retired from housekeeping at the age of 80. She was an avid children’s Sunday School Teacher for over 50 years, combined time between Canton Mennonite and First Church of the Nazarene in Canton, where she received a distinguished service award. She was involved in Women’s Home Bible Studies for 10 years and the Canton Christian Women’s Organization for 25 years. Naomi was also a great cook and was well known for her delicious pies and cookies.

No calling hours are planned. The Family will have a private graveside service. She will be buried beside her husband and son at Sunset Hills Burial Park. The Reed Funeral Home Canton Chapel is entrusted with the arrangements.”

I got to bed nice and early tonight.  I was very tired after only getting a few hours last night and having to be in the office so early today.  Dominica is planning on bringing the girls down to Irving and joining me for lunch at La Cima tomorrow.

After having worked in IT for decades you tend to accumulate anecdotes.  This is one of my favourites.

I once consulted for a major “new Ivy League” university on a project involving their database server cluster used to backend their core line of business application.  The university’s library completely relied on this system and it had to have near perfect uptimes and the data stored in it was critical.  Because of these needs the university had spared little expensive buying high end SAN, clustering the database, buying great hardware, hiring consulting engineers (like myself) and having a full time system administrator dedicated to this singular system.  The resources being thrown at this system were truly impressive which I found very surprising as I could not figure out how this system paid for itself – but that was a business decision whose factors I was not privy to so perhaps it was well justified.

The point here, however, is that the enormity of the resources thrown at the single system were staggering.  The SAN itself had to be over a half-million dollar investment alone.  Perhaps much more.

While I was consulting there one day the system administrator went into the other room to make some minor change at her console for the SAN and accidentally clicked on the wrong LUN.  One click.  One mistake.  Because it was a SAN this was a system that was regularly managed, but a “set and forget” storage system that would go years without human intervention.  This was a normal, every day change.  Nothing big.  No reason to not touch it during business hours.  No cause for concern.  No need for special safeties.  Except that clicking on the wrong line in the display was the difference between a temporarily LUN for testing and the production LUN on which the entire operation depended.

In the blink of an eye everything was gone.  One wrong click and all of the expensive, all of the redundancy, all of the planning went right out the window.  It was gone.  The entire database was just… gone.

So now we are operating in an “outage” state.  What is the plan?  What do we do from here?  We have no idea because instantly, upon realizing what had happened, the staff system administrator basically went into shock.  The stress of not only having the system go offline on her watch but to have done so due to an error that she made was too much and she was, quite instantly, useless.  She could not talk, stand or do anything.  She was so useless, in fact, that rather than dealing with the outage the remaining IT staff had to deal with getting her breathing normally and moved to a cafeteria or some other place out of the way so that she could recover and we could get to work repairing the damage.

So in this case, the human component not only caused the error (to err is human and all that) but upon having erred went into a “failure state.”  This was not good at all.

Fortunately a SAN administrator was available and able to start working on getting me available storage and I was able to start working on rebuilding the cluster.  After a few hours the systems were back online and working.  Data loss was relatively significant since the mid-priced storage and high levels of redundancy had lured them into believing that the system would not fail so the backups were not nearly as recent as one would have hoped but they were recent enough to bring the system back online eventually.

Had this event not happened on exactly the day that I was there consulting and immediately after having been walked through their cluster setup the order of magnitude of the disaster would have been much higher.  This could have been an outage of days or weeks.  There was little protection, either technically or via process, to protect against catastrophic human error and no redundancy for the most fragile piece of the system – the human.

I learned many lessons from this event that I have carried with me through my career.  These are great lessons and ones that everyone should know.  There are so many assumptions in IT that we often forget to step back and evaluate the big picture.  There are several things that could have prevented or mitigated this disaster including a better backup process, physical separations for environments and process controls to keep production changes limited to non-production hours of which there was plenty for this system since it only ran about 70-80 hours per week.  Plenty of time to have been doing work during off-hours.

Some of the things that I learned include that human error is our biggest fear – computers rarely fail as spectacularly as humans.  I learned that the presence of redundancy, even at every level, does not protect against intra-system failure – in this case the redundancy replicated the error to all points in the system instantly.  Humans need redundancy often more than computers do.  Humans need redundancy every day when they eat, sleep, travel, vacation, get sick, etc.  Computers need it only when something bad happens.  Processes are very important – had someone written down that development and production were on the same interface in a manual they probably would have realized that that was a bad idea.  And finally that having unbelievably reliable technology not only can cost more than it can possibly save but also that it makes it very tempting to trust the technology to protect against every scenario and fail to plan accordingly.

In this particular case this multi-million dollar system could have been reduced to a single, fifteen thousand dollar server and run faster and more reliably.  The complexity of the system ended up contributing to its downfall.