January 17, 2009: Pulling a Double on a Saturday

So much for a nice, relaxing weekend.  I was woken up at eight this morning by my boss calling to let me know that we had a massive disaster at work over the last few hours and that it was all hands on deck.  By a quarter after eight I was down in the basement office (where it is freezing cold) and on a conference call.

My day turned into not only a full work day but it was, in fact, a double work day.  I had originally had some work scheduled to take place this morning but that had to be pushed off as normal work can not occur while there is an outage going on.  That would just add to the chaos.

Today’s issue was caused by the extreme cold conditions in Manhattan this morning.  We were sub-zero before the sun rose this morning.  As often happens when things get this cold, pipes start to freeze which can cause ironic overheating situations when HVAC systems fail or flooding when ice blocks water moving in pipes.

I got a little break in the middle of the morning while waiting between conference calls when I got to play a little bit of Oblivion while keeping an eye on emails.  I didn’t really get to see Dominica, Liesl or Oreo at all today as they were all still asleep during the only break that I had.  I did get to come upstairs and eat lunch and dinner with them but that was about it.  As soon as food was finished I had to head back down to the basement to get back to work.

After fifteen hours things finally cooled down enough (figuratively as well as literally) at work that I was able to start on the work that I was originally going to do first thing this morning.  Wow.  This was just a ridiculously long day.  Almost all of it was spent on the phone.  I had my headset on almost all day.  I went through several handset batteries.

Work continued on into the evening.  In total I ended up working over sixteen hours today!  It was after midnight that my very last server came online in the datacenter.  I officially did nothing but work today.  This was one of the busiest days at the office that I can recall.

Dominica, Liesl and Oreo all went to bed before I was done working.  You know that it has been a long day when the people who got up after you are going to bed while you are still working.  I can barely keep my eyes open as I sit here wrapping up SGL for the day.  I am very, very tired of sitting at my desk.  This has been my longest stretch at my desk since we moved to Peekskill.

Tomorrow I am definitely attempting to sleep in and to take it easy all day.  I really need a break after today.  There is a very high liklihood that there will be residual problems from today’s disaster that will need my attention tomorrow but I anticipate that they will be few and far between.  As long as I get to sleep in in the morning then everything should be okay.

Okay, off to bed for me.  All appears to be right with the world.  I need some Boston Terrier snuggles.

January 16, 2009: Short Friday, Sort Of

Today I got the wonderful surprise of starting work and then finding out that not only is today a short day with the market closing at two in the afternoon but that Monday is a holiday!  So it is a long weekend for me.  Yay!  We are thinking about shopping for some window dressing for the living room this weekend.  We need to them to help insulate the house.  The big window in the living room is a big heat sink sucking the money right out of the house all winter.

The weekend might be a long and much needed break but today was a completely crazy, long day.  Everyone rushing to get work done prior to the weekend.  I spent the entire day just running from one request to another.  Very exhausting.

Lucking today ended on the early side (whatever that means) so I was able to wrap up around the time that I would be done on a normal day rather than on a normal Friday.

It occurred to me that I have now been working in IT for fifteen years (I started in 1994) and that I have been a consultant for a decade (started consulting in 1998 or 1999.)  Now I feel old.

After work Dominica decided that she wanted to read (she has some new novels from Amazon) and so I spent the evening playing Oblivion on the PS3.  I managed to knock of several quests tonight.

I have nothing major planned for the weekend.  It is my first weekend off in a while.  Ramona is planning on stopping up to Peekskill on Monday afternoon.  It will be her first trip to the new house and her first time seeing Liesl.

January 15, 2009: Getting Into the Post Holiday Groove

“The impact of the impossible is often rather small.”

Boy is it a cold day today.  I am so thankful that it is so much warmer here than so many other places that I have lived in the past.  We got down to around fifteen degrees in the middle of the day but the sun was out and I was still able to walk Oreo in my pajamas and fleece jacket and no gloves, hat, scarf, winter coat, etc.  Oreo was wearing his t-shirt and fuzzy sweater.  He did fine though.  He was not nearly as cold as he gets when the snow is deep and wet.  That is the worst for him.

The basement is extremely cold today.  My feet feel like they are going to fall off.  It would be very cost prohibitive to attempt to keep the basement warm.  There is no point, really, it is just me down here.  Oreo chooses to remain in the living room with the warm people when it gets this cold down here.

Today was an extremely busy day at the office.  Probably the busiest yet this year.  Dominica really needed some help with Liesl this afternoon, but I was so busy that I couldn’t leave the basement.

It was late by the time that I was able to go upstairs and join Dominica for dinner.  We ate butternut squash pasta and watched some of the second season of The Dick Van Dyke Show.  Then, after dinner, Dominica decided that she felt like reading so she read a book and I took the opportunity to play some Oblivion on the PS3.

This was my first time playing Oblivion on the new television and I can honestly say that the 720p cap output on that game from the PS3 really shows on the 1080p LCD.  It doesn’t really look any better than the old television did (bigger, but not really better) while the upgrade to Fable 2 was tremendous.  Anyone who says that you can’t see the difference between 720p and 1080p needs to be declared legally blind.  It is a significant difference that you should notice immediately.  I don’t have very good eyesight and it was glaringly obvious the moment that the game started that there were scaling issues and that the scenes were pixelated in comparison to Fable 2.  Now I am quite envious of Andy who played Oblivion on the PC with much higher resolution than my version and full 3d (with polarized glasses.)  Really makes the console version look sad.

Today was a really rough day for Liesl. She was quite unhappy all day long.  We don’t know if she just had a tummy ache or what but the entire day she was quite distraught.  We are all glad that today is finally over.

Managing Apache and Subversion Through Active Directory (Part 1 – Authentication)

In my previous article, Installing Subversion on RHEL5, we went over how to install the Subversion server and how to make it accessible through the Apache web server.  This solution is great but leaves us without any user authorization and authentication.  For most Subversion instances these are features that we will want to have.  We have many choices for our A/A solution and I have decided to integrate my example repository with a Microsoft Active Directory (AD) system running on Windows 2003.  This, I feel, is probably the most commonly desired scenario for enterprise shops although a non-AD based LDAP and Kerberos system may also be very popular.  We will start by address authentication via Kerberos in this article.

In addition to using Kerberos for secure authentication, we are also switching from using plain HTTP as our transport to HTTP over SSL so be aware that after applying the Apache configuration file here that you will need to access your Subversion directory with HTTPS rather than HTTP and that, unless otherwise configured, you will need to open your firewall both locally and remotely to allow port 443 traffic out instead of (or in addition to) port 80 traffic.

Installing Necessary Components

As with anything else in the Red Hat world, most of the heavy lifting is done by our friends at Red Hat engineering and we just need to leverage what they have already done for us.  We need to install the module for SSL transport and Kerberos authentication in Apache:

yum -y install mod_auth_kerb

This will automatically install the file /etc/httpd/conf.d/auth_kerb.conf which will take care of loading the Kerberos module into Apache and will provide a sample configuration if you want to learn more about Kerberos authentication in Apache.

Setting Up the Apache KeyTab File

Now we need to set up our Apache to Kerberos authentication table.  The Red Hat standard for this file is to be located at /etc/httpd/conf/keytab although you control its location through your Apache configuration.  We will not deviate from the standard here.

This file needs to contain

echo HTTP/[email protected] >> /etc/httpd/conf/keytab
chown apache.apache /etc/httpd/conf/keytab

Setting Access Control

The traditional examples will generally tell you to use the .htaccess file to manage your authentication mechanisms.  For most cases it is better to avoid the use of the .htaccess file and to switch to configuring these details within your <Location> section in your Apache configuration files.  This is better for performance reasons as well as for ease of security management.  Now you only need to worry about specifying your security information in a single location and Apache need not traverse the entire directory structure seeking out .htaccess files for each access attempt.

I use the file /etc/httpd/conf.d/subversion.conf for the configuration of my Subversion repository.  Here are its contents:

   <Location /svn>
     DAV svn
     SVNPath /var/projects/svn/
     AuthName "Active Directory Login"
     AuthType Kerberos
     Krb5Keytab /etc/httpd/conf/keytab
     KrbAuthRealm EXAMPLE.COM
     KrbMethodNegotiate Off
     KrbSaveCredentials off
     KrbVerifyKDC off
     Require valid-user
     SSLRequireSSL
   </Location>

Configuration of Kerberos

Kerberos is configured in Red Hat Linux in the /etc/krb5.conf file.  Obviously replace EXAMPLE.COM and ad.example.com with the name of your Domain and your KDC.  This file should have been created for you using almost exactly these settings by the RPM installer so there is very little here that needs to be changed.

[libdefaults]
 default_realm = EXAMPLE.COM
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 forwardable = yes

[realms]
 EXAMPLE.COM = {
  kdc = ad.example.com:88
 }

[domain_realm]
 example.com = EXAMPLE.COM
 .example.com = EXAMPLE.COM

[appdefaults]
 pam = {
   debug = false
   ticket_lifetime = 36000
   renew_lifetime = 36000
   forwardable = true
   krb4_convert = false
 }

Enable HTTPS Access Through Firewall

Use the Red Hat management tool to enable HTTPS connection through your host firewall.

system-config-securitylevel-tui

Restart Apache

Now, all that we need to do is to restart the web server to have it pick up the changes that we have made and voila, Kerberos authentication to Active Directory should be working.

/etc/init.d/httpd restart

Testing Your Connection

In order to test your connection you can use a web browser or use the Subversion command line client as below:

svn list https://localhost/svn/

Error Notes:

If you set KrbMethodNegotiate On then, in my experience, you will see Firefox work just fine but Internet Explorer (IE) and Chrome will fail with a 500 error.  In the logs I discovered the following entry:

gss_acquire_cred() failed: Unspecified GSS failure.  Minor code may provide more information (Unknown code krb5 213)

References:

Providing Active Directory Authentication via Kerberos Procol in Apache by Alex Yu, MVP, Microsoft Support

January 14, 2009: Dominica’s Shopping Day

It is pretty chilly here in the Hudson Valley this week but apparently what we are experiencing is nothing compared to what I have been hearing about from family back home or Twitter friends across the country.  Cold but not unreasonable.  It is times like this that I really appreciate living in downstate New York rather than Upstate.  I love Upstate but in January through March there is no comparison to the lower Hudson’s balmy climate.

We have been in desperate need of some shopping to be done so once I had a lull at work around lunch time Dominica decided to take off and spend about two hours running some errands.  We needed some basic stuff like some plastic file folder bins and file folders from Staples (which turned out to be on sale), some earplugs, new waste baskets and other miscellany such as that.  She decided to take Oreo with her so that he could visit PetsMart where he got his nails trimmed and his ears cleaned.  He doesn’t like getting his nails trimmed, but he is much more happy about having them do it than having us do it.  He does enjoy getting his ears cleaned.  They are so soft after a good cleaning.

Leaving me alone with Liesl for two hours would normally not be a big deal but she was screaming and very unhappy from the moment that Dominica walked out the door.  I had to wear earplugs the entire time as my head was swimming from the pain of my blown eardrums.  I am really scared that I am going to go deaf.  I really have to start wearing earplugs pretty much all of the time.

Liesl needed to be changed (really changed, like different outfits) immediately and needed to be fed, got sick and then I needed to be changed and a lot of laundry needed to be done.  It was a good thing that Oreo was not here since there was no way for me to have been taking care of him as well.  Liesl finally ate a bottle and was asking for another just as Dominica arrived home.

We had dinner and watched more of The Dick Van Dyke Show including watching the original pilot from 1958 with Carl Reiner playing the lead instead of Dick Van Dyck (the show was originally to be called “Head of the Family”.)  I love Carl Reiner but that pilot was awful.  It is amazing that the later show was ever made after seeing that.  It is also amazing how much better the show made in 1962 looked than the one from 1958.  A show from 1988 and 1992 look identical but there was a lot of advanced made in television filming at that point, I guess.  One was blurry and had no contrast and the other is an example of how good monochrome television could look.

After dinner, Dominica decided that Liesl was sleeping early and that this was going to be her one chance to run to the grocery store to do the week’s shopping.  So Liesl, Oreo and I stayed home alone while Dominica went out again.  Leaving me alone with Liesl twice in one day – this is a first, and for a total of over three hours!  That is just crazy.

Liesl slept pretty much the entire time that Dominica was gone but Oreo needed to be walked and given a lot of attention while she was out.  Once Oreo calmed down and settled in to chewing a dingo bone I fired up Fable 2 and spent some time wandering Knothole Island looking for Volume 2 of the history of Knothole Island which somehow I missed on my first pass through the quests.

It wasn’t until after Dominica got home that we finally found the final book and I acquired the Bibliophile achievement in Fable 2.  That is 25 additional gamer points.  Tomorrow I am hoping to get all of the items from the curiosity shop to complete every possible task in Knothole Island.  That should not take long at all.

My big project for today was wrapping up some Subversion via Apache with Active Directory via Kerberos integration.  It took some doing but I have it working really well now.  Using AD as an SSO solution works really well but setup is not always the easiest.  I am really glad to have that working now.

After Dominica was home and we located the final book we put on Hulu and I started carrying up paperwork from the basement that needed to be sorted through.  Dominica picked up those bins today so that we could sort through all of the paperwork that has amassed throughout the house and sort it into bins that can be stored someplace useful – like in the attic maybe.  Almost all of the paperwork is long-term storage “just in case” kind of stuff and nothing that we plan to ever need to go through again but hate to just throw away.

I sorted through several large piles of paperwork before calling it a night.  We are doing this now, with a high priority, because we realized that we have misplaced the American Express card.  It came in a stack of mail recently and we expect is lost in the piles and piles of paperwork all over the house.  So we are trying to dig it out.  There is so much, though, that it is going to be a challenge.