High IOWait on VMWare Server on Linux

In using VMWare Server running on Red Hat Enterprise Linux 5 (CentOS 5) I discovered a rather difficult problem.  My setup includes Red Hat Linux 5.2, Solaris 10 and Windows Server 2003 guests running on a Red Hat 5.2 host server all 64bit except for Windows running on AMD Opteron multicore processors on an HP Proliant DL145 G3.

The issue that I found was that the Windows guest was exhibiting serious performance issues.  The box would freeze regularly, networking would stop although pings continued but remote desktop (RDP) would be interrupted.  In the logs I consistently found symmpi errors in the System Event Log:

The device, \Device\Scsi\symmpi1, did not respond within the timeout period.

Because the issues were only exhibiting on Windows and not on Linux or Solaris guests I was convinced that the issue was Windows related.  I could see that the Linux host operating system was showing massive IOWait states (you can see this in top or with the iostat command from the sysstat package.)  I assumed that this was being caused by the Windows guest; it was not.

I turned off all three guest operating systems and noticed almost no drop in the IOWait levels, however if I turned of the VMWare Server process (/etc/init.d/vmware stop) the IOWait would drop almost instantly and return again as soon as I restart the process even without starting any virtual machine images.  Clearly the issue was with VMWare Server itself.

My first thought was to make sure that VMWare Server was up to date.  I have been running VMWare Server 1.0.7 and so downloaded and updated the very recent 1.0.8 update just to be sure that this issue was not addressed in that package.  It was not.  I am aware that the 2.0 series is available now but as this box is used a bit I am not interested yet in moving to the new series unless absolutely necessary.

Once I narrowed down that the issue was a problem with VMWare Server on Linux I was able to track down a solution.  Special thanks to Mr. Pointy for publishing the solution to this for Gutsy Gibson.  Red Hat and Ubuntu are sharing a problem in this case.

The issue is with memory configuration defaults with VMWare Server on this platform.  Very likely this will apply to Novell SUSE Linux, OpenSUSE, Fedora and others, but I have not tested it.  In the main VMWare Server configuration file (/etc/vmware/config) the following changes should be added:

prefvmx.useRecommendedLockedMemSize = “TRUE”
prefvmx.minVmMemPct = “100″

Then, in each of the individual virtual machine configuration files (*.vmx) you need to add:

sched.mem.pshare.enable = “FALSE”
mainMem.useNamedFile = “FALSE”
MemTrimRate = “0″
MemAllowAutoScaleDown = “FALSE”

These changes are taken directly from Mr. Pointy’s blog.  Once the changes are made you can restart VMWare Server (/etc/init.d/vmware restart) and the difference should be immediately visible.  Mr. Pointy posted his own sar results and here are mine.  You can clearly see the change in the %iowait column at 10:10pm when I restarted VMWare with the new configuration.  The numbers are low around 7:00pm because I had VMWare off much of that hour.

06:40:01 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
06:50:01 PM       all      0.16      0.00      1.77     43.15      0.00     54.92
07:00:01 PM       all      2.83      0.00      6.83      9.51      0.00     80.82
07:10:01 PM       all      0.10      0.00      1.38      4.93      0.00     93.59
07:20:01 PM       all      0.11      0.20      1.84     14.78      0.00     83.07
07:30:01 PM       all      0.10      0.00      2.08      8.84      0.00     88.98
07:40:02 PM       all      0.11      0.00      2.36     26.84      0.00     70.70
07:50:01 PM       all      0.11      0.00      2.32     28.54      0.00     69.04
08:00:01 PM       all      0.10      0.00      2.13     30.63      0.00     67.14
08:10:01 PM       all      0.10      0.00      2.06     22.74      0.00     75.10
08:20:01 PM       all      0.09      0.20      2.02     22.75      0.00     74.94
08:30:04 PM       all      0.09      0.00      2.21     23.22      0.00     74.48
08:40:01 PM       all      0.09      0.00      3.03     25.06      0.00     71.81
08:50:01 PM       all      0.09      0.00      3.09     27.21      0.00     69.61
09:00:01 PM       all      0.10      0.00      3.13     29.40      0.00     67.37
09:10:01 PM       all      0.09      0.00      3.11     25.56      0.00     71.23
09:20:01 PM       all      0.09      0.19      3.07     23.79      0.00     72.86
09:30:01 PM       all      0.09      0.00      2.98     21.50      0.00     75.43
09:40:01 PM       all      0.10      0.00      2.97     25.94      0.00     70.99
09:50:01 PM       all      0.10      0.00      3.28     32.70      0.00     63.93
10:00:01 PM       all      0.20      0.00      4.96     40.73      0.00     54.11
10:10:01 PM       all      0.69      0.00      8.57      1.23      0.00     89.50
10:20:01 PM       all      0.88      0.21      6.34      0.67      0.00     91.90
10:30:01 PM       all      0.81      0.00      6.04      0.26      0.00     92.89
10:40:01 PM       all      0.78      0.00      5.55      0.20      0.00     93.47
10:50:01 PM       all      0.77      0.00      5.47      0.07      0.00     93.69

After the change was complete I had no problem running i/o system intensive operations like disk compression, defragmentation, etc.

Original solution from: Mr. Pointy – Gutsy and VMWare Server – You’re In for Some Pain.

November 20, 2008: One More Day

1 Days to Baby Day! (39 Weeks and Six Days Pregnant)

“Sleep with the window open.” – Rule #285, 1001 Rules for My Unborn Son

One day to go.  It has felt like this day would not come for a very long time.  I decided that I really just needed to work from home tomorrow.  Everyone is expecting me to be home tomorrow anyway so it is no big deal.  I was just hoping to get in as much time in the office as possible before being home for a long stretch.  I really need a break from commuting and swimming though.  I am just worn out.

I slept in a bit this morning.  I really needed that.  I was awake around six thirty, but Oreo was just so snuggly.  I couldn’t bring myself to get out of bed for another hour.  He really misses me.  He got up immediately and came down to the basement and lay beside me while I worked.

At eight this morning the snow started.  Before it had started snowing I had been looking outside and thinking how “wintery” it looked.  This is our first snow in our new home.

I got Laborers for the Harvest set up today with a major WordPress code update, new theme and new plugins.  This is the first major site overhaul that we have done over there since the site first went live.

At noon, dad took Dominica down to the clinic downtown so that she could go to the final widwife appointment.  I can’t believe that we are down to the last one already.  One day short of nine months doesn’t seem very long.

Babies ‘R’ Us called while Dominica was out to let us know that our new furniture had arrived there.  The crib for the nursery and the glider-style rocking chair for our bedroom arrived.  Now we just have to figure out how to get it to the house.  It is some large stuff.

The word from the midwife is that everything looks to be fine but Dominica does not have any signs of approaching labor.  So the important thing, that the baby is healthy, is okay, but Dominica is very stressed and hoping to be able to deliver very soon.  She has an appointment to see the doctors on Monday for an ultrasound and something like a stress test in case she hasn’t gone into labor by then.

I was stuck in the basement pretty much all day working.  I’ve been working on a really tough issue with a VMWare Server instance that is producing some incredible drive activity that I am attempting to troubleshoot.  This is a particularly perplexing problem and is affecting a lot of stuff so I am desperate to get it fixed.  I will post a technical article about it once I get it figured out.

For dinner Dad and I drove out to the New City Diner in Yorktown Heights and got take out and brought food back for Dominica.  We watched a few episodes of the Mary Tyler Moore show and then Dominica went to bed around ten.  Dad put on Bones and I headed down to the basement to continue working.

I will be home tomorrow.  We are so close to the baby’s arrival that I should not be going into the office at this point.  Especially since I am able to get so much more work done when I don’t go into the office.  It is so hard to get a lot of work done when you commute for four hours.

Resizing VMWare Server Virtual Disk

Today I needed to resize a VMWare Virtual Disk (vmdk) for a Windows Server 2003 image running on a Red Hat Enterprise Linux 5 host using LVM to manage the physical, local disk space.  In my case, my logical volume was too small to accommodate the vmdk expansion and so I had to grow my logical volume before I could begin the VMWare portion of the work.

I must preface all of this, of course, by stating that you must make a complete backup of your virtual machine before doing something as invasive as this.  While this process is reasonably safe there is always the potential for disaster.  Take precautions.

The lvextend command is used to increase the size of the logical volume.  You can view your current logical volumes with lvdisplay.  I use the -L+ syntax as a safety measure to be sure that my drive is getting larger and not shrinking accidentally due to a typo.  In this example I am expanding the /dev/VolGroup00/lvxen logical volume by an additional 80GB.

lvextend -L+80G /dev/VolGroup00/lvvmware

This first step can be completed while the virtual machine is still running.  It will happily extend your available space in the background.  Our next step, however, requires that you power down your virtual machine before continuing.

Now that we have created space on our logical volume we need to expand the Linux local filesystem before we can expand the virtual filesystem running on top of it.  Assuming that we are using the current standard Ext3 this is very simple:

umount /dev/VolGroup00/lvvmware

e2fsck -f /dev/VolGroup00/lvvmware

resize2fs /dev/VolGroup00/lvvmware

mount /vmware/

Obviously for my purposes I use a /vmware directory structure for holding all of my disk images.  You will need to adjust as needed for your own setup.  /var/vmware is another common option.

Now we just enlarge the virtual disk itself.  We will do this through the vmware-vdiskmanager command.  You will need to execute this command on your vmdk and not your flat-vmdk even though this seems counter-intuitive when looking at your directory structure.

vmware-vdiskmanager -x 22GB ph-w2k3-ad.vmdk

This concludes the easy part.  Now you have plenty of logical disk space available for Windows but in order to expand the System drive of Windows you will need to use a third party tool.  Windows Server 2003 is unable to make partition changes that affect the running system.

If you are like me, you will want to fire up your virtual machine just to make sure that everything is okay after the disk change, but you will need to turn it off again before we make changes to the partition table.

There are many tools that can be used for this task but I decided to use GParted, which is available as a live CD which you can download for free.  For the version that I used, I just cd’d into /tmp and used this command to get my copy of GParted’s bootable CD ISO file.

wget http://downloads.sourceforge.net/gparted/gparted-live-0.3.9-4.iso?modtime=1222872844&big_mirror=0

Using your VMWare Server Console (or through the command line) you will need to set your Windows Server image to boot from the GParted ISO which you just downloaded.  Then go ahead and “start this virtual machine.”

You will likely need to hit “Esc” as soon as the virtual machine starts so that you can select to boot from CD.  I keep my Virtual BIOS set to boot directly to the hard drive under normal circumstances because it is faster and I don’t want to accidentally boot to some CD media unless I really, really mean it.

Once GParted starts you will be given a boot menu.  The default option works fine in more cases and worked fine for me.  You will need to select your keyboard layout and then you will be taken to GParted’s graphical partition manager screen.

Once in the GParted Partition Manager you should see the current partition that you had before we started, in my case called /dev/sda1 and marked as being an NTFS file system.  Mine also shows the “unallocated” partition space into which I will be expanding my /dev/sda1 partition.

Start by selection the partition which you are seeking to resize (sda1 for me) and then select “Resize/Move”.  This will open the Resize/Move window.  Do not alter the first numer – “Free Space Preceding”, this is for “moving” your partition.  You only want to alter the second number – “New Size.”  If you are doing like me and have created some empty space specifically for this purpose then you will simply set this number to the “Maximum Size” as displayed in the window.  Then select “Resize/Move” to continue.

Once you have completed that step you can visually confirm that the disk now looks the way that you want it to look.  If you look at the bottom of the window you will see that there is “1 operation pending.”  If everything looks alright go ahead and click “Apply” to commit your changes and to resize your partition.

Once the resizing completes you are safe to reboot your virtual machine into Windows again.  Double click the “Exit” button on the GParted desktop.  Reboot should already be selected so just choose OK to continue.

When Windows starts it will detect the drive configuration change and force a disk consistency check.  Allow it to run through this process and when it completes the system will restart automatically.  Once Windows restarts you should see that your drive has been resized.

November 19, 2008: New Basement Furniture

2 Days to Baby Day! (39 Weeks and Five Days Pregnant)

One Laptop Per Child XO Linux laptop is available again, this year from Amazon.  The program is just like it was last your.  For $199 you can donate an XO to a child somewhere in the developing world or, for $399, you can donate one and get one for yourself.  Makes a great gift for the techie in your life (this thing is just plain cool) or for a child.  If you are a technology collector like me this is a must have for your collection, and the first $199 is tax deductable and buying a laptop for a child is just plain awesome.

I forgot to set my alarm last night but fortunately woke up on my own before five thirty this morning.  That could have been very bad.  I got up, did some reading, deployed some packages for the office, got ready and Dominica drove me down to the train station.  We ended up being just a minute or two behind the train and ended up missing the 6:42.  It was really cold out this morning so Dominica waited at the train station with me and we sat in the car until I caught the 7:08 train instead.  That was crappy.

Katie turned me on to having my Twitter feed go directly into FaceBook so that my FaceBook Status is updated automatically by my tweets on Twitter.  Now I only have to update Twitter and the FaceBook users will see me automatically.  That is great since I have that handy Twitter command line client and, as of today, my office has opened up the firewall to let us access Twitter but not FaceBook.  Now I am only updating one microblogging location instead of two.  Much better.

Danielle got hooked up with Twitter today as well.

Dan and I went swimming over lunch today.  Katie was not able to go.  I did 1,500 yards today.  Not a major workout but it was a good hour of cardio which is what I need the most.

Our new basement theatre seating arrived today from Bob’s Discount Furniture.  The new chest that is going to our bedroom is back ordered and is expected to arrive on Friday.  Dad loves the new seating in the basement and is thinking about possibly sleeping down there tonight rather than in the living room.

Today was a brutal day for the stock markets.  The Dow Jones Industrial Average fell below the 8,000 mark as the US considers the very real possibility that it is going to let the big three American automakers just fail without any hope of a bailout.  Overall, Americans seem very favourable to watching the automakers fail.  No one wants their taxes to go up to pay for bloated, paperwork heavy, inefficient American manufacturing to be saved when they have had decades to figure out how to operate like lean, competitive companies.  Citi shares closed today at just $6.25.  This is a crazy market.

Consumer pricing was released today and we have had the biggest one month downturn in consumer prices since 1947.  So while the market is doing poorly and no one is making any money (whatever that means) the price of things is going down, way down.  Normal goods are getting cheaper and fuel is practically free compared to where it was just six months ago.

I was stuck in the office until after six thirty.  I made the 7:18 train out of Grand Central so Dominica was able to pick me up at 8:13.  I’ve been reading “Spellfire” by Ed Greenwood while on the train.  I’ve been reading this book for months (read: started it months ago and haven’t gotten back to it.)  So I am trying to finish it now.  Ramona will be very pleased that I am reading a fiction book for a change.  Although it really isn’t a very good one.

For dinner Dominica made Quorn fake-chicken with gruyere and goat cheese, mixed vegetables and cornbread stuffing.  It was delicious.

After dinner I went down to the basement to work on some projects.  There is always so much to be done.  I have no spare time now that I am commuting so much.  I feel like I get home just barely in time to get to bed.  My readers will notice that I have all but stopped watching any shows at all.  I am not getting a lot of sleep either.  I am looking forward to the more relaxed schedule of having a baby!  😉

Oreo has really been missing me.  He comes down to the basement to lay beside me or sits with me in the chair in the living room.  It has been a long time since he has been so snuggly.

I worked until well after eleven.  I spent the entire night trying to get connectivity to a VMWare Server.  VMWare has made even their simplest packages so hard to find on their website and so large and cumbersome to download that it took all evening to get anything at all working.  In the end I had to abandon even the simplest project for the evening as there was enough time to get one working install of the VMWare Client before it was well past time to go to bed.  I am going to be very tired tomorrow.

I may have to stay home simply because I am getting too tired with everything that is going on and all of the commuting.  At the current rate, Dominica is not going to be going into labor anytime soon and having me get worn out is not a good way to be prepared for things.

November 18, 2008: Big Swimming Day

3 Days to Baby Day! (39 Weeks and Four Days Pregnant)

On the ride home on the train last night I finished reading “Day of the Barbarians”.  This train riding is really giving me a chance to read a lot more than I was reading before.  That is my third book since Friday morning!

We grabbed dinner at Burger King on the way home from the train station.  There was still no power when dad and Dominica left to come to get me so there was no chance of cooking at home.  Once we got back, a little after eight, the lights were back on.  That was very good as Oreo is just about out of food and we needed to cook for him tonight.

Dominica taught me how to make Oreo’s special chicken stew (mostly chicken, sweet potatoes, celery, rosemary, garlic, carrots, peas, zucchini, barley, oatmeal and kelp.  The chicken is high grade skinless, boneless, of course.)  That took about half an hour.  I will be making his stew soon since Dominica will be busy with the baby.

My evening was very short.  I read a few chapters in “Agile Web Development with Rails” and then headed off to bed at twenty after ten.  Not much of an evening.

I woke up, on my own, at five thirty this morning.  Not too bad.  Twelve minutes ahead of the alarm.  That put me in good shape to grab the earlier train which I have been trying to grab for a week now.  I even managed to log into the office, get caught up with my mail and make sure that no one needed me before heading out.  Five thirty is, apparently, a good time to get up if I can do it.  Considering that I often don’t get home until between eight thirty and ten thirty it means that there is no way for me to even approach eight hours of sleep let alone get sleep and eat and do anything else whatsoever.

I managed to make the 6:42 train, which is harder to catch because the road closers earlier as there is an Amtrak train that comes through just before the Metro North line, and was at my desk in the office at a quarter after eight.  On the walk down Wall Street there was snow.  This is the first falling snow that I have seen this season.

Dad and Dominica made a run out to get wall-anchor parts today so that the basement bookshelves can be anchored to the wall so that we can begin to put books on them.  The books have to be up off of the floor tonight so that tomorrow, just after lunch, the new furniture can be delivered.  We have the new dresser for our bedroom coming as well as the multiplex theatre seating for the basement set to be delivered.

I forgot to mention that yesterday a new washer was delivered to our house.  We didn’t order it, of course, so it was a bit of a surprise.  Luckily Dad and Dominica were both home and were able to explain that it definitely was not supposed to be for us.  It turned out that it was for the people who owned the house before us.  We are guessing that they ordered it for their new house but somehow gave them our address.

This morning was pretty slow at the office.  It would figure that on a day when I managed to get in nice and early that no one was waiting for me at all.

Talked to Josh around lunch today, he said that he had six inches of snow this morning.  Ick.

No Dan today so I had to go to the gym by myself.  I managed to do 1,450 yard before Katie arrived and did 2,250 total by the end of my workout.  That was a really good workout for me.  Then we went to Flavors to get soup and salad.  Very healthy day for Scott.

The afternoon was not all that busy either.  I decided to not stay late today and managed to catch the 6:15 train out of Grand Central Terminal.  On the ride home I finished reading “The Business Analyst as Strategist” which I had started this morning.  Boy am I ever reading quickly.  When I was thinking about the impact of commuting by train like this it never occurred to me just how little time I normally actually had allotted to reading regular paper books and how much additional time I would now have.  I’ve read four books in the last five days!  Tomorrow I am taking along a fiction book which I have been reading slowly for some time.  I should be able to polish it off in no time.

Dominica made meatball subs tonight.  Boy are they good.  I am so glad that she figured out this new recipe.  It is delicious and not too complicated for her to make.  I figured out to get some BBQ potato chips to crush up and put into the sandwich.  It tastes just like a sandwich from Hot Truck in Ithaca.  I added a little mayo although we have no hot sauce in the house right now.  Pretty much, though, it tastes just like an MBC with Kate sauce from Hot Truck.  Awesome.  One of the few pre-vegetarian foods that I really missed and this tastes just as good as the original.  Can you believe that I haven’t eaten at hot truck in over seven years?

The evening was very short.  I helped dad a tiny bit as he worked on setting up the bookshelves in the basement.  He managed to repair our glass-doored bookshelf that had become unsettled during shipping from Newark and has all three of our basement shelving units in place and ready to be bolted to the wall tomorrow for safety.  The black, Ikea shelves look awesome in the basement and we are thinking that we will get three more exactly the same (two large shelving units and one, small corner unit that is half the width) to put behind Dominica’s desk so that they match.  We will probably expand the bookshelves by one more unit behind my desk as there are so many books to put on there that we have no idea where they will all go.  There is the low, teak bookshelf that will go upstairs as well for cookbooks.  That is the bookshelf on which Mr. Humphries, our little hamster, lived on several years ago in Geneseo.

My main job tonight was to move all of the stacks of books that were in the basement from the big, open area to the limited space behind the desks.  The books were in the way of where the furniture will be going tomorrow afternoon.  That will be exciting to come home to tomorrow night.

I managed to head to bed at eleven.  Not as early as I would like but if I go to bed much earlier I might as well not even bother to come home.  My evenings are extremely short.  I will be on Wall Street tomorrow.