The obvious steps to take include running NTP to control the local clock.  This, however, only works when the clock skews very slowly.  In my case, as in many, the clock drifts too rapidly for NTP to handle.  So we need another solution.  VMWare recommends installing VMWare Tools on the guest operating system and subsequently adding the following to your VMX configuration file:

tools.syncTime = true

This does not always work either.  You should also try changing you guest system clock type.  Most suggestions include adding clock=pit to the kernel options.  None of this worked for me.  I had to resort to a heavyhanded NTP trick – putting manual ntpdate updates into cron.  In my case, I set it to update every two minutes.  The clock still drifts heavily within the two minute interval but for me it is an acceptable amount.  You should adjust the update interval for your own needs.  Every five minutes may easily be enough but more frequently might be necessary.

Using crontab -e under the root user, add the following to your crontab:

*/2 * * * * /usr/sbin/ntpdate 0.centos.pool.ntp.org

For those unfamiliar with the use of */2 in the first column of this cron entry, that designates to run every two minutes.  For every five minutes you would use */5.  Remember that it takes a few minutes before cron changes take effect.  So don’t look for the time to begin syncing for a few minutes.

For me, this worked perfectly.  Ntpdate is not subject to the skew and offset issues that ntpd is.  So we don’t have to worry about the skew becoming too great and the sync process stopping.

If anyone has additional information on syncing Linux in this situation, please comment.  Keep in mind that this is for Red Hat Linux and the kernel with RHEL5 is 2.6.18 which does not include the latest kernel time updates that may be found in some distributions like Ubuntu.  Recent releases of Ubuntu likely do not suffer this issue as I expect OpenSuse 11.1 or the latest Fedora would not either.

The issue that I found was that the Windows guest was exhibiting serious performance issues.  The box would freeze regularly, networking would stop although pings continued but remote desktop (RDP) would be interrupted.  In the logs I consistently found symmpi errors in the System Event Log:

The device, \Device\Scsi\symmpi1, did not respond within the timeout period.

Because the issues were only exhibiting on Windows and not on Linux or Solaris guests I was convinced that the issue was Windows related.  I could see that the Linux host operating system was showing massive IOWait states (you can see this in top or with the iostat command from the sysstat package.)  I assumed that this was being caused by the Windows guest; it was not.

I turned off all three guest operating systems and noticed almost no drop in the IOWait levels, however if I turned of the VMWare Server process (/etc/init.d/vmware stop) the IOWait would drop almost instantly and return again as soon as I restart the process even without starting any virtual machine images.  Clearly the issue was with VMWare Server itself.

My first thought was to make sure that VMWare Server was up to date.  I have been running VMWare Server 1.0.7 and so downloaded and updated the very recent 1.0.8 update just to be sure that this issue was not addressed in that package.  It was not.  I am aware that the 2.0 series is available now but as this box is used a bit I am not interested yet in moving to the new series unless absolutely necessary.

Once I narrowed down that the issue was a problem with VMWare Server on Linux I was able to track down a solution.  Special thanks to Mr. Pointy for publishing the solution to this for Gutsy Gibson.  Red Hat and Ubuntu are sharing a problem in this case.

The issue is with memory configuration defaults with VMWare Server on this platform.  Very likely this will apply to Novell SUSE Linux, OpenSUSE, Fedora and others, but I have not tested it.  In the main VMWare Server configuration file (/etc/vmware/config) the following changes should be added:

prefvmx.useRecommendedLockedMemSize = “TRUE”
prefvmx.minVmMemPct = “100″

Then, in each of the individual virtual machine configuration files (*.vmx) you need to add:

sched.mem.pshare.enable = “FALSE”
mainMem.useNamedFile = “FALSE”
MemTrimRate = “0″
MemAllowAutoScaleDown = “FALSE”

These changes are taken directly from Mr. Pointy’s blog.  Once the changes are made you can restart VMWare Server (/etc/init.d/vmware restart) and the difference should be immediately visible.  Mr. Pointy posted his own sar results and here are mine.  You can clearly see the change in the %iowait column at 10:10pm when I restarted VMWare with the new configuration.  The numbers are low around 7:00pm because I had VMWare off much of that hour.

06:40:01 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
06:50:01 PM       all      0.16      0.00      1.77     43.15      0.00     54.92
07:00:01 PM       all      2.83      0.00      6.83      9.51      0.00     80.82
07:10:01 PM       all      0.10      0.00      1.38      4.93      0.00     93.59
07:20:01 PM       all      0.11      0.20      1.84     14.78      0.00     83.07
07:30:01 PM       all      0.10      0.00      2.08      8.84      0.00     88.98
07:40:02 PM       all      0.11      0.00      2.36     26.84      0.00     70.70
07:50:01 PM       all      0.11      0.00      2.32     28.54      0.00     69.04
08:00:01 PM       all      0.10      0.00      2.13     30.63      0.00     67.14
08:10:01 PM       all      0.10      0.00      2.06     22.74      0.00     75.10
08:20:01 PM       all      0.09      0.20      2.02     22.75      0.00     74.94
08:30:04 PM       all      0.09      0.00      2.21     23.22      0.00     74.48
08:40:01 PM       all      0.09      0.00      3.03     25.06      0.00     71.81
08:50:01 PM       all      0.09      0.00      3.09     27.21      0.00     69.61
09:00:01 PM       all      0.10      0.00      3.13     29.40      0.00     67.37
09:10:01 PM       all      0.09      0.00      3.11     25.56      0.00     71.23
09:20:01 PM       all      0.09      0.19      3.07     23.79      0.00     72.86
09:30:01 PM       all      0.09      0.00      2.98     21.50      0.00     75.43
09:40:01 PM       all      0.10      0.00      2.97     25.94      0.00     70.99
09:50:01 PM       all      0.10      0.00      3.28     32.70      0.00     63.93
10:00:01 PM       all      0.20      0.00      4.96     40.73      0.00     54.11
10:10:01 PM       all      0.69      0.00      8.57      1.23      0.00     89.50
10:20:01 PM       all      0.88      0.21      6.34      0.67      0.00     91.90
10:30:01 PM       all      0.81      0.00      6.04      0.26      0.00     92.89
10:40:01 PM       all      0.78      0.00      5.55      0.20      0.00     93.47
10:50:01 PM       all      0.77      0.00      5.47      0.07      0.00     93.69

After the change was complete I had no problem running i/o system intensive operations like disk compression, defragmentation, etc.

Original solution from: Mr. Pointy – Gutsy and VMWare Server – You’re In for Some Pain.

I must preface all of this, of course, by stating that you must make a complete backup of your virtual machine before doing something as invasive as this.  While this process is reasonably safe there is always the potential for disaster.  Take precautions.

The lvextend command is used to increase the size of the logical volume.  You can view your current logical volumes with lvdisplay.  I use the -L+ syntax as a safety measure to be sure that my drive is getting larger and not shrinking accidentally due to a typo.  In this example I am expanding the /dev/VolGroup00/lvxen logical volume by an additional 80GB.

lvextend -L+80G /dev/VolGroup00/lvvmware

This first step can be completed while the virtual machine is still running.  It will happily extend your available space in the background.  Our next step, however, requires that you power down your virtual machine before continuing.

Now that we have created space on our logical volume we need to expand the Linux local filesystem before we can expand the virtual filesystem running on top of it.  Assuming that we are using the current standard Ext3 this is very simple:

umount /dev/VolGroup00/lvvmware

e2fsck -f /dev/VolGroup00/lvvmware

resize2fs /dev/VolGroup00/lvvmware

mount /vmware/

Obviously for my purposes I use a /vmware directory structure for holding all of my disk images.  You will need to adjust as needed for your own setup.  /var/vmware is another common option.

Now we just enlarge the virtual disk itself.  We will do this through the vmware-vdiskmanager command.  You will need to execute this command on your vmdk and not your flat-vmdk even though this seems counter-intuitive when looking at your directory structure.

vmware-vdiskmanager -x 22GB ph-w2k3-ad.vmdk

This concludes the easy part.  Now you have plenty of logical disk space available for Windows but in order to expand the System drive of Windows you will need to use a third party tool.  Windows Server 2003 is unable to make partition changes that affect the running system.

If you are like me, you will want to fire up your virtual machine just to make sure that everything is okay after the disk change, but you will need to turn it off again before we make changes to the partition table.

There are many tools that can be used for this task but I decided to use GParted, which is available as a live CD which you can download for free.  For the version that I used, I just cd’d into /tmp and used this command to get my copy of GParted’s bootable CD ISO file.

wget http://downloads.sourceforge.net/gparted/gparted-live-0.3.9-4.iso?modtime=1222872844&big_mirror=0

Using your VMWare Server Console (or through the command line) you will need to set your Windows Server image to boot from the GParted ISO which you just downloaded.  Then go ahead and “start this virtual machine.”

You will likely need to hit “Esc” as soon as the virtual machine starts so that you can select to boot from CD.  I keep my Virtual BIOS set to boot directly to the hard drive under normal circumstances because it is faster and I don’t want to accidentally boot to some CD media unless I really, really mean it.

Once GParted starts you will be given a boot menu.  The default option works fine in more cases and worked fine for me.  You will need to select your keyboard layout and then you will be taken to GParted’s graphical partition manager screen.

Once in the GParted Partition Manager you should see the current partition that you had before we started, in my case called /dev/sda1 and marked as being an NTFS file system.  Mine also shows the “unallocated” partition space into which I will be expanding my /dev/sda1 partition.

Start by selection the partition which you are seeking to resize (sda1 for me) and then select “Resize/Move”.  This will open the Resize/Move window.  Do not alter the first numer – “Free Space Preceding”, this is for “moving” your partition.  You only want to alter the second number – “New Size.”  If you are doing like me and have created some empty space specifically for this purpose then you will simply set this number to the “Maximum Size” as displayed in the window.  Then select “Resize/Move” to continue.

Once you have completed that step you can visually confirm that the disk now looks the way that you want it to look.  If you look at the bottom of the window you will see that there is “1 operation pending.”  If everything looks alright go ahead and click “Apply” to commit your changes and to resize your partition.

Once the resizing completes you are safe to reboot your virtual machine into Windows again.  Double click the “Exit” button on the GParted desktop.  Reboot should already be selected so just choose OK to continue.

When Windows starts it will detect the drive configuration change and force a disk consistency check.  Allow it to run through this process and when it completes the system will restart automatically.  Once Windows restarts you should see that your drive has been resized.