I spend far too much time in meetings these days.  But I try to keep my Friday afternoons free of meetings – and often use this time to update my various servers and try out new configurations.  Today it is time to rebuild some of my storage:

Specifically I am throwing a bunch of disks into one of my servers and setting up a storage space to play around with (you can learn more about spaces here: http://channel9.msdn.com/events/BUILD/BUILD2011/SAC-474T).

These days I find that the most common cause of performance issues in my virtualized environments is the storage subsystem.  So I try to build systems with as much parallelism in the storage as possible – as I will be running dozens of virtual machines at the same time and a single disk just will not cut it for performance.

Cheers,
Ben

The last couple of days have been very strange and stressful for me.  The reason for this is that my home town – Brisbane, Queensland, Australia – is currently being flooded.  Thankfully all of my family is safe at this point in time – but it has been a very tense process as we have been following the news remotely and periodically trying to get in contact with different family members to find out what is happening.

While everyone is safe – my fathers business is not.  My father (Wayne Armstrong – though we all call him Joe) is joint-partner in a software development company called Bacchus Management Systems.  Two days ago the working day started normally for them, but they were soon notified that they had ~3 hours to evacuate the building before power would be cut ahead of incoming flood waters.

To put some further perspective on this – many areas of Brisbane saw flood waters arrive at a rate of over 1 meter an hour (1 foot every 20 minutes – for my American readers).

The people at Bacchus Management Systems tried to save as much as they could – but had to give up when the water reached the ground floor of the office building.  At which stage the water level was chest deep in the road that they needed to leave on.

Shortly after this I had the following online chat with my father:

Well, that is certainly a new selling point for virtualization!  Less hardware means a higher chance of escape from disaster.

But in all seriousness – while my family is safe, there are now many people in Queensland who are without a home.  Further more there will be many businesses (like my fathers) who will have significant losses.  As such I would like to make an unusual appeal for my blog:

If you are someone who is inclined to charity in these kinds of events – please visit http://www.qld.gov.au/floods/donate.html to see how you can donate to the recovery.

NOTE – MICROSOFT EMPLOYEES: I have learnt that Microsoft is matching employee contributions for this.  If you want to – please email me directly and I will let you know the process to ensure that your donation is matched.

Cheers,
Ben

P.S. One final strange thing about this whole event for me – was that I first received word that the floods were hitting Brisbane as I was practicing for a presentation I was about to give about how Hyper-V can be used for disaster recovery.

After a relaxing December on vacation I am now back at work – and ready to start blogging.  Or, at least, I thought I would be ready to start blogging.  Instead – I found that I needed to do a spot of cleaning first.

You see – somehow, in my absence, a lot of snow had gotten into my office:

…

Actually – it wasn’t snow (I am sure that came as a surprise to you) but a whole heap of shredded Styrofoam thoughtfully placed there by a colleague of mine.  At first I thought it looked quite pretty – and was intending to leave my office like this for a couple of days.  However – there were some issues:

1. The Styrofoam resulted in a massive build up of static electricity in my office.  There was actually a sizable amount of Styrofoam stuck to the walls – with nothing but static electricity.  Just walking into the office cause all of the hair on my arms and legs to stand on end.  Needless to say, I was terrified about the thought of turning a computer on (also notice that some of my computers do not have their cases on).
2. As a result of the static – whenever I left my office, I did so with a large covering of Styrofoam on my body.  I am sure that by the end of the week most of the Styrofoam would have naturally migrated from my office and out into the wilds of Microsoft.
3. Finally – as a result of the clingy Styrofoam, it was very easy to track me down this morning.  You just had to start at my office and follow the trails of Styrofoam down the hallways.

So, instead of writing something technical this afternoon – I spent some time vacuuming up my office (now with 90% less Styrofoam in it).  I will try to have something better to say tomorrow

Cheers,
Ben

For years now – whenever I have taken some time off from work I have gone through this process:

• A few weeks before my vacation: “I’m going to get blog posts pre-written for every day of my vacation!  No one will even know that I am gone!”
• One week before my vacation: “I’m going to get a couple of blog posts written – say, 1 or 2 a week – and people should not notice too much”
• One day before my vacation: “Well, maybe I will write some blog posts while I am on vacation”

To date – I do not think I have every posted a blog post while on vacation.  Strangely enough – it seems that once I am no longer at work it does not take very long at all for me to remember that I like spending time with my family & playing computer games more than I like to do work!

So with that in mind – I am taking some time off – and will not be posting here.  My vacation finishes at the end of December – so I will see you all in the new year.

Cheers,
Ben

Last week I had one of the hard disks in my Hyper-V server fail.  Having had to deal with this a couple of times over the last few years, I had little trouble working through the process of fixing my system, but I thought I would write up the details of my experience so that others can benefit.

It all started when my previously happy Windows Home Server started reporting a large number of file conflicts:

Having seen this before – the first thing I did was open the Hyper-V management console and connect to the Windows Home Server directly.  Logging into the virtual machine directly revealed that Windows was reporting a large number of write delay failures.

Write delay failures inside a virtual machine are almost always an indication of an underlying hardware problem with storage.  Unfortunately the virtual machine has no insight into what is actually going wrong with the physical disk – all it knows is that it tried to write to the disk and it failed. 

From here I went to the management operating system and bought up the event log.  The Windows system event log contained exactly the information I was looking for:

Clearly something was wrong with hard disk 2.  I shut down the Windows Home Server virtual machine (which was the only virtual machine in my system that was using this disk) and then used disk manager in the management operating system to take hard disk 2 offline:

I then updated the Windows Home Server virtual machine to remove the virtual hard disk that was stored here.  When I started the virtual machine back up – it was happy again (except for complaining about a missing hard disk).  Thankfully most of my data was configured for replication inside Windows Home Server.  Unfortunately, I had just disabled file replication on one of my Windows Home Server shares about one week earlier because I was running low on space – but that too was easy to address.

With the virtual machine up and running I then bought the problematic disk back online in the management operating system.  I used disk manager to connect the missing virtual hard disk to the management operating system and attempted to copy the missing files off of the virtual hard disk back onto the now running Windows Home Server.

This took a bit of work – as the hard disk was clearly on its last legs, and would regularly fill my event log with error messages and cause the virtual hard disk to get disconnected from the management operating system.  But each time I was able to reconnect the virtual hard disk and start the copy again.  After 3 or 4 attempts I was able to get all of the data off the disk.

Finally I used the Intel Storage Management tool to figure out the serial number for the faulty hard disk:

Then I shutdown the physical computer and removed the disk in question.

At the end of the day I had the system back up and running happily with no data lost.  I cannot complain about that!

Cheers,
Ben

Over the last couple of weeks I have been playing around with some new (for me) server technology.  This has been a lot of fun – and I have been learning a lot.  However, a couple of days ago things started to go wrong.

Specifically I started seeing all sorts of strange problems with my Hyper-V server.  Virtual machines started going offline randomly.  The whole system would go unresponsive for a couple of minutes and then come back.  Once or twice the management operating system stopped responding, while virtual machines continued to respond over the network.

This was concerning to say the least.

However, as I have been busy this week I did not have time to investigate properly.  After three nights dealing with this behavior, I was starting to wonder if I had some failing hardware, and was thinking about what I would need to do over the weekend to track down the cause of the problem.

Then - as I was driving home yesterday – I suddenly realized what the problem was.  Last weekend I had setup a login script for my domain that would install antivirus automatically on any domain-joined computer that did not have antivirus on it already.  When I got home I checked – and there it was.  I setup exclusions for Hyper-V and my server has been rock solid ever since.

Now, why am I feeling quite foolish to recount this story?  Apart from the obvious fact that it took me days to figure out what the problem was – I have already blogged about this exact problem.  Twice.

Sigh…

Cheers,
Ben

Next week I will be hanging out in New Orleans.  As well as doing my share of booth duty – and walking around talking to people about virtualization – I will be presenting on Dynamic Memory.  Here are the details:

Code	Session	Room	Day	Time
VIR304	Hyper-V and Dynamic Memory in Depth	Rm 295	Tuesday, June 8	1:30 PM - 2:45 PM
	Dynamic memory is a new feature of Hyper-V coming to Windows Server 2008 R2 SP1. Come to learn how Dynamic Memory enables Hyper-V to more efficiently utilize system resource, and how this can benefit your environment. Also learn how Dynamic Memory works under the covers and what you should be preparing for with the release of Windows Server 2008 R2 SP1.

See you there!

Cheers,
Ben

After a bit of turbulence, it looks like we are up and running on the new blog engine.  I am really happy about this – as it makes it a lot easier for me to customize the look and feel of the blog without having to hack things up too much.

For the time being – things should look pretty much the same as they use to – but there are some changes that I should point out:

• To comment on a blog post – you need to click on the post title, then you will see the comment section at the bottom
• To access files that are attached to a post – you also need to click on the post title
• If you look closely at the top of my blog posts you will see some new options:
  
  
  
  The two icons on the left let you choose between viewing full posts on my main page, or just viewing excerpts.  “Most Recent”, “Most Views” and “Most Comments” allow you to change the way posts are sorted.

I have also used the blog engine change as an opportunity to enable comments on all my posts (comments used to get disabled after 90 days).  I have done this because readers of my blog seem to be genuinely nice people who help each other out in the comments

Finally – we are now using Windows Live ID to sign in for the blogs, but you can also make anonymous comments if you want to.

Cheers,

Ben

A while ago I talked about how I use Hyper-V in my house, and more recently I talked about trying to backup my Hyper-V server.  This discussion of backup was actually in preparation for a significant rebuild of my home Hyper-V server, that I undertook a week and a half ago.  There were several goals that I had with this rebuild:

• Upgrade the system disk from an old stand alone disk to a newer RAID1 disk
• Add a new disk to my Windows Home Server virtual machine
• Pull my Domain Controller / DNS / DHCP server out of my parent partition and put it in a virtual machine

The challenges that I had with this process included:

• I did not have a second equivalent server handy to just migrate to.  Instead my goal was to utilize the machines sitting around my house as temporary staging locations while I did the rebuild (in fact I used my desktop as a file server and ran Hyper-V on my Mac Mini to act as my backup Hyper-V server).
• If this server is down – Internet is down in my house.  Any father / husband out there will tell you that you do not want to be responsible for breaking the household internet connection.
• My Windows Home Server virtual machine contains all of my families digital memories.  8 years of kid photos, videos of my children’s first steps, etc…  Losing this data is simply not an option.

It took over a week of planning and research but I ended off successfully rebuilding the system in the space of a weekend.  The process that I used looked something like this:

• Identify non-critical server virtual machines, shut them down and export them to a file share (on my desktop)
• Use SCVMM to move critical server virtual machines to Hyper-V on my Mac Mini
• Perform a configuration-only export of my Windows Home Server
• Do a bare metal backup of my physical computer
• Restore the bare metal backup into a virtual machine on my Mac Mini
• Shutdown my physical computer
• Get my Domain Controller / DNS / DHCP server running on the Mac Mini
• Swap in the new disks into the physical computer
• Install Windows and Hyper-V on the new disks
• Move all virtual machines back to the physical computer

To pull this off I used pretty much every trick that I know – so I thought it would be good to do a blog series on the process.

Before I get going – there are a few questions that I have been asked that I would like to answer upfront:

• Why are you using RAID1?!?  RAID10 would be much faster!
  
  To this I have two answers.  The first one is that I have a personal preference for minimizing the number of disks in my system (details that I prefer to not go into right now).  The second one is that you do not realize how bad the performance of my old system disk was.  I did some benchmarking on my old and new disk configuration to highlight the difference here:
  
  
  
• Why are you running your domain controller in a virtual machine?
  
  Read this post for my thoughts on this: http://blogs.msdn.com/virtual_pc_guy/archive/2008/11/24/the-domain-controller-dilemma.aspx
  
• Why don’t you just get some nice high end hardware?
  
  I know of people who do this, but I have many other things that I like to spend my money on.  Plus there is some level of satisfaction in getting results on a shoestring budget :-)

Cheers,
Ben

A while ago I talked about how I use Hyper-V in my house.  One of the problems that I identified with my current setup was that I had most of my virtual machines (except for the Windows Home Server) running on the same disk as the system disk for the management operating system.

Apart from being a bad practice in general – this has always concerned me as that disk represents a pretty large single point of failure in my server (if that disk fails I will lose my domain controller, FTP server, SCVMM server, SCOM server, MED-V server and WDS server).

Recently I also discovered that the disk in question is the oldest (and slowest) disk in the system – and this is causing performance issues for all of the virtual machines running off of it. 

Given all of this I decided to shuffle some disks out of other systems in my house and setup a higher performance two disk mirror for my system disk.  This would at least address the issues of performance and resiliency to disk failure.  The problem I faced was how to transfer my current system disk to a new RAID configuration.

After some failed attempts at using various cloning programs out there – it struck me that this was an ideal use of our backup technology.  I would just backup the current system disk – and restore it to the new physical disk.

As this was just going to be a “once off” backup – I did not want to spend the time to setup a full enterprise backup solution (like DPM) but just wanted to use Windows Server Backup.

I knew that Windows Server Backup does not support Hyper-V by default – so went off to get the details of how to enable this from the appropriate KB article (http://support.microsoft.com/kb/958662) and was pleasantly surprised to find that a “Fix it” has been made for this issue – so I was able to complete this step without too much trouble.

Side note: You may wonder what happens if you do not enable this fix it.  Simply put, by default Windows Server Backup will not engage our VSS backup components.  This means that it will just copy the files of the virtual machines without doing anything to prepare them for backup.  If your virtual machines are turned off – this is fine.  If your virtual machines are running – this can result in your backup having corrupt data in the virtual machines (but it will not affect the currently running virtual machines).

Once you enable the fix it – there is nothing in the Windows Server Backup user interface that indicates that anything is different.  But now when you backup a drive that contains virtual machines we will either use VSS inside the virtual machine in order to guarantee a valid backup is taken – or we will momentarily put the virtual machine into a saved state (if VSS is not supported by the guest operating system) and resume it after the backup is taken.

Most of my virtual machines support VSS, but I did fire up a Windows XP virtual machine just to watch the backup progress – otherwise there is no way for me to know that anything actually happened to the virtual machines :-)

I then fired up Windows Server Backup and requested to do a custom backup, and selected to only do a “Bare metal recovery” backup.  This meant that I was able to backup my system disk without backing up the (rather large) data disks used by my Windows Home Server virtual machine:

But then things started to go sideways.

On my first attempt, the backup failed after 10 minutes with an error message that stated:

“(0x81000101) The creation of a shadow copy has timed out. Try this operation again.”

Searching on this error message revealed nothing of particular interest – and as I was backing up the system due to slow performance of the disk I was trying to backup – I figured this was not too surprising.  So I decided to do as the error message advised – and try again.

The second attempt got further – about 30 minutes in – when it failed with an I/O error message.  A bit of investigation quickly revealed that the USB disk that I was trying to back up the system to had chosen this particular point in time to die.  Hmmm… Ominous.

For the third attempt I tried to backup over the network to my main desktop computer (after having to shuffle a lot of virtual machines around to make space).  This time I received an error message that stated:

“(0x80042336) The writer experienced a partial failure.”

Sigh.  At least I knew about this error message.  Basically – VSS (the backup infrastructure in Windows) prefers to have applications either succeed or fail an entire backup process.  The problem that we have is that we can succeed on all but a single virtual machine – in which case we need to report failure back to the backup application, but we also need to indicate that a specific virtual machine caused the problem.

Seeing this error message I went to check the event log.  Looking in the Admin section of the Hyper-V-VMMS log showed me that it was my FTP server that had caused the problem:

From here I went to look in the event log inside my FTP server. 

At first I checked the System log – and saw a number of error messages from the VDS Basic Provider that stated:

“Unexpected failure. Error code: 490@01010004”

One of these occurred around the time of the failed backup – but there were a number of other instances that did not appear to correlate to any backup activity.  A quick web search turned up this KB article:http://support.microsoft.com/kb/979391 that explains that this is a benign error message that can be safely ignored.

Next I checked the Application log – and saw an error message at the right time that looked like the culprit:

“Volume Shadow Copy Service error: Unexpected error calling routine.  IVssBackupComponents::SetContextInternal.  hr = 0x80042301, A function call was made when the object was in an incorrect state.”

Unfortunately searching on this error message revealed nothing but random people struggling with random variations of the error message – and none of them related to Hyper-V.  After reading through a number of these I decided that the layman's interpretation of this error message was “something went wrong deep in guts of VSS”.  With such insight in hand I decided that I would just give it another shot.

The fourth time the backup went through without a hitch.

I honestly did not expect this process to be so painful – but the nice thing is that (with the exception of my Windows XP virtual machine, which does not support VSS) through this whole process none of my running virtual machines were disturbed.  In fact – I was watching video streaming off of one of them for pretty much the entire time.

Unfortunately this story is yet to have a happy ending – as while I have been able to confirm that a valid and complete backup was taken (ironically by restoring the backup to a virtual machine on my Hyper-V server – which worked fine) I cannot get the darned thing to restore to my new disk configuration.

So for now my server continues to run a little slow on the old disk, and I am hunting down Windows Server Backup people to try and figure out why my restore is failing.  On the plus side – if I do have a hardware failure now I will have a valid backup to restore the system from (once I get that part working).

Cheers,
Ben

Update: This discussion is continued at http://blogs.msdn.com/virtual_pc_guy/archive/2010/03/10/adventures-in-backup-continued.aspx