One of the sets doesn’t have this error, but wouldn’t because they are physical servers.  They have been in production a long time, and we have plans to virtualize them. 

The second set doesn’t get the errors, but it is fewer servers and fewer users.

The third set:

• is virtual
• runs on 2008 R2 Hyper-V
• has more servers (6 as opposed to 4 or 5 for the other two)
• supports more users and more users per server (averages around 20 users per server during business hours)

Around November, we started upgrading our hosts from 2008 to 2008 R2.  The problems have been getting progressively worse peaking in the last 2 months.  Our last 2008 host was converted in March. 

After some event log review, we were able to correlate some of the issues to the following error in the event log:

Event Type:    Warning
Event Source:    netvsc
Event Category:    None
Event ID:    5
Date:        4/19/2010
Time:        3:49:53 PM
User:        N/A
Computer:    <ServerNameChangedToProtectTheGuilty>
Description:
The miniport ‘Microsoft Virtual Machine Bus Network Adapter #4′ hung.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 00 00 02 00 52 00   ……R.
0008: 00 00 00 00 05 00 00 80   …….€
0010: 00 00 00 00 00 00 00 00   ……..
0018: 00 00 00 00 00 00 00 00   ……..
0020: 00 00 00 00 00 00 00 00   ……..

and right behind that would be this message:

Event Type:    Information
Event Source:    netvsc
Event Category:    None
Event ID:    4
Date:        4/19/2010
Time:        3:49:53 PM
User:        N/A
Computer:    <ServerNameChangedToProtectTheGuilty>
Description:
The miniport ‘Microsoft Virtual Machine Bus Network Adapter #4′ reset.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 00 00 02 00 52 00   ……R.
0008: 00 00 00 00 04 00 00 40   …….@
0010: 00 00 00 00 00 00 00 00   ……..
0018: 00 00 00 00 00 00 00 00   ……..
0020: 00 00 00 00 00 00 00 00   ……..

After doing a bit of searching and getting a lot of nothing, and doing some on site troubleshooting without much luck, I finally broke down and called Microsoft.  I spent a day e-mailing back and forth with someone who was suggesting that I try all the things that I had already tried, so I contacted our TAM and had the case escalated. 

The technician then informed me that there was an internal hotfix that had not been fully tested yet, that related to my issue.  It seems that in 2008 R2 Hyper-V guests running Server 2003, the network adapter will hang and then reset under heavy load.  The hotfix has to be applied to the host and then the integration services on the guest have to be updated.  In my environment, when I updated, I had to remove the integration services from the guest before the updated NIC driver would install.  I reported this behavior to the technician I was working with, but he said that he couldn’t reproduce that particular problem and that he had no issues updating his test environment.

It is my understanding that the hotfix will be released under KB981836.  When you install this, it changes the integration services version from 6.1.7600.16385 to 6.1.7600.20683.  You can see this if you look at the driver version on the guest NIC.

After a bunch of searching, I saw a few people have gotten this, but no one had a solution, or at least I couldn’t find it.  I did see a similar message related to Adobe Reader, and that is what helped me find the answer to our problem.

One of the servers was not having the problem, but it DID have IE8.  hmmm..  It also had Adobe Reader 9 installed on it.  The others had Adobe Reader 8. 

So to fix the problem, required uninstalling IE8, updating to AR9, and then reinstalling IE8.  I did try just installing the updated Adobe Reader, but it still got the error until I uninstalled IE8. 

Now that the problem was understood, Microsoft and Citrix brainstormed on workarounds customers could apply while Citrix worked on an update to the ICA Client that would prevent the sharing violation. One workaround was to disable application prefetching and another was to write a logoff script that deletes the Ssonsvr.exe prefetch files. Citrix published the workarounds in this Citrix Knowledge Base article and Microsoft in this Microsoft Knowledge Base article. The update to the ICA Client, which was made available a few days later, changed the network provider DLL to 10 seconds after Ssonsvr.exe launches before returning control to Mpnotify.exe. Because Winlogon waits for Mpnotify to exit before logging on a user, the Logical Prefetcher won’t associate Winlogon’s accesses of the user’s hive with Ssonsvr.exe’s startup.

Mark’s Blog : The Case of the Temporary Registry Profiles

Ask the Core Team : DPM 2007 – Troubleshooting protection for Hyper-V

This post is about Windows Server 2008 with the Hyper-V role installed, that are being protected by System Center Data Protection Manager 2007.  There may be one or many Virtual Machines on each Host/Parent Partition, and they may be running Windows 2003 and/or Windows 2008. 

Ask the Core Team : DPM 2007 – Troubleshooting protection for Hyper-V