One of the sets doesn’t have this error, but wouldn’t because they are physical servers.  They have been in production a long time, and we have plans to virtualize them. 

The second set doesn’t get the errors, but it is fewer servers and fewer users.

The third set:

• is virtual
• runs on 2008 R2 Hyper-V
• has more servers (6 as opposed to 4 or 5 for the other two)
• supports more users and more users per server (averages around 20 users per server during business hours)

Around November, we started upgrading our hosts from 2008 to 2008 R2.  The problems have been getting progressively worse peaking in the last 2 months.  Our last 2008 host was converted in March. 

After some event log review, we were able to correlate some of the issues to the following error in the event log:

Event Type:    Warning
Event Source:    netvsc
Event Category:    None
Event ID:    5
Date:        4/19/2010
Time:        3:49:53 PM
User:        N/A
Computer:    <ServerNameChangedToProtectTheGuilty>
Description:
The miniport ‘Microsoft Virtual Machine Bus Network Adapter #4′ hung.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 00 00 02 00 52 00   ……R.
0008: 00 00 00 00 05 00 00 80   …….€
0010: 00 00 00 00 00 00 00 00   ……..
0018: 00 00 00 00 00 00 00 00   ……..
0020: 00 00 00 00 00 00 00 00   ……..

and right behind that would be this message:

Event Type:    Information
Event Source:    netvsc
Event Category:    None
Event ID:    4
Date:        4/19/2010
Time:        3:49:53 PM
User:        N/A
Computer:    <ServerNameChangedToProtectTheGuilty>
Description:
The miniport ‘Microsoft Virtual Machine Bus Network Adapter #4′ reset.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 00 00 02 00 52 00   ……R.
0008: 00 00 00 00 04 00 00 40   …….@
0010: 00 00 00 00 00 00 00 00   ……..
0018: 00 00 00 00 00 00 00 00   ……..
0020: 00 00 00 00 00 00 00 00   ……..

After doing a bit of searching and getting a lot of nothing, and doing some on site troubleshooting without much luck, I finally broke down and called Microsoft.  I spent a day e-mailing back and forth with someone who was suggesting that I try all the things that I had already tried, so I contacted our TAM and had the case escalated. 

The technician then informed me that there was an internal hotfix that had not been fully tested yet, that related to my issue.  It seems that in 2008 R2 Hyper-V guests running Server 2003, the network adapter will hang and then reset under heavy load.  The hotfix has to be applied to the host and then the integration services on the guest have to be updated.  In my environment, when I updated, I had to remove the integration services from the guest before the updated NIC driver would install.  I reported this behavior to the technician I was working with, but he said that he couldn’t reproduce that particular problem and that he had no issues updating his test environment.

It is my understanding that the hotfix will be released under KB981836.  When you install this, it changes the integration services version from 6.1.7600.16385 to 6.1.7600.20683.  You can see this if you look at the driver version on the guest NIC.