Michael Phillips Blog

Follow up to the DPM recovery point expiration issues

Previously, I blogged about issues I was having where old recovery points were not being expired/removed from my DPM servers.  I had to open a ticket with Microsoft, and worked with them to determine the cause, and since then, they have released a fix.

The fix that Microsoft developed is here: http://www.microsoft.com/downloads/details.aspx?FamilyID=aee949aa-d3e7-4b0f-b718-00b7c20f1257&displayLang=en

A few people have asked for the PowerShell script “show-pruneshadowcopies.ps1” that Microsoft provided and I mentioned in my previous post (here).  The script looks like this:

#displays all RP for data sources and shows which RP’s would be deleted by the regular pruneshadowcopies.ps1
# Outputs to a logfile:  C:\Program Files\Microsoft DPM\DPM\bin\SHOW-PRUNESHADOWCOPIES.LOG

#Author    : Mike J
#Date    : 02/24/2009
$version="V1.0"

$date=get-date
$logfile="SHOW-PRUNESHADOWCOPIES.LOG.txt"

function GetDistinctDays([Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.ProtectionGroup] $group,
[Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.Datasource] $ds)
{   
    if($group.ProtectionType -eq [Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.ProtectionType]::DiskToTape)
    {
        return 0
    }
    $scheduleList = get-policyschedule -ProtectionGroup $group -ShortTerm
    if($ds -is [Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.FileSystem.FsDataSource])
    {
        $jobType = [Microsoft.Internal.EnterpriseStorage.Dls.Intent.JobTypeType]::ShadowCopy
    }
    else
    {
        $jobType = [Microsoft.Internal.EnterpriseStorage.Dls.Intent.JobTypeType]::FullReplicationForApplication
        if($ds.ProtectionType -eq [Microsoft.Internal.EnterpriseStorage.Dls.Intent.ReplicaProtectionType]::ProtectFromDPM)
        {           
            return 2
        }
    }
    write-host   "Look for jobType $jobType"

    foreach($schedule in $scheduleList)
    {
        write-host("schedule jobType {0}" -f $schedule.JobType)
        if($schedule.JobType -eq $jobType)
        {
            return [Math]::Ceiling(($schedule.WeekDays.Length * $ds.RecoveryRangeinDays) / 7)
        }
    }

    return 0
}

function IsShadowCopyExternal($id)
{
    $result = $false;

    $ctx = New-Object -Typename Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlContext
    $ctx.Open()

    $cmd = $ctx.CreateCommand()
    $cmd.CommandText = "select COUNT(*) from tbl_RM_ShadowCopy where shadowcopyid = ‘$id’"  
    write-host $cmd.CommandText
    $countObj = $cmd.ExecuteScalar()
    write-host $countObj
    if ($countObj -eq 0)
    {
        $result = $true
    }
    $cmd.Dispose()
    $ctx.Close()

    return $result
}

function IsShadowCopyInUse($id)
{
    $result = $true;

    $ctx = New-Object -Typename Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlContext
    $ctx.Open()

    $cmd = $ctx.CreateCommand()
    $cmd.CommandText = "select ArchiveTaskId, RecoveryJobId from tbl_RM_ShadowCopy where ShadowCopyId = ‘$id’"  
    write-host $cmd.CommandText
    $reader = $cmd.ExecuteReader()
    while($reader.Read())
    {
        if ($reader.IsDBNull(0) -and $reader.IsDBNull(1))
        {
            $result = $false
        }
    }
    $cmd.Dispose()
    $ctx.Close()

    return $result
}

"**********************************" > $logfile
"Version $version" >> $logfile
get-date >> $logfile

$dpmservername = &"hostname"

$dpmsrv = connect-dpmserver $dpmservername

if (!$dpmsrv)
{
    write-host "Unable to connect to $dpmservername"
    exit 1
}

write-host $dpmservername
"Selected DPM server = $DPMservername" >> $logfile
$pgList = get-protectiongroup $dpmservername
if (!$pgList)
{
    write-host   "No PGs found"
    disconnect-dpmserver $dpmservername
    exit 2
}

write-host("Number of ProtectionGroups = {0}" -f $pgList.Length)
$replicaList = @{}
$latestScDateList = @{}

foreach($pg in $pgList)
{
    $dslist = get-datasource $pg
    if ($dslist.length -gt 0)
    {
    write-host("Number of datasources in this PG = {0}" -f $dslist.length)
    ("Number of datasources in this PG = {0}" -f $dslist.length) >> $logfile
     }
    Foreach ($ds in $dslist)
    {
       write-host("DS NAME=  $ds")
       ("DS NAME=  $ds") >>$logfile
    }
    foreach ($ds in $dslist)
    {       
        $rplist = get-recoverypoint $ds | where { $_.DataLocation -eq ‘Disk’ }
        write-host("Number of recovery points for $ds {0}" -f $rplist.length)
        ("Number of recovery points for $ds {0}" -f $rplist.length) >>$logfile 
        $countDistinctDays = GetDistinctDays $pg $ds
        write-host("Number of days with fulls = $countDistinctDays")
        ("Number of days with fulls = $countDistinctDays") >>$logfile
        if($countDistinctDays -eq 0)
        {
            write-host   "D2T PG. No recovery points to delete"
            "D2T PG. No recovery points to delete" >>$logfile
            continue;
        }
        $replicaList[$ds.ReplicaPath] = $ds.RecoveryRangeinDays
        $latestScDateList[$ds.ReplicaPath] = new-object DateTime 0,0
        $lastDayOfRetentionRange = ([DateTime]::UtcNow).AddDays($ds.RecoveryRangeinDays * -1);       
        write-host("Distinct days to count = {0}. LastDayOfRetentionRange = {1} " -f $countDistinctDays, $lastDayOfRetentionRange)
        ("Distinct days to count = {0}. LastDayOfRetentionRange = {1} " -f $countDistinctDays, $lastDayOfRetentionRange) >>$logfile
        $distinctDays = 0;
        $lastDistinctDay = (get-Date).Date
        $numberOfRecoveryPointsDeleted = 0

        if ($rplist)
        {
            foreach ($rp in ($rplist | sort-object -property UtcRepresentedPointInTime -descending))
            {                       
                if ($rp)
                {                   
                    if ($rp.UtcRepresentedPointInTime.Date -lt $lastDistinctDay)
                    {
                        $distinctDays += 1
                        $lastDistinctDay = $rp.UtcRepresentedPointInTime.Date
                    }
                    write-host(" $ds")
                    (" $ds") >>$logfile
                    write-host("  Recovery Point #$distinctdays RPtime={0}" -f $rp.UtcRepresentedPointInTime)
                    ("  Recovery Point #$distinctdays RPtime={0}" -f $rp.UtcRepresentedPointInTime) >>$logfile
                    if (($distinctDays -gt $countDistinctDays) -and ($rp.UtcRepresentedPointInTime -lt $lastDayOfRetentionRange))
                    {
                        write-host ("Recovery Point would be deleted ! – RPtime={0}" -f $rp.UtcRepresentedPointInTime)  -foregroundcolor red
                        ("Recovery Point would be deleted ! – RPtime={0} <<<<<<<" -f $rp.UtcRepresentedPointInTime) >>$logfile
#remove-recoverypoint $rp -ForceDeletion -confirm:$true | out-null
                        $numberOfRecoveryPointsDeleted += 1
                    }
                    else
                    {
                        write-host "    Recovery point not expired yet"
                        "    Recovery point not yet expired" >>$logfile
                    }
                }
                else
                {
                    write-host "Got a NULL rp"
                    "Got a NULL rp" >>$logfile
                }   
            }

            write-host "Number of RPs that would be deleted = $numberOfRecoveryPointsDeleted"  
            "Number of RPs that would be deleted = $numberOfRecoveryPointsDeleted" >>$logfile            
        }
    }
}

disconnect-dpmserver $dpmservername
write-host "Exiting from script"

exit

Hyper-V and DPM – Some issues that you may see

We have several (15 or so) Hyper-V hosts running a number (126 or so) guests.  We use DPM to backup our servers, but only a few of our VMs are backed up at the host level.  Most are backed up as regular clients.  I have been having trouble with a couple of the ones that we do backup at the host level and just got to looking for the answer to what is going on.  Lucky for me I waited long enough for the Core Team to come up with some suggestions:

Ask the Core Team : DPM 2007 – Troubleshooting protection for Hyper-V

This post is about Windows Server 2008 with the Hyper-V role installed, that are being protected by System Center Data Protection Manager 2007.  There may be one or many Virtual Machines on each Host/Parent Partition, and they may be running Windows 2003 and/or Windows 2008. 

Ask the Core Team : DPM 2007 – Troubleshooting protection for Hyper-V

Remote Server Administration Tools for Windows 7 Release Candidate (RC)

Admin tools for Windows 7 (RC).  Windows 7 rocks so far.  I just got to the point where I decided to install the admin tools though. 

Installing Remote Server Administration Tools for Windows 7 RC
You must be either a member of the Administrators group on the computer on which you want to install the Administration Tools pack, or you must be logged on to the computer by using the built-in Administrator account.
Important: Remove all versions of Administration Tools Pack or Remote Server Administration Tools for Windows Vista with SP1 from the computer before installing Remote Server Administration Tools for Windows 7 RC.
Important: Only one copy of Remote Server Administration Tools for Windows 7 RC can be installed on a computer at one time. Before you install a new package, remove any existing copies of Remote Server Administration Tools for Windows 7 RC. This includes any copies that are in different languages. To remove existing copies of Remote Server Administration Tools for Windows 7 RC, see Removing the complete Remote Server Administration Tools for Windows 7 RC pack on this page.
1. On a computer that is running the RC release of Windows 7 RC, download the Remote Server Administration Tools for Windows 7 RC package from the Microsoft Download Center.
2. Open the folder into which the package downloaded, and double-click the package to unpack it, and then start the Remote Server Administration Tools for Windows 7 RC Setup Wizard.
Important: You must accept the License Terms and Limited Warranty to start to install the Administration Tools pack.
3. Complete all the steps that you must follow by the wizard, and then click Finish to exit the wizard when installation is completed.
4. Click Start, click Control Panel, and then click Programs.
5. In the Programs and Features area, click Turn Windows features on or off.
6. If you are prompted by User Account Control to enable the Windows Features dialog box to open, click Continue.
7. In the Windows Features dialog box, expand Remote Server Administration Tools.
8. Select the remote management tools that you want to install.
9. Click OK.

Download details: Remote Server Administration Tools for Windows 7 Release Candidate (RC)

Time Travel?

Evidently, Windows Server 2003, running on Hyper-V, is confused and thinks it must be time traveling.  Read below for the details…

(Event ID 1054)
Windows cannot obtain the domain controller name for your computer network. (An unexpected network error occurred.). Group Policy processing aborted.

The customer explained to us that if he removes one of the Hyper-V virtual processors from his Windows Server 2003 Guest, the issue goes away. Based on this statement we asked the customer to gather a userenv log while forcing a group policy refresh with the additional virtual processor enabled, and this what we saw during the initial ping test before we process group policy:

USERENV(15c.858) 15:55:09:080 PingComputer: First time: 2069
USERENV(15c.858) 15:55:09:080 PingComputer: Second time: 2069
USERENV(15c.858) 15:55:09:080 PingComputer: First and second times match.
USERENV(15c.858) 15:55:09:080 PingComputer: First time: 2069
USERENV(15c.858) 15:55:09:080 PingComputer: Second time: -2069
USERENV(15c.858) 15:55:09:080 PingComputer: First time: -2069
USERENV(15c.858) 15:55:09:080 PingComputer: Second time: 0

We have a knowledgebase article that pertains to this issue on servers that uses dual-core or multiprocessor AMD Opteron processors:

938448 A Windows Server 2003-based server may experience time-stamp counter drift if the server uses dual-core AMD Opteron processors or multiprocessor AMD Opteron processors

http://support.microsoft.com/default.aspx?scid=kb;EN-US;938448

Now in the case of our customer they were not running an AMD Processor server so they felt this resolution did not apply to them. Even though the article did not apply to the type of processor in their servers, the behavior was identical so we applied the resolution outlined in the knowledgebase article and this resolved the customer’s issue. I am in the process of having a knowledgebase article created to specifically address this issue with Windows Server 2003 virtual machines running in Hyper-V.

So we did a little digging and found the following blog post from Tony Voellm, who is a Principal Software Test Engineer in the Windows Kernel development team:

Negative ping times in Windows VM’s – whats up?

http://blogs.msdn.com/tvoellm/archive/2008/06/05/negative-ping-times-in-windows-vm-s-whats-up.aspx

The following is from the above blog post:

If you see negative ping times in multiprocessor W2k3 guest OSes you might consider setting the /usepmtimer in the boot.ini file.

The root issue comes about from the Win32 QueryPerformanceCounter function.  By default it uses a time source called the TSC.  This is a CPU time source that essentially counts CPU cycles.  The TSC for each (virtual) processor can be different so there is no guarantee that reading TSC on one processor has anything to do with reading TSC on another processor.  This means back to back reads of TSC on different VP’s can actually go backwards. Hyper-V guarantees that TSC will not go backwards on a single VP.

So here the problem with negative ping times is the time source is using QueryPerformanceCounter which is using TSC.  By using the /usepmtimer boot.ini flag you change the time source for QueryPerformanceCounter from TSC to the PM timer which is a global time source.

Ask the Directory Services Team : Userenv 1054 events as a result of time-stamp counter drift on Windows Server 2003 guests running in Hyper-V

DPM v 3

I just watched a webcast on DPM v3 and thought I would share some of what I got from that.

In the last 18 months, DPM 2007 (v2) delivered application protection for Exchange ,SQL Server, SharePoint and virtualization environments running Virtual Server and Hyper-V.  Disaster recovery with Iron Mountain, Local Datasource Protection and Client backups have also come out through DPM 2007, its first feature update and Service Pack 1.  Now it is time to show what is coming next for DPM. 

A few top line items are support for the following:

• support for Exchange 14, and more granular restore
• protect the entire SQL instance, and auto discover new DB’s
• protect 1000’s of DBs per DPM server
• End User Recovery by the SQL Admin (role based access from the DPM console)
• Office 14
• AD appears as a data source in DPM UI
• Image restore from centrally managed DPM server – executed locally
• Support for Windows guest on VMware hosts
• SAP running on MS SQL

and some other improvements:

• up to 100 servers, 1000 laptops, 2000 databases per DPM server
• management pack updates
• automatic re-running of jobs and improved self-healing  —  This is a huge one in my book
• auto protect new sources for SQL and MOSS
• improved scheduling capabilities
• one click DPM DR failover and failback
• continued support for SAN (scripts/whitepapers)

platform requirements:

• DPM Server must be 64-bit Windows Server 2008 R2
• Integration capability with Windows EBS 2008 R2

“We’re Sorry”?

You would think that a company like Microsoft could come up with a better answer for a search on an error than:

We’re sorry

There is no additional information about this issue in the Error and Event Log Messages or Knowledge Base databases at this time. You can use the links in the Support area to determine whether any additional information might be available elsewhere.

---

Thank you for searching on this message; your search helps us identify those areas for which we need to provide more information.

P2V fails at Copy Hard Disk

I have been trying to get a P2V of a production system to use in our DR plan.  I have limited opportunity to do this, because I am not allowed to impact performance during production hours for this system, and the definition of production hours is fairly broad.  I have been trying for a couple of months to get this figured out.

We have our regularly scheduled maintenance once a month on the third Thursday of the month.  This is pretty awesome in that we are at liberty (most months) to take everything down from 6PM until 6AM.  I look at it as giving the company an evening off.

So, that being tonight, I had it in my mind that I was going to beat the OAS boxes.  (Oracle Application Servers, part of our new JD Edwards ERP system.)  They are an interesting setup, because they are using Apache, which as great as it may be, isn’t something I have much experience with.  They have a loopback adapter for use with the load balancing setup that they are in.  The load balancing is performed using our Cisco switches, which as great as they are, I don’t know very much about.  All in all, they are pretty complicated to troubleshoot in this case, because there are so many pieces that I am not completely familiar with. 

Such is life…

Anyway,  after a lot of hunting and a lot of posting in forums, I found an event that actually led to a solution. I probably should have found this before, and maybe I did, but didn’t pay enough attention… 

This is the exact symptoms that I had, and the errors in the event log were there, but the machine that I am trying to convert is a Windows 2003 Server, not Windows XP:

The P2V process fails at 40% when you try to run the P2V process by using Microsoft System Center Virtual Machine Manager 2008 on a source computer that is running Windows XP

You use Microsoft System Center Virtual Machine Manager 2008 to run the Physical-to-Virtual (P2V) process on a source computer that is running Windows XP. However, the process fails at 40% complete, and the following error is logged in the event log on the computer that has System Center Virtual Machine Manager (SCVMM) 2008 installed:

Type:		Warning
Date:		<Date>
Time:		<Time>
Event:		1706
Source:		Virtual Machine Manager
Category:	None
Computer:	<Computer Name>
Event Msg:	Job 7bfcd14a-884e-4a71-9984-3274622adeb7 (Physical-to-virtual conversion) failed to complete. 7bfcd14a-884e-4a71-9984-3274622adeb7 Physical-to-virtual conversion TaskFailed    

Additionally, you will find the following error logged in the event log on the source computer:

Type:		Error
Date:		<Date>
Time:		<Time>
Event:		15005
Source:		HTTP
Category:	None
Computer:	<Computer Name>
Event Msg:	Unable to bind to the underlying transport for 0.0.0.0:443. The IP Listen-Only list may contain a reference to an interface which may not exist on this machine.  The data field contains the error number.
Data:
 00 00 04 00 02 00 52 00 00 00 00 00 9D 3A 00 C0		 . . . . . . R . . . . . . . . À
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00		 . . . . . . . . . . . . . . . .
 00 00 00 00 00 00 00 00 43 00 00 C0				 . . . . . . . . C . . À

The P2V process fails at 40% when you try to run the P2V process by using Microsoft System Center Virtual Machine Manager 2008 on a source computer that is running Windows XP

Hyper-V certificate expiration and resolution

If you see this:

‘VMName’ failed to initialize.
Could not initialize machine remoting system. Error: ‘Unspecified error’ (0×80004005).
Could not find a usable certificate. Error: ‘Unspecified error’ (0×80004005).

Then this could be your answer:

Symptoms and resolution:

§ You may be unable to start or connect to virtual machines running on Windows Server 2008 or Microsoft Hyper-V Server 2008. This occurs when connecting using vmconnect. Connections made using remote desktop won’t be affected.  

§ KB Article 967902 has been created that details the symptoms and resolution.  This KB article provides a direct link to download the quickfix to resolve this error.

Important Notes:

§ Though this error may occur, the Hyper-V service will continue to operate.   Neither the Hyper-V host nor the running virtual machines will go offline.

§ It is not expected that this issue can be exploited for malicious purposes.

§ Customers running Windows Server 2008 R2 Hyper-V beta won’t experience this error.

That last line it my favorite…

I guess we should be running the beta in production…  and what’s with the ‘Unspecified error’?  

Windows Server Division WebLog : Hyper-V certificate expiration and resolution

Remove DPM agent from the DPM agent console

I blogged about this last year, but when I moved my blog, I lost part of the post (the picture) so I just deleted the post.  Then I noticed that Google is still sending people here to find the answer, so…

If you have DPM Protected Computer that goes away before you uninstall the agent, it isn’t obvious how you get the agent removed from the console.  Or at least it wasn’t immediately obvious to me.

1. In the Management/Agents tab, right click on the agent (it will have a red x and “Unavailable” in the Agent Status column) and select Uninstall…
2. Verify your list of agents (you can select more than one)
3. Click on “Uninstall Agents”
4. Enter the appropriate credentials.  This must be an account that has permissions to remove the agent from the DPM server, even though the Protected computer doesn’t exist, it still has to be a valid account.
5. Select the “Manually restart the selected servers later” radio button
6. Click ok.

So far, that isn’t any different than any other client uninstall.   At this point, you will have the option to close the window, and go on about your business.  And if the protected computer was still available, that would be perfectly fine to do.  But since the protected computer isn’t still available, you have to wait for the error to pop up.  First you will see that the uninstall failed and then you get this message:

Basically, it says, I couldn’t find that computer to remove the agent, you want me to just forget that it existed?  You click on “Yes” and then the entry for that computer is removed from the DPM database.  Now wasn’t that obvious?

New Features in System Center Virtual Machine Manager 2008 R2

Of the new features coming in the R2 versions of Windows Server 2008 and SCVMM, I think these two are the obvious winners:

Support for Live Migration: With Windows 2008 R2 adding support for Live migration, it’s now added as a new migration option in VMM R2. Live migration requires the source and destination host to be part of a failover cluster and that the VM is on a shared storage. Live migration means that there is no user perceived downtime; since the VM’s memory pages are being transferred, the hosts’ processors need to be the same (manufacturer and processor architecture). Our competition claims that Vmotion doesn’t require clustering but this only works for planned downtime and not for unplanned downtime. By combining Live migration and clustering, Hyper-V addresses both planned and unplanned downtime.

Multiple VMs per LUN: VMM 2008 didn’t allow placing multiple VMs per LUN even though Hyper-V allowed it and the reason was that the LUN ownership was on a per host basis. This meant that migrating any VM on that shared LUN would result in all other VMs being migrated as well which can result in a confusing user experience (I’ve blogged about this at length). With CSV (Clustered Shared Volumes) in Windows 2008 R2, a single LUN is accessible by all hosts within a cluster. This enables a VM that’s on a shared LUN to be migrated without affecting other VMs on that LUN. As a result, with VMM R2, we’ll allow multiple VMs to be placed on the same LUN if CSV is enabled on the cluster.

http://blogs.technet.com/rakeshm/archive/2009/03/16/scvmm-2008-r2-beta-is-available-now.aspx

That is from the beta release announcement for SCVMM.  I have downloaded the beta, but haven’t had time lately to get it setup.  I am hoping to work on that this coming week…