VMware Data Recovery Presented by Kroll Ontrack at WI Area VMware User’s Group Presented by Kroll Ontrack at WI Area VMware User’s Group

Kroll Ontrack Team Robert Bloomquist Principle Data Recovery Engineer Jodi Bredemeier Business Development Team Lead Ben Goluska Partnership Account Manager David Logue Sr. Remote Data Recovery Engineer Robert Bloomquist Principle Data Recovery Engineer Jodi Bredemeier Business Development Team Lead Ben Goluska Partnership Account Manager David Logue Sr. Remote Data Recovery Engineer

Common Data Loss Scenarios RAID/Drive problems VMFS File system corruption Format/reinstall Deleted virtual disks Internal VMDK or file corruption SQL, Exchange and Oracle Deleted files inside a Virtual Machine RAID/Drive problems VMFS File system corruption Format/reinstall Deleted virtual disks Internal VMDK or file corruption SQL, Exchange and Oracle Deleted files inside a Virtual Machine

RAID/Drive problems Scenario 1 RAID 5 – 3 (500GB) drives MS SQL data RAID Controller died Replaced controller and rebooted Drives all offline Forced drives online No volumes Scenario 1 RAID 5 – 3 (500GB) drives MS SQL data RAID Controller died Replaced controller and rebooted Drives all offline Forced drives online No volumes

Customer’s plan Force drives online and rebuild Restore from backup Recreate the data Force drives online and rebuild Restore from backup Recreate the data

Additional options Customer called Ontrack Data Recovery Ontrack suggested: Image the drives before rebuild If the rebuild fails Start a remote recovery Ship drives to Ontrack Customer called Ontrack Data Recovery Ontrack suggested: Image the drives before rebuild If the rebuild fails Start a remote recovery Ship drives to Ontrack

Customer’s challenges RAID config damged One drive out of date Rebuild would cause additional damage Backups did not include SQL data Time to recreate data 3 months to 2 years RAID config damged One drive out of date Rebuild would cause additional damage Backups did not include SQL data Time to recreate data 3 months to 2 years

Data Layout – RAID 5 HEALTHY DRIVE1Pointer 1 - VM 1Pointer 3PARITY VM 1 - 1VM1 - 3PARITYVM 2 - 1VM2 - 3 DRIVE2Pointer 2 - VM2PARITYPointer 5 VM1 - 2PARITYVM1 - 5VM2 - 2PARITY DRIVE3PARITYPointer 4Pointer 6 PARITYVM1 - 4VM1 - 6PARITYVM2 - 4 Contiguous Files

Data Layout – RAID 5 HEALTHY DRIVE1Pointer 1 - VM 1Pointer 3PARITY VM 1 - 1VM1 - 3PARITYVM1 - 4VM1 - 2 DRIVE2Pointer 2 - VM2PARITYPointer 5 VM 2 - 1PARITYVM1 - 5VM2 - 3PARITY DRIVE3PARITYPointer 4Pointer 6 PARITYVM2 - 4VM1 - 6PARITYVM2 - 2 Fragmented Files

Data Layout – RAID 5 DISORDERED DRIVE3PARITYPointer 4Pointer 6 PARITYVM1 - 4VM1 - 6PARITYVM2 - 4 DRIVE1Pointer 1 - VM 1Pointer 3PARITY VM 1 - 1VM1 - 3PARITYVM 2 - 1VM2 - 3 DRIVE2Pointer 2 - VM2PARITYPointer 5 VM1 - 2PARITYVM1 - 5VM2 - 2PARITY FILE1 VM 1 - 1VM1 - 2VM1 - 4VM1 - 3VM1 - 6VM1 - 5 FILE2 VM 2 - 1VM2 - 2VM2 - 4VM2 - 3

Resolution Ontrack used the original drives and was able to rebuild array 100% of data recovered If customer had proceeded with plans Parity would have been overwritten Data loss would have occurred Up to 1/3 of data would be missing or out of date Ontrack used the original drives and was able to rebuild array 100% of data recovered If customer had proceeded with plans Parity would have been overwritten Data loss would have occurred Up to 1/3 of data would be missing or out of date

File System Corruption Scenario 2 2 SAN LUNs - RAID 5 LUNs was accidentally formatted NTFS Scenario 2 2 SAN LUNs - RAID 5 LUNs was accidentally formatted NTFS

Customer’s plan Restore from backup to same volume Recreate the data Restore from backup to same volume Recreate the data

Additional options Customer called Ontrack Data Recovery Ontrack suggested: Restore data to different volume If the restore failed: Start a remote recovery Ship drives and/or tapes to Ontrack Customer called Ontrack Data Recovery Ontrack suggested: Restore data to different volume If the restore failed: Start a remote recovery Ship drives and/or tapes to Ontrack

Customer’s challenges Backups corrupt Good backups did not include SQL Time to recreate data 2 months Backups corrupt Good backups did not include SQL Time to recreate data 2 months

Resolution Ontrack was able to rebuild the VMFS volume 100% of critical data recovered If customer had proceeded with plans Existing structures would have been overwritten Data loss would have occurred No recovery from DR company Ontrack was able to rebuild the VMFS volume 100% of critical data recovered If customer had proceeded with plans Existing structures would have been overwritten Data loss would have occurred No recovery from DR company

Deleted Virtual Machine Scenario 3 Windows 2003 Server 2 virtual disks Virtual machine was deleted Scenario 3 Windows 2003 Server 2 virtual disks Virtual machine was deleted

Customer’s plan Restore from backup to same volume Recreate the data Restore from backup to same volume Recreate the data

Additional options Customer called Ontrack Data Recovery Ontrack Suggested: Restore data to different volume If the restore failed: Start a remote recovery Ship drives and/or tapes to Ontrack Customer called Ontrack Data Recovery Ontrack Suggested: Restore data to different volume If the restore failed: Start a remote recovery Ship drives and/or tapes to Ontrack

Customer’s challenges Backups corrupt Time to recreate data - 6 months Backups corrupt Time to recreate data - 6 months

Data Recovery - Deleted What happens when a VM is deleted

Data Layout – RAID 5 HEALTHY DRIVE1Pointer 1 - VM 1Pointer 3PARITY VM 1 - 1VM1 - 3PARITYVM 2 - 1VM2 - 3 DRIVE2Pointer 2 - VM2PARITYPointer 5 VM1 - 2PARITYVM1 - 5VM2 - 2PARITY DRIVE3PARITYPointer 4Pointer 6 PARITYVM1 - 4VM1 - 6PARITYVM2 - 4 Contiguous Files

Data Recovery - Deleted What happens when a VM is deleted Pointer is deleted or zeroed out Data area is marked as free space What happens when a VM is deleted Pointer is deleted or zeroed out Data area is marked as free space

Data Layout – RAID 5 DELETED DRIVE1DELETEDPointer 3PARITY FREE PARITYVM 2 - 1VM2 - 3 DRIVE2Pointer 2PARITYPointer 5 FREEPARITYFREEVM2 - 2PARITY DRIVE3PARITYPointer 4Pointer 6 PARITYFREE PARITYVM2 - 4 DRIVE1Pointer 1 - VM 1Pointer 3PARITY VM 1 - 1VM1 - 3PARITYVM 2 - 1VM2 - 3 DRIVE2Pointer 2 - VM2PARITYPointer 5 VM1 - 2PARITYVM1 - 5VM2 - 2PARITY DRIVE3PARITYPointer 4Pointer 6 PARITYVM1 - 4VM1 - 6PARITYVM2 - 4 Before After

Resolution Ontrack was able to rebuild deleted virtual disks Manually assembled multiple fragments 100% of critical data recovered If customer had proceeded with plans Existing data would have been overwritten Data loss would have occurred Possibly no recovery from DR company Ontrack was able to rebuild deleted virtual disks Manually assembled multiple fragments 100% of critical data recovered If customer had proceeded with plans Existing data would have been overwritten Data loss would have occurred Possibly no recovery from DR company

Internal Corruption Scenario 4 Large storm, city lost power UPS engaged and generator started Generator ran out of fuel UPS failed SQL server crashed during large update Scenario 4 Large storm, city lost power UPS engaged and generator started Generator ran out of fuel UPS failed SQL server crashed during large update

Customer’s plan Work with SAN vendor Work with database vendor Work with Microsoft Restore from backup to same volume Recreate the data Work with SAN vendor Work with database vendor Work with Microsoft Restore from backup to same volume Recreate the data

Additional options Customer called Ontrack Data Recovery Make copy of SQL database to server 2 Restore data to different volume Start a remote recovery Ship drives and/or tapes to Ontrack Customer called Ontrack Data Recovery Make copy of SQL database to server 2 Restore data to different volume Start a remote recovery Ship drives and/or tapes to Ontrack

Customer’s challenges RAID and VMFS volume OK SQL file too corrupt for vendor to repair SQL file to corrupt for Microsoft Backups corrupt Time to recreate data Could not recreate data RAID and VMFS volume OK SQL file too corrupt for vendor to repair SQL file to corrupt for Microsoft Backups corrupt Time to recreate data Could not recreate data

Resolution DR Company was able to reconstruct database structures/access row data 100% of critical row data recovered into a new database If customer had proceeded with plans Database would have been further corrupted (no good copy) Further data loss would have occurred No recovery from DR company DR Company was able to reconstruct database structures/access row data 100% of critical row data recovered into a new database If customer had proceeded with plans Database would have been further corrupted (no good copy) Further data loss would have occurred No recovery from DR company

Disaster Prevention

Create a Disaster Recovery Plan Define critical systems Time frames for restores Define critical contacts Internal and external Define procedures Define alternative options Backup site Create a Disaster Recovery Plan Define critical systems Time frames for restores Define critical contacts Internal and external Define procedures Define alternative options Backup site

Disaster Prevention Create a Disaster Recovery Plan Backup the Data Create a Disaster Recovery Plan Backup the Data

Disaster Prevention Create a Disaster Recovery Plan Backup the Data Test the Backups Create a Disaster Recovery Plan Backup the Data Test the Backups

Disaster Prevention Create a Disaster Recovery Plan Backup the Data Test the Backups Include Data Recovery in every Plan Create a Disaster Recovery Plan Backup the Data Test the Backups Include Data Recovery in every Plan

Questions?

Questions Toll free – Kroll Ontrack Free phone consultation Toll free – Kroll Ontrack Free phone consultation

Thank you

Ontrack One of the only companies to offer ESX server recovery Special tools recently developed for these recoveries Special training provided to engineers Development staff available to assist if needed Ability to handle complex, high end recoveries One of the only companies to offer ESX server recovery Special tools recently developed for these recoveries Special training provided to engineers Development staff available to assist if needed Ability to handle complex, high end recoveries

What is Virtualization Virtualization is: Software that allows the user to run multiple operating systems at the same time on the same physical computer by “virtualizing” the hardware of a complete physical computer. Virtualization is: Software that allows the user to run multiple operating systems at the same time on the same physical computer by “virtualizing” the hardware of a complete physical computer.

Terms to know Virtual (Guest) Machine A simulated computer in that it runs on a host computer but behaves as if it were a separate computer VMDK A single file that contains a virtual disk VMFS The VMware file system ESX The VMware operating system Virtual (Guest) Machine A simulated computer in that it runs on a host computer but behaves as if it were a separate computer VMDK A single file that contains a virtual disk VMFS The VMware file system ESX The VMware operating system