A virtual machine was deleted. Can it still be recovered?

Often yes, as long as the released space has not been overwritten. Stop writing to the datastore immediately, do not create new virtual machines and request an analysis.

Our snapshot chain is damaged and the VM no longer starts. What now?

Do not consolidate or delete snapshots. Rebuilding the chain can bring back the latest state, but only as long as all links are still there.

The volume uses deduplication and the files no longer open. Are they gone?

Usually not. The content is in the chunk store; what is damaged is often the references to it. Do not run garbage collection or other maintenance jobs, as they can permanently remove chunks.

Can you recover an Exchange or SQL database that was inside a virtual machine?

Yes. We work through the layers up to the database and check that it opens again. Do not run repair tools on the database beforehand; they can permanently discard damaged pages.

What is the difference with your three-tier approach for large volumes?

The three processes are about size: they make it possible to handle volumes of hundreds of terabytes. Multi-layer recovery is about structure: several layers stacked on top of each other. In large environments the two often come together.

In what form do we get the data back?

Your choice: as separate files, as a virtual disk you attach to your hypervisor again, or as a database you can mount again.

Can you restore damaged Veeam backups?

Often yes. If the backup files are on failed storage, we first recover that storage. If the backup files themselves are damaged, we make use of the redundancy they contain and, where needed, work in consultation with Veeam. Mention the version and, for encrypted backups, the password.

Multi-layer data recovery: virtualisation, deduplication and databases • Datarecuperatie

What is a multi-layer setup?

In a modern server environment every layer is a container for the next: the RAID provides a volume, the volume gets partitioning and a file system, that file system holds virtual disks, those virtual disks contain another file system, and on it sit the files of an application, such as an Exchange or SQL database.

Each layer interprets the bytes of the layer below. If one layer is damaged, everything above it becomes unreadable, even though the data itself is still fully there. The task is then to rebuild the damaged layer so that the layers above make sense again.

A damaged layer rarely means the data above it is lost. Usually it is the key to read it that is missing, not the content.

The layers, from bottom to top

Layer	Examples	What often goes wrong
Storage and RAID	Drives, RAID controller, NAS, SAN	Failed drives, lost RAID configuration
Partitioning	MBR, GPT, LVM, Storage Spaces	Overwritten partition table, disk initialised by mistake
Native file system	NTFS, ReFS, ext4, XFS, ZFS, Btrfs, VMFS	Damaged metadata, formatting, failed check
Deduplication	Windows Server deduplication, ZFS dedup, backup appliances	Damaged chunk store or index
Virtualisation	VMware vSphere, Hyper-V, Proxmox and KVM	Deleted virtual machine, broken snapshots
Virtual disk image	VMDK, VHDX, qcow2, raw	Scattered across the datastore, damaged header or block table
Guest file system	NTFS, ext4, XFS inside the virtual machine	The same problems as any file system
Application	Exchange (ESE), SQL Server, Oracle, mail archives	Damaged pages, missing log files
Backup	Veeam® (.vbk, .vib, .vrb) and other backup software	Damaged backup chain, backup files on failed storage

Technical: how virtual disks track their blocks

A virtual disk that is not fully allocated in advance (thin provisioned, sparse) keeps a table stating where each block of the virtual disk sits in the file: the grain directory and grain tables for VMDK, the Block Allocation Table (BAT) for VHDX, the L1 and L2 tables for qcow2.

Snapshots and differencing disks (VMware delta files, Hyper-V .avhdx) form a chain: each file holds only the changed blocks and points to its parent. If a link or the reference to it is damaged, the chain has to be rebuilt to get the current state of the disk.

The principle: the other layers tell what is missing

Every layer contains, deliberately or not, information about the layers around it. We use it in three directions:

Top-down. The content of a higher layer reveals how a lower layer was built. A file inside the virtual machine that runs across two fragments of the virtual disk confirms those fragments belong together; that is how a scattered virtual disk can be put in the right order, even when its block table is gone. And the file system inside a partition states its own size, which shows where the partition was, even if the partition table was overwritten.
Bottom-up. An intact lower layer narrows down the search in the higher one. If the datastore file system still knows which blocks belong to which virtual disk, we look for the structures of a damaged guest file system only in those blocks, and in the right order.
Within the layer. Many formats are built from fixed-size blocks or pages with their own signature or checksum. A SQL Server database page, for example, states its own page number. Separate pieces can thus be recognised, verified and put in order.

Technical: examples of backup copies and signatures

GPT keeps a backup copy of the header and partition table at the end of the disk. NTFS keeps a copy of the boot sector at the end of the volume, and a partial copy of the MFT (MFTMirr). ext4 and XFS keep copies of their superblock spread across the volume.

SQL Server uses 8 KB pages, with the file and page number in the header. Exchange (ESE) has used 32 KB pages since Exchange 2010, each with its own checksum. Such signatures make it possible to find pages among other data.

Deduplication: a layer that fragments everything

Deduplication stores each unique piece of data (chunk) only once. A file is then no longer a continuous run of blocks, but a list of references to chunks in a shared chunk store. With Windows Server deduplication, the files remain visible as references (reparse points), while their content lives in the chunk store.

If the index or chunk store is damaged, the files still seem to be there but will not open. Yet the chunk store holds the full content. The layer above, such as a virtual disk with a recognisable structure, then helps put the chunks back in the right order.

Backups as an extra layer

Often the backup itself is the last layer that matters: the backup files of Veeam®, Backup Exec, Macrium Reflect or other software sit on a NAS, a RAID or a deduplicated volume that failed. We then first recover the storage, then the backup files, and only then the data inside those backups, even if the backup files themselves are damaged.

All about recovering inaccessible backups →

Scenarios from practice

Deleted virtual machine

A virtual machine was deleted from a VMFS datastore. The space is released, but the blocks of the virtual disk are often still there; the structure of the guest file system is used to find and join them.

Broken snapshot chain

A Hyper-V or VMware environment with snapshots in which one link is damaged. The virtual machine no longer starts; rebuilding the chain brings back the latest state.

Virtualised Exchange server

RAID, VMFS, VMDK, NTFS and an Exchange database stacked on top of each other. One damaged layer is enough to make the mailboxes unreachable; the other layers help repair it, until the database is readable again.

Why standard software gets stuck here

Ordinary recovery software works one layer at a time and assumes the layers below are fine: a file system tool expects a correct partition, a VMDK tool a readable datastore, a database tool an intact virtual disk. In a multi-layer setup with damage at more than one level, that assumption no longer holds.

That is why we look at the whole stack at once, and determine per layer which information from the other layers is needed to reconstruct it.

What to do

Do

Stop the virtual machines and anything that writes to the affected storage.
Keep the configuration: VM settings (.vmx or the Hyper-V configuration), an overview of the datastores and snapshots.
Note the full stack: RAID, volumes, deduplication, virtualisation, applications.
Keep the logs of the hypervisor, the storage and the application.

Don't

Do not consolidate or delete snapshots.
Do not recreate or format a datastore or volume.
Do not run deduplication maintenance jobs, such as garbage collection.
Do not run repair tools inside the virtual machine or on the database, such as chkdsk or eseutil /p.

How a case runs

Analysis per layer. We map the full stack and determine which layers are damaged.
Copies. We make a full copy of every drive; physically damaged drives are repaired first in our Class 1 lab.
Reconstruction. We start at the lowest damaged layer and use the information from the other layers to rebuild it, layer by layer, up to the application.
Check at application level. A database must not only be back, it must also open. We check the result at the level where you use it.
Delivery. As files, as a virtual disk you can attach again, or as a database. You see the result before you pay.

For very large volumes we combine this with our three-tier approach.

More about large-volume recovery →

Multi-layer recovery