Repository navigation
Conversation
The KVM backup export (Veeam) only works for file-based pools: the hypervisor reads the volume files through a libvirt backup job, and the volume path is put together from the pool's mount point. Volumes on block storage like LINSTOR can't be read that way. Storage drivers can now serve the volumes of their pools themselves (new default methods on PrimaryDataStoreDriver): startBackupExport takes one crash consistent storage snapshot of the VM's volumes and makes each volume readable as a raw device on a host, finishBackupExport releases it, prepareVolumeForRestore makes a volume writable for a restore. For such VMs KVMBackupExportServiceImpl skips the libvirt backup job and the hypervisor checkpoints, keeps the per-volume export in backup_details and serves each volume with its own qemu-nbd on the host the driver chose, running and stopped VMs alike. VMs whose volumes mix both kinds of pools are rejected. qemu-nbd exports a block device as raw up to the volume size, as the device can be larger than the volume. Backups of these volumes can be incremental. The driver gets, per volume, the change tracking id of the volume's last backup at the VM's active checkpoint (volume detail backup.change.tracking.id, "<checkpoint id>:<change tracking id>") and can mark the export incremental. It can also name an extents source: the host's storage adaptor writes the extents of the export with the flags dirty (changed since the last backup) and zero (reads as zeros) to a file, which the image server serves as the extents of the transfer instead of a dirty bitmap and the allocation of the export. Without one, all of the export counts as changed data. LINSTOR implements it with changed block tracking snapshots (REST API 1.30): one multi snapshot with the CBT flag, a CBT session per volume on a host with a diskful copy, and the snapshot is deleted once the sessions are finished. The extents come from the CBT session of the snapshot: in mode dirty for an incremental backup, only if DRBD tracked the changes since the last backup's CBT id (else, e.g. after a failed backup, the volume is backed up in full), and in mode zero for a full backup if DRBD knows the never written extents (drbd-utils 9.36.0 and newer). Listing the dirty extents fails if they are not relative to that CBT id. The Veeam adapter accepts pools whose driver exports the volumes and reports LINSTOR pools as block storage domains. Their volumes are reported as cow (QCOW2) disks with backup incremental, whatever image format CloudStack records for them, as Veeam backs up only QCOW2 disks incrementally; their data goes through the image transfers as raw like for any disk. java-linstor 0.9.0 brings REST API 1.30; its resourceSnapshotDelete has an additional argument.
The agent takes the control socket answering as the image server being ready and hands out the transfer URL right away, but the server started the control socket thread first and only then bound the HTTP socket. The HTTP bind can take seconds: HTTPServer.server_bind() resolves the listen address with socket.getfqdn(), which took 5s on a test host whose DNS could not resolve it. Clients connecting in that window - the first transfer after the server starts - got "connection refused". Create the HTTP server (bound, listening, TLS wrapped) before starting the control socket and idle sweep threads; connections arriving before serve_forever() wait in the listen backlog. If the HTTP socket can't be bound, the control socket now never comes up, so the agent reports the failure instead of a server that can't serve.
…iver An upload with format cow (file backend: the image server writes the image as is) needs a file of the volume, so only raw images could be uploaded into volumes whose storage driver exports them as raw devices. Veeam uploads its worker image as qcow2, so deploying a Veeam worker on LINSTOR failed with "Only raw images can be uploaded". Storage drivers can now provide a temporary raw device for such an upload (new default methods PrimaryDataStoreDriver.createUploadStaging and deleteUploadStaging; without, only raw images can be uploaded, as before). The image is uploaded to it as is. On finalize the host checks that it is a qcow2 image without backing or data file and converts it into the volume's device with qemu-img convert, skipping zeros if the volume reads as zeros (KVMStoragePool.isVolumeZeroInitialized), and waits up to copy.volume.wait for it. The temporary device is kept as volume detail upload.staging and deleted on finalize, on cancel (without converting) and if the transfer can't be set up. LINSTOR creates a resource cs-upload-<transfer id> in the pool's resource group with one replica: on the host if it has storage of the group, else on another node and diskless on the host. Its size leaves room for the qcow2 metadata.
Veeam passes the worker's network configuration in the cloud-init user data (write_files /opt/VeeamBackupAgent/network/network_config.yml) and expects the worker to have that IP. saveInstanceAdditionalDetails keeps it as detail worker.vm.ip, which gives the worker's NIC that IP when it is added, but creating the VM called saveInstanceRestoreConfig instead, so the NIC got another IP of the network and Veeam could not reach the worker.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #14731 +/- ##
============================================
+ Coverage 20.81% 20.89% +0.08%
- Complexity 20923 21042 +119
============================================
Files 6053 6054 +1
Lines 548451 549089 +638
Branches 73279 73367 +88
============================================
+ Hits 114138 114737 +599
+ Misses 420948 420921 -27
- Partials 13365 13431 +66
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR lets the Veeam integration for KVM (#12991) back up and restore VMs whose volumes are on LINSTOR primary
storage, including incremental backups.
The KVM backup export only works for file-based pools: the host reads the volume files through a libvirt backup job,
and incremental backups need qcow2 dirty bitmaps. Volumes on block storage like LINSTOR can't be read that way.
Volumes exported by their storage driver. Storage drivers can now export the volumes of their pools for a backup
themselves, with new default methods on
PrimaryDataStoreDriver. Drivers that don't implement them work as before.startBackupExporttakes one crash consistent storage snapshot of the VM's volumes and makes each volume readable asa raw device on a host that the driver chooses.
finishBackupExportreleases it.prepareVolumeForRestoremakes a volume writable on a host for a restore.For such VMs,
KVMBackupExportServiceImplskips the libvirt backup job and the hypervisor checkpoints. It serves eachvolume with its own qemu-nbd on the driver's host, for running and stopped VMs alike. VMs with volumes on both kinds of
pools are rejected.
Incremental backups. For each volume, the driver gets the change tracking id of the volume's last backup at the
VM's active checkpoint, and can mark the export as incremental. It can also name a source for the extents: the host
then writes the extents of the export to a file, flagged dirty (changed since the last backup) or zero (reads as
zeros). The image server serves these as the extents of the transfer, instead of a dirty bitmap. Without such a
source, the whole export counts as changed.
LINSTOR implements this with changed block tracking (CBT) snapshots of DRBD (LINSTOR REST API 1.30).
copy. The snapshot is deleted when the backup is finalized.
the changes since then, for example after a failed backup, that volume is backed up in full.
backup, as Veeam only backs up QCOW2 disks incrementally. The data still goes through the image transfers as raw.
qcow2 uploads into these volumes. Veeam uploads the image of its worker VM as qcow2, which the image server
writes to a file. So deploying a worker onto LINSTOR failed with "Only raw images can be uploaded".
createUploadStaging,deleteUploadStaging).The image is written to it as is.
the volume with
qemu-img convert.Two fixes in the Veeam integration that apply to all storage:
the sign that the server is ready. Before, the first transfer after the server started could get "connection
refused": binding the HTTP socket took 5 s on a host whose DNS could not resolve its own name.
save that IP, so the worker's NIC got another IP of the network, and Veeam waited at "obtaining IP address".
java-linstor goes from 0.7.0 to 0.9.0 (REST API 1.30).
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
Unit tests, new and extended:
KVMBackupExportServiceImplTestLinstorPrimaryDataStoreDriverImplTest,LinstorStorageAdaptorTest,LinstorUtilTestServerAdapterTest,VolumeJoinVOToDiskConverterTestTest setup:
With Veeam Backup & Replication 13 (oVirt KVM plug-in), in an advanced zone:
written.
for blocks that the still running source had written after the backup.
With a script that drives the same API as Veeam, comparing every result byte for byte against the devices:
incremental;
How did you try to break this feature and the system with this change?
rather than with wrong extents.
can't be set up on the host, deletes the temporary device without converting (unit tests).
still released (unit tests).
(unit tests).
Found while testing, not changed in this PR:
the capacity of a VM that powered itself off until the next capacity check (up to 5 minutes later), so the start can
fail with insufficient capacity on small hosts.
through the ConfigDrive provider. The documentation only says that the network needs UserData.
POST /disks/{id}/reducereturns "not implemented". Veeam calls it for QCOW2 disks after a restore and toleratesthe error.