Repository navigation
vmm: Add clock_catchup to the restore config - #10
Closed
zili-simular wants to merge 11 commits into
Closed
zili-simular wants to merge 11 commits into
zili-simular wants to merge 11 commits into
Conversation
Build cloud-hypervisor on the dev branch for x86_64 and aarch64, then publish both static binaries to a rolling dev prerelease with checksums and build provenance. The binaries are built with the fw_cfg feature. On aarch64 that is what lets `--firmware` and `--kernel` be combined so EDK2 boots the kernel through fw_cfg and the guest gets ACPI; without `--fw-cfg-config` the feature changes nothing. The staged binary prints its enabled features so the release log shows the flag took effect. Signed-off-by: CMGS <ilskdw@gmail.com>
Every snapshot currently dumps the entire guest RAM, so the pause a snapshot imposes grows with the memory size regardless of how little the guest has changed. Iterative checkpointing and warm pre-copy (take a snapshot, let the guest run, take another) repeat that full-memory cost every round. Add a diff snapshot: `vm.snapshot` accepts a `snapshot_type` of `full` (default) or `diff`, exposed as `ch-remote snapshot --diff`. The first diff of a series writes a full baseline and enables dirty-page tracking; each subsequent diff writes only the pages dirtied since the previous one, so its pause is proportional to the dirtied memory rather than to the guest RAM size. Dirty pages are harvested after the device snapshot, so pages touched by snapshot side effects are included. The whole series lives in one directory, and every diff request of a series must target that directory. A delta `memory-ranges.diff.N` starts with a magic, an entry count and a table of (gpa, length) pairs, followed by the bytes of each entry, so it does not rely on the filesystem preserving holes. It is written under a temporary name, synced and renamed into place before `config.json` and `state.json` are replaced; both are synced under temporary names first and then renamed back to back. The memory manager state records how many deltas it covers, so an interrupted diff leaves the previous snapshot of the series restorable. The three syncs run inside the pause, in time proportional to the dirtied memory. A full snapshot, a memory layout change, a failed snapshot, a migration, a reboot, a shutdown or deleting the VM ends the series and stops dirty logging. Restore needs no new parameter: memory fills from the baseline (in any restore mode except ondemand, which a series rejects), then the deltas the state counts replay in order. A missing delta, or one whose table does not fit the snapshot's memory layout or its own file size, fails the restore before that delta touches memory. Dirty tracking is enabled lazily on the first diff rather than at boot, so a VM that never takes a diff snapshot pays no runtime cost; while a series is active the guest pays KVM dirty logging. Validated: an integration test writes tmpfs markers before and after the baseline and reads both back through a restore of the series directory; unit tests cover replaying only a delta's entries and rejecting foreign, out-of-layout and truncated delta files. Allow rename in the seccomp filter of the VMM thread: the delta and the staged config and state files are installed with it. A backend with DEVICE_STATE is stopped by the state save of every snapshot, and resuming it sends SET_FEATURES without F_LOG_ALL and a new SET_MEM_TABLE whose regions have no log bitmap. A series continues after that resume, so while a log region is active keep F_LOG_ALL, send the log base again after the memory table and enable vring logging before the vrings are enabled. Otherwise backend writes never reach a delta and a restore returns stale guest memory. A hot-added region can outgrow the log, so setup then stays unlogged and the next diff reports the layout change. Signed-off-by: CMGS <ilskdw@gmail.com>
io_uring can complete an O_DIRECT read with -EAGAIN on kernels without
commit b08e02045002 ("io_uring/rw: don't gate retry on completion
context"), which was merged for Linux 6.14. This happens on XFS for a
read that spans several extents when a concurrent allocating write
holds the inode lock: the bios for the first extents are already in
flight when mapping a later extent fails with -EAGAIN, iomap completes
the request from a workqueue, and io_uring does not retry it outside
the submitting task. virtio-blk then reports VIRTIO_BLK_S_IOERR to the
guest for data that is intact.
Reissue a read or write that completes with -EAGAIN or -EINTR once,
with IOSQE_ASYNC so that io-wq runs it in blocking mode, where it
cannot fail this way again. QEMU resubmits both errors as well.
Signed-off-by: doge <me@crackerben.com>
Ejecting a vhost-user device during a diff series drops its log with the pages not yet harvested. The next diff succeeds and a restore returns stale guest memory. End the series on a removal request, and fail the next diff after any eject during dirty logging, because the guest can eject a device on its own and later than the request. vDPA and vfio-user devices track no dirty pages, and the DMA log of a VFIO device behind a virtual IOMMU has no translation to guest addresses. Fail a diff request while one of them is present. A kernel VFIO device added during a series is not logged at all, so adding one ends the series too. None of this changes full snapshots or live migration. Stop the dirty log and run the diff check without holding the device tree lock while locking devices. A vCPU moving a BAR takes these locks in the other order, and these paths now run on a running VM.
A caller that gives up on a diff series, such as an aborted migration attempt, has no way to end it without a side effect. The series stays active, dirty logging stays on and every diff to a new directory fails. PUT /vm.end-snapshot-series ends the series and stops dirty logging. It returns 204 also when no series is active, so a caller can run it at the start of every attempt.
The 16550 and PL011 models queue host input in a buffer with no limit. When a client writes faster than the guest reads, the buffer grows on the serial-manager thread until a reallocation needs mremap, which its seccomp filter rejects, and the VMM dies with SIGSYS. Cap the buffer at 4 KiB, allocated up front, and make queue_input_bytes() report how many bytes it took. serial-manager keeps the rest, stops polling the input fd and retries every 10 ms until the guest has read enough. The client then waits on the full socket or PTY buffer and no input byte is lost. Signed-off-by: CMGS <ilskdw@gmail.com>
Squash of the part of cloud-hypervisor#9122 by Isaku Yamahata that upstream has not taken yet, ported onto upstream's common VFIO BAR relocation and its GuestRegion-based shared memory BARs. The upstream PR was closed in favour of smaller ones; the commits that went in that way are left out. The ported series is kept on the pci-bar-rework-1119 branch. BAR moves are split into prepare and commit phases, a BAR is released when the guest disables its decoding, and pending installs are retried after a release or an eject. Windows guests that disable and re-enable several NICs at once swap their BARs; without this, CH refuses the move ("Failed moving device BAR ... keeping old BAR") and leaves those NICs in Code 10. Signed-off-by: Isaku Yamahata <iyamahata@crusoe.ai> Signed-off-by: CMGS <ilskdw@gmail.com>
A VM restored with memory_restore_mode=copyonwrite maps the snapshot file MAP_PRIVATE over its guest RAM. MADV_DONTNEED there only drops the private copy: the next guest touch of a released page, even a write, faults the page back in from the snapshot file with read-around. With free page reporting on, every page the guest frees and reuses pays that read. Map anonymous memory over the 2 MiB-aligned part of each released range instead, so a reused page is zero-filled without I/O. The head and tail below 2 MiB keep MADV_DONTNEED. This covers free page reporting, which reports pageblock-sized ranges. Inflate releases at most 256 pages (1 MiB) per batch, so it releases each batch with MADV_DONTNEED at once and tracks the inflated 4 KiB pages per 2 MiB chunk across batches. A chunk the guest inflates completely is remapped the same way, and a deflated page clears its bit. The tracking costs at most one 64-byte bitmap per partly inflated chunk. In the worst case, alternate 2 MiB blocks reported, that gives one mapping per 2 MiB of guest RAM, which stays under the default vm.max_map_count for guests up to about 100 GiB. The new mapping keeps the region's MAP_NORESERVE, and a failed mmap falls back to MADV_DONTNEED. Kernels before 6.12 can leave a hole if MAP_FIXED fails after the unmap. The remapped range gets MADV_DONTDUMP like the rest of guest RAM, and MADV_NOHUGEPAGE: a private copy of a file page is 4 KiB today, and hugepage memory never takes the copy-on-write restore path, so a reused page keeps costing 4 KiB instead of a 2 MiB host page. An overlay region still records itself as anonymous, so the memory manager notes that the overlay was mapped and the balloon remaps only then. OnDemand restore never remaps, which would drop the userfaultfd registration. Gates: fork CI on e6221b9 and node1 fmt, typos, clippy (4 feature sets), fuzz check and virtio-devices tests; 16-clone hardware A/B and an inflate/deflate integrity run on a testbed host. Signed-off-by: CMGS <ilskdw@gmail.com>
A copy-on-write restore maps the snapshot file MAP_PRIVATE over guest RAM, and marked that mapping MADV_HUGEPAGE when thp is on. A page the guest writes becomes a private anonymous copy of one host base page (4 KiB on x86_64) with or without the hint. On a filesystem with large folios, the hint lets read faults map PMD-sized file folios read-only, but a read fault that misses the page cache can then request readahead of two PMD-sized folios, about 4 MiB, including the faults the restore path takes. Measured on x86_64 with Linux 6.12 and XFS, 16 copy-on-write clones of a 2 GiB guest, with and without the hint: - a restore on an idle host took p50 5.6 ms instead of 8.5 ms, and p50 8.9 ms instead of 47.8 ms with 16 clones restoring at once; - random reads of read-mostly guest memory ran about 3.5% slower, since the file-backed PMD mappings (about 448 MB per clone) are gone; streaming reads and CPU-bound work were unchanged. Restore latency sits on the clone path, so drop the hint from the overlay. The reserve policy is still applied, and the docs state the trade-off. Signed-off-by: CMGS <ilskdw@gmail.com>
Restoring a snapshot advances the guest clock by the wall time since the snapshot was taken. On x86_64 with KVM this also moves the TSC, and with it every clock a guest derives from the TSC, such as the uptime of a Windows guest. Add ClockRestoreMode::SnapshotRestoreNoCatchUp for a restore that continues from the saved clock instead. On x86_64 with KVM, kvmclock is set without KVM_CLOCK_REALTIME, so KVM does not add the elapsed time, and the TSC stays at the value restored with the vCPUs. On aarch64 the virtual counter stays at its restored value. MSHV has no catch-up, so its restore is unchanged. Nothing selects the new mode yet. Link: cocoonstack#9 Signed-off-by: zili-simular <zili@simular.ai>
A Windows guest whose uptime jumps forward by 2^31 to 2^32 ms (24.9 to 49.7 days) on restore can resume with its TCP retransmission timers stopped, and a golden snapshot of that age is ordinary. Add `clock_catchup=on|off` to the restore config, on by default. With `off`, the guest clock continues from the snapshot, as if no time had passed, and the guest's wall clock is behind by the age of the snapshot until something in the guest sets it. A received migration always catches up. Link: cocoonstack#9 Signed-off-by: zili-simular <zili@simular.ai>
zili-simular
force-pushed
the
restore-clock-catchup
branch
from
November 19, 2026 09:35
7e8e84c to
f363038
Compare
CMGS
reviewed
Nov 20, 2026
CMGS
left a comment
Collaborator
There was a problem hiding this comment.
no blocker. the no-catch-up path, the restore flag and the docs look right. the clock api this builds on is upstream, so the two commits can go upstream as they are. one note inline.
| pub(crate) external_fds: ExternalFds, | ||
| } | ||
|
|
||
| impl Default for RestoreConfig { |
Collaborator
There was a problem hiding this comment.
this default now lists every field by hand. a newtype bool whose Default is true would keep #[derive(Default)] and drop the list.
Collaborator
|
landed on dev as 29bf71d, squashed to one commit. the two commits stay on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refs #9. Adds a restore option that leaves the guest clock at its saved value, for guests that cannot take a large forward jump of their monotonic clock. The default is unchanged.
vm.restore {source_url, memory_restore_mode, ...} + clock_catchup: bool = true # CLI: clock_catchup=on|off Vm::new - saved_clock.mode = SnapshotRestore + saved_clock.mode = clock_catchup ? SnapshotRestore : SnapshotRestoreNoCatchUp vm.resume restore_clock(mode) # x86_64, KVM + if mode == SnapshotRestoreNoCatchUp + KVM_SET_CLOCK without KVM_CLOCK_REALTIME # kvmclock continues from the saved value + return # TSC and TSC deadlines stay as restored with the vCPUs KVM_SET_CLOCK with KVM_CLOCK_REALTIME # KVM adds the time since the snapshot advance the TSC by the same amount, re-arm TSC deadlinesvm.restoreonly.RestoreConfigfield, OpenAPI and docs.Evidence
A Windows 10 guest restored from a snapshot that is 28 days old, on an AMD EPYC 7443P host (Ubuntu 24.04, kernel 7.0.0-31). The probe counts the SYNs the guest sends to a blackholed address in 5.5 s; a healthy Windows guest sends three. Details in #9.
clock_catchupnot sent, this branch): guest uptime 28.03 days, date correct, 1 SYN. Same as2ca4cb08.After (
clock_catchup=false, sent by fix(cloudhypervisor): a Windows clone or restore resumes from the snapshot's clock cocoon#341): guest uptime 162 s, date is the snapshot's, 3 SYNs at 0 s, 1 s and 3 s. 2 of 2 restores. AfterSet-Datein the guest the date is correct, the uptime is 168 s, and the probe still sees 3 SYNs.f3630387(ondeva8f410a1), built withcargo build --release --features fw_cfg. The same three results held on the branch before the rebase, on2ca4cb08.kvm::tests::test_restore_clock_without_catch_up_keeps_saved_clockpasses on that host, next totest_restore_clock_rearms_tsc_deadline: with a one-hour gap, neither kvmclock nor the TSC moves by the gap, and the same state restored with catch-up moves both.config::tests::test_restore_parsingandtest_restore_config_serdecoverclock_catchup=off, the JSON field and the default. Thevmmunit tests matchingrestoreandscm_rightspass on that host, 24 of 24.cargo clippy -p hypervisor -p vmm --features kvm --all-targets -- -D warnings(x86_64),cargo +nightly fmt --all -- --checkandgitlintare clean.Not run: the integration tests, the full workspace test suite, MSHV, and anything on aarch64 beyond
cargo check -p hypervisor --features kvm --tests.Merge Danger
Door: two-way
The field is additive and defaults to today's behavior. A client that sends it to a build without it is not refused:
RestoreConfigignores unknown fields, and that build catches up.Blast Radius: restore
With
clock_catchup=offthe guest's wall clock is behind by the snapshot's age until something in the guest sets it. The caller owns that.LLM disclosure: the code, tests and this description were drafted with an AI coding agent (Amp). Draft until reviewed by a human.