Skip to content

vmm: Add clock_catchup to the restore config - #10

Closed
zili-simular wants to merge 11 commits into
cocoonstack:devfrom
zili-simular:restore-clock-catchup
Closed

zili-simular wants to merge 11 commits into
cocoonstack:devfrom
zili-simular:restore-clock-catchup

Conversation

@zili-simular

@zili-simular zili-simular commented Nov 19, 2026 •

Copy link
Copy Markdown

Summary

Refs #9. Adds a restore option that leaves the guest clock at its saved value, for guests that cannot take a large forward jump of their monotonic clock. The default is unchanged.

 vm.restore {source_url, memory_restore_mode, ...}
+            clock_catchup: bool = true            # CLI: clock_catchup=on|off
   Vm::new
-    saved_clock.mode = SnapshotRestore
+    saved_clock.mode = clock_catchup ? SnapshotRestore : SnapshotRestoreNoCatchUp
 vm.resume
   restore_clock(mode)                              # x86_64, KVM
+    if mode == SnapshotRestoreNoCatchUp
+      KVM_SET_CLOCK without KVM_CLOCK_REALTIME     # kvmclock continues from the saved value
+      return                                       # TSC and TSC deadlines stay as restored with the vCPUs
     KVM_SET_CLOCK with KVM_CLOCK_REALTIME          # KVM adds the time since the snapshot
     advance the TSC by the same amount, re-arm TSC deadlines
  • aarch64: the virtual counter stays at the value restored with the vCPUs. MSHV: unchanged, it has no catch-up.
  • Migration receive, which includes a restore from an offload daemon, always catches up. The option is on vm.restore only.
  • Two commits: the hypervisor mode with its KVM test, then the RestoreConfig field, OpenAPI and docs.

Evidence

A Windows 10 guest restored from a snapshot that is 28 days old, on an AMD EPYC 7443P host (Ubuntu 24.04, kernel 7.0.0-31). The probe counts the SYNs the guest sends to a blackholed address in 5.5 s; a healthy Windows guest sends three. Details in #9.

  • Before (clock_catchup not sent, this branch): guest uptime 28.03 days, date correct, 1 SYN. Same as 2ca4cb08.
    After (clock_catchup=false, sent by fix(cloudhypervisor): a Windows clone or restore resumes from the snapshot's clock cocoon#341): guest uptime 162 s, date is the snapshot's, 3 SYNs at 0 s, 1 s and 3 s. 2 of 2 restores. After Set-Date in the guest the date is correct, the uptime is 168 s, and the probe still sees 3 SYNs.
  • Both runs used this branch at f3630387 (on dev a8f410a1), built with cargo build --release --features fw_cfg. The same three results held on the branch before the rebase, on 2ca4cb08.
  • kvm::tests::test_restore_clock_without_catch_up_keeps_saved_clock passes on that host, next to test_restore_clock_rearms_tsc_deadline: with a one-hour gap, neither kvmclock nor the TSC moves by the gap, and the same state restored with catch-up moves both.
  • config::tests::test_restore_parsing and test_restore_config_serde cover clock_catchup=off, the JSON field and the default. The vmm unit tests matching restore and scm_rights pass on that host, 24 of 24.
  • cargo clippy -p hypervisor -p vmm --features kvm --all-targets -- -D warnings (x86_64), cargo +nightly fmt --all -- --check and gitlint are clean.

Not run: the integration tests, the full workspace test suite, MSHV, and anything on aarch64 beyond cargo check -p hypervisor --features kvm --tests.

Merge Danger

Door: two-way

The field is additive and defaults to today's behavior. A client that sends it to a build without it is not refused: RestoreConfig ignores unknown fields, and that build catches up.

Blast Radius: restore

With clock_catchup=off the guest's wall clock is behind by the snapshot's age until something in the guest sets it. The caller owns that.

LLM disclosure: the code, tests and this description were drafted with an AI coding agent (Amp). Draft until reviewed by a human.

CMGS and others added 9 commits November 19, 2026 10:58
Build cloud-hypervisor on the dev branch for x86_64 and aarch64, then
publish both static binaries to a rolling dev prerelease with checksums
and build provenance.

The binaries are built with the fw_cfg feature. On aarch64 that is what
lets `--firmware` and `--kernel` be combined so EDK2 boots the kernel
through fw_cfg and the guest gets ACPI; without `--fw-cfg-config` the
feature changes nothing. The staged binary prints its enabled features
so the release log shows the flag took effect.

Signed-off-by: CMGS <ilskdw@gmail.com>
Every snapshot currently dumps the entire guest RAM, so the pause a
snapshot imposes grows with the memory size regardless of how little
the guest has changed. Iterative checkpointing and warm pre-copy (take
a snapshot, let the guest run, take another) repeat that full-memory
cost every round.

Add a diff snapshot: `vm.snapshot` accepts a `snapshot_type` of `full`
(default) or `diff`, exposed as `ch-remote snapshot --diff`. The first
diff of a series writes a full baseline and enables dirty-page
tracking; each subsequent diff writes only the pages dirtied since the
previous one, so its pause is proportional to the dirtied memory
rather than to the guest RAM size. Dirty pages are harvested after the
device snapshot, so pages touched by snapshot side effects are
included.

The whole series lives in one directory, and every diff request of a
series must target that directory. A delta `memory-ranges.diff.N`
starts with a magic, an entry count and a table of (gpa, length)
pairs, followed by the bytes of each entry, so it does not rely on the
filesystem preserving holes. It is written under a temporary name,
synced and renamed into place before `config.json` and `state.json`
are replaced; both are synced under temporary names first and then
renamed back to back. The memory manager state records how many
deltas it covers, so an interrupted diff leaves the previous snapshot
of the series restorable. The three syncs run inside the pause, in
time proportional to the dirtied memory. A full snapshot, a memory layout
change, a failed snapshot, a migration, a reboot, a shutdown or deleting
the VM ends the series and stops dirty logging.

Restore needs no new parameter: memory fills from the baseline (in any
restore mode except ondemand, which a series rejects), then the deltas
the state counts replay in order. A missing delta, or one whose table
does not fit the snapshot's memory layout or its own file size, fails
the restore before that delta touches memory.

Dirty tracking is enabled lazily on the first diff rather than at
boot, so a VM that never takes a diff snapshot pays no runtime cost;
while a series is active the guest pays KVM dirty logging.

Validated: an integration test writes tmpfs markers before and after
the baseline and reads both back through a restore of the series
directory; unit tests cover replaying only a delta's entries and
rejecting foreign, out-of-layout and truncated delta files.

Allow rename in the seccomp filter of the VMM thread: the delta and the
staged config and state files are installed with it.

A backend with DEVICE_STATE is stopped by the state save of every
snapshot, and resuming it sends SET_FEATURES without F_LOG_ALL and a new
SET_MEM_TABLE whose regions have no log bitmap. A series continues after
that resume, so while a log region is active keep F_LOG_ALL, send the log
base again after the memory table and enable vring logging before the
vrings are enabled. Otherwise backend writes never reach a delta and a
restore returns stale guest memory. A hot-added region can outgrow the log,
so setup then stays unlogged and the next diff reports the layout change.

Signed-off-by: CMGS <ilskdw@gmail.com>
io_uring can complete an O_DIRECT read with -EAGAIN on kernels without
commit b08e02045002 ("io_uring/rw: don't gate retry on completion
context"), which was merged for Linux 6.14. This happens on XFS for a
read that spans several extents when a concurrent allocating write
holds the inode lock: the bios for the first extents are already in
flight when mapping a later extent fails with -EAGAIN, iomap completes
the request from a workqueue, and io_uring does not retry it outside
the submitting task. virtio-blk then reports VIRTIO_BLK_S_IOERR to the
guest for data that is intact.

Reissue a read or write that completes with -EAGAIN or -EINTR once,
with IOSQE_ASYNC so that io-wq runs it in blocking mode, where it
cannot fail this way again. QEMU resubmits both errors as well.

Signed-off-by: doge <me@crackerben.com>
Ejecting a vhost-user device during a diff series drops its log with the
pages not yet harvested. The next diff succeeds and a restore returns
stale guest memory. End the series on a removal request, and fail the
next diff after any eject during dirty logging, because the guest can
eject a device on its own and later than the request.

vDPA and vfio-user devices track no dirty pages, and the DMA log of a
VFIO device behind a virtual IOMMU has no translation to guest addresses.
Fail a diff request while one of them is present. A kernel VFIO device
added during a series is not logged at all, so adding one ends the series
too. None of this changes full snapshots or live migration.

Stop the dirty log and run the diff check without holding the device tree
lock while locking devices. A vCPU moving a BAR takes these locks in the
other order, and these paths now run on a running VM.
A caller that gives up on a diff series, such as an aborted migration
attempt, has no way to end it without a side effect. The series stays
active, dirty logging stays on and every diff to a new directory fails.

PUT /vm.end-snapshot-series ends the series and stops dirty logging. It
returns 204 also when no series is active, so a caller can run it at the
start of every attempt.
The 16550 and PL011 models queue host input in a buffer with no
limit. When a client writes faster than the guest reads, the buffer
grows on the serial-manager thread until a reallocation needs
mremap, which its seccomp filter rejects, and the VMM dies with
SIGSYS.

Cap the buffer at 4 KiB, allocated up front, and make
queue_input_bytes() report how many bytes it took. serial-manager
keeps the rest, stops polling the input fd and retries every 10 ms
until the guest has read enough. The client then waits on the full
socket or PTY buffer and no input byte is lost.

Signed-off-by: CMGS <ilskdw@gmail.com>
Squash of the part of cloud-hypervisor#9122 by Isaku
Yamahata that upstream has not taken yet, ported onto upstream's common
VFIO BAR relocation and its GuestRegion-based shared memory BARs. The
upstream PR was closed in favour of smaller ones; the commits that went
in that way are left out. The ported series is kept on the
pci-bar-rework-1119 branch.

BAR moves are split into prepare and commit phases, a BAR is released
when the guest disables its decoding, and pending installs are retried
after a release or an eject. Windows guests that disable and re-enable
several NICs at once swap their BARs; without this, CH refuses the move
("Failed moving device BAR ... keeping old BAR") and leaves those NICs
in Code 10.

Signed-off-by: Isaku Yamahata <iyamahata@crusoe.ai>
Signed-off-by: CMGS <ilskdw@gmail.com>
A VM restored with memory_restore_mode=copyonwrite maps the snapshot
file MAP_PRIVATE over its guest RAM. MADV_DONTNEED there only drops the
private copy: the next guest touch of a released page, even a write,
faults the page back in from the snapshot file with read-around. With
free page reporting on, every page the guest frees and reuses pays that
read.

Map anonymous memory over the 2 MiB-aligned part of each released range
instead, so a reused page is zero-filled without I/O. The head and tail
below 2 MiB keep MADV_DONTNEED. This covers free page reporting, which
reports pageblock-sized ranges. Inflate releases at most 256 pages (1
MiB) per batch, so it releases each batch with MADV_DONTNEED at once and
tracks the inflated 4 KiB pages per 2 MiB chunk across batches. A chunk
the guest inflates completely is remapped the same way, and a deflated
page clears its bit. The tracking costs at most one 64-byte bitmap per
partly inflated chunk. In the worst case, alternate 2 MiB blocks
reported, that gives one mapping per 2 MiB of guest RAM, which stays
under the default vm.max_map_count for guests up to about 100 GiB. The
new mapping keeps the region's MAP_NORESERVE, and a failed mmap falls
back to MADV_DONTNEED. Kernels before 6.12 can leave a hole if MAP_FIXED
fails after the unmap.

The remapped range gets MADV_DONTDUMP like the rest of guest RAM, and
MADV_NOHUGEPAGE: a private copy of a file page is 4 KiB today, and
hugepage memory never takes the copy-on-write restore path, so a reused
page keeps costing 4 KiB instead of a 2 MiB host page.

An overlay region still records itself as anonymous, so the memory
manager notes that the overlay was mapped and the balloon remaps only
then. OnDemand restore never remaps, which would drop the userfaultfd
registration.

Gates: fork CI on e6221b9 and node1 fmt, typos, clippy (4 feature
sets), fuzz check and virtio-devices tests; 16-clone hardware A/B and
an inflate/deflate integrity run on a testbed host.

Signed-off-by: CMGS <ilskdw@gmail.com>
A copy-on-write restore maps the snapshot file MAP_PRIVATE over guest
RAM, and marked that mapping MADV_HUGEPAGE when thp is on. A page the
guest writes becomes a private anonymous copy of one host base page
(4 KiB on x86_64) with or without the hint. On a filesystem with large
folios, the hint lets read faults map PMD-sized file folios read-only,
but a read fault that misses the page cache can then request readahead
of two PMD-sized folios, about 4 MiB, including the faults the restore
path takes.

Measured on x86_64 with Linux 6.12 and XFS, 16 copy-on-write clones of
a 2 GiB guest, with and without the hint:
- a restore on an idle host took p50 5.6 ms instead of 8.5 ms, and
  p50 8.9 ms instead of 47.8 ms with 16 clones restoring at once;
- random reads of read-mostly guest memory ran about 3.5% slower,
  since the file-backed PMD mappings (about 448 MB per clone) are
  gone; streaming reads and CPU-bound work were unchanged.

Restore latency sits on the clone path, so drop the hint from the
overlay. The reserve policy is still applied, and the docs state the
trade-off.

Signed-off-by: CMGS <ilskdw@gmail.com>
Restoring a snapshot advances the guest clock by the wall time since
the snapshot was taken. On x86_64 with KVM this also moves the TSC, and
with it every clock a guest derives from the TSC, such as the uptime of
a Windows guest.

Add ClockRestoreMode::SnapshotRestoreNoCatchUp for a restore that
continues from the saved clock instead. On x86_64 with KVM, kvmclock is
set without KVM_CLOCK_REALTIME, so KVM does not add the elapsed time,
and the TSC stays at the value restored with the vCPUs. On aarch64 the
virtual counter stays at its restored value. MSHV has no catch-up, so
its restore is unchanged.

Nothing selects the new mode yet.

Link: cocoonstack#9
Signed-off-by: zili-simular <zili@simular.ai>
A Windows guest whose uptime jumps forward by 2^31 to 2^32 ms (24.9 to
49.7 days) on restore can resume with its TCP retransmission timers
stopped, and a golden snapshot of that age is ordinary.

Add `clock_catchup=on|off` to the restore config, on by default. With
`off`, the guest clock continues from the snapshot, as if no time had
passed, and the guest's wall clock is behind by the age of the snapshot
until something in the guest sets it. A received migration always
catches up.

Link: cocoonstack#9
Signed-off-by: zili-simular <zili@simular.ai>

@CMGS CMGS left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no blocker. the no-catch-up path, the restore flag and the docs look right. the clock api this builds on is upstream, so the two commits can go upstream as they are. one note inline.

Comment thread vmm/src/config.rs
pub(crate) external_fds: ExternalFds,
}

impl Default for RestoreConfig {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this default now lists every field by hand. a newtype bool whose Default is true would keep #[derive(Default)] and drop the list.

@CMGS

CMGS commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

landed on dev as 29bf71d, squashed to one commit. the two commits stay on clock-catchup, on top of upstream main, for upstream.

@CMGS CMGS closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants