Skip to content

test(ci): measure the slow tier with #806's CI changes alone - #812

Closed
earakely-scale wants to merge 7 commits into
mainfrom
edgararakelyan/measure-ci-time
Closed

earakely-scale wants to merge 7 commits into
mainfrom
edgararakelyan/measure-ci-time

Conversation

@earakely-scale

Copy link
Copy Markdown
Collaborator

Measurement only: not for review or merge. I'll close it once CI has run.

This is main plus #806's three CI commits and nothing else:

  • a composed disk whose base has no published disk takes the recipe's partition, with no base disk made;
  • computer fixtures compose containers only, as agent-env run does for local Docker;
  • split_in cuts the three longest slow-tier files in two, and the durations table is refreshed from main's last green run.

#806 also changes the ubuntu-24.04 base, so its CI builds the base cold in every shard. This branch leaves the base unchanged, so its slow tier runs as main's would with these changes. It measures them.

🤖 Generated with Claude Code

earakely-scale and others added 3 commits November 19, 2026 01:02
…ipe's partition, with no base disk made

A computer's disk keeps only its base disk's EFI system partition and size. With no published base disk, as on
every run after the base's sources change until a release publishes them, the composer made a whole base disk
first: the base's root filesystem written into a 16 GB disk and exported, then converted back to raw so the
composed disk could cut out its 64 MiB partition. That took 4 to over 10 minutes on CI's runners, enough to time
out the slow tier's composing fixtures. The composed disk now takes the recipe's own partition (its uki stage) at
the base's size, which is what the made base disk held; the disk's key is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… does, its containers alone

Each computer fixture composed every form of its base: the container and the disk of each arch, the native ones
probed by gate 3, though only the launcher's tests boot a fixture's disk image. Building and booting a disk added
about 2 minutes to each of the slow tier's fixtures. They now compose as `agent-env run` does for local Docker
(Composing.for_run), so a test that deploys the disk form composes it first, as a run's deploy would; the launcher's
tests (and the Modal disk tests) take desktop_with_disk, which composes the disk too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ach, and refresh the durations table

Each slow-tier shard ran a file's tests together, so the three longest files, chromium_app_test (about 15
minutes on main), browser_multi_env_test (14) and browser_computer_test (14), each held a shard to at least
that long whatever the rest weighed. The durations table can now name a file in split_in, which cuts it into
that many runs of neighbouring tests, each charged the setup of the file's module fixtures that its runner
does anew, and these three are cut in two. With main's latest green run's times, rewritten into the table by
slow_test_durations.py from all of its JUnit reports, the slowest shard of twelve goes from 14.8 to 11.9
minutes of tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale and others added 2 commits November 19, 2026 02:16
… boots it alone

A computer app env that names a computer and a driver, as agentenv's chromium does, was composed alone whenever a
bundle wrote it, though most runs deploy it only on a desktop, which composes it into its own image. A run's
compose now leaves it to the first deploy that boots it alone (compose_unstarted), as it leaves a form no deploy
boots; a register, with or without a --sandbox, still composes it, since others deploy what it writes. In CI,
composing chromium alone took 2 to 10 minutes in each shard whose fixtures used it only as a desktop's app.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pytest-xdist's loadfile hands a worker the next file as it frees, in collected order, so a shard whose longest
file was collected last ran it last: shard 10 of a run held 20 minutes of tests, about 10 for each of its two
workers, and took 13.4, its 7-minute run_bundle_claude_code_test starting late. The shard hook now orders a
shard's files by the seconds the durations table gives them, heaviest first, each file's tests together in their
collected order, which packs the two workers as evenly as the files allow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale added a commit that referenced this pull request Nov 19, 2026
…uns of the new fixtures

The durations table was main's, whose fixtures composed disks and every app alone, and where a file that reused
a computer another file on its shard had composed looked short: browser_clock_test weighed 2 minutes and took 8
on a shard where it composed chromium alone first. The table is now each unit's longer time over two runs of
these changes (#812's 44504100677 and 44506657938), each written by slow_test_durations.py from all of a run's
JUnit reports, so a file is weighed as it costs where it composes its own computers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale and others added 2 commits November 19, 2026 02:50
…uns of the new fixtures

The durations table was main's, whose fixtures composed disks and every app alone, and where a file that reused
a computer another file on its shard had composed looked short: browser_clock_test weighed 2 minutes and took 8
on a shard where it composed chromium alone first. The table is now each unit's longer time over two runs of
these changes (#812's 44504100677 and 44506657938), each written by slow_test_durations.py from all of a run's
JUnit reports, so a file is weighed as it costs where it composes its own computers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A computer app that names a computer and a driver, as agentenv's chromium does, was composed alone by every
fixture whose bundle put it on a desktop, though those fixtures deploy it only on the desktop, which composes it
into its own image: 2 to 10 minutes of each such shard. While a fixture writes its bundle, composer.compose now
composes no computer app alone but the fixture's own (env_id, as the browser computer fixture's is chromium), or
every one with disks; a test that deploys another alone has its deploy compose it first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale

Copy link
Copy Markdown
Collaborator Author

Measurement done: the numbers are in #806's description. With #806's CI changes alone on main (no base change), the Local backends run took 12.9 minutes (slow shards 8.1–12.7) in run 44513863413, against about 17 on main's last green run. Closing; the branch stays for reference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant