Skip to content

test: wait for the Autobahn server to start - #599

Open
mafredri wants to merge 8 commits into
masterfrom
mafredri/autobahn-startup-wait
Open

mafredri wants to merge 8 commits into
masterfrom
mafredri/autobahn-startup-wait

Conversation

@mafredri

@mafredri mafredri commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

TestAutobahn gave the wstest container 5s after docker run to accept a WebSocket handshake. On GitHub's ubuntu-26.04 runners that is often not enough, and the daily test jobs fail with autobahn_test.go:72: context deadline exceeded (#598). Since the daily jobs moved to 26.04 on 2026-10-30, 13 of 27 failed this way.

The image is already pulled when the wait starts. The time goes to the first container start after docker pull: on 26.04 (Docker 29.4.2 with the containerd image store), that start took about 2.8s on AMD EPYC 7763 VMs and 2.9s to 7.3s on EPYC 9V45, EPYC 9V74 and Xeon 8573C VMs. wstest starts listening about 1.1s later. Under the full ci/test.sh load, the server took 2.4s to 13.5s to accept a handshake. On 24.04 (Docker 28.0.4), the same start took under 0.5s on every VM type. The daily failures follow the VM type: 0 of 11 jobs on EPYC 7763 and 0 of 1 on Xeon 8370C failed, and 13 of 15 on EPYC 9V45, EPYC 9V74 and Xeon 8573C.

wstestServer now returns once the server accepts a handshake. It dials every 100ms instead of in a busy loop, fails as soon as docker run exits, and gives up after 2 minutes.

Killing the docker run client does not stop the container, so each run left a wstest container running. closeFn now removes the container and its anonymous volumes. The container gets a random name, because another run can get the same port.

Refs #598

Verification
  • Startup timings on runners: 43594056234 (24.04 and 26.04 under the test load), 43594595034 (first start without load), 43594994307 (any image or network mode), 43595595196 (time since pull, not since boot), 43598451098 (runner image and VM type).
  • AUTOBAHN=1 make test on ubuntu-26.04 with earlier versions of this branch: 43599366425 (5 minute limit, same container removal), 8 of 8 passed (5 on the slower VM types), with no container left afterwards. 43595298461 (1 minute limit), 6 of 6 passed. Master's test in the same run passed 5 of 6 and failed once with the 5s error.
  • The daily workflow on this branch: 43606350993 passed on an earlier head, 44090002726 on the current head.
  • Locally, AUTOBAHN=fast passes and leaves no container or volume. With an invalid wstest mode, the test fails after 1.8s with wstest exited: exit status 1. With the limit set to 3s and an unused port, it fails with timed out after 3s and removes the container.
  • make lint passed.
Decision log
  • 2 minutes: the wait covers only the container start and wstest's startup, not a download. The longest measured time to accept a handshake was 13.5s. A server that fails to start makes docker run exit, which ends the wait right away.
  • What Docker 29 does during the delay is not known. It happens before containerd starts the container's shim, for any image and network mode, and not when 60s pass between the pull and the run. The test does not work around it with a throwaway container or a sleep after the pull, because that would build one Docker version's behavior into the test.
  • Not pinning the daily jobs to ubuntu-24.04: that hides the slow start until 24.04 is retired.
  • The busy loop did not cause the delay: dialing every 50ms gave the same 2.6s to 8.7s start times. It made 15k to 64k dials per startup.

🤖 This PR was created with the help of Coder Agents.

@mafredri
mafredri marked this pull request as ready for review November 14, 2026 16:45
@mafredri

Copy link
Copy Markdown
Member Author

/coder-agents-review

@coder-agents-review

coder-agents-review Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Chat: Review posted | View chat
Requested: 2026-11-17 12:41 UTC by @mafredri
Spend: $7.95 / $100.00

Review history
  • R1 (2026-11-17): 11 reviewers, 2 P3, 2 P4, COMMENT. Review

deep-review v0.13.0 | Round 1 | 7463847..1f8a5d0

Last posted: Round 1, 4 findings (2 P3, 2 P4), COMMENT. Review

Finding inventory

Finding inventory, PR #599

Findings

# Sev Status Location Summary Round Reviewer Posted
CRF-1 P3 Open autobahn_test.go:217 docker rm --force without --volumes leaves the image's two anonymous volumes on every run R1 Mafuuu Yes
CRF-2 P3 Open autobahn_test.go:99 waitWS doc says it waits until ctx is done but omits its own 2 minute limit R1 Gon P3, Leorio Nit Yes
CRF-3 P4 Open autobahn_test.go:114 Timeout error says only "context deadline exceeded", not that the 2 minute startup limit expired R1 Leorio Yes
CRF-4 P4 Open autobahn_test.go:176 Port-derived container name can collide across concurrent runs; the failing run's closeFn removes the other run's container R1 Kite Yes
CRF-5 P4 Open PR description "13 of 27 failed" does not match the per-VM breakdown, which totals 26 jobs R1 Leorio Yes (body, no diff line)
CRF-6 Note Dropped by orchestrator (informational, asks for no change) autobahn_test.go:216 closeFn no longer reports the docker run exit status; mid-run crash still fails via case dials R1 Netero No
CRF-7 P4 Dropped by orchestrator (reason sits in the doc comment 3 lines above; file uses inline durations) autobahn_test.go:103 2 minute limit as a named constant R1 Gon No
CRF-8 P4 Dropped by orchestrator (PR measured that dial rate does not affect startup, so changing it has no harmful consequence) autobahn_test.go:115 100ms dial interval has no stated reason R1 Gon No
CRF-9 Nit Dropped by orchestrator (no project naming rule; the single call site reads clearly) autobahn_test.go:206 Rename exited to cancelWait R1 Gon No
CRF-10 P4 Dropped by orchestrator (title matches repository title style, e.g. #584; squash body describes container removal) PR title Title omits container removal R1 Leorio No
CRF-11 Nit Dropped by orchestrator (comment is adjacent to the waitCtx it describes and reads correctly) autobahn_test.go:204 Reword "the wait ends then" comment R1 Leorio No
CRF-12 OOS Out of scope (body) main_test.go:23 TestMain goroutine check races the exec.CommandContext watchCtx goroutine (line 189); TestAutobahn run alone fails intermittently, base too R1 Komugi, Killua No
CRF-13 OOS Out of scope (body) autobahn_test.go:63 Deferred closeFn does not run on test timeout panic or SIGINT, so the container keeps running R1 Killua, Bisky No
CRF-14 OOS Out of scope (body) autobahn_test.go:147 1 hour context cancel is called only on error R1 Mafuuu No
CRF-15 OOS Out of scope (body) autobahn_test.go:295 unusedListenAddr closes the listener before docker binds the port R1 Komugi No
CRF-16 OOS Out of scope (body) autobahn_test.go:77 i := i is unneeded since Go 1.22 R1 Netero, Ging-Go No
CRF-17 OOS Out of scope (body) autobahn_test.go:63 context.Background with 1 hour timeout instead of t.Context R1 Ging-Go No
CRF-18 OOS Out of scope (body) autobahn_test.go:185 --webport=0 comment does not say which server it disables R1 Gon No

Contested and acknowledged

None.

Round log

Round 1

Full panel (round 1). Netero: no findings, 1 Note. Law not run (+45 effective). Panel: ging-go, takumi, killua, ryosuke (Go specialists), pariston, mafuuu (core), bisky (test code), gon, leorio (once-per-PR floor), komugi (flake fix), kite (wildcard). Size 11 exceeds the focused target of 6-8 because 4 specialists, Bisky, and the Gon/Leorio floor are required. 2 P3, 3 P4 open; 6 dropped; 7 out of scope. Orchestrator reproduced CRF-1 on Docker 27.5.1 (docker rm --force keeps both volumes, --force --volumes removes them). Komugi P3 on the watchCtx goroutine moved to out of scope: base fails the same way and the fix line (189) is unchanged. Reviewed against 7463847..1f8a5d0.

About deep-review

CRF = Coder Review Finding (P0-P4, Nit, Note)

Reviewer Focus
Bisky tests
Chopper ops/errors
Churn-guard change verification
Ging language modernization
Gon naming
Hisoka edge cases
Killua perf
Kite change integrity
Knov contracts
Knuckle SQL
Komugi flake/determinism
Kurapika security
Law decomposition
Leorio docs
Luffy product
Mafu-san process
Mafuuu contracts
Melody dispatch/pairing
Meruem structural
Nami frontend
Netero mechanical checks
Pariston premise testing
Pen-botter product gaps
Razor verification
Robin duplication
Ryosuke Go arch
Takumi concurrency
Zoro shape

🤖 Managed by Coder Agents.

coder-agents-review[bot]

This comment was marked as outdated.

TestAutobahn gave the wstest container 5s to accept a handshake after
docker run. Since the daily jobs moved to ubuntu-26.04 (Docker 29.4.2),
the first container start on a fresh runner takes 2.8s to 12s, then
wstest needs about 1s more to listen. On 24.04 (Docker 28.0.4) the
first start took under 0.5s. The test failed in 13 of 27 daily test jobs
since 2026-10-30.

waitWS now allows a minute, sleeps 100ms between dials instead of
dialing in a busy loop, and returns the last dial error.
Killing the docker run client leaves the container running, so each
TestAutobahn run left a wstest container behind. closeFn now removes
the container by name.

wstestServer now returns once the server accepts handshakes, and stops
waiting when docker run exits, so a server that fails to start fails
the test right away. That lets the wait allow 5 minutes for a slow
start.
The image is pulled before docker run, so the wait covers only the
container start and wstest's startup, which took at most 13.5s on
the runners measured. 2 minutes is enough for that.

docker run exiting now cancels the wait's context with the exit error
as its cause, so waitWS needs no separate channel.
docker rm --force keeps the image's anonymous /config and /reports
volumes, which --rm would remove, so every run left two volumes.
closeFn now passes --volumes.

The container name came from the port, and unusedListenAddr releases
the port before docker binds it, so another run could get the same
name, and a failing run's closeFn would remove the other run's
container. The name is now random.

The 2 minute limit moves from waitWS to wstestServer, next to the
measurement that explains it, and its cause names the limit in the
error instead of a second "context deadline exceeded".
@mafredri
mafredri force-pushed the mafredri/autobahn-startup-wait branch from 1f8a5d0 to 6742435 Compare November 17, 2026 13:10
Comment thread autobahn_test.go Resolved
Comment thread autobahn_test.go Resolved
Comment thread autobahn_test.go Resolved
@mafredri
mafredri requested a review from code-asher November 17, 2026 14:26

@code-asher code-asher left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants