Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 17 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ When it ends:

## Status

Week 1 (Oct 19 to 23, 2026), plus the first part of week 2.
Through week 4 (Nov 2026): the runner, the sandbox, the engine and crash resume.

| Piece | File | State |
| --- | --- | --- |
Expand All @@ -82,7 +82,9 @@ Week 1 (Oct 19 to 23, 2026), plus the first part of week 2.
| `approval.requested` after `task.finished` | [runner/src/runner.ts](runner/src/runner.ts) | Done (contracts v0.2.0). Real and fake runners, never after a stop or a failure |
| Repo memory: notes read before a run, a note added after | [runner/src/memory/](runner/src/memory/) | Done. `MemoryStore` with an in-memory stub and a file store. Stopped and failed runs add nothing, and a memory error never fails a run |
| Repo profile: language, package manager, test command, Node version | [runner/src/memory/profile.ts](runner/src/memory/profile.ts) | Done. Read from files in the worktree, no model call. The sandbox uses it to pick the image and decide whether to install |
| Sandbox: run a repo's tests in a locked Docker box | [runner/src/runtime/](runner/src/runtime/) | Done. `runInSandbox()`: no network, 1 GB with no swap, 1 CPU, 256 processes, read-only repo and system, non-root, your timezone, 60 s. npm only. Nothing calls it yet: the engine's test step will |
| Sandbox: run a repo's tests in a locked Docker box | [runner/src/runtime/](runner/src/runtime/) | Done. `runInSandbox()`: no network, 1 GB with no swap, 1 CPU, 256 processes, read-only repo and system, non-root, your timezone, 60 s. npm only. The engine's test step calls it |
| Engine: analyze → plan → implement → test → review | [runner/src/engine/](runner/src/engine/) | Done. `createEngine()` is a `Runner` for `agent: "codehands"`. Claude Code does implement through the runner, the sandbox does test, and one model call each does analyze, plan and review. Once the change exists, nothing after it throws it away: failing tests or a failed review still end in `awaiting_approval` |
| Crash resume: kill the daemon, it finishes anyway | [runner/src/engine/run-store.ts](runner/src/engine/run-store.ts) | Done. The engine saves `run.json` before every event. After a restart, `engine.recover()` lists the runs a crash cut off, and `resume()` carries on from the first step that didn't finish: no step is paid for twice, Claude Code's own session is continued, `seq` never repeats, and `approve` waits for test and review |

Next, from the [design](https://github.com/devweekends/codehands-docs#who-owns-what): a model per phase, naming provider errors as one `provider.status` event (`rate_limited`, `auth_expired`, `usage_limit`, `subscription_inactive`, `down`), memory and user preferences, and two models on one task.

Expand Down Expand Up @@ -119,7 +121,10 @@ const { branch, commit } = await runner.approve(sessionId, "Fix the date bug");
| `createFakeRunner({ delayMs? })` | Emits 12 scripted events, 300 ms apart, then `approval.requested`, and supports every method including stop and resume. Costs nothing, needs no repo. Build against it, and keep it as the demo backup |
| `EventSequence` | Hands out events with the next `seq` |
| `SessionStore`, `SessionRecord` | Session records with the legal-move check from the contracts. Saved as files when given a root, otherwise in memory (the fake runner) |
| `RecoverableRunner`, `RecoveryReport` | What `createRunner` returns: the contract's `Runner` plus `recover()`, and what `recover()` reports |
| `RecoverableRunner`, `RecoveryReport`, `RecoverOptions` | What `createRunner` and `createEngine` return: the contract's `Runner` plus `recover()`, `worktree()` and `status()`. `recover({ keep })` keeps a running session its owner will continue as `stopped` instead of failing it, and lists it in `report.resumable` |
| `createEngine(options?)`, `EngineOptions`, `modelFor`, `DEFAULT_MODEL` | The engine. `runner`, `model`, `sandbox`, `memory` and `runsRoot` can each be swapped, which is how the tests run it for $0. `modelFor(o, step)`: the step's own model, then the run's, then the default |
| `RunStore`, `RunState` | The engine's `run.json`: each finished step's result, the run's status, and the last `seq` sent |
| `createClaudeModel()`, `createCannedModel(answers)`, `ModelClient` | Who answers analyze, plan and review: `claude -p` with no tools, or fixed text for tests and `--free` |
| `branchName(prompt, sessionId)`, `slugify(text)` | `codehands/fix-the-failing-test-3f2a9c1e` |
| `MemoryStore`, `createMemoryStub()`, `createFileMemory(root?)` | Repo memory: `notes`, `addNote`, `profile`, `saveProfile`. The stub keeps it in a `Map` for tests |
| `formatNote`, `runNote`, `trimNotes`, `notesForPrompt` | How a note is written, built from a run, trimmed to about 200 lines, and capped at 6,000 characters for the prompt |
Expand All @@ -135,6 +140,8 @@ The daemon owns the real database. The runner keeps only what it needs to finish

When the daemon starts it should call `recover()` once, before anything else. Sessions still marked `queued` or `running` become `failed`, and their ids come back in `report.failed`. Failed sessions lose their worktree, along with anything uncommitted in it, because `failed` is final and nothing else would remove it. Their branch goes too, unless the agent committed to it. `awaiting_approval` and `stopped` sessions are left alone. A worktree that can't be removed comes back in `report.cleanupErrors` and is tried again next time.

**With the engine (Midhat):** on startup call `engine.recover()`, not `runner.recover()` (the engine calls the runner's itself), then `resume()` every id in `report.resumable`. Each resume is a normal event stream: store its events like any run's.

### How Claude Code's output becomes events

Each stdout line from `claude -p --output-format stream-json --verbose` is one JSON object. The parser in [claude-code.ts](runner/src/agents/claude-code.ts) maps it like this:
Expand Down Expand Up @@ -168,6 +175,8 @@ Some details that matter:
| `~/.codehands/memory/<repoKey>/notes.md` | Per-repo notes, newest last, trimmed to about 200 lines by dropping the oldest whole notes |
| `~/.codehands/memory/<repoKey>/profile.json` | The repo profile from the latest run |
| `~/.codehands/sessions/<sessionId>.json` | What the runner needs to finish a session after a restart |
| `~/.codehands/runs/<sessionId>/run.json` | The engine's saved state: each finished step's result, the status and the last `seq`. Written before every event, through a temp file, so a crash never leaves half of it |
| `~/.codehands/runs/<sessionId>/<n>-<step>.json` | One file per engine step: what went in, what came out, model, cost and time |
| `~/.codehands/runs/<sessionId>/test-<n>.json` | Each sandbox test run: passed, exit code, timed out, out of memory, the last 64 KB of output |
| Docker containers `ch-<first 8 of sessionId>-<n>` | One per sandbox run, removed right after. `-install` on the end for the install box |
| `codehands/<slug>-<first 8 of sessionId>` | The task's branch, in your repo |
Expand Down Expand Up @@ -219,6 +228,8 @@ Run these in `runner/`.
| `pnpm sandbox [repoPath] [--tz <zone>]` | Runs a repo's tests in the locked box once and prints the result. Defaults to the fixture. `--tz UTC` shows why the box needs your timezone | No |
| `pnpm fake` | Prints the fake runner's events, one JSON object per line. This is exactly what the daemon receives | No |
| `pnpm try [repoPath] [prompt] [--approve \| --reject]` | One real run through the runner, one line per event, then the diff stat, then approve, reject or keep | Yes, capped at $0.50 |
| `pnpm engine [repoPath] [prompt] [--free | --canned] [--wrong-fix] [--slow] [--approve | --reject]` | One run through the engine, one event per line. `--free` costs nothing: canned thinking, the recorded Claude run as implement, and the real Docker box. `--canned` pays only for implement. `--wrong-fix` shows failing tests. `--slow` waits 10 s in every step so you can kill it halfway | Only without `--free` |
| `pnpm engine --recover [--free | --canned] [--approve | --reject]` | What the daemon does after a restart: finds the runs a crash cut off and finishes them. Use the same flags the run started with | Only without `--free` |
| `pnpm build` | Compiles to `dist/` | No |
| `pnpm typecheck` | Type check only | No |

Expand All @@ -236,7 +247,9 @@ The scripts in `runner/scripts/`:
| File | Covers |
| --- | --- |
| [claude-parser.test.ts](runner/test/claude-parser.test.ts) | The whole recorded run replays in order; each mapping above; blocked commands; failed tests hidden behind exit code 0; error results; broken lines; `raw` on every event |
| [runner.test.ts](runner/test/runner.test.ts) | A worktree on a new branch with your repo untouched; clean failures for a repo with no commits, an agent with no result line, a missing binary and an unsupported agent; diff with new files; approve commits on the branch and leaves `main` alone; reject deletes the branch; nothing to approve; stop kills the agent and its children, keeps the worktree and refuses a session that is not running; resume continues the numbering, passes the prompt, can be stopped again and refuses a session that is not stopped; `approval.requested` right after `task.finished`; memory: the second run gets the first run's note, the profile is saved, and failed or stopped runs add no note; after a restart: approve, reject, diff and resume still work, and `recover()` fails interrupted sessions, removes their worktrees, keeps branches with commits, leaves live sessions alone, and reports what it can't clean or read |
| [runner.test.ts](runner/test/runner.test.ts) | A worktree on a new branch with your repo untouched; clean failures for a repo with no commits, an agent with no result line, a missing binary and an unsupported agent; diff with new files; approve commits on the branch and leaves `main` alone; reject deletes the branch; nothing to approve; stop kills the agent and its children, keeps the worktree and refuses a session that is not running; resume continues the numbering, passes the prompt, can be stopped again and refuses a session that is not stopped; `approval.requested` right after `task.finished`; memory: the second run gets the first run's note, the profile is saved, and failed or stopped runs add no note; after a restart: approve, reject, diff and resume still work, and `recover()` fails interrupted sessions, removes their worktrees, keeps branches with commits, leaves live sessions alone, and reports what it can't clean or read; `recover({ keep })` keeps a session as `stopped` and it resumes; `status()` |
| [engine.test.ts](runner/test/engine.test.ts) | The five steps in order ending in `awaiting_approval`; Claude Code's events tagged with the step; each step gets the one before; model per step; a file per step and `run.json`; wrong agent and a repo with no commits; analyze, implement, tests, sandbox and review failing, with the change kept once it exists; Claude Code's own failure says why; approve waits for the whole run; reject; stop during implement and resume; tests run in the worktree with the change |
| [engine-recover.test.ts](runner/test/engine-recover.test.ts) | A crash at every point: during analyze, after plan, during implement (Claude Code's session continued), right after Claude Code finished, during review, between `task.finished` and `approval.requested`. Each time no finished step runs again and `seq` carries on. Approve refused until the run finishes; finished and failed runs not resumed; a run stopped before the restart still resumes; a second resume refused; `RunStore` never reads outside its folder |
| [session-store.test.ts](runner/test/session-store.test.ts) | A new store finds what the old one saved; the pid is never saved; moves are still checked after loading; ids that aren't plain names never reach the disk; no root means memory only |
| [fake-runner.test.ts](runner/test/fake-runner.test.ts) | Event order and `seq`; approve only after finishing; stop then resume with `seq` continuing; no `approval.requested` after a stop |
| [memory.test.ts](runner/test/memory.test.ts) | The stub keeps notes in order and repos apart; trimming drops whole notes and keeps the newest; the prompt cap keeps the newest end; a note's summary stays on one line; the file store reads back from disk and treats missing files as empty |
Expand Down
77 changes: 57 additions & 20 deletions runner/scripts/engine.ts
Original file line number Diff line number Diff line change
@@ -1,57 +1,94 @@
// pnpm engine [repoPath] [prompt] [--free | --canned] [--wrong-fix] [--approve | --reject]: one run through the engine, one event per line.
// pnpm engine [repoPath] [prompt] [--free | --canned] [--wrong-fix] [--slow] [--approve | --reject]: one run, one event per line.
// pnpm engine --recover [--free | --canned] [--approve | --reject]: what the daemon does after a restart.
// It finds the runs a crash cut off and finishes them.
// --canned answers analyze, plan and review from canned text, so only the implement step costs money.
// --free costs nothing: canned thinking, your recorded Claude run as implement (it also makes the real
// one-line fix), and the real Docker box. Add --wrong-fix to see what failing tests look like.
// --slow waits 10 seconds in every thinking step and before Claude Code answers, so you can kill it halfway.
// Without --approve or --reject it keeps the worktree. Run scripts/reset-fixture.sh afterwards.
import { randomUUID } from "node:crypto";
import { homedir } from "node:os";
import { join } from "node:path";
import { setTimeout as sleep } from "node:timers/promises";
import { fileURLToPath } from "node:url";
import { createCannedModel, createEngine, createRunner } from "../src/index.js";
import type { RunnerEvent } from "@codehands/contracts";
import { createCannedModel, createEngine, createRunner, type ModelClient } from "../src/index.js";

const flags = process.argv.slice(2).filter((a) => a.startsWith("--"));
const [repoPath = join(homedir(), "codehands-fixtures", "date-bug"), prompt = "Fix the failing test. Run npm test to check."] =
process.argv.slice(2).filter((a) => !a.startsWith("--"));
const free = flags.includes("--free");
const slow = flags.includes("--slow");

const canned = createCannedModel({
analyze: "The test expects the day of the month in UTC. src/date.js probably reads it in local time.",
plan: "1. Read src/date.js and test/date.test.js\n2. Read the day in UTC\n3. Run npm test",
review: `VERDICT: ${flags.includes("--wrong-fix") ? "concerns" : "approve"}\n(canned review: no model was asked)`,
});
const thinking: ModelClient = slow ? { complete: async (r) => (await sleep(10_000), canned.complete(r)) } : canned;

// --free: instead of Claude Code, node prints your recorded run. First it makes the change the
// recorded run made: getDate() becomes getUTCDate() in src/date.js. --wrong-fix only adds a comment.
// It prints task.started at once and the rest after --slow's 10 seconds: the window to kill it in implement.
// The same stand-in answers a resume, so a run cut off during implement also finishes for free.
const recording = fileURLToPath(new URL("../test/fixtures/spike-1.jsonl", import.meta.url));
const [from, to] = flags.includes("--wrong-fix") ? ["getDate();", "getDate(); // the day"] : ["getDate()", "getUTCDate()"];
const replay = `const fs=require("fs"),f="src/date.js";if(fs.existsSync(f))fs.writeFileSync(f,fs.readFileSync(f,"utf8").replace(${JSON.stringify(from)},${JSON.stringify(to)}));process.stdout.write(fs.readFileSync(process.argv[1],"utf8"))`;
const runner = free ? createRunner({ agentCommand: () => ({ file: process.execPath, args: ["-e", replay, recording] }) }) : undefined;
const replay = `const fs=require("fs"),f="src/date.js",lines=fs.readFileSync(process.argv[1],"utf8").split("\\n");
process.stdout.write(lines.slice(0,2).join("\\n")+"\\n");
setTimeout(()=>{
if(fs.existsSync(f))fs.writeFileSync(f,fs.readFileSync(f,"utf8").replace(${JSON.stringify(from)},${JSON.stringify(to)}));
process.stdout.write(lines.slice(2).join("\\n"));
},${slow ? 10_000 : 0});`;
const standIn = () => ({ file: process.execPath, args: ["-e", replay, recording] });
const runner = free ? createRunner({ agentCommand: standIn, resumeCommand: standIn }) : undefined;

const engine = createEngine({ runner, model: free || flags.includes("--canned") ? canned : undefined });
const sessionId = randomUUID();
const engine = createEngine({ runner, model: free || flags.includes("--canned") ? thinking : undefined });

let finished = false;
for await (const e of engine.start({ sessionId, repoPath, prompt, agent: "codehands", model: "sonnet", budgetUsd: 0.5 })) {
function print(e: RunnerEvent) {
const p = e.payload;
const step = typeof p.step === "string" ? `[${p.step}]` : "";
const detail = p.reason ?? p.command ?? p.path ?? p.model ?? (typeof p.text === "string" ? p.text.slice(0, 70) : "");
const extra = e.type === "step.finished" ? ` ok=${p.ok}${p.costUsd ? ` $${Number(p.costUsd).toFixed(4)}` : ""}`
: e.type === "test.result" && p.step === "test" ? ` ran=${p.ran} passed=${p.passed}${p.error ? ` ${p.error}` : ""}`
: e.type === "task.finished" ? ` verdict=${p.verdict} $${Number(p.total_cost_usd).toFixed(4)}` : "";
console.log(`${String(e.seq).padStart(3)} ${e.type.padEnd(17)} ${step.padEnd(12)} ${String(detail).replace(/\s+/g, " ")}${extra}`);
if (e.type === "task.finished") finished = true;
}
if (!finished) process.exit(1);

console.log(`\nsession: ${sessionId}`);
console.log(`step files: ${join(homedir(), ".codehands", "runs", sessionId)}`);
if (free) console.log("cost: $0. The cost above is the recorded run's, replayed");
if (flags.includes("--approve")) {
const { branch, commit } = await engine.approve(sessionId, prompt);
console.log(`approved: ${branch} @ ${commit.slice(0, 7)}`);
} else if (flags.includes("--reject")) {
await engine.reject(sessionId);
console.log("rejected: worktree and branch deleted");
// Prints a run's events, then approves, rejects or keeps it. false when it didn't finish.
async function watch(sessionId: string, events: AsyncIterable<RunnerEvent>): Promise<boolean> {
let finished = false;
for await (const e of events) {
print(e);
if (e.type === "task.finished") finished = true;
}
if (!finished) return false;
console.log(`\nsession: ${sessionId}`);
console.log(`step files: ${join(homedir(), ".codehands", "runs", sessionId)}`);
if (free) console.log("cost: $0. The cost above is the recorded run's, replayed");
if (flags.includes("--approve")) {
const { branch, commit } = await engine.approve(sessionId, prompt);
console.log(`approved: ${branch} @ ${commit.slice(0, 7)}`);
} else if (flags.includes("--reject")) {
await engine.reject(sessionId);
console.log("rejected: worktree and branch deleted");
} else {
console.log("kept: the worktree is still there, see `git worktree list` in the repo");
}
return true;
}

if (flags.includes("--recover")) {
if (slow) console.log(`pid ${process.pid} · slow: kill me halfway too, then run --recover again\n`);
const report = await engine.recover();
console.log(`recover: ${report.resumable.length} to resume, ${report.failed.length} failed, ${report.cleanupErrors.length} cleanup errors`);
let ok = true;
for (const sessionId of report.resumable) {
console.log(`\nresuming ${sessionId}`);
ok = (await watch(sessionId, engine.resume(sessionId))) && ok;
}
if (!ok) process.exit(1);
} else {
console.log("kept: the worktree is still there, see `git worktree list` in the repo");
const sessionId = randomUUID();
console.log(`pid ${process.pid} · session ${sessionId}${slow ? " · slow: kill me halfway, then run with --recover" : ""}\n`);
if (!(await watch(sessionId, engine.start({ sessionId, repoPath, prompt, agent: "codehands", model: "sonnet", budgetUsd: 0.5 })))) process.exit(1);
}
Loading
Loading