Hono replay: 16 agents on honojs/hono's own pull requests #

MEASURED on https://staging.firetrident.com (build 1a51ebe), 2026-10-19: in-platform agents calling real models, review on, a scripted person answering the review queue, graded by Hono's own tests under real vitest. The workload is demo/hono-replay/workload.json (scripts/hono_replay_select.rb): honojs/hono at ab590e4ec8be (2026-07-10) and 15 pull requests merged into main after it, each redone as one intent (source human, equal priority) with its title, its description as the why and the files it changed under src/ as its footprint. Each pull request changes source that another one also changes; most shared: src/request.ts (5), src/client/client.ts (3), src/router/trie-router/node.ts (2), src/utils/cookie.ts (2). Each run started on its own fresh colony from the same seed (B's tree, 346 files) and intents.

Operator's note: Staging build 1a51ebe (the 8-agent run's fixes plus accounts, which operator colonies do not use); colonies at regionDepth 3; #5236 dropped (15 intents). Flags: FT_LATERAL_INHIBITION=off, FT_MEGAFILE=off (as Oct 17), FT_RED_BRAKE, FT_REVIEW_VIEW (with FT_QUOTED_ACCEPTANCE), FT_INHIBIT_STRONGEST, FT_AGE_CAP, FT_PRELAND on; FT_DEFAULT_SPEND_CAP 5.

Each colony was created at region depth 3 (--region-depth, default 3). Admission is decided per region (a region admits while its working agents are fewer than max(3, 2 x its landings in the last minute)), and every file of this workload is under src/: at depth 1 the whole workload is one region and a run holds about three working agents whatever its agent count.

Left out of each colony's trunk certification, not of the grade (they pass under Node and fail in workerd): src/helper/proxy/index.test.ts > Proxy Middleware > proxy > client disconnect, src/request.test.ts > cloneRawRequest > clones consumed request object.

ruby scripts/hono_replay.rb --base https://staging.firetrident.com --clone <clone> --modules <modules> --agents 16 --name hono-replay --note 'Staging build 1a51ebe (the 8-agent run'\''s fixes plus accounts, which operator colonies do not use); colonies at regionDepth 3; #5236 dropped (15 intents). Flags: FT_LATERAL_INHIBITION=off, FT_MEGAFILE=off (as Oct 17), FT_RED_BRAKE, FT_REVIEW_VIEW (with FT_QUOTED_ACCEPTANCE), FT_INHIBIT_STRONGEST, FT_AGE_CAP, FT_PRELAND on; FT_DEFAULT_SPEND_CAP 5.'

The grade never reads FireTrident's certification: after a run, the final trunk is cloned with stock git, what the run changed since the seed is laid over B in a scratch Hono worktree, and for each intent the pull request's test files are copied in and run with real vitest (Hono's main project, B's dependencies). Those hidden test files, which no agent saw, are the pull request's test files as its history left them right after it, with the selected pull requests laid on B in merge order. An intent passes when every one of its own tests passes: the tests in those files that fail right before it and pass right after it. "Hidden test files pass whole" says whether every test in them passed (B's tests and the earlier pull requests' tests in the same files as well).

The scripted person answers the review queue every 30.0 s: a change waiting at least 120.0 s gets its tests run again on the platform (the review card's "Run again"); it is approved with "scripted person: the visible tests pass" when they pass (or when every failing test fails on its base too), and returned naming the failing tests otherwise. When no test can run on it, it is approved when every file it writes is in its intent's footprint and returned naming the files outside otherwise. A change approved 3 times that comes back again is returned to be redone on the current trunk, and an agent's report waiting as long is dismissed with a note to do the intent as written. Every decision is in run-<n>.json under person.

Written by scripts/hono_replay.rb (numbers: scripts/contention_report.rb, whose header defines each one) and copied through packages/bench/src/evidence.ts redact. Files: hono-replay.json (every number), run-<n>.json (each run's record: events, certifications, proposals, agents, the scripted person's decisions, the grade), run.log.

Runs side by side #

16 agents (colony hono-replay-16) #

Model mix @cf/moonshotai/kimi-k2.7-code=1,@cf/zai-org/glm-5.3-flash=1,@cf/openai/gpt-oss-120b=2,@cf/google/gemma-4-26b-a4b-it; region depth 3; flags on: FT_A2A, FT_ACCOUNTS, FT_AGENT_CALL_BUDGET, FT_AGE_CAP, FT_ALREADY_ON_TRUNK, FT_BISECT, FT_CREDIT_TRAILERS, FT_CULPRITS, FT_DEFAULT_SPEND_CAP, FT_DELETION_GUARD, FT_ESCALATION_BATCH, FT_FIELD_GUIDE, FT_GOAL_CONFLICT, FT_HEADS_UP, FT_HEDGE, FT_INHIBIT_STRONGEST, FT_INLINE, FT_INTENT_SPEND, FT_LESSON_SCOPE, FT_MERGE_INSERTIONS, FT_MODEL_STEPUP, FT_PERSON_WAIT, FT_PRELAND, FT_PUBLIC_GIT, FT_PUBLIC_MCP, FT_QUOTED_ACCEPTANCE, FT_RED_BRAKE, FT_REPRODUCE, FT_RESOLVER, FT_RETURN_HUNKS, FT_REVIEW_VIEW, FT_TEXT_MERGE, FT_TRUNK_TESTS, FT_TWINS, FT_VIRTUAL_CHECKOUT; seed 346 files in 7 inline changes, digest 2955e216b062, trunk held it exactly; spend cap set.

Returns: 8 (and 0 redirects). Cause repeats by cause: stale read 0/0, text-merge conflict 0/0, review 2/5, rule 0/0, other 0/0.

Human queue: 1 escalations, 0 held merges, 0 agent reports; 0 waiting at the end. The scripted person approved 0 and returned 1 (0 of them by the no-tests rule), asked for 1 test runs (0 refused) and dismissed 0 reports.

Spend by model, with the in-platform agents on it and their calls: @cf/google/gemma-4-26b-a4b-it $0.1433 (12, 50); @cf/moonshotai/kimi-k2.7-code $0.5439 (1, 15); @cf/openai/gpt-oss-120b $0.1021 (2, 13); @cf/zai-org/glm-5.3-flash $0.0154 (1, 4); @cf/cloudflare/clef $0.0082 (0, 0); @cf/zai-org/glm-4.7-flash $0.0007 (0, 0). By layer: agents $0.7920, review $0.0216, judge $0.0000, resolver $0.0000, other $0.0000.

The workload #

Base ab590e4ec8be (2026-07-10T18:15:02+09:00): fix(types): strip extra properties from array types in JSONParsed (#5103). The pull requests, in merge order, with the source files each changed, its hidden test files and how many of its own tests they hold:

Shared source files: src/client/client.ts (#5202, #5215, #5244); src/request.ts (#5133, #5161, #5268, #5282, #5288); src/router/trie-router/node.ts (#5102, #5189); src/utils/cookie.ts (#5164, #5246); src/utils/stream.ts (#4988, #5274); src/utils/url.ts (#5161, #5209).

Dropped from the workload after selection: #5236: its src/hono.test.ts is 109,470 characters at B and 109,990 right after it, more than the 86,016 characters an in-platform agent can write back in one reply (packages/worker/src/agents/prompt.ts MAX_REWRITE_CHARS), so no agent can produce the file and its intent can never land.

Workload note: #5236 was dropped by hand after selection (considered.dropped says why). The selection now skips a pull request that changes a file longer than an agent can write back (scripts/hono_replay_select.rb), but it was not run again: that needs GitHub reads and fresh vitest runs, and could choose another pull request in its place. Workload note: The pull requests after #5236 were laid on B again without it (stock git, the same fixed identity and dates), so their laid commits and the target changed; every hidden file's blob id is unchanged, and the overlap and the API scan were recomputed from the remaining pull requests.

The seed is B's tree, 346 files, but for 79 under benchmarks/, 1 over the 256 KiB inline limit, 6 under docs/images/, 12 under perf-measures/, 46 under runtime-tests/: those stay in the scratch worktree the grade runs in.

What this does not measure #

• One repository and one stretch of its history; the numbers are one contended point on real code, not a curve.
• The person is a script: it approves what passes the visible tests and returns what fails them, which a careful reviewer would not always do.
• Model calls are what this platform counted for the colony (agents, review, the resolver, the judge). The grade is real vitest on Node with B's dependencies, outside the platform.
• The pull requests were merged upstream; their descriptions are what each intent says, and a description can say less than the change did.
• Models answer differently between runs: a second run of the same command will not give the same numbers.

In this folder #

• README.md
• hono-replay.json
• run-16.json
• run.log