Paper-review checks of the simulator, 2026-10-13 #

The 2026-10-13 paper reviews questioned the simulator numbers in README line 167: "plain optimistic claims at about 42% rework against 22% for exclusive locks; decaying inhibition and co-change learning bring it to about 31% (26% with a stronger co-change weight) with no loss of throughput, and conflicts per 1,000 submissions fall below the lock design's (285 vs 318)".

This folder re-runs each question with the existing harness: 10 seeds per configuration, at 100, 1,000 and 10,000 agents. It adds three analysis options that are not built in the platform: a classifier for why changes come back, import-aware take(), and assumption revalidation.

Every number here is SIMULATED. None comes from a deployment. The reviewers' numbers came from their own runs or estimates, and this is the first reproduction.

Verdicts #

Unless noted, figures are at 1,000 agents, the mean of seeds 1-10 ± the 95% interval. "Deployed" means group commit plus co-change weight 2 plus lateral inhibition, which is what production runs.

The README's own numbers still hold on their own terms. Plain optimistic claims rework 43.2 ± 1.2%, inhibition plus co-change 30.4 ± 1.2%, co-change weight 5 25.9 ± 1.4%, and conflicts are 273 per 1,000 against 310 for locks. All of these come from per-lane commits with the unequal retry costs.

What was run #

Every number in this folder is SIMULATED: the coordination simulator in packages/bench/src/sim (FireSim for FireTrident, BaselineSim for locks), on the Hono workload of the 2026-10-12 benchmark. Nothing here was measured on a deployment.

• Workload. Tasks drawn from 2,285 Hono commits (3.01 files per change) in a monorepo of 100 package copies, region depth 3. Each task reads the direct relative imports of the files it writes (at most 8). Agent work per attempt is log-normal with a 60 s median (sigma 0.8). Timings are the in-platform calibration in data/calibration.json, with checks free (testMs 0).
• Settings. The same as results/2026-10-12-inhibition, which produced every number in README line 167: 20 simulated minutes with the first 5 as warm-up, a backlog of 1,000 open tasks, co-change weight 2 (top 8 neighbors with coupling of at least 0.1, mined from Hono's last 200 commits), lateral inhibition at 0.5.
• Seeds and statistics. Seeds 1 to 10 for every configuration, at 100, 1,000 and 10,000 agents. Tables show the mean over seeds ± the 95% Student t half-width. A change between two configurations is the difference of their means with a Welch 95% interval.
• Two trunk writers.
  • Per-lane commits (model firetrident): each lane batch is its own commit, and one writer serves them one at a time. The README's 42%, 31%, 26%, "no loss of throughput" and "285 vs 318" all come from this model. Review 2610.04779 calls it "per-proposal lanes".
  • Group commit (model firetrident-group): the writer folds every waiting batch into one commit. Production works this way with one difference: it writes one chained commit per batch and pushes the tip once with compare-and-swap (TrunkDO and planCycle in packages/worker/src/trunk/plan.ts).
• Reproduction check. 81 rows here repeat a configuration of results/2026-10-12-inhibition/sim-steady.json (same variant, agents and seed, default retry cost). All 81 are identical to the published rows: landings, rework, conflicts, write-write, claim-to-land p50 and submissions. The new options are off by default, and the return classifier does not change a run.

New simulator code (analysis only; none of it is built in the platform) #

• FireOptions.onReturn (sim/fire.ts) reports why each returned proposal went back. It classifies the return as one of three kinds:
  • read-only staleness: only files it read moved, on trunk or by a proposal ahead of it in the same commit;
  • write-write: a file it writes moved, or another proposal of the commit writes it;
  • order: it writes a file that a proposal ahead of it in the same group commit read.
• FireOptions.assumptions and BaselineOptions.assumptions add assumption revalidation. A change that is stale only through files it read lands anyway when its assumption on each of those files held through every change since the agent read it. Each change is one independent draw at the hold probability.
• sim/imports.ts adds import-aware repulsion: weight times core neighborRepulsion over each written file's direct imports and importers (weight 1 per edge), added to co-change repulsion the way ColonyDO.take adds it.
• sim/papers.ts is the driver and the report. The tests are in test/papers.test.ts.

Q1. Equal retry costs: the 42% vs 22% compares a full redo with half a redo #

sweep.ts sets rework: model === "baseline" ? 0.5 : 1.0. A locked change that comes back is reworked at half its work time; a FireTrident change that comes back is redone in full. Here both models run with the same retry cost, 1.0 and then 0.5.

• Locks, 1,000 agents: 34.8% with a full redo against the published 21.3% with half a redo. Landings and conflicts barely move (51.9/min; 314 and 310 conflicts per 1,000), because the merge queue limits locks.
• FireTrident with inhibition and co-change: below locks at both costs, with either trunk writer.
  • Per-lane commits: 30.4% (full) and 17.8% (half).
  • Group commit, as deployed: 30.5% (full) and 17.4% (half).
  • This comes with fewer conflicts (194 to 273 per 1,000, against 310 to 314) and 1.7 to 4.0 times the landings.
• Plain optimistic claims rework more than locks: 43.2% against 34.8% with a full redo, and 27.5% against 21.3% with half. The gap is 6 to 8 points, not the 20 points the README implies.
• A cheaper retry also raises throughput under group commit (deployed: 175 to 208 landed/min), because agents spend less time redoing work. With per-lane commits the trunk writer is the limit (about 86/min either way).
• At 100 agents every design reworks 5 to 11%. Locks are not better there either: 10.3% against 9.5-11.4% with a full redo.
• At 100 agents co-change weights 2 and 5 give identical runs. A one-seed probe showed weight 10 does too: a stronger weight changes no agent's choice there.

Q1. Equal retry costs (SIMULATED, seeds 1, 2, 3, 4, 5, 6, 7, 8, 9, 10; mean ± 95% CI over seeds) #

100 agents:

1,000 agents:

10,000 agents:

Q2. Lateral inhibition costs throughput under group commit #

• With per-lane commits (the README's numbers), inhibition costs nothing: +0.6% landed/min, with a 95% interval of +0.2 to +0.9 at 1,000 agents.
• With group commit, the trunk writer folds batches into one commit. Throughput then depends on how much independent work agents find, and inhibition costs real landings:
  • −7.2% at 1,000 agents with co-change, as deployed (−11.0 to −3.5);
  • −10.8% at 10,000 agents (−14.0 to −7.5);
  • −15 to −16% without co-change.
  • It is still about 3.4 times locks' landings (175 against 52 per minute).
• The rework it buys is about 10 points: 40.9% to 30.5% deployed, 45.9% to 31.6% without co-change.
• Why it costs landings (from results/2026-10-12-inhibition): inhibited intents sit at the head of the 200-oldest candidate window until their files' claims clear.
• At 100 agents the runs with and without inhibition are identical. With few agents a claimed intent never outranks a free one.

Q2. What lateral inhibition costs in throughput (SIMULATED, seeds 1, 2, 3, 4, 5, 6, 7, 8, 9, 10; change = difference of means with a Welch 95% CI) #

Q3. What still comes back: stale reads, mostly on three hub files #

Each return is classified as it happens, using FireOptions.onReturn. The classifier has no effect on a run: all 81 rows that repeat a published configuration are identical.

• With inhibition, no return is a write conflict. The simulator's agents heartbeat, so every claim is fresh, and inhibition never lets two agents hold a claim on the same file at the same time.
  • Per-lane commits: 100.0% of returns are read-only staleness.
  • Group commit: 99.8%. The other 0.2% are proposals that write a file a proposal ahead of them in the same commit read. The simulated group-commit screen returns these, while planCycle lands that case across lane batches.
  • Without inhibition, 36-41% of returns are write-write.
• The published writeWritePer1000 counter misleads. It counts every return the trunk writer finds as write-write (finishTick passes true).
  • With per-lane commits, about half of all returns are found there. That is where the reviewer's "51-52% read-only" came from; the true share is 100%.
  • Under group commit about 1% are found there, so that counter's 98.7% is close to the classified 99.8%.
• The three most-imported files take about 70% of the stale reads.
  • src/context.ts (73 importers), src/hono.ts (69) and src/types.ts (63) account for 71.2% of the changed reads behind read-only returns in the deployed configuration, and 68-75% in every other configuration and agent count.
  • 77.5% of the read-only returns touch at least one of them.
  • A read-only return has 1.32 changed reads on average under group commit, and 1.18 with per-lane commits.

Q3. What sends changes back (SIMULATED, the Q2 runs; shares of returns after warm-up, mean ± 95% CI over seeds) #

Q4. Import-aware take() (ANALYSIS, not built) #

The new term is weight × neighborRepulsion(footprint, import neighbors, claims). Each direct import, or importer, of a written file that another agent holds a claim on adds that claim, at weight 1. It is added to the co-change term the way ColonyDO.take adds co-change repulsion, and it only repels: it never inhibits.

• Rework.
  • Weight 1 roughly halves it: deployed 30.5% falls to 16.2%, per-lane commits 30.4% to 16.7%.
  • Weight 2 nearly removes it: 1.9% deployed, 6.3% with per-lane commits.
  • The tail shortens too: deployed claim-to-land p99 falls from 683 s to 382 s at weight 2.
• Throughput.
  • Under group commit it costs 9.8% (weight 1) and 12.6% (weight 2) of landings a minute at 1,000 agents, and 3.9% and 11.8% at 10,000.
  • With per-lane commits it costs nothing.
  • At 100 agents it removes all rework and adds about 9-10% landings.
• These numbers match review 2606.00953's probe. The probe had 16.8% and 1.1% at about 11% fewer landings, and 7.0% for per-lane commits at weight 2.
• Best case. The simulator's read sets are drawn from the same import edges, so the term sees every read. Real agents read more than their direct imports.

Q4. Import-aware take() (ANALYSIS, not built; SIMULATED, seeds 1, 2, 3, 4, 5, 6, 7, 8, 9, 10; mean ± 95% CI over seeds) #

Q5. Assumption revalidation (ANALYSIS, not built) #

A change that is stale only through files it read lands anyway when its assumption on each changed file held through every change since the agent read it. Each change is one independent draw at the hold rate. The rates come from the probe on 2,285 Hono commits: 68.7% when the signatures of the imported names held, and 57.6% when test importers also need the bodies unchanged. "Always" is the upper bound, ignoring reads entirely.

• Deployed.
  • Rework falls from 30.5% to 18.1% (57.6%) or 14.2% (68.7%).
  • Landings rise 19-24%, because work that used to be redone now lands.
  • Ignoring reads altogether would give 0.1% rework and +51% landings.
• Why the estimate was too optimistic.
  • The review's 13.5-15% and about 10% assumed each stale return lands at the hold rate.
  • A returned change has 1.32 changed reads on average, and a hot file can change more than once while a change waits.
  • So only 46.1% and 57.9% of read-only-stale changes land.
• Locks benefit just as much: 21.3% falls to 10.1% or 7.4%, with +6-7% landings. Assumptions do not favor optimism over locks.
• Plain optimistic claims keep their write conflicts: 43.2% falls only to 29.6-31.9%, and to 21.2% even with every assumption holding. Assumptions complement inhibition; they do not replace it.
• False landings cost nothing here. Real assumption checks would let some broken changes through, and certification would have to catch them.

Q5. Assumption revalidation (ANALYSIS, not built; SIMULATED, seeds 1, 2, 3, 4, 5, 6, 7, 8, 9, 10; mean ± 95% CI over seeds) #

Caveats #

1. Footprints and reads are idealized. A task's declared footprint is exactly its write set, an oracle footprint. Its reads are exactly the direct imports of its written files. Real agents read more, and their footprints are predictions. Every take-time remedy (inhibition, co-change, imports) is therefore at its best here.
2. Import-aware take is a best case. It repels on the same import edges the simulator draws read sets from.
3. Assumption revalidation costs nothing when it is wrong. The model has no false landings: a kept change that real code would break costs nothing. The hold rate is uniform across files. On Hono the hottest files hold less often for signatures (src/context.ts 50.6%, src/hono.ts 56.1%) and src/types.ts more often (85.5%). The probe weights each modification by its importer's touches, roughly the simulator's own exposure, so the average is a fair first-order rate. There is one draw per change to a changed file, not one per importing file, so a change whose two files both import a changed file gets a single draw.
4. The published write-write counter is misleading. writeWritePer1000 counts every return found by the trunk writer as write-write. Q3 classifies each return directly, and the published counter is shown beside it.
5. The simulated group commit is slightly stricter than production. Its screen returns a proposal that writes a file a proposal ahead of it in the same commit read ("order"). planCycle lands that case across lane batches. Q3 shows how rare it is.
6. Rework is agent work only. It includes no model or review cost for a redo, and no cost for revalidating assumptions.
7. Locks saturate. The locks model gives identical runs at 1,000 and 10,000 agents. The merge queue serves about 52 changes a minute, claims go oldest first, and the extra agents only wait. The published runs show the same.

Files #

Every JSON file carries "label": "SIMULATED", the command that wrote it, the workload, the settings and the variants.

Reproduce #

Run these from the repository root. Nothing touches the network or a deployment. Runs are deterministic, so the same command writes the same rows; only generatedAt and wallMs change. These are the 16 shards behind this folder. They ran 6 at a time in about 40 minutes, about 225 CPU-minutes in all.

OUT=packages/bench/results/2026-10-13-papers
node packages/bench/src/sim/papers.ts --steps q1 --out $OUT --agents 100 --tag n100
node packages/bench/src/sim/papers.ts --steps q1 --out $OUT --agents 1000 --tag n1000
node packages/bench/src/sim/papers.ts --steps q1 --out $OUT --agents 10000 --seeds 1,2,3,4,5 --tag n10000-a
node packages/bench/src/sim/papers.ts --steps q1 --out $OUT --agents 10000 --seeds 6,7,8,9,10 --tag n10000-b
node packages/bench/src/sim/papers.ts --steps q2 --out $OUT --agents 100 --tag n100
node packages/bench/src/sim/papers.ts --steps q2 --out $OUT --agents 1000 --tag n1000
node packages/bench/src/sim/papers.ts --steps q2 --out $OUT --agents 10000 --tag n10000
node packages/bench/src/sim/papers.ts --steps q4 --out $OUT --agents 100 --tag n100
node packages/bench/src/sim/papers.ts --steps q4 --out $OUT --agents 1000 --seeds 1,2,3,4,5 --tag n1000-a
node packages/bench/src/sim/papers.ts --steps q4 --out $OUT --agents 1000 --seeds 6,7,8,9,10 --tag n1000-b
node packages/bench/src/sim/papers.ts --steps q4 --out $OUT --agents 10000 --seeds 1,2,3 --tag n10000-a
node packages/bench/src/sim/papers.ts --steps q4 --out $OUT --agents 10000 --seeds 4,5,6 --tag n10000-b
node packages/bench/src/sim/papers.ts --steps q4 --out $OUT --agents 10000 --seeds 7,8,9,10 --tag n10000-c
node packages/bench/src/sim/papers.ts --steps q5 --out $OUT --agents 100 --tag n100
node packages/bench/src/sim/papers.ts --steps q5 --out $OUT --agents 1000 --tag n1000
node packages/bench/src/sim/papers.ts --steps q5 --out $OUT --agents 10000 --tag n10000
# summary.json, the check against results/2026-10-12-inhibition, and the tables above (markdown, on stdout)
node packages/bench/src/sim/papers.ts --steps report --out $OUT

A single node packages/bench/src/sim/papers.ts --out <empty folder> runs every step in one process at the default agent counts and seeds 1-10, then prints the report.

The report counts each configuration and seed once. Re-running a shard, or mixing tagged and untagged files, does not double-count.

Tests: npx vitest run --root packages/bench. test/papers.test.ts covers the return classifier, assumption revalidation in both models and import-aware repulsion.

In this folder #

• README.md
• q1-n100.json
• q1-n1000.json
• q1-n10000-a.json
• q1-n10000-b.json
• q2-n100.json
• q2-n1000.json
• q2-n10000.json
• q4-n100.json
• q4-n1000-a.json
• q4-n1000-b.json
• q4-n10000-a.json
• q4-n10000-b.json
• q4-n10000-c.json
• q5-n100.json
• q5-n1000.json
• q5-n10000.json
• summary.json