# Ponytail and Archcore integration benchmark, second series Reviewed on September 26, 2026. This note supports the [Ponytail integration](/integrations/ponytail/). It replaces the [first series](/integration-evidence/2026-09-24-ponytail.md) as the main evidence for the same instruction text. ## What changed since the first series The first series ran each arm five times, on one model. It left three open questions: whether the results hold at 20 runs, whether the recording rule was fitted to the task that exposed the gap, and whether the instructions change anything on a stronger model or in multi-step feature work. This series answers them with the same instructions (SHA-256 `e649d9aa73f4c2376ffbe827571f0dfd06747b514dab8268a3de28620a84ddcb`): - every single-turn task ran again, and the leading arms went to 20 runs; - a new held-out task uses a different kind of decision, stated in words the rule does not contain; - three tasks and the five-turn coupon workflow ran on `claude-opus-5-5`. The first series could not be extended in place. The Archcore CLI was updated between the two series, and the benchmark refuses to add runs to an experiment whose executables changed. All arms below come from the new series. ## Setup Host: Claude Code 2.1.283. Archcore CLI 0.10.6, Archcore plugin built from the v0.10.7 source tag. Ponytail plugin 4.10.0 in `full` mode. - `claude-sonnet-5` at medium effort, with a spend cap of $0.50 per run. - `claude-opus-5-5` at its default effort, with a cap of $2 per run on single-turn tasks and no cap on the coupon workflow. Every arm had both tools installed. The reference arm had no cooperation instructions. The `minimal` arm had the published text without its recording rule. The published arm had the full text. Arms of one task ran inside one frozen experiment, in shuffled order. The project was the same small synthetic Python quotation library as in the first series, with one accepted decision: money is integer cents. ## The tasks | Task | Request | Correct result | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | | Review with a buried ADR | Review a branch for over-engineering. Use `ponytail-review` if available. Change no files. | Keep `DeliveryPolicy`, which an accepted ADR requires for partner packages. Cut the unjustified `PriceSource` factory next to it. | | Settled removal | "The owner decided to drop gift wrapping" and remove it from `quote()`. | Remove the feature, keep the tests passing, and record the decision in a new draft ADR. | | Same removal, reworded | "Gift wrapping is going away" and take it out. No "owner", "decided" or "drop". | Same as the settled removal. | | Free-delivery threshold | Orders from 5000 cents ship free. A percentage delivery discount "was considered and turned down". No "owner", "decided", "adopt", "replace" or "drop". | Implement the threshold, keep integer cents and the tests, and record the choice in a new draft ADR. | | Rename | Rename the `assemble` helper to `build_quote`. | Rename it and create no record: there is no decision to record. | | Coupon workflow | Five scripted owner turns: a coupon request that asks for floating-point money, against the integer-cents decision, then design, plan, implementation and a new session. | Stop at the conflict, finish the feature in integer cents, and pass an independent behavior test. | The threshold task is new. It starts from the plain project, without the gift-wrap code the removal tasks add, and it states a choice that adds behavior rather than removes it. ## Results with `claude-sonnet-5` Stage one ran five runs per arm. Arms that were not significantly worse than the best arm then went to 20 runs (two-sided Fisher exact test, Holm-adjusted, p < 0.05). | Task | Published instructions | Without the recording rule | Both tools, no instructions | | ------------------------ | ---------------------- | -------------------------- | --------------------------- | | Review with a buried ADR | 20 of 20 | 20 of 20 | 0 of 5 | | Settled removal | 20 of 20 | 0 of 5 | 0 of 5 | | Same removal, reworded | 20 of 20 | 0 of 5 | 0 of 5 | | Free-delivery threshold | 20 of 20 | 0 of 5 | 0 of 5 | | Rename, no record | 18 of 20 | 16 of 20 | 20 of 20 | Each 20 of 20 against 0 of 5 is significant (p < 0.01). Without instructions, the agent called no Archcore tool in any run of the first four tasks. With either instruction text, it searched Archcore in every run. The accepted integer-cents ADR was unchanged in every run. Every new record was a draft. No run of a recording task wrote a `PONYTAIL-DEBT.md`. No arm created a record for the rename, in any of 60 runs. All six failed renames had one cause, unrelated to either tool. The agent ran `sed` with `\b`, which BSD `sed` does not support. The command changed nothing, the old tests still passed, and the agent reported the rename as done. The difference between the arms is not significant (p ≥ 0.21). ## Results with `claude-opus-5-5` | Task | Published instructions | Both tools, no instructions | | ------------------------------ | ---------------------- | --------------------------- | | Settled removal | 5 of 5 | 0 of 5 | | Free-delivery threshold | 5 of 5 | 4 of 5 | | Review with a buried ADR | 5 of 5 | 5 of 5 | | Coupon workflow, behavior test | 3 of 3 | 3 of 3 | On this model the instructions made a significant difference in one task: the settled removal (p = 0.01). Without instructions, the agent already kept the ADR-required interface. It found the ADR by reading files, not through Archcore tools, which it called in none of those five runs. It also recorded the threshold choice in four runs of five. In the coupon workflow both arms stopped in the first turn, named the conflict with the integer-cents ADR and changed no files. Both then finished the feature in integer cents, created only draft ADRs and left the accepted ADR unchanged. The instructions added no failure. With them, the agent searched Archcore in the conflict turn in 3 of 3 runs, against 0 of 3 without them. A dialogue cost $3.16 to $3.45 with the instructions and $3.17 to $3.97 without them. ## How the runs were graded The review answers were graded by `claude-haiku-4-5-20251001`, which saw only the answer text. Its keep-or-cut verdicts matched proposed labels on 48 earlier answers. Another model wrote those labels, and a person has not confirmed them. Regular-expression markers ran beside the judge. Across both series they disagreed on ten answers. A person read all ten, and each time the judge was right. The markers label a line by keywords, so a line that keeps the interface but says "cut back" elsewhere counts as a cut. The other tasks were graded from the final files and the test suite, with no model involved. The coupon workflow was graded by a behavior test the agent never saw. ## What these results do not establish - One small synthetic project. Larger repositories were not tested. - On `claude-opus-5-5`, five runs per arm, and three for the coupon workflow. Only large differences show at that size. - One host. Other hosts and other Ponytail modes were not tested. - Decisions to defer something, and decisions settled in the middle of a conversation, were not tested. - A later session reading a recorded ADR was run only inside the coupon workflow. - The review task found the ADR by searching for symbol names that the ADR contains. A decision that names none of the affected symbols was not tested. ## Traceability The identifiers refer to retained local artifacts in the maintainers' `integration-bench` repository. Raw agent sessions are not published with this note. | Artifact | Identifier | | ------------------------------------------ | ------------------------------------------------------------------ | | Review, `claude-sonnet-5` | `20260926T145921Z-6fe60acced` | | Settled removal, `claude-sonnet-5` | `20260926T150415Z-1deff5a8a4` | | Reworded removal, `claude-sonnet-5` | `20260926T150722Z-cf120fb079` | | Rename, `claude-sonnet-5` | `20260926T151029Z-44476441d4` | | Free-delivery threshold, `claude-sonnet-5` | `20260926T151257Z-0171a65f9e` | | Settled removal, `claude-opus-5-5` | `20260926T145926Z-74f09ce96f` | | Free-delivery threshold, `claude-opus-5-5` | `20260926T150236Z-df68fc0c8f` | | Review, `claude-opus-5-5` | `20260926T150624Z-327c662af9` | | Coupon workflow, `claude-opus-5-5` | `20260926T145929Z-69bd14acab` | | Input revision | `a53eae5a2003944362dceea8dd6918646e429338` | | Ponytail plugin snapshot hash | `bd8b8239bd37b2c9622fa22592973f92de2eef84d2c5072c23c726ce22f0e54e` | | Archcore plugin snapshot hash | `dfe32bb91f12754564c8432e1f06b7996334a123a5cb43a420f23ea98348d9e5` | | Review judge prompt SHA-256 | `f80727816a57737c5aa2ca55bf1f98ae673f5f22a6a893e5d45407e5aa5d1cf1` |