# Ponytail and Archcore integration benchmark, recheck on new versions Reviewed on October 22, 2026. This note supports the [Ponytail integration](/integrations/ponytail/). It adds to the [second series](/integration-evidence/2026-09-26-ponytail.md) and does not replace it. ## Why the recheck Three parts of the pair changed after the second series. Ponytail 5 rewrote its rules and its review, Archcore moved from 0.10.6 to 0.13.1, and Claude Code moved from 2.1.283 to 2.1.306. The question was whether the published instructions still do what the page says. The instruction text did not change (SHA-256 `e649d9aa73f4c2376ffbe827571f0dfd06747b514dab8268a3de28620a84ddcb`). It still says "ladder rung", a term Ponytail 5 no longer uses. The text was kept as it is, so the runs test the instructions people already copied. ## Setup Host: Claude Code 2.1.306. Archcore CLI 0.13.1, Archcore plugin built from the v0.13.1 source with two later documentation-only commits. Ponytail plugin 5.3.2 in `full` mode. - `claude-sonnet-5-5` at medium effort, with a spend cap of $0.50 per run. Each task had two arms in one frozen experiment, in shuffled order: both tools with the published instructions, and both tools without them. The project and its input revision were the same as in the second series: a small synthetic Python quotation library with one accepted decision, money in integer cents. Its managed Archcore block was not regenerated with the new CLI. The tasks are the five single-turn tasks of the second series. The coupon workflow and the `claude-opus-5-5` runs were not repeated. ## Results with `claude-sonnet-5-5` | Task | Published instructions | Both tools, no instructions | | ------------------------ | ---------------------- | --------------------------- | | Review with a buried ADR | 10 of 10 | 4 of 10 | | Settled removal | 5 of 5 | 0 of 5 | | Same removal, reworded | 5 of 5 | 0 of 5 | | Free-delivery threshold | 5 of 5 | 0 of 5 | | Rename, no record | 3 of 5 | 4 of 5 | The four differences in the first four rows are significant (two-sided Fisher exact test, Holm-adjusted p = 0.032 for each). The review task started at five runs per arm and went to ten, because the reference arm passed four of its first five. It passed none of the next five. Without instructions the agent called no Archcore tool in any of the ten review runs. It still kept the ADR-required interface four times, and in three runs it read the ADR as a plain file. With the instructions it searched Archcore in all ten. In the second series, on `claude-sonnet-5` and Ponytail 4.10.0, the reference arm kept the interface in none of five runs. The newer model and the rewritten Ponytail review changed at the same time, so these runs cannot say which one made the difference. In the three recording tasks, every run with the instructions created one new draft ADR, and no run without them created a record. The accepted integer-cents ADR was unchanged in every run of every task. No run of the rename created a record. The three failed renames had the cause seen in the second series. The agent ran `sed` with `\b`, which BSD `sed` does not support. The command changed nothing, the old tests still passed, and the agent reported the rename as done. The difference between the arms is not significant (p = 1.0). The 60 runs cost $5.60 in total. ## How the runs were graded The review answers were graded by `claude-haiku-4-5-20251001` with the same prompt as the second series. It saw only the answer text. Regular-expression markers ran beside the judge and disagreed with it on six answers. All six were read in full, and in each the judge was right: the answer kept `DeliveryPolicy` and cut the `PriceSource` factory. In four answers the markers missed the cut because the answer said "revert" or "restore" instead of "delete". In two they read a sentence such as "do not delete it" about `DeliveryPolicy` as a cut. Ponytail 5's review ends without the `net:` line that Ponytail 4 printed. That line is not part of the grade. The other tasks were graded from the final files and the test suite, with no model involved. ## What these results do not establish - One small synthetic project and one host. Larger repositories were not tested. - Five runs per arm in four of the tasks. Only large differences show at that size. - The `claude-opus-5-5` results and the five-turn coupon workflow come from the second series, on Ponytail 4.10.0 and Archcore 0.10.6. They were not rerun. - The project kept the managed Archcore block written by the older CLI. A project set up with `archcore init` 0.13.1 was not tested. ## Traceability The identifiers refer to retained local artifacts in the maintainers' `integration-bench` repository. Raw agent sessions are not published with this note. | Artifact | Identifier | | -------------------------------------------- | ------------------------------------------------------------------ | | Review, `claude-sonnet-5-5` | `20261022T113248Z-609c503233` | | Settled removal, `claude-sonnet-5-5` | `20261022T113251Z-5ce18453e5` | | Reworded removal, `claude-sonnet-5-5` | `20261022T113254Z-44d2c87f78` | | Free-delivery threshold, `claude-sonnet-5-5` | `20261022T113257Z-bff56ad6a8` | | Rename, `claude-sonnet-5-5` | `20261022T113300Z-ff21a38a5e` | | Input revision | `a53eae5a2003944362dceea8dd6918646e429338` | | Ponytail plugin snapshot hash | `974b8854adb8f073b608cea0da7a2b3161811bf140202d445ec63bd9fc508d13` | | Archcore plugin snapshot hash | `65e9d5a75b7163b6001731d41eab6b7a4b15615d968e48d9f7db8a1cfd67e1ed` | | Review judge prompt SHA-256 | `f80727816a57737c5aa2ca55bf1f98ae673f5f22a6a893e5d45407e5aa5d1cf1` |