From a List of Annoyances to a Self-Growing Toolbelt
From a List of Annoyances to a Self-Growing Toolbelt
How the "Candidates for Deterministic Software" doc became working enforcement — and what we expect it to change
Date: 2026-09-15 Author: Claude (Kimi), with Douglas Status: Wave 1 live and enforcing
The short version
A week ago we wrote down a list of things agents keep doing by hand — the same fiddly command ceremonies, re-derived from scratch every session, run once, and forgotten. Today, five of those ceremonies are real command-line tools, the agents are redirected to them automatically when they reach for the raw equivalent, and the system measures how much re-derivation is being avoided. This document is the story of how that happened, why we did it, and what we expect it to improve.
1. The observation (September 8)
We ran a census over the agent session transcripts on this machine — 831 Kimi sessions, 1,160 Claude sessions, 368 memory files, the full cron inventory. The finding, in numbers:
| Pattern | Sessions containing it (Kimi) | (Claude) |
|---|---|---|
| UPDATE/INSERT SQL | 790 | 279+ |
| KB drafts | 789 | — |
| Email sends | 651 | 40 |
srv git pushes | 520 | 27 |
az vm run-command | 516 | 79 |
| docker exec / docker cp | 410 | 55 |
| The capcom2 script bridge (base64 → cp → exec) | 349 | (same recipe) |
Every one of those was a ceremony: a multi-step incantation with failure modes you only learn by hitting them (a 4KB output truncation, a run-command slot that only takes one caller at a time, a SAS string that 403s if you sign it by hand). Agents — me included — were re-deriving these from memory files and old transcripts every single session, burning turns, tokens, and occasionally getting them wrong (three shared-clone git clobbers in ten days; an SMTP password typed into a transcript).
The rule Douglas set that day:
An agent does a thing one-off only until we know it's a repeating task. Then it graduates. Judgment stays with the agent; mechanics graduate to software.
The deliverable was deterministic-software-candidates.md: 26 ranked candidates plus the design for the machine that detects repeats — the "refire mechanism."
2. The sensor (September 9)
The audit's uncomfortable finding: nothing on this box logged what agents actually do. No hooks, no invocation ledger. You can't graduate what you can't count.
So wave 0 was the sensor itself. A small hook now fires on every operational tool call in both Claude Code and Kimi, normalizes it into a signature (capcom2-exec, srv-push, im-ticket-write…), redacts anything that looks like a secret, and appends one JSON line to a ledger. A nightly cron (04:37 UTC) counts signatures over a trailing 30 days and publishes a report: anything seen ≥3 times in ≥2 sessions is flagged as a graduation candidate.
The report posts to capcom, where a "Deterministic Tools Watcher" service watches the watcher (an hourly freshness check, so a dead cron page-outs instead of going quiet).
3. The uncomfortable review (September 15)
A week of data gave us a top-10 list, and we sat down to pick what to build. The honest answer surprised the framing: most of the top 10 should never become tools.
| Signature | Uses/week | Verdict |
|---|---|---|
shell:cd, shell:grep, shell:ls, shell:cat, shell:sed | ~5,700 | Not tasks — the agent's hands. Zero ceremony to eliminate. |
github-read, MCP search tools | ~1,300 | Already tools. Volume is usage, not re-derivation. |
capcom2-exec | 755 | Graduate — the base64→cp→exec bridge, rediscovered constantly. |
az-run-command | 577 | Graduate — same family. |
capcom-pg-sql, capcom-mssql-sql | 1,129 | The resolver is the pain (client→server→db→credential), not the SQL. |
srv-push, git-push | 353 | Graduate — and make the safe sequence the only sequence. |
shell:ssh | 511 | Half debugging (keep), half absorbed by the bridge. |
The graduation test isn't frequency alone — it's frequency × determinism × pain (how much re-derivation each occurrence costs). cd fails it. The base64 bridge passes it spectacularly.
We also made every verdict first-class instead of leaving it in chat: a registry in capcom. Each repeating signature is either mapped (this helper covers it — keep measuring) or explained (classified once, with a reason, and never reported again — the report dropped from ~130 noisy rows to ~28 actionable ones overnight). You can see all of this on the AgentOps → Tools Watcher page (Observations tab) and on Release Management → Helpers.
4. The tools (September 15, wave 1)
Five helpers shipped, each verified end-to-end against the live estate:
run-on-vm— the run-command bridge. Ships a script to any VM,
docker-cps into a container, executes, streams output back, retries the transient failures with jittered backoff (it fought real contention on the capcom VM during its own testing and won), and pulls big outputs back in gzip chunks instead of truncating them at 4KB.
srv-push— the git closeout. Typecheck gate, pathspec-scoped commits
(never add -A), push, and on a race it rebases through a temp worktree instead of clobbering the shared clone. The safe sequence is now the only sequence. It committed itself to git — twice.
client-sql— client database resolver + runner. Saycoxand it
finds the server, database, and credential path (AAD service principal via the capcom container), runs the query read-only by default, and logs anything else to an audit file.
send-ses-mail— email with the SMTP password resolved internally.
(That password will never appear in a transcript again.) Attachments over 10MB automatically become blob links.
blob-sas— upload + 7-day link, signed by the Azure CLI instead of
by hand (the old 403 class).
They live in a new git repo (tc-agent-ops on the srv mirror) and each one is also a service — registered on the Release Management → Helpers tab with a Deploy button. Douglas pressed Deploy on all thirteen; that "armed" them, and each one switched to active enforcement the moment its binary landed.
5. The enforcement (this is the part that makes it real)
Tools nobody remembers to call are shelf-ware. So the same registry drives two enforcement layers:
- In the CLIs (Claude Code + Kimi): a hook inspects every operational
call before it runs. If the call's family has graduated and the helper is live, the call is denied with a pointer to the helper — the agent reads the message and calls the tool instead. Within minutes of wave 1 shipping, the log showed another session getting redirected on a raw git push — it works in the wild, not just in tests. There's a documented one-off escape (HELPERS_BYPASS=1) for fire drills, and the gate fails open: a bug in the enforcer can never wedge a session.
- In Tapestry (the Designer): every MCP tool call now checks the
registry first. A live helper answers the call through a small stdin/stdout contract; if the helper declines (exit 42), the call falls through to the raw substrate. Flipping a helper to live on the Releases card starts routing the Designer automatically — no per-helper code in Tapestry.
A subtle point: enforcement only blocks the invocation, never data. Quoting docker exec capcom2 inside a JSON document or a heredoc doesn't trip it — the matcher strips quoted spans before deciding.
6. The measurement
You can't improve what you don't count (twice now, that lesson). Three signals, all automatic:
- Adoption curve — every night, per helper: hand-runs remaining vs
helper runs vs redirections. Baseline today: run-on-vm has ~1,680 raw calls to absorb, srv-push ~410, client-sql ~460. Those numbers should fall daily.
- Redirection log — every enforcement denial, so we can see friction
(a helper people keep routing around is a badly designed helper).
- Token model — each helper run verifiably replaces one multi-call
ceremony: saved ≈ runs × (calls-in-ceremony − 1) × tokens-per-call. It shows on the Releases cards as "N runs · ~X tokens saved." The counts are measured; the per-call token constant is a deliberately conservative estimate (refinable from transcripts). And the model deliberately excludes the biggest effect: every avoided ceremony is turns that never enter the session context, and context size is what drives the quadratic cost of long sessions. The printed number is a floor.
7. What we expect this to improve
- Spend. Fewer re-derived ceremonies = fewer turns = smaller context
growth. The watcher already suppresses ~16,500 of ~19,000 weekly invocations from the noisy report; wave 1 targets the ~3,000/week that were pure re-derivation.
- Correctness. The safe sequence is now the only sequence — the git
clobber class, the 4KB truncation surprise, the hand-signed SAS 403, and the credential-in-transcript exposure are all closed by construction, not by remembering.
- Speed. A helper call is one round-trip with zero rediscovery, and it
carries the retry logic that used to cost sessions their flow.
- A toolbelt that grows itself. The loop is the point: the watcher
notices a new repeat → it appears on the Tools Watcher page with evidence → it graduates to a helper → enforcement routes agents to it → the adoption curve proves (or disproves) it. Wave 2 is already queued by the data: im-ticket (~330), vm-docker-redeploy (~160), queue-refire (~115), xlsx-export, acr-build-tag-bump.
8. The honest gaps
- Codex (OpenAI's CLI) has no hook mechanism, so it can't be hard-gated
the way Claude and Kimi are. Its use here is harness-driven and small; when wave 2 lands we'll add a prompt-level directive and mirror the helpers to the capcom VM where those runs execute.
- The token figure is an estimate on top of measured counts — we'll ground
the constant with a transcript sample rather than pretend precision.
- Helpers that turn out to be wrong for a job have an exit: the contract's
passthrough code returns the call to the agent's judgment, and the registry can retire a helper as easily as it deployed one. Judgment stays with the agent. That's the rule everything else is built around.
Where to look: Release Management → Helpers (cards + deploy buttons) · AgentOps → Tools Watcher → Observations (the nightly evidence report) · ~/deterministic-software-candidates.md (the original census + backlog) · ~/agent-ops/HELPER-CONTRACT.md (the tool contract) · repo tc-agent-ops on the srv mirror.