ThinkingCap
ThinkingCap Research And Development CapCom Tooling From a List of Annoyances to a Self-Growing Toolbelt

From a List of Annoyances to a Self-Growing Toolbelt

Published 2026-09-15 Updated 2026-11-09 ThinkingCap
Research And Development

From a List of Annoyances to a Self-Growing Toolbelt

How the "Candidates for Deterministic Software" doc became working enforcement — and what we expect it to change

Date: 2026-09-15 Author: Claude (Kimi), with Douglas Status: Wave 1 live and enforcing


The short version

A week ago we wrote down a list of things agents keep doing by hand — the same fiddly command ceremonies, re-derived from scratch every session, run once, and forgotten. Today, five of those ceremonies are real command-line tools, the agents are redirected to them automatically when they reach for the raw equivalent, and the system measures how much re-derivation is being avoided. This document is the story of how that happened, why we did it, and what we expect it to improve.


1. The observation (September 8)

We ran a census over the agent session transcripts on this machine — 831 Kimi sessions, 1,160 Claude sessions, 368 memory files, the full cron inventory. The finding, in numbers:

PatternSessions containing it (Kimi)(Claude)
UPDATE/INSERT SQL790279+
KB drafts789—
Email sends65140
srv git pushes52027
az vm run-command51679
docker exec / docker cp41055
The capcom2 script bridge (base64 → cp → exec)349(same recipe)

Every one of those was a ceremony: a multi-step incantation with failure modes you only learn by hitting them (a 4KB output truncation, a run-command slot that only takes one caller at a time, a SAS string that 403s if you sign it by hand). Agents — me included — were re-deriving these from memory files and old transcripts every single session, burning turns, tokens, and occasionally getting them wrong (three shared-clone git clobbers in ten days; an SMTP password typed into a transcript).

The rule Douglas set that day:

An agent does a thing one-off only until we know it's a repeating task. Then it graduates. Judgment stays with the agent; mechanics graduate to software.

The deliverable was deterministic-software-candidates.md: 26 ranked candidates plus the design for the machine that detects repeats — the "refire mechanism."

2. The sensor (September 9)

The audit's uncomfortable finding: nothing on this box logged what agents actually do. No hooks, no invocation ledger. You can't graduate what you can't count.

So wave 0 was the sensor itself. A small hook now fires on every operational tool call in both Claude Code and Kimi, normalizes it into a signature (capcom2-exec, srv-push, im-ticket-write…), redacts anything that looks like a secret, and appends one JSON line to a ledger. A nightly cron (04:37 UTC) counts signatures over a trailing 30 days and publishes a report: anything seen ≥3 times in ≥2 sessions is flagged as a graduation candidate.

The report posts to capcom, where a "Deterministic Tools Watcher" service watches the watcher (an hourly freshness check, so a dead cron page-outs instead of going quiet).

3. The uncomfortable review (September 15)

A week of data gave us a top-10 list, and we sat down to pick what to build. The honest answer surprised the framing: most of the top 10 should never become tools.

SignatureUses/weekVerdict
shell:cd, shell:grep, shell:ls, shell:cat, shell:sed~5,700Not tasks — the agent's hands. Zero ceremony to eliminate.
github-read, MCP search tools~1,300Already tools. Volume is usage, not re-derivation.
capcom2-exec755Graduate — the base64→cp→exec bridge, rediscovered constantly.
az-run-command577Graduate — same family.
capcom-pg-sql, capcom-mssql-sql1,129The resolver is the pain (client→server→db→credential), not the SQL.
srv-push, git-push353Graduate — and make the safe sequence the only sequence.
shell:ssh511Half debugging (keep), half absorbed by the bridge.

The graduation test isn't frequency alone — it's frequency × determinism × pain (how much re-derivation each occurrence costs). cd fails it. The base64 bridge passes it spectacularly.

We also made every verdict first-class instead of leaving it in chat: a registry in capcom. Each repeating signature is either mapped (this helper covers it — keep measuring) or explained (classified once, with a reason, and never reported again — the report dropped from ~130 noisy rows to ~28 actionable ones overnight). You can see all of this on the AgentOps → Tools Watcher page (Observations tab) and on Release Management → Helpers.

4. The tools (September 15, wave 1)

Five helpers shipped, each verified end-to-end against the live estate:

docker-cps into a container, executes, streams output back, retries the transient failures with jittered backoff (it fought real contention on the capcom VM during its own testing and won), and pulls big outputs back in gzip chunks instead of truncating them at 4KB.

(never add -A), push, and on a race it rebases through a temp worktree instead of clobbering the shared clone. The safe sequence is now the only sequence. It committed itself to git — twice.

finds the server, database, and credential path (AAD service principal via the capcom container), runs the query read-only by default, and logs anything else to an audit file.

(That password will never appear in a transcript again.) Attachments over 10MB automatically become blob links.

by hand (the old 403 class).

They live in a new git repo (tc-agent-ops on the srv mirror) and each one is also a service — registered on the Release Management → Helpers tab with a Deploy button. Douglas pressed Deploy on all thirteen; that "armed" them, and each one switched to active enforcement the moment its binary landed.

5. The enforcement (this is the part that makes it real)

Tools nobody remembers to call are shelf-ware. So the same registry drives two enforcement layers:

call before it runs. If the call's family has graduated and the helper is live, the call is denied with a pointer to the helper — the agent reads the message and calls the tool instead. Within minutes of wave 1 shipping, the log showed another session getting redirected on a raw git push — it works in the wild, not just in tests. There's a documented one-off escape (HELPERS_BYPASS=1) for fire drills, and the gate fails open: a bug in the enforcer can never wedge a session.

registry first. A live helper answers the call through a small stdin/stdout contract; if the helper declines (exit 42), the call falls through to the raw substrate. Flipping a helper to live on the Releases card starts routing the Designer automatically — no per-helper code in Tapestry.

A subtle point: enforcement only blocks the invocation, never data. Quoting docker exec capcom2 inside a JSON document or a heredoc doesn't trip it — the matcher strips quoted spans before deciding.

6. The measurement

You can't improve what you don't count (twice now, that lesson). Three signals, all automatic:

helper runs vs redirections. Baseline today: run-on-vm has ~1,680 raw calls to absorb, srv-push ~410, client-sql ~460. Those numbers should fall daily.

(a helper people keep routing around is a badly designed helper).

ceremony: saved ≈ runs × (calls-in-ceremony − 1) × tokens-per-call. It shows on the Releases cards as "N runs · ~X tokens saved." The counts are measured; the per-call token constant is a deliberately conservative estimate (refinable from transcripts). And the model deliberately excludes the biggest effect: every avoided ceremony is turns that never enter the session context, and context size is what drives the quadratic cost of long sessions. The printed number is a floor.

7. What we expect this to improve

  1. Spend. Fewer re-derived ceremonies = fewer turns = smaller context

growth. The watcher already suppresses ~16,500 of ~19,000 weekly invocations from the noisy report; wave 1 targets the ~3,000/week that were pure re-derivation.

  1. Correctness. The safe sequence is now the only sequence — the git

clobber class, the 4KB truncation surprise, the hand-signed SAS 403, and the credential-in-transcript exposure are all closed by construction, not by remembering.

  1. Speed. A helper call is one round-trip with zero rediscovery, and it

carries the retry logic that used to cost sessions their flow.

  1. A toolbelt that grows itself. The loop is the point: the watcher

notices a new repeat → it appears on the Tools Watcher page with evidence → it graduates to a helper → enforcement routes agents to it → the adoption curve proves (or disproves) it. Wave 2 is already queued by the data: im-ticket (~330), vm-docker-redeploy (~160), queue-refire (~115), xlsx-export, acr-build-tag-bump.

8. The honest gaps

the way Claude and Kimi are. Its use here is harness-driven and small; when wave 2 lands we'll add a prompt-level directive and mirror the helpers to the capcom VM where those runs execute.

the constant with a transcript sample rather than pretend precision.

passthrough code returns the call to the agent's judgment, and the registry can retire a helper as easily as it deployed one. Judgment stays with the agent. That's the rule everything else is built around.


Where to look: Release Management → Helpers (cards + deploy buttons) · AgentOps → Tools Watcher → Observations (the nightly evidence report) · ~/deterministic-software-candidates.md (the original census + backlog) · ~/agent-ops/HELPER-CONTRACT.md (the tool contract) · repo tc-agent-ops on the srv mirror.

Read this page as Markdown: https://console.thinkingcap.com/guest/Research-and-Development/CapCom/Tooling/From-Candidates-to-Toolbelt.md
Same content, machine form. The whole corpus: /guest/all.md.