--- title: Candidates for Deterministic Software description: "Status 2026-09-09: Wave 0 (the Refire Mechanism, §2) is LIVE — invocation hooks wired into both Claude + Kimi settings, ledger at /agent-ops/ledger.jsonl, nightly repeat detector (cron 04:37) publishing…" created: 2026-09-15 updated: 2026-09-21 authors: ThinkingCap R&D topics: [CapCom] status: published canonical: https://console.thinkingcap.com/rd/CapCom/Tooling/deterministic-software-candidates date: 2026-09-15 --- # Candidates for Deterministic Software ### What the agents keep doing by hand — and what should become tools > **Status 2026-09-09: Wave 0 (the Refire Mechanism, §2) is LIVE** — invocation > hooks wired into both Claude + Kimi settings, ledger at `~/agent-ops/ledger.jsonl`, > nightly repeat detector (cron 04:37) publishing `~/agent-ops/candidates.md`. > Waves 1+ land in `~/agent-ops/bin/` as signatures cross the threshold. **Date:** 2026-09-08 **Prepared by:** Claude (kimi-k3), at Douglas's request **Corpus reviewed:** 831 Kimi session transcripts (1.2 GB JSONL), 1,160 Claude session transcripts (102 MB), 368 memory files, full cron/systemd automation inventory. All staff sessions (Douglas, Campbell, Sejal, Smarandita, Radu, Cal, Cynthia, Isobel, Lia). --- ## 1. The thesis Every day the agents do dev-grade work by hand: push to git, file and update Issue Manager tickets, run UPDATEs against client databases, build ACR images, email exports. Each time, the *mechanics* are re-derived from scratch — the same base64→docker-cp→docker-exec incantation, the same closeout SQL, the same nodemailer boilerplate — rebuilt in `/tmp`, run once, and lost. The measured scale of re-derivation, from the transcript census: | Pattern (sessions containing it) | Kimi (831) | Claude (1,160) | |---|---|---| | UPDATE/INSERT SQL | 790 | 279+ | | KB drafts (`kb_draft_save`) | 789 | — | | SES / email sends | 651 | 40 (nodemailer) | | `srv` bare pushes | 520 | 27 (git push) | | `worker_types` metadata edits | 458 | — | | `az vm run-command` | 516 | 79 | | docker exec / docker cp | 410 | 55 | | SAS link generation | 378 | — | | capcom2 `.cjs` bridge (base64→cp→exec) | 349 | (same recipe, older corpus) | | xlsx export builds | 291 | 16 | | `az acr build` | 111 | 69 | | systemctl restarts | 293 | 18 | Tool-call census (Kimi): `capcom` MCP 7,057 calls (postgres SQL 3,495, sqlserver SQL 1,081), `github_*` 2,870 combined, `issue_sql` 672, `pg_sql` 406, `exec` 1,150. **Operating rule (Douglas's directive):** an agent does a thing one-off *only until we know it's a repeating task*. Then it graduates. The rest of this document is (a) the mechanism that detects repeats and fires the graduation, and (b) the ranked candidate list the review found. --- ## 2. The Refire Mechanism (candidate #0 — build this first) **Finding from the automation audit: nothing on this box logs agent task invocations.** No hooks in any settings.json (Claude or Kimi), no task ledger. The closest existing pieces are `teamCollectPush.ts` (05:10 cron, ships session *digests* to capcom for the 06:00 review) and `dirkGitPush.ts` (05:20, commits). Session-level, not action-level. The refire loop has no sensor. ### Proposed design ``` ACT SENSE DETECT GRADUATE agent does task → hook appends to → nightly counter finds → script/tool built, by hand (1st, agent_ops_ledger signatures seen signature next fires 2nd time) (one JSON line per ≥3× in ≥2 sessions the TOOL, not the agent mutating call) (refire) ``` 1. **Ledger.** A `PostToolUse` hook in `~/.claude/settings.json` and `~/.claude-kimi/settings.json` appends one JSON line per operational call to `~/agent-ops/ledger.jsonl`: `{ts, session, user, tool, signature, params_hash, target}`. The *signature* is the normalized task name — `capcom2-exec`, `im-ticket-response`, `srv-push`, `ses-send`, `acr-build`, `queue-refire` — derived by a small deterministic mapper (tool name + command head + target host/queue/db). Params are hashed, never stored raw (credentials stay out). 2. **Repeat detector.** Nightly cron (piggyback the 05:10 team-collect) runs `detect-repeats.py`: counts per signature over a trailing 30 days, emits `candidates.md` — "seen N times in M sessions by K users, last seen ". Threshold: **N ≥ 3 in ≥ 2 sessions** → flag for graduation. Also watches for *new* signatures crossing the threshold, so the list grows continuously — this is the "grow new software continuously" part. 3. **Graduation states.** Every task is in exactly one state: - **one-off** — agent does it inline (novel work, debugging, judgment). - **repeat-detected** — ledger flagged it; a script gets written to `~/agent-ops/bin/` with a real CLI contract (flags, `--dry-run` default, exit codes). - **standing tool** — promoted to a versioned home (`tc-agent-ops` repo or capcom `listActions()`), callable by any agent *or* fired by cron/queue/webhook. The agent's job becomes parsing the request into the tool's parameters and validating the result. 4. **Refire.** Each standing tool declares its trigger: `manual` (CLI), `cron `, `queue ` (a capcom queue message fires it), or `webhook `. Repeat incidents then refire the *tool* — no agent re-derivation. The auto-attendant SOP model and the data-cartographer daemon are the existing proofs this works. **Effort:** ledger hook + mapper ≈ 1 day. Detector ≈ half a day. The tools below are the backlog it will drain. --- ## 3. The candidate catalog Ranked by (frequency × determinism × pain). Each entry: what agents do by hand today → the proposed deterministic form, its input contract, and its refire trigger. Graduation status: **ad-hoc** (rebuilt every time) / **recipe** (documented in memory, still hand-run) / **scripted** (exists, scattered) / **automated** (no agent involved). --- ### P0 — Universal substrate (used by nearly every session) **1. `run-in-capcom2` — remote script bridge** — *ad-hoc, 349 Kimi sessions, ~20 memory recipes* - Today: base64 a `.cjs` → `az vm run-command invoke -g PATCH -n capcom` → `docker cp capcom2:/app/` → `docker exec capcom2 node` → parse `value[0].message`; plus per-session rediscovery of: 4 KB stdout truncation, single-slot contention (retry loop + unique output token), `/app` path requirement, 40613/40197 retry. - Tool: `run-in-capcom2 [--container capcom2] [--vm capcom] [--timeout N]` — ships, executes, streams output, built-in bounded retry, artifact mode for >4 KB (auto blob+SAS). - Refire: `manual` + library import for every other tool below. **This is the highest-leverage single build — it is the substrate under candidates 4, 5, 6, 8, 10, 13.** **2. `client-sql` — client database resolver + query runner** — *ad-hoc, spine of all ticket work (790 sessions w/ DML)* - Today: agent looks up the client in patch PG (`clients` row via `base_url ILIKE`), decodes server/db (`client_db_credentials`, nasql/us-sql/euro-sql), picks a credential path (AAD SP `SCORM_NODE_WRITER_*` via capcom2, or pymssql `thinkingcap@`), then hand-writes the query boilerplate. Re-resolved from scratch per session. - Tool: `client-sql [--write]` — resolves client → server → db → credential path, executes read-only by default, `--write` gated and logged. Table-name helper included (`tbl{Suffix}` encoder/decoder). - Refire: `manual`; called by 4, 6, 13, 17. **3. `srv-push` — the git closeout that ends every coding task** — *ad-hoc, 520 sessions* - Today: typecheck (`tsc --noEmit` / test suite) → pathspec-scoped commit → `git push srv main` → on race: `git fetch srv` + temp detached worktree + cherry-pick/re-apply → push → worktree cleanup → report "operator deploy needed". Identical in 9 of the 40 most recent Kimi sessions alone. - Tool: `srv-push -m ` — runs the typecheck gate, scoped commit, push, auto-rebase-via-worktree on non-fast-forward, `git show --stat` verification, and prints the operator-deploy handoff line. - Refire: `manual`. Also fixes the shared-clone race class (3 clobber incidents in 10 days) by making the safe sequence the *only* sequence. **4. `im-ticket` — Issue Manager closeout ceremony** — *recipe, ~20 tickets in 3 weeks, 87 ticket-session files* - Today (byte-identical every close): INSERT response row (as Douglas uid 4 / CapGPT uid 7435 via `SQL_ADMIN_*` from the ticketer container env) → UPDATE status (11=RFA, or Patch Update Reqd + Back End Queue) → INSERT internal note → reassign → *remember* that direct SQL sends no client notification email → send the email separately (candidate 5). Filing new tickets = the verified INSERT shape (project 181, status 1, priority 6/8, assigned 1450, group 5, `is_story='0'`). - Tool: `im-ticket get ` / `im-ticket respond --file resp.md --status 11 [--reassign uid] [--email]` / `im-ticket file --title --desc --priority` — one call does the full ceremony transactionally. - Refire: `manual` + later `queue` (failed-job auto-tickets already flow here). **5. `send-ses-mail` + `blob-sas` — email and large-attachment delivery** — *recipe (3 overlapping memories), 651 sessions* - Today: pull `SMTP_*` env from capcom2 or `~/devops-monthly/.env` → hand-write nodemailer script → multipart send → >10 MB → upload to client blob storage → hand-generate 7-day SAS (hand-rolled SAS 403s — must use `@azure/storage-blob`). **Security note: a raw SMTP password appeared inline in at least one transcript** — a helper kills credential-in-prompt permanently. - Tool: `send-ses-mail --to [--cc] --subject --md-file|--attach ` (env resolved internally; >10 MB attachments auto-route to `blob-sas` + link). `blob-sas [--container claudexfer] [--days 7]` (account key read from `/etc/smbcredentials`, never echoed). - Refire: `manual`; embedded in 4, 6, 13, 14, 15. --- ### P1 — High-frequency recipes **6. `xlsx-export` — query → workbook → delivery** — *recipe, 291 sessions, scripts deliberately ephemeral in /tmp* - Today: stream query in capcom2 → build xlsx in-container (inline strings, numFmt 164 dates, deflate+CRC32, self-validate) → ≤7 MB base64 attachment else blob+SAS → ET-labeled filename → cleanup temp scripts/blobs. Rebuilt for RECO (09-08), ESA (08-28), OMVIC proctoring (recurring ticket type!), Lingoda (the one that graduated to cron). - Tool: `xlsx-export --client --sql --out name.xlsx [--sheet-per-x] --deliver email:|sas` — wraps 1+2+5. The OMVIC proctoring export becomes `xlsx-export --profile omvic-proctoring `. - Refire: `manual`; `cron` for any profile that repeats monthly (Lingoda pattern). **7. `queue-refire` — job re-enqueue / dual-write ingress** — *recipe+scripts, used across scormapi / priority-notifications / reportrequests / rolluplps / queuejobs* - Today: copy exact `_requestData` shape from Archive tables → INSERT waiting row in `tblQueuedRequests` (status 'waiting', processedTimestamp 9999-12-31) → POST base64 `` envelope to the queue → dedup by `_requestID` → **pilot 10 → verify drain + zero new failed rows → bulk** → verify archived `_status=completed` + log table + PG events. Scripts exist scattered: `~/bgjobs-rerun.sh`, `~/requeue_97993.py`, `refire-rollup.mjs`, `reset_job.py`. - Tool: `queue-refire --queue --worklist ids.json [--pilot 10] [--execute]` (dry-run default, pilot-then-bulk built in, archive verification automatic). Absorbs the four scattered scripts. - Refire: `manual` + `queue` (failed-job triage auto-refires transient classes). **8. `vm-docker-redeploy` + `live-roll-unblock` — hand-roll & pin-move** — *recipe (3+ memories), ~12 hand-rolls in 4 weeks* - Today: docker inspect live env (`printf %q`+eval, drop IMAGE_TAG/GIT_REV) → stop/rm → recreate on concrete tag → probe `/health` for exact tag on *both* pair nodes. Separately: maestro pin move = next numeric ACR tag + INSERT `pool_deployments` success row → `docker rm -f` → maestro recreates ~1 min. These exist because roll machinery bugs (silent resident roll, env-less roll, gate wedge) are still operator-deploy-pending — the workaround is standing, so it must be safe and uniform. - Tool: `vm-docker-redeploy --rg --vm --container --image [--health-path /health]` and `live-roll-unblock --worker [--bump-fifo]`. - Refire: `manual` (incident-driven); delete these tools when the machinery fixes deploy. **9. `acr-build-tag-bump` — build → tag check → worker_types metadata** — *ad-hoc, 111 sessions (`az acr build`), 458 touching worker_types* - Today: `git rev-parse --short HEAD` → `az acr build -r patchacr -t img:latest -t img: .` → `show-tags --orderby time_desc` → pg `UPDATE worker_types SET …` → redeploy. 7 of 40 recent sessions. - Tool: one parameterized flow with the digest/tag arithmetic and the worker_types UPDATE built in (respecting the build/deploy gate: it *prepares*, the operator promotes). - Refire: `manual`. **10. `control-flip` — compliance control status + audit row** — *recipe, ~6 flip events incl. a 16-control batch* - Today: capcom2 node script, one transaction = UPDATE `compliance_controls` + INSERT audit row (requester email, review dates) → verify via the read path → delete script copies; never UPDATE the audit log. Hand-run because the web route is session-gated and no API path exists. - Tool: `control-flip --status --justification "" --requester [--review-date]` — same tx, same audit rules, executed via candidate 1. Better: a CAPCOM_API_KEY-gated API route so pipelines can flip too. - Refire: `manual` + `webhook` (Secureframe-style external systems later). **11. Fleet/shift-start status report — *recipe, designed for daily, never cronned*** - Today: pull queue_samples + vm_container_health + failed_job_tickets + infra_alerts from patch PG + ticket counts → fill the [SHIFT START] GOOD/WATCH/CRITICAL template → SES to Liam ~08:00 / Cal ~11:15 EDT. Sent 3× by hand 09-03/04; memory says "to make this daily, ask." - Tool: `fleet-status-report [--send liam|cal]` — then simply add two cron lines. - Refire: **`cron`** — this is the cheapest graduation on the list; the template and send path are proven. **12. `new-dbscript` — LMS migration scaffolder** — *convention, dozens of uses* - Today: find max script number in `Campus/DatabaseUpdateScripts/` → create `.sql` → add the `EmbeddedResource` line to `Campus.csproj` (append-only, both rules in memory; violations break upgrades). - Tool: `new-dbscript ` — allocates the number, writes the file with the no-GO header convention, patches the csproj. - Refire: `manual`. --- ### P2 — Campaign-grade kits (partially scripted — formalize, don't rewrite) **13. Report-worker optimization harness** — *playbook + per-report /tmp harnesses, ~10 report repos in 2 weeks* - Playbook is written (`docs/REPORT-OPTIMIZATION-PLAYBOOK.md`); the A/B harness (old DLL vs new on real client data), the dev-e2e re-enqueue, and the failed-report requeue are rebuilt per report. Formalize the harness as `report-harness [--ab] [--e2e ]`. Refire: `manual`. **14. Split/carve kit** — *scripted, scattered (`~/tc-report-split/`, carve-worker.sh, onboard CSVs, request-fix-ci.sh)* - 6 splits in August (bgjob, bulk, express, requests=90 types, reports, catalog). Move the kit into a versioned repo with the cutover ceremony (enable carves → disable monolith → enable router → aftercare) as a checklist runner. Refire: `manual`. **15. Credbroker rollout driver** — *scripted (`/tmp/wire/*.py` went 14/14 hands-off), ~250+ services wired* - The driver works and is in /tmp. Version it (`wire --family --repos r.csv [--smoke]`), keep operator mint/promote gates. Refire: `manual`. **16. Surface scaffold/onboard** — *scripted (scaffold.sh, sync-bridge.sh, ONBOARDING.md), ~26 instances* - Mostly codified; remaining by-hand: operator stand-up/Onboard/grant steps and the LV per-surface wave recipe. Add `surface-onboard --check` (verify-only mode against the runbook). Refire: `manual`. **17. Failed-job triage** — *partially scripted (`bgjobs-rerun.sh`), 754-ticket triage 09-01* - Add `failed-triage --queue [--since]` → census (deterministic REST enumeration, `_processorID` attribution) → signature classification (noise vs real) → report + optional auto-refire of transient classes via candidate 7. Refire: `manual` + `cron` weekly digest. **18. TinyAward PG reload** — *runbook + scripts, 4 runs ("Campbell asks each time")* - Already ~15 min and documented; formalize as `ta-pg-reload [--verify-only]` in a repo (currently lives in runbook + memory corrections). Refire: `manual`. **19. Metadata-fields extraction** — *scripted, client-parameterized; ARCA gate GREEN* - The extract→validate→review chain is built; the Azure upload phase is unbuilt. Finish it: `metadata-extract --upload`. Refire: `manual` + `cron` per onboarded client. **20. Monitor templates** — *ad-hoc per concern* - watch.py skeletons (keyset-paged census, blob canary, bounded 40613/40197 retry, alerts-only) exist per incident and are re-derived per concern. Template-ize: `new-watch --probe sql|blob|http --threshold ...` emitting an arm-ready Monitor script + re-arm command. Refire: `manual` (armed per incident). --- ### P3 — Hygiene & glue **21. Shared-clone git guard** — `~/capcom` hosts ~32 concurrent sessions; 3 clobber incidents in 10 days. A wrapper (or pre-push hook) enforcing pathspec-only commits + `git show --stat` verify + fetch-rebase-before-push. Subsumed by candidate 3 if `srv-push` is mandatory there. **22. Deliverables router** — `deliver --to ` encoding the rules now kept in 3 memory files: `/media/shared//` via `sudo cp` (never For-Douglas for others), Liam compliance docs → claudexfer (md draft + PDF), Liam transcripts → claudexfer `sessions/` only. Misrouted twice this month. **23. TCOV schema trio lint+upload** — `schema-publish `: lint via `POST /api/schema/lint`, validate Draft 2020-12 + sample, upload trio to `tcsettings/schemas` with the layout rules (no extra prefixes, category match), drift check local-vs-Azure. 88 sessions touched schema lint/upload. **24. Disk-sweep stale-clone rule** — hourly housekeeping is automated, but the 09-08 manual sweep freed 142 GB from stale clones the script doesn't cover. Add the SAFE-to-delete rule (clean porcelain + no unpushed vs `git ls-remote` srv + tar-before-delete) to the timer. **25. Front Door cert rotation helper** — `cert-rotate ` writing the `tls_certificates`/`cert_deploy_jobs` rows in the audited shape + post-propagation re-probe (apex ~45 min). Partially daemonized already. **26. Loom git-ops** — `loom-git `: temp-clone merge, push, `reset --hard` in loom clone, npm ci. Recipe-documented, 4+ uses. --- ## 4. Already graduated (proof the model works — do not re-propose) ~48 active cron entries + 4 custom systemd timers on this box, all built by prior agent work: | Automated task | Trigger | |---|---| | ~32 compliance evidence pipelines (collect → publish-to-capcom) | daily crons 03:30–08:45 | | VM-02/VM-05 vuln collectors, trivy (VMs + ACR), Semgrep sweep | daily/weekly | | PI suite + PI-03-1 quarterly collector | daily 06:45 / quarterly | | suspend_data truncator (XML-shred) | weekly Sun 02:00 systemd | | Lingoda review-notes export → SES | monthly `13 8 1 * *` | | team session collect (05:10) + review (06:00); dirk git push (05:20) | daily | | dirk brief 06:30, crew-nightly 03:17, compliance digest + obligations-nag | daily/weekly | | devops-checks bucket (disk alerts/cleanup); disk housekeeping | hourly | | IM mirror sync | daily 05:00 | | capgpt on-prem index | daily 02:30 systemd | | Auto Attendant SOP execution (autonomous remediation) | 60 s event poll | | data cartographer daemon (scan + onboard-on-build) | 15 min incremental | | baker builds + capcom web self-deploy | per-push CI | | DevOps monthly report | Friday-gated cron | **Found broken during the audit (fix while here):** `gdpr35-privacy-vendors` cron points at a `refresh-and-publish.sh` that no longer exists on disk; `mother-vision.timer` is a dangling symlink (unit file deleted). --- ## 5. What must NOT be codified The boundary of the model. Agents keep: novel root-cause debugging (every deadlock, plan-lottery, and CPU cliff was different), first-instance ticket investigation, schema *authoring* (judgment about the domain), incident command (the watcher/dump/rollback *tools* get codified; the decisions don't), and anything touching credentials policy or the operator deploy gate. The rule is: **judgment stays with the agent; mechanics graduate to software.** The operator deploy gate itself is not a candidate. It appeared in ~60 memory entries and 32/40 recent sessions — it is policy, not missing tooling. What the candidates do is shrink everything *up to* the gate so the gate is the only manual step left. --- ## 6. Proposed build order | Wave | Items | Why first | |---|---|---| | 0 | Refire Mechanism (ledger hook + repeat detector) | The sensor; without it this list is a snapshot, not a process | | 1 | 1 `run-in-capcom2`, 2 `client-sql`, 3 `srv-push`, 5 `send-ses-mail`+`blob-sas` | Substrate everything else calls; kills credential-in-transcript exposure | | 2 | 4 `im-ticket`, 6 `xlsx-export`, 7 `queue-refire`, 11 fleet-status cron | Highest-frequency ceremonies; #11 is two cron lines | | 3 | 8, 9, 10, 12 | Incident/deploy mechanics | | 4 | 13–20 campaign kits | Formalize, mostly not rewrite | | 5 | 21–26 hygiene | As touched | Each wave ships with its signature registered in the ledger mapper, so the detector measures the *drop* in hand-runs — that's how we know the graduation is working. --- *Method note: quantitative census via tool-call extraction over all 831 Kimi JSONL transcripts and pattern counts over all 1,160 Claude session transcripts; qualitative recipes extracted from 368 memory files, 16 sampled Claude transcripts across 12 filename clusters, and the 40 most recent Kimi sessions; automation inventory from crontab, systemd, and the ~/ pipeline directories. Full extraction detail in session transcript `~/.claude/sessions/2026-09-08-deterministic-software-candidates.txt`.*