Cleanup Retention Scope
homeboy cleanup --include terminal-runs is the lifecycle owner for terminal
observation records, and since #10316 it is the only surface for them: the
homeboy runs retention specialist carried no argument the aggregate could not
express and was deleted. Its dry-run output includes each candidate run, its
registered persisted-artifact cleanup plan, and any agent-task lifecycle
directory. Apply revalidates local artifact paths, removes eligible artifact
bytes and lifecycle directories, then removes the terminal database records.
Unsafe existing local artifact paths keep the run and its lifecycle directory.
The existing cleanup inventory remains the only planner. This change does not add a second cleanup engine.
Durable Aggregate Apply
homeboy cleanup --apply submits the existing aggregate inventory to the local
controller’s durable job queue and returns a compact receipt immediately. The
daemon continues the apply if the submitting terminal disconnects. Use
homeboy cleanup status <job-id> for compact progress,
homeboy cleanup status <job-id> --full for the durable job evidence, and
homeboy cleanup resume <job-id> to start a queued recovered job. The durable
controller job retains the full category evidence; normal submission and status
responses deliberately report only totals and retrieval commands. The
runner-downloads category remains explicit opt-in in durable applies.
Retention resolves the observation store and the artifact root once per sweep.
Both used to be re-derived inside every per-run artifact plan, so a default
--limit 1000 sweep opened thousands of SQLite connections and re-ran the
migration ladder for each one. The apply pass still re-plans each run
immediately before deleting its bytes: the planning loop walks every candidate
before any deletion happens, so the earliest plans are already stale by then.
Orphaned Artifact Bytes
homeboy cleanup --include orphaned-artifact-bytes
homeboy cleanup --include orphaned-artifact-bytes --applyEvery other artifact-root cleanup surface is driven by artifacts rows, so
bytes written before their row existed are structurally invisible to all of
them. This category reclaims the two artifact-root path families that are
created outside any durable journal and are otherwise only reclaimed by an
in-process cleanup branch, which SIGKILL and OOM skip:
<artifact_root>/<run_id>/.artifact-<uuid>.staging— the staging sibling written before a file artifact is hard-linked into place.<artifact_root>/_scratch/patch-<label>-<uuid>/— the working-tree baseline copy taken by daemon patch capture, reclaimed only byimpl Drop.
It is deliberately not a generic "filesystem path with no database row"
reaper. The artifact root is a shared namespace: runner/, runner-attach/,
runner-exec-attach/, agent-task/, agent-task-loop-controller/,
controller-scratch-recovery/, recovered-runner-artifacts/,
executor-finalized/, preview-consumer/, and _scratch/ are all owned by
other subsystems and carry no artifact row at those paths. Artifact bytes are
also written before their row is inserted by design, so row absence is the
normal state of a publication that is currently succeeding. Row absence
therefore proves nothing, and the database is not consulted.
Ownership is instead proven by name shape — both families come from a single private constructor whose format no other writer emits — plus a fixed 24-hour age floor covering the in-flight window. The floor is not operator-overridable. Anything else at those paths, including a lookalike name without a parseable UUID, a staging-shaped directory, a scratch-shaped file, or a symlink, is reported and kept.
Reported sizes are advisory. Removal is a pure function of name shape, entry
type, age, symlink-freedom, and root containment; a failed size measurement
reports size_measured: false and does not change the verdict in either
direction.
Runner Download Cache Retention
homeboy cleanup --include runner-downloads
homeboy cleanup --include runner-downloads --applyThis category is opt-in only: a bare homeboy cleanup --apply does not
sweep it.
<artifact-root>/runner is produced by exactly one writer — the default output
path of homeboy_lab_runner::evidence::download::download_remote_artifact,
which lays bytes down as <artifact-root>/runner/<runner-id>/<run-id>/<file>.
Every reachable caller of it is a fetch someone asked for (runs artifact get,
runs artifacts <run-id> --pull, lab apply, evidence mirroring, the HTTP
artifact endpoint), and runs artifact get reports that path back as the
location of the operator’s file. So this is not scratch; it is the operator’s
copy.
Before #10564 the implementation was an unconditional fs::remove_dir_all of
the whole root. Its only checks were path-containment ones — --run-id requires
--runner, each filter must be a single normal path component, the root must be
a real directory — which prove the deletion stays inside the cache and prove
nothing about whether the bytes are dead. A bare sweep therefore deleted
artifacts pulled seconds earlier.
The predicate now requires all of:
- Ownership by name shape. A candidate must be a real directory at exactly
runner/<a>/<b>with no symlink at either level — the only shape the single writer emits. Loose files, symlinks, and non-canonical depths are reported and never removed. As with orphaned artifact bytes, the database is deliberately not joined: bytes here are written before, and usually without, any localartifactsrow, so row absence is the normal state of a download that is succeeding. - Age floor over the newest byte. The newest modification time anywhere in
the cache directory must be at least 24 hours old (the shared
cleanup::RUNNER_MIN_AGE_HOURS). Taking the newest, not the oldest, is what makes one fresh pull re-arm the whole cache directory. The floor is not operator-overridable. - Non-terminal-run veto. The observation store is consulted only in the
retain direction. A running run matching the
<run-id>component retains the cache; a missing row never authorizes removal. - Fail closed. An unreadable or future-dated mtime, an unwalkable subtree, a path that does not resolve inside the artifact root, an observation store that cannot be opened, or a running-run scan that hit its bound all retain.
Removal is per cache directory, so a stale cache and a fresh one under the same
runner are decided independently; the cache root itself is never removed, and an
emptied <runner-id> directory is pruned only by a non-recursive remove_dir
that refuses a non-empty directory. Sizes are advisory: a failed measurement
reports size_measured: false and moves the verdict in neither direction.
--runner and --run-id narrow which candidates are considered; they never
waive a check.
The remaining gap, and the reason for opt-in: the writer emits the same name shape for an operator’s deliberate pull and for an internal auto-fetch, so "reclaimable transient" and "the operator’s copy" are not distinguishable by name. Making them distinguishable requires the writer to tag its output.
Lab Failure Retention
Lab offloads delete run-scoped workspaces on every known terminal outcome by
default. --preserve-workspace-on-failure is the bounded debugging profile: it
keeps failed or cancelled materialization state, registers it as
delete_after_ttl in the workspace lifecycle metadata, and uses
lab.runner_workspace_ttl (default P7D) for existing runner workspace
pruning. The terminal report identifies the policy, outcome, lifecycle owner,
retained location, and homeboy runner workspace prune <runner> --apply --min-age-hours 0 reclaim command.
Detached, in-flight, and otherwise uncertain daemon ownership always relinquishes the local cleanup handle. Those paths remain fail-closed and are never treated as terminal deletion or debug-retention outcomes.
Runner Binary Cache Retention
homeboy cleanup --include runner-binary-caches composes the runner-owned
binary cache lifecycle into the aggregate planner. Each configured runner is
inventoried independently through direct local or SSH execution; an unavailable
runner becomes a scoped skipped category and does not abort healthy runners.
Only canonical refresh (homeboy-*) and dev-sync (dev/<16-hex>) slots with a
regular expected binary are eligible after the fixed 24-hour age floor. The
configured homeboy_path, slots with open files or process working directories,
symlinks, malformed or partial layouts, and candidates that change identity are
retained. Apply revalidates selection, inode identity, layout, symlink state,
and process ownership immediately before removal.
Runtime Temp Root Relocation
The runtime temp root is what Homeboy exports as TMPDIR, TMP, and TEMP to
every child process it launches, so it absorbs build intermediates from
workloads such as cargo build. Through #11125 it defaulted to
<config root>/runtime/tmp while the remote-shell protocol in the same module
resolved $HOMEBOY_DATA_DIR/runtime/tmp. The two resolvers disagreeing meant an
operator who moved storage to a large volume moved everything except the bytes
that actually grow. It now defaults to <data root>/runtime/tmp, matching the
remote protocol. HOMEBOY_RUNTIME_TMPDIR remains the highest-precedence
override; see Configuration Reference.
Entries an older binary left under the config root are not orphaned. They are
tracked by owner metadata and are only ever reclaimed by the runtime-temp sweep,
so homeboy cleanup --include runtime-tmp — and the aggregate homeboy cleanup
— drains the superseded root as part of its normal pass. There is no flag and no
separate command. The drain reuses the full ownership protocol rather than
deleting blindly: active pins, live owner PIDs, active invocation leases, running
persisted runs, and the corrupt-metadata quarantine grace all protect a
superseded entry exactly as they protect an active one, and the configured
retention window still applies. The sweep shares one bounded row budget across
roots, restarts the superseded root from the beginning each invocation (nothing
allocates there, so it needs no cursor), and removes the superseded root
directory once it is empty. It never creates that directory: a root that was
never used is left alone.
The drain is scoped to default resolution. If HOMEBOY_RUNTIME_TMPDIR is set,
that root is the only one swept, because the old resolver honored the same
override and there is no superseded default in play. Residue under the config
root from before an override was adopted is reported by
homeboy cleanup retained-storage.
Retained Storage Accounting
homeboy cleanup retained-storage explains where disk went without deleting
anything. It accumulates read-only plans from the controller-runtime,
shared-Cargo-target, controller-scratch, and runtime-temp categories, from the
SQLite observation index, and — since #10316 — from the artifact root itself
through persisted-run-artifacts, runner-downloads, and
orphaned-artifact-bytes. Before that the one command whose purpose is "where
did my disk go" never called artifacts::root() at all.
Records carry a liveness, and reclaimable is reported separately from the
retained totals. "Cleanup cannot free this" and "cleanup has not freed this yet"
are different answers, so retained_bytes and reclaimable_bytes are never
summed together. safe_next_commands names a reclaim command for every category
the report can produce a record for.
The SQLite row is explicitly scoped to the index, not the artifact bytes it indexes; the artifact payloads are accounted for by the artifact-root categories and dwarf the database.
Protected persisted artifacts are summarized as one counted record rather than one zero-byte row each: only rows classified for removal carry a measured size, and an unmeasured size is reported as zero and never inferred.
The additive filesystem object is the root-accounting contract. It inventories
every top-level data-root entry, including artifacts, cargo-targets,
controller-runtimes, controller-scratch, and the SQLite store, and labels
unrecognized entries unknown/unmanaged. It also includes configured artifact
roots outside the data root as managed/external. apparent_bytes is the sum
of file lengths; physical_bytes is allocated filesystem blocks. Sparse files,
hard links shared across top-level stores, directory metadata, and allocation
granularity can make the top-level sum differ from root physical usage; the
report states the direction and byte difference rather than hiding it. Existing
retained_bytes and lifecycle aggregates remain unchanged for consumers.
Since #11126 the config root (~/.config/homeboy) is probed as a fourth
root, exposed as filesystem.config_root and as a config_root entry in
top_level. It is usually a different volume from the data root and holds
rigs/{id}.state/logs/ (unbounded append), rig-packages/, backups/,
agent-runtimes/, runner-sessions/, runner-lease-evidence/,
runner-job-execution-context/, and preview-ingress/routes/. It was never
probed before, so during the 2026-07-29 outage this report described a healthy
system while the volume it omitted was full. Like the artifact and Cargo roots,
it is classified managed/external and excluded from the data-root
reconciliation totals, because it is not part of the walk those totals balance
against. Double-counting is prevented by the same starts_with containment test
the other external roots use, applied in both directions: two roots that merely
share a filesystem occupy disjoint inodes and are not double-counted, but a
config root nested in the data root (or a data root nested in the config root)
is listed once.
runtime/tmp used to live on the config root and no longer does (#11125), so
after that change the config root should be small. It is still probed: an
unprobed volume is an invisible one, and everything else in the list above
still lives there.
Remaining Scope
The following Issue #8648 portions remain independently owned and are not implemented by terminal-run retention:
- Crash-orphaned
/tmp/hb-<uid>invocation-root inventory and age-pruning. - Reverse reconciliation for the subsystem-owned artifact-root subtrees listed above. Each needs its own lifecycle owner; none of them can be reclaimed by a row join.
- Controller-scratch index compaction for missing or deleted terminal tombstones.
- Aging removed task-worktree records out of workspace registries.
- Re-evaluating whether
runner-downloadscan rejoin the bare sweep now that the writer tags its output (#10585). The tag exists and cleanup reads it, but every cache directory predating the tag is untagged and therefore retained, so the default-sweep decision needs its own evidence after a backfill window. - Detecting collisions between task-worktree and adopted-workspace registry handles.