`homeboy runner`

Synopsis

sh
homeboy runner <COMMAND>

runner manages durable execution backends. SSH runners are a capability on a homeboy server record, so the common Lab flow uses one ID for the machine and its runner. Local runners remain standalone because they describe this machine rather than an SSH server. Both storage shapes share the same runner capability contract: workspace root, settings, environment, secret references, resources, and policy.

Runner configuration separates printable environment from secrets:

  • env is for non-secret values that are useful in diagnostics, such as HOMEBOY_PUBLIC_ARTIFACT_BASE_URL.
  • secret_env is for execution-time secret references like { "env": "NAME" } or { "file": "~/.config/homeboy/secrets/name" }.
  • Command output redacts sensitive names in env and prints only secret_env references, never resolved secret values.

Subcommands

Daemon Recovery Migration

For daemon child recovery, use the canonical exact-evidence command:

sh
homeboy daemon recover-missing-child-identity --lease-id <expected-lease> 
  --recorded-daemon-pid <recorded-daemon-pid> 
  --recorded-daemon-endpoint <recorded-daemon-endpoint> --job-id <job-id> 
  --child-pid <child-pid> --child-starttime-ticks <child-starttime-ticks>

The released homeboy daemon adopt-orphan --recover-missing-child-identity and --confirm-untracked-child-dead <job-id> flags are accepted only as paired migration aliases. They return this exact remediation and never terminalize work. Runner connection recovery should surface the exact daemon command rather than retrying those aliases.

lifecycle

Inspect runner workspace lifecycle and finalization readiness.

add

sh
homeboy runner add <id> --workspace-root <path>
homeboy runner add <server-id> --server <server-id> --workspace-root <path>
homeboy runner add --json <spec>

Options:

  • --kind local|ssh: explicit runner kind. Defaults to ssh when --server is set, otherwise local.
  • --server <server-id>: existing homeboy server record for SSH runners. For SSH runners, <id> must match <server-id>.
  • --workspace-root <path>: workspace root on the runner machine.
  • --homeboy-path <path>: Homeboy binary path on the runner machine.
  • --daemon: marks the runner as daemon-preferred for future commands.
  • --concurrency-limit <n>: maximum concurrent workflows this runner should accept.
  • --artifact-policy <label>: artifact policy label reserved for future execution commands.

enable

sh
homeboy runner enable <server-id> --workspace-root <path>
homeboy runner enable <server-id> --workspace-root <path> --concurrency-limit 4 --artifact-policy copy

Enables runner capability on an existing SSH server. This is the recommended onboarding path for any machine that should accept Homeboy runner work:

sh
homeboy server create <runner-id> --host <host> --user <user> --port 22
homeboy runner enable <runner-id> --workspace-root <workspace-root> --concurrency-limit 4 --artifact-policy copy
homeboy runner connect <runner-id>

After this, <runner-id> is both the server ID and the runner ID.

Commands that are both resource-policy hot and portable for Lab offload (agent-task controller from-spec --resume, agent-task controller run-from-spec, agent-task controller materialize, audit, full lint, test, bench run, fuzz run, and trace) auto-select a default runner when --runner is omitted. Selection is conservative:

  • --runner <id> always wins.
  • --placement local keeps a portable hot command local even when a default Lab runner is available, unless a command-specific host policy denies local execution. For benchmarks, homeboy config set /bench/local_execution '"denied"' makes local homeboy bench execution fail closed until the global config is changed back.
  • lab.preferred_runner is used when it names an SSH runner, even if that runner is not connected yet.
  • Without lab.preferred_runner, Homeboy auto-selects only when exactly one SSH runner is configured or exactly one SSH runner is already connected.
  • With a preferred or uniquely configured Lab runner, homeboy bench <component> routes to Lab directly; --runner <id> is only needed to override an ambiguous or non-default runner selection.
  • Local runners are never auto-selected.
  • If the auto-selected runner is disconnected, Homeboy attempts a short bounded runner connect before execution. Connection failure prints the reason and falls back to local execution.
  • Explicit --runner <id> also attempts to connect a disconnected runner, but connection failure remains a command error instead of falling back silently.

Observation metadata records the routing decision under metadata.lab_offload when an observed run is created. The stable contract is schema: "homeboy/lab-offload/v1": source is automatic or explicit; status is offloaded, skipped, or fallback; successful offloads include runner_id plus remote_workspace; local fallback records the runner and fallback_reason; skipped local execution records why no automatic offload was used, such as placement_local_override or no_default_runner. The same object also carries plan_id and plan-derived phase fields including sync_mode, capability_preflight, extension_parity, and patch_captured.

Commands launched through non-local homeboy runner exec run with HOMEBOY_RUNNER_HOSTED_EXEC=1 and the private HOMEBOY_RUNNER_PLACEMENT_RESOLVED=1 marker. Nested runner commands such as homeboy agent-task cook pass the non-interactive resource preflight without a public placement override because the work is already intentionally hosted on the selected runner.

For reviewer-facing output, treat runner exec stdout as operator context and record durable artifacts through the run/artifact loop. Pass --run-id <id> plus --artifact <path> for files/directories and --summary <path> for compact summary evidence produced by the runner command. JSON summary files are echoed in the structured runner.exec response under structured_summaries, with the run id, runner id, command, declared path, and promoted artifact id/path so LLM workflows can consume the compact result without spelunking remote stdout or artifact directories. See Artifact loop for runner and matrix workflows for static HTML and matrix examples, plus proposed attach/promote command shapes for successful runner jobs that produced files but registered zero artifacts.

Lab offload support is intentionally command-specific:

CommandAuto offloadExplicit --runnerDecision
agent-task cook / agent-task run-planYesYesPortable agent-task execution when the command has deterministic gates where required.
agent-task controller from-spec --resume / agent-task controller run-from-spec / agent-task controller materializeYesYesPortable controller spec materialization defaults to Lab when a default runner exists.
agent-task controller resumeNoYesRunner-resident controller lifecycle work.
agent-task retry --runYesYesRetry plus execution follows the portable agent-task run path.
agent-task run / run-next / status / logs / artifacts / review / providersNoYesRunner-resident inspection and queue interaction.
agent-task auth statusNoYesRunner-resident auth diagnostics.
audit full workspaceYesYesSafe single-workspace replay after snapshot sync.
audit --changed-sinceNoNoRuns locally for now because changed-since audit depends on git base refs that Lab sync may not have fetched. The Lab plan records the skipped local-only decision.
bench run / default bench runYesYesSafe single-workspace replay; local baseline/ratchet writes are treated as mutation flags.
buildNoNoLocal quality gate; extension-owned build commands are not represented in the portable Lab contract yet.
fuzz run / default fuzz runYesYesSafe single-workspace replay with fuzz-specific case/result artifacts.
lint full workspaceYesYesSafe single-workspace replay; --fix is treated as a mutation flag.
lint --changed-since / lint --changed-onlyNoNoRuns locally for now because changed-file scopes are not represented in the Lab portability contract yet. The Lab plan records the skipped local-only decision.
refactor --from audit / refactor --allNoYesSource refactor runs with command-owned mutation flags and runner-side workspace controls.
reviewYesYesScoped audit/lint/test umbrella using the same portable quality gates.
rig checkNoYesRunner-side rig validation without starting local services.
test full workspaceYesYesSafe single-workspace replay with runner extension parity preflight.
test --changed-sinceNoNoRuns locally for now because changed-since test selection depends on git base refs that Lab sync may not have fetched. The Lab plan records the skipped local-only decision.
traceYesYesSafe single-workspace replay with Playwright/browser capability gate.
tunnel preview-consumer runNoYesRunner-hosted preview consumer execution.
tunnel service expose / tunnel service startNoYesRunner-hosted private service tunnel lifecycle.
rig upNoNoStays local because rig pipelines manage local services, leases, ports, and declared filesystem paths that the current single-workspace snapshot cannot safely mirror.
fleet execNoNoStays local because fleet execution depends on local fleet/project/server config before opening SSH sessions to each project; runner-side config parity is not guaranteed.

Local-only resource-pressure commands still get resource-policy warnings, but those warnings explain why Lab offload is unavailable instead of suggesting --runner.

Stale daemon diagnostics and local recovery

When a connected SSH runner has a non-fresh daemon admission session, normal runner exec remains fenced: it does not rotate or submit work through the stale daemon. Use the diagnostic transport for bounded direct SSH inspection:

sh
homeboy runner exec <runner-id> --ssh -- homeboy daemon status

Diagnostic SSH is available only when the runner is disconnected or its daemon admission is non-fresh. It bypasses daemon job admission, does not materialize extensions or rotate the tunnel, and preserves the runner id, command, and an optional --run-id in the normal runner execution evidence.

If runner disconnect reports endpoint_identity_mismatch, it retains local state and never sends a lifecycle mutation to the foreign endpoint. After the read-only SSH status probe proves an authoritative zero active-job count, retire only the controller-local state with:

sh
homeboy runner disconnect <runner-id> --local-recovery

This operation does not claim the remote daemon was stopped or torn down.

Configure a preferred runner with:

sh
homeboy config set /lab/preferred_runner '"<runner-id>"'

doctor

sh
homeboy runner doctor local
homeboy runner doctor <runner-id>
homeboy runner doctor <runner-id> --path <component-path> --extension rust
homeboy runner doctor <runner-id> --require-tool zip --require-tool unzip
homeboy runner doctor <runner-id> --scope lab-offload
homeboy runner doctor <runner-id> --scope lab-offload --repair

Diagnoses a local or configured SSH runner without mutating it. A connected session is read once from controller state and checked through a bounded /health request; doctor does not reconcile generations, create an exec job, replace the session, or tear down its tunnel. Use local, localhost, or self to inspect this machine without creating a runner record. The JSON payload uses command: "runner.doctor" and includes runner_id, status, capabilities, and warning/error details when a capability probe fails. status: "error" means the runner is not ready and exits with status 1; ok and non-blocking warn reports exit 0. The command has no report-only mode.

Use doctor before connect when you need to know whether Homeboy, Git, SSH, and the configured workspace root are usable on the target machine.

preflight

sh
homeboy runner preflight --request '{"schema":"homeboy/placement-readiness/v2","runner_id":"homeboy-lab","allow_queue":false,"durable_workload":false,"invocation":{"kind":"agent_task_cook","provider":"openai","source_path":"/workspace","selector":"openai.default","model":"gpt-5"}}'
homeboy runner preflight --request '{"schema":"homeboy/placement-readiness/v2","runner_id":"homeboy-lab","allow_queue":true,"durable_workload":true,"invocation":{"kind":"capability_audit","source_path":"/workspace","capability_id":"browser"}}'

preflight creates no run, rig lease, runner job, workspace, or connection. Its homeboy/placement-readiness/v2 response is ready, queueable, or blocked with exact predicates and typed executable recovery actions (including action ID, argv, safety, required confirmations, and evidence). Queueing is limited to a durable workload on a full reverse runner. Execution revalidates live admission facts through the same compiled admission plan before reserving a daemon admission, so this result is evidence, not a reservation. Wrapper order is preflight, durable run persistence, then setup and materialization.

Safety manifest metadata marks runner connect and runner work as explicit operator lifecycle actions. They do not currently expose dry-run contracts; use runner doctor for preflight diagnostics before changing runner lifecycle state.

Use --scope lab-offload before serious Lab evidence runs. It adds checks for the configured runner Homeboy command, bare homeboy PATH resolution, preferred runner binaries, connected daemon health readiness, and Sample Runtime runner-path freshness signals. When Homeboy can identify a safe exact recovery command, the check includes that remediation instead of leaving the operator to infer it from logs.

Use --scope lab-offload --repair for the narrow self-healing path. Today this reconnects a failing direct Lab runner daemon and reruns the daemon exec probe. It does not upgrade binaries, rewrite runner paths, or refresh Sample Runtime caches; those remain explicit operator actions because they can be expensive or depend on environment-specific paths.

runner status includes generic selected_lab_runner.executable_requirements from declared agent runtime manifests. Each entry names the runtime, requirement id, non-secret env vars, candidate executable names/paths, optional version arguments, and install hint. Homeboy owns the generic semantics: runtimes declare the executable they need; runner diagnostics expose the declaration without embedding product-specific checks in core.

runner status also includes selected_lab_runner.selected_runtime_runtime for the legacy runner job environment projection. That block shows the configured WP Sandbox CLI, managed cache source/binary, expected @automattic/sample-runtime-playground and @automattic/sample-runtime-core paths, a runner-side probe command that prints the exact effective runtime and git SHA, and warnings when configured paths mix a stale checkout with the managed cache.

refresh-homeboy

sh
homeboy runner refresh-homeboy <runner-id> --ref main --reconnect
homeboy runner refresh-homeboy <runner-id> --source https://github.com/Extra-Chill/homeboy.git --ref <branch-or-sha>
homeboy runner refresh-homeboy <runner-id> --select /path/to/homeboy --reconnect
homeboy runner refresh-homeboy <runner-id> --ref <branch-or-sha> --dry-run

Builds or selects the Homeboy binary used by runner/Lab job execution without mutating a runner’s primary checkout. Materialize mode clones/fetches the Homeboy source into a managed runner-side cache under the runner workspace root unless --target-dir is supplied, hard-resets that isolated checkout to the requested ref, builds target/release/homeboy, probes self identity, and then updates the runner homeboy_path to the clean binary. Select mode skips the build and probes the exact binary passed to --select before updating homeboy_path.

Use --reconnect when the active daemon/session should be refreshed immediately. Without it, the JSON output includes follow-up disconnect, connect, and status commands so operators can restart a session at the right time. runner status also reports controller, configured executable, active daemon version, build identity, drift signals, and refresh commands under selected_lab_runner.runner_homeboy.

Reconnect rollback restores the exact previously selected binary and publishes its session. If any rollback phase fails, Homeboy retains runner-sessions/<runner-id>/refresh-partial.json; the error names that path and one refresh-homeboy --select <previous-path> --reconnect continuation command. Do not delete the receipt as a recovery step.

When a configured SSH runner is disconnected, refresh-homeboy automatically uses its configured SSH server transport to run the same managed build/select script with a bounded deadline. The selected binary and runner configuration are updated only after that script reports a clean identity matching the resolved source commit. The fallback is restricted to configured SSH runners; local runners retain their existing direct execution path.

Refreshing back to a release changes only the selected homeboy_path; existing runner environment and resource provenance remain intact.

dev-sync

sh
homeboy runner dev-sync <runner-id> --homeboy-source /path/to/homeboy --reconnect
homeboy runner dev-sync <runner-id> --homeboy-binary /path/to/target/release/homeboy
homeboy runner dev-sync <runner-id> --homeboy-source . --extensions sample-runtime=/path/to/runtime

runner dev-sync is the fast-loop handoff for iterating against a Lab runner without cutting a release. It builds the controller-local Homeboy source checkout unless --homeboy-binary points at a prebuilt local binary, hashes that binary, uploads it to a runner-managed content-addressed slot under <workspace_root>/_homeboy_binaries/dev/<hash>/homeboy, probes it with self identity, and selects that exact runner-side binary using the same selection machinery as runner refresh-homeboy --select.

The selected dev binary becomes authoritative for both daemon and child executor processes: Homeboy updates runner.homeboy_path, prepends the selected binary directory to the runner PATH, and sets HOMEBOY_COMMAND to the selected absolute binary path. This avoids stale homeboy binaries shadowing runner jobs through inherited PATH state.

The command records dev provenance under the runner resources.dev_sync block: binary SHA-256, content hash, runner-side path, source revision, dirty flag, and deferred extension specs. runner show exposes the full runner record and runner status surfaces the same block under selected_lab_runner.runner_homeboy.dev_sync so operators can see dev mode at a glance.

--extensions id=path is accepted and recorded in the dev-sync provenance. The binary handoff is first-class in this release; extension source relink/parity is deferred and remains visible in extensions_deferred output rather than silently pretending extension parity was changed.

The JSON output includes next_actions, including a ready-to-copy runner status command and the release/ref refresh command that reverts the runner from dev mode.

Pass one or more --require-tool <command> values when a provider or job path knows it needs additional runner-side commands before starting expensive work. Doctor resolves each command on the runner PATH and reports missing requested tools as tool.required.<command> errors with install/setup remediation. This is generic: provider layers declare their own tools; Homeboy core only checks command availability.

Programmatic runner execution can use the same generic boundary through RunnerCapabilityPreflight.required_commands. Those commands are checked before remote execution starts, alongside existing required tools, components, and environment variables.

Runner-managed dependency sources

Portable Lab/evidence runs should declare runner dependency sources instead of smuggling local checkout paths through workflow-specific environment variables. Homeboy’s generic contract is the extension/component setting validation_dependencies: each entry is a component id or explicit checkout path that the runner workspace sync treats as managed source input.

For each declared dependency, homeboy runner workspace sync:

  • resolves a sibling checkout, registered component, or deterministic clone;
  • rejects dirty, stale, divergent, missing-upstream, or ambiguous Git state;
  • runs the dependency install/build lifecycle in a prepared copy;
  • materializes the prepared dependency beside the primary runner workspace; and The JSON output includes validation_dependencies, with each dependency’s id, role, controller local_path, runner remote_path, and evidence_path. Bench, trace, eval, and provider layers can consume that generic output to populate their own path settings without knowing Lab filesystem layouts or dependency-specific environment variable names.

Example manifest fragment:

json
{
  "extensions": {
    "provider-id": {
      "settings": {
        "validation_dependencies": ["runtime-component"]
      }
    }
  }
}

Pass one or more --extension <id> values to validate extension parity before Lab offload. Doctor runs the same homeboy extension show <id> contract on the target runner that test offload uses at execution time. --path sets the probe working directory when the extension should resolve from a specific component checkout. Missing extensions are reported as extension.parity errors with an install command such as homeboy extension install <source> --id rust.

Lab offload for portable resource-pressure commands uses the same capability vocabulary before running on an explicit --runner. Homeboy currently gates lint, test, audit, bench, and trace against the source worktree’s lightweight tool signals:

  • package.json requires node and npm.
  • pnpm-lock.yaml requires node and pnpm.
  • composer.json requires php and composer.
  • Docker/Compose files require docker.
  • trace requires Playwright plus browser binaries.

When an explicit runner is missing required tools, the command fails before workspace sync with a runner_capabilities validation error and remediation. The same central policy returns a local fallback reason for future automatic Lab offload selection.

refresh-plan

sh
homeboy runner refresh-plan --runner <runner-id> --workspace . --runner-cwd /runner/workspaces/app --run-id matrix-refresh-1 --produces artifacts/review --summary artifacts/review/summary.json -- npm test

Plans a runner-backed refresh loop before dispatching matrix-style work, without executing the workload. It composes the existing runner/workspace/run artifact primitives into one envelope: a RunnerExecutionEnvelope, a handoff describing workspace mapping and Homeboy binary provenance (controller CLI vs. runner configured binary vs. active daemon, with version/build drift diagnostics), declared evidence/artifact paths, and the ordered next_commands to verify the runner, sync the workspace, run the refresh, and inspect the produced evidence. Source and fixture paths passed with --source/--fixture must exist before the plan is emitted; --sync-mode accepts snapshot, snapshot-git, or git.

Declared produced evidence uses --produces and --summary. --produces was spelled --output through 0.321.0, which collided with the process-wide global --output <PATH> ("write the JSON result envelope to this file") and left both meanings unreachable in a clap-defined order (#10566). On this command --output now unambiguously means the global envelope path; use --produces to declare a produced artifact.

connect

sh
homeboy runner connect <runner-id>
homeboy runner connect <runner-id> --adopt-orphan-lease <lease-id>
homeboy runner connect <runner-id> --adopt-orphan-lease <lease-id> --confirm-untracked-child-dead <job-id>
homeboy runner connect <runner-id> --reconcile-leaseless-orphans
homeboy daemon reconcile-leaseless-orphans
homeboy runner connect <runner-id> --recover-missing-lease-state <lease-id> --recorded-pid <pid> --recorded-endpoint <loopback-host:port>
homeboy daemon recover-missing-lease-state --lease-id <lease-id> --recorded-pid <pid> --recorded-endpoint <loopback-host:port>
homeboy runner connect <controller-id> --reverse --reverse-runner <runner-id> --broker-url <url>

--confirm-pid-dead, --confirm-no-daemon-owner, and --confirm-control-plane-lost are deprecated no-ops on both surfaces; the runner proves each fact itself before mutating anything. See the deprecation table in daemon.md. They remain accepted for one release, so released runbooks and composed repair commands keep working, but supplying one without its recovery mode is refused — a confirmation selects no recovery on its own. --recorded-pid and --recorded-endpoint are not confirmations: once a runner’s state record is gone it cannot recompute them, so they stay required. --confirm-untracked-child-dead <job-id> also stays required where it applies — it names an exact job with no recorded child PID, which is evidence homeboy does not have.

Starts a loopback-only Homeboy daemon on the runner and opens an SSH tunnel to it. This is the preferred Lab execution path because later runner exec calls can use the daemon session instead of ad-hoc SSH command execution. The JSON payload uses command: "runner.connect" and reports connection state such as the runner ID, tunnel endpoint, daemon endpoint, and persisted session metadata.

Set HOMEBOY_CONTROLLER_PROXY to an explicit http, https, socks5, or socks5h controller-local proxy URL to expose it to a direct SSH runner. Connect then owns a runner-loopback reverse forward and injects its credential-free loopback URL as HTTP_PROXY, HTTPS_PROXY, and ALL_PROXY for daemon jobs that do not already set those values. Session status records the forward PID and URL; rotation preserves it and disconnect tears it down. With the variable unset, runner connections and job environments are unchanged.

Reconnect distinguishes a healthy endpoint from a lost local tunnel, a live exact-owner PID whose recorded listener no longer answers, a dead PID with a retained lease, and an endpoint whose lease/PID identity differs from the controller record. Only the exact lease, PID, loopback address, configured build, and authoritatively zero durable jobs permit automatic unhealthy-listener replacement. The lease-bound stop rechecks the startup ownership token and job store before every signal. Active work, PID/lease drift, and ambiguous listener ownership remain fail-closed.

When a controller session was lost while a remote daemon lease is dead and its durable jobs remain, inspect the remote homeboy daemon status first. Recovery requires the explicit exact-lease form shown above. It verifies the recorded PID is dead itself — that is not an operator assertion. If a listed active job has neither a terminal result nor a recorded child PID, inspect its process identity independently and supply one repeated --confirm-untracked-child-dead <job-id> value for every such exact job. The confirmation is rejected for terminal, PID-backed, foreign-lease, or unknown jobs; it records an operator-confirmed interruption in each reconciled job’s durable events. It probes the supplied concrete loopback endpoint and requires it to be unreachable, then acquires the remote daemon lifecycle lock before preserving job events and terminalizing jobs from that dead control plane. The response retains the exact lease, affected job IDs, endpoint probe, owner-lock proof, snapshot, and replacement identity before the fresh local session is recorded. Exact, lease-less, and orphan-adoption recovery modes are mutually exclusive. Ordinary runner connect never infers orphan ownership; live, ambiguous, and lease-mismatched daemon records remain protected.

Exact state-loss recovery writes a lease-keyed receipt before terminalizing jobs. If replacement startup fails, the receipt retains the reconciled evidence and the same exact lease, PID, and endpoint inputs resume only replacement startup on retry. Different replay inputs fail closed, and a zero-active-job invocation without a matching receipt remains invalid.

Every daemon returned by a replacement operation is journaled before tunnel health and endpoint identity checks. A retry authenticates and publishes those exact coordinates. If the recorded PID has died, a newer lease has superseded it, or the exact idle replacement listener is unhealthy, the stale journal is retired under the runner lock so it cannot pin a dead endpoint.

When daemon status reports stale_reason_code: lease_missing, active jobs, and unavailable recovery evidence, run the explicit daemon reconciliation on the configured runner host. It acquires the daemon lifecycle lock, requires the affirmative confirmation flag, independently checks for a homeboy daemon serve process and Homeboy TCP listener, snapshots jobs.json, retains all job events and artifact references, terminalizes every active job as control-plane-lost, and returns each original lease ID plus the affected job mapping before starting one replacement daemon. Historical lease IDs do not establish a current owner. Live or ambiguous probes abort without changing the store.

Reverse runner connections record the runner-initiated session substrate and use the controller daemon as the broker. A reverse runner can register itself with POST /runner/sessions; the controller then reports that runner as connected and routes runner exec through brokered jobs instead of a direct daemon URL. The broker exposes POST /runner/jobs, POST /runner/jobs/claim, POST /runner/jobs/<job-id>/events, and POST /runner/jobs/<job-id>/finish so controllers can queue work and reverse runners can claim, stream progress, and return results without inbound access to the lab machine.

Use Homeboy wrappers for broker maintenance and artifact lookup instead of raw HTTP calls:

sh
homeboy runner job reconcile <runner-id>
homeboy runner job artifacts <runner-id> <job-id> <artifact-id>

Broker authentication and pairing

Every /runner/* broker route requires an authenticated, scoped bearer token (#2990). The trust model is:

  • Pairing. On the broker host, mint a credential with homeboy runner broker pair <id> --runner-id <runner-id> [--work] [--submit]. Pairing prints a one-time bearer token and stores only its SHA-256 hash in ~/.config/homeboy/broker_auth.json (0600). The plaintext token is shown once and never re-displayed or logged.
  • Scopes. --work authorizes worker routes (register, claim, events, finish, heartbeat); --submit authorizes controller job submission and cancellation (POST /runner/jobs, POST /runner/jobs/<id>/cancel). A credential may carry both. When neither flag is given, --work is assumed.
  • Runner-id binding. Worker routes carry a runner_id; the presented token must belong to a credential paired to that exact runner_id, so a paired runner can never claim, progress, or finish another runner’s jobs.
  • Revocation. homeboy runner broker revoke <id> disables a credential. homeboy runner broker list shows non-secret credential metadata (never tokens or hashes).
  • Secure by default. With no credentials configured, the broker rejects all /runner/* traffic with a structured broker.auth_denied (401) error. To keep an existing loopback-only smoke setup working without auth, set "allow_unauthenticated_loopback": true in broker_auth.json; this is only honored when the broker is bound to loopback.

Workers send the token via --broker-token <token> or the HOMEBOY_BROKER_TOKEN environment variable; controllers read their submit token from HOMEBOY_BROKER_TOKEN. Tokens travel as both X-Homeboy-Broker-Token and Authorization: Bearer <token> so they survive proxies that strip one form. Unauthenticated, mismatched-runner, or wrong-scope requests are rejected with a structured broker.auth_denied error and never expose the stored secret.

Daemon and broker HTTP responses use one canonical envelope. The outer response reports transport success and the endpoint payload always lives under data.body; runner clients require that shape and do not parse legacy direct data payloads.

lifecycle

Runner lifecycle output is transport-neutral even when the underlying route is a direct daemon tunnel or a reverse broker. JSON consumers should prefer these generic contracts:

  • RunnerSession: persisted connection/session ownership, including mode, role, controller/broker references, process IDs, version identity, and heartbeat.
  • RunnerJob: active or inspected job state with lifecycle owner, claim lease, command, status, durable run linkage, and artifact refs.
  • RunnerWorkspaceLease: controller-owned local-to-runner workspace materialization with source ref/commit/dirty metadata.
  • RunnerWorkspaceLeaseSet: named workspace leases for jobs that materialize more than one tree. primary names the default execution workspace; entries in leases bind names such as source, rig, or runtime to their own RunnerWorkspaceLease. Existing single-workspace output keeps workspace_lease while migrations add workspace_leases.
  • RunnerResult: terminal command result with exit status, stream sizes, mirrored run linkage, and artifact refs.
  • RunnerArtifactRef: stable artifact identity plus path/URL/hash/transport metadata for later retrieval.
  • RunnerMutationArtifacts: transport-independent mutation return shape. When a runner command captures source changes, mutation_artifacts.patch_ref points at the patch artifact; future file-bundle and operation-log returns use file_bundle_ref and operation_log_ref without changing caller logic.
  • LabRunnerHandoff: the controller-to-runner handoff envelope tying transport, owner, job, workspace lease, and result together.

Compatibility fields such as active_jobs, job, job_id, job_events, and workspace_lease remain present for existing callers. New integrations should read active_runner_jobs, runner_job, workspace_leases, runner_result, mutation_artifacts, artifact_ref, and handoff so direct SSH and reverse broker implementations stay hidden behind the same lifecycle vocabulary.

Connected daemon status also reports stale runtime-path signals. When the daemon started with configured HOMEBOY_*_COMPONENT_PATH, HOMEBOY_*_PLUGIN_PATH, HOMEBOY_*_PROVIDER_PATH, or HOMEBOY_*_RUNTIME_PATH values, /version captures start-time fingerprints and compares them with the current on-disk paths. homeboy runner status <runner-id> surfaces those differences in stale_daemon.stale_runtime_paths, and surfaces runner config path changes in stale_daemon.changed_runtime_paths. This is intentionally read-only: refresh a development runner with homeboy runner disconnect <runner-id> followed by homeboy runner connect <runner-id> after rebuilding runner-side runtime code.

Daemon compatibility has three states, not two

stale_daemon.verification names which question the report answered:

  • compared — the runner’s configured identity was read and compared. The version and build-identity fields are observations, and a mismatch fences admission.
  • probe_failed — the identity probe itself failed. Nothing was compared; stale_daemon.probe_error carries the verbatim failure and current_homeboy_version is the sentinel unverified rather than a substituted value. A runner that has a probe and could not answer it is still fenced.
  • unavailable — this runner has no controller-side identity probe at all. Reverse-connected sessions are reached through the broker, which exposes no identity endpoint, so their compatibility is reported as compatibility_reason: reverse_unverified. This is surfaced and ranked below every verified runner in default Lab selection, but it does not fence admission — fencing an entire runner class on the strength of a gap rather than a mismatch would take every reverse lab out of service.

Reports with either unverified verdict carry severity: unknown and a read-only homeboy runner doctor <runner-id> --scope lab-offload recovery: replacing a binary is not the answer to "I could not look at it".

Controller identity and a dirty working tree

Runner convergence is normally proven by comparing the controller’s embedded build commit against the runner’s configured job command binary. A controller built from a dirty working tree records the last commit, not the tree it was actually built from, so that commit names no reproducible build and can equal no runner’s — which is "the comparison is unavailable", not "this runner is stale". Homeboy reports the two states separately:

  • The controller↔runner version check still applies in full. A runner whose configured binary is a different Homeboy version is still refused, with stale_daemon.compatibility_reason = "controller_configured_version_skew" and a runner-side homeboy runner refresh-homeboy recovery.
  • The exact-commit check is skipped while the controller cannot name its own build, so a dirty controller no longer marks every runner stale. The runner-internal check (its live daemon against its own configured binary) is untouched and still blocking.
  • stale_daemon.compatibility_reason = "controller_identity_unverifiable" reports an unavailable comparison. It carries no recovery command, because the remediation is to commit or stash the controller working tree and rerun homeboy runner status <runner-id> — not to reinstall the controller.
json
{
  "success": true,
  "data": {
    "status": 200,
    "endpoint": "runner.jobs.submit",
    "body": {
      "job": {},
      "poll": {}
    }
  }
}

For the generic controller-to-runner operator path, see Controller to runner reverse-runner setup. That guide is machine-agnostic and intentionally explicit about what is available now and what remains gated by #2990, #2991, #2992, and #2947 before production broker exposure.

work

sh
homeboy runner work <runner-id> --broker-url <url> --broker-token <token>
homeboy runner work <runner-id> --broker-url <url> --project <project-id> --lease-ms 30000
HOMEBOY_BROKER_TOKEN=<token> homeboy runner work <runner-id> --broker-url <url> --loop

Pass the paired token with --broker-token or the HOMEBOY_BROKER_TOKEN environment variable. The token is required whenever the broker enforces auth; omit it only for a loopback-open smoke broker.

Claims one brokered reverse-runner job for the runner, executes it on the runner machine under the runner’s local policy, streams a progress event, and finishes the broker job with stdout, stderr, and exit code. This is the runner-side half of reverse runner exec; it uses outbound HTTP from the lab to the controller broker and does not require inbound SSH or a public listening port on the lab.

The command exits 0 when no job is available, with claimed: false in the JSON payload. When a job is claimed, the process exit code matches the executed command’s exit code.

Use --loop for a long-running reverse runner service. Loop mode emits one structured JSON status line per lifecycle event to stderr so systemd/journald can index startup, idle backoff, job completion, transient broker failures, and shutdown without mixing those events into the final stdout JSON payload. Empty queues use exponential backoff controlled by --idle-backoff-ms and --max-idle-backoff-ms, so workers do not hot-spin when no work is available. Transient broker failures sleep for --broker-failure-backoff-ms and exit non-zero after --broker-retry-limit consecutive failures. SIGINT and SIGTERM request graceful shutdown after the current claim attempt or job.

broker

sh
homeboy runner broker pair lab-1 --runner-id homeboy-lab --work
homeboy runner broker pair ctrl-1 --runner-id homeboy-lab --submit
homeboy runner broker revoke lab-1
homeboy runner broker list

Manage reverse runner broker authentication and pairing on the broker host. pair mints a one-time scoped bearer token (printed once; only its SHA-256 hash is stored under ~/.config/homeboy/broker_auth.json). revoke disables a credential by id, and list reports non-secret credential metadata. See Broker authentication and pairing for the full trust model.

job

sh
homeboy runner job logs <runner-id> <job-id>
homeboy runner job logs <runner-id> <job-id> --follow --poll-ms 1000
homeboy runner job logs <runner-id> <job-id> --follow --cursor 65028
homeboy runner job cancel <runner-id> <job-id>

Inspects or cancels durable runner daemon jobs after runner exec or Lab offload has submitted work to a connected runner. logs fetches the persisted job plus its event stream; --follow keeps polling until the job reaches a terminal state and prints newly observed events as they arrive. Use this when a controller exits after dispatching runner work and you need to inspect the already-started job.

cancel requests cancellation for a queued or running durable runner daemon job through the connected runner daemon.

Follow tracks the highest displayed daemon event sequence. A controller tunnel loss opens a new ephemeral SSH tunnel to the generation recorded as the job owner, and verifies that generation’s lease and PID through /health before reading. It never reuses or probes the recorded local tunnel port. Replayed events are sorted and emitted only when their sequence exceeds the cursor. The final JSON output includes next_cursor and resume_command; Ctrl-C prints the same continuation command without cancelling remote work. A job absent after a successful authoritative recovery is reported as retention/eviction rather than a transient connection failure. cancel uses the same owner recovery path when its current controller tunnel has disappeared.

Minimal Homeboy Lab systemd unit:

ini
[Unit]
Description=Homeboy reverse runner worker
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=deploy
WorkingDirectory=/home/user
Environment=HOMEBOY_BROKER_TOKEN=<paired-token>
ExecStart=/usr/local/bin/homeboy runner work homeboy-lab --broker-url https://controller.example.com --loop --idle-backoff-ms 1000 --max-idle-backoff-ms 30000 --broker-retry-limit 12
Restart=on-failure
RestartSec=10
KillSignal=SIGTERM

[Install]
WantedBy=multi-user.target

The worker authenticates with the broker using the paired bearer token. Prefer delivering it through HOMEBOY_BROKER_TOKEN — via a systemd EnvironmentFile=/etc/homeboy/broker.env with 0600 permissions rather than an inline Environment= line — so the secret never lands in systemctl show output. Pair the credential on the broker host with homeboy runner broker pair (see Broker authentication and pairing).

status

sh
homeboy runner status <runner-id>

Shows the persisted tunnel/session state for a runner. Use this to determine whether runner exec will use a connected daemon or needs an explicit fallback. The JSON payload uses command: "runner.status" and reports whether a saved session exists, whether the tunnel still appears live, and the recorded endpoint details.

Status output also carries the former Lab readiness handoff. extra includes preferred_lab_runner, selected_lab_runner.runner_homeboy, and managed_followups with copyable commands for recent run discovery, evidence inspection, runner doctor --scope lab-offload, runner environment inspection, daemon refresh, binary upgrade, managed runner exec, and workspace sync. Use this output before assuming a runner has picked up a merged Homeboy CLI fix.

Runner-side extension updates use the generic runner/extension primitives instead of a Lab-specific sync command:

sh
homeboy runner exec <runner-id> -- homeboy extension install <source> --id <extension-id> --ref <ref> --replace
homeboy runner exec <runner-id> -- homeboy extension refresh <source> --id <extension-id> --ref <ref>

Status output is also the operator handoff for active runner jobs. When the connected daemon or reverse broker can report active work, each active job includes lifecycle timing fields such as updated_at_ms, heartbeat_age_ms, claim_id, claimed_by_runner_id, claimed_at_ms, claim_expires_at_ms, and claim_expires_in_ms. The top-level operator_commands array gives copyable next actions for the current session, including job log following, cancellation, broker claim reconciliation for reverse runners, and artifact lookup/fetch commands when the status payload has enough context.

reconcile

sh
homeboy runner reconcile <runner-id>

Reconciles persisted direct-runner generation state and retires daemon generations that have been verified as drained. Use the remediation command reported by homeboy runner status after it has established that no active work still belongs to the generation.

disconnect

sh
homeboy runner disconnect <runner-id>

Closes a persisted runner tunnel session and removes its local session state. The JSON payload uses command: "runner.disconnect" and reports which session state was removed. This is safe to run when no live session exists; it is the explicit cleanup counterpart to runner connect.

list

sh
homeboy runner list

show

sh
homeboy runner show <id>

set

sh
homeboy runner set <id> --json <JSON>
homeboy runner set <id> --base64 <BASE64_JSON>
homeboy runner set <id> --json '{"workspace_root":"/srv/homeboy","concurrency_limit":4}'

Updates a runner by merging a JSON object into the runner config. SSH runner settings live under servers/<id>.json as the server’s runner capability; local runners live under runners/<id>.json. Arbitrary runner updates must use --json or --base64; positional key=value and trailing arbitrary --key value updates are not accepted.

trust

sh
homeboy runner trust <runner-id> --project <project-id> --command test --command bench --allow-raw-exec false
homeboy runner trust <runner-id> --workspace-root <runner-workspace-root> --artifact-policy metadata
homeboy runner trust <runner-id> --peer <controller-server-id> --fingerprint SHA256:...

Persists controller-side trust policy for a runner. Policy is stored in the runner config as policy, not in transient CLI state. Repeated values are appended without duplicates.

Policy fields:

  • --project <id> allows a project to use the runner.
  • --command <family> allows a command family such as test, bench, lint, audit, trace, cargo, or runner.exec.
  • --allow-raw-exec <true|false> controls arbitrary runner exec shell access. SSH runner raw exec is denied by default until this is explicitly true.
  • --workspace-root <path> limits execution to one or more approved runner workspace roots.
  • --artifact-policy <label> records artifact behavior; none and deny block patch capture.
  • --peer <id> records accepted peer/controller server IDs for reverse-runner pairing.
  • --fingerprint <value> records expected peer host keys or equivalent fingerprints.

pair

sh
homeboy runner pair <runner-id> --peer <controller-server-id> --accept-project <project-id> --workspace-root <runner-workspace-root>
homeboy runner pair <runner-id> --fingerprint SHA256:... --allow-raw-exec false

Persists runner-side pairing policy for trusted controllers. pair writes the same durable policy object as trust, using runner-side option names for accepted peer IDs, accepted project IDs, peer fingerprints, workspace roots, and raw exec policy.

remove

sh
homeboy runner remove <id>

exec

sh
homeboy runner exec [HOMEBOY_OPTIONS] <runner-id> -- <command...>
homeboy runner exec --project <project-id> --cwd /runner/workspace/project <runner-id> -- <command...>
homeboy runner exec --sync-workspace /local/project@patch --require-path /runner/resources/input <runner-id> -- <command...>
homeboy runner exec --ssh --cwd /runner/workspace/project <runner-id> -- <command...>
homeboy runner exec --run-id ssi-fixture-matrix-summary <runner-id> -- <command...>
homeboy runner exec --cwd /runner/workspace/project --require-path /runner/workspace/project <runner-id> -- <command...>
homeboy runner env <runner-id>

exec submits the command to the connected runner daemon when homeboy runner connect <runner-id> has established a live loopback tunnel. If no daemon session is connected, local runners execute directly and SSH runners require explicit diagnostic --ssh. SSH runner raw exec is policy-denied by default until policy.allow_raw_exec is explicitly true.

recipe-run

sh
homeboy runner recipe-run <runner-id> --provider <provider-id> --sync-workspace <local-workspace> --recipe <relative-recipe> --artifacts <relative-artifact-directory> --run-id <durable-run-id>

recipe-run snapshots its declared workspace once, resolves the recipe and artifact directory beneath that materialized root, and dispatches an extension-owned provider. The provider descriptor declares its executable and version; Homeboy preflights the executable through the normal runner execution path and records the provider, resolved argv, source snapshot, terminal JSON result, and promoted artifacts on the durable run. Recipe and artifact paths must be non-empty relative paths without parent traversal. runner exec remains available for low-level invocation.

The Homeboy contract is the installed extension manifest’s recipe_run_providers array. Each descriptor has id, version, executable, and a command argv array. {recipe} and {artifacts} tokens expand to the validated workspace-relative paths without shell parsing. Homeboy discovers these descriptors from installed extensions at execution time; product extensions can therefore supply providers without being linked into the Homeboy binary.

Inspect the installed provider inventory before dispatching:

sh
homeboy runner recipe-providers

The JSON output lists each declaration’s provider ID and version, owning extension, installed source reference, executable, structural resolvable state, and validation state. resolvable does not probe executable readiness on a selected runner; execution uses the normal runner preflight path. Invalid and duplicate declarations remain visible but unresolvable; recipe-run fails closed for either. Unknown provider errors include this exact inventory command and a bounded list of available provider IDs.

--script-file <path> reads the controller-side source, then materializes it as a private, content-addressed script-<sha256>.sh under ${XDG_RUNTIME_DIR:-/tmp}/homeboy-runner-exec/<job-id>/ on the runner. Bash executes that file directly, so $0 is its runner path. The job exports HOMEBOY_RUNNER_EXEC_SCRIPT with the same path and HOMEBOY_RUNNER_EXEC_SCRIPT_SHA256 as sha256:<digest>; the wrapper records the digest in its durable command evidence, makes the file mode 0500, and removes it when the runner job exits.

--script-file - reads stdin verbatim on the controller with the same bounded capture and materialization behavior; it does not stream stdin through bash -s. Zero-byte stdin is a validation error before Homeboy builds or submits a runner execution plan. Whitespace-only stdin is valid and is materialized verbatim, including newlines.

Place Homeboy options before the runner ID and use -- before the remote command. Without the separator, Homeboy diagnoses known exec options that appear after a remote command; the separator preserves remote flags with names such as --cwd, --raw, and --run-id.

Path rules:

  • SSH runners require workspace_root so local paths are not silently reused remotely.
  • SSH --cwd must be an absolute path under the configured workspace_root.
  • Omitting --cwd on an SSH runner uses the runner workspace_root.
  • --sync-workspace <local-worktree> snapshots a controller-side worktree first, including dirty edits allowed by the normal snapshot safety excludes, then executes from the returned runner remote_path. It is mutually exclusive with --cwd because Homeboy owns the execution cwd for that one command.
  • --require-path <path> preflights one or more runner-side paths before execution. Use it when a command references a lab worktree path so missing controller-only paths fail with a structured require_path error instead of an empty command failure.
  • --project <id> feeds the runner trust policy project allowlist check.
  • --run-id <id> sets the persisted controller-side runner-exec run id for ad hoc evidence commands. When omitted, Homeboy derives a run id prefix from the command domain, such as trace-matrix-summary.
  • --ssh is the explicit diagnostic fallback when connect is unavailable; daemon execution is preferred because it records job metadata and supports artifact-oriented workflows.
  • Diagnostic SSH output serializes as mode: "diagnostic_ssh" and does not include job/event evidence.
  • Raw SSH execution remains intentionally explicit and should not be used as production Lab/offload evidence; use connected daemon or reverse broker execution for job/event/artifact-compatible output.
  • Mutation-capable executions expose returned source changes through mutation_artifacts.patch_ref when a patch artifact is available. Direct daemon and reverse broker results use the same field; callers should not branch on transport mode to find returned mutations.

Dirty local worktree against runner-side resources:

sh
homeboy runner exec --sync-workspace /local/project@patch 
  --require-path /runner/resources/input 
  --capture-patch 
  <runner-id> 
  -- ./run-workload --input /runner/resources/input

This is the generic one-command form for “run my local patch on the runner”: Homeboy snapshots the local source, runs the command from the materialized runner copy, preflights any runner-owned resource paths the caller declares, and attaches source snapshot metadata to patch capture output. The resource path meaning remains caller-owned; Homeboy only checks that the path exists on the runner.

Runner job environment:

  • homeboy runner env <runner-id> shows configured public runner env plus secret_env keys/references for runner jobs. It does not resolve or print secret values.
  • Public env values are redacted because legacy configs may still contain tokens; secret_env remains references-only.
  • homeboy ssh <server> -- printenv NAME inspects the server login shell environment. It does not include runner job env unless the variable is also configured on the server shell.
  • Use homeboy runner exec <runner-id> -- printenv NAME for final execution-time proof when debugging resolved runner job environment.

Runner metrics:

  • Local runner execution, connected daemon jobs, and reverse-runner worker results include a metrics object with duration_ms, sample_count, and lightweight resource fields when available.
  • On Linux runners, metrics are sampled from /proc for the command process tree and include peak_rss_bytes, child_process_count_peak, cpu_user_ms, and cpu_system_ms.
  • CPU accounting is sampled and can miss very short-lived child processes between samples; duration is always recorded, and non-Linux runners report source: "duration_only".

workspace list

sh
homeboy runner workspace list <runner-id>
homeboy runner workspace list <runner-id> --limit 5

workspace list shows recent runner-side Lab workspace directories under the runner’s configured workspace_root and includes a reusable runner exec --cwd <remote-path> command for each entry. Use it after a manual or matrix session when the previous remote_path scrolled out of view and you need to run another command against the same runner-side checkout.

workspace sync

sh
homeboy runner workspace sync <runner-id> --path <local-worktree>
homeboy runner workspace sync <runner-id> --path <local-worktree> --mode snapshot
homeboy runner workspace sync <runner-id> --path <local-worktree> --mode git

workspace sync materializes a controller-side worktree under the runner’s configured workspace_root so runner execution can run against an explicit remote path while Git operations and canonical edits stay local.

Modes:

  • snapshot copies the current local tree, including dirty edits, through a tar stream from the controller. Use this for private or proxy-dependent sources because the runner does not need repository access.
  • git requires a clean local tree, then clones or refreshes remote.origin.url on the runner and checks out local HEAD detached. Use this only when the runner is allowed to fetch the remote directly.

Private/proxied sources:

  • Private or proxy-dependent source access stays on the controller machine.
  • Materialize those sources with homeboy runner workspace sync <runner-id> --path <local-worktree> --mode snapshot.
  • Use the returned remote_path for downstream runner exec --cwd or job inputs.
  • Runner-side Git fetches for configured private/proxied hosts are refused with an actionable diagnostic. The default host list includes github.example.com; override with HOMEBOY_PRIVATE_PROXIED_SOURCE_HOSTS only when a runner is explicitly allowed to fetch those sources.

Safety rules:

  • The remote path is deterministic and lives under <workspace_root>/_lab_workspaces/.
  • Snapshot sync excludes dependency directories, build outputs, caches, .git, and common secret file patterns such as .env*, *.pem, and *.key.
  • Runner policy can add project-specific generated-state patterns with snapshot_excludes; configured patterns are merged with the default snapshot safety excludes and affect snapshot hashing, stats, and materialization.
  • Git sync refuses to overwrite an existing dirty runner-side checkout by default. Use --allow-dirty-lab-workspace only for noisy investigation where discarding runner-side changes is intentional; Lab metadata records that override.
  • Output includes local_path, remote_path, sync_mode, snapshot_identity, and snapshot files / bytes when available.
  • The runner workspace is execution-only; this command does not push branches, commit, or make the runner authoritative for source changes.

workspace prune

sh
homeboy runner workspace prune <runner-id> --min-age-hours 24 --limit 5
homeboy runner workspace prune <runner-id> --apply --min-age-hours 24 --limit 5 --passes 10
homeboy runner workspace prune <runner-id> --cursor <opaque-cursor>

workspace prune only removes metadata-backed, inactive Lab workspaces that are old enough, have no pending patch or diff artifact, and are orphaned or have an expired lifecycle TTL. It never removes outside _lab_workspaces.

--min-age-hours defaults to the shared runner age floor (24 hours) that homeboy cleanup --include remote-lab-workspaces and homeboy runner cache-prune also apply; it is one named constant rather than a literal repeated per command. --limit defaults to the shared page size (25) and bounds the workspace directories inspected, sized, retained in memory, and removed in each pass; it is a page size, not a delete budget, which is why the aggregate --limit record budget is deliberately not wired to it. --passes bounds apply passes. Output now includes scanned_workspace_count and scan_complete: when scan_complete is false, has_more is true, continuation_cursor is an opaque token that resumes strictly after the last inspected workspace, and total_candidate_* plus remaining_candidate_* describe only the inspected window. Use the emitted next_command or drain_command unchanged; both carry the cursor when needed.

cache-prune

sh
homeboy runner cache-prune <runner-id>
homeboy runner cache-prune <runner-id> --apply --min-age-hours 24

Inventories stale managed Homeboy binary slots on a runner. --apply removes only unselected slots that meet the shared minimum-age floor; omit it to inspect the eligible slots without changing the runner.

workspace apply

sh
homeboy runner workspace apply <runner-apply.json>
homeboy runner workspace apply <runner-apply.json> --force

workspace apply brings a runner-generated fix artifact back to the local source worktree recorded in the artifact’s source_snapshot.local_path. It is local-only: it does not commit, push, or make the runner canonical. Reviewability stays in normal local Git via git status and git diff.

Safety rules:

  • The artifact must identify the local source worktree through source_snapshot.local_path.
  • Homeboy recalculates the current local source_snapshot.snapshot_hash before applying.
  • If the local source worktree drifted since the Lab snapshot, apply is refused unless --force is explicit.
  • Unified diffs are checked with git apply --check before mutation, so conflicts do not partially apply.
  • Delta paths must be relative and stay inside the source worktree.
  • Output includes apply_status, modified_files, expected_snapshot_hash, and current_snapshot_hash.

Temporary Wave 4 adapter contract, until the runner fix-capture contract settles:

json
{
  "source_snapshot": {
    "runner_id": "lab-a",
    "local_path": "/path/to/project@branch",
    "remote_path": "/srv/homeboy/_lab_workspaces/project-abc123",
    "git_sha": "...",
    "dirty": false,
    "sync_mode": "snapshot",
    "snapshot_hash": "sha256:...",
    "synced_at": "2026-05-16T00:00:00Z",
    "sync_excludes": [".git/", "node_modules/"]
  },
  "patch": {
    "format": "unified_diff",
    "content": "diff --git a/file.txt b/file.txtn..."
  }
}

Delta form is also accepted for explicit file replacement/deletion:

json
{
  "source_snapshot": { "...": "..." },
  "delta": {
    "files": [
      { "path": "src/file.txt", "content_base64": "Li4u" },
      { "path": "obsolete.txt", "delete": true }
    ]
  }
}

Runner Shape

SSH runner records are stored on their server as runner capability config under ~/.config/homeboy/servers/<id>.json.

json
{
  "id": "runner-a",
  "host": "runner.example.internal",
  "user": "runner",
  "port": 22,
  "runner": {
    "workspace_root": "/srv/homeboy/workspaces",
    "homeboy_path": "/usr/local/bin/homeboy",
    "daemon": false,
    "concurrency_limit": 4,
    "artifact_policy": "copy",
    "env": {},
    "resources": {}
  }
}

Standalone local runner records are still stored under ~/.config/homeboy/runners/.

json
{
  "id": "lab-local",
  "kind": "local",
  "server_id": null,
  "workspace_root": "/srv/homeboy/workspaces",
  "homeboy_path": "/usr/local/bin/homeboy",
  "daemon": false,
  "concurrency_limit": 2,
  "artifact_policy": "copy",
  "env": {},
  "resources": {}
}

Rules:

  • kind is local or ssh.
  • ssh runner IDs are server IDs; a single SSH machine does not need a separate runner ID.
  • concurrency_limit, when set, must be greater than zero.
  • env and resources are metadata maps for future connect, doctor, exec, and Desktop workflows.

JSON Output

All command output is wrapped in the global JSON envelope described in the JSON output contract. The data payload uses the generic entity CRUD shape:

  • command: action identifier such as runner.add, runner.enable, runner.list, runner.show, runner.set, runner.remove, runner.doctor, runner.connect, runner.status, runner.disconnect, runner.exec, runner.workspace.sync, or runner.workspace.apply
  • id: present for single-runner actions
  • entity: runner configuration for single-runner read/write actions
  • entities: list for list
  • updated_fields: list of updated field names for writes
  • deleted: list of removed runner IDs

Bounded Read-Only Probes

runner status, runs list --include-active-runner-jobs, and agent-task status are read-only inspection commands. Every remote probe they issue is bounded by a wall-clock deadline (default 15s per probe, overridable with HOMEBOY_READONLY_PROBE_TIMEOUT_SECONDS). When a probe hits its bound the SSH child process group is terminated and the command still returns, with the reason attached:

json
"probe_degradations": [
  {
    "probe": "runner_homeboy_identity",
    "runner_id": "homeboy-lab",
    "reason_code": "readonly_probe.timeout",
    "timeout_seconds": 15,
    "detail": "read-only probe `runner_homeboy_identity` exceeded its 15s bound ..."
  }
]

runner status --full also surfaces each degradation as a PARTIAL STATUS (...) entry in operator_hints. A non-empty probe_degradations list means the answer is partial — the runner did not respond in time — which is how an operator distinguishes "the Lab is wedged" from "there is nothing to report".

The same ledger carries readonly_probe.unavailable, which is not a remote timeout: a probe that reads the controller’s own ambient environment could not run there at all. Its timeout_seconds is 0 and its hint is labelled DEGRADED PROBE (...) so it cannot be mistaken for a wedged runner. The controller-side git ancestry probe (controller_git_ancestry) reports this way when runner status --full runs outside a Homeboy checkout: recovery guidance falls back to the controller’s own build ref, git’s fatal: not a git repository is captured into detail instead of being printed in front of the result envelope, and the runner answer itself is unaffected (#10525).

runs list --include-active-runner-jobs deliberately reads the latency-bounded indexed active-job snapshot rather than the full status report: it does not reconcile the daemon generation ledger, so a wedged Lab cannot block a listing.