Controller to runner reverse-runner setup
This guide is the operator runbook for using any Homeboy-managed controller
machine as a broker while any configured runner machine claims work by making an
outbound connection to that controller. A controller can be a VPS, a workstation,
or another server; a runner can be a lab box, build host, desktop, or any machine
with a Homeboy runner record. Extra Chill VPS to homeboy-lab is only the first
concrete deployment profile for this path. The broker, worker, pairing,
and workspace sync model is reusable. It tracks the end-to-end shape from issue
#2993 and the parent epic #2950.
Status
The current main branch contains the reverse-runner broker/job substrate and controller-side runner policy commands, but the production setup is gated by follow-up work:
| Area | Status | Gate |
|---|---|---|
| Broker authentication and pairing | Not production-safe yet | #2990 |
| Long-running runner worker service | Not available as a durable service yet | #2991 |
| Controller broker service deployment | Not available as a packaged service recipe yet | #2992 |
| Reverse workspace sync | Not available for controller-to-runner workspaces without direct SSH | #2947 |
Until those issues land, use this guide as scaffolding for local/private smoke tests and operator preparation. Do not expose the broker publicly or rely on this path for production hot-command offload.
Topology
Controller machine
homeboy daemon / reverse broker
runner trust policy
job submission
^
| HTTPS or private tunnel, authenticated after #2990
|
Runner machine
runner pairing policy
reverse worker service after #2991
workspace materialization after #2947The runner initiates all broker traffic. The controller does not need SSH access to the runner for the reverse path, and the runner should not be opened to the public internet.
Example profile: Extra Chill
The examples below use placeholders so the same runbook applies to any VPS. For the current Extra Chill deployment, use these values:
| Placeholder | Extra Chill value |
|---|---|
<project-id> | extrachill |
<controller-server-id> | extra-chill |
<controller-label> | extra-chill-vps |
<runner-id> | homeboy-lab |
<runner-workspace-root> | /home/chubes/Developer |
<runner-user> | chubes |
1. Controller broker setup
Run this on the controller machine.
Confirm Homeboy can see the project and controller server record:
shhomeboy project show <project-id> --output /tmp/homeboy-controller-project.json homeboy server show <controller-server-id> --output /tmp/homeboy-controller-server.jsonEnable a broker-capable daemon in a private or protected network context. The exact service wrapper and safe public binding are gated by #2992 and broker auth is gated by #2990. Until those land, keep the daemon loopback-only or behind a private tunnel:
shhomeboy daemon start --addr 127.0.0.1:0 --output /tmp/homeboy-reverse-broker.jsonRecord the broker URL label operators will use in evidence. For private smoke tests this can be a tunnel label rather than a public URL:
shexport HOMEBOY_RUNNER_BROKER_LABEL=<controller-label> export HOMEBOY_RUNNER_BROKER_URL=https://broker.example.invalidTrust the runner for only the needed project, command families, and workspace roots. This records policy on the controller side; secure token material and pairing enforcement are completed by #2990:
shhomeboy runner trust <runner-id> --peer <controller-server-id> --project <project-id> --command runner.exec --command audit --command lint --command test --command bench --command trace --workspace-root <runner-workspace-root> --allow-raw-exec false --artifact-policy metadataAfter #2992 lands, install the broker as a service and place it behind the documented TLS/private tunnel/auth boundary. The service should persist job store state, expose health/status, and make restart behavior explicit for active claims.
2. Runner pairing and auth
Confirm Homeboy can see the project and controller server record:
Confirm the runner record exists and has a confined workspace root:
shhomeboy runner show <runner-id> --output /tmp/homeboy-runner.json homeboy runner doctor <runner-id> --path <runner-workspace-root>/homeboy --extension rustPair the runner with the controller policy. This is the runner-side counterpart to
runner trust; #2990 supplies the secured token and enforcement model:shhomeboy runner pair <runner-id> --peer <controller-server-id> --accept-project <project-id> --workspace-root <runner-workspace-root> --allow-raw-exec falseStore any future broker tokens through the Homeboy auth/secrets path described by #2990. Do not put broker tokens in shell history, systemd unit files, or normal command output.
3. Runner worker service setup
Enable a broker-capable daemon in a private or protected network context. The exact service wrapper and safe public binding are gated by #2992 and broker auth is gated by #2990. Until those land, keep the daemon loopback-only or behind a private tunnel:
Record the broker URL label operators will use in evidence. For private smoke tests this can be a tunnel label rather than a public URL:
homeboy project show <project-id> --output /tmp/homeboy-controller-project.json
homeboy server show <controller-server-id> --output /tmp/homeboy-controller-server.jsonTrust the runner for only the needed project, command families, and workspace roots. This records policy on the controller side; secure token material and pairing enforcement are completed by #2990:
homeboy daemon start --addr 127.0.0.1:0 --output /tmp/homeboy-reverse-broker.jsonAfter #2992 lands, install the broker as a service and place it behind the documented TLS/private tunnel/auth boundary. The service should persist job store state, expose health/status, and make restart behavior explicit for active claims.
4. Reverse workspace sync
Confirm Homeboy can see the project and controller server record:
Confirm the runner record exists and has a confined workspace root:
git: the runner materializes a repo-backed clean commit/ref itself.snapshot-over-tunnel: the controller streams a filtered archive through the reverse session.patch-over-tunnel: the controller sends a clean base plus a diff for uncommitted work.
Pair the runner with the controller policy. This is the
runner-side counterpart to runner trust; #2990 supplies the secured token
and enforcement model:
5. Minimal end-to-end smoke
Store any future broker tokens through the Homeboy auth/secrets path described by #2990. Do not put broker tokens in shell history, systemd unit files, or normal command output.
Start or verify the controller broker service:
shhomeboy runner status <runner-id> --output /tmp/homeboy-runner-status.jsonStart or verify the runner worker service:
shsystemctl --user status homeboy-reverse-runner.service journalctl --user -u homeboy-reverse-runner.service -n 50 --no-pagerSubmit a minimal command from the controller:
shhomeboy runner exec <runner-id> --project <project-id> --cwd <runner-workspace-root> --output /tmp/homeboy-runner-smoke.json -- /bin/sh -lc 'printf "homeboy-runner-smoke\n"'Expected evidence shape:
json{ "command": "runner.exec", "runner_id": "<runner-id>", "mode": "reverse_broker", "transport": "reverse_broker", "broker_label": "<controller-label>", "job_id": "...", "exit_code": 0, "stdout_sample": "homeboy-runner-smoke" }
Enable a broker-capable daemon in a private or protected network context. The exact service wrapper and safe public binding are gated by #2992 and broker auth is gated by #2990. Until those land, keep the daemon loopback-only or behind a private tunnel:
Record the broker URL label operators will use in evidence. For private smoke tests this can be a tunnel label rather than a public URL:
Troubleshooting
| Symptom | Check | Likely fix |
|---|---|---|
| Auth failure | Broker returns a structured auth or forbidden error. | Re-pair the runner/controller after #2990, rotate token, and confirm runner ID/project policy match. |
| No jobs claimed | Worker logs show empty claims or no claim attempts. | Confirm broker URL, runner ID, worker service status, and that submitted jobs target the expected runner. |
| Stale claims | Job remains claimed with no finish event. | Restart the worker, inspect broker job status, and use the #2992 recovery command once available. |
| Worker offline | runner status or service status shows no active worker heartbeat. | Restart the runner worker service and inspect journald for broker/network/auth errors. |
| Workspace sync failure | Error names transport, materialization, policy, or command execution. | Use git mode for clean repo-backed work, or wait for #2947 for reverse snapshot/patch transport. |
| Policy denial | runner.exec reports project, command, raw exec, or workspace-root denial. | Update runner trust on the controller and runner pair on the runner with the narrow missing permission. |
Cleanup and restart
Trust the runner for only the needed project, command families, and workspace roots. This records policy on the controller side; secure token material and pairing enforcement are completed by #2990:
Stop the runner worker:
shsystemctl --user stop homeboy-reverse-runner.serviceDisconnect stale local runner session metadata when direct or reverse session metadata is wrong:
shhomeboy runner disconnect <runner-id>Restart the controller broker service after #2992 supplies the packaged unit:
shsystemctl restart homeboy-reverse-broker.serviceStart the runner worker again:
shsystemctl --user start homeboy-reverse-runner.serviceRe-run the minimal smoke and compare
job_id,runner_id,mode,exit_code, and stdout evidence before enabling automatic hot-command offload.