Controller to runner reverse-runner setup

This guide is the operator runbook for using any Homeboy-managed controller machine as a broker while any configured runner machine claims work by making an outbound connection to that controller. A controller can be a VPS, a workstation, or another server; a runner can be a lab box, build host, desktop, or any machine with a Homeboy runner record. Extra Chill VPS to homeboy-lab is only the first concrete deployment profile for this path. The broker, worker, pairing, and workspace sync model is reusable. It tracks the end-to-end shape from issue #2993 and the parent epic #2950.

Status

The current main branch contains the reverse-runner broker/job substrate and controller-side runner policy commands, but the production setup is gated by follow-up work:

AreaStatusGate
Broker authentication and pairingNot production-safe yet#2990
Long-running runner worker serviceNot available as a durable service yet#2991
Controller broker service deploymentNot available as a packaged service recipe yet#2992
Reverse workspace syncNot available for controller-to-runner workspaces without direct SSH#2947

Until those issues land, use this guide as scaffolding for local/private smoke tests and operator preparation. Do not expose the broker publicly or rely on this path for production hot-command offload.

Topology

text
Controller machine
  homeboy daemon / reverse broker
  runner trust policy
  job submission
        ^
        | HTTPS or private tunnel, authenticated after #2990
        |
Runner machine
  runner pairing policy
  reverse worker service after #2991
  workspace materialization after #2947

The runner initiates all broker traffic. The controller does not need SSH access to the runner for the reverse path, and the runner should not be opened to the public internet.

Example profile: Extra Chill

The examples below use placeholders so the same runbook applies to any VPS. For the current Extra Chill deployment, use these values:

PlaceholderExtra Chill value
<project-id>extrachill
<controller-server-id>extra-chill
<controller-label>extra-chill-vps
<runner-id>homeboy-lab
<runner-workspace-root>/home/chubes/Developer
<runner-user>chubes

1. Controller broker setup

Run this on the controller machine.

  1. Confirm Homeboy can see the project and controller server record:

    sh
    homeboy project show <project-id> --output /tmp/homeboy-controller-project.json
    homeboy server show <controller-server-id> --output /tmp/homeboy-controller-server.json
    
  2. Enable a broker-capable daemon in a private or protected network context. The exact service wrapper and safe public binding are gated by #2992 and broker auth is gated by #2990. Until those land, keep the daemon loopback-only or behind a private tunnel:

    sh
    homeboy daemon start --addr 127.0.0.1:0 --output /tmp/homeboy-reverse-broker.json
    
  3. Record the broker URL label operators will use in evidence. For private smoke tests this can be a tunnel label rather than a public URL:

    sh
    export HOMEBOY_RUNNER_BROKER_LABEL=<controller-label>
    export HOMEBOY_RUNNER_BROKER_URL=https://broker.example.invalid
    
  4. Trust the runner for only the needed project, command families, and workspace roots. This records policy on the controller side; secure token material and pairing enforcement are completed by #2990:

    sh
    homeboy runner trust <runner-id> 
      --peer <controller-server-id> 
      --project <project-id> 
      --command runner.exec 
      --command audit 
      --command lint 
      --command test 
      --command bench 
      --command trace 
      --workspace-root <runner-workspace-root> 
      --allow-raw-exec false 
      --artifact-policy metadata
    
  5. After #2992 lands, install the broker as a service and place it behind the documented TLS/private tunnel/auth boundary. The service should persist job store state, expose health/status, and make restart behavior explicit for active claims.

2. Runner pairing and auth

Confirm Homeboy can see the project and controller server record:

  1. Confirm the runner record exists and has a confined workspace root:

    sh
    homeboy runner show <runner-id> --output /tmp/homeboy-runner.json
    homeboy runner doctor <runner-id> --path <runner-workspace-root>/homeboy --extension rust
    
  2. Pair the runner with the controller policy. This is the runner-side counterpart to runner trust; #2990 supplies the secured token and enforcement model:

    sh
    homeboy runner pair <runner-id> 
      --peer <controller-server-id> 
      --accept-project <project-id> 
      --workspace-root <runner-workspace-root> 
      --allow-raw-exec false
    
  3. Store any future broker tokens through the Homeboy auth/secrets path described by #2990. Do not put broker tokens in shell history, systemd unit files, or normal command output.

3. Runner worker service setup

Enable a broker-capable daemon in a private or protected network context. The exact service wrapper and safe public binding are gated by #2992 and broker auth is gated by #2990. Until those land, keep the daemon loopback-only or behind a private tunnel:

Record the broker URL label operators will use in evidence. For private smoke tests this can be a tunnel label rather than a public URL:

sh
homeboy project show <project-id> --output /tmp/homeboy-controller-project.json
homeboy server show <controller-server-id> --output /tmp/homeboy-controller-server.json

Trust the runner for only the needed project, command families, and workspace roots. This records policy on the controller side; secure token material and pairing enforcement are completed by #2990:

sh
homeboy daemon start --addr 127.0.0.1:0 --output /tmp/homeboy-reverse-broker.json

After #2992 lands, install the broker as a service and place it behind the documented TLS/private tunnel/auth boundary. The service should persist job store state, expose health/status, and make restart behavior explicit for active claims.

4. Reverse workspace sync

Confirm Homeboy can see the project and controller server record:

Confirm the runner record exists and has a confined workspace root:

  • git: the runner materializes a repo-backed clean commit/ref itself.
  • snapshot-over-tunnel: the controller streams a filtered archive through the reverse session.
  • patch-over-tunnel: the controller sends a clean base plus a diff for uncommitted work.

Pair the runner with the controller policy. This is the runner-side counterpart to runner trust; #2990 supplies the secured token and enforcement model:

5. Minimal end-to-end smoke

Store any future broker tokens through the Homeboy auth/secrets path described by #2990. Do not put broker tokens in shell history, systemd unit files, or normal command output.

  1. Start or verify the controller broker service:

    sh
    homeboy runner status <runner-id> --output /tmp/homeboy-runner-status.json
    
  2. Start or verify the runner worker service:

    sh
    systemctl --user status homeboy-reverse-runner.service
    journalctl --user -u homeboy-reverse-runner.service -n 50 --no-pager
    
  3. Submit a minimal command from the controller:

    sh
    homeboy runner exec <runner-id> 
      --project <project-id> 
      --cwd <runner-workspace-root> 
      --output /tmp/homeboy-runner-smoke.json 
      -- /bin/sh -lc 'printf "homeboy-runner-smoke\n"'
    
  4. Expected evidence shape:

    json
    {
      "command": "runner.exec",
      "runner_id": "<runner-id>",
      "mode": "reverse_broker",
      "transport": "reverse_broker",
      "broker_label": "<controller-label>",
      "job_id": "...",
      "exit_code": 0,
      "stdout_sample": "homeboy-runner-smoke"
    }
    

Enable a broker-capable daemon in a private or protected network context. The exact service wrapper and safe public binding are gated by #2992 and broker auth is gated by #2990. Until those land, keep the daemon loopback-only or behind a private tunnel:

Record the broker URL label operators will use in evidence. For private smoke tests this can be a tunnel label rather than a public URL:

Troubleshooting

SymptomCheckLikely fix
Auth failureBroker returns a structured auth or forbidden error.Re-pair the runner/controller after #2990, rotate token, and confirm runner ID/project policy match.
No jobs claimedWorker logs show empty claims or no claim attempts.Confirm broker URL, runner ID, worker service status, and that submitted jobs target the expected runner.
Stale claimsJob remains claimed with no finish event.Restart the worker, inspect broker job status, and use the #2992 recovery command once available.
Worker offlinerunner status or service status shows no active worker heartbeat.Restart the runner worker service and inspect journald for broker/network/auth errors.
Workspace sync failureError names transport, materialization, policy, or command execution.Use git mode for clean repo-backed work, or wait for #2947 for reverse snapshot/patch transport.
Policy denialrunner.exec reports project, command, raw exec, or workspace-root denial.Update runner trust on the controller and runner pair on the runner with the narrow missing permission.

Cleanup and restart

Trust the runner for only the needed project, command families, and workspace roots. This records policy on the controller side; secure token material and pairing enforcement are completed by #2990:

  1. Stop the runner worker:

    sh
    systemctl --user stop homeboy-reverse-runner.service
    
  2. Disconnect stale local runner session metadata when direct or reverse session metadata is wrong:

    sh
    homeboy runner disconnect <runner-id>
    
  3. Restart the controller broker service after #2992 supplies the packaged unit:

    sh
    systemctl restart homeboy-reverse-broker.service
    
  4. Start the runner worker again:

    sh
    systemctl --user start homeboy-reverse-runner.service
    
  5. Re-run the minimal smoke and compare job_id, runner_id, mode, exit_code, and stdout evidence before enabling automatic hot-command offload.