# Running Hashiru on a server

How to run Hashiru on a dedicated Linux machine, from install to upgrades.
Everything here was rehearsed on Ubuntu 24.04 with Docker Engine: install, the
system service, load tests, a Docker restart and a reboot (see "Tested" below).

## 1. The machine

- **OS:** Linux (x86_64 or arm64) with systemd and Docker Engine.
  - Ubuntu: `apt install docker.io git curl`. Docker's own packages work too.
  - Docker Desktop isn't needed on Linux.
- **CPU and memory:** budget each runner's `cpus` and `memory`, plus about 2 GB for the system.
  - Example: 16 cores and 64 GB handles 4 runners at 4 CPUs and 12 GB each.
- **Disk:** 100 GB or more on an SSD. The space goes to Docker images (the runner image is about 3 GB), git mirrors, toolchain caches and the actions/cache store (`cache_server.max_gb`, 20 GB by default).
- **Network:** outbound HTTPS to github.com, api.github.com, `*.actions.githubusercontent.com` and ghcr.io. Nothing inbound is needed: runners pull jobs from GitHub.

## 2. Install

```sh
curl -fsSL https://hashiru.sh/install.sh | sudo HASHIRU_INSTALL_DIR=/usr/local/bin sh
```

## 3. Credentials

Pick one of these. Store secrets in a root-only file, not in the config:

| | Use when | Permissions |
|---|---|---|
| Fine-grained PAT | A few repos | Repository: Administration read/write, Contents read, Actions read, Metadata read |
| Classic PAT | Quick start | `repo` (repo runners) or `admin:org` (org runners) |
| GitHub App | Orgs, long-term | Repository: Administration read/write, Contents read, Actions read. Org: Self-hosted runners read/write (for org URLs) |

Tokens expire, so prefer an App for a machine you won't touch for months. App
tokens refresh themselves. A PAT that expires stops new runners from
registering.

```sh
sudo install -d -m 700 /etc/hashiru
sudo sh -c 'cat > /etc/hashiru/token'        # paste the token, then Ctrl-D
sudo chmod 600 /etc/hashiru/token
```

After rotating a token, run `sudo systemctl reload hashiru`. This re-reads the config and the token without touching running jobs.

## 4. Config

`/etc/hashiru/hashiru.yaml`:

```yaml
listen: 127.0.0.1:8484        # status page, /api/status, /metrics
cache_dir: /var/lib/hashiru   # git mirrors, action archives, actions/cache
max_runners: 4                # all pools together; size to the machine
min_free_gb: 20               # pause new runners when disk gets this low

github:
  token_command: cat /etc/hashiru/token
  # or:
  # app:
  #   client_id: Iv23li...
  #   installation_id: 12345678
  #   private_key_path: /etc/hashiru/app.pem

pools:
  - name: hashiru                 # runs-on: hashiru
    urls:
      - https://github.com/your-org/repo-a
      - https://github.com/your-org/repo-b
    min: 1                        # warm runners per repo
    max: 4                        # per repo; max_runners caps the total
    docker:
      image: hashiru/runner:latest  # built from the binary on first start
      cpus: "4"
      memory: 12g
      network: host
      docker_socket: true
```

Then run:

```sh
sudo hashiru doctor -c /etc/hashiru/hashiru.yaml    # every problem comes with a fix
sudo hashiru service install -c /etc/hashiru/hashiru.yaml
sudo hashiru status -c /etc/hashiru/hashiru.yaml
journalctl -u hashiru -f
```

As root, `service install` writes `/etc/systemd/system/hashiru.service`. The service starts at boot, restarts if it crashes, and restarts along with Docker. As a regular user (in the `docker` group), it writes a user unit and enables lingering, so the service keeps running after logout.

On first start Hashiru builds `hashiru/runner:latest`. This takes about a minute on a fast machine.

## 5. Things to decide

**Capacity.** A pool's `max` applies to each repo, so 5 repos at `max: 4` could start 20 runners. Set `max_runners` to what the machine can run (roughly CPUs ÷ `cpus` per runner). Extra jobs wait in GitHub's queue and start as slots free up. `hashiru doctor` warns if the numbers don't add up.

**Networking.** `network: host` (the default) lets job steps reach service containers on `127.0.0.1`, like GitHub-hosted runners. Concurrent jobs share the host's ports, so two jobs that both map Postgres to `5432:5432` collide. Let Docker pick the host port instead:

```yaml
services:
  postgres:
    image: postgres:17
    ports: ["5432"]   # random host port
steps:
  - run: psql -h 127.0.0.1 -p ${{ job.services.postgres.ports['5432'] }} ...
```

Bridge networking (leave `network` unset) isolates jobs, but steps can't reach service containers on localhost. Hashiru's cache server listens on the Docker bridge gateway, so caching works either way.

**Trust.** Only run repos you trust on these runners:
- With `docker_socket: true`, a job controls the host's Docker daemon, which is effectively root on the machine.
- Each job gets a fresh container, but jobs share the caches on this machine.
- For public repos, fork pull requests can run code here (`hashiru doctor` flags public repos). Require approval for outside contributors, or keep public repos off self-hosted runners.

## 6. Operating it

- **Status:** `hashiru status`, or `curl localhost:8484/api/status`.
- **Metrics:** Prometheus text at `localhost:8484/metrics`.
  - Per pool: `hashiru_up`, `hashiru_runners{state}`, `hashiru_jobs_assigned`, `hashiru_jobs_running`, `hashiru_jobs_completed_total`, `hashiru_last_error_timestamp_seconds`.
  - Machine-wide: `hashiru_host_runners`, `hashiru_cache_bytes`, `hashiru_goroutines`, `hashiru_heap_bytes`.
  - Alerts worth having: `hashiru_up == 0` for 5 minutes; `hashiru_jobs_assigned > 0` with `hashiru_runners == 0` for 10 minutes.
  - The endpoint has no auth. Keep `listen` on loopback and scrape through an SSH tunnel or a local agent.
- **Health:** `GET /healthz`.
- **Logs:** `journalctl -u hashiru`. Set `log: {format: json}` for log shippers.
- **Upgrades:** re-run the install command, then `sudo systemctl restart hashiru`. Running jobs keep going: runners are separate containers, and the new process adopts them.
- **Runner image:** rebuilt (or re-pulled) daily, because GitHub refuses runners more than 30 days behind the latest release, and Hashiru turns off the runner's self-update. New runners use the new image.
- **Disk:**
  - Caches: `hashiru cache status` shows what they use; `hashiru cache clear` empties them (they refill).
  - Docker: `docker system prune` cleans up images that jobs built through the socket.
  - New runners pause below `min_free_gb`, and `hashiru status` says why.
- **Backups:** only the config and the token/key. Everything under `cache_dir` can be rebuilt.
- **Remove everything:** `sudo hashiru service uninstall`, then `sudo hashiru cleanup --delete-scale-sets -c /etc/hashiru/hashiru.yaml`.

## 7. Known limits

- If a runner dies mid-job (host reboot, OOM kill of the whole container), GitHub marks the job failed after about 10 minutes. GitHub has no API to fail a single job sooner.
- `actions/cache` runs on this machine for github.com only. GitHub Enterprise Server jobs use GHES's own cache.
- Linux jobs only. macOS and Windows jobs stay on GitHub-hosted runners (`hashiru workflows` points them out).

## Tested

From the October 2026 production rehearsal, on Ubuntu 24.04 (17 cores, Docker 29) and macOS with OrbStack:

**Linux:**
- 19 jobs passed with `max_runners: 4`: 16 load jobs, 2 bridge-network jobs with caching, and 1 Postgres service job.
- The service was restarted mid-run.
- Peak was 4 runners, and the daemon used about 22 MB of memory.
- **Docker daemon restart:** Hashiru restarted with Docker and was back in 3 seconds. Dead runners were removed from GitHub.
- **Reboot:** the service started at boot, and pools were listening 10 seconds after the machine came up.

**macOS:**
- A 24-job burst across 3 repos with `max_runners: 6`: never more than 6 runners, and the daemon peaked at 33 MB with 39 goroutines.
- The daemon was hard-restarted mid-run and adopted all running runners.
- A runner was killed mid-job. Only that job failed, and the rest of the run finished.
- Afterwards, it drained back to the minimum with no leaked containers or offline runners.

**Unit stress tests** (`go test -race ./...`):
- 32 concurrent jobs saving and restoring caches under eviction pressure.
- 400 jobs through 4 scale sets sharing a host cap, with 5% of runner starts failing.