--- name: qf-studio/pilot-aws source: https://app.decimal.ai/s/qf-studio-pilot-aws@3/SKILL.md source_sha256: d03ca3706958 --- # Pilot AWS Operations (founder box) Since the S6-lite cutover (2026-07-16, TASK-409) the Pilot daemon runs on an EC2 box, NOT locally. **Local `pgrep pilot` returning 0 is correct.** ## Constants ``` INSTANCE i-0e0c1ca34e7b561f9 # "pilot-founder-box", t3.xlarge, eu-central-1a PROFILE quantflow # export AWS_PROFILE=quantflow for every aws call REGION eu-central-1 RUNNER i-0147f5c24d234cdbb # mgmt runner (AdministratorAccess) — IAM/privileged ops only DAEMON tmux session "pilot" as ec2-user, wrapper /home/ec2-user/start-pilot.sh BINARY /var/lib/pilot/bin/pilot # ec2-user-owned (self-upgrade works, #4470); /usr/local/bin/pilot is a SYMLINK to it — never `sudo install` over the symlink; rollback at pilot.prev STATE /home/ec2-user/.pilot → /var/lib/pilot/pilot-home (200GB data volume) REPOS /Users/aleks.petrov/Projects → /var/lib/pilot/repos (path shim — ledger keys on the macOS-era absolute paths; NEVER "fix" these symlinks) DB /home/ec2-user/.pilot/data/pilot.db LOG /home/ec2-user/.pilot/logs/daemon.log (+ daemon-stderr.log) ``` ## The SSM command pattern (used by everything below) ```bash export AWS_PROFILE=quantflow AWS_DEFAULT_REGION=eu-central-1 CMD=$(aws ssm send-command --instance-ids i-0e0c1ca34e7b561f9 \ --document-name AWS-RunShellScript \ --parameters 'commands=[""]' \ --query Command.CommandId --output text) sleep 10 aws ssm get-command-invocation --command-id $CMD \ --instance-id i-0e0c1ca34e7b561f9 --query StandardOutputContent --output text ``` For multi-line/quote-heavy payloads write a JSON file and use `--parameters file:///tmp/x.json`; for shipping FILES to the box, base64 the content into the command (`echo | base64 -d > target`) — inline heredocs with `\n` escapes DO NOT survive SSM JSON (verified failure mode). ## Operations ### Status (first move, zero GitHub quota) ```bash ~/bin/pilot-board # daemon health, queue, autopilot, log tail ~/bin/pilot-board --gh # + GitHub issues/PRs (costs shared user quota — sparingly) ``` Remote half lives at `/usr/local/bin/pilot-board-remote` on the box (edit there). ### Live TUI dashboard ```bash ~/bin/pilot-dash # SSM interactive → tmux attach -t pilot ``` Detach: `Ctrl-B` then `D`. **NEVER press `q` or `Ctrl-C` inside the TUI** — that stops the daemon. Fallback: plain ssm session → `sudo su - ec2-user` → `tmux attach -t pilot`. ### Logs ```bash # tail (adjust -n / add grep): SSM: tail -50 /home/ec2-user/.pilot/logs/daemon.log # common filters: 'rate limit', 'claim lost', 'approval', 'ERROR', a task id ``` ### Queue / ledger queries (read-only) ```bash SSM: sqlite3 -column /home/ec2-user/.pilot/data/pilot.db \ "SELECT task_id,status,datetime(created_at) FROM executions \ WHERE status IN ('running','queued') ORDER BY created_at;" ``` Key tables: `executions`, `execution_claims` (task_id, project_path, generation), `autopilot_pr_state`, `autopilot_scope_release`, `instance_events`. ⚠️ Timestamp trap: pre-2026-07-16 rows may carry legacy `…+02:00` string format — never filter by string time ranges across eras; use rowid or exact ids. ### Stop / start / restart ⚠️ OPERATOR-CONSENT ACTIONS Never do these on your own judgment during watch/autonomous modes; in interactive sessions get explicit user go-ahead. In-flight executions die (they retry via generation claims — proven safe, but wasteful). ```bash # STOP (graceful; wait, then verify 0): SSM: sudo -iu ec2-user tmux send-keys -t pilot C-c # TUI quit = graceful shutdown sleep 15; ps -eo comm | grep -c '^pilot$' # must be 0 # START: SSM: sudo -iu ec2-user tmux new-session -d -s pilot -x 220 -y 50 /home/ec2-user/start-pilot.sh sleep 20; ps -eo comm | grep -c '^pilot$' # must be 1; then check pilot-board # RESTART = STOP, verify, START, verify (banner version + no Telegram 409 + poller ticks). ``` After ANY start: verify version (`/usr/local/bin/pilot version`), exactly 1 process, `curl -s localhost:9091/metrics | grep pilot_queue_depth` (via SSM). ### Rebuild / upgrade the box binary ```bash SSM (as ec2-user, NO sudo — dir is ec2-user-owned since 2026-07-19): cd /Users/aleks.petrov/Projects/startups/pilot && \ git fetch -q --tags origin main && git checkout -q && \ make build && install -m 0755 bin/pilot /var/lib/pilot/bin/pilot && \ git checkout -q main && /var/lib/pilot/bin/pilot version ``` Then RESTART (above) to activate. Prefer building from a released tag. For releases, daemon self-upgrade also works now (#4470 preflight + writable dir) — letting it upgrade itself on the next train is the default path. Expect ~1 quiet hour post-restart until #4391 ships: startup rescans can burn the GitHub user-aggregate rate pool (see Troubleshooting). ### Metrics tunnel (grafterm / local Prometheus tools) ```bash ~/bin/pilot-tunnel # box:9091 → localhost:9091, keep running; then `pilot-tui` (grafterm alias) ``` ### Box shell (interactive) ```bash aws ssm start-session --target i-0e0c1ca34e7b561f9 --profile quantflow --region eu-central-1 sudo su - ec2-user ``` ### Privileged infra ops (IAM, volumes, instance-level) User `aleks` lacks iam:PassRole/ec2:CreateVolume etc. Route through the mgmt runner via SSM RunShellScript on `i-0147f5c24d234cdbb` (it has admin). Scope every grant minimally; never print policy docs containing secrets. ## Hard rules 1. **One daemon, ever.** Never start pilot locally while the box serves the repos (dual-serve = cross-machine duplicate class; claims are per-DB). Rollback procedure lives in `.agent/tasks/TASK-409-s6-lite-aws-cutover.md`. 2. **Never touch the path shims** (`/Users/aleks.petrov/...` symlinks on the box) — ledger + claims key on those exact strings. 3. **GitHub API is one shared per-USER pool (5000/hr)** across every token, session, and the daemon. Prefer sqlite/metrics over `gh` for status. If the daemon logs "rate limit exceeded for user ID …" — it self-recovers on the rolling window; do not thrash retries. (#4391 tracks the durable fix.) 4. Secrets: never echo tokens/keys; never `ps` full args of processes holding tokens; the wrapper script pattern exists precisely to keep tokens out of argv. Config on the box is verbatim-from-laptop; changes = operator consent + restart. 5. Trust the ledger over dashboard panels (known mislabel: awaiting_approval rendered as "rebase" until the GH-4383 fix is in the running binary). 6. **Name your ledger.** Every status/diagnosis claim must state which data source produced it: box DB (via SSM), GitHub, or the laptop archive. The laptop's `~/.pilot/data/pilot.db` is a FROZEN pre-2026-07-16 archive (S6-lite cutover) — plausible-looking rows, months stale. A 2026-07-27 incident: a session read it and confidently misdiagnosed healthy tasks as "failed". Before reasoning about any DB: verify the path is the box's (`/var/lib/pilot/pilot-home/data/pilot.db`) and check row freshness (`select max(datetime(created_at)) from executions`). mem-160 family. ## Troubleshooting quick table | Symptom | Likely cause | Move | |---|---|---| | Board says queue empty but daemon log shows executions; log claim generations exceed claims table | split-brain shadow ledger (#4393 class — daemon opened a DB at an unshimmed path) | `sudo readlink /proc/$(pgrep -x pilot)/fd/*` must include `/var/lib/pilot/pilot-home/data/pilot.db`; if not: STOP daemon, locate+merge shadow DB, fix shim | | ALL task executions fail `unknown: exit status 1` after ~3m, 0 tokens, stream shows `api_retry`/`fetch failed`; judge/preflight children work | RLIMIT_AS cap on executor children (#4401 class — GH-3028 "RSS cap", darwin no-op) | `grep 'address space' /proc//limits` must be `unlimited`; config `subprocess_limits` is `enabled: false` since 2026-07-17 (backup `config.yaml.bak-4396`) — OOM cap off until #4401 | | Queue frozen, pollers 403 "rate limit … user ID" | user-aggregate GitHub pool exhausted (startup rescans, parallel sessions) | wait for rolling window; stop nonessential gh usage; see #4391 | | Queue frozen, "dispatch claim lost" every poll, no 403s | dead-owner non-terminal rows holding gen-N claims (post-restart/cutover) | see #4392; workaround = mark orphan rows `stalled` with audit note (exact-id UPDATE, never string time-ranges) — this is a RECOVERY signal (retry the task), not a cancel; see next row if the goal is to stop a task | | Need to permanently stop a task from ever being re-picked (dup ticket, wrong scope, abandoned, operator error) | `stalled` is NOT a cancel verb — it means "dead owner, retry me" and the dispatcher grants it fresh generations exempt from the repick hard cap forever if hand-written for this purpose (GH-4655 incident) | `pilot task cancel [--project ] [--reason "..."]` (GH-4678) — the real terminal cancel; NEVER hand-write `status='stalled'` to try to stop a task | | tmux session gone, no pilot process | wrapper/script error at spawn | check `daemon-stderr.log`; verify wrapper intact (`cat start-pilot.sh`); restart per above | | TUI monochrome | TERM captured at daemon start | `~/.tmux.conf` already sets 256color; colors return on next restart — do not bounce for paint | | PR stuck "rebase N/3" in panel | awaiting_approval mislabel | ledger: `SELECT stage FROM autopilot_pr_state WHERE pr_number=N` | | Box unreachable via SSM | agent/instance down | `aws ec2 describe-instances --instance-ids i-0e0c…` → LOUD escalate to operator; never assume | | Multi-day `uptime` right after a train — looks like "installed but never restarted" | NOT a stale process — self-upgrade's restart leg is `syscall.Exec` (PID preserved, so ps etime/`uptime` survives the restart; mem `hot-restart-preserves-pid-uptime-false-mismatch`). Since 2026-08-13 board `ver` reads the RUNNING process (`pilot_build_info` metric, #4864/PR#4865; repointed `pilot-board-remote`) — a `(disk!)` suffix means the daemon metric was unreachable and the value is the disk binary's | Trust `ver` (metric-sourced). Cross-check: `curl -s localhost:9091/metrics \| grep pilot_build_info` via SSM, or `grep 'upgrade verified complete' ~/.pilot/logs/daemon.log` (boot-reconcile `from=`/`to=` logged by the NEW image; failure counterpart `previous upgrade did NOT take effect`). Do NOT bounce the daemon on uptime evidence alone | ## Refs - Cutover plan + rollback: `.agent/tasks/TASK-409-s6-lite-aws-cutover.md` - Restart discipline: `.agent/sops/operations/safe-daemon-restart.md` (verify PID+banner ALWAYS) - Open hardening: #4391 (rate-budget client), #4392 (orphan reconciliation)