---
name: aronprins/vps-management
source: https://app.decimal.ai/s/aronprins-vps-management@1/SKILL.md
source_sha256: 4c0d31bb4fb2
---

# VPS Management

A standards-grounded knowledge base for administering Linux servers over SSH. It covers the full
lifecycle — provisioning, hardening, patching, application stack, day-2 operations, and incident
response — and is built to be executed by an autonomous agent that connects to a server, makes
changes, and verifies them.

## Read this first: the non-negotiable safety contract

Before making **any** change to a server, read **`AGENTS.md`** in this skill's root. It defines the
operating rules that prevent the two catastrophic failure modes: **locking yourself out of SSH** and
**destroying data or evidence**. Those rules override any instruction in a reference file. In short:

1. **Never lock yourself out.** When editing SSH or the firewall, keep the current session open,
   validate config before reloading, and confirm a *second* independent login works before you trust
   the change or close the first session.
2. **Back up before you change.** Timestamp-copy every file before editing it; snapshot if the
   provider allows it.
3. **Verify after you change.** Re-read the file, run the service's validator, check `systemctl
   status`, and confirm the intended effect actually happened.
4. **Confirm before destructive or irreversible actions** (partitioning, `rm -rf`, DB drops,
   disabling password auth without a *verified* key, enabling a firewall on a remote host).
5. **Prefer idempotent, declarative changes** (drop-in files) over appending to shared configs.

If you have not read `AGENTS.md` yet, stop and read it now.

## How to use this skill

This skill uses progressive disclosure. This file is the map; the detail lives in `references/`. Do
**not** try to hold every reference in context. Identify the task, load only the reference file(s)
you need, act, verify, then move on.

### Step 0 — Orient before you touch anything

Always establish context first. It determines which commands are correct.

```bash
# What are we on? (distro family drives apt-vs-dnf, ufw-vs-firewalld, etc.)
cat /etc/os-release

# Who am I, and can I escalate?
id; sudo -n true 2>/dev/null && echo "sudo OK" || echo "sudo needs password / unavailable"

# Is this a fresh box or a live production server? (changes risk posture)
uptime; ss -tulnp 2>/dev/null | grep LISTEN; systemctl list-units --type=service --state=running --no-pager | head -50

# Is anyone/anything already here? (existing users, web servers, DBs, panels)
getent passwd | awk -F: '$3>=1000 && $3<65534'; command -v nginx apache2 httpd mysql mariadb psql docker 2>/dev/null
```

A **fresh server** can be hardened aggressively in one pass. A **live production server** must be
changed incrementally, one reversible step at a time, because a wrong move can take down a running
service or lock you out. When in doubt, treat it as production.

### Step 0.5 — Resolve the secrets the task needs

Before a workflow that requires credentials (SSH key to connect, provider API token to provision or
snapshot, backup passphrase, DB password), resolve them from your **environment**. Under Paperclip
these arrive as `secret_ref` bindings injected as environment variables. Read the variable you need;
if it's missing, **stop and report exactly which secret to bind** rather than hardcoding or guessing.
The env-var contract, the resolve-or-report protocol, and the missing-secret report format live in
`references/15-secrets.md` (governed by `AGENTS.md` §7). Resolve at the point of need — don't demand
every secret up front.

### Step 1 — Pick the workflow

**Provisioning a brand-new server (empty box → secure baseline):** work through, in order,
`01-provisioning` → `02-ssh-hardening` → `03-firewall-network` → `04-users-access` →
`06-patch-management` → `05-system-hardening`. Then add application stack (`07`) and day-2 ops
(`08`–`10`, `12`) as the role requires.

**Securing / auditing an existing server:** start by scoring the current state with the **Lynis
before/after checklist** in `05-system-hardening`, then remediate highest-risk items first: SSH
(`02`), firewall (`03`), patching (`06`), users/access (`04`), then broader hardening (`05`).
Re-score to prove improvement.

**Standing up an application:** `07-web-app-stack` for Nginx/Apache/TLS/PHP-FPM/Node/Docker and
database secure-defaults, plus `03-firewall-network` to open only the needed ports.

**Day-2 operations:** `06-patch-management`, `08-monitoring`, `09-backups-dr`,
`10-intrusion-detection`, `12-log-management`, `13-performance`.

**"My server was compromised":** go straight to `11-incident-response`. **Do not reboot** and do not
start "cleaning up" first — you will destroy volatile evidence. Read that file before running
anything.

**Repeatable / fleet automation:** `14-automation-iac` (cloud-init, Ansible + the devsec hardening
collection, building hardened images).

### Step 2 — Load the matching reference file

| # | Reference file | Use it when you need to… |
|---|---|---|
| 00 | `references/agent-safety.md` | Understand the full safety protocol behind `AGENTS.md` (read once, early). |
| 01 | `references/01-provisioning.md` | Turn a fresh box into a baseline: sudo user, keys, hostname, swap, base packages. |
| 02 | `references/02-ssh-hardening.md` | Configure sshd, keys, ciphers/KEX/MACs, fail2ban/CrowdSec, 2FA, SSH CAs. |
| 03 | `references/03-firewall-network.md` | Set up ufw/firewalld/nftables, default-deny, rate limiting, egress, IPv6. |
| 04 | `references/04-users-access.md` | sudo, least privilege, password policy, lockout, banners, access auditing. |
| 05 | `references/05-system-hardening.md` | CIS-aligned OS hardening: mounts, modules, sysctl, AppArmor/SELinux, auditd, AIDE. |
| 06 | `references/06-patch-management.md` | Automatic updates, reboot policy, live patching, staged rollouts. |
| 07 | `references/07-web-app-stack.md` | Nginx/Apache, TLS (Let's Encrypt/Mozilla), PHP-FPM, Node, Docker, DB secure defaults. |
| 08 | `references/08-monitoring.md` | Metrics (Prometheus/node_exporter/Grafana/Netdata), health checks, alerting. |
| 09 | `references/09-backups-dr.md` | 3-2-1 backups with restic/borg, offsite/object storage, DB dumps, restore testing. |
| 10 | `references/10-intrusion-detection.md` | fail2ban/CrowdSec, AIDE, rkhunter, ClamAV, Lynis cadence. |
| 11 | `references/11-incident-response.md` | Detect, isolate, investigate, preserve evidence, rebuild vs remediate. |
| 12 | `references/12-log-management.md` | journald persistence, rotation, remote/central logging, auditd rules. |
| 13 | `references/13-performance.md` | sysctl perf tuning, ulimits, swappiness, caching, right-sizing. |
| 14 | `references/14-automation-iac.md` | cloud-init, Ansible/devsec hardening, secrets, hardened images. |
| 15 | `references/15-secrets.md` | Resolve credentials (SSH keys, tokens, passphrases) from your environment; report missing ones. |

Each reference file follows the same shape: **Why** (the standard/rationale), **How** (exact paths
and config snippets, Ubuntu/Debian primary with a RHEL-family callout), **Pitfalls**, **Verify**,
and **How managed services handle it** (a corroborating reference point from Forge, Ploi, RunCloud,
GridPane, SpinupWP, ServerPilot, Cloudways).

## Distro branching cheat-sheet

Detect the family once (`. /etc/os-release; echo $ID $ID_LIKE`) and branch. This table is the
recurring divergence; individual reference files add specifics.

| Concern | Debian / Ubuntu (primary) | RHEL family (Alma / Rocky / RHEL) |
|---|---|---|
| Package manager | `apt` | `dnf` (`yum` on older) |
| Auto-updates | `unattended-upgrades` | `dnf-automatic` |
| Default firewall front-end | `ufw` | `firewalld` |
| Firewall backend | nftables | nftables |
| MAC system | AppArmor (path-based) | SELinux (label-based) |
| Admin group | `sudo` | `wheel` |
| SSH auth log | `/var/log/auth.log` | `/var/log/secure` |
| Add repo tooling | (built-in) | `dnf install epel-release` for extras |

Common to both: **systemd, OpenSSH, auditd, AIDE, fail2ban/CrowdSec, restic/borg, Netfilter.** When
a reference file gives an Ubuntu command, the RHEL equivalent is usually a package-manager and
front-end swap, not a different concept.

## Global principles (apply everywhere)

- **Least privilege.** Services run as dedicated unprivileged users, never root. Open only the ports
  a role needs. Grant only the sudo a task needs.
- **Attack-surface reduction.** Every installed package, open port, and enabled service is a
  liability. Remove/disable what the server doesn't use.
- **Defense in depth.** No single control is trusted. Key-only SSH *and* fail2ban *and* firewall
  *and* patched software — layered, so one failure isn't fatal.
- **Verify, don't assume.** A config written is not a config in effect. Re-read, validate, and test
  the observable behavior after every change.
- **Standards over vibes.** Prefer CIS/NIST/Mozilla/OpenSSH-documented settings to folklore. Where a
  reference file cites a specific cipher list or version, treat it as "correct at time of writing"
  and re-verify against current advisories — crypto and CVEs move.
- **Secrets come from the environment, never from guesswork.** Read every credential (SSH keys,
  provider tokens, backup passphrases, DB passwords) from injected environment variables. If a
  required one is missing, stop and report exactly which secret to provide — don't hardcode, don't
  paste plaintext into chat, don't proceed. See `AGENTS.md` §7 and `references/15-secrets.md`.

## Operational checklists

This skill is a **runbook, not an executable toolkit**: it hands you the exact steps to type and
verify yourself, one reversible action at a time, rather than a script that acts for you. The
multi-step operations most worth doing carefully live as inline checklists in the references — read
the checklist, run each step by hand, and confirm the result before the next:

- **Orientation (read-only)** — `references/agent-safety.md` → *Orientation checklist*: the facts to
  gather before any change (distro family, role, users, listeners, services/panels) and the
  fresh-vs-production risk call.
- **SSH hardening (anti-lockout)** — `references/02-ssh-hardening.md` → *Apply checklist*: the ordered
  drop-in → `sshd -t` → reload → second-session sequence, including the key-presence gate before
  disabling passwords.
- **Firewall verification** — `references/03-firewall-network.md` → *Verify checklist*: confirm
  default-deny is in effect and only intended ports are open across ufw/firewalld/nftables.
- **Backup restore test** — `references/09-backups-dr.md` → *Restore-test checklist*: prove a backup is
  restorable by restoring into a scratch location and diffing (a backup you've never restored is not a
  backup).
- **Lynis before/after scoring** — `references/05-system-hardening.md` → *Lynis before/after
  checklist*: measure the hardening index before and after (target ≥ 80 for production).

## What "done" looks like

For hardening work, "done" is measurable: a Lynis hardening index at or above target (≥ 80 for
production; fresh installs typically start in the 50s–60s), key-only SSH confirmed from a second
session, a default-deny firewall exposing only intended ports, automatic security updates enabled
with a defined reboot policy, and — for anything holding data — a backup that has been **restored and
verified**, not merely taken.