hantim-server/README.md

# hantim-server

Server provisioning and app management for the hantim webserver. This repo
lives at `/opt/hantim` on the server and contains Docker Compose configs, deploy
scripts, nginx configs, and Gitea Actions workflows for each app.

See ARCHITECTURE.md for technical deep-dives, USECASES.md for expected behaviors.

## Infrastructure overview

Two servers and a cloud secrets service:

- **Hantim** (Vultr, Rocky Linux 9) — production web server running nginx,
  app containers, and a Garage storage node via Docker Compose
- **Argento** (home server) — Gitea instance (`git.timothykim.net`) with
  Docker registry, CI runner, and a second Garage storage node (with web UI)
- **Bitwarden Secrets Manager** (cloud) — all secrets, fetched via the `bws` CLI

All app containers share a `shared` Docker network. Only nginx binds host ports
(80/443). Garage provides S3-compatible media storage replicated across both
nodes.

## Repo structure

```
scripts/
  bootstrap.sh         # First-time server setup (manual, uses Bitwarden)
  configure.sh         # Idempotent server config (CI-safe)
  deploy.sh            # Deploy + provision (called via SSH)
tools/
  app.sh               # Add a new static site (run from dev machine)
  service.sh           # Manage Docker services by subcommand (dns, files, nginx, cert, all)
  remove-app.sh        # Remove a static site (run from dev machine)
docker/
  nginx/               # Reverse proxy (nginx:alpine) + SSL config
    nginx.conf         # Main nginx config
    conf.d/            # Per-app server blocks (e.g. example.com.conf)
    compose.yml
  <domain>/            # Per-app compose files (e.g. example.com)
    compose.yml
  goatcounter/         # GoatCounter analytics
  garage/              # Garage S3-compatible object storage
    compose.yml
    Dockerfile
    garage.toml
    .env.keys
tests/                 # pytest integration tests (run post-deploy)
.gitea/workflows/      # Per-app deploy workflows + provision.yml
```

## Provisioning a new server

On a fresh Rocky Linux 9 install:

```bash
dnf install -y git
git clone https://git.timothykim.net/hantim/hantim-server.git /opt/hantim
bash /opt/hantim/scripts/bootstrap.sh
```

`bootstrap.sh` runs once manually. It installs dependencies, sets up the `bws`
CLI, creates the `deploy` user, and calls `configure.sh`.

`configure.sh` is idempotent and CI-safe. It opens firewall ports, sets up
certbot renewal, and deploys all services (nginx first, then apps).

**Prerequisites:**
- DNS for all configured domains must point to the server
- Bitwarden Secrets Manager access token for the `hantim-server` machine account

Restoring from backup is identical — run the same three commands on a fresh
server. Everything is idempotent.

## Adding a new static site

```bash
./tools/app.sh all <domain>
```

This single command handles everything: DNS records (Vultr), Gitea repo
(from `static-site-template`), nginx/compose/workflow configs, SSL cert,
Garage media bucket, first build, verification, and UptimeRobot monitoring.

**Prerequisites:** domain nameservers pointed to Vultr, `bws`/`jq`/`dig`
installed, `BWS_ACCESS_TOKEN` env var or token at `~/.config/hantim/bws-token`.

After the site is live, clone the app repo and customize:

```bash
git clone git@git.timothykim.net:hantim/<domain>.git
cd <domain>
# edit content
git add . && git commit -m "initial content" && git push
```

## Deploying changes

**App code change** — push to an app repo (e.g., `example.com`):
- `build.yml` builds the image, pushes to registry, SSHes to server to deploy

**Compose/nginx config change** — push to this repo:
- Path-matched workflows trigger and SSH to server
- nginx deploys test config before applying (`nginx -t`)

**Setup change** — push changes to `scripts/configure.sh`:
- `provision.yml` triggers and re-runs `configure.sh`

## Media hosting

Media files are stored in Garage and served via nginx at `/media/` on each
domain. This keeps large files out of git repos.

**Endpoints:**
- `s3.hantim.net` — S3 API for uploads (authenticated)
- `garage.hantim.net` — Admin API for bucket management
- `https://www.<domain>/media/<path>` — public file access

**Uploading:**

```bash
# One-time AWS CLI setup
aws configure  # key ID + secret from bws, region: garage, output: json
# Then add to ~/.aws/config under [default]:
#   endpoint_url = https://s3.hantim.net

# Upload
aws s3 cp photo.jpg s3://example.com/photo.jpg
aws s3 sync static/media/ s3://example.com/
```

**Local development:** store media in `static/media/` (gitignored) so files
are accessible at `/media/` when serving locally.

**Creating buckets manually** (for existing domains without buckets):

```bash
# Get admin token and media key ID from bws
# Create bucket
curl -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"globalAlias": "example.com"}' https://garage.hantim.net/v2/CreateBucket
# Note the bucket ID from the response, then:
curl -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"bucketId": "BUCKET_ID", "accessKeyId": "MEDIA_KEY_ID", "permissions": {"read": true, "write": true, "owner": false}}' \
  https://garage.hantim.net/v2/AllowBucketKey
curl -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"websiteAccess": {"enabled": true, "indexDocument": "index.html"}}' \
  "https://garage.hantim.net/v2/UpdateBucket?id=BUCKET_ID"
```

## Garage cluster initialization

After provisioning a fresh server, Garage connects to argento automatically.
The cluster layout must be assigned manually once (node IDs change per install):

```bash
docker exec garage /garage status
docker exec garage /garage layout assign <hantim-node-id> -z hantim -c 1G
docker exec garage /garage layout assign <argento-node-id> -z argento -c 1G
docker exec garage /garage layout apply --version 1
```

If only hantim is reprovisioned, argento retains the layout and hantim
reconnects automatically.

## Testing

Tests live in `tests/` and run with [uv](https://docs.astral.sh/uv/):

```bash
# Install uv: curl -LsSf https://astral.sh/uv/install.sh | sh

# Live site tests (security headers, TLS, default server)
uvx pytest tests/ -v --ignore=tests/test_app_e2e.py

# E2E tests for tools/app.sh (creates real resources)
uvx pytest tests/test_app_e2e.py -v -x
```

### Live site tests

These run against live sites and verify infrastructure invariants:

- **`test_security_headers.py`** — Checks that all sites including
  `security-headers.inc` return the expected headers (HSTS, X-Frame-Options,
  etc.). Sites are auto-discovered from `docker/nginx/conf.d/*.conf`.
- **`test_tls.py`** — Verifies TLS 1.3 and 1.2 work, TLS 1.1 is rejected.
- **`test_default_server.py`** — Verifies unknown Host headers get dropped
  (connection reset) on both HTTP and HTTPS.

The `deploy-nginx.yml` workflow runs these automatically after each nginx
deploy. If tests fail, Gitea sends an email notification.

### E2E tests

`test_app_e2e.py` tests `tools/app.sh` against real services (Vultr, Gitea,
Garage, UptimeRobot). Uses `kgfamily.com` as a throwaway test domain.

**Prerequisites:**
- CLI tools: `bws`, `jq`, `dig`, `ssh`, `curl`, `git`
- `BWS_ACCESS_TOKEN` env var or token at `~/.config/hantim/bws-token`
- Vultr API key must allow requests from the machine's IP
  (check https://my.vultr.com/settings/#settingsapi)

**Structure:**

1. **TestAppSubcommands** — Runs each `app.sh` subcommand individually
   (dns, repo, files, cert, garage, build, monitor), verifies side effects
   via API, then cleans up.
2. **TestAppAll** — Runs `app.sh all kgfamily.com`, verifies the full flow
   including site liveness and git commit, then cleans up.

Cleanup runs before and after each phase via APIs (Vultr, Gitea, Garage,
UptimeRobot) + local file deletion. Does not use `remove-app.sh` to avoid
coupling.

## Secrets

### Bitwarden Secrets Manager

Project: `hantim`. Fetched via `bws` CLI on the server.

| Secret | Purpose | Used by |
|---|---|---|
| `hantim-ci-registry-push` | Gitea token for Docker registry | `bootstrap.sh` |
| `hantim-deploy-ssh-public-key` | Deploy user's SSH public key | `bootstrap.sh` |
| `hantim-deploy-ssh-private-key` | Deploy user's SSH private key | `app.sh` |
| `hantim-new-app-script` | Gitea API token for creating repos | `app.sh` |
| `hantim-vultr-api-key` | Vultr API key for DNS management | `app.sh` |
| `hantim-garage-rpc-secret` | Garage cluster RPC secret | `deploy.sh` |
| `hantim-garage-argento-node-id` | Argento's Garage node ID + address | `deploy.sh` |
| `hantim-garage-admin-token` | Garage admin API token | `deploy.sh`, `app.sh` |
| `hantim-garage-media-key-id` | S3 access key ID for media uploads | `app.sh`, `aws` CLI |
| `hantim-garage-media-secret-key` | S3 secret key for media uploads | `aws` CLI |
| `hantim-uptimerobot-api-key` | UptimeRobot API key for monitors | `app.sh` |
| `hantim-goatcounter-password` | GoatCounter admin password | `deploy.sh` |

Machine accounts: `hantim-server` (token at `/etc/bws-token`), `hantim-ci` (reserved).

### Service secrets (`.env.keys`)

Services that need secrets declare them in `.env.keys` (format:
`ENV_VAR=bws-secret-uuid`, one per line, with `# secret-name` comments).
`deploy.sh` fetches each secret by UUID from bws and generates `.env`
before starting the service.

### Gitea org-level secrets

Stored in the `hantim` Gitea org for direct use in CI workflows:

| Name | Type | Purpose |
|---|---|---|
| `DEPLOY_HOST` | Variable | Server IP or hostname |
| `DEPLOY_HOST_KEY` | Variable | Server SSH host public key (for host key pinning) |
| `DEPLOY_SSH_KEY` | Secret | SSH private key for the deploy user |
| `CI_REGISTRY_TOKEN` | Secret | Gitea token for Docker registry login |

After reprovisioning hantim (new host key), update `DEPLOY_HOST_KEY`:

```bash
ssh-keyscan -t ed25519 hantim.net 2>/dev/null
```