diff --git a/CLAUDE.md b/CLAUDE.md index 89ba01f..b025c51 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -20,6 +20,7 @@ Recovery steps are in `RUNBOOK.md`. Repo overview and workflows are in `README.m - **ZFS parent dataset mountpoints**: Pools have parent datasets (e.g., `nextcloud`, `threeteras`) that create mountpoints like `/mnt/nextcloud`. These are empty and expected — do not delete them. - **msmtp**: Replaces sendmail for cron MAILTO and smartd email alerts. Configured via `/etc/msmtprc` (Fastmail SMTP). The symlink `/usr/sbin/sendmail -> /usr/bin/msmtp` is set up in recovery step 2. - **Beszel agent**: System monitoring agent that reports to the beszel hub on hantim (`beszel.hantim.net`). Uses host network, listens on port 45876. No nginx proxy — it's not a web app. +- **Diun**: Watches all running containers daily at 06:00 for new image digests, emails via Fastmail SMTP (credentials in `docker/diun/.env`). State DB in `docker/diun/data/` tracks seen digests; `firstCheckNotif: false` suppresses the flood on first run after a restore. - **hantim-server**: Sister repo with similar nginx/certbot patterns. Located at `~/dev/hantim-server`. Useful as reference for deploy patterns. ## Remotes diff --git a/README.md b/README.md index 7332331..021c12e 100644 --- a/README.md +++ b/README.md @@ -64,6 +64,34 @@ docker exec nginx nginx -s reload git add . && git commit && git push ``` +**Updating an app (Diun notification received):** + +Diun emails when a watched image has a new digest on the registry. The email +names the image (e.g. `postgres:17-alpine`), not the container. To find which +container uses it: + +```bash +docker ps --filter "ancestor=:" --format '{{.Names}}' +``` + +Then apply the update from that app's directory: + +```bash +cd /opt/argento/docker/ +docker compose pull +docker compose up -d +docker ps # verify container is healthy +``` + +Then smoke-test the app (open the UI, or `curl -I https://`). + +Caveats before pulling: + +- Diun only reports that a digest changed -- it does not say the update is safe. For apps with schema migrations (Nextcloud, Immich, Gitea, Postgres), skim the release notes first. +- For major version bumps, run `./scripts/backup.sh` manually before pulling so the USB has a fresh snapshot. +- Postgres major version bumps (e.g. 17 -> 18) cannot reuse the old data dir -- a plain `compose up -d` will fail to start. See [RUNBOOK.md](RUNBOOK.md#maintenance-postgres-major-version-upgrade). +- After several updates, reclaim disk with `docker image prune -f`. + **Adding a new app:** 1. Create `docker//compose.yaml` with `container_name` and `shared` network diff --git a/RUNBOOK.md b/RUNBOOK.md index 840d805..e0114c0 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -306,3 +306,53 @@ curl -I http://minecraft.thekims.family # Verify SMART, ZFS, and disk space are healthy /opt/argento/scripts/disk-health-check.sh ``` + +--- + +## Maintenance: Postgres major version upgrade + +A postgres major bump (e.g. 17 -> 18) cannot reuse the old data dir -- the +on-disk format changes. Re-init and reload from `pg_dumpall`. `backup.sh` +already produces a daily dump for both Nextcloud and Immich. + +Procedure (Nextcloud; Immich notes at the end): + +```bash +cd /opt/argento/docker/nextcloud + +# 1. Force a fresh dump so it reflects current state, not yesterday's. +/opt/argento/scripts/backup.sh + +# 2. Stop the stack. Nothing will write to the DB after this. +docker compose down + +# 3. Move the dump out of the data dir before nuking it. +cp db/pg_dumpall.sql /tmp/nc-pg_dumpall.sql +mv db db.old + +# 4. Bump the postgres image tag in compose.yaml (e.g. 17-alpine -> 18-alpine). + +# 5. Start only postgres so it initializes a fresh data dir, then wait for it. +docker compose up -d postgres +docker compose logs -f postgres # wait for "ready to accept connections", Ctrl-C + +# 6. Restore. Connect to the default 'postgres' DB so the dump can recreate +# the 'nextcloud' DB itself. +docker exec -i nextcloud-db psql -U nextcloud -d postgres < /tmp/nc-pg_dumpall.sql + +# 7. Bring up the rest of the stack and verify the app end-to-end. +docker compose up -d + +# 8. Once verified, remove the old data dir and the temp dump. +rm -rf db.old /tmp/nc-pg_dumpall.sql +``` + +**Immich differences:** + +- Data dir is `${DB_DATA_LOCATION}` from `.env` (not under `docker/immich/`), + so step 3 is just `mv "$DB_DATA_LOCATION" "$DB_DATA_LOCATION.old"` -- the + dump at `docker/immich/db/pg_dumpall.sql` is in a separate dir and stays put. +- DB container is `immich_postgres`, user is `${DB_USERNAME}` from `.env`. +- Immich pins to a specific postgres image (`ghcr.io/immich-app/postgres`) + with a digest -- check Immich release notes before bumping; they coordinate + the upgrade, you usually just follow their version. diff --git a/docker/diun/diun.yml b/docker/diun/diun.yml index 6c20cd0..084b04a 100644 --- a/docker/diun/diun.yml +++ b/docker/diun/diun.yml @@ -1,6 +1,6 @@ watch: workers: 20 - schedule: "0 0 6 * * *" # daily at 09:00; six-field cron (sec min hr dom mon dow) + schedule: "0 0 6 * * *" # daily at 06:00; six-field cron (sec min hr dom mon dow) firstCheckNotif: false # don't email on first run for every image jitter: 30s