doc update

.gitignore

@@ -2,3 +2,8 @@ smb/
 **/.env
 system/samba-private/
 
+# vim swap/backup files
+*.swp
+*.swo
+*~
+

CLAUDE.md

@@ -2,7 +2,7 @@
 
 ## What this repo is
 
-Git-based provisioning and recovery repo for the "argento" home server (Rocky Linux 9, ZFS, Docker). The repo lives at `/opt/argento/` on the server and contains the actual running configs. This dev copy at `~/dev/argento` is where edits happen, then files are manually copied to the server.
+Git-based provisioning and recovery repo for the "argento" home server (Rocky Linux 9, ZFS, Docker). The repo lives at `/opt/argento/` on the server and contains the actual running configs. This dev copy at `~/dev/argento` is where edits happen, then synced to the server via git push/pull (Timothy runs the git commands).
 
 Recovery steps are in `RUNBOOK.md`. Repo overview and workflows are in `README.md`.
 
@@ -20,6 +20,7 @@ Recovery steps are in `RUNBOOK.md`. Repo overview and workflows are in `README.m
 - **ZFS parent dataset mountpoints**: Pools have parent datasets (e.g., `nextcloud`, `threeteras`) that create mountpoints like `/mnt/nextcloud`. These are empty and expected — do not delete them.
 - **msmtp**: Replaces sendmail for cron MAILTO and smartd email alerts. Configured via `/etc/msmtprc` (Fastmail SMTP). The symlink `/usr/sbin/sendmail -> /usr/bin/msmtp` is set up in recovery step 2.
 - **Beszel agent**: System monitoring agent that reports to the beszel hub on hantim (`beszel.hantim.net`). Uses host network, listens on port 45876. No nginx proxy — it's not a web app.
+- **Diun**: Watches all running containers daily at 06:00 for new image digests, emails via Fastmail SMTP (credentials in `docker/diun/.env`). State DB in `docker/diun/data/` tracks seen digests; `firstCheckNotif: false` suppresses the flood on first run after a restore.
 - **hantim-server**: Sister repo with similar nginx/certbot patterns. Located at `~/dev/hantim-server`. Useful as reference for deploy patterns.
 
 ## Remotes

README.md

@@ -20,6 +20,7 @@ docker/            Docker app configs (compose files, nginx confs)
   minecraft/       MCSManager
   immich/          Immich photo server + PostgreSQL
   beszel-agent/    System monitoring agent (reports to beszel.hantim.net)
+  diun/            Daily image-digest watcher (emails when updates available)
 smb/               Samba share (ZFS mountpoint, not in git)
 system/            System config snapshots (smb.conf, sanoid.conf, firewalld, etc.)
 ```
@@ -31,20 +32,19 @@ system/            System config snapshots (smb.conf, sanoid.conf, firewalld, et
 | `RUNBOOK.md` | Full recovery steps, top to bottom |
 | `scripts/backup.sh` | Daily USB backup (DB dumps + rsync) |
 | `scripts/issue-cert.sh` | Issue SSL certs via certbot webroot |
-| `scripts/disk-health-check.sh` | SMART, ZFS, disk space monitoring (daily alerts + weekly reports via cron) |
+| `scripts/disk-health-check.sh` | SMART, ZFS, quota, disk space monitoring (daily alerts + weekly reports via cron) |
 | `scripts/sync-system.sh` | Auto-sync system configs to git |
 | `system/tracked-configs` | Maps system config paths to repo paths |
 | `docker/nginx/conf.d/_template.conf.example` | Template for new nginx app confs |
 
 ## Workflows
 
-**Compose files** -- edit in `/opt/argento/`, apply, commit:
+**Compose files** -- edit in `/opt/argento/`, apply:
 
 ```bash
 cd /opt/argento/docker/<app>
 vim compose.yaml
 docker compose up -d
-git add . && git commit && git push
 ```
 
 **System configs** -- edit in place, auto-synced daily:
@@ -61,9 +61,25 @@ systemctl restart smb
 vim /opt/argento/docker/nginx/conf.d/<domain>.conf
 docker exec nginx nginx -t
 docker exec nginx nginx -s reload
-git add . && git commit && git push
 ```
 
+**Updating an app (Diun notification received):**
+
+Diun emails name the image (e.g. `postgres:17-alpine`), not the container. Find it:
+
+```bash
+docker ps --filter "ancestor=<image>:<tag>" --format '{{.Names}}'
+```
+Then update:
+
+```bash
+cd /opt/argento/docker/<app>
+docker compose pull --ignore-buildable
+docker compose build --pull # if repo contains dockerfile
+docker compose up -d
+```
+Postgres major version bumps (e.g. 17 -> 18) need extra steps -- see [RUNBOOK.md](RUNBOOK.md#maintenance-postgres-major-version-upgrade).
+
 **Adding a new app:**
 
 1. Create `docker/<app>/compose.yaml` with `container_name` and `shared` network
@@ -71,7 +87,6 @@ git add . && git commit && git push
 3. `./scripts/issue-cert.sh <domain>`
 4. `cp docker/nginx/conf.d/_template.conf.example docker/nginx/conf.d/<domain>.conf` and fill in placeholders
 5. `docker exec nginx nginx -t && docker exec nginx nginx -s reload`
-6. Commit and push
 
 ## Recovery
 
@@ -79,15 +94,15 @@ See [RUNBOOK.md](RUNBOOK.md) for full disaster recovery steps.
 
 ## What's protected where
 
-| Data | Protection | Recovery |
-|------|-----------|----------|
-| Configs (compose, nginx, system) | Git (Gitea bare repo on USB backup) | `git clone` from USB |
-| Nextcloud files | ZFS mirror + sanoid snapshots + USB | `rsync` from USB |
-| Media library | ZFS raidz2 + sanoid snapshots + USB | `rsync` from USB |
-| Nextcloud DB (PostgreSQL) | ZFS dataset + `pg_dumpall` + USB | `rsync` from USB |
-| Immich library (photos/videos) | ZFS raidz2 + sanoid snapshots + USB | `rsync` from USB |
-| Immich DB (PostgreSQL) | `pg_dumpall` + USB | `rsync` from USB |
-| Garage S3 data | 2-node replication + USB | Replication or USB |
-| Gitea repos + DB | SQLite `.backup` + USB | `rsync` from USB |
-| Secrets (.env files) | USB backup + Bitwarden | `rsync` from USB, or recreate from Bitwarden |
-| SSL certificates | Re-issued from Let's Encrypt | `./scripts/issue-cert.sh` |
+| Data | Protection |
+|------|-----------|
+| Configs (compose, nginx, system) | Git (Gitea bare repo on USB backup) |
+| Nextcloud files | ZFS mirror + sanoid snapshots + USB |
+| Media library | ZFS raidz2 + sanoid snapshots + USB |
+| Nextcloud DB (PostgreSQL) | ZFS dataset + `pg_dumpall` + USB |
+| Immich library (photos/videos) | ZFS raidz2 + sanoid snapshots + USB |
+| Immich DB (PostgreSQL) | `pg_dumpall` + USB |
+| Garage S3 data | 2-node replication + USB |
+| Gitea repos + DB | SQLite `.backup` + USB |
+| Secrets (.env files) | USB backup + Bitwarden |
+| SSL certificates | Re-issued from Let's Encrypt |

RUNBOOK.md

@@ -306,3 +306,53 @@ curl -I http://minecraft.thekims.family
 # Verify SMART, ZFS, and disk space are healthy
 /opt/argento/scripts/disk-health-check.sh
 ```
+
+---
+
+## Maintenance: Postgres major version upgrade
+
+A postgres major bump (e.g. 17 -> 18) cannot reuse the old data dir -- the
+on-disk format changes. Re-init and reload from `pg_dumpall`. `backup.sh`
+already produces a daily dump for both Nextcloud and Immich.
+
+Procedure (Nextcloud; Immich notes at the end):
+
+```bash
+cd /opt/argento/docker/nextcloud
+
+# 1. Force a fresh dump so it reflects current state, not yesterday's.
+/opt/argento/scripts/backup.sh
+
+# 2. Stop the stack. Nothing will write to the DB after this.
+docker compose down
+
+# 3. Move the dump out of the data dir before nuking it.
+cp db/pg_dumpall.sql /tmp/nc-pg_dumpall.sql
+mv db db.old
+
+# 4. Bump the postgres image tag in compose.yaml (e.g. 17-alpine -> 18-alpine).
+
+# 5. Start only postgres so it initializes a fresh data dir, then wait for it.
+docker compose up -d postgres
+docker compose logs -f postgres   # wait for "ready to accept connections", Ctrl-C
+
+# 6. Restore. Connect to the default 'postgres' DB so the dump can recreate
+#    the 'nextcloud' DB itself.
+docker exec -i nextcloud-db psql -U nextcloud -d postgres < /tmp/nc-pg_dumpall.sql
+
+# 7. Bring up the rest of the stack and verify the app end-to-end.
+docker compose up -d
+
+# 8. Once verified, remove the old data dir and the temp dump.
+rm -rf db.old /tmp/nc-pg_dumpall.sql
+```
+
+**Immich differences:**
+
+- Data dir is `${DB_DATA_LOCATION}` from `.env` (not under `docker/immich/`),
+  so step 3 is just `mv "$DB_DATA_LOCATION" "$DB_DATA_LOCATION.old"` -- the
+  dump at `docker/immich/db/pg_dumpall.sql` is in a separate dir and stays put.
+- DB container is `immich_postgres`, user is `${DB_USERNAME}` from `.env`.
+- Immich pins to a specific postgres image (`ghcr.io/immich-app/postgres`)
+  with a digest -- check Immich release notes before bumping; they coordinate
+  the upgrade, you usually just follow their version.

docker/diun/compose.yaml

@@ -11,6 +11,7 @@ services:
     volumes:
       - "./data:/data"
       - "./diun.yml:/diun.yml:ro"
+      - "./diun.list.yml:/diun.list.yml:ro"
       - "/var/run/docker.sock:/var/run/docker.sock"
     environment:
       - "TZ=America/New_York"

docker/diun/diun.list.yml

@@ -0,0 +1,9 @@
+# Base images for locally-built containers. The docker provider doesn't see
+# these because the built image has a local-only tag. ctn_names metadata is set
+# so the email subject still points at the compose dir that needs rebuilding.
+- name: nextcloud:latest
+  metadata:
+    ctn_names: nextcloud-base
+- name: githubyumao/mcsmanager-daemon:latest
+  metadata:
+    ctn_names: mcsmanager-daemon-base

docker/diun/diun.yml

@@ -1,6 +1,6 @@
 watch:
   workers: 20
-  schedule: "0 0 6 * * *"     # daily at 09:00; six-field cron (sec min hr dom mon dow)
+  schedule: "0 0 6 * * *"     # daily at 06:00; six-field cron (sec min hr dom mon dow)
   firstCheckNotif: false      # don't email on first run for every image
   jitter: 30s
 
@@ -8,6 +8,11 @@ providers:
   docker:
     watchByDefault: true      # watch every running container without needing labels
     watchStopped: false
+  file:
+    # Explicit watches for base images of locally-built containers (apps with a
+    # Dockerfile in their compose dir). The docker provider can't see these
+    # because the built image has a local-only tag with no registry digest.
+    filename: /diun.list.yml
 
 notif:
   mail:
@@ -19,4 +24,13 @@ notif:
     from: argento@fastmail.com
     to:
       - timothykim@fastmail.fm
+    # Subject leads with container name so the compose dir is obvious from the inbox
+    # (e.g. "nextcloud-redis" -> docker/nextcloud/). Default template uses the full
+    # canonical ref with @sha256 digest, which is unreadable.
+    templateTitle: '[{{ .Entry.Status }}] {{ .Entry.Metadata.ctn_names }} ({{ .Entry.Image.Path }}:{{ .Entry.Image.Tag }})'
+    templateBody: |
+      Container: {{ .Entry.Metadata.ctn_names }}
+      Image:     {{ .Entry.Image.Path }}:{{ .Entry.Image.Tag }}
+      {{ if .Entry.Image.HubLink }}Link:      {{ .Entry.Image.HubLink }}{{ end }}
+      Host:      {{ .Meta.Hostname }}
 

docker/minecraft/Dockerfile

@@ -1,9 +1,9 @@
 FROM githubyumao/mcsmanager-daemon:latest
 
 RUN apt-get update && \
-    apt-get install -y wget apt-transport-https && \
-    wget -qO - https://packages.adoptium.net/artifactory/api/gpg/key/public | apt-key add - && \
-    echo "deb https://packages.adoptium.net/artifactory/deb noble main" > /etc/apt/sources.list.d/adoptium.list && \
+    apt-get install -y wget apt-transport-https gpg && \
+    wget -qO - https://packages.adoptium.net/artifactory/api/gpg/key/public | gpg --dearmor -o /usr/share/keyrings/adoptium.gpg && \
+    echo "deb [signed-by=/usr/share/keyrings/adoptium.gpg] https://packages.adoptium.net/artifactory/deb noble main" > /etc/apt/sources.list.d/adoptium.list && \
     apt-get update && \
     apt-get install -y temurin-25-jre && \
     apt-get clean

scripts/disk-health-check.sh

@@ -1,20 +1,18 @@
 #!/bin/bash
-# server-health-check.sh — Argento health monitoring
-# Checks: SMART disk health, ZFS pool status, disk space
-# Outputs to stdout by default, or sends email with --email.
-# Use --error-only to suppress output when everything is healthy.
+# disk-health-check.sh — argento health monitoring
+# Checks SMART health (SATA and NVMe), ZFS pool state and error counters,
+# ZFS dataset quotas, and non-ZFS disk space.
+#
+# Output is plain text to stdout (ANSI-colored when stdout is a TTY),
+# or an HTML email to $MAILTO with --email.
 #
 # Usage:
-#   ./server-health-check.sh               # full report to stdout
-#   ./server-health-check.sh --error-only  # only print errors (silent if healthy)
-#   ./server-health-check.sh --email       # full report to email
-#   ./server-health-check.sh --error-only --email # email errors (no email if healthy)
+#   disk-health-check.sh                        # full report to stdout
+#   disk-health-check.sh --error-only           # problems only; silent if healthy
+#   disk-health-check.sh --email                # full report by email
+#   disk-health-check.sh --error-only --email   # email only if problems found
 #
-# Install:
-#   chmod +x /opt/server-health-check.sh
-#   crontab -e:
-#     0 6 * * * /opt/server-health-check.sh --error-only --email  # daily alert
-#     0 8 * * 1 /opt/server-health-check.sh --email               # weekly report
+# Cron on the server runs --error-only --email daily and --email weekly.
 #
 # Dependencies: smartmontools, zfs, msmtp (as sendmail)
 
@@ -23,6 +21,8 @@ set -uo pipefail
 MAILTO="timothykim@fastmail.fm"
 HOSTNAME=$(hostname)
 SPACE_THRESHOLD=85  # percent used — alert above this
+ZFS_CAP_THRESHOLD=80  # ZFS write performance degrades on nearly-full pools
+QUOTA_THRESHOLD=85  # percent of dataset quota — alert above this
 
 ERROR_ONLY=false
 USE_EMAIL=false
@@ -37,8 +37,97 @@ done
 PROBLEMS=""
 STATUS=""
 
+# HTML table rows for the email body, built alongside the plain-text STATUS.
+# Styles are inlined on every element because most mail clients strip
+# <style> blocks. nowrap keeps device paths and sizes from wrapping.
+TH='style="padding:2px 10px;text-align:left;border-bottom:1px solid #ccc;white-space:nowrap"'
+TD='style="padding:2px 10px;text-align:left;white-space:nowrap"'
+TDR='style="padding:2px 10px;text-align:right;white-space:nowrap"'
+TABLE='style="border-collapse:collapse;margin:4px 0 16px"'
+HTML_SATA=""
+HTML_NVME=""
+HTML_ZFS=""
+HTML_SPACE=""
+
+# Map a usage percentage to a hex color: green at 0%, amber at 50%,
+# red at 100%, linearly interpolated per RGB channel. Endpoints are the
+# same dark shades used by state_html so they stay readable on white.
+# Done in bash because email clients can't compute colors in CSS.
+usage_color() {
+    local p=$1 r g b
+    [ "$p" -ge 0 ] 2>/dev/null || { printf '#222222'; return; }
+    [ "$p" -gt 100 ] && p=100
+    if [ "$p" -le 50 ]; then
+        # 1a7f37 (green) -> 9a6700 (amber)
+        r=$(( 26 + (154 - 26) * p / 50 ))
+        g=$(( 127 + (103 - 127) * p / 50 ))
+        b=$(( 55 + (0 - 55) * p / 50 ))
+    else
+        # 9a6700 (amber) -> cf222e (red)
+        r=$(( 154 + (207 - 154) * (p - 50) / 50 ))
+        g=$(( 103 + (34 - 103) * (p - 50) / 50 ))
+        b=$(( 0 + (46 - 0) * (p - 50) / 50 ))
+    fi
+    printf '#%02x%02x%02x' "$r" "$g" "$b"
+}
+
+# Green for healthy, red for anything else. UNKNOWN is amber because it
+# means smartctl couldn't read the device, not that the device failed.
+state_html() {
+    case "$1" in
+        PASSED|ONLINE) printf '<span style="color:#1a7f37;font-weight:bold">%s</span>' "$1" ;;
+        UNKNOWN)       printf '<span style="color:#9a6700;font-weight:bold">%s</span>' "$1" ;;
+        *)             printf '<span style="color:#cf222e;font-weight:bold">%s</span>' "$1" ;;
+    esac
+}
+
+# Terminal colors (same palette as the HTML), only when stdout is a TTY
+# so piped/redirected output stays clean.
+COLOR=false
+C_BOLD=""
+C_RED=""
+C_RESET=""
+C_DIM=""
+if [ -t 1 ]; then
+    COLOR=true
+    C_BOLD=$'\e[1m'
+    C_DIM=$'\e[2m'
+    C_RED=$'\e[1;38;2;207;34;46m'
+    C_RESET=$'\e[0m'
+fi
+
+state_txt() {
+    if [ "$COLOR" = true ]; then
+        local c
+        case "$1" in
+            PASSED|ONLINE) c=$'\e[1;38;2;26;127;55m' ;;
+            UNKNOWN)       c=$'\e[1;38;2;154;103;0m' ;;
+            *)             c=$'\e[1;38;2;207;34;46m' ;;
+        esac
+        printf '%s%s%s' "$c" "$1" "$C_RESET"
+    else
+        printf '%s' "$1"
+    fi
+}
+
+# $1 = percentage (drives the gradient), $2 = text to print. Callers pad
+# $2 to its column width BEFORE calling — ANSI codes are invisible but
+# still count as characters, so padding after coloring breaks alignment.
+usage_txt() {
+    if [ "$COLOR" = true ]; then
+        local hex
+        hex=$(usage_color "$1")
+        printf '\e[38;2;%d;%d;%dm%s%s' \
+            "$((16#${hex:1:2}))" "$((16#${hex:3:2}))" "$((16#${hex:5:2}))" \
+            "$2" "$C_RESET"
+    else
+        printf '%s' "$2"
+    fi
+}
+
 # ─── SMART Disk Health (SATA/SAS) ───────────────────────────────
-STATUS+="SMART (SATA):\n"
+STATUS+="${C_BOLD}SMART (SATA):${C_RESET}\n"
+STATUS+="${C_DIM}$(printf '  %-13s %-24s %5s %7s  %s' Disk Model Temp Hours Health)${C_RESET}\n"
 for disk in /dev/sd?; do
     [ -b "$disk" ] || continue
 
@@ -50,11 +139,15 @@ for disk in /dev/sd?; do
     temp=$(smartctl -A "$disk" 2>/dev/null | awk '$1 == 194 || $1 == 190 {print $10; exit}') || true
     temp="${temp:-?}"
     hours=$(smartctl -A "$disk" 2>/dev/null | awk '$1 == 9 {print $10; exit}') || true
+    # Some Seagates report attribute 9 raw as "11170h+42m+51.807s" —
+    # keep only the whole hours.
+    hours=${hours%%[^0-9]*}
     hours="${hours:-?}"
     model=$(smartctl -i "$disk" 2>/dev/null | awk -F': +' '/Device Model|Model Number/{print $2; exit}') || true
     model="${model:-unknown}"
 
-    STATUS+="  $disk ($model): $health, ${temp}C, ${hours}h\n"
+    STATUS+=$(printf '  %-13s %-24s %5s %7s  ' "$disk" "$model" "${temp}C" "${hours}h")$(state_txt "$health")"\n"
+    HTML_SATA+="<tr><td $TD>$disk</td><td $TD>$model</td><td $TD>$(state_html "$health")</td><td $TDR>${temp}C</td><td $TDR>${hours}h</td></tr>"$'\n'
 
     if [ "$health" != "PASSED" ] && [ "$health" != "UNKNOWN" ]; then
         PROBLEMS+="[SMART] $disk: overall health check FAILED ($health)\n"
@@ -80,7 +173,8 @@ for disk in /dev/sd?; do
 done
 
 # ─── SMART Disk Health (NVMe) ───────────────────────────────────
-STATUS+="\nSMART (NVMe):\n"
+STATUS+="\n${C_BOLD}SMART (NVMe):${C_RESET}\n"
+STATUS+="${C_DIM}$(printf '  %-13s %-24s %5s %7s  %s' Disk Model Temp Hours Health)${C_RESET}\n"
 for disk in /dev/nvme[0-9]*; do
     # Match only controller devices (nvme0, nvme1), not namespaces (nvme0n1)
     [[ "$disk" =~ ^/dev/nvme[0-9]+$ ]] || continue
@@ -93,14 +187,18 @@ for disk in /dev/nvme[0-9]*; do
     health="${health:-UNKNOWN}"
     model=$(smartctl -i "$ns" 2>/dev/null | awk -F': +' '/Model Number/{print $2; exit}') || true
     model="${model:-unknown}"
+    # smartctl prints "38 Celsius" — keep just the number to match the SATA column
     temp=$(smartctl -A "$ns" 2>/dev/null | awk -F': +' '/Temperature:/{print $2; exit}') || true
+    temp=${temp%% *}
     temp="${temp:-?}"
-    hours=$(smartctl -A "$ns" 2>/dev/null | awk -F': +' '/Power On Hours/{print $2; exit}') || true
+    # strip thousands separator ("9,852")
+    hours=$(smartctl -A "$ns" 2>/dev/null | awk -F': +' '/Power On Hours/{print $2; exit}' | tr -d ',') || true
     hours="${hours:-?}"
     # NVMe percentage used — 100% means full rated write endurance consumed
     pct_used=$(smartctl -A "$ns" 2>/dev/null | awk -F': +' '/Percentage Used/{print $2; exit}' | tr -d '%') || true
 
-    STATUS+="  $ns ($model): $health, ${temp}, ${hours}h\n"
+    STATUS+=$(printf '  %-13s %-24s %5s %7s  ' "$ns" "$model" "${temp}C" "${hours}h")$(state_txt "$health")"\n"
+    HTML_NVME+="<tr><td $TD>$ns</td><td $TD>$model</td><td $TD>$(state_html "$health")</td><td $TDR>${temp}C</td><td $TDR>${hours}h</td></tr>"$'\n'
 
     if [ "$health" != "PASSED" ] && [ "$health" != "UNKNOWN" ]; then
         PROBLEMS+="[SMART] $ns: overall health check FAILED ($health)\n"
@@ -112,17 +210,30 @@ for disk in /dev/nvme[0-9]*; do
 done
 
 # ─── ZFS Pool Health ─────────────────────────────────────────────
-STATUS+="\nZFS Pools:\n"
+STATUS+="\n${C_BOLD}ZFS Pools:${C_RESET}\n"
+STATUS+="${C_DIM}$(printf '  %-12s %7s %7s %7s %5s  %s' Pool Size Alloc Free Cap State)${C_RESET}\n"
 while IFS= read -r pool; do
     [ -n "$pool" ] || continue
     state=$(zpool list -H -o health "$pool" 2>/dev/null) || true
+    state="${state:-UNKNOWN}"
     pool_info=$(zpool list -H -o name,size,alloc,free,cap "$pool" 2>/dev/null) || true
-    STATUS+="  $pool_info  $state\n"
+
+    # zpool -H output is tab-separated
+    IFS=$'\t' read -r p_name p_size p_alloc p_free p_cap <<< "$pool_info" || true
+    STATUS+=$(printf '  %-12s %7s %7s %7s ' "${p_name:-$pool}" "$p_size" "$p_alloc" "$p_free")$(usage_txt "${p_cap%\%}" "$(printf '%5s' "$p_cap")")"  "$(state_txt "$state")"\n"
+    cap_cell="<span style=\"color:$(usage_color "${p_cap%\%}")\">$p_cap</span>"
+    HTML_ZFS+="<tr><td $TD>${p_name:-$pool}</td><td $TDR>$p_size</td><td $TDR>$p_alloc</td><td $TDR>$p_free</td><td $TDR>$cap_cell</td><td $TD>$(state_html "$state")</td></tr>"$'\n'
 
     if [ "$state" != "ONLINE" ]; then
         PROBLEMS+="[ZFS] Pool '$pool' state: $state\n"
     fi
 
+    # Datasets are excluded from the df check below, so pool cap is the
+    # only capacity alert for ZFS data.
+    if [ "${p_cap%\%}" -gt "$ZFS_CAP_THRESHOLD" ] 2>/dev/null; then
+        PROBLEMS+="[ZFS] Pool '$pool' is ${p_cap} full\n"
+    fi
+
     # Check for errors via zpool status -p (parseable).
     # Parse the per-vdev lines: columns are NAME STATE READ WRITE CKSUM.
     # Flag any vdev that is not ONLINE, or has non-zero error counters.
@@ -145,8 +256,31 @@ while IFS= read -r pool; do
     done < <(zpool status -p "$pool" 2>/dev/null | awk '/NAME.*STATE.*READ/{found=1; next} found && /^[[:space:]]+[^ ]/{print} /^$/{found=0}' || true)
 done < <(zpool list -H -o name 2>/dev/null || true)
 
+# ─── ZFS Dataset Quotas ──────────────────────────────────────────
+# Quotas are hard limits — writes fail at 100% — so warn before the kids
+# hit the wall. Lists every dataset with a quota set (with -p, an unset
+# quota reads as 0).
+STATUS+="\n${C_BOLD}Quotas:${C_RESET}\n"
+STATUS+="${C_DIM}$(printf '  %-18s %4s %7s %7s' Dataset 'Use%' Used Quota)${C_RESET}\n"
+HTML_QUOTA=""
+while IFS=$'\t' read -r ds used quota; do
+    [ -n "$ds" ] || continue
+    [ "$quota" -gt 0 ] 2>/dev/null || continue
+    pct=$(( used * 100 / quota ))
+    used_h=$(numfmt --to=iec "$used" 2>/dev/null || echo "$used")
+    quota_h=$(numfmt --to=iec "$quota" 2>/dev/null || echo "$quota")
+    STATUS+=$(printf '  %-18s ' "$ds")$(usage_txt "$pct" "$(printf '%4s' "${pct}%")")$(printf ' %7s %7s' "$used_h" "$quota_h")"\n"
+    weight=""
+    if [ "$pct" -gt "$QUOTA_THRESHOLD" ]; then
+        PROBLEMS+="[QUOTA] $ds is at ${pct}% of its ${quota_h} quota\n"
+        weight=";font-weight:bold"
+    fi
+    HTML_QUOTA+="<tr><td $TD>$ds</td><td $TDR>$used_h</td><td $TDR>$quota_h</td><td $TDR><span style=\"color:$(usage_color "$pct")$weight\">${pct}%</span></td></tr>"$'\n'
+done < <(zfs list -Hp -o name,used,quota -t filesystem 2>/dev/null || true)
+
 # ─── Disk Space ──────────────────────────────────────────────────
-STATUS+="\nDisk Space:\n"
+STATUS+="\n${C_BOLD}Disk Space:${C_RESET}\n"
+STATUS+="${C_DIM}$(printf '  %-12s %4s %7s %7s %7s' Mount 'Use%' Used Size Avail)${C_RESET}\n"
 while IFS= read -r line; do
     [ -n "$line" ] || continue
     usage=$(echo "$line" | awk '{print $5}' | tr -d '%')
@@ -154,11 +288,17 @@ while IFS= read -r line; do
     size=$(echo "$line" | awk '{print $2}')
     used=$(echo "$line" | awk '{print $3}')
     avail=$(echo "$line" | awk '{print $4}')
-    STATUS+="  $mount: ${usage}% (${used}/${size}, ${avail} free)\n"
+    STATUS+=$(printf '  %-12s ' "$mount")$(usage_txt "$usage" "$(printf '%4s' "${usage}%")")$(printf ' %7s %7s %7s' "$used" "$size" "$avail")"\n"
+    weight=""
     if [ "$usage" -gt "$SPACE_THRESHOLD" ] 2>/dev/null; then
         PROBLEMS+="[SPACE] $mount is ${usage}% full\n"
+        weight=";font-weight:bold"
     fi
-done < <(df -h --output=source,size,used,avail,pcent,target -x tmpfs -x devtmpfs -x overlay -x efivarfs 2>/dev/null | tail -n +2 || true)
+    usage_cell="<span style=\"color:$(usage_color "$usage")$weight\">${usage}%</span>"
+    HTML_SPACE+="<tr><td $TD>$mount</td><td $TDR>$usage_cell</td><td $TDR>$used</td><td $TDR>$size</td><td $TDR>$avail</td></tr>"$'\n'
+# ZFS datasets are excluded: their df percentages are computed against
+# shared pool free space, so they're misleading — pool cap above covers them.
+done < <(df -h --output=source,size,used,avail,pcent,target -x tmpfs -x devtmpfs -x overlay -x efivarfs -x zfs 2>/dev/null | tail -n +2 || true)
 
 # ─── Output ──────────────────────────────────────────────────────
 if [ "$ERROR_ONLY" = true ]; then
@@ -166,7 +306,7 @@ if [ "$ERROR_ONLY" = true ]; then
         exit 0
     fi
     SUBJECT="[ALERT] $HOSTNAME health check failed"
-    BODY="Health check found issues on $HOSTNAME at $(date):\n\n$PROBLEMS\n---\n"
+    BODY="Health check found issues on $HOSTNAME at $(date):\n\n${C_RED}$PROBLEMS${C_RESET}\n---\n"
 else
     if [ -n "$PROBLEMS" ]; then
         SUBJECT="[REPORT] $HOSTNAME health — ISSUES FOUND"
@@ -175,18 +315,56 @@ else
     fi
     BODY="Health report for $HOSTNAME at $(date):\n\n"
     if [ -n "$PROBLEMS" ]; then
-        BODY+="*** ISSUES ***\n\n$PROBLEMS\n"
+        BODY+="${C_RED}*** ISSUES ***\n\n$PROBLEMS${C_RESET}\n"
     fi
     BODY+="--- Status ---\n\n$STATUS\n---\n"
 fi
 
 if [ "$USE_EMAIL" = true ]; then
+    HTML_PROBLEMS=""
+    while IFS= read -r p; do
+        [ -n "$p" ] || continue
+        HTML_PROBLEMS+="<li style=\"color:#cf222e;font-weight:bold;margin:2px 0\">$p</li>"$'\n'
+    done < <(printf '%b' "$PROBLEMS")
+
+    # Built with a newline after each row/section — SMTP caps lines at
+    # ~1000 chars and one giant line can get mangled in transit.
+    HTML_BODY="<div style=\"font-family:Arial,Helvetica,sans-serif;font-size:14px;color:#222\">"$'\n'
+    if [ "$ERROR_ONLY" = true ]; then
+        HTML_BODY+="<p>Health check found issues on <b>$HOSTNAME</b> at $(date):</p>"$'\n'
+        HTML_BODY+="<ul style=\"padding-left:20px\">"$'\n'"$HTML_PROBLEMS</ul>"$'\n'
+    else
+        HTML_BODY+="<p>Health report for <b>$HOSTNAME</b> at $(date):</p>"$'\n'
+        if [ -n "$HTML_PROBLEMS" ]; then
+            HTML_BODY+="<h3 style=\"color:#cf222e\">Issues</h3>"$'\n'
+            HTML_BODY+="<ul style=\"padding-left:20px\">"$'\n'"$HTML_PROBLEMS</ul>"$'\n'
+        fi
+        HTML_BODY+="<h3>SMART (SATA)</h3><table $TABLE>"$'\n'
+        HTML_BODY+="<tr><th $TH>Disk</th><th $TH>Model</th><th $TH>Health</th><th $TH>Temp</th><th $TH>Hours</th></tr>"$'\n'
+        HTML_BODY+="$HTML_SATA</table>"$'\n'
+        HTML_BODY+="<h3>SMART (NVMe)</h3><table $TABLE>"$'\n'
+        HTML_BODY+="<tr><th $TH>Disk</th><th $TH>Model</th><th $TH>Health</th><th $TH>Temp</th><th $TH>Hours</th></tr>"$'\n'
+        HTML_BODY+="$HTML_NVME</table>"$'\n'
+        HTML_BODY+="<h3>ZFS Pools</h3><table $TABLE>"$'\n'
+        HTML_BODY+="<tr><th $TH>Pool</th><th $TH>Size</th><th $TH>Alloc</th><th $TH>Free</th><th $TH>Cap</th><th $TH>State</th></tr>"$'\n'
+        HTML_BODY+="$HTML_ZFS</table>"$'\n'
+        HTML_BODY+="<h3>Quotas</h3><table $TABLE>"$'\n'
+        HTML_BODY+="<tr><th $TH>Dataset</th><th $TH>Used</th><th $TH>Quota</th><th $TH>Used %</th></tr>"$'\n'
+        HTML_BODY+="$HTML_QUOTA</table>"$'\n'
+        HTML_BODY+="<h3>Disk Space</h3><table $TABLE>"$'\n'
+        HTML_BODY+="<tr><th $TH>Mount</th><th $TH>Used %</th><th $TH>Used</th><th $TH>Size</th><th $TH>Free</th></tr>"$'\n'
+        HTML_BODY+="$HTML_SPACE</table>"$'\n'
+    fi
+    HTML_BODY+="</div>"
+
     {
         echo "Subject: $SUBJECT"
         echo "From: $HOSTNAME <argento@fastmail.com>"
         echo "To: $MAILTO"
+        echo "MIME-Version: 1.0"
+        echo "Content-Type: text/html; charset=utf-8"
         echo ""
-        printf '%b' "$BODY"
+        printf '%s\n' "$HTML_BODY"
     } | sendmail "$MAILTO"
 else
     printf '%b' "$BODY"

scripts/update-docker.sh

@@ -0,0 +1,241 @@
+#!/bin/bash
+# update-docker.sh — pull/build the latest images for every app under docker/,
+# recreate only the containers whose image changed, validate they come back
+# healthy, and run Nextcloud's post-upgrade occ steps. Prints a per-app summary
+# (updated / no update / failed) and exits non-zero if anything failed.
+#
+# In-place by design: it never runs a global `docker compose down`, because the
+# `shared` network is created by the nginx compose file and every other app
+# joins it as external. Tearing it all down at once would break that network and
+# cause needless downtime. `docker compose up -d` recreates only the changed
+# containers and leaves the rest (and the network) alone.
+#
+# Usage:
+#   ./update-docker.sh                 # update every app under docker/
+#   ./update-docker.sh nextcloud nginx # update only the named apps
+#   ./update-docker.sh --prune         # also reclaim dangling images afterward
+#
+# Env overrides:
+#   DOCKER_DIR=/path/to/docker   # default: ../docker relative to this script
+#   HEALTH_TIMEOUT=180           # seconds to wait for containers to be healthy
+#
+# Runs on the server (Rocky Linux 9), where each app's .env is present so compose
+# can resolve secrets. Run it from anywhere; it cd's into each app dir itself.
+
+set -uo pipefail
+
+SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+DOCKER_DIR="${DOCKER_DIR:-$(cd "$SCRIPT_DIR/.." && pwd)/docker}"
+HEALTH_TIMEOUT="${HEALTH_TIMEOUT:-180}"
+
+PRUNE=false
+REQUESTED=()
+for arg in "$@"; do
+    case "$arg" in
+        --prune) PRUNE=true ;;
+        -*) echo "unknown option: $arg" >&2; exit 2 ;;
+        *) REQUESTED+=("$arg") ;;
+    esac
+done
+
+# `docker compose pull --ignore-buildable` cleanly skips services with a build
+# section (added in Compose v2.30). Without it, pulling minecraft would error on
+# its local-only image tag — harmless here since we treat pull failures as
+# non-fatal and rebuild buildable services anyway, but the flag keeps output clean.
+PULL_FLAGS=""
+if docker compose pull --help 2>/dev/null | grep -q -- '--ignore-buildable'; then
+    PULL_FLAGS="--ignore-buildable"
+fi
+
+UPDATED=()
+NOUPDATE=()
+FAILED=()
+
+# Stable "ref=imageID" listing of every image the project resolves to, so we can
+# tell whether a pull/build actually changed anything.
+image_signature() {
+    local img id
+    while IFS= read -r img; do
+        [ -z "$img" ] && continue
+        id=$(docker image inspect -f '{{.Id}}' "$img" 2>/dev/null || echo '<none>')
+        printf '%s=%s\n' "$img" "$id"
+    done < <(docker compose config --images 2>/dev/null) | sort
+}
+
+compose_has_build() {
+    docker compose config 2>/dev/null | grep -qE '^[[:space:]]+build:'
+}
+
+# Wait until every container in the project is running, and (if it has a
+# healthcheck) healthy. Returns non-zero on timeout.
+validate_app() {
+    local deadline cid cname state health all_ok pending
+    deadline=$(( SECONDS + HEALTH_TIMEOUT ))
+    while :; do
+        all_ok=true
+        pending=""
+        while IFS= read -r cid; do
+            [ -z "$cid" ] && continue
+            cname=$(docker inspect -f '{{.Name}}' "$cid" 2>/dev/null | sed 's#^/##')
+            state=$(docker inspect -f '{{.State.Status}}' "$cid" 2>/dev/null)
+            health=$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$cid" 2>/dev/null)
+            if [ "$state" != "running" ]; then
+                all_ok=false; pending+=" $cname=$state"
+            elif [ "$health" = "starting" ] || [ "$health" = "unhealthy" ]; then
+                all_ok=false; pending+=" $cname=health:$health"
+            fi
+        done < <(docker compose ps -a -q)
+
+        $all_ok && return 0
+        if [ "$SECONDS" -ge "$deadline" ]; then
+            echo "    validation timed out after ${HEALTH_TIMEOUT}s:$pending"
+            return 1
+        fi
+        sleep 5
+    done
+}
+
+# Nextcloud post-upgrade steps.
+#
+# The official image's entrypoint already runs `occ upgrade` on start when it
+# detects a version change, and holds the instance in maintenance mode while it
+# does (https://github.com/nextcloud/docker README). The old upgrade.sh failed
+# because it ran `occ upgrade` after a fixed `sleep 10` — i.e. while the
+# entrypoint was still mid-upgrade. So here we first wait for maintenance mode to
+# clear, then run occ ourselves.
+#
+# Exit codes from core/Command/Upgrade.php (current master): 0 = success AND
+# "no upgrade required" (ERROR_UP_TO_DATE == 0), 2 = maintenance mode,
+# 5 = failure. So a non-zero from `occ upgrade` is a genuine problem, not the
+# benign "already up to date" it used to be in older versions (which returned 3).
+nextcloud_occ() {
+    local rc=0 deadline
+    deadline=$(( SECONDS + HEALTH_TIMEOUT ))
+    echo "    occ: waiting for the entrypoint's auto-upgrade / maintenance mode to clear"
+    while docker compose exec -T -u www-data nextcloud php occ maintenance:mode 2>/dev/null \
+            | grep -q 'currently enabled'; do
+        if [ "$SECONDS" -ge "$deadline" ]; then
+            echo "    still in maintenance mode after ${HEALTH_TIMEOUT}s — aborting occ steps"
+            return 1
+        fi
+        sleep 5
+    done
+
+    # Safety-net upgrade. Normally a no-op (the entrypoint already migrated), so
+    # this should print "No upgrade required" and return 0. A non-zero exit is real.
+    echo "    occ: upgrade"
+    docker compose exec -T -u www-data nextcloud php occ upgrade 2>&1 | sed 's/^/      /'
+    local up=${PIPESTATUS[0]}
+    if [ "$up" -ne 0 ]; then
+        echo "    occ upgrade exited $up (2=maintenance, 5=failure) — see output above"
+        rc=1
+    fi
+
+    # Documented post-upgrade index optimization; idempotent and safe to repeat.
+    echo "    occ: add missing DB indices"
+    docker compose exec -T -u www-data nextcloud php occ db:add-missing-indices 2>&1 | sed 's/^/      /'
+    [ "${PIPESTATUS[0]}" -ne 0 ] && rc=1
+
+    echo "    occ: status"
+    docker compose exec -T -u www-data nextcloud php occ status 2>&1 | sed 's/^/      /'
+    [ "${PIPESTATUS[0]}" -ne 0 ] && rc=1
+
+    return $rc
+}
+
+update_app() {
+    local name="$1"
+    local dir="$DOCKER_DIR/$name"
+
+    if ! cd "$dir" 2>/dev/null; then
+        FAILED+=("$name (cannot cd into $dir)"); return
+    fi
+
+    echo "==> $name: checking for updates"
+    local before after
+    before=$(image_signature)
+
+    # Pull registry images. Lenient: a warning/error here (e.g. an image that is
+    # only built locally) shouldn't abort the run — validation catches real breakage.
+    docker compose pull $PULL_FLAGS 2>&1 | sed 's/^/    /'
+
+    if compose_has_build; then
+        echo "    rebuilding local images against latest base"
+        docker compose build --pull 2>&1 | sed 's/^/    /'
+        if [ "${PIPESTATUS[0]}" -ne 0 ]; then
+            FAILED+=("$name (build failed)"); return
+        fi
+    fi
+
+    after=$(image_signature)
+    if [ "$before" = "$after" ]; then
+        echo "    no update needed"
+        NOUPDATE+=("$name"); return
+    fi
+
+    echo "    new image(s) detected — recreating containers"
+    docker compose up -d 2>&1 | sed 's/^/    /'
+    if [ "${PIPESTATUS[0]}" -ne 0 ]; then
+        FAILED+=("$name (up -d failed)"); return
+    fi
+
+    if ! validate_app; then
+        FAILED+=("$name (unhealthy after update)"); return
+    fi
+
+    if [ "$name" = "nextcloud" ]; then
+        if ! nextcloud_occ; then
+            FAILED+=("$name (occ steps failed)"); return
+        fi
+    fi
+
+    echo "    updated and validated"
+    UPDATED+=("$name")
+}
+
+# Build the work list: every dir under docker/ that has a compose file, with
+# nginx first so the shared network owner is touched before its dependents.
+all_apps=()
+while IFS= read -r d; do
+    name=$(basename "$d")
+    if compgen -G "$d/compose.y*ml" >/dev/null || compgen -G "$d/docker-compose.y*ml" >/dev/null; then
+        all_apps+=("$name")
+    fi
+done < <(find "$DOCKER_DIR" -mindepth 1 -maxdepth 1 -type d | sort)
+
+if [ "${#REQUESTED[@]}" -gt 0 ]; then
+    # Validate requested names against what actually exists.
+    apps=()
+    for req in "${REQUESTED[@]}"; do
+        found=false
+        for a in "${all_apps[@]}"; do [ "$a" = "$req" ] && found=true; done
+        if $found; then apps+=("$req"); else
+            echo "no such app under docker/: $req" >&2; exit 2
+        fi
+    done
+else
+    apps=("${all_apps[@]}")
+fi
+
+# nginx first, the rest after.
+ordered=()
+for a in "${apps[@]}"; do [ "$a" = nginx ] && ordered+=("nginx"); done
+for a in "${apps[@]}"; do [ "$a" != nginx ] && ordered+=("$a"); done
+
+for name in "${ordered[@]}"; do
+    update_app "$name"
+done
+
+echo
+echo "================ summary ================"
+echo "updated:    ${UPDATED[*]:-none}"
+echo "no update:  ${NOUPDATE[*]:-none}"
+echo "failed:     ${FAILED[*]:-none}"
+
+if $PRUNE && [ "${#UPDATED[@]}" -gt 0 ]; then
+    echo
+    echo "pruning dangling images"
+    docker image prune -f 2>&1 | sed 's/^/    /'
+fi
+
+[ "${#FAILED[@]}" -eq 0 ]