argento/RUNBOOK.md

Argento Recovery Runbook

Full top-to-bottom rebuild. Follow these steps in order.

1. Install Rocky Linux 9

Standard minimal install on NVMe.

2. Repos and base packages

# EPEL
dnf install -y epel-release

# Docker CE
dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo

# ZFS — must use zfs-testing repo (stable repo's version can't import the existing pools)
dnf install -y https://zfsonlinux.org/epel/zfs-release-2-3$(rpm --eval "%{dist}").noarch.rpm
dnf config-manager --enable zfs-testing
dnf install -y kernel-devel zfs  # uses DKMS — kernel-devel is required

# NVIDIA (CUDA repo — provides nvidia-driver and container toolkit)
dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo

# Install everything
# Sanoid (not in EPEL — COPR repo, see https://github.com/jimsalterjrs/sanoid for manual install)
dnf copr enable -y orrisroot/zfs

dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin docker-buildx-plugin \
    certbot msmtp smartmontools \
    jq git sanoid samba \
    nvidia-driver nvidia-container-toolkit \
    cockpit

# Load ZFS kernel module
modprobe zfs

# Configure NVIDIA container runtime for Docker
nvidia-ctk runtime configure --runtime=docker

# Set up msmtp as sendmail replacement (for cron MAILTO and smartd alerts)
ln -sf /usr/bin/msmtp /usr/sbin/sendmail

3. Enable services

systemctl enable --now docker smb firewalld cockpit.socket sshd smartd
systemctl enable --now zfs-import-cache zfs-mount zfs-share zfs-zed
systemctl enable --now sanoid.timer

4. Firewall

cp /opt/argento/system/firewalld/public.xml /etc/firewalld/zones/public.xml
firewall-cmd --reload

5. Mount USB backup drive

echo 'LABEL=argento-backup /mnt/backup ext4 defaults,nofail 0 2' >> /etc/fstab
mkdir -p /mnt/backup
mount /mnt/backup

6. Get the repo

# Clone from the Gitea bare repo on the USB backup
git clone /mnt/backup/argento/docker/gitea/data/git/repositories/timothykim/argento.git /opt/argento

Clone first -- ZFS datasets will mount into this directory tree in the next step.

7. ZFS pools

If ZFS disks are intact (OS drive failure only) -- the common case:

# ZFS stores pool metadata on the disks — import auto-detects them
# Run "zpool import" (no args) first to list available pools
zpool import nextcloud
zpool import threeteras

# Verify mountpoints (update if the old /opt/docker/ paths are still set):
zfs set mountpoint=/opt/argento/docker/nextcloud/data nextcloud/data
zfs set mountpoint=/opt/argento/smb/media threeteras/media
zfs set mountpoint=/opt/argento/smb/yireh threeteras/yireh
zfs set mountpoint=/opt/argento/smb/isaac threeteras/isaac
zfs set mountpoint=/opt/argento/docker/garage/data threeteras/garage
zfs set mountpoint=/opt/argento/docker/immich/library threeteras/immich

zpool status  # verify pools are ONLINE

If ZFS disks are lost (full rebuild):

Important: The pools use feature flags that require ZFS from the zfs-testing repo (step 2). If zpool import fails with a feature flag error, verify zfs-testing is enabled and you have ZFS 2.4.0+.

# Find disk IDs (use these instead of /dev/sdX — they're stable across reboots):
ls -l /dev/disk/by-id/ | grep -v part

# Nextcloud pool — mirror of 2x 7.3TB drives
zpool create nextcloud mirror /dev/disk/by-id/<disk1> /dev/disk/by-id/<disk2>
zfs set compression=lz4 nextcloud
zfs create -o mountpoint=/opt/argento/docker/nextcloud/data nextcloud/data

# Threeteras pool — raidz2 of 6x 2.7TB drives
zpool create threeteras raidz2 /dev/disk/by-id/<disk3> ... /dev/disk/by-id/<disk8>
zfs set compression=lz4 threeteras
zfs create -o mountpoint=/opt/argento/smb/media threeteras/media
zfs create -o mountpoint=/opt/argento/smb/yireh -o quota=1T threeteras/yireh
zfs create -o mountpoint=/opt/argento/smb/isaac -o quota=1T threeteras/isaac
zfs create -o mountpoint=/opt/argento/docker/garage/data threeteras/garage
zfs create -o mountpoint=/opt/argento/docker/immich/library threeteras/immich

zpool status  # verify pools are ONLINE

# Shared music dir. Lives inside the media dataset (so it rides the media
# snapshot policy) but is exported as its own writable [music] share.
# Ownership is set in step 7b — sambagroup doesn't exist yet.
mkdir -p /opt/argento/smb/media/Music

# SELinux contexts for samba — fresh ZFS datasets get unlabeled_t and samba
# refuses to serve them. Persistent rules (semanage) survive full relabels;
# restorecon applies them now.
semanage fcontext -a -t samba_share_t "/opt/argento/smb/media(/.*)?"
semanage fcontext -a -t samba_share_t "/opt/argento/smb/yireh(/.*)?"
semanage fcontext -a -t samba_share_t "/opt/argento/smb/isaac(/.*)?"
restorecon -Rv /opt/argento/smb/

7b. Restore users

System users and the samba passdb are not part of the OS install or the /opt/argento/ rsync. Recreate them now, BEFORE the data rsync in step 7c, so file ownership lands on the correct UIDs.

/opt/argento/scripts/restore-users.sh

# Share dirs are root-owned on freshly created datasets. These must run AFTER
# restore-users.sh — the users and sambagroup don't exist before it. Skip if
# the datasets were imported intact; step 7c's rsync preserves ownership.
for u in yireh isaac; do
    chown "$u":sambagroup "/opt/argento/smb/$u"
    chmod 2770 "/opt/argento/smb/$u"
done

# No chmod on Music: threeteras/media is acltype=nfsv4 aclmode=restricted, so
# chmod fails with EPERM. aclinherit=passthrough already copied the parent's
# ACL onto the dir, which is what grants sambagroup write. The other datasets
# are acltype=posix aclmode=discard, so chmod works normally there.
chown timothykim:sambagroup /opt/argento/smb/media/Music

This recreates timothykim, yireh, isaac, hannah, and the sambagroup group with their original UIDs/GIDs, then restores the samba password hashes from /mnt/backup/argento/system/samba-private/ if present (mirrored there by sync-system.sh, gitignored so the hashes aren't in version control).

If the samba passdb backup is missing, set passwords manually:

for u in timothykim yireh isaac hannah; do
    smbpasswd -a "$u"
    smbpasswd -e "$u"
done

Set timothykim's Linux login password (not in any backup):

passwd timothykim

7c. Restore data from USB

If ZFS disks were intact:

# Skip the big ZFS-backed dirs — that data is already on disk
rsync -a --exclude='docker/nextcloud/data/' --exclude='docker/garage/data/' \
    --exclude='docker/immich/library/' --exclude='smb/' \
    /mnt/backup/argento/ /opt/argento/

If ZFS was rebuilt from scratch:

# Restore everything (ZFS mountpoints are set, data lands in the right place)
rsync -a /mnt/backup/argento/ /opt/argento/

8. Restore .env files

Secrets are not in the repo -- they live in .env files on the server. Restore from USB backup:

# .env files are backed up alongside app data
# After the rsync in step 7, they should already be in place. Verify:
ls /opt/argento/docker/*/.env

# If USB backup is unavailable (full from-scratch rebuild),
# recreate .env files manually from Bitwarden.
# Each app's compose.yaml or runner-config.yaml references its .env file.

9. System configs

cp /opt/argento/system/docker-daemon.json /etc/docker/daemon.json
cp /opt/argento/system/smb.conf /etc/samba/smb.conf
cp /opt/argento/system/sanoid.conf /etc/sanoid/sanoid.conf
cp /opt/argento/system/msmtprc /etc/msmtprc
chmod 600 /etc/msmtprc
cp /opt/argento/system/smartd.conf /etc/smartd.conf
systemctl restart docker smb smartd
crontab /opt/argento/system/root-crontab

10. Nginx + SSL

# Create shared network
docker network create shared

# Move SSL confs aside (nginx can't load them — cert files don't exist yet)
# HTTP-only confs (minecraft) and non-SSL files (.inc, .example) can stay
cd /opt/argento/docker/nginx
mkdir -p conf.d/.hold certbot/www
for f in conf.d/*.conf; do
    grep -q ssl_certificate "$f" && mv "$f" conf.d/.hold/
done

# Start nginx (only HTTP-only confs and the default catch-all are loaded)
docker compose up -d

# Issue certs (add any new domains that have been added since this was written)
/opt/argento/scripts/issue-cert.sh \
    nextcloud.timothykim.net git.timothykim.net jellyfin.timothykim.net \
    argento.timothykim.net garage.timothykim.net photos.thekims.family \
    music.thekims.family

# Restore SSL confs now that certs exist
mv conf.d/.hold/*.conf conf.d/
rmdir conf.d/.hold
docker exec nginx nginx -t && docker exec nginx nginx -s reload

# Set up cert renewal
mkdir -p /etc/letsencrypt/renewal-hooks/deploy
cat > /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh <<'HOOK'
#!/bin/bash
docker exec nginx nginx -s reload
HOOK
chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
systemctl enable --now certbot-renew.timer

# Verify renewal works
certbot renew --dry-run

11. Docker apps

Deploy each app (nginx is already running and will return 502 until they start):

# Add any new apps that have been added since this was written
for app in nextcloud gitea jellyfin navidrome garage minecraft immich beszel-agent diun; do
    cd /opt/argento/docker/$app
    docker compose pull
    docker compose build
    docker compose up -d
    echo "$app started"
done
docker ps  # verify all containers are healthy

12. Router

Set up port forwarding on the home router:

• 80 -> 192.168.2.122:80
• 443 -> 192.168.2.122:443
• 222 -> 192.168.2.122:222 (Gitea SSH)
• 3901 -> 192.168.2.122:3901 (Garage RPC)
• 25500-25600 -> 192.168.2.122:25500-25600 (Minecraft)

13. DNS

CNAME records pointing to the DDNS hostname. These don't need updating when the IP changes -- only the DDNS record does.

Record	Type	Value
nextcloud.timothykim.net	CNAME	argento.ddns.net
git.timothykim.net	CNAME	argento.ddns.net
jellyfin.timothykim.net	CNAME	argento.ddns.net
argento.timothykim.net	CNAME	argento.ddns.net
garage.timothykim.net	CNAME	argento.ddns.net
minecraft.thekims.family	CNAME	argento.ddns.net
photos.thekims.family	CNAME	argento.ddns.net
music.thekims.family	CNAME	argento.ddns.net

14. Verify

curl -I https://nextcloud.timothykim.net
curl -I https://git.timothykim.net
curl -I https://jellyfin.timothykim.net
curl -I https://argento.timothykim.net
curl -I https://garage.timothykim.net
curl -I https://photos.thekims.family
curl -I https://music.thekims.family
curl -I http://minecraft.thekims.family

# Verify SMART, ZFS, and disk space are healthy
/opt/argento/scripts/disk-health-check.sh

---

Maintenance: Postgres major version upgrade

A postgres major bump (e.g. 17 -> 18) cannot reuse the old data dir -- the on-disk format changes. Re-init and reload from pg_dumpall. backup.sh already produces a daily dump for both Nextcloud and Immich.

Procedure (Nextcloud; Immich notes at the end):

cd /opt/argento/docker/nextcloud

# 1. Force a fresh dump so it reflects current state, not yesterday's.
/opt/argento/scripts/backup.sh

# 2. Stop the stack. Nothing will write to the DB after this.
docker compose down

# 3. Move the dump out of the data dir before nuking it.
cp db/pg_dumpall.sql /tmp/nc-pg_dumpall.sql
mv db db.old

# 4. Bump the postgres image tag in compose.yaml (e.g. 17-alpine -> 18-alpine).

# 5. Start only postgres so it initializes a fresh data dir, then wait for it.
docker compose up -d postgres
docker compose logs -f postgres   # wait for "ready to accept connections", Ctrl-C

# 6. Restore. Connect to the default 'postgres' DB so the dump can recreate
#    the 'nextcloud' DB itself.
docker exec -i nextcloud-db psql -U nextcloud -d postgres < /tmp/nc-pg_dumpall.sql

# 7. Bring up the rest of the stack and verify the app end-to-end.
docker compose up -d

# 8. Once verified, remove the old data dir and the temp dump.
rm -rf db.old /tmp/nc-pg_dumpall.sql

Immich differences:

• Data dir is ${DB_DATA_LOCATION} from .env (not under docker/immich/), so step 3 is just mv "$DB_DATA_LOCATION" "$DB_DATA_LOCATION.old" -- the dump at docker/immich/db/pg_dumpall.sql is in a separate dir and stays put.
• DB container is immich_postgres, user is ${DB_USERNAME} from .env.
• Immich pins to a specific postgres image (ghcr.io/immich-app/postgres) with a digest -- check Immich release notes before bumping; they coordinate the upgrade, you usually just follow their version.