# Argento Recovery Runbook

Full top-to-bottom rebuild. Follow these steps in order.

## 1. Install Rocky Linux 9

Standard minimal install on NVMe.

## 2. Repos and base packages

```bash
# EPEL
dnf install -y epel-release

# Docker CE
dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo

# ZFS — must use zfs-testing repo (stable repo's version can't import the existing pools)
dnf install -y https://zfsonlinux.org/epel/zfs-release-2-3$(rpm --eval "%{dist}").noarch.rpm
dnf config-manager --enable zfs-testing
dnf install -y kernel-devel zfs  # uses DKMS — kernel-devel is required

# NVIDIA (CUDA repo — provides nvidia-driver and container toolkit)
dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo

# Install everything
# Sanoid (not in EPEL — COPR repo, see https://github.com/jimsalterjrs/sanoid for manual install)
dnf copr enable -y orrisroot/zfs

dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin docker-buildx-plugin \
    certbot msmtp smartmontools \
    jq git sanoid samba \
    nvidia-driver nvidia-container-toolkit \
    cockpit

# Load ZFS kernel module
modprobe zfs

# Configure NVIDIA container runtime for Docker
nvidia-ctk runtime configure --runtime=docker

# Set up msmtp as sendmail replacement (for cron MAILTO and smartd alerts)
ln -sf /usr/bin/msmtp /usr/sbin/sendmail
```

## 3. Enable services

```bash
systemctl enable --now docker smb firewalld cockpit.socket sshd smartd
systemctl enable --now zfs-import-cache zfs-mount zfs-share zfs-zed
systemctl enable --now sanoid.timer
```

## 4. Firewall

```bash
cp /opt/argento/system/firewalld/public.xml /etc/firewalld/zones/public.xml
firewall-cmd --reload
```

## 5. Mount USB backup drive

```bash
echo 'LABEL=argento-backup /mnt/backup ext4 defaults,nofail 0 2' >> /etc/fstab
mkdir -p /mnt/backup
mount /mnt/backup
```

## 6. Get the repo

```bash
# Clone from the Gitea bare repo on the USB backup
git clone /mnt/backup/argento/docker/gitea/data/git/repositories/timothykim/argento.git /opt/argento
```

Clone first -- ZFS datasets will mount into this directory tree in the next step.

## 7. ZFS pools

**If ZFS disks are intact (OS drive failure only)** -- the common case:

```bash
# ZFS stores pool metadata on the disks — import auto-detects them
# Run "zpool import" (no args) first to list available pools
zpool import nextcloud
zpool import threeteras

# Verify mountpoints (update if the old /opt/docker/ paths are still set):
zfs set mountpoint=/opt/argento/docker/nextcloud/data nextcloud/data
zfs set mountpoint=/opt/argento/smb/media threeteras/media
zfs set mountpoint=/opt/argento/smb/yireh threeteras/yireh
zfs set mountpoint=/opt/argento/smb/isaac threeteras/isaac
zfs set mountpoint=/opt/argento/docker/garage/data threeteras/garage
zfs set mountpoint=/opt/argento/docker/immich/library threeteras/immich

zpool status  # verify pools are ONLINE
```

**If ZFS disks are lost (full rebuild):**

**Important:** The pools use feature flags that require ZFS from the `zfs-testing` repo (step 2). If `zpool import` fails with a feature flag error, verify `zfs-testing` is enabled and you have ZFS 2.4.0+.

```bash
# Find disk IDs (use these instead of /dev/sdX — they're stable across reboots):
ls -l /dev/disk/by-id/ | grep -v part

# Nextcloud pool — mirror of 2x 7.3TB drives
zpool create nextcloud mirror /dev/disk/by-id/<disk1> /dev/disk/by-id/<disk2>
zfs set compression=lz4 nextcloud
zfs create -o mountpoint=/opt/argento/docker/nextcloud/data nextcloud/data

# Threeteras pool — raidz2 of 6x 2.7TB drives
zpool create threeteras raidz2 /dev/disk/by-id/<disk3> ... /dev/disk/by-id/<disk8>
zfs set compression=lz4 threeteras
zfs create -o mountpoint=/opt/argento/smb/media threeteras/media
zfs create -o mountpoint=/opt/argento/smb/yireh -o quota=1T threeteras/yireh
zfs create -o mountpoint=/opt/argento/smb/isaac -o quota=1T threeteras/isaac
zfs create -o mountpoint=/opt/argento/docker/garage/data threeteras/garage
zfs create -o mountpoint=/opt/argento/docker/immich/library threeteras/immich

zpool status  # verify pools are ONLINE

# Per-user SMB share dirs need ownership + setgid (rsync in step 7c will
# preserve these if a USB backup exists, but freshly-created datasets are
# root-owned and need this fixup).
for u in yireh isaac; do
    chown "$u":sambagroup "/opt/argento/smb/$u"
    chmod 2770 "/opt/argento/smb/$u"
done

# SELinux contexts for samba — fresh ZFS datasets get unlabeled_t and samba
# refuses to serve them. Persistent rules (semanage) survive full relabels;
# restorecon applies them now.
semanage fcontext -a -t samba_share_t "/opt/argento/smb/media(/.*)?"
semanage fcontext -a -t samba_share_t "/opt/argento/smb/yireh(/.*)?"
semanage fcontext -a -t samba_share_t "/opt/argento/smb/isaac(/.*)?"
restorecon -Rv /opt/argento/smb/
```

## 7b. Restore users

System users and the samba passdb are not part of the OS install or the
`/opt/argento/` rsync. Recreate them now, BEFORE the data rsync in step 7c,
so file ownership lands on the correct UIDs.

```bash
/opt/argento/scripts/restore-users.sh
```

This recreates timothykim, yireh, isaac, and the sambagroup group with
their original UIDs/GIDs, then restores the samba password hashes from
`/mnt/backup/argento/system/samba-private/` if present (mirrored there
by sync-system.sh, gitignored so the hashes aren't in version control).

If the samba passdb backup is missing, set passwords manually:

```bash
for u in timothykim yireh isaac; do
    smbpasswd -a "$u"
    smbpasswd -e "$u"
done
```

Set timothykim's Linux login password (not in any backup):

```bash
passwd timothykim
```

## 7c. Restore data from USB

**If ZFS disks were intact:**

```bash
# Skip the big ZFS-backed dirs — that data is already on disk
rsync -a --exclude='docker/nextcloud/data/' --exclude='docker/garage/data/' \
    --exclude='docker/immich/library/' --exclude='smb/' \
    /mnt/backup/argento/ /opt/argento/
```

**If ZFS was rebuilt from scratch:**

```bash
# Restore everything (ZFS mountpoints are set, data lands in the right place)
rsync -a /mnt/backup/argento/ /opt/argento/
```

## 8. Restore .env files

Secrets are not in the repo -- they live in `.env` files on the server. Restore from USB backup:

```bash
# .env files are backed up alongside app data
# After the rsync in step 7, they should already be in place. Verify:
ls /opt/argento/docker/*/.env

# If USB backup is unavailable (full from-scratch rebuild),
# recreate .env files manually from Bitwarden.
# Each app's compose.yaml or runner-config.yaml references its .env file.
```

## 9. System configs

```bash
cp /opt/argento/system/docker-daemon.json /etc/docker/daemon.json
cp /opt/argento/system/smb.conf /etc/samba/smb.conf
cp /opt/argento/system/sanoid.conf /etc/sanoid/sanoid.conf
cp /opt/argento/system/msmtprc /etc/msmtprc
chmod 600 /etc/msmtprc
cp /opt/argento/system/smartd.conf /etc/smartd.conf
systemctl restart docker smb smartd
crontab /opt/argento/system/root-crontab
```

## 10. Nginx + SSL

```bash
# Create shared network
docker network create shared

# Move SSL confs aside (nginx can't load them — cert files don't exist yet)
# HTTP-only confs (minecraft) and non-SSL files (.inc, .example) can stay
cd /opt/argento/docker/nginx
mkdir -p conf.d/.hold certbot/www
for f in conf.d/*.conf; do
    grep -q ssl_certificate "$f" && mv "$f" conf.d/.hold/
done

# Start nginx (only HTTP-only confs and the default catch-all are loaded)
docker compose up -d

# Issue certs (add any new domains that have been added since this was written)
/opt/argento/scripts/issue-cert.sh \
    nextcloud.timothykim.net git.timothykim.net jellyfin.timothykim.net \
    argento.timothykim.net garage.timothykim.net photos.thekims.family

# Restore SSL confs now that certs exist
mv conf.d/.hold/*.conf conf.d/
rmdir conf.d/.hold
docker exec nginx nginx -t && docker exec nginx nginx -s reload

# Set up cert renewal
mkdir -p /etc/letsencrypt/renewal-hooks/deploy
cat > /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh <<'HOOK'
#!/bin/bash
docker exec nginx nginx -s reload
HOOK
chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
systemctl enable --now certbot-renew.timer

# Verify renewal works
certbot renew --dry-run
```

## 11. Docker apps

Deploy each app (nginx is already running and will return 502 until they start):

```bash
# Add any new apps that have been added since this was written
for app in nextcloud gitea jellyfin garage minecraft immich beszel-agent; do
    cd /opt/argento/docker/$app
    docker compose pull
    docker compose build
    docker compose up -d
    echo "$app started"
done
docker ps  # verify all containers are healthy
```

## 12. Router

Set up port forwarding on the home router:

- 80 -> 192.168.2.122:80
- 443 -> 192.168.2.122:443
- 222 -> 192.168.2.122:222 (Gitea SSH)
- 3901 -> 192.168.2.122:3901 (Garage RPC)
- 25500-25600 -> 192.168.2.122:25500-25600 (Minecraft)

## 13. DNS

CNAME records pointing to the DDNS hostname. These don't need updating when the IP changes -- only the DDNS record does.

| Record | Type | Value |
|--------|------|-------|
| `nextcloud.timothykim.net` | CNAME | `argento.ddns.net` |
| `git.timothykim.net` | CNAME | `argento.ddns.net` |
| `jellyfin.timothykim.net` | CNAME | `argento.ddns.net` |
| `argento.timothykim.net` | CNAME | `argento.ddns.net` |
| `garage.timothykim.net` | CNAME | `argento.ddns.net` |
| `minecraft.thekims.family` | CNAME | `argento.ddns.net` |
| `photos.thekims.family` | CNAME | `argento.ddns.net` |

## 14. Verify

```bash
curl -I https://nextcloud.timothykim.net
curl -I https://git.timothykim.net
curl -I https://jellyfin.timothykim.net
curl -I https://argento.timothykim.net
curl -I https://garage.timothykim.net
curl -I https://photos.thekims.family
curl -I http://minecraft.thekims.family

# Verify SMART, ZFS, and disk space are healthy
/opt/argento/scripts/disk-health-check.sh
```

---

## Maintenance: Postgres major version upgrade

A postgres major bump (e.g. 17 -> 18) cannot reuse the old data dir -- the
on-disk format changes. Re-init and reload from `pg_dumpall`. `backup.sh`
already produces a daily dump for both Nextcloud and Immich.

Procedure (Nextcloud; Immich notes at the end):

```bash
cd /opt/argento/docker/nextcloud

# 1. Force a fresh dump so it reflects current state, not yesterday's.
/opt/argento/scripts/backup.sh

# 2. Stop the stack. Nothing will write to the DB after this.
docker compose down

# 3. Move the dump out of the data dir before nuking it.
cp db/pg_dumpall.sql /tmp/nc-pg_dumpall.sql
mv db db.old

# 4. Bump the postgres image tag in compose.yaml (e.g. 17-alpine -> 18-alpine).

# 5. Start only postgres so it initializes a fresh data dir, then wait for it.
docker compose up -d postgres
docker compose logs -f postgres   # wait for "ready to accept connections", Ctrl-C

# 6. Restore. Connect to the default 'postgres' DB so the dump can recreate
#    the 'nextcloud' DB itself.
docker exec -i nextcloud-db psql -U nextcloud -d postgres < /tmp/nc-pg_dumpall.sql

# 7. Bring up the rest of the stack and verify the app end-to-end.
docker compose up -d

# 8. Once verified, remove the old data dir and the temp dump.
rm -rf db.old /tmp/nc-pg_dumpall.sql
```

**Immich differences:**

- Data dir is `${DB_DATA_LOCATION}` from `.env` (not under `docker/immich/`),
  so step 3 is just `mv "$DB_DATA_LOCATION" "$DB_DATA_LOCATION.old"` -- the
  dump at `docker/immich/db/pg_dumpall.sql` is in a separate dir and stays put.
- DB container is `immich_postgres`, user is `${DB_USERNAME}` from `.env`.
- Immich pins to a specific postgres image (`ghcr.io/immich-app/postgres`)
  with a digest -- check Immich release notes before bumping; they coordinate
  the upgrade, you usually just follow their version.