# Argento Recovery Runbook

Full top-to-bottom rebuild. Follow these steps in order.

## 1. Install Rocky Linux 9

Standard minimal install on NVMe.

## 2. Repos and base packages

```bash
# EPEL
dnf install -y epel-release

# Docker CE
dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo

# ZFS — must use zfs-testing repo (stable repo's version can't import the existing pools)
dnf install -y https://zfsonlinux.org/epel/zfs-release-2-3$(rpm --eval "%{dist}").noarch.rpm
dnf config-manager --enable zfs-testing
dnf install -y kernel-devel zfs  # uses DKMS — kernel-devel is required

# NVIDIA (CUDA repo — provides nvidia-driver and container toolkit)
dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo

# Install everything
# Sanoid (not in EPEL — COPR repo, see https://github.com/jimsalterjrs/sanoid for manual install)
dnf copr enable -y orrisroot/zfs

dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin docker-buildx-plugin \
    certbot msmtp smartmontools \
    jq git sanoid samba \
    nvidia-driver nvidia-container-toolkit \
    cockpit

# Load ZFS kernel module
modprobe zfs

# Configure NVIDIA container runtime for Docker
nvidia-ctk runtime configure --runtime=docker

# Set up msmtp as sendmail replacement (for cron MAILTO and smartd alerts)
ln -sf /usr/bin/msmtp /usr/sbin/sendmail
```

## 3. Enable services

```bash
systemctl enable --now docker smb firewalld cockpit.socket sshd smartd
systemctl enable --now zfs-import-cache zfs-mount zfs-share zfs-zed
systemctl enable --now sanoid.timer
```

## 4. Firewall

```bash
cp /opt/argento/system/firewalld/public.xml /etc/firewalld/zones/public.xml
firewall-cmd --reload
```

## 5. Mount USB backup drive

```bash
echo 'LABEL=argento-backup /mnt/backup ext4 defaults,nofail 0 2' >> /etc/fstab
mkdir -p /mnt/backup
mount /mnt/backup
```

## 6. Get the repo

```bash
git clone https://github.com/<user>/argento-server.git /opt/argento

# If GitHub is unavailable, clone from the Gitea bare repo on the USB backup:
# git clone /mnt/backup/argento/docker/gitea/data/git/repositories/timothykim/argento.git /opt/argento
```

Clone first -- ZFS datasets will mount into this directory tree in the next step.

## 7. ZFS pools + data restore

**If ZFS disks are intact (OS drive failure only)** -- the common case:

```bash
# ZFS stores pool metadata on the disks — import auto-detects them
# Run "zpool import" (no args) first to list available pools
zpool import nextcloud
zpool import threeteras

# Verify mountpoints (update if the old /opt/docker/ paths are still set):
zfs set mountpoint=/opt/argento/docker/nextcloud/data nextcloud/data
zfs set mountpoint=/opt/argento/smb/media threeteras/media
zfs set mountpoint=/opt/argento/docker/garage/data threeteras/garage

zpool status  # verify pools are ONLINE

# Restore non-ZFS app data (Gitea, Minecraft, Jellyfin config) from USB:
rsync -a --exclude='docker/nextcloud/data/' --exclude='docker/garage/data/' --exclude='smb/' \
    /mnt/backup/argento/ /opt/argento/
```

**If ZFS disks are lost (full rebuild):**

**Important:** The pools use feature flags that require ZFS from the `zfs-testing` repo (step 2). If `zpool import` fails with a feature flag error, verify `zfs-testing` is enabled and you have ZFS 2.4.0+.

```bash
# Find disk IDs (use these instead of /dev/sdX — they're stable across reboots):
ls -l /dev/disk/by-id/ | grep -v part

# Nextcloud pool — mirror of 2x 7.3TB drives
zpool create nextcloud mirror /dev/disk/by-id/<disk1> /dev/disk/by-id/<disk2>
zfs set compression=lz4 nextcloud
zfs create -o mountpoint=/opt/argento/docker/nextcloud/data nextcloud/data

# Threeteras pool — raidz2 of 6x 2.7TB drives
zpool create threeteras raidz2 /dev/disk/by-id/<disk3> ... /dev/disk/by-id/<disk8>
zfs set compression=lz4 threeteras
zfs create -o mountpoint=/opt/argento/smb/media threeteras/media
zfs create -o mountpoint=/opt/argento/docker/garage/data threeteras/garage

zpool status  # verify pools are ONLINE

# Restore everything from USB (ZFS mountpoints are set, data lands in the right place):
rsync -a /mnt/backup/argento/ /opt/argento/
```

## 8. Restore .env files

Secrets are not in the repo -- they live in `.env` files on the server. Restore from USB backup:

```bash
# .env files are backed up alongside app data
# After the rsync in step 7, they should already be in place. Verify:
ls /opt/argento/docker/*/.env

# If USB backup is unavailable (full from-scratch rebuild),
# recreate .env files manually from Bitwarden.
# Each app's compose.yaml or runner-config.yaml references its .env file.
```

## 9. System configs

```bash
cp /opt/argento/system/docker-daemon.json /etc/docker/daemon.json
cp /opt/argento/system/smb.conf /etc/samba/smb.conf
cp /opt/argento/system/sanoid.conf /etc/sanoid/sanoid.conf
cp /opt/argento/system/msmtprc /etc/msmtprc
chmod 600 /etc/msmtprc
cp /opt/argento/system/smartd.conf /etc/smartd.conf
systemctl restart docker smb smartd
crontab /opt/argento/system/root-crontab
```

## 10. Nginx + SSL

```bash
# Create shared network
docker network create shared

# Move SSL confs aside (nginx can't load them — cert files don't exist yet)
# HTTP-only confs (minecraft) and non-SSL files (.inc, .example) can stay
cd /opt/argento/docker/nginx
mkdir -p conf.d/.hold certbot/www
for f in conf.d/*.conf; do
    grep -q ssl_certificate "$f" && mv "$f" conf.d/.hold/
done

# Start nginx (only HTTP-only confs and the default catch-all are loaded)
docker compose up -d

# Issue certs (add any new domains that have been added since this was written)
/opt/argento/scripts/issue-cert.sh \
    nextcloud.timothykim.net git.timothykim.net jellyfin.timothykim.net \
    argento.timothykim.net garage.timothykim.net

# Restore SSL confs now that certs exist
mv conf.d/.hold/*.conf conf.d/
rmdir conf.d/.hold
docker exec nginx nginx -t && docker exec nginx nginx -s reload

# Set up cert renewal
mkdir -p /etc/letsencrypt/renewal-hooks/deploy
cat > /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh <<'HOOK'
#!/bin/bash
docker exec nginx nginx -s reload
HOOK
chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
systemctl enable --now certbot-renew.timer

# Verify renewal works
certbot renew --dry-run
```

## 11. Docker apps

Deploy each app (nginx is already running and will return 502 until they start):

```bash
# Add any new apps that have been added since this was written
for app in nextcloud gitea jellyfin garage minecraft beszel-agent; do
    cd /opt/argento/docker/$app
    docker compose pull
    docker compose build
    docker compose up -d
    echo "$app started"
done
docker ps  # verify all containers are healthy
```

## 12. Router

Set up port forwarding on the home router:

- 80 -> 192.168.2.122:80
- 443 -> 192.168.2.122:443
- 222 -> 192.168.2.122:222 (Gitea SSH)
- 3901 -> 192.168.2.122:3901 (Garage RPC)
- 25500-25600 -> 192.168.2.122:25500-25600 (Minecraft)

## 13. DNS

CNAME records pointing to the DDNS hostname. These don't need updating when the IP changes -- only the DDNS record does.

| Record | Type | Value |
|--------|------|-------|
| `nextcloud.timothykim.net` | CNAME | `argento.ddns.net` |
| `git.timothykim.net` | CNAME | `argento.ddns.net` |
| `jellyfin.timothykim.net` | CNAME | `argento.ddns.net` |
| `argento.timothykim.net` | CNAME | `argento.ddns.net` |
| `garage.timothykim.net` | CNAME | `argento.ddns.net` |
| `minecraft.thekims.family` | CNAME | `argento.ddns.net` |

## 14. Verify

```bash
curl -I https://nextcloud.timothykim.net
curl -I https://git.timothykim.net
curl -I https://jellyfin.timothykim.net
curl -I https://argento.timothykim.net
curl -I https://garage.timothykim.net
curl -I http://minecraft.thekims.family

# Verify SMART, ZFS, and disk space are healthy
/opt/argento/scripts/disk-health-check.sh
```