246 lines
7.7 KiB
Markdown
246 lines
7.7 KiB
Markdown
# Argento Recovery Runbook
|
|
|
|
Full top-to-bottom rebuild. Follow these steps in order.
|
|
|
|
## 1. Install Rocky Linux 9
|
|
|
|
Standard minimal install on NVMe.
|
|
|
|
## 2. Repos and base packages
|
|
|
|
```bash
|
|
# EPEL
|
|
dnf install -y epel-release
|
|
|
|
# Docker CE
|
|
dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
|
|
|
|
# ZFS — must use zfs-testing repo (stable repo's version can't import the existing pools)
|
|
dnf install -y https://zfsonlinux.org/epel/zfs-release-2-3$(rpm --eval "%{dist}").noarch.rpm
|
|
dnf config-manager --enable zfs-testing
|
|
dnf install -y kernel-devel zfs # uses DKMS — kernel-devel is required
|
|
|
|
# NVIDIA (CUDA repo — provides nvidia-driver and container toolkit)
|
|
dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
|
|
|
|
# Install everything
|
|
# Sanoid (not in EPEL — COPR repo, see https://github.com/jimsalterjrs/sanoid for manual install)
|
|
dnf copr enable -y orrisroot/zfs
|
|
|
|
dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin docker-buildx-plugin \
|
|
certbot msmtp smartmontools \
|
|
jq git sanoid samba \
|
|
nvidia-driver nvidia-container-toolkit \
|
|
cockpit
|
|
|
|
# Load ZFS kernel module
|
|
modprobe zfs
|
|
|
|
# Configure NVIDIA container runtime for Docker
|
|
nvidia-ctk runtime configure --runtime=docker
|
|
|
|
# Set up msmtp as sendmail replacement (for cron MAILTO and smartd alerts)
|
|
ln -sf /usr/bin/msmtp /usr/sbin/sendmail
|
|
```
|
|
|
|
## 3. Enable services
|
|
|
|
```bash
|
|
systemctl enable --now docker smb firewalld cockpit.socket sshd smartd
|
|
systemctl enable --now zfs-import-cache zfs-mount zfs-share zfs-zed
|
|
systemctl enable --now sanoid.timer
|
|
```
|
|
|
|
## 4. Firewall
|
|
|
|
```bash
|
|
cp /opt/argento/system/firewalld/public.xml /etc/firewalld/zones/public.xml
|
|
firewall-cmd --reload
|
|
```
|
|
|
|
## 5. Mount USB backup drive
|
|
|
|
```bash
|
|
echo 'LABEL=argento-backup /mnt/backup ext4 defaults,nofail 0 2' >> /etc/fstab
|
|
mkdir -p /mnt/backup
|
|
mount /mnt/backup
|
|
```
|
|
|
|
## 6. Get the repo
|
|
|
|
```bash
|
|
git clone https://github.com/<user>/argento-server.git /opt/argento
|
|
|
|
# If GitHub is unavailable, clone from the Gitea bare repo on the USB backup:
|
|
# git clone /mnt/backup/argento/docker/gitea/data/git/repositories/timothykim/argento.git /opt/argento
|
|
```
|
|
|
|
Clone first -- ZFS datasets will mount into this directory tree in the next step.
|
|
|
|
## 7. ZFS pools + data restore
|
|
|
|
**If ZFS disks are intact (OS drive failure only)** -- the common case:
|
|
|
|
```bash
|
|
# ZFS stores pool metadata on the disks — import auto-detects them
|
|
# Run "zpool import" (no args) first to list available pools
|
|
zpool import nextcloud
|
|
zpool import threeteras
|
|
|
|
# Verify mountpoints (update if the old /opt/docker/ paths are still set):
|
|
zfs set mountpoint=/opt/argento/docker/nextcloud/data nextcloud/data
|
|
zfs set mountpoint=/opt/argento/smb/media threeteras/media
|
|
zfs set mountpoint=/opt/argento/docker/garage/data threeteras/garage
|
|
|
|
zpool status # verify pools are ONLINE
|
|
|
|
# Restore non-ZFS app data (Gitea, Minecraft, Jellyfin config) from USB:
|
|
rsync -a --exclude='docker/nextcloud/data/' --exclude='docker/garage/data/' --exclude='smb/' \
|
|
/mnt/backup/argento/ /opt/argento/
|
|
```
|
|
|
|
**If ZFS disks are lost (full rebuild):**
|
|
|
|
**Important:** The pools use feature flags that require ZFS from the `zfs-testing` repo (step 2). If `zpool import` fails with a feature flag error, verify `zfs-testing` is enabled and you have ZFS 2.4.0+.
|
|
|
|
```bash
|
|
# Find disk IDs (use these instead of /dev/sdX — they're stable across reboots):
|
|
ls -l /dev/disk/by-id/ | grep -v part
|
|
|
|
# Nextcloud pool — mirror of 2x 7.3TB drives
|
|
zpool create nextcloud mirror /dev/disk/by-id/<disk1> /dev/disk/by-id/<disk2>
|
|
zfs set compression=lz4 nextcloud
|
|
zfs create -o mountpoint=/opt/argento/docker/nextcloud/data nextcloud/data
|
|
|
|
# Threeteras pool — raidz2 of 6x 2.7TB drives
|
|
zpool create threeteras raidz2 /dev/disk/by-id/<disk3> ... /dev/disk/by-id/<disk8>
|
|
zfs set compression=lz4 threeteras
|
|
zfs create -o mountpoint=/opt/argento/smb/media threeteras/media
|
|
zfs create -o mountpoint=/opt/argento/docker/garage/data threeteras/garage
|
|
|
|
zpool status # verify pools are ONLINE
|
|
|
|
# Restore everything from USB (ZFS mountpoints are set, data lands in the right place):
|
|
rsync -a /mnt/backup/argento/ /opt/argento/
|
|
```
|
|
|
|
## 8. Restore .env files
|
|
|
|
Secrets are not in the repo -- they live in `.env` files on the server. Restore from USB backup:
|
|
|
|
```bash
|
|
# .env files are backed up alongside app data
|
|
# After the rsync in step 7, they should already be in place. Verify:
|
|
ls /opt/argento/docker/*/.env
|
|
|
|
# If USB backup is unavailable (full from-scratch rebuild),
|
|
# recreate .env files manually from Bitwarden.
|
|
# Each app's compose.yaml or runner-config.yaml references its .env file.
|
|
```
|
|
|
|
## 9. System configs
|
|
|
|
```bash
|
|
cp /opt/argento/system/docker-daemon.json /etc/docker/daemon.json
|
|
cp /opt/argento/system/smb.conf /etc/samba/smb.conf
|
|
cp /opt/argento/system/sanoid.conf /etc/sanoid/sanoid.conf
|
|
cp /opt/argento/system/msmtprc /etc/msmtprc
|
|
chmod 600 /etc/msmtprc
|
|
cp /opt/argento/system/smartd.conf /etc/smartd.conf
|
|
systemctl restart docker smb smartd
|
|
crontab /opt/argento/system/root-crontab
|
|
```
|
|
|
|
## 10. Nginx + SSL
|
|
|
|
```bash
|
|
# Create shared network
|
|
docker network create shared
|
|
|
|
# Move SSL confs aside (nginx can't load them — cert files don't exist yet)
|
|
# HTTP-only confs (minecraft) and non-SSL files (.inc, .example) can stay
|
|
cd /opt/argento/docker/nginx
|
|
mkdir -p conf.d/.hold certbot/www
|
|
for f in conf.d/*.conf; do
|
|
grep -q ssl_certificate "$f" && mv "$f" conf.d/.hold/
|
|
done
|
|
|
|
# Start nginx (only HTTP-only confs and the default catch-all are loaded)
|
|
docker compose up -d
|
|
|
|
# Issue certs (add any new domains that have been added since this was written)
|
|
/opt/argento/scripts/issue-cert.sh \
|
|
nextcloud.timothykim.net git.timothykim.net jellyfin.timothykim.net \
|
|
argento.timothykim.net garage.timothykim.net
|
|
|
|
# Restore SSL confs now that certs exist
|
|
mv conf.d/.hold/*.conf conf.d/
|
|
rmdir conf.d/.hold
|
|
docker exec nginx nginx -t && docker exec nginx nginx -s reload
|
|
|
|
# Set up cert renewal
|
|
mkdir -p /etc/letsencrypt/renewal-hooks/deploy
|
|
cat > /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh <<'HOOK'
|
|
#!/bin/bash
|
|
docker exec nginx nginx -s reload
|
|
HOOK
|
|
chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
|
|
systemctl enable --now certbot-renew.timer
|
|
|
|
# Verify renewal works
|
|
certbot renew --dry-run
|
|
```
|
|
|
|
## 11. Docker apps
|
|
|
|
Deploy each app (nginx is already running and will return 502 until they start):
|
|
|
|
```bash
|
|
# Add any new apps that have been added since this was written
|
|
for app in nextcloud gitea jellyfin garage minecraft beszel-agent; do
|
|
cd /opt/argento/docker/$app
|
|
docker compose pull
|
|
docker compose build
|
|
docker compose up -d
|
|
echo "$app started"
|
|
done
|
|
docker ps # verify all containers are healthy
|
|
```
|
|
|
|
## 12. Router
|
|
|
|
Set up port forwarding on the home router:
|
|
|
|
- 80 -> 192.168.2.122:80
|
|
- 443 -> 192.168.2.122:443
|
|
- 222 -> 192.168.2.122:222 (Gitea SSH)
|
|
- 3901 -> 192.168.2.122:3901 (Garage RPC)
|
|
- 25500-25600 -> 192.168.2.122:25500-25600 (Minecraft)
|
|
|
|
## 13. DNS
|
|
|
|
CNAME records pointing to the DDNS hostname. These don't need updating when the IP changes -- only the DDNS record does.
|
|
|
|
| Record | Type | Value |
|
|
|--------|------|-------|
|
|
| `nextcloud.timothykim.net` | CNAME | `argento.ddns.net` |
|
|
| `git.timothykim.net` | CNAME | `argento.ddns.net` |
|
|
| `jellyfin.timothykim.net` | CNAME | `argento.ddns.net` |
|
|
| `argento.timothykim.net` | CNAME | `argento.ddns.net` |
|
|
| `garage.timothykim.net` | CNAME | `argento.ddns.net` |
|
|
| `minecraft.thekims.family` | CNAME | `argento.ddns.net` |
|
|
|
|
## 14. Verify
|
|
|
|
```bash
|
|
curl -I https://nextcloud.timothykim.net
|
|
curl -I https://git.timothykim.net
|
|
curl -I https://jellyfin.timothykim.net
|
|
curl -I https://argento.timothykim.net
|
|
curl -I https://garage.timothykim.net
|
|
curl -I http://minecraft.thekims.family
|
|
|
|
# Verify SMART, ZFS, and disk space are healthy
|
|
/opt/argento/scripts/disk-health-check.sh
|
|
```
|