timothykim
bf02e29fd6
Secrets were removed from the repo and moved to .env files on the server. Added runbook step for restoring them from USB backup (or recreating from Bitwarden), updated the protection table in README, and added architecture note in CLAUDE.md.
246 lines
7.7 KiB
Markdown
246 lines
7.7 KiB
Markdown
# Argento Recovery Runbook
|
|
|
|
Full top-to-bottom rebuild. Follow these steps in order.
|
|
|
|
## 1. Install Rocky Linux 9
|
|
|
|
Standard minimal install on NVMe.
|
|
|
|
## 2. Repos and base packages
|
|
|
|
```bash
|
|
# EPEL
|
|
dnf install -y epel-release
|
|
|
|
# Docker CE
|
|
dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
|
|
|
|
# ZFS — must use zfs-testing repo (stable repo's version can't import the existing pools)
|
|
dnf install -y https://zfsonlinux.org/epel/zfs-release-2-3$(rpm --eval "%{dist}").noarch.rpm
|
|
dnf config-manager --enable zfs-testing
|
|
dnf install -y kernel-devel zfs # uses DKMS — kernel-devel is required
|
|
|
|
# NVIDIA (CUDA repo — provides nvidia-driver and container toolkit)
|
|
dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
|
|
|
|
# Install everything
|
|
# Sanoid (not in EPEL — COPR repo, see https://github.com/jimsalterjrs/sanoid for manual install)
|
|
dnf copr enable -y orrisroot/zfs
|
|
|
|
dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin docker-buildx-plugin \
|
|
certbot msmtp smartmontools \
|
|
jq git sanoid samba \
|
|
nvidia-driver nvidia-container-toolkit \
|
|
cockpit
|
|
|
|
# Load ZFS kernel module
|
|
modprobe zfs
|
|
|
|
# Configure NVIDIA container runtime for Docker
|
|
nvidia-ctk runtime configure --runtime=docker
|
|
|
|
# Set up msmtp as sendmail replacement (for cron MAILTO and smartd alerts)
|
|
ln -sf /usr/bin/msmtp /usr/sbin/sendmail
|
|
```
|
|
|
|
## 3. Enable services
|
|
|
|
```bash
|
|
systemctl enable --now docker smb firewalld cockpit.socket sshd smartd
|
|
systemctl enable --now zfs-import-cache zfs-mount zfs-share zfs-zed
|
|
systemctl enable --now sanoid.timer
|
|
```
|
|
|
|
## 4. Firewall
|
|
|
|
```bash
|
|
cp /opt/argento/system/firewalld/public.xml /etc/firewalld/zones/public.xml
|
|
firewall-cmd --reload
|
|
```
|
|
|
|
## 5. Mount USB backup drive
|
|
|
|
```bash
|
|
echo 'LABEL=argento-backup /mnt/backup ext4 defaults,nofail 0 2' >> /etc/fstab
|
|
mkdir -p /mnt/backup
|
|
mount /mnt/backup
|
|
```
|
|
|
|
## 6. Get the repo
|
|
|
|
```bash
|
|
git clone https://github.com/<user>/argento-server.git /opt/argento
|
|
|
|
# If GitHub is unavailable, clone from the Gitea bare repo on the USB backup:
|
|
# git clone /mnt/backup/argento/docker/gitea/data/git/repositories/timothykim/argento.git /opt/argento
|
|
```
|
|
|
|
Clone first -- ZFS datasets will mount into this directory tree in the next step.
|
|
|
|
## 7. ZFS pools + data restore
|
|
|
|
**If ZFS disks are intact (OS drive failure only)** -- the common case:
|
|
|
|
```bash
|
|
# ZFS stores pool metadata on the disks — import auto-detects them
|
|
# Run "zpool import" (no args) first to list available pools
|
|
zpool import nextcloud
|
|
zpool import threeteras
|
|
|
|
# Verify mountpoints (update if the old /opt/docker/ paths are still set):
|
|
zfs set mountpoint=/opt/argento/docker/nextcloud/data nextcloud/data
|
|
zfs set mountpoint=/opt/argento/smb/media threeteras/media
|
|
zfs set mountpoint=/opt/argento/docker/garage/data threeteras/garage
|
|
|
|
zpool status # verify pools are ONLINE
|
|
|
|
# Restore non-ZFS app data (Gitea, Minecraft, Jellyfin config) from USB:
|
|
rsync -a --exclude='docker/nextcloud/data/' --exclude='docker/garage/data/' --exclude='smb/' \
|
|
/mnt/backup/argento/ /opt/argento/
|
|
```
|
|
|
|
**If ZFS disks are lost (full rebuild):**
|
|
|
|
**Important:** The pools use feature flags that require ZFS from the `zfs-testing` repo (step 2). If `zpool import` fails with a feature flag error, verify `zfs-testing` is enabled and you have ZFS 2.4.0+.
|
|
|
|
```bash
|
|
# Find disk IDs (use these instead of /dev/sdX — they're stable across reboots):
|
|
ls -l /dev/disk/by-id/ | grep -v part
|
|
|
|
# Nextcloud pool — mirror of 2x 7.3TB drives
|
|
zpool create nextcloud mirror /dev/disk/by-id/<disk1> /dev/disk/by-id/<disk2>
|
|
zfs set compression=lz4 nextcloud
|
|
zfs create -o mountpoint=/opt/argento/docker/nextcloud/data nextcloud/data
|
|
|
|
# Threeteras pool — raidz2 of 6x 2.7TB drives
|
|
zpool create threeteras raidz2 /dev/disk/by-id/<disk3> ... /dev/disk/by-id/<disk8>
|
|
zfs set compression=lz4 threeteras
|
|
zfs create -o mountpoint=/opt/argento/smb/media threeteras/media
|
|
zfs create -o mountpoint=/opt/argento/docker/garage/data threeteras/garage
|
|
|
|
zpool status # verify pools are ONLINE
|
|
|
|
# Restore everything from USB (ZFS mountpoints are set, data lands in the right place):
|
|
rsync -a /mnt/backup/argento/ /opt/argento/
|
|
```
|
|
|
|
## 8. Restore .env files
|
|
|
|
Secrets are not in the repo -- they live in `.env` files on the server. Restore from USB backup:
|
|
|
|
```bash
|
|
# .env files are backed up alongside app data
|
|
# After the rsync in step 7, they should already be in place. Verify:
|
|
ls /opt/argento/docker/*/.env
|
|
|
|
# If USB backup is unavailable (full from-scratch rebuild),
|
|
# recreate .env files manually from Bitwarden.
|
|
# Each app's compose.yaml or runner-config.yaml references its .env file.
|
|
```
|
|
|
|
## 9. System configs
|
|
|
|
```bash
|
|
cp /opt/argento/system/docker-daemon.json /etc/docker/daemon.json
|
|
cp /opt/argento/system/smb.conf /etc/samba/smb.conf
|
|
cp /opt/argento/system/sanoid.conf /etc/sanoid/sanoid.conf
|
|
cp /opt/argento/system/msmtprc /etc/msmtprc
|
|
chmod 600 /etc/msmtprc
|
|
cp /opt/argento/system/smartd.conf /etc/smartd.conf
|
|
systemctl restart docker smb smartd
|
|
crontab /opt/argento/system/root-crontab
|
|
```
|
|
|
|
## 10. Nginx + SSL
|
|
|
|
```bash
|
|
# Create shared network
|
|
docker network create shared
|
|
|
|
# Move SSL confs aside (nginx can't load them — cert files don't exist yet)
|
|
# HTTP-only confs (minecraft) and non-SSL files (.inc, .example) can stay
|
|
cd /opt/argento/docker/nginx
|
|
mkdir -p conf.d/.hold certbot/www
|
|
for f in conf.d/*.conf; do
|
|
grep -q ssl_certificate "$f" && mv "$f" conf.d/.hold/
|
|
done
|
|
|
|
# Start nginx (only HTTP-only confs and the default catch-all are loaded)
|
|
docker compose up -d
|
|
|
|
# Issue certs (add any new domains that have been added since this was written)
|
|
/opt/argento/scripts/issue-cert.sh \
|
|
nextcloud.timothykim.net git.timothykim.net jellyfin.timothykim.net \
|
|
argento.timothykim.net garage.timothykim.net
|
|
|
|
# Restore SSL confs now that certs exist
|
|
mv conf.d/.hold/*.conf conf.d/
|
|
rmdir conf.d/.hold
|
|
docker exec nginx nginx -t && docker exec nginx nginx -s reload
|
|
|
|
# Set up cert renewal
|
|
mkdir -p /etc/letsencrypt/renewal-hooks/deploy
|
|
cat > /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh <<'HOOK'
|
|
#!/bin/bash
|
|
docker exec nginx nginx -s reload
|
|
HOOK
|
|
chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
|
|
systemctl enable --now certbot-renew.timer
|
|
|
|
# Verify renewal works
|
|
certbot renew --dry-run
|
|
```
|
|
|
|
## 11. Docker apps
|
|
|
|
Deploy each app (nginx is already running and will return 502 until they start):
|
|
|
|
```bash
|
|
# Add any new apps that have been added since this was written
|
|
for app in nextcloud gitea jellyfin garage minecraft; do
|
|
cd /opt/argento/docker/$app
|
|
docker compose pull
|
|
docker compose build
|
|
docker compose up -d
|
|
echo "$app started"
|
|
done
|
|
docker ps # verify all containers are healthy
|
|
```
|
|
|
|
## 12. Router
|
|
|
|
Set up port forwarding on the home router:
|
|
|
|
- 80 -> 192.168.2.122:80
|
|
- 443 -> 192.168.2.122:443
|
|
- 222 -> 192.168.2.122:222 (Gitea SSH)
|
|
- 3901 -> 192.168.2.122:3901 (Garage RPC)
|
|
- 25500-25600 -> 192.168.2.122:25500-25600 (Minecraft)
|
|
|
|
## 13. DNS
|
|
|
|
CNAME records pointing to the DDNS hostname. These don't need updating when the IP changes -- only the DDNS record does.
|
|
|
|
| Record | Type | Value |
|
|
|--------|------|-------|
|
|
| `nextcloud.timothykim.net` | CNAME | `argento.ddns.net` |
|
|
| `git.timothykim.net` | CNAME | `argento.ddns.net` |
|
|
| `jellyfin.timothykim.net` | CNAME | `argento.ddns.net` |
|
|
| `argento.timothykim.net` | CNAME | `argento.ddns.net` |
|
|
| `garage.timothykim.net` | CNAME | `argento.ddns.net` |
|
|
| `minecraft.thekims.family` | CNAME | `argento.ddns.net` |
|
|
|
|
## 14. Verify
|
|
|
|
```bash
|
|
curl -I https://nextcloud.timothykim.net
|
|
curl -I https://git.timothykim.net
|
|
curl -I https://jellyfin.timothykim.net
|
|
curl -I https://argento.timothykim.net
|
|
curl -I https://garage.timothykim.net
|
|
curl -I http://minecraft.thekims.family
|
|
|
|
# Verify SMART, ZFS, and disk space are healthy
|
|
/opt/argento/scripts/disk-health-check.sh
|
|
```
|