diff --git a/docs/how-to/operations/troubleshoot-sifaka-nfs.md b/docs/how-to/operations/troubleshoot-sifaka-nfs.md new file mode 100644 index 0000000..85514d4 --- /dev/null +++ b/docs/how-to/operations/troubleshoot-sifaka-nfs.md @@ -0,0 +1,73 @@ +--- +title: Troubleshoot Sifaka NFS +modified: 2026-03-28 +last-reviewed: 2026-03-28 +tags: + - how-to + - storage + - nfs +--- + +# Troubleshoot Sifaka NFS + +How to diagnose and fix NFS permission failures on [[sifaka]]. + +## Symptom + +NFS mounts from ringtail (or any Tailscale client) to sifaka fail with "Permission denied". Frigate shows empty storage stats. The `frigate-storage` check in `mise run services-check` fails. Existing mounts go stale — even `ls` on the mount point returns EACCES. + +## Root Cause: Tailscale Userspace Networking + +Sifaka runs Tailscale via the Synology DSM package. On DSM 7, the package can run in two modes: + +| Mode | `TUN` flag | NFS sees source IP as | NFS result | +|------|-----------|----------------------|------------| +| **TUN (kernel)** | `True` | Client's Tailscale IP (e.g. `100.121.200.77`) | Works — matches `100.64.0.0/10` export rule | +| **Userspace** | `False` | `127.0.0.1` (loopback) | Fails — doesn't match any export rule | + +In userspace mode, Tailscale proxies connections through loopback. The NFS daemon sees `127.0.0.1` as the source IP, which doesn't match the `100.64.0.0/10` or `192.168.1.0/24` export rules, so it rejects the mount. + +## Diagnosis + +```bash +# Check Tailscale mode on sifaka +ssh sifaka '/var/packages/Tailscale/target/bin/tailscale status --json' | python3 -c "import sys,json; print('TUN:', json.load(sys.stdin).get('TUN'))" + +# If TUN: False, that's the problem + +# Confirm NFS lease failures on ringtail +ssh ringtail 'sudo dmesg | grep -i nfs | tail -5' +# Look for: "check lease failed on NFSv4 server sifaka with error 13" +``` + +## Fix + +The DSM Task Scheduler has a boot-up task ("Enable tailscale outbound TUN") that runs: + +```bash +/var/packages/Tailscale/target/bin/tailscale configure-host; +synosystemctl restart pkgctl-Tailscale.service +``` + +`configure-host` grants the Tailscale package permission to open `/dev/net/tun` (which is `crw-------` root-only by default on DSM 7). The service restart then picks up TUN mode. + +**To fix immediately:** In DSM, go to Control Panel > Task Scheduler, select "Enable tailscale outbound TUN", and click Run. + +**Note:** Running this task restarts Tailscale, which briefly drops all Tailscale connections to sifaka. SSH sessions over Tailscale will disconnect but reconnect within seconds. + +After Tailscale restarts, restart the affected pods to get fresh NFS mounts: + +```bash +kubectl --context=k3s-ringtail rollout restart deployment/frigate -n frigate +``` + +## Why It Recurs + +The "Update Tailscale" scheduled task runs nightly (`tailscale update --yes`). Package updates can reset the TUN device permissions, reverting to userspace mode. The boot-up task only runs at boot, not after updates. + +If this keeps recurring, consider adding `tailscale configure-host` to the update task as well, or running it on a schedule. + +## Related + +- [[sifaka]] — NAS reference card +- [[frigate]] — Primary NFS consumer affected by this issue diff --git a/docs/reference/storage/sifaka.md b/docs/reference/storage/sifaka.md index 31fe90a..b3387c1 100644 --- a/docs/reference/storage/sifaka.md +++ b/docs/reference/storage/sifaka.md @@ -1,7 +1,7 @@ --- title: Sifaka -modified: 2026-02-09 -last-reviewed: 2026-03-23 +modified: 2026-03-28 +last-reviewed: 2026-03-28 tags: - storage --- @@ -28,14 +28,19 @@ Synology NAS providing network storage and backup target. | music | `/volume1/music` | Music library | [[navidrome]] | | allisonflix | `/volume1/allisonflix` | Video library | [[jellyfin]] | | photos | `/volume1/photos` | Photo library | [[immich]] | +| frigate | `/volume1/frigate` | NVR recordings, clips, models | [[frigate]] | ## NFS Exports -| Export | Allowed Clients | Purpose | -|--------|-----------------|---------| -| `/volume1/torrents` | 192.168.1.0/24, 100.64.0.0/10 | k8s pods via Docker NAT | -| `/volume1/music` | 192.168.1.0/24, 100.64.0.0/10 | k8s pods via Docker NAT | -| `/volume1/photos` | 192.168.1.0/24, 100.64.0.0/10 | k8s pods via Docker NAT | +| Export | Allowed Clients | Squash | Purpose | +|--------|-----------------|--------|---------| +| `/volume1/torrents` | 192.168.1.0/24, 100.64.0.0/10 | Map all users to admin | k8s pods | +| `/volume1/music` | 192.168.1.0/24, 100.64.0.0/10 | Map all users to admin | k8s pods | +| `/volume1/photos` | 192.168.1.0/24, 100.64.0.0/10 | Map all users to admin | k8s pods | +| `/volume1/frigate` | 192.168.1.0/24, 100.64.0.0/10 | Map all users to admin | k8s pods on ringtail | +| `/volume1/reports` | 192.168.1.0/24, 100.64.0.0/10 | Map all users to admin | Prowler reports | + +All exports use `all_squash` mapping to uid 1024 (`admin`) / gid 100 (`users`). If NFS mounts fail with permission denied, see [[troubleshoot-sifaka-nfs]]. ## Monitoring @@ -108,6 +113,7 @@ Synology uses `/dev/sata*` (e.g., `/dev/sata1` through `/dev/sata4`) instead of - Tag: `tag:nas` - ACL: `tag:homelab` can access for backups +- **Must run in TUN mode** for NFS — see [[troubleshoot-sifaka-nfs]] ## Backup @@ -117,8 +123,10 @@ Data protection for sifaka itself currently relies on the Synology RAID 5 config ## Related +- [[troubleshoot-sifaka-nfs]] - NFS permission denied troubleshooting - [[backups|Backups]] - Backup policy - [[borgmatic]] - Backup system +- [[frigate]] - NVR recordings consumer - [[immich]] - Photo consumer - [[jellyfin]] - Media consumer - [[navidrome]] - Music consumer