From 0018d3c5932c73778f17d6de2f8c881a997c86bd Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 3 Feb 2026 13:22:19 -0800 Subject: [PATCH 01/12] Add Reference section with 24 technical reference cards Phase 2 of documentation restructuring. Creates docs/reference/ with: Services (16): - alloy, argocd, borgmatic, 1password, forgejo, grafana - jellyfin, kiwix, loki, miniflux, navidrome, postgresql - prometheus, teslamate, transmission, zot Infrastructure (3): - hosts - Device inventory - tailscale - ACLs, groups, tags - routing - DNS domains and port mappings Kubernetes (2): - cluster - Minikube specs - apps - ArgoCD application registry Storage (2): - sifaka - Synology NAS configuration - backups - Backup policy All cards use wiki-links for cross-referencing and include YAML frontmatter with title and tags for Quartz. Co-Authored-By: Claude Opus 4.5 --- docs/README.md | 14 ++-- .../docs-phase-2-reference.feature.md | 1 + docs/reference/index.md | 54 ++++++++++++ docs/reference/infrastructure/hosts.md | 71 ++++++++++++++++ docs/reference/infrastructure/routing.md | 79 +++++++++++++++++ docs/reference/infrastructure/tailscale.md | 67 +++++++++++++++ docs/reference/kubernetes/apps.md | 65 ++++++++++++++ docs/reference/kubernetes/cluster.md | 73 ++++++++++++++++ docs/reference/services/1password.md | 58 +++++++++++++ docs/reference/services/alloy.md | 52 ++++++++++++ docs/reference/services/argocd.md | 50 +++++++++++ docs/reference/services/borgmatic.md | 60 +++++++++++++ docs/reference/services/forgejo.md | 58 +++++++++++++ docs/reference/services/grafana.md | 50 +++++++++++ docs/reference/services/jellyfin.md | 51 +++++++++++ docs/reference/services/kiwix.md | 52 ++++++++++++ docs/reference/services/loki.md | 51 +++++++++++ docs/reference/services/miniflux.md | 49 +++++++++++ docs/reference/services/navidrome.md | 52 ++++++++++++ docs/reference/services/postgresql.md | 68 +++++++++++++++ docs/reference/services/prometheus.md | 54 ++++++++++++ docs/reference/services/teslamate.md | 58 +++++++++++++ docs/reference/services/transmission.md | 53 ++++++++++++ docs/reference/services/zot.md | 66 +++++++++++++++ docs/reference/storage/backups.md | 84 +++++++++++++++++++ docs/reference/storage/sifaka.md | 51 +++++++++++ 26 files changed, 1435 insertions(+), 6 deletions(-) create mode 100644 docs/changelog.d/docs-phase-2-reference.feature.md create mode 100644 docs/reference/index.md create mode 100644 docs/reference/infrastructure/hosts.md create mode 100644 docs/reference/infrastructure/routing.md create mode 100644 docs/reference/infrastructure/tailscale.md create mode 100644 docs/reference/kubernetes/apps.md create mode 100644 docs/reference/kubernetes/cluster.md create mode 100644 docs/reference/services/1password.md create mode 100644 docs/reference/services/alloy.md create mode 100644 docs/reference/services/argocd.md create mode 100644 docs/reference/services/borgmatic.md create mode 100644 docs/reference/services/forgejo.md create mode 100644 docs/reference/services/grafana.md create mode 100644 docs/reference/services/jellyfin.md create mode 100644 docs/reference/services/kiwix.md create mode 100644 docs/reference/services/loki.md create mode 100644 docs/reference/services/miniflux.md create mode 100644 docs/reference/services/navidrome.md create mode 100644 docs/reference/services/postgresql.md create mode 100644 docs/reference/services/prometheus.md create mode 100644 docs/reference/services/teslamate.md create mode 100644 docs/reference/services/transmission.md create mode 100644 docs/reference/services/zot.md create mode 100644 docs/reference/storage/backups.md create mode 100644 docs/reference/storage/sifaka.md diff --git a/docs/README.md b/docs/README.md index 4403aea..dc472e2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -64,14 +64,16 @@ The documentation is being restructured to follow the [Diataxis](https://diataxi **Docs URL:** https://docs.ops.eblu.me -### Phase 2: Reference +### Phase 2: Reference (Complete) Information-oriented technical descriptions. Built first so other docs can link to reference material. -- [ ] Create `reference/` directory -- [ ] Service reference pages (migrate from zk cards) -- [ ] Infrastructure inventory -- [ ] Configuration reference -- [ ] API/CLI reference for mise tasks +- [x] Create `reference/` directory with index +- [x] Service reference pages (16 services: alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot) +- [x] Infrastructure inventory (hosts, tailscale, routing) +- [x] Kubernetes reference (cluster, apps) +- [x] Storage reference (sifaka, backups) + +**Reference URL:** https://docs.ops.eblu.me/reference/ ### Phase 3: Tutorials Learning-oriented content for getting started. diff --git a/docs/changelog.d/docs-phase-2-reference.feature.md b/docs/changelog.d/docs-phase-2-reference.feature.md new file mode 100644 index 0000000..39bf20f --- /dev/null +++ b/docs/changelog.d/docs-phase-2-reference.feature.md @@ -0,0 +1 @@ +Add Reference section with 24 technical reference cards covering services, infrastructure, kubernetes, and storage diff --git a/docs/reference/index.md b/docs/reference/index.md new file mode 100644 index 0000000..1662f1b --- /dev/null +++ b/docs/reference/index.md @@ -0,0 +1,54 @@ +--- +title: Reference +tags: + - reference +--- + +# Reference + +Technical specifications, inventories, and configuration details for BlumeOps infrastructure. + +## Services + +Individual service reference cards with URLs, configuration, and operational details. + +| Service | Description | Location | +|---------|-------------|----------| +| [[services/alloy\|Alloy]] | Observability collector (metrics & logs) | indri + k8s | +| [[services/argocd\|ArgoCD]] | GitOps continuous delivery | k8s | +| [[services/borgmatic\|Borgmatic]] | Backup system | indri | +| [[services/1password\|1Password]] | Secrets management | cloud + k8s | +| [[services/forgejo\|Forgejo]] | Git forge & CI/CD | indri | +| [[services/grafana\|Grafana]] | Dashboards & visualization | k8s | +| [[services/jellyfin\|Jellyfin]] | Media server | indri | +| [[services/kiwix\|Kiwix]] | Offline Wikipedia & ZIM archives | k8s | +| [[services/loki\|Loki]] | Log aggregation | k8s | +| [[services/miniflux\|Miniflux]] | RSS feed reader | k8s | +| [[services/navidrome\|Navidrome]] | Music streaming | k8s | +| [[services/postgresql\|PostgreSQL]] | Database cluster | k8s | +| [[services/prometheus\|Prometheus]] | Metrics collection | k8s | +| [[services/teslamate\|TeslaMate]] | Tesla data logger | k8s | +| [[services/transmission\|Transmission]] | BitTorrent daemon | k8s | +| [[services/zot\|Zot]] | Container registry | indri | + +## Infrastructure + +Host inventory and network configuration. + +- [[infrastructure/hosts\|Hosts]] - Device inventory (indri, gilbert, sifaka, etc.) +- [[infrastructure/tailscale\|Tailscale]] - ACLs, groups, tags +- [[infrastructure/routing\|Routing]] - DNS domains, port mappings + +## Kubernetes + +Cluster configuration and application registry. + +- [[kubernetes/cluster\|Cluster]] - Minikube specs, storage, networking +- [[kubernetes/apps\|Apps]] - ArgoCD application registry + +## Storage + +Network storage and backup configuration. + +- [[storage/sifaka\|Sifaka]] - Synology NAS configuration +- [[storage/backups\|Backups]] - Backup policy and schedule diff --git a/docs/reference/infrastructure/hosts.md b/docs/reference/infrastructure/hosts.md new file mode 100644 index 0000000..9589b40 --- /dev/null +++ b/docs/reference/infrastructure/hosts.md @@ -0,0 +1,71 @@ +--- +title: Host Inventory +tags: + - infrastructure +--- + +# Host Inventory + +All devices connected via [Tailscale](https://login.tailscale.com/) tailnet `tail8d86e.ts.net`. + +## Devices + +| Host | Description | Notes | +|------|-------------|-------| +| **Indri** | Mac Mini M1, 2020 | Primary server, 2TB internal disk | +| **[[storage/sifaka\|Sifaka]]** | Synology NAS | 10.9TB RAID 5, backup target | +| **Gilbert** | 13" MacBook Air M4, 2025 | Primary workstation | +| **Mouse** | 13" MacBook Air M2 | Allison's laptop | +| **UniFi** | UniFi Express 7 | Home WiFi network | +| **Dwarf** | iPad Air | Employer-provided, off tailnet | + +## Indri Details + +| Property | Value | +|----------|-------| +| **Model** | Mac mini M1, 2020 (Macmini9,1) | +| **Storage** | 2TB internal SSD | +| **macOS** | 15.7.3 (Sequoia) | +| **Role** | Primary server | +| **Tailscale IP** | 100.98.163.89 | + +### Services Hosted + +**Native (via Ansible):** +- [[services/forgejo\|Forgejo]] - Git forge +- [[services/zot\|Zot]] - Container registry +- [[services/jellyfin\|Jellyfin]] - Media server +- [[services/borgmatic\|Borgmatic]] - Backup system +- [[services/alloy\|Alloy]] - Metrics/logs collector +- Caddy - Reverse proxy + +**Kubernetes (via minikube):** +- [[kubernetes/apps\|All k8s applications]] + +### Sleep Prevention + +Indri uses Amphetamine (App Store) to prevent sleep. Configuration: +- Start Session At Launch: enabled +- Default Duration: indefinite +- Allow Closed-Display Sleep: enabled + +## Gilbert Details + +| Property | Value | +|----------|-------| +| **Model** | 13" MacBook Air M4, 2025 | +| **Role** | Development workstation | +| **User** | eblume | + +### Development Tools + +Managed via `Brewfile` and `mise.toml`. + +Fish abbreviations: +- `ki` -> `kubectl --context=minikube-indri` +- `k9i` -> `k9s --context=minikube-indri` + +## Related + +- [[infrastructure/tailscale\|Tailscale]] - Network configuration +- [[storage/sifaka\|Sifaka]] - NAS details diff --git a/docs/reference/infrastructure/routing.md b/docs/reference/infrastructure/routing.md new file mode 100644 index 0000000..51d7133 --- /dev/null +++ b/docs/reference/infrastructure/routing.md @@ -0,0 +1,79 @@ +--- +title: Service Routing +tags: + - infrastructure + - network +--- + +# Service Routing + +Services are accessible via two DNS domains with different reachability. + +## DNS Domains + +| Domain | Proxy | Reachable From | +|--------|-------|----------------| +| `*.ops.eblu.me` | Caddy on indri | k8s pods, docker containers, tailnet clients | +| `*.tail8d86e.ts.net` | Tailscale MagicDNS | Tailnet clients only | + +**Use `*.ops.eblu.me`** for services that need pod-to-service communication. + +## Caddy Services (`*.ops.eblu.me`) + +DNS points to indri's Tailscale IP (100.98.163.89). TLS via Let's Encrypt (ACME DNS-01 with Gandi). + +| Service | URL | Description | +|---------|-----|-------------| +| Homepage | https://go.ops.eblu.me | Service dashboard | +| [[services/forgejo\|Forgejo]] | https://forge.ops.eblu.me | Git hosting (SSH: 2222) | +| [[services/zot\|Zot]] | https://registry.ops.eblu.me | Container registry | +| [[services/grafana\|Grafana]] | https://grafana.ops.eblu.me | Dashboards | +| [[services/argocd\|ArgoCD]] | https://argocd.ops.eblu.me | GitOps CD | +| [[services/prometheus\|Prometheus]] | https://prometheus.ops.eblu.me | Metrics | +| [[services/loki\|Loki]] | https://loki.ops.eblu.me | Logs | +| [[services/miniflux\|Miniflux]] | https://feed.ops.eblu.me | RSS reader | +| [[services/kiwix\|Kiwix]] | https://kiwix.ops.eblu.me | Offline Wikipedia | +| [[services/transmission\|Transmission]] | https://torrent.ops.eblu.me | BitTorrent | +| [[services/teslamate\|TeslaMate]] | https://tesla.ops.eblu.me | Tesla logger | +| [[services/navidrome\|Navidrome]] | https://dj.ops.eblu.me | Music streaming | +| [[services/jellyfin\|Jellyfin]] | https://jellyfin.ops.eblu.me | Media server | +| [[services/postgresql\|PostgreSQL]] | pg.ops.eblu.me:5432 | Database | +| [[storage/sifaka\|Sifaka]] | https://nas.ops.eblu.me | NAS dashboard | + +## Tailscale-Only Services + +| Service | URL | Description | +|---------|-----|-------------| +| Kubernetes | https://k8s.tail8d86e.ts.net | Minikube API | + +## Port Map (Indri) + +| Port | Service | Protocol | Binding | Notes | +|------|---------|----------|---------|-------| +| 443 | Caddy | HTTPS | 0.0.0.0 | Reverse proxy | +| 2222 | Caddy L4 | TCP | 0.0.0.0 | SSH proxy to Forgejo | +| 5432 | Caddy L4 | TCP | 0.0.0.0 | PostgreSQL proxy | +| 2200 | Forgejo SSH | TCP | localhost | Built-in SSH server | +| 3001 | Forgejo | HTTP | localhost | Web UI | +| 5050 | Zot | HTTP | localhost | Registry API | +| 8096 | Jellyfin | HTTP | localhost | Media server | +| 44491 | K8s API | HTTPS | 0.0.0.0 | Minikube API server | + +## Adding New Services + +### Indri Services (via Caddy) +1. Host service on localhost +2. Add to `ansible/roles/caddy/defaults/main.yml` +3. Run `mise run provision-indri -- --tags caddy` + +### K8s Services (via Tailscale Ingress) +1. Create manifests in `argocd/manifests//` +2. Add ArgoCD Application in `argocd/apps/` +3. Add Tailscale Ingress annotation +4. Add Caddy proxy entry +5. Sync via ArgoCD + +## Related + +- [[infrastructure/tailscale\|Tailscale]] - ACL configuration +- [[infrastructure/hosts\|Hosts]] - Where services run diff --git a/docs/reference/infrastructure/tailscale.md b/docs/reference/infrastructure/tailscale.md new file mode 100644 index 0000000..7e2d49a --- /dev/null +++ b/docs/reference/infrastructure/tailscale.md @@ -0,0 +1,67 @@ +--- +title: Tailscale +tags: + - infrastructure + - network +--- + +# Tailscale + +Tailnet `tail8d86e.ts.net` provides secure networking for all BlumeOps infrastructure. + +## ACL Management + +ACLs managed via Pulumi in `pulumi/policy.hujson`. + +```bash +mise run tailnet-preview # Preview changes +mise run tailnet-up # Apply changes +``` + +## Groups + +| Group | Members | Purpose | +|-------|---------|---------| +| `group:allisonflix` | admin, member | [[services/jellyfin\|Jellyfin]] media access | + +## Device Tags + +| Tag | Devices | Purpose | +|-----|---------|---------| +| `tag:homelab` | indri | Server infrastructure | +| `tag:nas` | sifaka | Network-attached storage | +| `tag:blumeops` | indri, sifaka | Pulumi IaC managed resources | +| `tag:registry` | indri | Container registry access | +| `tag:k8s-api` | indri | Kubernetes API server access | + +**Important:** Don't tag user-owned devices (like gilbert). Tagging converts them to "tagged devices" which lose user identity and break user-based SSH rules. + +## Access Matrix + +| Source | Kiwix | Forge | PyPI | Miniflux | PostgreSQL | NAS | Grafana | Loki | +|--------|-------|-------|------|----------|------------|-----|---------|------| +| `autogroup:admin` | Y | Y | Y | Y | Y | Y | Y | Y | +| `autogroup:member` | Y | Y | Y | Y | Y | - | - | - | +| `tag:homelab` | - | - | - | - | - | Y | - | - | + +- **Admins** - full access to all services +- **Members** - member services only, no Grafana/Loki/NAS + +## SSH Access + +| Source | Destinations | Auth | +|--------|--------------|------| +| `autogroup:member` | `autogroup:self` | check | +| `autogroup:admin` | `tag:homelab` | check (12h) | +| `autogroup:admin` | `tag:nas` | check (12h) | + +## OAuth Credentials + +Pulumi uses OAuth client from 1Password (blumeops vault): +- Scopes: acl, dns, devices, services +- Auto-applies `tag:blumeops` to IaC-managed resources + +## Related + +- [[infrastructure/routing\|Routing]] - Service URLs +- [[infrastructure/hosts\|Hosts]] - Device inventory diff --git a/docs/reference/kubernetes/apps.md b/docs/reference/kubernetes/apps.md new file mode 100644 index 0000000..de6f679 --- /dev/null +++ b/docs/reference/kubernetes/apps.md @@ -0,0 +1,65 @@ +--- +title: ArgoCD Applications +tags: + - kubernetes + - argocd +--- + +# ArgoCD Applications + +Registry of all applications deployed via [[services/argocd\|ArgoCD]]. + +## Application Registry + +| App | Namespace | Path/Source | Service | +|-----|-----------|-------------|---------| +| `apps` | argocd | `argocd/apps/` | App-of-apps root | +| `argocd` | argocd | `argocd/manifests/argocd/` | [[services/argocd\|ArgoCD]] | +| `tailscale-operator` | tailscale | `argocd/manifests/tailscale-operator/` | Tailscale k8s operator | +| `1password-connect` | 1password | `argocd/manifests/1password-connect/` | [[services/1password\|1Password]] | +| `external-secrets` | external-secrets | Helm chart | [[services/1password\|1Password]] | +| `external-secrets-config` | external-secrets | `argocd/manifests/external-secrets-config/` | [[services/1password\|1Password]] | +| `cloudnative-pg` | cnpg-system | Helm chart (forge mirror) | PostgreSQL operator | +| `blumeops-pg` | databases | `argocd/manifests/databases/` | [[services/postgresql\|PostgreSQL]] | +| `prometheus` | monitoring | `argocd/manifests/prometheus/` | [[services/prometheus\|Prometheus]] | +| `loki` | monitoring | `argocd/manifests/loki/` | [[services/loki\|Loki]] | +| `grafana` | monitoring | Helm chart (forge mirror) | [[services/grafana\|Grafana]] | +| `grafana-config` | monitoring | `argocd/manifests/grafana-config/` | [[services/grafana\|Grafana]] | +| `alloy-k8s` | alloy | `argocd/manifests/alloy-k8s/` | [[services/alloy\|Alloy]] | +| `kube-state-metrics` | monitoring | `argocd/manifests/kube-state-metrics/` | K8s metrics | +| `miniflux` | miniflux | `argocd/manifests/miniflux/` | [[services/miniflux\|Miniflux]] | +| `kiwix` | kiwix | `argocd/manifests/kiwix/` | [[services/kiwix\|Kiwix]] | +| `torrent` | torrent | `argocd/manifests/torrent/` | [[services/transmission\|Transmission]] | +| `navidrome` | navidrome | `argocd/manifests/navidrome/` | [[services/navidrome\|Navidrome]] | +| `teslamate` | teslamate | `argocd/manifests/teslamate/` | [[services/teslamate\|TeslaMate]] | +| `forgejo-runner` | forgejo-runner | `argocd/manifests/forgejo-runner/` | [[services/forgejo\|Forgejo]] CI | + +## Sync Policies + +| Application | Policy | Rationale | +|-------------|--------|-----------| +| `apps` | Automated | Picks up new Application manifests | +| All others | Manual | Explicit control over deployments | + +## Common Commands + +```bash +argocd app list # List all apps +argocd app get # Get details +argocd app diff # Preview changes +argocd app sync # Deploy changes +``` + +## PR Workflow + +1. Create feature branch, modify manifests +2. Push to forge +3. Sync apps application: `argocd app sync apps` +4. Point service at branch: `argocd app set --revision feature/branch` +5. Test: `argocd app sync ` +6. After merge, reset: `argocd app set --revision main` + +## Related + +- [[services/argocd\|ArgoCD]] - GitOps platform details +- [[kubernetes/cluster\|Cluster]] - Kubernetes infrastructure diff --git a/docs/reference/kubernetes/cluster.md b/docs/reference/kubernetes/cluster.md new file mode 100644 index 0000000..f904fac --- /dev/null +++ b/docs/reference/kubernetes/cluster.md @@ -0,0 +1,73 @@ +--- +title: Kubernetes Cluster +tags: + - kubernetes +--- + +# Kubernetes Cluster + +Single-node Minikube cluster running on [[infrastructure/hosts\|Indri]]. + +## Cluster Specifications + +| Property | Value | +|----------|-------| +| **Driver** | docker | +| **Container Runtime** | docker | +| **Kubernetes Version** | v1.34.0 | +| **CPUs** | 6 | +| **Memory** | 11GB | +| **Disk** | 200GB | +| **API Server** | https://k8s.tail8d86e.ts.net | + +**Prerequisites:** Docker Desktop with at least 12GB memory allocated. + +## Remote Access + +From gilbert: + +```bash +mise run ensure-minikube-indri-kubectl-config +``` + +Fish abbreviations: +- `ki` -> `kubectl --context=minikube-indri` +- `k9i` -> `k9s --context=minikube-indri` + +## Volume Mounting + +Pods mount NFS directly from [[storage/sifaka\|Sifaka]]: + +```yaml +volumes: + - name: torrents + nfs: + server: sifaka + path: /volume1/torrents +``` + +Docker NATs outbound traffic through indri's LAN IP (192.168.1.50), allowing access to Sifaka's NFS exports. + +## Registry Mirror + +Containerd uses [[services/zot\|Zot]] as a pull-through cache: +- Endpoint: `host.minikube.internal:5050` +- Config: `/etc/containerd/certs.d//hosts.toml` + +Mirrors configured: `registry.ops.eblu.me`, `docker.io`, `ghcr.io`, `quay.io` + +## Useful Commands (on indri) + +```bash +minikube status # Cluster status +minikube start # Start cluster +minikube stop # Stop cluster +minikube ssh # SSH into node +minikube logs # View logs +``` + +## Related + +- [[kubernetes/apps\|Apps]] - ArgoCD applications +- [[services/argocd\|ArgoCD]] - GitOps deployment +- [[services/zot\|Zot]] - Registry mirror diff --git a/docs/reference/services/1password.md b/docs/reference/services/1password.md new file mode 100644 index 0000000..59c5b57 --- /dev/null +++ b/docs/reference/services/1password.md @@ -0,0 +1,58 @@ +--- +title: 1Password +tags: + - service + - secrets +--- + +# 1Password + +Root credential store for all BlumeOps secrets, synced to Kubernetes via External Secrets Operator. + +## Architecture + +``` +1Password Cloud + | + v +1Password Connect (namespace: 1password) + | + v +External Secrets Operator (namespace: external-secrets) + | + v +Native Kubernetes Secrets +``` + +## Vault + +The `blumeops` vault contains all infrastructure credentials. + +## Kubernetes Integration + +**ClusterSecretStore:** `onepassword-blumeops` + +Services reference 1Password items via `ExternalSecret` manifests. Example: `argocd/manifests/devpi/external-secret.yaml` + +## CLI Usage + +```bash +# Get a secret field +op --vault blumeops item get --fields --reveal + +# Inject into a template +op inject -i secret.yaml.tpl | kubectl apply -f - +``` + +## Bootstrap (Disaster Recovery) + +1. Create Connect server: `op connect server create blumeops --vaults blumeops` +2. Create token: `op connect token create blumeops --server --vault blumeops` +3. Store credentials in 1Password item "1Password Connect" +4. Apply bootstrap secret to k8s +5. Sync apps: 1password-connect, external-secrets-crds, external-secrets, external-secrets-config + +## Related + +- [[argocd\|ArgoCD]] - Uses secrets for git access +- [[postgresql\|PostgreSQL]] - Database credentials diff --git a/docs/reference/services/alloy.md b/docs/reference/services/alloy.md new file mode 100644 index 0000000..fb8dcaf --- /dev/null +++ b/docs/reference/services/alloy.md @@ -0,0 +1,52 @@ +--- +title: Grafana Alloy +tags: + - service + - observability +--- + +# Grafana Alloy + +Unified observability collector for metrics and logs with two deployments: +1. **Indri (host)** - System metrics and service logs from macOS host +2. **Kubernetes (DaemonSet)** - Automatic pod log collection and service health probes + +## Quick Reference + +| Property | Value | +|----------|-------| +| **Indri Binary** | `~/.local/bin/alloy` | +| **Indri Config** | `~/.config/grafana-alloy/config.alloy` | +| **K8s Namespace** | `alloy` | +| **K8s Image** | `grafana/alloy:v1.8.2` | +| **ArgoCD App** | `alloy-k8s` | + +## Metrics Collected + +### From Indri +- System metrics via `prometheus.exporter.unix` +- Textfile collector: `minikube.prom`, `borgmatic.prom`, `zot.prom`, `jellyfin.prom` +- Zot registry metrics from `http://localhost:5050/metrics` +- Pushed to [[prometheus\|Prometheus]] via remote_write + +### From Kubernetes +- All pod logs via `loki.source.kubernetes` +- Service health probes: miniflux, kiwix, transmission, devpi, argocd + +## Logs Collected + +**Brew services:** forgejo, tailscale + +**mcquack LaunchAgents:** alloy, borgmatic, zot, jellyfin + +Logs pushed to [[loki\|Loki]] at `https://loki.tail8d86e.ts.net/loki/api/v1/push`. + +## Why Built from Source + +The Homebrew bottle uses `CGO_ENABLED=0`, which breaks Tailscale MagicDNS. Building with `CGO_ENABLED=1` uses the macOS native resolver. + +## Related + +- [[prometheus\|Prometheus]] - Metrics storage +- [[loki\|Loki]] - Log storage +- [[grafana\|Grafana]] - Visualization diff --git a/docs/reference/services/argocd.md b/docs/reference/services/argocd.md new file mode 100644 index 0000000..874653e --- /dev/null +++ b/docs/reference/services/argocd.md @@ -0,0 +1,50 @@ +--- +title: ArgoCD +tags: + - service + - gitops +--- + +# ArgoCD + +GitOps continuous delivery platform for the [[kubernetes/cluster\|Kubernetes cluster]]. + +## Quick Reference + +| Property | Value | +|----------|-------| +| **URL** | https://argocd.ops.eblu.me | +| **Tailscale URL** | https://argocd.tail8d86e.ts.net | +| **Namespace** | `argocd` | +| **Git Source** | `ssh://forgejo@indri.tail8d86e.ts.net:2200/eblume/blumeops.git` | +| **Manifests Path** | `argocd/` | + +## Sync Policy + +| Application | Sync Policy | Rationale | +|-------------|-------------|-----------| +| `apps` | Automated | Picks up new Application manifests | +| All workloads | Manual | Explicit control over deployments | + +## CLI Commands + +```bash +# Login +argocd login argocd.ops.eblu.me --username admin --password "$(op ...)" + +# Common operations +argocd app list +argocd app diff +argocd app sync +argocd app get +``` + +## Credentials + +- Admin password: 1Password (blumeops vault) +- Git deploy key (SSH): 1Password + +## Related + +- [[kubernetes/apps\|Apps]] - Full application registry +- [[forgejo\|Forgejo]] - Git source diff --git a/docs/reference/services/borgmatic.md b/docs/reference/services/borgmatic.md new file mode 100644 index 0000000..e7afdd4 --- /dev/null +++ b/docs/reference/services/borgmatic.md @@ -0,0 +1,60 @@ +--- +title: Borgmatic +tags: + - service + - backup +--- + +# Borgmatic + +Daily backup system using Borg backup, running on indri. + +## Quick Reference + +| Property | Value | +|----------|-------| +| **Install** | mise (pipx) | +| **Config** | `~/.config/borgmatic/config.yaml` | +| **Schedule** | Daily at 2:00 AM | +| **Repository** | `/Volumes/backups/borg/` on [[storage/sifaka\|Sifaka]] | + +## What Gets Backed Up + +**Directories:** +- `~/code/personal/zk` - Zettelkasten +- `/opt/homebrew/var/forgejo` - Git forge data +- `~/.config/borgmatic` - Borgmatic config +- `~/Documents` - Personal documents +- `~/Pictures` - Photos + +**Databases:** +- `miniflux` on [[postgresql\|PostgreSQL]] +- `teslamate` on [[postgresql\|PostgreSQL]] + +**Not backed up (by design):** +- ZIM archives (re-downloadable) +- Prometheus metrics (ephemeral) +- Loki logs (ephemeral) + +## Retention Policy + +| Period | Count | +|--------|-------| +| Daily | 7 | +| Monthly | 12 | +| Yearly | 1000 | + +## Monitoring + +Metrics exposed via textfile collector to [[prometheus\|Prometheus]]: +- `borgmatic_up` - Repository accessibility +- `borgmatic_last_archive_timestamp` - Last backup time +- `borgmatic_repo_deduplicated_size_bytes` - Disk usage + +Dashboard: "Borgmatic Backups" in [[grafana\|Grafana]] + +## Related + +- [[storage/backups\|Backups]] - Full backup policy +- [[storage/sifaka\|Sifaka]] - Backup target +- [[postgresql\|PostgreSQL]] - Database backups diff --git a/docs/reference/services/forgejo.md b/docs/reference/services/forgejo.md new file mode 100644 index 0000000..c90ff03 --- /dev/null +++ b/docs/reference/services/forgejo.md @@ -0,0 +1,58 @@ +--- +title: Forgejo +tags: + - service + - git + - cicd +--- + +# Forgejo + +Git forge and CI/CD platform. **Primary source of truth for blumeops** (mirrored to GitHub). + +## Quick Reference + +| Property | Value | +|----------|-------| +| **URL** | https://forge.ops.eblu.me | +| **SSH** | `ssh://forgejo@forge.ops.eblu.me:2222` | +| **Local Ports** | 3001 (HTTP), 2200 (SSH) | +| **Config** | `ansible/roles/forgejo/templates/app.ini.j2` | + +## Repositories + +| Repo | Description | +|------|-------------| +| `eblume/blumeops` | Infrastructure as code (primary) | +| `eblume/alloy` | Grafana Alloy fork (CGO build) | +| `eblume/tesla_auth` | Tesla OAuth helper | +| Helm chart mirrors | cloudnative-pg-charts, grafana-helm-charts | + +## CI/CD (Forgejo Actions) + +**Runner:** Kubernetes pod with Docker-in-Docker sidecar +- Namespace: `forgejo-runner` +- Labels: `k8s` +- ArgoCD app: `forgejo-runner` + +**Workflows:** `.forgejo/workflows/` +- `build-container.yaml` - Container image builds on tag + +**Container release:** +```bash +mise run container-list # List containers +mise run container-release runner v1.0.0 # Tag and build +``` + +## Ansible Management + +```bash +mise run provision-indri -- --tags forgejo +``` + +Secrets fetched from 1Password: `lfs-jwt-secret`, `internal-token`, `oauth2-jwt-secret`, `runner_reg` + +## Related + +- [[argocd\|ArgoCD]] - Uses Forgejo as git source +- [[zot\|Zot]] - Container registry for built images diff --git a/docs/reference/services/grafana.md b/docs/reference/services/grafana.md new file mode 100644 index 0000000..f0181f3 --- /dev/null +++ b/docs/reference/services/grafana.md @@ -0,0 +1,50 @@ +--- +title: Grafana +tags: + - service + - observability +--- + +# Grafana + +Dashboards and visualization for BlumeOps observability. + +## Quick Reference + +| Property | Value | +|----------|-------| +| **URL** | https://grafana.ops.eblu.me | +| **Tailscale URL** | https://grafana.tail8d86e.ts.net | +| **Namespace** | `monitoring` | +| **Helm Chart** | grafana (mirrored to forge) | +| **Values** | `argocd/manifests/grafana/values.yaml` | + +## Datasources + +| Name | Type | Target | +|------|------|--------| +| Prometheus | prometheus | `prometheus.monitoring.svc.cluster.local:9090` | +| Loki | loki | `loki.monitoring.svc.cluster.local:3100` | +| TeslaMate | postgres | `blumeops-pg-rw.databases.svc.cluster.local:5432` | + +## Dashboard Provisioning + +Dashboards are ConfigMaps with label `grafana_dashboard: "1"`. + +Location: `argocd/manifests/grafana-config/dashboards/` + +Optional annotation: `grafana_folder: "FolderName"` + +## Key Dashboards + +- macOS System - Host metrics for indri +- Minikube - Kubernetes cluster overview +- Borgmatic Backups - Backup status and trends +- Services Health - HTTP probe results +- TeslaMate (18 dashboards) - Vehicle data + +## Related + +- [[prometheus\|Prometheus]] - Metrics datasource +- [[loki\|Loki]] - Logs datasource +- [[alloy\|Alloy]] - Data collector diff --git a/docs/reference/services/jellyfin.md b/docs/reference/services/jellyfin.md new file mode 100644 index 0000000..7a28b69 --- /dev/null +++ b/docs/reference/services/jellyfin.md @@ -0,0 +1,51 @@ +--- +title: Jellyfin +tags: + - service + - media +--- + +# Jellyfin + +Open-source media server running natively on indri for VideoToolbox hardware transcoding. + +## Quick Reference + +| Property | Value | +|----------|-------| +| **URL** | https://jellyfin.ops.eblu.me | +| **Local Port** | 8096 | +| **Data** | `~/Library/Application Support/jellyfin` | +| **Media** | `/Volumes/allisonflix` (NFS from sifaka) | +| **LaunchAgent** | `mcquack.jellyfin` | + +## Hardware Transcoding + +Apple VideoToolbox on M1 Mac Mini. + +| Codec | Support | +|-------|---------| +| H.264 encode/decode | Hardware | +| HEVC (H.265) encode/decode | Hardware | +| AV1 decode | Software (requires M3+) | +| HDR to SDR tone mapping | VPP (hardware) | + +Concurrent 4K streams with HDR tonemapping: ~3 + +## Configuration + +Dashboard > Playback: +1. Hardware Acceleration: Apple VideoToolbox +2. Allow hardware encoding: Enabled +3. VPP Tone mapping: Enabled + +## Observability + +- Metrics: `jellyfin_metrics` ansible role +- Logs: Forwarded via [[alloy\|Alloy]] +- Dashboard: "Jellyfin Media Server" in [[grafana\|Grafana]] + +## Related + +- [[navidrome\|Navidrome]] - Music streaming +- [[storage/sifaka\|Sifaka]] - Media storage diff --git a/docs/reference/services/kiwix.md b/docs/reference/services/kiwix.md new file mode 100644 index 0000000..9f66d25 --- /dev/null +++ b/docs/reference/services/kiwix.md @@ -0,0 +1,52 @@ +--- +title: Kiwix +tags: + - service + - knowledge +--- + +# Kiwix + +Offline Wikipedia and ZIM archive server. + +## Quick Reference + +| Property | Value | +|----------|-------| +| **URL** | https://kiwix.ops.eblu.me | +| **Tailscale URL** | https://kiwix.tail8d86e.ts.net | +| **Namespace** | `kiwix` | +| **Image** | `ghcr.io/kiwix/kiwix-serve:3.8.1` | +| **Storage** | NFS from [[storage/sifaka\|Sifaka]] (`/volume1/torrents`) | + +## Architecture + +| Component | Purpose | +|-----------|---------| +| kiwix-serve | Serves ZIM files on port 80 | +| torrent-sync | Sidecar syncing ZIM torrents to [[transmission\|Transmission]] | +| zim-watcher | CronJob (hourly) to restart on new ZIMs | + +## Configured Archives + +- Wikipedia top 1M English articles with images +- Project Gutenberg (60,000+ books) +- iFixit repair guides +- Stack Exchange (SuperUser, Math, etc.) +- LibreTexts textbooks +- DevDocs developer documentation + +Full list: `argocd/manifests/kiwix/configmap-zim-torrents.yaml` + +## Adding Archives + +1. Edit `configmap-zim-torrents.yaml` +2. Add torrent URL from https://download.kiwix.org/zim/ +3. Sync: `argocd app sync kiwix` +4. Torrent-sync adds to [[transmission\|Transmission]] +5. zim-watcher restarts kiwix when download completes + +## Related + +- [[transmission\|Transmission]] - Downloads ZIM files +- [[storage/sifaka\|Sifaka]] - ZIM storage diff --git a/docs/reference/services/loki.md b/docs/reference/services/loki.md new file mode 100644 index 0000000..c13fea2 --- /dev/null +++ b/docs/reference/services/loki.md @@ -0,0 +1,51 @@ +--- +title: Loki +tags: + - service + - observability +--- + +# Loki + +Log aggregation system for BlumeOps infrastructure. + +## Quick Reference + +| Property | Value | +|----------|-------| +| **URL** | https://loki.ops.eblu.me | +| **Tailscale URL** | https://loki.tail8d86e.ts.net | +| **Namespace** | `monitoring` | +| **Image** | `grafana/loki:3.4.2` | +| **Storage** | 50Gi PVC | +| **Retention** | 31 days | + +## Architecture + +- Single-node deployment with filesystem storage +- TSDB index with 24h period +- Logs collected by [[alloy\|Alloy]] and pushed via Loki API +- Queried via [[grafana\|Grafana]] + +## Log Sources + +**From Indri (via Alloy):** +- forgejo, tailscale (brew services) +- alloy, borgmatic, zot, jellyfin (LaunchAgents) + +**From Kubernetes (via Alloy DaemonSet):** +- All pods in all namespaces + +## Query Examples (LogQL) + +```logql +{service="forgejo"} # All forgejo logs +{service="borgmatic", stream="stderr"} # Borgmatic errors +{host="indri"} |= "error" # All logs containing "error" +``` + +## Related + +- [[alloy\|Alloy]] - Log collector +- [[grafana\|Grafana]] - Log visualization +- [[prometheus\|Prometheus]] - Metrics counterpart diff --git a/docs/reference/services/miniflux.md b/docs/reference/services/miniflux.md new file mode 100644 index 0000000..543e59f --- /dev/null +++ b/docs/reference/services/miniflux.md @@ -0,0 +1,49 @@ +--- +title: Miniflux +tags: + - service + - rss +--- + +# Miniflux + +Minimalist RSS/Atom feed reader. + +## Quick Reference + +| Property | Value | +|----------|-------| +| **URL** | https://feed.ops.eblu.me | +| **Tailscale URL** | https://feed.tail8d86e.ts.net | +| **Namespace** | `miniflux` | +| **Image** | `ghcr.io/miniflux/miniflux:latest` | +| **Database** | [[postgresql\|PostgreSQL]] | + +## Features + +- Keyboard shortcuts for efficient reading +- Fever and Google Reader API compatible +- Mobile-friendly web interface +- OPML import/export +- Content scraping for full articles + +## Database + +Uses CloudNativePG cluster at `pg.ops.eblu.me`. + +Database user password stored in `blumeops-pg-app` secret (auto-generated by CNPG). + +## Backup + +Feed subscriptions and read state backed up via [[borgmatic\|Borgmatic]] PostgreSQL hook. + +## Health Check + +```bash +curl https://feed.ops.eblu.me/healthcheck +``` + +## Related + +- [[postgresql\|PostgreSQL]] - Database backend +- [[borgmatic\|Borgmatic]] - Data backup diff --git a/docs/reference/services/navidrome.md b/docs/reference/services/navidrome.md new file mode 100644 index 0000000..ccfac06 --- /dev/null +++ b/docs/reference/services/navidrome.md @@ -0,0 +1,52 @@ +--- +title: Navidrome +tags: + - service + - media +--- + +# Navidrome + +Self-hosted music streaming server. + +## Quick Reference + +| Property | Value | +|----------|-------| +| **URL** | https://dj.ops.eblu.me | +| **Tailscale URL** | https://dj.tail8d86e.ts.net | +| **Namespace** | `navidrome` | +| **Manifests** | `argocd/manifests/navidrome/` | + +## Storage + +| Mount | Type | Source | Access | +|-------|------|--------|--------| +| /music | NFS PV | sifaka:/volume1/music | Read-only | +| /data | Local PVC (10Gi) | minikube storage | Read-write | + +The `/data` directory contains SQLite database, configuration, and cache. + +## Configuration + +| Variable | Value | +|----------|-------| +| `ND_SCANSCHEDULE` | 1h | +| `ND_LOGLEVEL` | info | +| `ND_MUSICFOLDER` | /music | +| `ND_DATAFOLDER` | /data | + +## Initial Setup + +On first access, Navidrome prompts to create an admin user. No default credentials. + +## Verify NFS Mount + +```bash +kubectl --context=minikube-indri -n navidrome exec deploy/navidrome -- ls /music +``` + +## Related + +- [[jellyfin\|Jellyfin]] - Video streaming +- [[storage/sifaka\|Sifaka]] - Music storage diff --git a/docs/reference/services/postgresql.md b/docs/reference/services/postgresql.md new file mode 100644 index 0000000..ae46b78 --- /dev/null +++ b/docs/reference/services/postgresql.md @@ -0,0 +1,68 @@ +--- +title: PostgreSQL +tags: + - service + - database +--- + +# PostgreSQL + +Database cluster via CloudNativePG operator. + +## Quick Reference + +| Property | Value | +|----------|-------| +| **URL** | `tcp://pg.ops.eblu.me:5432` | +| **Metrics** | `http://cnpg-metrics.tail8d86e.ts.net:9187/metrics` | +| **Namespace** | `databases` | +| **Cluster** | `blumeops-pg` | +| **Operator** | CloudNativePG | + +## Databases + +| Database | Owner | Purpose | +|----------|-------|---------| +| miniflux | miniflux | [[miniflux\|Miniflux]] feed data | +| teslamate | teslamate | [[teslamate\|TeslaMate]] vehicle data | + +## Users + +| User | Role | Purpose | +|------|------|---------| +| postgres | superuser | CNPG internal | +| miniflux | app owner | Owns miniflux database | +| teslamate | superuser | TeslaMate (needs extensions) | +| eblume | superuser | Admin access | +| borgmatic | pg_read_all_data | [[borgmatic\|Backup]] access | + +## Quick Connect + +```bash +PGPASSWORD=$(op --vault blumeops item get --fields password --reveal) \ + psql -h pg.ops.eblu.me -U eblume -d miniflux +``` + +## Backup + +Backed up via [[borgmatic\|Borgmatic]] `postgresql_databases` hook. + +Borgmatic streams `pg_dump` directly to Borg (no intermediate files, no downtime). + +## Credentials + +**1Password items:** +- `guxu3j7ajhjyey6xxl2ovsl2ui` - eblume password +- `mw2bv5we7woicjza7hc6s44yvy` - borgmatic password + +**CNPG-managed secrets:** +- `blumeops-pg-app` - miniflux user +- `blumeops-pg-eblume` - eblume superuser +- `blumeops-pg-borgmatic` - borgmatic backup user +- `blumeops-pg-teslamate` - teslamate user + +## Related + +- [[miniflux\|Miniflux]] - Feed reader database +- [[teslamate\|TeslaMate]] - Vehicle data database +- [[borgmatic\|Borgmatic]] - Database backup diff --git a/docs/reference/services/prometheus.md b/docs/reference/services/prometheus.md new file mode 100644 index 0000000..75a7379 --- /dev/null +++ b/docs/reference/services/prometheus.md @@ -0,0 +1,54 @@ +--- +title: Prometheus +tags: + - service + - observability +--- + +# Prometheus + +Metrics storage and querying for BlumeOps infrastructure. + +## Quick Reference + +| Property | Value | +|----------|-------| +| **URL** | https://prometheus.ops.eblu.me | +| **Tailscale URL** | https://prometheus.tail8d86e.ts.net | +| **Namespace** | `monitoring` | +| **Image** | `prom/prometheus:v3.2.1` | +| **Storage** | 50Gi PVC | + +## Data Sources + +### Remote Write (from Alloy) +- Indri system metrics via [[alloy\|Alloy]] remote_write +- Textfile metrics: minikube, borgmatic, zot, jellyfin + +### Scrape Targets +| Target | Metrics | +|--------|---------| +| `sifaka:9100` | [[storage/sifaka\|Sifaka]] NAS (node_exporter) | +| `cnpg-metrics.tail8d86e.ts.net:9187` | [[postgresql\|CloudNativePG]] metrics | +| `kube-state-metrics.monitoring.svc:8080` | Kubernetes resource metrics | + +## Query API + +```bash +# Check targets +curl -s https://prometheus.ops.eblu.me/api/v1/targets | jq '.data.activeTargets[].scrapeUrl' +``` + +## ArgoCD Management + +```bash +argocd app sync prometheus +``` + +Manifests: `argocd/manifests/prometheus/` + +## Related + +- [[alloy\|Alloy]] - Metrics collector +- [[grafana\|Grafana]] - Visualization +- [[loki\|Loki]] - Logs counterpart diff --git a/docs/reference/services/teslamate.md b/docs/reference/services/teslamate.md new file mode 100644 index 0000000..34b8a66 --- /dev/null +++ b/docs/reference/services/teslamate.md @@ -0,0 +1,58 @@ +--- +title: TeslaMate +tags: + - service + - vehicle +--- + +# TeslaMate + +Self-hosted Tesla data logger collecting vehicle telemetry from the Tesla Owner API. + +## Quick Reference + +| Property | Value | +|----------|-------| +| **URL** | https://tesla.ops.eblu.me | +| **Tailscale URL** | https://tesla.tail8d86e.ts.net | +| **Namespace** | `teslamate` | +| **Image** | `teslamate/teslamate:2.2.0` | +| **Database** | [[postgresql\|PostgreSQL]] | + +## Data Collected + +- Battery level, state of charge, range estimates +- Charging sessions (location, energy, cost, duration) +- Drives (distance, efficiency, routes) +- Climate/HVAC usage +- Software update history +- Vampire drain analysis +- Vehicle states (asleep, driving, charging, online) + +## Grafana Dashboards + +18 dashboards in the "TeslaMate" folder: +- Overview, Charges, Drives, Efficiency, States +- Battery Health, Vampire Drain, Statistics +- Charge Level, Locations, Trip, Mileage +- Drive Stats, Charging Stats, Projected Range +- Timeline, Updates, Visited + +Dashboards use PostgreSQL datasource (not Prometheus). + +## Authentication + +Uses Tesla Owner API via OAuth: +1. Access https://tesla.ops.eblu.me +2. Click "Sign in with Tesla" +3. Tokens encrypted with ENCRYPTION_KEY + +## Credentials + +**1Password:** `TeslaMate` item with `db_password` and `api_enc_key` + +## Related + +- [[postgresql\|PostgreSQL]] - Data storage +- [[grafana\|Grafana]] - Dashboards +- [[borgmatic\|Borgmatic]] - Database backup diff --git a/docs/reference/services/transmission.md b/docs/reference/services/transmission.md new file mode 100644 index 0000000..7ef6ebc --- /dev/null +++ b/docs/reference/services/transmission.md @@ -0,0 +1,53 @@ +--- +title: Transmission +tags: + - service + - torrent +--- + +# Transmission + +BitTorrent daemon, primarily for downloading ZIM archives for [[kiwix\|Kiwix]]. + +## Quick Reference + +| Property | Value | +|----------|-------| +| **URL** | https://torrent.ops.eblu.me | +| **Tailscale URL** | https://torrent.tail8d86e.ts.net | +| **Namespace** | `torrent` | +| **Image** | `lscr.io/linuxserver/transmission:latest` | +| **Storage** | NFS PVC from [[storage/sifaka\|Sifaka]] | + +## Storage Layout + +NFS share on sifaka (`/volume1/torrents`): + +| Path | Purpose | +|------|---------| +| `/downloads/` | Active downloads and metadata | +| `/downloads/complete/` | Completed downloads | +| `/config/` | Transmission configuration | +| `/watch/` | Watch directory for .torrent files | + +[[kiwix\|Kiwix]] reads from `/downloads/complete/` to serve ZIM archives. + +## Integration with Kiwix + +The Kiwix deployment includes a torrent-sync sidecar that: +1. Reads ZIM torrent list from ConfigMap +2. Adds missing torrents via RPC +3. Runs on startup and every 30 minutes + +When downloads complete, the zim-watcher CronJob detects new ZIMs and restarts Kiwix. + +## Monitoring + +Basic uptime via blackbox probe in [[alloy\|Alloy]] k8s (Services Health dashboard). + +Web UI shows: active/seeding/paused counts, speeds, disk usage. + +## Related + +- [[kiwix\|Kiwix]] - ZIM archive consumer +- [[storage/sifaka\|Sifaka]] - Download storage diff --git a/docs/reference/services/zot.md b/docs/reference/services/zot.md new file mode 100644 index 0000000..a09b362 --- /dev/null +++ b/docs/reference/services/zot.md @@ -0,0 +1,66 @@ +--- +title: Zot +tags: + - service + - registry +--- + +# Zot + +OCI-native container registry providing pull-through cache and private image storage. + +## Quick Reference + +| Property | Value | +|----------|-------| +| **URL** | https://registry.ops.eblu.me | +| **Local Port** | 5050 | +| **Data** | `~/zot` | +| **Config** | `~/.config/zot/config.json` | +| **LaunchAgent** | mcquack | + +## Namespace Convention + +| Path | Source | +|------|--------| +| `registry.ops.eblu.me/docker.io/*` | Cached from Docker Hub | +| `registry.ops.eblu.me/ghcr.io/*` | Cached from GHCR | +| `registry.ops.eblu.me/quay.io/*` | Cached from Quay | +| `registry.ops.eblu.me/blumeops/*` | Private images | + +## Pull-Through Cache + +When [[kubernetes/cluster\|minikube]] pulls an image: +1. Containerd checks zot first (`host.minikube.internal:5050`) +2. If cached, returns immediately +3. If not, zot fetches from upstream, caches, returns + +## Private Images + +```bash +# Build and push from gilbert +podman build -t registry.ops.eblu.me/blumeops/myapp:v1 . +podman push registry.ops.eblu.me/blumeops/myapp:v1 + +# Use in k8s manifest +image: registry.ops.eblu.me/blumeops/myapp:v1 +``` + +## Security Model + +Network access only (no authentication). Defense is the Tailscale ACL boundary. + +## Useful Commands + +```bash +# List all images +curl -s http://indri:5050/v2/_catalog | jq + +# List tags +curl -s http://indri:5050/v2/blumeops/devpi/tags/list | jq +``` + +## Related + +- [[forgejo\|Forgejo]] - Container build CI +- [[kubernetes/cluster\|Cluster]] - Registry consumer diff --git a/docs/reference/storage/backups.md b/docs/reference/storage/backups.md new file mode 100644 index 0000000..c35095e --- /dev/null +++ b/docs/reference/storage/backups.md @@ -0,0 +1,84 @@ +--- +title: Backup Policy +tags: + - storage + - backup +--- + +# Backup Policy + +Daily automated backups from [[infrastructure/hosts\|Indri]] to [[storage/sifaka\|Sifaka]] NAS. + +## Schedule + +| Time | Frequency | System | +|------|-----------|--------| +| 2:00 AM | Daily | [[services/borgmatic\|Borgmatic]] | + +## What Gets Backed Up + +### Directories + +| Path | Description | Priority | +|------|-------------|----------| +| `~/code/personal/zk` | Zettelkasten notes | Critical | +| `/opt/homebrew/var/forgejo` | Git repositories | Critical | +| `~/.config/borgmatic` | Backup config | High | +| `~/Documents` | Personal documents | High | +| `~/Pictures` | Photos | Medium | + +### Databases + +| Database | Host | Method | +|----------|------|--------| +| miniflux | [[services/postgresql\|pg.ops.eblu.me]] | pg_dump stream | +| teslamate | [[services/postgresql\|pg.ops.eblu.me]] | pg_dump stream | + +## What Is NOT Backed Up + +| Data | Reason | +|------|--------| +| ZIM archives (`~/transmission/`) | Re-downloadable via torrent | +| Prometheus metrics | Ephemeral, in k8s PVC | +| Loki logs | Ephemeral, in k8s PVC | +| devpi cache | Re-fetchable from PyPI | + +## Retention Policy + +| Period | Retention | +|--------|-----------| +| Daily | 7 backups | +| Monthly | 12 backups | +| Yearly | 1000 backups | + +## Backup Target + +Repository: `/Volumes/backups/borg/` on [[storage/sifaka\|Sifaka]] + +## Monitoring + +Metrics exposed to [[services/prometheus\|Prometheus]]: +- `borgmatic_up` - Repository accessible +- `borgmatic_last_archive_timestamp` - Last backup time +- `borgmatic_repo_deduplicated_size_bytes` - Disk usage + +Dashboard: "Borgmatic Backups" in [[services/grafana\|Grafana]] + +## Recovery + +```bash +# List archives +ssh indri 'mise x -- borgmatic list' + +# Extract specific path from latest +ssh indri 'mise x -- borgmatic extract --archive latest --path /some/path' + +# Check repository health +ssh indri 'mise x -- borgmatic check' +``` + +## Related + +- [[services/borgmatic\|Borgmatic]] - Backup system details +- [[storage/sifaka\|Sifaka]] - Backup storage +- [[services/postgresql\|PostgreSQL]] - Database backups diff --git a/docs/reference/storage/sifaka.md b/docs/reference/storage/sifaka.md new file mode 100644 index 0000000..2e18793 --- /dev/null +++ b/docs/reference/storage/sifaka.md @@ -0,0 +1,51 @@ +--- +title: Sifaka NAS +tags: + - storage +--- + +# Sifaka NAS + +Synology NAS providing network storage and backup target. + +## Quick Reference + +| Property | Value | +|----------|-------| +| **Dashboard** | https://nas.ops.eblu.me | +| **Model** | Synology | +| **Storage** | 10.9TB RAID 5 | +| **Role** | Backup target, media storage | + +## Network Shares + +| Share | Path | Purpose | Consumers | +|-------|------|---------|-----------| +| backups | `/volume1/backups` | Borg backup repository | [[services/borgmatic\|Borgmatic]] | +| torrents | `/volume1/torrents` | ZIM downloads | [[services/kiwix\|Kiwix]], [[services/transmission\|Transmission]] | +| music | `/volume1/music` | Music library | [[services/navidrome\|Navidrome]] | +| allisonflix | `/volume1/allisonflix` | Video library | [[services/jellyfin\|Jellyfin]] | +| photos | `/volume1/photos` | Photo library | Immich | + +## NFS Exports + +| Export | Allowed Clients | Purpose | +|--------|-----------------|---------| +| `/volume1/torrents` | 192.168.1.0/24, 100.64.0.0/10 | k8s pods via Docker NAT | +| `/volume1/music` | 192.168.1.0/24, 100.64.0.0/10 | k8s pods via Docker NAT | + +## Monitoring + +Node exporter running in Docker container, scraped by [[services/prometheus\|Prometheus]] at `sifaka:9100`. + +## Tailscale + +- Tag: `tag:nas` +- ACL: `tag:homelab` can access for backups + +## Related + +- [[storage/backups\|Backups]] - Backup policy +- [[services/borgmatic\|Borgmatic]] - Backup system +- [[services/jellyfin\|Jellyfin]] - Media consumer +- [[services/navidrome\|Navidrome]] - Music consumer -- 2.50.1 (Apple Git-155) From 649ae15204f9f46593f6ac93ba5f22b60682f474 Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 3 Feb 2026 13:33:14 -0800 Subject: [PATCH 02/12] Fix escaped pipes in wiki-links Remove backslash escaping from pipe characters in wiki-links. Correct: [[services/forgejo|Forgejo]] Wrong: [[services/forgejo\|Forgejo]] Co-Authored-By: Claude Opus 4.5 --- docs/reference/index.md | 46 +++++++++++----------- docs/reference/infrastructure/hosts.md | 18 ++++----- docs/reference/infrastructure/routing.md | 32 +++++++-------- docs/reference/infrastructure/tailscale.md | 6 +-- docs/reference/kubernetes/apps.md | 38 +++++++++--------- docs/reference/kubernetes/cluster.md | 12 +++--- docs/reference/services/1password.md | 4 +- docs/reference/services/alloy.md | 10 ++--- docs/reference/services/argocd.md | 6 +-- docs/reference/services/borgmatic.md | 16 ++++---- docs/reference/services/forgejo.md | 4 +- docs/reference/services/grafana.md | 6 +-- docs/reference/services/jellyfin.md | 8 ++-- docs/reference/services/kiwix.md | 10 ++--- docs/reference/services/loki.md | 10 ++--- docs/reference/services/miniflux.md | 8 ++-- docs/reference/services/navidrome.md | 4 +- docs/reference/services/postgresql.md | 14 +++---- docs/reference/services/prometheus.md | 12 +++--- docs/reference/services/teslamate.md | 8 ++-- docs/reference/services/transmission.md | 12 +++--- docs/reference/services/zot.md | 6 +-- docs/reference/storage/backups.md | 20 +++++----- docs/reference/storage/sifaka.md | 18 ++++----- 24 files changed, 164 insertions(+), 164 deletions(-) diff --git a/docs/reference/index.md b/docs/reference/index.md index 1662f1b..c8da610 100644 --- a/docs/reference/index.md +++ b/docs/reference/index.md @@ -14,41 +14,41 @@ Individual service reference cards with URLs, configuration, and operational det | Service | Description | Location | |---------|-------------|----------| -| [[services/alloy\|Alloy]] | Observability collector (metrics & logs) | indri + k8s | -| [[services/argocd\|ArgoCD]] | GitOps continuous delivery | k8s | -| [[services/borgmatic\|Borgmatic]] | Backup system | indri | -| [[services/1password\|1Password]] | Secrets management | cloud + k8s | -| [[services/forgejo\|Forgejo]] | Git forge & CI/CD | indri | -| [[services/grafana\|Grafana]] | Dashboards & visualization | k8s | -| [[services/jellyfin\|Jellyfin]] | Media server | indri | -| [[services/kiwix\|Kiwix]] | Offline Wikipedia & ZIM archives | k8s | -| [[services/loki\|Loki]] | Log aggregation | k8s | -| [[services/miniflux\|Miniflux]] | RSS feed reader | k8s | -| [[services/navidrome\|Navidrome]] | Music streaming | k8s | -| [[services/postgresql\|PostgreSQL]] | Database cluster | k8s | -| [[services/prometheus\|Prometheus]] | Metrics collection | k8s | -| [[services/teslamate\|TeslaMate]] | Tesla data logger | k8s | -| [[services/transmission\|Transmission]] | BitTorrent daemon | k8s | -| [[services/zot\|Zot]] | Container registry | indri | +| [[services/alloy|Alloy]] | Observability collector (metrics & logs) | indri + k8s | +| [[services/argocd|ArgoCD]] | GitOps continuous delivery | k8s | +| [[services/borgmatic|Borgmatic]] | Backup system | indri | +| [[services/1password|1Password]] | Secrets management | cloud + k8s | +| [[services/forgejo|Forgejo]] | Git forge & CI/CD | indri | +| [[services/grafana|Grafana]] | Dashboards & visualization | k8s | +| [[services/jellyfin|Jellyfin]] | Media server | indri | +| [[services/kiwix|Kiwix]] | Offline Wikipedia & ZIM archives | k8s | +| [[services/loki|Loki]] | Log aggregation | k8s | +| [[services/miniflux|Miniflux]] | RSS feed reader | k8s | +| [[services/navidrome|Navidrome]] | Music streaming | k8s | +| [[services/postgresql|PostgreSQL]] | Database cluster | k8s | +| [[services/prometheus|Prometheus]] | Metrics collection | k8s | +| [[services/teslamate|TeslaMate]] | Tesla data logger | k8s | +| [[services/transmission|Transmission]] | BitTorrent daemon | k8s | +| [[services/zot|Zot]] | Container registry | indri | ## Infrastructure Host inventory and network configuration. -- [[infrastructure/hosts\|Hosts]] - Device inventory (indri, gilbert, sifaka, etc.) -- [[infrastructure/tailscale\|Tailscale]] - ACLs, groups, tags -- [[infrastructure/routing\|Routing]] - DNS domains, port mappings +- [[infrastructure/hosts|Hosts]] - Device inventory (indri, gilbert, sifaka, etc.) +- [[infrastructure/tailscale|Tailscale]] - ACLs, groups, tags +- [[infrastructure/routing|Routing]] - DNS domains, port mappings ## Kubernetes Cluster configuration and application registry. -- [[kubernetes/cluster\|Cluster]] - Minikube specs, storage, networking -- [[kubernetes/apps\|Apps]] - ArgoCD application registry +- [[kubernetes/cluster|Cluster]] - Minikube specs, storage, networking +- [[kubernetes/apps|Apps]] - ArgoCD application registry ## Storage Network storage and backup configuration. -- [[storage/sifaka\|Sifaka]] - Synology NAS configuration -- [[storage/backups\|Backups]] - Backup policy and schedule +- [[storage/sifaka|Sifaka]] - Synology NAS configuration +- [[storage/backups|Backups]] - Backup policy and schedule diff --git a/docs/reference/infrastructure/hosts.md b/docs/reference/infrastructure/hosts.md index 9589b40..cea3105 100644 --- a/docs/reference/infrastructure/hosts.md +++ b/docs/reference/infrastructure/hosts.md @@ -13,7 +13,7 @@ All devices connected via [Tailscale](https://login.tailscale.com/) tailnet `tai | Host | Description | Notes | |------|-------------|-------| | **Indri** | Mac Mini M1, 2020 | Primary server, 2TB internal disk | -| **[[storage/sifaka\|Sifaka]]** | Synology NAS | 10.9TB RAID 5, backup target | +| **[[storage/sifaka|Sifaka]]** | Synology NAS | 10.9TB RAID 5, backup target | | **Gilbert** | 13" MacBook Air M4, 2025 | Primary workstation | | **Mouse** | 13" MacBook Air M2 | Allison's laptop | | **UniFi** | UniFi Express 7 | Home WiFi network | @@ -32,15 +32,15 @@ All devices connected via [Tailscale](https://login.tailscale.com/) tailnet `tai ### Services Hosted **Native (via Ansible):** -- [[services/forgejo\|Forgejo]] - Git forge -- [[services/zot\|Zot]] - Container registry -- [[services/jellyfin\|Jellyfin]] - Media server -- [[services/borgmatic\|Borgmatic]] - Backup system -- [[services/alloy\|Alloy]] - Metrics/logs collector +- [[services/forgejo|Forgejo]] - Git forge +- [[services/zot|Zot]] - Container registry +- [[services/jellyfin|Jellyfin]] - Media server +- [[services/borgmatic|Borgmatic]] - Backup system +- [[services/alloy|Alloy]] - Metrics/logs collector - Caddy - Reverse proxy **Kubernetes (via minikube):** -- [[kubernetes/apps\|All k8s applications]] +- [[kubernetes/apps|All k8s applications]] ### Sleep Prevention @@ -67,5 +67,5 @@ Fish abbreviations: ## Related -- [[infrastructure/tailscale\|Tailscale]] - Network configuration -- [[storage/sifaka\|Sifaka]] - NAS details +- [[infrastructure/tailscale|Tailscale]] - Network configuration +- [[storage/sifaka|Sifaka]] - NAS details diff --git a/docs/reference/infrastructure/routing.md b/docs/reference/infrastructure/routing.md index 51d7133..b6042a2 100644 --- a/docs/reference/infrastructure/routing.md +++ b/docs/reference/infrastructure/routing.md @@ -25,20 +25,20 @@ DNS points to indri's Tailscale IP (100.98.163.89). TLS via Let's Encrypt (ACME | Service | URL | Description | |---------|-----|-------------| | Homepage | https://go.ops.eblu.me | Service dashboard | -| [[services/forgejo\|Forgejo]] | https://forge.ops.eblu.me | Git hosting (SSH: 2222) | -| [[services/zot\|Zot]] | https://registry.ops.eblu.me | Container registry | -| [[services/grafana\|Grafana]] | https://grafana.ops.eblu.me | Dashboards | -| [[services/argocd\|ArgoCD]] | https://argocd.ops.eblu.me | GitOps CD | -| [[services/prometheus\|Prometheus]] | https://prometheus.ops.eblu.me | Metrics | -| [[services/loki\|Loki]] | https://loki.ops.eblu.me | Logs | -| [[services/miniflux\|Miniflux]] | https://feed.ops.eblu.me | RSS reader | -| [[services/kiwix\|Kiwix]] | https://kiwix.ops.eblu.me | Offline Wikipedia | -| [[services/transmission\|Transmission]] | https://torrent.ops.eblu.me | BitTorrent | -| [[services/teslamate\|TeslaMate]] | https://tesla.ops.eblu.me | Tesla logger | -| [[services/navidrome\|Navidrome]] | https://dj.ops.eblu.me | Music streaming | -| [[services/jellyfin\|Jellyfin]] | https://jellyfin.ops.eblu.me | Media server | -| [[services/postgresql\|PostgreSQL]] | pg.ops.eblu.me:5432 | Database | -| [[storage/sifaka\|Sifaka]] | https://nas.ops.eblu.me | NAS dashboard | +| [[services/forgejo|Forgejo]] | https://forge.ops.eblu.me | Git hosting (SSH: 2222) | +| [[services/zot|Zot]] | https://registry.ops.eblu.me | Container registry | +| [[services/grafana|Grafana]] | https://grafana.ops.eblu.me | Dashboards | +| [[services/argocd|ArgoCD]] | https://argocd.ops.eblu.me | GitOps CD | +| [[services/prometheus|Prometheus]] | https://prometheus.ops.eblu.me | Metrics | +| [[services/loki|Loki]] | https://loki.ops.eblu.me | Logs | +| [[services/miniflux|Miniflux]] | https://feed.ops.eblu.me | RSS reader | +| [[services/kiwix|Kiwix]] | https://kiwix.ops.eblu.me | Offline Wikipedia | +| [[services/transmission|Transmission]] | https://torrent.ops.eblu.me | BitTorrent | +| [[services/teslamate|TeslaMate]] | https://tesla.ops.eblu.me | Tesla logger | +| [[services/navidrome|Navidrome]] | https://dj.ops.eblu.me | Music streaming | +| [[services/jellyfin|Jellyfin]] | https://jellyfin.ops.eblu.me | Media server | +| [[services/postgresql|PostgreSQL]] | pg.ops.eblu.me:5432 | Database | +| [[storage/sifaka|Sifaka]] | https://nas.ops.eblu.me | NAS dashboard | ## Tailscale-Only Services @@ -75,5 +75,5 @@ DNS points to indri's Tailscale IP (100.98.163.89). TLS via Let's Encrypt (ACME ## Related -- [[infrastructure/tailscale\|Tailscale]] - ACL configuration -- [[infrastructure/hosts\|Hosts]] - Where services run +- [[infrastructure/tailscale|Tailscale]] - ACL configuration +- [[infrastructure/hosts|Hosts]] - Where services run diff --git a/docs/reference/infrastructure/tailscale.md b/docs/reference/infrastructure/tailscale.md index 7e2d49a..a852ddb 100644 --- a/docs/reference/infrastructure/tailscale.md +++ b/docs/reference/infrastructure/tailscale.md @@ -22,7 +22,7 @@ mise run tailnet-up # Apply changes | Group | Members | Purpose | |-------|---------|---------| -| `group:allisonflix` | admin, member | [[services/jellyfin\|Jellyfin]] media access | +| `group:allisonflix` | admin, member | [[services/jellyfin|Jellyfin]] media access | ## Device Tags @@ -63,5 +63,5 @@ Pulumi uses OAuth client from 1Password (blumeops vault): ## Related -- [[infrastructure/routing\|Routing]] - Service URLs -- [[infrastructure/hosts\|Hosts]] - Device inventory +- [[infrastructure/routing|Routing]] - Service URLs +- [[infrastructure/hosts|Hosts]] - Device inventory diff --git a/docs/reference/kubernetes/apps.md b/docs/reference/kubernetes/apps.md index de6f679..c10bbd8 100644 --- a/docs/reference/kubernetes/apps.md +++ b/docs/reference/kubernetes/apps.md @@ -7,32 +7,32 @@ tags: # ArgoCD Applications -Registry of all applications deployed via [[services/argocd\|ArgoCD]]. +Registry of all applications deployed via [[services/argocd|ArgoCD]]. ## Application Registry | App | Namespace | Path/Source | Service | |-----|-----------|-------------|---------| | `apps` | argocd | `argocd/apps/` | App-of-apps root | -| `argocd` | argocd | `argocd/manifests/argocd/` | [[services/argocd\|ArgoCD]] | +| `argocd` | argocd | `argocd/manifests/argocd/` | [[services/argocd|ArgoCD]] | | `tailscale-operator` | tailscale | `argocd/manifests/tailscale-operator/` | Tailscale k8s operator | -| `1password-connect` | 1password | `argocd/manifests/1password-connect/` | [[services/1password\|1Password]] | -| `external-secrets` | external-secrets | Helm chart | [[services/1password\|1Password]] | -| `external-secrets-config` | external-secrets | `argocd/manifests/external-secrets-config/` | [[services/1password\|1Password]] | +| `1password-connect` | 1password | `argocd/manifests/1password-connect/` | [[services/1password|1Password]] | +| `external-secrets` | external-secrets | Helm chart | [[services/1password|1Password]] | +| `external-secrets-config` | external-secrets | `argocd/manifests/external-secrets-config/` | [[services/1password|1Password]] | | `cloudnative-pg` | cnpg-system | Helm chart (forge mirror) | PostgreSQL operator | -| `blumeops-pg` | databases | `argocd/manifests/databases/` | [[services/postgresql\|PostgreSQL]] | -| `prometheus` | monitoring | `argocd/manifests/prometheus/` | [[services/prometheus\|Prometheus]] | -| `loki` | monitoring | `argocd/manifests/loki/` | [[services/loki\|Loki]] | -| `grafana` | monitoring | Helm chart (forge mirror) | [[services/grafana\|Grafana]] | -| `grafana-config` | monitoring | `argocd/manifests/grafana-config/` | [[services/grafana\|Grafana]] | -| `alloy-k8s` | alloy | `argocd/manifests/alloy-k8s/` | [[services/alloy\|Alloy]] | +| `blumeops-pg` | databases | `argocd/manifests/databases/` | [[services/postgresql|PostgreSQL]] | +| `prometheus` | monitoring | `argocd/manifests/prometheus/` | [[services/prometheus|Prometheus]] | +| `loki` | monitoring | `argocd/manifests/loki/` | [[services/loki|Loki]] | +| `grafana` | monitoring | Helm chart (forge mirror) | [[services/grafana|Grafana]] | +| `grafana-config` | monitoring | `argocd/manifests/grafana-config/` | [[services/grafana|Grafana]] | +| `alloy-k8s` | alloy | `argocd/manifests/alloy-k8s/` | [[services/alloy|Alloy]] | | `kube-state-metrics` | monitoring | `argocd/manifests/kube-state-metrics/` | K8s metrics | -| `miniflux` | miniflux | `argocd/manifests/miniflux/` | [[services/miniflux\|Miniflux]] | -| `kiwix` | kiwix | `argocd/manifests/kiwix/` | [[services/kiwix\|Kiwix]] | -| `torrent` | torrent | `argocd/manifests/torrent/` | [[services/transmission\|Transmission]] | -| `navidrome` | navidrome | `argocd/manifests/navidrome/` | [[services/navidrome\|Navidrome]] | -| `teslamate` | teslamate | `argocd/manifests/teslamate/` | [[services/teslamate\|TeslaMate]] | -| `forgejo-runner` | forgejo-runner | `argocd/manifests/forgejo-runner/` | [[services/forgejo\|Forgejo]] CI | +| `miniflux` | miniflux | `argocd/manifests/miniflux/` | [[services/miniflux|Miniflux]] | +| `kiwix` | kiwix | `argocd/manifests/kiwix/` | [[services/kiwix|Kiwix]] | +| `torrent` | torrent | `argocd/manifests/torrent/` | [[services/transmission|Transmission]] | +| `navidrome` | navidrome | `argocd/manifests/navidrome/` | [[services/navidrome|Navidrome]] | +| `teslamate` | teslamate | `argocd/manifests/teslamate/` | [[services/teslamate|TeslaMate]] | +| `forgejo-runner` | forgejo-runner | `argocd/manifests/forgejo-runner/` | [[services/forgejo|Forgejo]] CI | ## Sync Policies @@ -61,5 +61,5 @@ argocd app sync # Deploy changes ## Related -- [[services/argocd\|ArgoCD]] - GitOps platform details -- [[kubernetes/cluster\|Cluster]] - Kubernetes infrastructure +- [[services/argocd|ArgoCD]] - GitOps platform details +- [[kubernetes/cluster|Cluster]] - Kubernetes infrastructure diff --git a/docs/reference/kubernetes/cluster.md b/docs/reference/kubernetes/cluster.md index f904fac..b796987 100644 --- a/docs/reference/kubernetes/cluster.md +++ b/docs/reference/kubernetes/cluster.md @@ -6,7 +6,7 @@ tags: # Kubernetes Cluster -Single-node Minikube cluster running on [[infrastructure/hosts\|Indri]]. +Single-node Minikube cluster running on [[infrastructure/hosts|Indri]]. ## Cluster Specifications @@ -36,7 +36,7 @@ Fish abbreviations: ## Volume Mounting -Pods mount NFS directly from [[storage/sifaka\|Sifaka]]: +Pods mount NFS directly from [[storage/sifaka|Sifaka]]: ```yaml volumes: @@ -50,7 +50,7 @@ Docker NATs outbound traffic through indri's LAN IP (192.168.1.50), allowing acc ## Registry Mirror -Containerd uses [[services/zot\|Zot]] as a pull-through cache: +Containerd uses [[services/zot|Zot]] as a pull-through cache: - Endpoint: `host.minikube.internal:5050` - Config: `/etc/containerd/certs.d//hosts.toml` @@ -68,6 +68,6 @@ minikube logs # View logs ## Related -- [[kubernetes/apps\|Apps]] - ArgoCD applications -- [[services/argocd\|ArgoCD]] - GitOps deployment -- [[services/zot\|Zot]] - Registry mirror +- [[kubernetes/apps|Apps]] - ArgoCD applications +- [[services/argocd|ArgoCD]] - GitOps deployment +- [[services/zot|Zot]] - Registry mirror diff --git a/docs/reference/services/1password.md b/docs/reference/services/1password.md index 59c5b57..f85c107 100644 --- a/docs/reference/services/1password.md +++ b/docs/reference/services/1password.md @@ -54,5 +54,5 @@ op inject -i secret.yaml.tpl | kubectl apply -f - ## Related -- [[argocd\|ArgoCD]] - Uses secrets for git access -- [[postgresql\|PostgreSQL]] - Database credentials +- [[argocd|ArgoCD]] - Uses secrets for git access +- [[postgresql|PostgreSQL]] - Database credentials diff --git a/docs/reference/services/alloy.md b/docs/reference/services/alloy.md index fb8dcaf..f8953a0 100644 --- a/docs/reference/services/alloy.md +++ b/docs/reference/services/alloy.md @@ -27,7 +27,7 @@ Unified observability collector for metrics and logs with two deployments: - System metrics via `prometheus.exporter.unix` - Textfile collector: `minikube.prom`, `borgmatic.prom`, `zot.prom`, `jellyfin.prom` - Zot registry metrics from `http://localhost:5050/metrics` -- Pushed to [[prometheus\|Prometheus]] via remote_write +- Pushed to [[prometheus|Prometheus]] via remote_write ### From Kubernetes - All pod logs via `loki.source.kubernetes` @@ -39,7 +39,7 @@ Unified observability collector for metrics and logs with two deployments: **mcquack LaunchAgents:** alloy, borgmatic, zot, jellyfin -Logs pushed to [[loki\|Loki]] at `https://loki.tail8d86e.ts.net/loki/api/v1/push`. +Logs pushed to [[loki|Loki]] at `https://loki.tail8d86e.ts.net/loki/api/v1/push`. ## Why Built from Source @@ -47,6 +47,6 @@ The Homebrew bottle uses `CGO_ENABLED=0`, which breaks Tailscale MagicDNS. Build ## Related -- [[prometheus\|Prometheus]] - Metrics storage -- [[loki\|Loki]] - Log storage -- [[grafana\|Grafana]] - Visualization +- [[prometheus|Prometheus]] - Metrics storage +- [[loki|Loki]] - Log storage +- [[grafana|Grafana]] - Visualization diff --git a/docs/reference/services/argocd.md b/docs/reference/services/argocd.md index 874653e..9090907 100644 --- a/docs/reference/services/argocd.md +++ b/docs/reference/services/argocd.md @@ -7,7 +7,7 @@ tags: # ArgoCD -GitOps continuous delivery platform for the [[kubernetes/cluster\|Kubernetes cluster]]. +GitOps continuous delivery platform for the [[kubernetes/cluster|Kubernetes cluster]]. ## Quick Reference @@ -46,5 +46,5 @@ argocd app get ## Related -- [[kubernetes/apps\|Apps]] - Full application registry -- [[forgejo\|Forgejo]] - Git source +- [[kubernetes/apps|Apps]] - Full application registry +- [[forgejo|Forgejo]] - Git source diff --git a/docs/reference/services/borgmatic.md b/docs/reference/services/borgmatic.md index e7afdd4..da84c6e 100644 --- a/docs/reference/services/borgmatic.md +++ b/docs/reference/services/borgmatic.md @@ -16,7 +16,7 @@ Daily backup system using Borg backup, running on indri. | **Install** | mise (pipx) | | **Config** | `~/.config/borgmatic/config.yaml` | | **Schedule** | Daily at 2:00 AM | -| **Repository** | `/Volumes/backups/borg/` on [[storage/sifaka\|Sifaka]] | +| **Repository** | `/Volumes/backups/borg/` on [[storage/sifaka|Sifaka]] | ## What Gets Backed Up @@ -28,8 +28,8 @@ Daily backup system using Borg backup, running on indri. - `~/Pictures` - Photos **Databases:** -- `miniflux` on [[postgresql\|PostgreSQL]] -- `teslamate` on [[postgresql\|PostgreSQL]] +- `miniflux` on [[postgresql|PostgreSQL]] +- `teslamate` on [[postgresql|PostgreSQL]] **Not backed up (by design):** - ZIM archives (re-downloadable) @@ -46,15 +46,15 @@ Daily backup system using Borg backup, running on indri. ## Monitoring -Metrics exposed via textfile collector to [[prometheus\|Prometheus]]: +Metrics exposed via textfile collector to [[prometheus|Prometheus]]: - `borgmatic_up` - Repository accessibility - `borgmatic_last_archive_timestamp` - Last backup time - `borgmatic_repo_deduplicated_size_bytes` - Disk usage -Dashboard: "Borgmatic Backups" in [[grafana\|Grafana]] +Dashboard: "Borgmatic Backups" in [[grafana|Grafana]] ## Related -- [[storage/backups\|Backups]] - Full backup policy -- [[storage/sifaka\|Sifaka]] - Backup target -- [[postgresql\|PostgreSQL]] - Database backups +- [[storage/backups|Backups]] - Full backup policy +- [[storage/sifaka|Sifaka]] - Backup target +- [[postgresql|PostgreSQL]] - Database backups diff --git a/docs/reference/services/forgejo.md b/docs/reference/services/forgejo.md index c90ff03..5b8edd3 100644 --- a/docs/reference/services/forgejo.md +++ b/docs/reference/services/forgejo.md @@ -54,5 +54,5 @@ Secrets fetched from 1Password: `lfs-jwt-secret`, `internal-token`, `oauth2-jwt- ## Related -- [[argocd\|ArgoCD]] - Uses Forgejo as git source -- [[zot\|Zot]] - Container registry for built images +- [[argocd|ArgoCD]] - Uses Forgejo as git source +- [[zot|Zot]] - Container registry for built images diff --git a/docs/reference/services/grafana.md b/docs/reference/services/grafana.md index f0181f3..04c8114 100644 --- a/docs/reference/services/grafana.md +++ b/docs/reference/services/grafana.md @@ -45,6 +45,6 @@ Optional annotation: `grafana_folder: "FolderName"` ## Related -- [[prometheus\|Prometheus]] - Metrics datasource -- [[loki\|Loki]] - Logs datasource -- [[alloy\|Alloy]] - Data collector +- [[prometheus|Prometheus]] - Metrics datasource +- [[loki|Loki]] - Logs datasource +- [[alloy|Alloy]] - Data collector diff --git a/docs/reference/services/jellyfin.md b/docs/reference/services/jellyfin.md index 7a28b69..5ff0e51 100644 --- a/docs/reference/services/jellyfin.md +++ b/docs/reference/services/jellyfin.md @@ -42,10 +42,10 @@ Dashboard > Playback: ## Observability - Metrics: `jellyfin_metrics` ansible role -- Logs: Forwarded via [[alloy\|Alloy]] -- Dashboard: "Jellyfin Media Server" in [[grafana\|Grafana]] +- Logs: Forwarded via [[alloy|Alloy]] +- Dashboard: "Jellyfin Media Server" in [[grafana|Grafana]] ## Related -- [[navidrome\|Navidrome]] - Music streaming -- [[storage/sifaka\|Sifaka]] - Media storage +- [[navidrome|Navidrome]] - Music streaming +- [[storage/sifaka|Sifaka]] - Media storage diff --git a/docs/reference/services/kiwix.md b/docs/reference/services/kiwix.md index 9f66d25..1b6e5f7 100644 --- a/docs/reference/services/kiwix.md +++ b/docs/reference/services/kiwix.md @@ -17,14 +17,14 @@ Offline Wikipedia and ZIM archive server. | **Tailscale URL** | https://kiwix.tail8d86e.ts.net | | **Namespace** | `kiwix` | | **Image** | `ghcr.io/kiwix/kiwix-serve:3.8.1` | -| **Storage** | NFS from [[storage/sifaka\|Sifaka]] (`/volume1/torrents`) | +| **Storage** | NFS from [[storage/sifaka|Sifaka]] (`/volume1/torrents`) | ## Architecture | Component | Purpose | |-----------|---------| | kiwix-serve | Serves ZIM files on port 80 | -| torrent-sync | Sidecar syncing ZIM torrents to [[transmission\|Transmission]] | +| torrent-sync | Sidecar syncing ZIM torrents to [[transmission|Transmission]] | | zim-watcher | CronJob (hourly) to restart on new ZIMs | ## Configured Archives @@ -43,10 +43,10 @@ Full list: `argocd/manifests/kiwix/configmap-zim-torrents.yaml` 1. Edit `configmap-zim-torrents.yaml` 2. Add torrent URL from https://download.kiwix.org/zim/ 3. Sync: `argocd app sync kiwix` -4. Torrent-sync adds to [[transmission\|Transmission]] +4. Torrent-sync adds to [[transmission|Transmission]] 5. zim-watcher restarts kiwix when download completes ## Related -- [[transmission\|Transmission]] - Downloads ZIM files -- [[storage/sifaka\|Sifaka]] - ZIM storage +- [[transmission|Transmission]] - Downloads ZIM files +- [[storage/sifaka|Sifaka]] - ZIM storage diff --git a/docs/reference/services/loki.md b/docs/reference/services/loki.md index c13fea2..af5c77d 100644 --- a/docs/reference/services/loki.md +++ b/docs/reference/services/loki.md @@ -24,8 +24,8 @@ Log aggregation system for BlumeOps infrastructure. - Single-node deployment with filesystem storage - TSDB index with 24h period -- Logs collected by [[alloy\|Alloy]] and pushed via Loki API -- Queried via [[grafana\|Grafana]] +- Logs collected by [[alloy|Alloy]] and pushed via Loki API +- Queried via [[grafana|Grafana]] ## Log Sources @@ -46,6 +46,6 @@ Log aggregation system for BlumeOps infrastructure. ## Related -- [[alloy\|Alloy]] - Log collector -- [[grafana\|Grafana]] - Log visualization -- [[prometheus\|Prometheus]] - Metrics counterpart +- [[alloy|Alloy]] - Log collector +- [[grafana|Grafana]] - Log visualization +- [[prometheus|Prometheus]] - Metrics counterpart diff --git a/docs/reference/services/miniflux.md b/docs/reference/services/miniflux.md index 543e59f..9468036 100644 --- a/docs/reference/services/miniflux.md +++ b/docs/reference/services/miniflux.md @@ -17,7 +17,7 @@ Minimalist RSS/Atom feed reader. | **Tailscale URL** | https://feed.tail8d86e.ts.net | | **Namespace** | `miniflux` | | **Image** | `ghcr.io/miniflux/miniflux:latest` | -| **Database** | [[postgresql\|PostgreSQL]] | +| **Database** | [[postgresql|PostgreSQL]] | ## Features @@ -35,7 +35,7 @@ Database user password stored in `blumeops-pg-app` secret (auto-generated by CNP ## Backup -Feed subscriptions and read state backed up via [[borgmatic\|Borgmatic]] PostgreSQL hook. +Feed subscriptions and read state backed up via [[borgmatic|Borgmatic]] PostgreSQL hook. ## Health Check @@ -45,5 +45,5 @@ curl https://feed.ops.eblu.me/healthcheck ## Related -- [[postgresql\|PostgreSQL]] - Database backend -- [[borgmatic\|Borgmatic]] - Data backup +- [[postgresql|PostgreSQL]] - Database backend +- [[borgmatic|Borgmatic]] - Data backup diff --git a/docs/reference/services/navidrome.md b/docs/reference/services/navidrome.md index ccfac06..cf51184 100644 --- a/docs/reference/services/navidrome.md +++ b/docs/reference/services/navidrome.md @@ -48,5 +48,5 @@ kubectl --context=minikube-indri -n navidrome exec deploy/navidrome -- ls /music ## Related -- [[jellyfin\|Jellyfin]] - Video streaming -- [[storage/sifaka\|Sifaka]] - Music storage +- [[jellyfin|Jellyfin]] - Video streaming +- [[storage/sifaka|Sifaka]] - Music storage diff --git a/docs/reference/services/postgresql.md b/docs/reference/services/postgresql.md index ae46b78..5a2fb3f 100644 --- a/docs/reference/services/postgresql.md +++ b/docs/reference/services/postgresql.md @@ -23,8 +23,8 @@ Database cluster via CloudNativePG operator. | Database | Owner | Purpose | |----------|-------|---------| -| miniflux | miniflux | [[miniflux\|Miniflux]] feed data | -| teslamate | teslamate | [[teslamate\|TeslaMate]] vehicle data | +| miniflux | miniflux | [[miniflux|Miniflux]] feed data | +| teslamate | teslamate | [[teslamate|TeslaMate]] vehicle data | ## Users @@ -34,7 +34,7 @@ Database cluster via CloudNativePG operator. | miniflux | app owner | Owns miniflux database | | teslamate | superuser | TeslaMate (needs extensions) | | eblume | superuser | Admin access | -| borgmatic | pg_read_all_data | [[borgmatic\|Backup]] access | +| borgmatic | pg_read_all_data | [[borgmatic|Backup]] access | ## Quick Connect @@ -45,7 +45,7 @@ PGPASSWORD=$(op --vault blumeops item get --fields password --reveal) ## Backup -Backed up via [[borgmatic\|Borgmatic]] `postgresql_databases` hook. +Backed up via [[borgmatic|Borgmatic]] `postgresql_databases` hook. Borgmatic streams `pg_dump` directly to Borg (no intermediate files, no downtime). @@ -63,6 +63,6 @@ Borgmatic streams `pg_dump` directly to Borg (no intermediate files, no downtime ## Related -- [[miniflux\|Miniflux]] - Feed reader database -- [[teslamate\|TeslaMate]] - Vehicle data database -- [[borgmatic\|Borgmatic]] - Database backup +- [[miniflux|Miniflux]] - Feed reader database +- [[teslamate|TeslaMate]] - Vehicle data database +- [[borgmatic|Borgmatic]] - Database backup diff --git a/docs/reference/services/prometheus.md b/docs/reference/services/prometheus.md index 75a7379..b0a1435 100644 --- a/docs/reference/services/prometheus.md +++ b/docs/reference/services/prometheus.md @@ -22,14 +22,14 @@ Metrics storage and querying for BlumeOps infrastructure. ## Data Sources ### Remote Write (from Alloy) -- Indri system metrics via [[alloy\|Alloy]] remote_write +- Indri system metrics via [[alloy|Alloy]] remote_write - Textfile metrics: minikube, borgmatic, zot, jellyfin ### Scrape Targets | Target | Metrics | |--------|---------| -| `sifaka:9100` | [[storage/sifaka\|Sifaka]] NAS (node_exporter) | -| `cnpg-metrics.tail8d86e.ts.net:9187` | [[postgresql\|CloudNativePG]] metrics | +| `sifaka:9100` | [[storage/sifaka|Sifaka]] NAS (node_exporter) | +| `cnpg-metrics.tail8d86e.ts.net:9187` | [[postgresql|CloudNativePG]] metrics | | `kube-state-metrics.monitoring.svc:8080` | Kubernetes resource metrics | ## Query API @@ -49,6 +49,6 @@ Manifests: `argocd/manifests/prometheus/` ## Related -- [[alloy\|Alloy]] - Metrics collector -- [[grafana\|Grafana]] - Visualization -- [[loki\|Loki]] - Logs counterpart +- [[alloy|Alloy]] - Metrics collector +- [[grafana|Grafana]] - Visualization +- [[loki|Loki]] - Logs counterpart diff --git a/docs/reference/services/teslamate.md b/docs/reference/services/teslamate.md index 34b8a66..8d3766f 100644 --- a/docs/reference/services/teslamate.md +++ b/docs/reference/services/teslamate.md @@ -17,7 +17,7 @@ Self-hosted Tesla data logger collecting vehicle telemetry from the Tesla Owner | **Tailscale URL** | https://tesla.tail8d86e.ts.net | | **Namespace** | `teslamate` | | **Image** | `teslamate/teslamate:2.2.0` | -| **Database** | [[postgresql\|PostgreSQL]] | +| **Database** | [[postgresql|PostgreSQL]] | ## Data Collected @@ -53,6 +53,6 @@ Uses Tesla Owner API via OAuth: ## Related -- [[postgresql\|PostgreSQL]] - Data storage -- [[grafana\|Grafana]] - Dashboards -- [[borgmatic\|Borgmatic]] - Database backup +- [[postgresql|PostgreSQL]] - Data storage +- [[grafana|Grafana]] - Dashboards +- [[borgmatic|Borgmatic]] - Database backup diff --git a/docs/reference/services/transmission.md b/docs/reference/services/transmission.md index 7ef6ebc..4aa9849 100644 --- a/docs/reference/services/transmission.md +++ b/docs/reference/services/transmission.md @@ -7,7 +7,7 @@ tags: # Transmission -BitTorrent daemon, primarily for downloading ZIM archives for [[kiwix\|Kiwix]]. +BitTorrent daemon, primarily for downloading ZIM archives for [[kiwix|Kiwix]]. ## Quick Reference @@ -17,7 +17,7 @@ BitTorrent daemon, primarily for downloading ZIM archives for [[kiwix\|Kiwix]]. | **Tailscale URL** | https://torrent.tail8d86e.ts.net | | **Namespace** | `torrent` | | **Image** | `lscr.io/linuxserver/transmission:latest` | -| **Storage** | NFS PVC from [[storage/sifaka\|Sifaka]] | +| **Storage** | NFS PVC from [[storage/sifaka|Sifaka]] | ## Storage Layout @@ -30,7 +30,7 @@ NFS share on sifaka (`/volume1/torrents`): | `/config/` | Transmission configuration | | `/watch/` | Watch directory for .torrent files | -[[kiwix\|Kiwix]] reads from `/downloads/complete/` to serve ZIM archives. +[[kiwix|Kiwix]] reads from `/downloads/complete/` to serve ZIM archives. ## Integration with Kiwix @@ -43,11 +43,11 @@ When downloads complete, the zim-watcher CronJob detects new ZIMs and restarts K ## Monitoring -Basic uptime via blackbox probe in [[alloy\|Alloy]] k8s (Services Health dashboard). +Basic uptime via blackbox probe in [[alloy|Alloy]] k8s (Services Health dashboard). Web UI shows: active/seeding/paused counts, speeds, disk usage. ## Related -- [[kiwix\|Kiwix]] - ZIM archive consumer -- [[storage/sifaka\|Sifaka]] - Download storage +- [[kiwix|Kiwix]] - ZIM archive consumer +- [[storage/sifaka|Sifaka]] - Download storage diff --git a/docs/reference/services/zot.md b/docs/reference/services/zot.md index a09b362..95cd220 100644 --- a/docs/reference/services/zot.md +++ b/docs/reference/services/zot.md @@ -30,7 +30,7 @@ OCI-native container registry providing pull-through cache and private image sto ## Pull-Through Cache -When [[kubernetes/cluster\|minikube]] pulls an image: +When [[kubernetes/cluster|minikube]] pulls an image: 1. Containerd checks zot first (`host.minikube.internal:5050`) 2. If cached, returns immediately 3. If not, zot fetches from upstream, caches, returns @@ -62,5 +62,5 @@ curl -s http://indri:5050/v2/blumeops/devpi/tags/list | jq ## Related -- [[forgejo\|Forgejo]] - Container build CI -- [[kubernetes/cluster\|Cluster]] - Registry consumer +- [[forgejo|Forgejo]] - Container build CI +- [[kubernetes/cluster|Cluster]] - Registry consumer diff --git a/docs/reference/storage/backups.md b/docs/reference/storage/backups.md index c35095e..86b19ea 100644 --- a/docs/reference/storage/backups.md +++ b/docs/reference/storage/backups.md @@ -7,13 +7,13 @@ tags: # Backup Policy -Daily automated backups from [[infrastructure/hosts\|Indri]] to [[storage/sifaka\|Sifaka]] NAS. +Daily automated backups from [[infrastructure/hosts|Indri]] to [[storage/sifaka|Sifaka]] NAS. ## Schedule | Time | Frequency | System | |------|-----------|--------| -| 2:00 AM | Daily | [[services/borgmatic\|Borgmatic]] | +| 2:00 AM | Daily | [[services/borgmatic|Borgmatic]] | ## What Gets Backed Up @@ -31,8 +31,8 @@ Daily automated backups from [[infrastructure/hosts\|Indri]] to [[storage/sifaka | Database | Host | Method | |----------|------|--------| -| miniflux | [[services/postgresql\|pg.ops.eblu.me]] | pg_dump stream | -| teslamate | [[services/postgresql\|pg.ops.eblu.me]] | pg_dump stream | +| miniflux | [[services/postgresql|pg.ops.eblu.me]] | pg_dump stream | +| teslamate | [[services/postgresql|pg.ops.eblu.me]] | pg_dump stream | ## What Is NOT Backed Up @@ -53,16 +53,16 @@ Daily automated backups from [[infrastructure/hosts\|Indri]] to [[storage/sifaka ## Backup Target -Repository: `/Volumes/backups/borg/` on [[storage/sifaka\|Sifaka]] +Repository: `/Volumes/backups/borg/` on [[storage/sifaka|Sifaka]] ## Monitoring -Metrics exposed to [[services/prometheus\|Prometheus]]: +Metrics exposed to [[services/prometheus|Prometheus]]: - `borgmatic_up` - Repository accessible - `borgmatic_last_archive_timestamp` - Last backup time - `borgmatic_repo_deduplicated_size_bytes` - Disk usage -Dashboard: "Borgmatic Backups" in [[services/grafana\|Grafana]] +Dashboard: "Borgmatic Backups" in [[services/grafana|Grafana]] ## Recovery @@ -79,6 +79,6 @@ ssh indri 'mise x -- borgmatic check' ## Related -- [[services/borgmatic\|Borgmatic]] - Backup system details -- [[storage/sifaka\|Sifaka]] - Backup storage -- [[services/postgresql\|PostgreSQL]] - Database backups +- [[services/borgmatic|Borgmatic]] - Backup system details +- [[storage/sifaka|Sifaka]] - Backup storage +- [[services/postgresql|PostgreSQL]] - Database backups diff --git a/docs/reference/storage/sifaka.md b/docs/reference/storage/sifaka.md index 2e18793..df2387c 100644 --- a/docs/reference/storage/sifaka.md +++ b/docs/reference/storage/sifaka.md @@ -21,10 +21,10 @@ Synology NAS providing network storage and backup target. | Share | Path | Purpose | Consumers | |-------|------|---------|-----------| -| backups | `/volume1/backups` | Borg backup repository | [[services/borgmatic\|Borgmatic]] | -| torrents | `/volume1/torrents` | ZIM downloads | [[services/kiwix\|Kiwix]], [[services/transmission\|Transmission]] | -| music | `/volume1/music` | Music library | [[services/navidrome\|Navidrome]] | -| allisonflix | `/volume1/allisonflix` | Video library | [[services/jellyfin\|Jellyfin]] | +| backups | `/volume1/backups` | Borg backup repository | [[services/borgmatic|Borgmatic]] | +| torrents | `/volume1/torrents` | ZIM downloads | [[services/kiwix|Kiwix]], [[services/transmission|Transmission]] | +| music | `/volume1/music` | Music library | [[services/navidrome|Navidrome]] | +| allisonflix | `/volume1/allisonflix` | Video library | [[services/jellyfin|Jellyfin]] | | photos | `/volume1/photos` | Photo library | Immich | ## NFS Exports @@ -36,7 +36,7 @@ Synology NAS providing network storage and backup target. ## Monitoring -Node exporter running in Docker container, scraped by [[services/prometheus\|Prometheus]] at `sifaka:9100`. +Node exporter running in Docker container, scraped by [[services/prometheus|Prometheus]] at `sifaka:9100`. ## Tailscale @@ -45,7 +45,7 @@ Node exporter running in Docker container, scraped by [[services/prometheus\|Pro ## Related -- [[storage/backups\|Backups]] - Backup policy -- [[services/borgmatic\|Borgmatic]] - Backup system -- [[services/jellyfin\|Jellyfin]] - Media consumer -- [[services/navidrome\|Navidrome]] - Music consumer +- [[storage/backups|Backups]] - Backup policy +- [[services/borgmatic|Borgmatic]] - Backup system +- [[services/jellyfin|Jellyfin]] - Media consumer +- [[services/navidrome|Navidrome]] - Music consumer -- 2.50.1 (Apple Git-155) From ad7e9af45347e1324076ae2029c5263d3e4cf517 Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 3 Feb 2026 13:44:26 -0800 Subject: [PATCH 03/12] Trim operational content from reference cards Reference docs should describe "what things are", not "how to do things". Removed: - CLI command examples - Operational workflows (PR workflow, sync commands) - Setup/bootstrap procedures - "Useful Commands" sections Also split hosts.md into individual cards: - infrastructure/indri.md - Primary server details - infrastructure/gilbert.md - Development workstation This content belongs in how-to guides (future phase). Co-Authored-By: Claude Opus 4.5 --- docs/reference/index.md | 6 +- docs/reference/infrastructure/gilbert.md | 27 +++++++++ docs/reference/infrastructure/hosts.md | 64 +++------------------- docs/reference/infrastructure/indri.md | 38 +++++++++++++ docs/reference/infrastructure/routing.md | 16 +----- docs/reference/infrastructure/tailscale.md | 5 -- docs/reference/kubernetes/apps.md | 18 ------ docs/reference/kubernetes/cluster.md | 40 +------------- docs/reference/services/1password.md | 24 +------- docs/reference/services/argocd.md | 15 +---- docs/reference/services/forgejo.md | 18 ++---- docs/reference/services/miniflux.md | 18 ++---- docs/reference/services/navidrome.md | 12 +--- docs/reference/services/postgresql.md | 23 +++----- docs/reference/services/prometheus.md | 27 +++------ docs/reference/services/zot.md | 28 +--------- docs/reference/storage/backups.md | 15 +---- 17 files changed, 113 insertions(+), 281 deletions(-) create mode 100644 docs/reference/infrastructure/gilbert.md create mode 100644 docs/reference/infrastructure/indri.md diff --git a/docs/reference/index.md b/docs/reference/index.md index c8da610..abb6661 100644 --- a/docs/reference/index.md +++ b/docs/reference/index.md @@ -10,7 +10,7 @@ Technical specifications, inventories, and configuration details for BlumeOps in ## Services -Individual service reference cards with URLs, configuration, and operational details. +Individual service reference cards with URLs and configuration details. | Service | Description | Location | |---------|-------------|----------| @@ -35,7 +35,9 @@ Individual service reference cards with URLs, configuration, and operational det Host inventory and network configuration. -- [[infrastructure/hosts|Hosts]] - Device inventory (indri, gilbert, sifaka, etc.) +- [[infrastructure/hosts|Hosts]] - Device inventory +- [[infrastructure/indri|Indri]] - Primary server +- [[infrastructure/gilbert|Gilbert]] - Development workstation - [[infrastructure/tailscale|Tailscale]] - ACLs, groups, tags - [[infrastructure/routing|Routing]] - DNS domains, port mappings diff --git a/docs/reference/infrastructure/gilbert.md b/docs/reference/infrastructure/gilbert.md new file mode 100644 index 0000000..01b1b65 --- /dev/null +++ b/docs/reference/infrastructure/gilbert.md @@ -0,0 +1,27 @@ +--- +title: Gilbert +tags: + - infrastructure + - host +--- + +# Gilbert + +Primary development workstation. + +## Specifications + +| Property | Value | +|----------|-------| +| **Model** | 13" MacBook Air M4, 2025 | +| **User** | eblume | +| **Role** | Development workstation | + +## Development Tools + +Managed via `Brewfile` and `mise.toml` in the blumeops repo. + +## Related + +- [[infrastructure/indri|Indri]] - Server accessed from gilbert +- [[kubernetes/cluster|Cluster]] - Remote k8s access diff --git a/docs/reference/infrastructure/hosts.md b/docs/reference/infrastructure/hosts.md index cea3105..7b89345 100644 --- a/docs/reference/infrastructure/hosts.md +++ b/docs/reference/infrastructure/hosts.md @@ -10,62 +10,16 @@ All devices connected via [Tailscale](https://login.tailscale.com/) tailnet `tai ## Devices -| Host | Description | Notes | -|------|-------------|-------| -| **Indri** | Mac Mini M1, 2020 | Primary server, 2TB internal disk | -| **[[storage/sifaka|Sifaka]]** | Synology NAS | 10.9TB RAID 5, backup target | -| **Gilbert** | 13" MacBook Air M4, 2025 | Primary workstation | -| **Mouse** | 13" MacBook Air M2 | Allison's laptop | -| **UniFi** | UniFi Express 7 | Home WiFi network | -| **Dwarf** | iPad Air | Employer-provided, off tailnet | - -## Indri Details - -| Property | Value | -|----------|-------| -| **Model** | Mac mini M1, 2020 (Macmini9,1) | -| **Storage** | 2TB internal SSD | -| **macOS** | 15.7.3 (Sequoia) | -| **Role** | Primary server | -| **Tailscale IP** | 100.98.163.89 | - -### Services Hosted - -**Native (via Ansible):** -- [[services/forgejo|Forgejo]] - Git forge -- [[services/zot|Zot]] - Container registry -- [[services/jellyfin|Jellyfin]] - Media server -- [[services/borgmatic|Borgmatic]] - Backup system -- [[services/alloy|Alloy]] - Metrics/logs collector -- Caddy - Reverse proxy - -**Kubernetes (via minikube):** -- [[kubernetes/apps|All k8s applications]] - -### Sleep Prevention - -Indri uses Amphetamine (App Store) to prevent sleep. Configuration: -- Start Session At Launch: enabled -- Default Duration: indefinite -- Allow Closed-Display Sleep: enabled - -## Gilbert Details - -| Property | Value | -|----------|-------| -| **Model** | 13" MacBook Air M4, 2025 | -| **Role** | Development workstation | -| **User** | eblume | - -### Development Tools - -Managed via `Brewfile` and `mise.toml`. - -Fish abbreviations: -- `ki` -> `kubectl --context=minikube-indri` -- `k9i` -> `k9s --context=minikube-indri` +| Host | Description | Card | +|------|-------------|------| +| **Indri** | Mac Mini M1, 2020 - Primary server | [[infrastructure/indri|Details]] | +| **Gilbert** | MacBook Air M4, 2025 - Workstation | [[infrastructure/gilbert|Details]] | +| **[[storage/sifaka|Sifaka]]** | Synology NAS - Storage & backups | [[storage/sifaka|Details]] | +| **Mouse** | MacBook Air M2 - Allison's laptop | - | +| **UniFi** | UniFi Express 7 - Home WiFi | - | +| **Dwarf** | iPad Air - Employer-provided, off tailnet | - | ## Related - [[infrastructure/tailscale|Tailscale]] - Network configuration -- [[storage/sifaka|Sifaka]] - NAS details +- [[infrastructure/routing|Routing]] - Service URLs diff --git a/docs/reference/infrastructure/indri.md b/docs/reference/infrastructure/indri.md new file mode 100644 index 0000000..d5215a1 --- /dev/null +++ b/docs/reference/infrastructure/indri.md @@ -0,0 +1,38 @@ +--- +title: Indri +tags: + - infrastructure + - host +--- + +# Indri + +Primary BlumeOps server. Mac Mini M1 (2020). + +## Specifications + +| Property | Value | +|----------|-------| +| **Model** | Mac mini M1, 2020 (Macmini9,1) | +| **Storage** | 2TB internal SSD | +| **macOS** | 15.7.3 (Sequoia) | +| **Tailscale IP** | 100.98.163.89 | +| **Tailscale Tag** | `tag:homelab` | + +## Services Hosted + +**Native (via Ansible):** +- [[services/forgejo|Forgejo]] - Git forge +- [[services/zot|Zot]] - Container registry +- [[services/jellyfin|Jellyfin]] - Media server +- [[services/borgmatic|Borgmatic]] - Backup system +- [[services/alloy|Alloy]] - Metrics/logs collector +- Caddy - Reverse proxy for `*.ops.eblu.me` + +**Kubernetes (via minikube):** +- [[kubernetes/apps|All k8s applications]] + +## Related + +- [[infrastructure/routing|Routing]] - Port mappings +- [[kubernetes/cluster|Cluster]] - Minikube details diff --git a/docs/reference/infrastructure/routing.md b/docs/reference/infrastructure/routing.md index b6042a2..23e90f7 100644 --- a/docs/reference/infrastructure/routing.md +++ b/docs/reference/infrastructure/routing.md @@ -59,21 +59,7 @@ DNS points to indri's Tailscale IP (100.98.163.89). TLS via Let's Encrypt (ACME | 8096 | Jellyfin | HTTP | localhost | Media server | | 44491 | K8s API | HTTPS | 0.0.0.0 | Minikube API server | -## Adding New Services - -### Indri Services (via Caddy) -1. Host service on localhost -2. Add to `ansible/roles/caddy/defaults/main.yml` -3. Run `mise run provision-indri -- --tags caddy` - -### K8s Services (via Tailscale Ingress) -1. Create manifests in `argocd/manifests//` -2. Add ArgoCD Application in `argocd/apps/` -3. Add Tailscale Ingress annotation -4. Add Caddy proxy entry -5. Sync via ArgoCD - ## Related - [[infrastructure/tailscale|Tailscale]] - ACL configuration -- [[infrastructure/hosts|Hosts]] - Where services run +- [[infrastructure/indri|Indri]] - Where services run diff --git a/docs/reference/infrastructure/tailscale.md b/docs/reference/infrastructure/tailscale.md index a852ddb..21688ca 100644 --- a/docs/reference/infrastructure/tailscale.md +++ b/docs/reference/infrastructure/tailscale.md @@ -13,11 +13,6 @@ Tailnet `tail8d86e.ts.net` provides secure networking for all BlumeOps infrastru ACLs managed via Pulumi in `pulumi/policy.hujson`. -```bash -mise run tailnet-preview # Preview changes -mise run tailnet-up # Apply changes -``` - ## Groups | Group | Members | Purpose | diff --git a/docs/reference/kubernetes/apps.md b/docs/reference/kubernetes/apps.md index c10bbd8..fb22816 100644 --- a/docs/reference/kubernetes/apps.md +++ b/docs/reference/kubernetes/apps.md @@ -41,24 +41,6 @@ Registry of all applications deployed via [[services/argocd|ArgoCD]]. | `apps` | Automated | Picks up new Application manifests | | All others | Manual | Explicit control over deployments | -## Common Commands - -```bash -argocd app list # List all apps -argocd app get # Get details -argocd app diff # Preview changes -argocd app sync # Deploy changes -``` - -## PR Workflow - -1. Create feature branch, modify manifests -2. Push to forge -3. Sync apps application: `argocd app sync apps` -4. Point service at branch: `argocd app set --revision feature/branch` -5. Test: `argocd app sync ` -6. After merge, reset: `argocd app set --revision main` - ## Related - [[services/argocd|ArgoCD]] - GitOps platform details diff --git a/docs/reference/kubernetes/cluster.md b/docs/reference/kubernetes/cluster.md index b796987..abfb828 100644 --- a/docs/reference/kubernetes/cluster.md +++ b/docs/reference/kubernetes/cluster.md @@ -6,7 +6,7 @@ tags: # Kubernetes Cluster -Single-node Minikube cluster running on [[infrastructure/hosts|Indri]]. +Single-node Minikube cluster running on [[infrastructure/indri|Indri]]. ## Cluster Specifications @@ -22,50 +22,16 @@ Single-node Minikube cluster running on [[infrastructure/hosts|Indri]]. **Prerequisites:** Docker Desktop with at least 12GB memory allocated. -## Remote Access - -From gilbert: - -```bash -mise run ensure-minikube-indri-kubectl-config -``` - -Fish abbreviations: -- `ki` -> `kubectl --context=minikube-indri` -- `k9i` -> `k9s --context=minikube-indri` - ## Volume Mounting -Pods mount NFS directly from [[storage/sifaka|Sifaka]]: - -```yaml -volumes: - - name: torrents - nfs: - server: sifaka - path: /volume1/torrents -``` - -Docker NATs outbound traffic through indri's LAN IP (192.168.1.50), allowing access to Sifaka's NFS exports. +Pods mount NFS directly from [[storage/sifaka|Sifaka]]. Docker NATs outbound traffic through indri's LAN IP (192.168.1.50), allowing access to Sifaka's NFS exports. ## Registry Mirror -Containerd uses [[services/zot|Zot]] as a pull-through cache: -- Endpoint: `host.minikube.internal:5050` -- Config: `/etc/containerd/certs.d//hosts.toml` +Containerd uses [[services/zot|Zot]] as a pull-through cache at `host.minikube.internal:5050`. Mirrors configured: `registry.ops.eblu.me`, `docker.io`, `ghcr.io`, `quay.io` -## Useful Commands (on indri) - -```bash -minikube status # Cluster status -minikube start # Start cluster -minikube stop # Stop cluster -minikube ssh # SSH into node -minikube logs # View logs -``` - ## Related - [[kubernetes/apps|Apps]] - ArgoCD applications diff --git a/docs/reference/services/1password.md b/docs/reference/services/1password.md index f85c107..0be5940 100644 --- a/docs/reference/services/1password.md +++ b/docs/reference/services/1password.md @@ -32,27 +32,9 @@ The `blumeops` vault contains all infrastructure credentials. **ClusterSecretStore:** `onepassword-blumeops` -Services reference 1Password items via `ExternalSecret` manifests. Example: `argocd/manifests/devpi/external-secret.yaml` - -## CLI Usage - -```bash -# Get a secret field -op --vault blumeops item get --fields --reveal - -# Inject into a template -op inject -i secret.yaml.tpl | kubectl apply -f - -``` - -## Bootstrap (Disaster Recovery) - -1. Create Connect server: `op connect server create blumeops --vaults blumeops` -2. Create token: `op connect token create blumeops --server --vault blumeops` -3. Store credentials in 1Password item "1Password Connect" -4. Apply bootstrap secret to k8s -5. Sync apps: 1password-connect, external-secrets-crds, external-secrets, external-secrets-config +Services reference 1Password items via `ExternalSecret` manifests. ## Related -- [[argocd|ArgoCD]] - Uses secrets for git access -- [[postgresql|PostgreSQL]] - Database credentials +- [[services/argocd|ArgoCD]] - Uses secrets for git access +- [[services/postgresql|PostgreSQL]] - Database credentials diff --git a/docs/reference/services/argocd.md b/docs/reference/services/argocd.md index 9090907..4934d69 100644 --- a/docs/reference/services/argocd.md +++ b/docs/reference/services/argocd.md @@ -26,19 +26,6 @@ GitOps continuous delivery platform for the [[kubernetes/cluster|Kubernetes clus | `apps` | Automated | Picks up new Application manifests | | All workloads | Manual | Explicit control over deployments | -## CLI Commands - -```bash -# Login -argocd login argocd.ops.eblu.me --username admin --password "$(op ...)" - -# Common operations -argocd app list -argocd app diff -argocd app sync -argocd app get -``` - ## Credentials - Admin password: 1Password (blumeops vault) @@ -47,4 +34,4 @@ argocd app get ## Related - [[kubernetes/apps|Apps]] - Full application registry -- [[forgejo|Forgejo]] - Git source +- [[services/forgejo|Forgejo]] - Git source diff --git a/docs/reference/services/forgejo.md b/docs/reference/services/forgejo.md index 5b8edd3..3c4b705 100644 --- a/docs/reference/services/forgejo.md +++ b/docs/reference/services/forgejo.md @@ -38,21 +38,11 @@ Git forge and CI/CD platform. **Primary source of truth for blumeops** (mirrored **Workflows:** `.forgejo/workflows/` - `build-container.yaml` - Container image builds on tag -**Container release:** -```bash -mise run container-list # List containers -mise run container-release runner v1.0.0 # Tag and build -``` +## Secrets -## Ansible Management - -```bash -mise run provision-indri -- --tags forgejo -``` - -Secrets fetched from 1Password: `lfs-jwt-secret`, `internal-token`, `oauth2-jwt-secret`, `runner_reg` +Managed via 1Password: `lfs-jwt-secret`, `internal-token`, `oauth2-jwt-secret`, `runner_reg` ## Related -- [[argocd|ArgoCD]] - Uses Forgejo as git source -- [[zot|Zot]] - Container registry for built images +- [[services/argocd|ArgoCD]] - Uses Forgejo as git source +- [[services/zot|Zot]] - Container registry for built images diff --git a/docs/reference/services/miniflux.md b/docs/reference/services/miniflux.md index 9468036..2b4f0f7 100644 --- a/docs/reference/services/miniflux.md +++ b/docs/reference/services/miniflux.md @@ -17,7 +17,7 @@ Minimalist RSS/Atom feed reader. | **Tailscale URL** | https://feed.tail8d86e.ts.net | | **Namespace** | `miniflux` | | **Image** | `ghcr.io/miniflux/miniflux:latest` | -| **Database** | [[postgresql|PostgreSQL]] | +| **Database** | [[services/postgresql|PostgreSQL]] | ## Features @@ -29,21 +29,13 @@ Minimalist RSS/Atom feed reader. ## Database -Uses CloudNativePG cluster at `pg.ops.eblu.me`. - -Database user password stored in `blumeops-pg-app` secret (auto-generated by CNPG). +Uses CloudNativePG cluster at `pg.ops.eblu.me`. Database user password stored in `blumeops-pg-app` secret (auto-generated by CNPG). ## Backup -Feed subscriptions and read state backed up via [[borgmatic|Borgmatic]] PostgreSQL hook. - -## Health Check - -```bash -curl https://feed.ops.eblu.me/healthcheck -``` +Feed subscriptions and read state backed up via [[services/borgmatic|Borgmatic]] PostgreSQL hook. ## Related -- [[postgresql|PostgreSQL]] - Database backend -- [[borgmatic|Borgmatic]] - Data backup +- [[services/postgresql|PostgreSQL]] - Database backend +- [[services/borgmatic|Borgmatic]] - Data backup diff --git a/docs/reference/services/navidrome.md b/docs/reference/services/navidrome.md index cf51184..e3f46b7 100644 --- a/docs/reference/services/navidrome.md +++ b/docs/reference/services/navidrome.md @@ -36,17 +36,7 @@ The `/data` directory contains SQLite database, configuration, and cache. | `ND_MUSICFOLDER` | /music | | `ND_DATAFOLDER` | /data | -## Initial Setup - -On first access, Navidrome prompts to create an admin user. No default credentials. - -## Verify NFS Mount - -```bash -kubectl --context=minikube-indri -n navidrome exec deploy/navidrome -- ls /music -``` - ## Related -- [[jellyfin|Jellyfin]] - Video streaming +- [[services/jellyfin|Jellyfin]] - Video streaming - [[storage/sifaka|Sifaka]] - Music storage diff --git a/docs/reference/services/postgresql.md b/docs/reference/services/postgresql.md index 5a2fb3f..031ac2b 100644 --- a/docs/reference/services/postgresql.md +++ b/docs/reference/services/postgresql.md @@ -23,8 +23,8 @@ Database cluster via CloudNativePG operator. | Database | Owner | Purpose | |----------|-------|---------| -| miniflux | miniflux | [[miniflux|Miniflux]] feed data | -| teslamate | teslamate | [[teslamate|TeslaMate]] vehicle data | +| miniflux | miniflux | [[services/miniflux|Miniflux]] feed data | +| teslamate | teslamate | [[services/teslamate|TeslaMate]] vehicle data | ## Users @@ -34,20 +34,11 @@ Database cluster via CloudNativePG operator. | miniflux | app owner | Owns miniflux database | | teslamate | superuser | TeslaMate (needs extensions) | | eblume | superuser | Admin access | -| borgmatic | pg_read_all_data | [[borgmatic|Backup]] access | - -## Quick Connect - -```bash -PGPASSWORD=$(op --vault blumeops item get --fields password --reveal) \ - psql -h pg.ops.eblu.me -U eblume -d miniflux -``` +| borgmatic | pg_read_all_data | [[services/borgmatic|Backup]] access | ## Backup -Backed up via [[borgmatic|Borgmatic]] `postgresql_databases` hook. - -Borgmatic streams `pg_dump` directly to Borg (no intermediate files, no downtime). +Backed up via [[services/borgmatic|Borgmatic]] `postgresql_databases` hook. Streams `pg_dump` directly to Borg (no intermediate files, no downtime). ## Credentials @@ -63,6 +54,6 @@ Borgmatic streams `pg_dump` directly to Borg (no intermediate files, no downtime ## Related -- [[miniflux|Miniflux]] - Feed reader database -- [[teslamate|TeslaMate]] - Vehicle data database -- [[borgmatic|Borgmatic]] - Database backup +- [[services/miniflux|Miniflux]] - Feed reader database +- [[services/teslamate|TeslaMate]] - Vehicle data database +- [[services/borgmatic|Borgmatic]] - Database backup diff --git a/docs/reference/services/prometheus.md b/docs/reference/services/prometheus.md index b0a1435..86a95c3 100644 --- a/docs/reference/services/prometheus.md +++ b/docs/reference/services/prometheus.md @@ -18,37 +18,24 @@ Metrics storage and querying for BlumeOps infrastructure. | **Namespace** | `monitoring` | | **Image** | `prom/prometheus:v3.2.1` | | **Storage** | 50Gi PVC | +| **Manifests** | `argocd/manifests/prometheus/` | ## Data Sources ### Remote Write (from Alloy) -- Indri system metrics via [[alloy|Alloy]] remote_write +- Indri system metrics via [[services/alloy|Alloy]] remote_write - Textfile metrics: minikube, borgmatic, zot, jellyfin ### Scrape Targets + | Target | Metrics | |--------|---------| | `sifaka:9100` | [[storage/sifaka|Sifaka]] NAS (node_exporter) | -| `cnpg-metrics.tail8d86e.ts.net:9187` | [[postgresql|CloudNativePG]] metrics | +| `cnpg-metrics.tail8d86e.ts.net:9187` | [[services/postgresql|CloudNativePG]] metrics | | `kube-state-metrics.monitoring.svc:8080` | Kubernetes resource metrics | -## Query API - -```bash -# Check targets -curl -s https://prometheus.ops.eblu.me/api/v1/targets | jq '.data.activeTargets[].scrapeUrl' -``` - -## ArgoCD Management - -```bash -argocd app sync prometheus -``` - -Manifests: `argocd/manifests/prometheus/` - ## Related -- [[alloy|Alloy]] - Metrics collector -- [[grafana|Grafana]] - Visualization -- [[loki|Loki]] - Logs counterpart +- [[services/alloy|Alloy]] - Metrics collector +- [[services/grafana|Grafana]] - Visualization +- [[services/loki|Loki]] - Logs counterpart diff --git a/docs/reference/services/zot.md b/docs/reference/services/zot.md index 95cd220..1da1d3e 100644 --- a/docs/reference/services/zot.md +++ b/docs/reference/services/zot.md @@ -30,37 +30,13 @@ OCI-native container registry providing pull-through cache and private image sto ## Pull-Through Cache -When [[kubernetes/cluster|minikube]] pulls an image: -1. Containerd checks zot first (`host.minikube.internal:5050`) -2. If cached, returns immediately -3. If not, zot fetches from upstream, caches, returns - -## Private Images - -```bash -# Build and push from gilbert -podman build -t registry.ops.eblu.me/blumeops/myapp:v1 . -podman push registry.ops.eblu.me/blumeops/myapp:v1 - -# Use in k8s manifest -image: registry.ops.eblu.me/blumeops/myapp:v1 -``` +When [[kubernetes/cluster|minikube]] pulls an image, containerd checks zot first. If cached, returns immediately. If not, zot fetches from upstream, caches it, then returns. ## Security Model Network access only (no authentication). Defense is the Tailscale ACL boundary. -## Useful Commands - -```bash -# List all images -curl -s http://indri:5050/v2/_catalog | jq - -# List tags -curl -s http://indri:5050/v2/blumeops/devpi/tags/list | jq -``` - ## Related -- [[forgejo|Forgejo]] - Container build CI +- [[services/forgejo|Forgejo]] - Container build CI - [[kubernetes/cluster|Cluster]] - Registry consumer diff --git a/docs/reference/storage/backups.md b/docs/reference/storage/backups.md index 86b19ea..81c608f 100644 --- a/docs/reference/storage/backups.md +++ b/docs/reference/storage/backups.md @@ -7,7 +7,7 @@ tags: # Backup Policy -Daily automated backups from [[infrastructure/hosts|Indri]] to [[storage/sifaka|Sifaka]] NAS. +Daily automated backups from [[infrastructure/indri|Indri]] to [[storage/sifaka|Sifaka]] NAS. ## Schedule @@ -64,19 +64,6 @@ Metrics exposed to [[services/prometheus|Prometheus]]: Dashboard: "Borgmatic Backups" in [[services/grafana|Grafana]] -## Recovery - -```bash -# List archives -ssh indri 'mise x -- borgmatic list' - -# Extract specific path from latest -ssh indri 'mise x -- borgmatic extract --archive latest --path /some/path' - -# Check repository health -ssh indri 'mise x -- borgmatic check' -``` - ## Related - [[services/borgmatic|Borgmatic]] - Backup system details -- 2.50.1 (Apple Git-155) From 0eac3c0384dfa916af0332ea1c7a0f2d45d5b8d8 Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 3 Feb 2026 13:46:20 -0800 Subject: [PATCH 04/12] Add storage/postgresql redirect card Links to services/postgresql for discoverability from storage section. Co-Authored-By: Claude Opus 4.5 --- docs/reference/index.md | 1 + docs/reference/storage/postgresql.md | 10 ++++++++++ 2 files changed, 11 insertions(+) create mode 100644 docs/reference/storage/postgresql.md diff --git a/docs/reference/index.md b/docs/reference/index.md index abb6661..39dbd52 100644 --- a/docs/reference/index.md +++ b/docs/reference/index.md @@ -53,4 +53,5 @@ Cluster configuration and application registry. Network storage and backup configuration. - [[storage/sifaka|Sifaka]] - Synology NAS configuration +- [[storage/postgresql|PostgreSQL]] - Database cluster - [[storage/backups|Backups]] - Backup policy and schedule diff --git a/docs/reference/storage/postgresql.md b/docs/reference/storage/postgresql.md new file mode 100644 index 0000000..e1f965e --- /dev/null +++ b/docs/reference/storage/postgresql.md @@ -0,0 +1,10 @@ +--- +title: PostgreSQL +tags: + - storage + - database +--- + +# PostgreSQL + +See [[services/postgresql|PostgreSQL]] in Services. -- 2.50.1 (Apple Git-155) From 6f8fb33122386f0e1d2574803a7c020124c6f45d Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 3 Feb 2026 13:47:12 -0800 Subject: [PATCH 05/12] Add kubernetes/external-secrets redirect card Links to services/1password for discoverability from kubernetes section. Co-Authored-By: Claude Opus 4.5 --- docs/reference/index.md | 1 + docs/reference/kubernetes/external-secrets.md | 10 ++++++++++ 2 files changed, 11 insertions(+) create mode 100644 docs/reference/kubernetes/external-secrets.md diff --git a/docs/reference/index.md b/docs/reference/index.md index 39dbd52..57ee816 100644 --- a/docs/reference/index.md +++ b/docs/reference/index.md @@ -47,6 +47,7 @@ Cluster configuration and application registry. - [[kubernetes/cluster|Cluster]] - Minikube specs, storage, networking - [[kubernetes/apps|Apps]] - ArgoCD application registry +- [[kubernetes/external-secrets|External Secrets]] - Secrets management ## Storage diff --git a/docs/reference/kubernetes/external-secrets.md b/docs/reference/kubernetes/external-secrets.md new file mode 100644 index 0000000..aa97fdb --- /dev/null +++ b/docs/reference/kubernetes/external-secrets.md @@ -0,0 +1,10 @@ +--- +title: External Secrets +tags: + - kubernetes + - secrets +--- + +# External Secrets + +See [[services/1password|1Password]] in Services. -- 2.50.1 (Apple Git-155) From 10b062051d378113e95d62011dc944dc88bbc527 Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 3 Feb 2026 13:58:41 -0800 Subject: [PATCH 06/12] Add operations reference cards - observability - links to prometheus, loki, alloy, grafana - backup - links to borgmatic, sifaka, backup policy - disaster-recovery - TBD, links to borgmatic, 1password, forgejo Co-Authored-By: Claude Opus 4.5 --- docs/reference/index.md | 8 ++++++++ docs/reference/operations/backup.md | 15 +++++++++++++++ .../reference/operations/disaster-recovery.md | 19 +++++++++++++++++++ docs/reference/operations/observability.md | 16 ++++++++++++++++ 4 files changed, 58 insertions(+) create mode 100644 docs/reference/operations/backup.md create mode 100644 docs/reference/operations/disaster-recovery.md create mode 100644 docs/reference/operations/observability.md diff --git a/docs/reference/index.md b/docs/reference/index.md index 57ee816..a4b211b 100644 --- a/docs/reference/index.md +++ b/docs/reference/index.md @@ -56,3 +56,11 @@ Network storage and backup configuration. - [[storage/sifaka|Sifaka]] - Synology NAS configuration - [[storage/postgresql|PostgreSQL]] - Database cluster - [[storage/backups|Backups]] - Backup policy and schedule + +## Operations + +Operational concerns and their components. + +- [[operations/observability|Observability]] - Metrics, logs, dashboards +- [[operations/backup|Backup]] - Data protection +- [[operations/disaster-recovery|Disaster Recovery]] - Recovery procedures (TBD) diff --git a/docs/reference/operations/backup.md b/docs/reference/operations/backup.md new file mode 100644 index 0000000..da79caa --- /dev/null +++ b/docs/reference/operations/backup.md @@ -0,0 +1,15 @@ +--- +title: Backup +tags: + - operations +--- + +# Backup + +Daily automated backups of BlumeOps data. + +## Components + +- [[services/borgmatic|Borgmatic]] - Backup orchestration +- [[storage/sifaka|Sifaka]] - Backup target (NAS) +- [[storage/backups|Backup Policy]] - What gets backed up and retention diff --git a/docs/reference/operations/disaster-recovery.md b/docs/reference/operations/disaster-recovery.md new file mode 100644 index 0000000..e110a30 --- /dev/null +++ b/docs/reference/operations/disaster-recovery.md @@ -0,0 +1,19 @@ +--- +title: Disaster Recovery +tags: + - operations +--- + +# Disaster Recovery + +TBD. Current state: + +- [[services/borgmatic|Borgmatic]] provides daily backups to [[storage/sifaka|Sifaka]] +- Infrastructure can be rebootstrapped using the blumeops repo +- Detailed DR procedures not yet documented + +## Components + +- [[services/borgmatic|Borgmatic]] - Backup restoration +- [[services/1password|1Password]] - Credential recovery +- [[services/forgejo|Forgejo]] - Source of truth for infrastructure code diff --git a/docs/reference/operations/observability.md b/docs/reference/operations/observability.md new file mode 100644 index 0000000..388ae47 --- /dev/null +++ b/docs/reference/operations/observability.md @@ -0,0 +1,16 @@ +--- +title: Observability +tags: + - operations +--- + +# Observability + +Metrics, logs, and dashboards for BlumeOps infrastructure. + +## Components + +- [[services/prometheus|Prometheus]] - Metrics storage and querying +- [[services/loki|Loki]] - Log aggregation +- [[services/alloy|Alloy]] - Metrics and log collection +- [[services/grafana|Grafana]] - Dashboards and visualization -- 2.50.1 (Apple Git-155) From cfbf8e480398971e8ef7ae9a159a4326e008f009 Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 3 Feb 2026 14:03:00 -0800 Subject: [PATCH 07/12] Add note about Alloy CGO build possibly being obsolete With *.ops.eblu.me URLs routed via Caddy, Tailscale MagicDNS resolution may no longer be needed. Should be tested. Co-Authored-By: Claude Opus 4.5 --- docs/reference/services/alloy.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/reference/services/alloy.md b/docs/reference/services/alloy.md index f8953a0..1135c66 100644 --- a/docs/reference/services/alloy.md +++ b/docs/reference/services/alloy.md @@ -45,6 +45,8 @@ Logs pushed to [[loki|Loki]] at `https://loki.tail8d86e.ts.net/loki/api/v1/push` The Homebrew bottle uses `CGO_ENABLED=0`, which breaks Tailscale MagicDNS. Building with `CGO_ENABLED=1` uses the macOS native resolver. +**Note:** This may no longer be needed now that services use `*.ops.eblu.me` URLs (routed via Caddy) instead of `*.tail8d86e.ts.net`. Should be tested in the future. + ## Related - [[prometheus|Prometheus]] - Metrics storage -- 2.50.1 (Apple Git-155) From 1665fa90e00cefeb5a5fb32f9a98ba5780640d8b Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 3 Feb 2026 14:11:47 -0800 Subject: [PATCH 08/12] Link postgresql backup section to operations/backup Co-Authored-By: Claude Opus 4.5 --- docs/reference/services/postgresql.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/services/postgresql.md b/docs/reference/services/postgresql.md index 031ac2b..c8b412c 100644 --- a/docs/reference/services/postgresql.md +++ b/docs/reference/services/postgresql.md @@ -38,7 +38,7 @@ Database cluster via CloudNativePG operator. ## Backup -Backed up via [[services/borgmatic|Borgmatic]] `postgresql_databases` hook. Streams `pg_dump` directly to Borg (no intermediate files, no downtime). +Backed up via [[services/borgmatic|Borgmatic]] `postgresql_databases` hook. Streams `pg_dump` directly to Borg (no intermediate files, no downtime). See [[operations/backup|Backup]] for overall backup policy. ## Credentials -- 2.50.1 (Apple Git-155) From 4838da600e4751606b7f9680fb06a5247f28262e Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 3 Feb 2026 14:17:21 -0800 Subject: [PATCH 09/12] Add Immich service card - New services/immich.md card - Update sifaka to link photos share to Immich - Add Immich to services index Co-Authored-By: Claude Opus 4.5 --- docs/reference/index.md | 1 + docs/reference/services/immich.md | 26 ++++++++++++++++++++++++++ docs/reference/storage/sifaka.md | 3 ++- 3 files changed, 29 insertions(+), 1 deletion(-) create mode 100644 docs/reference/services/immich.md diff --git a/docs/reference/index.md b/docs/reference/index.md index a4b211b..0423242 100644 --- a/docs/reference/index.md +++ b/docs/reference/index.md @@ -20,6 +20,7 @@ Individual service reference cards with URLs and configuration details. | [[services/1password|1Password]] | Secrets management | cloud + k8s | | [[services/forgejo|Forgejo]] | Git forge & CI/CD | indri | | [[services/grafana|Grafana]] | Dashboards & visualization | k8s | +| [[services/immich|Immich]] | Photo management | k8s | | [[services/jellyfin|Jellyfin]] | Media server | indri | | [[services/kiwix|Kiwix]] | Offline Wikipedia & ZIM archives | k8s | | [[services/loki|Loki]] | Log aggregation | k8s | diff --git a/docs/reference/services/immich.md b/docs/reference/services/immich.md new file mode 100644 index 0000000..5413dbd --- /dev/null +++ b/docs/reference/services/immich.md @@ -0,0 +1,26 @@ +--- +title: Immich +tags: + - service + - media +--- + +# Immich + +Self-hosted photo and video management. + +## Quick Reference + +| Property | Value | +|----------|-------| +| **URL** | https://photos.ops.eblu.me | +| **Namespace** | `immich` | +| **Deployment** | Helm chart | +| **Database** | [[services/postgresql|PostgreSQL]] (CNPG) | +| **Storage** | [[storage/sifaka|Sifaka]] photos volume | + +## Related + +- [[services/postgresql|PostgreSQL]] - Database backend +- [[storage/sifaka|Sifaka]] - Photo storage +- [[services/jellyfin|Jellyfin]] - Video streaming (separate service) diff --git a/docs/reference/storage/sifaka.md b/docs/reference/storage/sifaka.md index df2387c..73998d5 100644 --- a/docs/reference/storage/sifaka.md +++ b/docs/reference/storage/sifaka.md @@ -25,7 +25,7 @@ Synology NAS providing network storage and backup target. | torrents | `/volume1/torrents` | ZIM downloads | [[services/kiwix|Kiwix]], [[services/transmission|Transmission]] | | music | `/volume1/music` | Music library | [[services/navidrome|Navidrome]] | | allisonflix | `/volume1/allisonflix` | Video library | [[services/jellyfin|Jellyfin]] | -| photos | `/volume1/photos` | Photo library | Immich | +| photos | `/volume1/photos` | Photo library | [[services/immich|Immich]] | ## NFS Exports @@ -47,5 +47,6 @@ Node exporter running in Docker container, scraped by [[services/prometheus|Prom - [[storage/backups|Backups]] - Backup policy - [[services/borgmatic|Borgmatic]] - Backup system +- [[services/immich|Immich]] - Photo consumer - [[services/jellyfin|Jellyfin]] - Media consumer - [[services/navidrome|Navidrome]] - Music consumer -- 2.50.1 (Apple Git-155) From 3c72f1d2ae6d00c8c3dfea4c3b6c63c0e9cd9e51 Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 3 Feb 2026 14:21:42 -0800 Subject: [PATCH 10/12] Add Backup section to sifaka card Clarifies that sifaka is the backup TARGET, not source. Notes RAID 5 for current protection, offsite planned for future. Co-Authored-By: Claude Opus 4.5 --- docs/reference/storage/sifaka.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/reference/storage/sifaka.md b/docs/reference/storage/sifaka.md index 73998d5..1481b88 100644 --- a/docs/reference/storage/sifaka.md +++ b/docs/reference/storage/sifaka.md @@ -43,6 +43,12 @@ Node exporter running in Docker container, scraped by [[services/prometheus|Prom - Tag: `tag:nas` - ACL: `tag:homelab` can access for backups +## Backup + +Sifaka is the **target** for [[operations/backup|backups]], not a backup source. [[services/borgmatic|Borgmatic]] sends backups TO sifaka, not OF sifaka. + +Data protection for sifaka itself currently relies on the Synology RAID 5 configuration, which provides single-disk fault tolerance. Future plans include offsite duplication for additional resiliency. + ## Related - [[storage/backups|Backups]] - Backup policy -- 2.50.1 (Apple Git-155) From 9ca698087c6843dd073c162faa7bbc1bc36e53f2 Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 3 Feb 2026 14:24:17 -0800 Subject: [PATCH 11/12] Add Immich to k8s apps registry - Clarify immich deployment is k8s Helm chart - Add immich entry to kubernetes/apps.md Co-Authored-By: Claude Opus 4.5 --- docs/reference/kubernetes/apps.md | 1 + docs/reference/services/immich.md | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/kubernetes/apps.md b/docs/reference/kubernetes/apps.md index fb22816..149d685 100644 --- a/docs/reference/kubernetes/apps.md +++ b/docs/reference/kubernetes/apps.md @@ -25,6 +25,7 @@ Registry of all applications deployed via [[services/argocd|ArgoCD]]. | `loki` | monitoring | `argocd/manifests/loki/` | [[services/loki|Loki]] | | `grafana` | monitoring | Helm chart (forge mirror) | [[services/grafana|Grafana]] | | `grafana-config` | monitoring | `argocd/manifests/grafana-config/` | [[services/grafana|Grafana]] | +| `immich` | immich | Helm chart | [[services/immich|Immich]] | | `alloy-k8s` | alloy | `argocd/manifests/alloy-k8s/` | [[services/alloy|Alloy]] | | `kube-state-metrics` | monitoring | `argocd/manifests/kube-state-metrics/` | K8s metrics | | `miniflux` | miniflux | `argocd/manifests/miniflux/` | [[services/miniflux|Miniflux]] | diff --git a/docs/reference/services/immich.md b/docs/reference/services/immich.md index 5413dbd..0c4ba00 100644 --- a/docs/reference/services/immich.md +++ b/docs/reference/services/immich.md @@ -15,7 +15,7 @@ Self-hosted photo and video management. |----------|-------| | **URL** | https://photos.ops.eblu.me | | **Namespace** | `immich` | -| **Deployment** | Helm chart | +| **Deployment** | Helm chart (k8s) | | **Database** | [[services/postgresql|PostgreSQL]] (CNPG) | | **Storage** | [[storage/sifaka|Sifaka]] photos volume | -- 2.50.1 (Apple Git-155) From f2c7c2914661a158996c4db5a81e1320de20f960 Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 3 Feb 2026 14:25:33 -0800 Subject: [PATCH 12/12] Add photos NFS export to sifaka card Verified against actual k8s PVs - all three NFS exports now listed: - /volume1/torrents (kiwix, transmission) - /volume1/music (navidrome) - /volume1/photos (immich) Co-Authored-By: Claude Opus 4.5 --- docs/reference/storage/sifaka.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/reference/storage/sifaka.md b/docs/reference/storage/sifaka.md index 1481b88..e17083e 100644 --- a/docs/reference/storage/sifaka.md +++ b/docs/reference/storage/sifaka.md @@ -33,6 +33,7 @@ Synology NAS providing network storage and backup target. |--------|-----------------|---------| | `/volume1/torrents` | 192.168.1.0/24, 100.64.0.0/10 | k8s pods via Docker NAT | | `/volume1/music` | 192.168.1.0/24, 100.64.0.0/10 | k8s pods via Docker NAT | +| `/volume1/photos` | 192.168.1.0/24, 100.64.0.0/10 | k8s pods via Docker NAT | ## Monitoring -- 2.50.1 (Apple Git-155)