Phase 2: Add Reference section with 24 technical reference cards (#88)

## Summary
- Create `docs/reference/` section with 24 technical reference cards
- Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot
- Infrastructure (3): hosts, tailscale, routing
- Kubernetes (2): cluster, apps
- Storage (2): sifaka, backups
- Update README to mark Phase 2 as complete
- Add towncrier changelog fragment

## Deployment and Testing
- [ ] Build docs locally to verify wiki-links resolve
- [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
This commit is contained in:
Erich Blume 2026-02-03 14:27:37 -08:00
commit 254b93096a
34 changed files with 1385 additions and 6 deletions

View file

@ -64,14 +64,16 @@ The documentation is being restructured to follow the [Diataxis](https://diataxi
**Docs URL:** https://docs.ops.eblu.me **Docs URL:** https://docs.ops.eblu.me
### Phase 2: Reference ### Phase 2: Reference (Complete)
Information-oriented technical descriptions. Built first so other docs can link to reference material. Information-oriented technical descriptions. Built first so other docs can link to reference material.
- [ ] Create `reference/` directory - [x] Create `reference/` directory with index
- [ ] Service reference pages (migrate from zk cards) - [x] Service reference pages (16 services: alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot)
- [ ] Infrastructure inventory - [x] Infrastructure inventory (hosts, tailscale, routing)
- [ ] Configuration reference - [x] Kubernetes reference (cluster, apps)
- [ ] API/CLI reference for mise tasks - [x] Storage reference (sifaka, backups)
**Reference URL:** https://docs.ops.eblu.me/reference/
### Phase 3: Tutorials ### Phase 3: Tutorials
Learning-oriented content for getting started. Learning-oriented content for getting started.

View file

@ -0,0 +1 @@
Add Reference section with 24 technical reference cards covering services, infrastructure, kubernetes, and storage

67
docs/reference/index.md Normal file
View file

@ -0,0 +1,67 @@
---
title: Reference
tags:
- reference
---
# Reference
Technical specifications, inventories, and configuration details for BlumeOps infrastructure.
## Services
Individual service reference cards with URLs and configuration details.
| Service | Description | Location |
|---------|-------------|----------|
| [[services/alloy|Alloy]] | Observability collector (metrics & logs) | indri + k8s |
| [[services/argocd|ArgoCD]] | GitOps continuous delivery | k8s |
| [[services/borgmatic|Borgmatic]] | Backup system | indri |
| [[services/1password|1Password]] | Secrets management | cloud + k8s |
| [[services/forgejo|Forgejo]] | Git forge & CI/CD | indri |
| [[services/grafana|Grafana]] | Dashboards & visualization | k8s |
| [[services/immich|Immich]] | Photo management | k8s |
| [[services/jellyfin|Jellyfin]] | Media server | indri |
| [[services/kiwix|Kiwix]] | Offline Wikipedia & ZIM archives | k8s |
| [[services/loki|Loki]] | Log aggregation | k8s |
| [[services/miniflux|Miniflux]] | RSS feed reader | k8s |
| [[services/navidrome|Navidrome]] | Music streaming | k8s |
| [[services/postgresql|PostgreSQL]] | Database cluster | k8s |
| [[services/prometheus|Prometheus]] | Metrics collection | k8s |
| [[services/teslamate|TeslaMate]] | Tesla data logger | k8s |
| [[services/transmission|Transmission]] | BitTorrent daemon | k8s |
| [[services/zot|Zot]] | Container registry | indri |
## Infrastructure
Host inventory and network configuration.
- [[infrastructure/hosts|Hosts]] - Device inventory
- [[infrastructure/indri|Indri]] - Primary server
- [[infrastructure/gilbert|Gilbert]] - Development workstation
- [[infrastructure/tailscale|Tailscale]] - ACLs, groups, tags
- [[infrastructure/routing|Routing]] - DNS domains, port mappings
## Kubernetes
Cluster configuration and application registry.
- [[kubernetes/cluster|Cluster]] - Minikube specs, storage, networking
- [[kubernetes/apps|Apps]] - ArgoCD application registry
- [[kubernetes/external-secrets|External Secrets]] - Secrets management
## Storage
Network storage and backup configuration.
- [[storage/sifaka|Sifaka]] - Synology NAS configuration
- [[storage/postgresql|PostgreSQL]] - Database cluster
- [[storage/backups|Backups]] - Backup policy and schedule
## Operations
Operational concerns and their components.
- [[operations/observability|Observability]] - Metrics, logs, dashboards
- [[operations/backup|Backup]] - Data protection
- [[operations/disaster-recovery|Disaster Recovery]] - Recovery procedures (TBD)

View file

@ -0,0 +1,27 @@
---
title: Gilbert
tags:
- infrastructure
- host
---
# Gilbert
Primary development workstation.
## Specifications
| Property | Value |
|----------|-------|
| **Model** | 13" MacBook Air M4, 2025 |
| **User** | eblume |
| **Role** | Development workstation |
## Development Tools
Managed via `Brewfile` and `mise.toml` in the blumeops repo.
## Related
- [[infrastructure/indri|Indri]] - Server accessed from gilbert
- [[kubernetes/cluster|Cluster]] - Remote k8s access

View file

@ -0,0 +1,25 @@
---
title: Host Inventory
tags:
- infrastructure
---
# Host Inventory
All devices connected via [Tailscale](https://login.tailscale.com/) tailnet `tail8d86e.ts.net`.
## Devices
| Host | Description | Card |
|------|-------------|------|
| **Indri** | Mac Mini M1, 2020 - Primary server | [[infrastructure/indri|Details]] |
| **Gilbert** | MacBook Air M4, 2025 - Workstation | [[infrastructure/gilbert|Details]] |
| **[[storage/sifaka|Sifaka]]** | Synology NAS - Storage & backups | [[storage/sifaka|Details]] |
| **Mouse** | MacBook Air M2 - Allison's laptop | - |
| **UniFi** | UniFi Express 7 - Home WiFi | - |
| **Dwarf** | iPad Air - Employer-provided, off tailnet | - |
## Related
- [[infrastructure/tailscale|Tailscale]] - Network configuration
- [[infrastructure/routing|Routing]] - Service URLs

View file

@ -0,0 +1,38 @@
---
title: Indri
tags:
- infrastructure
- host
---
# Indri
Primary BlumeOps server. Mac Mini M1 (2020).
## Specifications
| Property | Value |
|----------|-------|
| **Model** | Mac mini M1, 2020 (Macmini9,1) |
| **Storage** | 2TB internal SSD |
| **macOS** | 15.7.3 (Sequoia) |
| **Tailscale IP** | 100.98.163.89 |
| **Tailscale Tag** | `tag:homelab` |
## Services Hosted
**Native (via Ansible):**
- [[services/forgejo|Forgejo]] - Git forge
- [[services/zot|Zot]] - Container registry
- [[services/jellyfin|Jellyfin]] - Media server
- [[services/borgmatic|Borgmatic]] - Backup system
- [[services/alloy|Alloy]] - Metrics/logs collector
- Caddy - Reverse proxy for `*.ops.eblu.me`
**Kubernetes (via minikube):**
- [[kubernetes/apps|All k8s applications]]
## Related
- [[infrastructure/routing|Routing]] - Port mappings
- [[kubernetes/cluster|Cluster]] - Minikube details

View file

@ -0,0 +1,65 @@
---
title: Service Routing
tags:
- infrastructure
- network
---
# Service Routing
Services are accessible via two DNS domains with different reachability.
## DNS Domains
| Domain | Proxy | Reachable From |
|--------|-------|----------------|
| `*.ops.eblu.me` | Caddy on indri | k8s pods, docker containers, tailnet clients |
| `*.tail8d86e.ts.net` | Tailscale MagicDNS | Tailnet clients only |
**Use `*.ops.eblu.me`** for services that need pod-to-service communication.
## Caddy Services (`*.ops.eblu.me`)
DNS points to indri's Tailscale IP (100.98.163.89). TLS via Let's Encrypt (ACME DNS-01 with Gandi).
| Service | URL | Description |
|---------|-----|-------------|
| Homepage | https://go.ops.eblu.me | Service dashboard |
| [[services/forgejo|Forgejo]] | https://forge.ops.eblu.me | Git hosting (SSH: 2222) |
| [[services/zot|Zot]] | https://registry.ops.eblu.me | Container registry |
| [[services/grafana|Grafana]] | https://grafana.ops.eblu.me | Dashboards |
| [[services/argocd|ArgoCD]] | https://argocd.ops.eblu.me | GitOps CD |
| [[services/prometheus|Prometheus]] | https://prometheus.ops.eblu.me | Metrics |
| [[services/loki|Loki]] | https://loki.ops.eblu.me | Logs |
| [[services/miniflux|Miniflux]] | https://feed.ops.eblu.me | RSS reader |
| [[services/kiwix|Kiwix]] | https://kiwix.ops.eblu.me | Offline Wikipedia |
| [[services/transmission|Transmission]] | https://torrent.ops.eblu.me | BitTorrent |
| [[services/teslamate|TeslaMate]] | https://tesla.ops.eblu.me | Tesla logger |
| [[services/navidrome|Navidrome]] | https://dj.ops.eblu.me | Music streaming |
| [[services/jellyfin|Jellyfin]] | https://jellyfin.ops.eblu.me | Media server |
| [[services/postgresql|PostgreSQL]] | pg.ops.eblu.me:5432 | Database |
| [[storage/sifaka|Sifaka]] | https://nas.ops.eblu.me | NAS dashboard |
## Tailscale-Only Services
| Service | URL | Description |
|---------|-----|-------------|
| Kubernetes | https://k8s.tail8d86e.ts.net | Minikube API |
## Port Map (Indri)
| Port | Service | Protocol | Binding | Notes |
|------|---------|----------|---------|-------|
| 443 | Caddy | HTTPS | 0.0.0.0 | Reverse proxy |
| 2222 | Caddy L4 | TCP | 0.0.0.0 | SSH proxy to Forgejo |
| 5432 | Caddy L4 | TCP | 0.0.0.0 | PostgreSQL proxy |
| 2200 | Forgejo SSH | TCP | localhost | Built-in SSH server |
| 3001 | Forgejo | HTTP | localhost | Web UI |
| 5050 | Zot | HTTP | localhost | Registry API |
| 8096 | Jellyfin | HTTP | localhost | Media server |
| 44491 | K8s API | HTTPS | 0.0.0.0 | Minikube API server |
## Related
- [[infrastructure/tailscale|Tailscale]] - ACL configuration
- [[infrastructure/indri|Indri]] - Where services run

View file

@ -0,0 +1,62 @@
---
title: Tailscale
tags:
- infrastructure
- network
---
# Tailscale
Tailnet `tail8d86e.ts.net` provides secure networking for all BlumeOps infrastructure.
## ACL Management
ACLs managed via Pulumi in `pulumi/policy.hujson`.
## Groups
| Group | Members | Purpose |
|-------|---------|---------|
| `group:allisonflix` | admin, member | [[services/jellyfin|Jellyfin]] media access |
## Device Tags
| Tag | Devices | Purpose |
|-----|---------|---------|
| `tag:homelab` | indri | Server infrastructure |
| `tag:nas` | sifaka | Network-attached storage |
| `tag:blumeops` | indri, sifaka | Pulumi IaC managed resources |
| `tag:registry` | indri | Container registry access |
| `tag:k8s-api` | indri | Kubernetes API server access |
**Important:** Don't tag user-owned devices (like gilbert). Tagging converts them to "tagged devices" which lose user identity and break user-based SSH rules.
## Access Matrix
| Source | Kiwix | Forge | PyPI | Miniflux | PostgreSQL | NAS | Grafana | Loki |
|--------|-------|-------|------|----------|------------|-----|---------|------|
| `autogroup:admin` | Y | Y | Y | Y | Y | Y | Y | Y |
| `autogroup:member` | Y | Y | Y | Y | Y | - | - | - |
| `tag:homelab` | - | - | - | - | - | Y | - | - |
- **Admins** - full access to all services
- **Members** - member services only, no Grafana/Loki/NAS
## SSH Access
| Source | Destinations | Auth |
|--------|--------------|------|
| `autogroup:member` | `autogroup:self` | check |
| `autogroup:admin` | `tag:homelab` | check (12h) |
| `autogroup:admin` | `tag:nas` | check (12h) |
## OAuth Credentials
Pulumi uses OAuth client from 1Password (blumeops vault):
- Scopes: acl, dns, devices, services
- Auto-applies `tag:blumeops` to IaC-managed resources
## Related
- [[infrastructure/routing|Routing]] - Service URLs
- [[infrastructure/hosts|Hosts]] - Device inventory

View file

@ -0,0 +1,48 @@
---
title: ArgoCD Applications
tags:
- kubernetes
- argocd
---
# ArgoCD Applications
Registry of all applications deployed via [[services/argocd|ArgoCD]].
## Application Registry
| App | Namespace | Path/Source | Service |
|-----|-----------|-------------|---------|
| `apps` | argocd | `argocd/apps/` | App-of-apps root |
| `argocd` | argocd | `argocd/manifests/argocd/` | [[services/argocd|ArgoCD]] |
| `tailscale-operator` | tailscale | `argocd/manifests/tailscale-operator/` | Tailscale k8s operator |
| `1password-connect` | 1password | `argocd/manifests/1password-connect/` | [[services/1password|1Password]] |
| `external-secrets` | external-secrets | Helm chart | [[services/1password|1Password]] |
| `external-secrets-config` | external-secrets | `argocd/manifests/external-secrets-config/` | [[services/1password|1Password]] |
| `cloudnative-pg` | cnpg-system | Helm chart (forge mirror) | PostgreSQL operator |
| `blumeops-pg` | databases | `argocd/manifests/databases/` | [[services/postgresql|PostgreSQL]] |
| `prometheus` | monitoring | `argocd/manifests/prometheus/` | [[services/prometheus|Prometheus]] |
| `loki` | monitoring | `argocd/manifests/loki/` | [[services/loki|Loki]] |
| `grafana` | monitoring | Helm chart (forge mirror) | [[services/grafana|Grafana]] |
| `grafana-config` | monitoring | `argocd/manifests/grafana-config/` | [[services/grafana|Grafana]] |
| `immich` | immich | Helm chart | [[services/immich|Immich]] |
| `alloy-k8s` | alloy | `argocd/manifests/alloy-k8s/` | [[services/alloy|Alloy]] |
| `kube-state-metrics` | monitoring | `argocd/manifests/kube-state-metrics/` | K8s metrics |
| `miniflux` | miniflux | `argocd/manifests/miniflux/` | [[services/miniflux|Miniflux]] |
| `kiwix` | kiwix | `argocd/manifests/kiwix/` | [[services/kiwix|Kiwix]] |
| `torrent` | torrent | `argocd/manifests/torrent/` | [[services/transmission|Transmission]] |
| `navidrome` | navidrome | `argocd/manifests/navidrome/` | [[services/navidrome|Navidrome]] |
| `teslamate` | teslamate | `argocd/manifests/teslamate/` | [[services/teslamate|TeslaMate]] |
| `forgejo-runner` | forgejo-runner | `argocd/manifests/forgejo-runner/` | [[services/forgejo|Forgejo]] CI |
## Sync Policies
| Application | Policy | Rationale |
|-------------|--------|-----------|
| `apps` | Automated | Picks up new Application manifests |
| All others | Manual | Explicit control over deployments |
## Related
- [[services/argocd|ArgoCD]] - GitOps platform details
- [[kubernetes/cluster|Cluster]] - Kubernetes infrastructure

View file

@ -0,0 +1,39 @@
---
title: Kubernetes Cluster
tags:
- kubernetes
---
# Kubernetes Cluster
Single-node Minikube cluster running on [[infrastructure/indri|Indri]].
## Cluster Specifications
| Property | Value |
|----------|-------|
| **Driver** | docker |
| **Container Runtime** | docker |
| **Kubernetes Version** | v1.34.0 |
| **CPUs** | 6 |
| **Memory** | 11GB |
| **Disk** | 200GB |
| **API Server** | https://k8s.tail8d86e.ts.net |
**Prerequisites:** Docker Desktop with at least 12GB memory allocated.
## Volume Mounting
Pods mount NFS directly from [[storage/sifaka|Sifaka]]. Docker NATs outbound traffic through indri's LAN IP (192.168.1.50), allowing access to Sifaka's NFS exports.
## Registry Mirror
Containerd uses [[services/zot|Zot]] as a pull-through cache at `host.minikube.internal:5050`.
Mirrors configured: `registry.ops.eblu.me`, `docker.io`, `ghcr.io`, `quay.io`
## Related
- [[kubernetes/apps|Apps]] - ArgoCD applications
- [[services/argocd|ArgoCD]] - GitOps deployment
- [[services/zot|Zot]] - Registry mirror

View file

@ -0,0 +1,10 @@
---
title: External Secrets
tags:
- kubernetes
- secrets
---
# External Secrets
See [[services/1password|1Password]] in Services.

View file

@ -0,0 +1,15 @@
---
title: Backup
tags:
- operations
---
# Backup
Daily automated backups of BlumeOps data.
## Components
- [[services/borgmatic|Borgmatic]] - Backup orchestration
- [[storage/sifaka|Sifaka]] - Backup target (NAS)
- [[storage/backups|Backup Policy]] - What gets backed up and retention

View file

@ -0,0 +1,19 @@
---
title: Disaster Recovery
tags:
- operations
---
# Disaster Recovery
TBD. Current state:
- [[services/borgmatic|Borgmatic]] provides daily backups to [[storage/sifaka|Sifaka]]
- Infrastructure can be rebootstrapped using the blumeops repo
- Detailed DR procedures not yet documented
## Components
- [[services/borgmatic|Borgmatic]] - Backup restoration
- [[services/1password|1Password]] - Credential recovery
- [[services/forgejo|Forgejo]] - Source of truth for infrastructure code

View file

@ -0,0 +1,16 @@
---
title: Observability
tags:
- operations
---
# Observability
Metrics, logs, and dashboards for BlumeOps infrastructure.
## Components
- [[services/prometheus|Prometheus]] - Metrics storage and querying
- [[services/loki|Loki]] - Log aggregation
- [[services/alloy|Alloy]] - Metrics and log collection
- [[services/grafana|Grafana]] - Dashboards and visualization

View file

@ -0,0 +1,40 @@
---
title: 1Password
tags:
- service
- secrets
---
# 1Password
Root credential store for all BlumeOps secrets, synced to Kubernetes via External Secrets Operator.
## Architecture
```
1Password Cloud
|
v
1Password Connect (namespace: 1password)
|
v
External Secrets Operator (namespace: external-secrets)
|
v
Native Kubernetes Secrets
```
## Vault
The `blumeops` vault contains all infrastructure credentials.
## Kubernetes Integration
**ClusterSecretStore:** `onepassword-blumeops`
Services reference 1Password items via `ExternalSecret` manifests.
## Related
- [[services/argocd|ArgoCD]] - Uses secrets for git access
- [[services/postgresql|PostgreSQL]] - Database credentials

View file

@ -0,0 +1,54 @@
---
title: Grafana Alloy
tags:
- service
- observability
---
# Grafana Alloy
Unified observability collector for metrics and logs with two deployments:
1. **Indri (host)** - System metrics and service logs from macOS host
2. **Kubernetes (DaemonSet)** - Automatic pod log collection and service health probes
## Quick Reference
| Property | Value |
|----------|-------|
| **Indri Binary** | `~/.local/bin/alloy` |
| **Indri Config** | `~/.config/grafana-alloy/config.alloy` |
| **K8s Namespace** | `alloy` |
| **K8s Image** | `grafana/alloy:v1.8.2` |
| **ArgoCD App** | `alloy-k8s` |
## Metrics Collected
### From Indri
- System metrics via `prometheus.exporter.unix`
- Textfile collector: `minikube.prom`, `borgmatic.prom`, `zot.prom`, `jellyfin.prom`
- Zot registry metrics from `http://localhost:5050/metrics`
- Pushed to [[prometheus|Prometheus]] via remote_write
### From Kubernetes
- All pod logs via `loki.source.kubernetes`
- Service health probes: miniflux, kiwix, transmission, devpi, argocd
## Logs Collected
**Brew services:** forgejo, tailscale
**mcquack LaunchAgents:** alloy, borgmatic, zot, jellyfin
Logs pushed to [[loki|Loki]] at `https://loki.tail8d86e.ts.net/loki/api/v1/push`.
## Why Built from Source
The Homebrew bottle uses `CGO_ENABLED=0`, which breaks Tailscale MagicDNS. Building with `CGO_ENABLED=1` uses the macOS native resolver.
**Note:** This may no longer be needed now that services use `*.ops.eblu.me` URLs (routed via Caddy) instead of `*.tail8d86e.ts.net`. Should be tested in the future.
## Related
- [[prometheus|Prometheus]] - Metrics storage
- [[loki|Loki]] - Log storage
- [[grafana|Grafana]] - Visualization

View file

@ -0,0 +1,37 @@
---
title: ArgoCD
tags:
- service
- gitops
---
# ArgoCD
GitOps continuous delivery platform for the [[kubernetes/cluster|Kubernetes cluster]].
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://argocd.ops.eblu.me |
| **Tailscale URL** | https://argocd.tail8d86e.ts.net |
| **Namespace** | `argocd` |
| **Git Source** | `ssh://forgejo@indri.tail8d86e.ts.net:2200/eblume/blumeops.git` |
| **Manifests Path** | `argocd/` |
## Sync Policy
| Application | Sync Policy | Rationale |
|-------------|-------------|-----------|
| `apps` | Automated | Picks up new Application manifests |
| All workloads | Manual | Explicit control over deployments |
## Credentials
- Admin password: 1Password (blumeops vault)
- Git deploy key (SSH): 1Password
## Related
- [[kubernetes/apps|Apps]] - Full application registry
- [[services/forgejo|Forgejo]] - Git source

View file

@ -0,0 +1,60 @@
---
title: Borgmatic
tags:
- service
- backup
---
# Borgmatic
Daily backup system using Borg backup, running on indri.
## Quick Reference
| Property | Value |
|----------|-------|
| **Install** | mise (pipx) |
| **Config** | `~/.config/borgmatic/config.yaml` |
| **Schedule** | Daily at 2:00 AM |
| **Repository** | `/Volumes/backups/borg/` on [[storage/sifaka|Sifaka]] |
## What Gets Backed Up
**Directories:**
- `~/code/personal/zk` - Zettelkasten
- `/opt/homebrew/var/forgejo` - Git forge data
- `~/.config/borgmatic` - Borgmatic config
- `~/Documents` - Personal documents
- `~/Pictures` - Photos
**Databases:**
- `miniflux` on [[postgresql|PostgreSQL]]
- `teslamate` on [[postgresql|PostgreSQL]]
**Not backed up (by design):**
- ZIM archives (re-downloadable)
- Prometheus metrics (ephemeral)
- Loki logs (ephemeral)
## Retention Policy
| Period | Count |
|--------|-------|
| Daily | 7 |
| Monthly | 12 |
| Yearly | 1000 |
## Monitoring
Metrics exposed via textfile collector to [[prometheus|Prometheus]]:
- `borgmatic_up` - Repository accessibility
- `borgmatic_last_archive_timestamp` - Last backup time
- `borgmatic_repo_deduplicated_size_bytes` - Disk usage
Dashboard: "Borgmatic Backups" in [[grafana|Grafana]]
## Related
- [[storage/backups|Backups]] - Full backup policy
- [[storage/sifaka|Sifaka]] - Backup target
- [[postgresql|PostgreSQL]] - Database backups

View file

@ -0,0 +1,48 @@
---
title: Forgejo
tags:
- service
- git
- cicd
---
# Forgejo
Git forge and CI/CD platform. **Primary source of truth for blumeops** (mirrored to GitHub).
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://forge.ops.eblu.me |
| **SSH** | `ssh://forgejo@forge.ops.eblu.me:2222` |
| **Local Ports** | 3001 (HTTP), 2200 (SSH) |
| **Config** | `ansible/roles/forgejo/templates/app.ini.j2` |
## Repositories
| Repo | Description |
|------|-------------|
| `eblume/blumeops` | Infrastructure as code (primary) |
| `eblume/alloy` | Grafana Alloy fork (CGO build) |
| `eblume/tesla_auth` | Tesla OAuth helper |
| Helm chart mirrors | cloudnative-pg-charts, grafana-helm-charts |
## CI/CD (Forgejo Actions)
**Runner:** Kubernetes pod with Docker-in-Docker sidecar
- Namespace: `forgejo-runner`
- Labels: `k8s`
- ArgoCD app: `forgejo-runner`
**Workflows:** `.forgejo/workflows/`
- `build-container.yaml` - Container image builds on tag
## Secrets
Managed via 1Password: `lfs-jwt-secret`, `internal-token`, `oauth2-jwt-secret`, `runner_reg`
## Related
- [[services/argocd|ArgoCD]] - Uses Forgejo as git source
- [[services/zot|Zot]] - Container registry for built images

View file

@ -0,0 +1,50 @@
---
title: Grafana
tags:
- service
- observability
---
# Grafana
Dashboards and visualization for BlumeOps observability.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://grafana.ops.eblu.me |
| **Tailscale URL** | https://grafana.tail8d86e.ts.net |
| **Namespace** | `monitoring` |
| **Helm Chart** | grafana (mirrored to forge) |
| **Values** | `argocd/manifests/grafana/values.yaml` |
## Datasources
| Name | Type | Target |
|------|------|--------|
| Prometheus | prometheus | `prometheus.monitoring.svc.cluster.local:9090` |
| Loki | loki | `loki.monitoring.svc.cluster.local:3100` |
| TeslaMate | postgres | `blumeops-pg-rw.databases.svc.cluster.local:5432` |
## Dashboard Provisioning
Dashboards are ConfigMaps with label `grafana_dashboard: "1"`.
Location: `argocd/manifests/grafana-config/dashboards/`
Optional annotation: `grafana_folder: "FolderName"`
## Key Dashboards
- macOS System - Host metrics for indri
- Minikube - Kubernetes cluster overview
- Borgmatic Backups - Backup status and trends
- Services Health - HTTP probe results
- TeslaMate (18 dashboards) - Vehicle data
## Related
- [[prometheus|Prometheus]] - Metrics datasource
- [[loki|Loki]] - Logs datasource
- [[alloy|Alloy]] - Data collector

View file

@ -0,0 +1,26 @@
---
title: Immich
tags:
- service
- media
---
# Immich
Self-hosted photo and video management.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://photos.ops.eblu.me |
| **Namespace** | `immich` |
| **Deployment** | Helm chart (k8s) |
| **Database** | [[services/postgresql|PostgreSQL]] (CNPG) |
| **Storage** | [[storage/sifaka|Sifaka]] photos volume |
## Related
- [[services/postgresql|PostgreSQL]] - Database backend
- [[storage/sifaka|Sifaka]] - Photo storage
- [[services/jellyfin|Jellyfin]] - Video streaming (separate service)

View file

@ -0,0 +1,51 @@
---
title: Jellyfin
tags:
- service
- media
---
# Jellyfin
Open-source media server running natively on indri for VideoToolbox hardware transcoding.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://jellyfin.ops.eblu.me |
| **Local Port** | 8096 |
| **Data** | `~/Library/Application Support/jellyfin` |
| **Media** | `/Volumes/allisonflix` (NFS from sifaka) |
| **LaunchAgent** | `mcquack.jellyfin` |
## Hardware Transcoding
Apple VideoToolbox on M1 Mac Mini.
| Codec | Support |
|-------|---------|
| H.264 encode/decode | Hardware |
| HEVC (H.265) encode/decode | Hardware |
| AV1 decode | Software (requires M3+) |
| HDR to SDR tone mapping | VPP (hardware) |
Concurrent 4K streams with HDR tonemapping: ~3
## Configuration
Dashboard > Playback:
1. Hardware Acceleration: Apple VideoToolbox
2. Allow hardware encoding: Enabled
3. VPP Tone mapping: Enabled
## Observability
- Metrics: `jellyfin_metrics` ansible role
- Logs: Forwarded via [[alloy|Alloy]]
- Dashboard: "Jellyfin Media Server" in [[grafana|Grafana]]
## Related
- [[navidrome|Navidrome]] - Music streaming
- [[storage/sifaka|Sifaka]] - Media storage

View file

@ -0,0 +1,52 @@
---
title: Kiwix
tags:
- service
- knowledge
---
# Kiwix
Offline Wikipedia and ZIM archive server.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://kiwix.ops.eblu.me |
| **Tailscale URL** | https://kiwix.tail8d86e.ts.net |
| **Namespace** | `kiwix` |
| **Image** | `ghcr.io/kiwix/kiwix-serve:3.8.1` |
| **Storage** | NFS from [[storage/sifaka|Sifaka]] (`/volume1/torrents`) |
## Architecture
| Component | Purpose |
|-----------|---------|
| kiwix-serve | Serves ZIM files on port 80 |
| torrent-sync | Sidecar syncing ZIM torrents to [[transmission|Transmission]] |
| zim-watcher | CronJob (hourly) to restart on new ZIMs |
## Configured Archives
- Wikipedia top 1M English articles with images
- Project Gutenberg (60,000+ books)
- iFixit repair guides
- Stack Exchange (SuperUser, Math, etc.)
- LibreTexts textbooks
- DevDocs developer documentation
Full list: `argocd/manifests/kiwix/configmap-zim-torrents.yaml`
## Adding Archives
1. Edit `configmap-zim-torrents.yaml`
2. Add torrent URL from https://download.kiwix.org/zim/
3. Sync: `argocd app sync kiwix`
4. Torrent-sync adds to [[transmission|Transmission]]
5. zim-watcher restarts kiwix when download completes
## Related
- [[transmission|Transmission]] - Downloads ZIM files
- [[storage/sifaka|Sifaka]] - ZIM storage

View file

@ -0,0 +1,51 @@
---
title: Loki
tags:
- service
- observability
---
# Loki
Log aggregation system for BlumeOps infrastructure.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://loki.ops.eblu.me |
| **Tailscale URL** | https://loki.tail8d86e.ts.net |
| **Namespace** | `monitoring` |
| **Image** | `grafana/loki:3.4.2` |
| **Storage** | 50Gi PVC |
| **Retention** | 31 days |
## Architecture
- Single-node deployment with filesystem storage
- TSDB index with 24h period
- Logs collected by [[alloy|Alloy]] and pushed via Loki API
- Queried via [[grafana|Grafana]]
## Log Sources
**From Indri (via Alloy):**
- forgejo, tailscale (brew services)
- alloy, borgmatic, zot, jellyfin (LaunchAgents)
**From Kubernetes (via Alloy DaemonSet):**
- All pods in all namespaces
## Query Examples (LogQL)
```logql
{service="forgejo"} # All forgejo logs
{service="borgmatic", stream="stderr"} # Borgmatic errors
{host="indri"} |= "error" # All logs containing "error"
```
## Related
- [[alloy|Alloy]] - Log collector
- [[grafana|Grafana]] - Log visualization
- [[prometheus|Prometheus]] - Metrics counterpart

View file

@ -0,0 +1,41 @@
---
title: Miniflux
tags:
- service
- rss
---
# Miniflux
Minimalist RSS/Atom feed reader.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://feed.ops.eblu.me |
| **Tailscale URL** | https://feed.tail8d86e.ts.net |
| **Namespace** | `miniflux` |
| **Image** | `ghcr.io/miniflux/miniflux:latest` |
| **Database** | [[services/postgresql|PostgreSQL]] |
## Features
- Keyboard shortcuts for efficient reading
- Fever and Google Reader API compatible
- Mobile-friendly web interface
- OPML import/export
- Content scraping for full articles
## Database
Uses CloudNativePG cluster at `pg.ops.eblu.me`. Database user password stored in `blumeops-pg-app` secret (auto-generated by CNPG).
## Backup
Feed subscriptions and read state backed up via [[services/borgmatic|Borgmatic]] PostgreSQL hook.
## Related
- [[services/postgresql|PostgreSQL]] - Database backend
- [[services/borgmatic|Borgmatic]] - Data backup

View file

@ -0,0 +1,42 @@
---
title: Navidrome
tags:
- service
- media
---
# Navidrome
Self-hosted music streaming server.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://dj.ops.eblu.me |
| **Tailscale URL** | https://dj.tail8d86e.ts.net |
| **Namespace** | `navidrome` |
| **Manifests** | `argocd/manifests/navidrome/` |
## Storage
| Mount | Type | Source | Access |
|-------|------|--------|--------|
| /music | NFS PV | sifaka:/volume1/music | Read-only |
| /data | Local PVC (10Gi) | minikube storage | Read-write |
The `/data` directory contains SQLite database, configuration, and cache.
## Configuration
| Variable | Value |
|----------|-------|
| `ND_SCANSCHEDULE` | 1h |
| `ND_LOGLEVEL` | info |
| `ND_MUSICFOLDER` | /music |
| `ND_DATAFOLDER` | /data |
## Related
- [[services/jellyfin|Jellyfin]] - Video streaming
- [[storage/sifaka|Sifaka]] - Music storage

View file

@ -0,0 +1,59 @@
---
title: PostgreSQL
tags:
- service
- database
---
# PostgreSQL
Database cluster via CloudNativePG operator.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | `tcp://pg.ops.eblu.me:5432` |
| **Metrics** | `http://cnpg-metrics.tail8d86e.ts.net:9187/metrics` |
| **Namespace** | `databases` |
| **Cluster** | `blumeops-pg` |
| **Operator** | CloudNativePG |
## Databases
| Database | Owner | Purpose |
|----------|-------|---------|
| miniflux | miniflux | [[services/miniflux|Miniflux]] feed data |
| teslamate | teslamate | [[services/teslamate|TeslaMate]] vehicle data |
## Users
| User | Role | Purpose |
|------|------|---------|
| postgres | superuser | CNPG internal |
| miniflux | app owner | Owns miniflux database |
| teslamate | superuser | TeslaMate (needs extensions) |
| eblume | superuser | Admin access |
| borgmatic | pg_read_all_data | [[services/borgmatic|Backup]] access |
## Backup
Backed up via [[services/borgmatic|Borgmatic]] `postgresql_databases` hook. Streams `pg_dump` directly to Borg (no intermediate files, no downtime). See [[operations/backup|Backup]] for overall backup policy.
## Credentials
**1Password items:**
- `guxu3j7ajhjyey6xxl2ovsl2ui` - eblume password
- `mw2bv5we7woicjza7hc6s44yvy` - borgmatic password
**CNPG-managed secrets:**
- `blumeops-pg-app` - miniflux user
- `blumeops-pg-eblume` - eblume superuser
- `blumeops-pg-borgmatic` - borgmatic backup user
- `blumeops-pg-teslamate` - teslamate user
## Related
- [[services/miniflux|Miniflux]] - Feed reader database
- [[services/teslamate|TeslaMate]] - Vehicle data database
- [[services/borgmatic|Borgmatic]] - Database backup

View file

@ -0,0 +1,41 @@
---
title: Prometheus
tags:
- service
- observability
---
# Prometheus
Metrics storage and querying for BlumeOps infrastructure.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://prometheus.ops.eblu.me |
| **Tailscale URL** | https://prometheus.tail8d86e.ts.net |
| **Namespace** | `monitoring` |
| **Image** | `prom/prometheus:v3.2.1` |
| **Storage** | 50Gi PVC |
| **Manifests** | `argocd/manifests/prometheus/` |
## Data Sources
### Remote Write (from Alloy)
- Indri system metrics via [[services/alloy|Alloy]] remote_write
- Textfile metrics: minikube, borgmatic, zot, jellyfin
### Scrape Targets
| Target | Metrics |
|--------|---------|
| `sifaka:9100` | [[storage/sifaka|Sifaka]] NAS (node_exporter) |
| `cnpg-metrics.tail8d86e.ts.net:9187` | [[services/postgresql|CloudNativePG]] metrics |
| `kube-state-metrics.monitoring.svc:8080` | Kubernetes resource metrics |
## Related
- [[services/alloy|Alloy]] - Metrics collector
- [[services/grafana|Grafana]] - Visualization
- [[services/loki|Loki]] - Logs counterpart

View file

@ -0,0 +1,58 @@
---
title: TeslaMate
tags:
- service
- vehicle
---
# TeslaMate
Self-hosted Tesla data logger collecting vehicle telemetry from the Tesla Owner API.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://tesla.ops.eblu.me |
| **Tailscale URL** | https://tesla.tail8d86e.ts.net |
| **Namespace** | `teslamate` |
| **Image** | `teslamate/teslamate:2.2.0` |
| **Database** | [[postgresql|PostgreSQL]] |
## Data Collected
- Battery level, state of charge, range estimates
- Charging sessions (location, energy, cost, duration)
- Drives (distance, efficiency, routes)
- Climate/HVAC usage
- Software update history
- Vampire drain analysis
- Vehicle states (asleep, driving, charging, online)
## Grafana Dashboards
18 dashboards in the "TeslaMate" folder:
- Overview, Charges, Drives, Efficiency, States
- Battery Health, Vampire Drain, Statistics
- Charge Level, Locations, Trip, Mileage
- Drive Stats, Charging Stats, Projected Range
- Timeline, Updates, Visited
Dashboards use PostgreSQL datasource (not Prometheus).
## Authentication
Uses Tesla Owner API via OAuth:
1. Access https://tesla.ops.eblu.me
2. Click "Sign in with Tesla"
3. Tokens encrypted with ENCRYPTION_KEY
## Credentials
**1Password:** `TeslaMate` item with `db_password` and `api_enc_key`
## Related
- [[postgresql|PostgreSQL]] - Data storage
- [[grafana|Grafana]] - Dashboards
- [[borgmatic|Borgmatic]] - Database backup

View file

@ -0,0 +1,53 @@
---
title: Transmission
tags:
- service
- torrent
---
# Transmission
BitTorrent daemon, primarily for downloading ZIM archives for [[kiwix|Kiwix]].
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://torrent.ops.eblu.me |
| **Tailscale URL** | https://torrent.tail8d86e.ts.net |
| **Namespace** | `torrent` |
| **Image** | `lscr.io/linuxserver/transmission:latest` |
| **Storage** | NFS PVC from [[storage/sifaka|Sifaka]] |
## Storage Layout
NFS share on sifaka (`/volume1/torrents`):
| Path | Purpose |
|------|---------|
| `/downloads/` | Active downloads and metadata |
| `/downloads/complete/` | Completed downloads |
| `/config/` | Transmission configuration |
| `/watch/` | Watch directory for .torrent files |
[[kiwix|Kiwix]] reads from `/downloads/complete/` to serve ZIM archives.
## Integration with Kiwix
The Kiwix deployment includes a torrent-sync sidecar that:
1. Reads ZIM torrent list from ConfigMap
2. Adds missing torrents via RPC
3. Runs on startup and every 30 minutes
When downloads complete, the zim-watcher CronJob detects new ZIMs and restarts Kiwix.
## Monitoring
Basic uptime via blackbox probe in [[alloy|Alloy]] k8s (Services Health dashboard).
Web UI shows: active/seeding/paused counts, speeds, disk usage.
## Related
- [[kiwix|Kiwix]] - ZIM archive consumer
- [[storage/sifaka|Sifaka]] - Download storage

View file

@ -0,0 +1,42 @@
---
title: Zot
tags:
- service
- registry
---
# Zot
OCI-native container registry providing pull-through cache and private image storage.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://registry.ops.eblu.me |
| **Local Port** | 5050 |
| **Data** | `~/zot` |
| **Config** | `~/.config/zot/config.json` |
| **LaunchAgent** | mcquack |
## Namespace Convention
| Path | Source |
|------|--------|
| `registry.ops.eblu.me/docker.io/*` | Cached from Docker Hub |
| `registry.ops.eblu.me/ghcr.io/*` | Cached from GHCR |
| `registry.ops.eblu.me/quay.io/*` | Cached from Quay |
| `registry.ops.eblu.me/blumeops/*` | Private images |
## Pull-Through Cache
When [[kubernetes/cluster|minikube]] pulls an image, containerd checks zot first. If cached, returns immediately. If not, zot fetches from upstream, caches it, then returns.
## Security Model
Network access only (no authentication). Defense is the Tailscale ACL boundary.
## Related
- [[services/forgejo|Forgejo]] - Container build CI
- [[kubernetes/cluster|Cluster]] - Registry consumer

View file

@ -0,0 +1,71 @@
---
title: Backup Policy
tags:
- storage
- backup
---
# Backup Policy
Daily automated backups from [[infrastructure/indri|Indri]] to [[storage/sifaka|Sifaka]] NAS.
## Schedule
| Time | Frequency | System |
|------|-----------|--------|
| 2:00 AM | Daily | [[services/borgmatic|Borgmatic]] |
## What Gets Backed Up
### Directories
| Path | Description | Priority |
|------|-------------|----------|
| `~/code/personal/zk` | Zettelkasten notes | Critical |
| `/opt/homebrew/var/forgejo` | Git repositories | Critical |
| `~/.config/borgmatic` | Backup config | High |
| `~/Documents` | Personal documents | High |
| `~/Pictures` | Photos | Medium |
### Databases
| Database | Host | Method |
|----------|------|--------|
| miniflux | [[services/postgresql|pg.ops.eblu.me]] | pg_dump stream |
| teslamate | [[services/postgresql|pg.ops.eblu.me]] | pg_dump stream |
## What Is NOT Backed Up
| Data | Reason |
|------|--------|
| ZIM archives (`~/transmission/`) | Re-downloadable via torrent |
| Prometheus metrics | Ephemeral, in k8s PVC |
| Loki logs | Ephemeral, in k8s PVC |
| devpi cache | Re-fetchable from PyPI |
## Retention Policy
| Period | Retention |
|--------|-----------|
| Daily | 7 backups |
| Monthly | 12 backups |
| Yearly | 1000 backups |
## Backup Target
Repository: `/Volumes/backups/borg/` on [[storage/sifaka|Sifaka]]
## Monitoring
Metrics exposed to [[services/prometheus|Prometheus]]:
- `borgmatic_up` - Repository accessible
- `borgmatic_last_archive_timestamp` - Last backup time
- `borgmatic_repo_deduplicated_size_bytes` - Disk usage
Dashboard: "Borgmatic Backups" in [[services/grafana|Grafana]]
## Related
- [[services/borgmatic|Borgmatic]] - Backup system details
- [[storage/sifaka|Sifaka]] - Backup storage
- [[services/postgresql|PostgreSQL]] - Database backups

View file

@ -0,0 +1,10 @@
---
title: PostgreSQL
tags:
- storage
- database
---
# PostgreSQL
See [[services/postgresql|PostgreSQL]] in Services.

View file

@ -0,0 +1,59 @@
---
title: Sifaka NAS
tags:
- storage
---
# Sifaka NAS
Synology NAS providing network storage and backup target.
## Quick Reference
| Property | Value |
|----------|-------|
| **Dashboard** | https://nas.ops.eblu.me |
| **Model** | Synology |
| **Storage** | 10.9TB RAID 5 |
| **Role** | Backup target, media storage |
## Network Shares
| Share | Path | Purpose | Consumers |
|-------|------|---------|-----------|
| backups | `/volume1/backups` | Borg backup repository | [[services/borgmatic|Borgmatic]] |
| torrents | `/volume1/torrents` | ZIM downloads | [[services/kiwix|Kiwix]], [[services/transmission|Transmission]] |
| music | `/volume1/music` | Music library | [[services/navidrome|Navidrome]] |
| allisonflix | `/volume1/allisonflix` | Video library | [[services/jellyfin|Jellyfin]] |
| photos | `/volume1/photos` | Photo library | [[services/immich|Immich]] |
## NFS Exports
| Export | Allowed Clients | Purpose |
|--------|-----------------|---------|
| `/volume1/torrents` | 192.168.1.0/24, 100.64.0.0/10 | k8s pods via Docker NAT |
| `/volume1/music` | 192.168.1.0/24, 100.64.0.0/10 | k8s pods via Docker NAT |
| `/volume1/photos` | 192.168.1.0/24, 100.64.0.0/10 | k8s pods via Docker NAT |
## Monitoring
Node exporter running in Docker container, scraped by [[services/prometheus|Prometheus]] at `sifaka:9100`.
## Tailscale
- Tag: `tag:nas`
- ACL: `tag:homelab` can access for backups
## Backup
Sifaka is the **target** for [[operations/backup|backups]], not a backup source. [[services/borgmatic|Borgmatic]] sends backups TO sifaka, not OF sifaka.
Data protection for sifaka itself currently relies on the Synology RAID 5 configuration, which provides single-disk fault tolerance. Future plans include offsite duplication for additional resiliency.
## Related
- [[storage/backups|Backups]] - Backup policy
- [[services/borgmatic|Borgmatic]] - Backup system
- [[services/immich|Immich]] - Photo consumer
- [[services/jellyfin|Jellyfin]] - Media consumer
- [[services/navidrome|Navidrome]] - Music consumer