Add Reference section with 24 technical reference cards

Phase 2 of documentation restructuring. Creates docs/reference/ with:

Services (16):
- alloy, argocd, borgmatic, 1password, forgejo, grafana
- jellyfin, kiwix, loki, miniflux, navidrome, postgresql
- prometheus, teslamate, transmission, zot

Infrastructure (3):
- hosts - Device inventory
- tailscale - ACLs, groups, tags
- routing - DNS domains and port mappings

Kubernetes (2):
- cluster - Minikube specs
- apps - ArgoCD application registry

Storage (2):
- sifaka - Synology NAS configuration
- backups - Backup policy

All cards use wiki-links for cross-referencing and include YAML
frontmatter with title and tags for Quartz.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Erich Blume 2026-02-03 13:22:19 -08:00
commit ce1f696bd8
26 changed files with 1435 additions and 6 deletions

View file

@ -64,14 +64,16 @@ The documentation is being restructured to follow the [Diataxis](https://diataxi
**Docs URL:** https://docs.ops.eblu.me
### Phase 2: Reference
### Phase 2: Reference (Complete)
Information-oriented technical descriptions. Built first so other docs can link to reference material.
- [ ] Create `reference/` directory
- [ ] Service reference pages (migrate from zk cards)
- [ ] Infrastructure inventory
- [ ] Configuration reference
- [ ] API/CLI reference for mise tasks
- [x] Create `reference/` directory with index
- [x] Service reference pages (16 services: alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot)
- [x] Infrastructure inventory (hosts, tailscale, routing)
- [x] Kubernetes reference (cluster, apps)
- [x] Storage reference (sifaka, backups)
**Reference URL:** https://docs.ops.eblu.me/reference/
### Phase 3: Tutorials
Learning-oriented content for getting started.

View file

@ -0,0 +1 @@
Add Reference section with 24 technical reference cards covering services, infrastructure, kubernetes, and storage

54
docs/reference/index.md Normal file
View file

@ -0,0 +1,54 @@
---
title: Reference
tags:
- reference
---
# Reference
Technical specifications, inventories, and configuration details for BlumeOps infrastructure.
## Services
Individual service reference cards with URLs, configuration, and operational details.
| Service | Description | Location |
|---------|-------------|----------|
| [[services/alloy\|Alloy]] | Observability collector (metrics & logs) | indri + k8s |
| [[services/argocd\|ArgoCD]] | GitOps continuous delivery | k8s |
| [[services/borgmatic\|Borgmatic]] | Backup system | indri |
| [[services/1password\|1Password]] | Secrets management | cloud + k8s |
| [[services/forgejo\|Forgejo]] | Git forge & CI/CD | indri |
| [[services/grafana\|Grafana]] | Dashboards & visualization | k8s |
| [[services/jellyfin\|Jellyfin]] | Media server | indri |
| [[services/kiwix\|Kiwix]] | Offline Wikipedia & ZIM archives | k8s |
| [[services/loki\|Loki]] | Log aggregation | k8s |
| [[services/miniflux\|Miniflux]] | RSS feed reader | k8s |
| [[services/navidrome\|Navidrome]] | Music streaming | k8s |
| [[services/postgresql\|PostgreSQL]] | Database cluster | k8s |
| [[services/prometheus\|Prometheus]] | Metrics collection | k8s |
| [[services/teslamate\|TeslaMate]] | Tesla data logger | k8s |
| [[services/transmission\|Transmission]] | BitTorrent daemon | k8s |
| [[services/zot\|Zot]] | Container registry | indri |
## Infrastructure
Host inventory and network configuration.
- [[infrastructure/hosts\|Hosts]] - Device inventory (indri, gilbert, sifaka, etc.)
- [[infrastructure/tailscale\|Tailscale]] - ACLs, groups, tags
- [[infrastructure/routing\|Routing]] - DNS domains, port mappings
## Kubernetes
Cluster configuration and application registry.
- [[kubernetes/cluster\|Cluster]] - Minikube specs, storage, networking
- [[kubernetes/apps\|Apps]] - ArgoCD application registry
## Storage
Network storage and backup configuration.
- [[storage/sifaka\|Sifaka]] - Synology NAS configuration
- [[storage/backups\|Backups]] - Backup policy and schedule

View file

@ -0,0 +1,71 @@
---
title: Host Inventory
tags:
- infrastructure
---
# Host Inventory
All devices connected via [Tailscale](https://login.tailscale.com/) tailnet `tail8d86e.ts.net`.
## Devices
| Host | Description | Notes |
|------|-------------|-------|
| **Indri** | Mac Mini M1, 2020 | Primary server, 2TB internal disk |
| **[[storage/sifaka\|Sifaka]]** | Synology NAS | 10.9TB RAID 5, backup target |
| **Gilbert** | 13" MacBook Air M4, 2025 | Primary workstation |
| **Mouse** | 13" MacBook Air M2 | Allison's laptop |
| **UniFi** | UniFi Express 7 | Home WiFi network |
| **Dwarf** | iPad Air | Employer-provided, off tailnet |
## Indri Details
| Property | Value |
|----------|-------|
| **Model** | Mac mini M1, 2020 (Macmini9,1) |
| **Storage** | 2TB internal SSD |
| **macOS** | 15.7.3 (Sequoia) |
| **Role** | Primary server |
| **Tailscale IP** | 100.98.163.89 |
### Services Hosted
**Native (via Ansible):**
- [[services/forgejo\|Forgejo]] - Git forge
- [[services/zot\|Zot]] - Container registry
- [[services/jellyfin\|Jellyfin]] - Media server
- [[services/borgmatic\|Borgmatic]] - Backup system
- [[services/alloy\|Alloy]] - Metrics/logs collector
- Caddy - Reverse proxy
**Kubernetes (via minikube):**
- [[kubernetes/apps\|All k8s applications]]
### Sleep Prevention
Indri uses Amphetamine (App Store) to prevent sleep. Configuration:
- Start Session At Launch: enabled
- Default Duration: indefinite
- Allow Closed-Display Sleep: enabled
## Gilbert Details
| Property | Value |
|----------|-------|
| **Model** | 13" MacBook Air M4, 2025 |
| **Role** | Development workstation |
| **User** | eblume |
### Development Tools
Managed via `Brewfile` and `mise.toml`.
Fish abbreviations:
- `ki` -> `kubectl --context=minikube-indri`
- `k9i` -> `k9s --context=minikube-indri`
## Related
- [[infrastructure/tailscale\|Tailscale]] - Network configuration
- [[storage/sifaka\|Sifaka]] - NAS details

View file

@ -0,0 +1,79 @@
---
title: Service Routing
tags:
- infrastructure
- network
---
# Service Routing
Services are accessible via two DNS domains with different reachability.
## DNS Domains
| Domain | Proxy | Reachable From |
|--------|-------|----------------|
| `*.ops.eblu.me` | Caddy on indri | k8s pods, docker containers, tailnet clients |
| `*.tail8d86e.ts.net` | Tailscale MagicDNS | Tailnet clients only |
**Use `*.ops.eblu.me`** for services that need pod-to-service communication.
## Caddy Services (`*.ops.eblu.me`)
DNS points to indri's Tailscale IP (100.98.163.89). TLS via Let's Encrypt (ACME DNS-01 with Gandi).
| Service | URL | Description |
|---------|-----|-------------|
| Homepage | https://go.ops.eblu.me | Service dashboard |
| [[services/forgejo\|Forgejo]] | https://forge.ops.eblu.me | Git hosting (SSH: 2222) |
| [[services/zot\|Zot]] | https://registry.ops.eblu.me | Container registry |
| [[services/grafana\|Grafana]] | https://grafana.ops.eblu.me | Dashboards |
| [[services/argocd\|ArgoCD]] | https://argocd.ops.eblu.me | GitOps CD |
| [[services/prometheus\|Prometheus]] | https://prometheus.ops.eblu.me | Metrics |
| [[services/loki\|Loki]] | https://loki.ops.eblu.me | Logs |
| [[services/miniflux\|Miniflux]] | https://feed.ops.eblu.me | RSS reader |
| [[services/kiwix\|Kiwix]] | https://kiwix.ops.eblu.me | Offline Wikipedia |
| [[services/transmission\|Transmission]] | https://torrent.ops.eblu.me | BitTorrent |
| [[services/teslamate\|TeslaMate]] | https://tesla.ops.eblu.me | Tesla logger |
| [[services/navidrome\|Navidrome]] | https://dj.ops.eblu.me | Music streaming |
| [[services/jellyfin\|Jellyfin]] | https://jellyfin.ops.eblu.me | Media server |
| [[services/postgresql\|PostgreSQL]] | pg.ops.eblu.me:5432 | Database |
| [[storage/sifaka\|Sifaka]] | https://nas.ops.eblu.me | NAS dashboard |
## Tailscale-Only Services
| Service | URL | Description |
|---------|-----|-------------|
| Kubernetes | https://k8s.tail8d86e.ts.net | Minikube API |
## Port Map (Indri)
| Port | Service | Protocol | Binding | Notes |
|------|---------|----------|---------|-------|
| 443 | Caddy | HTTPS | 0.0.0.0 | Reverse proxy |
| 2222 | Caddy L4 | TCP | 0.0.0.0 | SSH proxy to Forgejo |
| 5432 | Caddy L4 | TCP | 0.0.0.0 | PostgreSQL proxy |
| 2200 | Forgejo SSH | TCP | localhost | Built-in SSH server |
| 3001 | Forgejo | HTTP | localhost | Web UI |
| 5050 | Zot | HTTP | localhost | Registry API |
| 8096 | Jellyfin | HTTP | localhost | Media server |
| 44491 | K8s API | HTTPS | 0.0.0.0 | Minikube API server |
## Adding New Services
### Indri Services (via Caddy)
1. Host service on localhost
2. Add to `ansible/roles/caddy/defaults/main.yml`
3. Run `mise run provision-indri -- --tags caddy`
### K8s Services (via Tailscale Ingress)
1. Create manifests in `argocd/manifests/<service>/`
2. Add ArgoCD Application in `argocd/apps/`
3. Add Tailscale Ingress annotation
4. Add Caddy proxy entry
5. Sync via ArgoCD
## Related
- [[infrastructure/tailscale\|Tailscale]] - ACL configuration
- [[infrastructure/hosts\|Hosts]] - Where services run

View file

@ -0,0 +1,67 @@
---
title: Tailscale
tags:
- infrastructure
- network
---
# Tailscale
Tailnet `tail8d86e.ts.net` provides secure networking for all BlumeOps infrastructure.
## ACL Management
ACLs managed via Pulumi in `pulumi/policy.hujson`.
```bash
mise run tailnet-preview # Preview changes
mise run tailnet-up # Apply changes
```
## Groups
| Group | Members | Purpose |
|-------|---------|---------|
| `group:allisonflix` | admin, member | [[services/jellyfin\|Jellyfin]] media access |
## Device Tags
| Tag | Devices | Purpose |
|-----|---------|---------|
| `tag:homelab` | indri | Server infrastructure |
| `tag:nas` | sifaka | Network-attached storage |
| `tag:blumeops` | indri, sifaka | Pulumi IaC managed resources |
| `tag:registry` | indri | Container registry access |
| `tag:k8s-api` | indri | Kubernetes API server access |
**Important:** Don't tag user-owned devices (like gilbert). Tagging converts them to "tagged devices" which lose user identity and break user-based SSH rules.
## Access Matrix
| Source | Kiwix | Forge | PyPI | Miniflux | PostgreSQL | NAS | Grafana | Loki |
|--------|-------|-------|------|----------|------------|-----|---------|------|
| `autogroup:admin` | Y | Y | Y | Y | Y | Y | Y | Y |
| `autogroup:member` | Y | Y | Y | Y | Y | - | - | - |
| `tag:homelab` | - | - | - | - | - | Y | - | - |
- **Admins** - full access to all services
- **Members** - member services only, no Grafana/Loki/NAS
## SSH Access
| Source | Destinations | Auth |
|--------|--------------|------|
| `autogroup:member` | `autogroup:self` | check |
| `autogroup:admin` | `tag:homelab` | check (12h) |
| `autogroup:admin` | `tag:nas` | check (12h) |
## OAuth Credentials
Pulumi uses OAuth client from 1Password (blumeops vault):
- Scopes: acl, dns, devices, services
- Auto-applies `tag:blumeops` to IaC-managed resources
## Related
- [[infrastructure/routing\|Routing]] - Service URLs
- [[infrastructure/hosts\|Hosts]] - Device inventory

View file

@ -0,0 +1,65 @@
---
title: ArgoCD Applications
tags:
- kubernetes
- argocd
---
# ArgoCD Applications
Registry of all applications deployed via [[services/argocd\|ArgoCD]].
## Application Registry
| App | Namespace | Path/Source | Service |
|-----|-----------|-------------|---------|
| `apps` | argocd | `argocd/apps/` | App-of-apps root |
| `argocd` | argocd | `argocd/manifests/argocd/` | [[services/argocd\|ArgoCD]] |
| `tailscale-operator` | tailscale | `argocd/manifests/tailscale-operator/` | Tailscale k8s operator |
| `1password-connect` | 1password | `argocd/manifests/1password-connect/` | [[services/1password\|1Password]] |
| `external-secrets` | external-secrets | Helm chart | [[services/1password\|1Password]] |
| `external-secrets-config` | external-secrets | `argocd/manifests/external-secrets-config/` | [[services/1password\|1Password]] |
| `cloudnative-pg` | cnpg-system | Helm chart (forge mirror) | PostgreSQL operator |
| `blumeops-pg` | databases | `argocd/manifests/databases/` | [[services/postgresql\|PostgreSQL]] |
| `prometheus` | monitoring | `argocd/manifests/prometheus/` | [[services/prometheus\|Prometheus]] |
| `loki` | monitoring | `argocd/manifests/loki/` | [[services/loki\|Loki]] |
| `grafana` | monitoring | Helm chart (forge mirror) | [[services/grafana\|Grafana]] |
| `grafana-config` | monitoring | `argocd/manifests/grafana-config/` | [[services/grafana\|Grafana]] |
| `alloy-k8s` | alloy | `argocd/manifests/alloy-k8s/` | [[services/alloy\|Alloy]] |
| `kube-state-metrics` | monitoring | `argocd/manifests/kube-state-metrics/` | K8s metrics |
| `miniflux` | miniflux | `argocd/manifests/miniflux/` | [[services/miniflux\|Miniflux]] |
| `kiwix` | kiwix | `argocd/manifests/kiwix/` | [[services/kiwix\|Kiwix]] |
| `torrent` | torrent | `argocd/manifests/torrent/` | [[services/transmission\|Transmission]] |
| `navidrome` | navidrome | `argocd/manifests/navidrome/` | [[services/navidrome\|Navidrome]] |
| `teslamate` | teslamate | `argocd/manifests/teslamate/` | [[services/teslamate\|TeslaMate]] |
| `forgejo-runner` | forgejo-runner | `argocd/manifests/forgejo-runner/` | [[services/forgejo\|Forgejo]] CI |
## Sync Policies
| Application | Policy | Rationale |
|-------------|--------|-----------|
| `apps` | Automated | Picks up new Application manifests |
| All others | Manual | Explicit control over deployments |
## Common Commands
```bash
argocd app list # List all apps
argocd app get <app> # Get details
argocd app diff <app> # Preview changes
argocd app sync <app> # Deploy changes
```
## PR Workflow
1. Create feature branch, modify manifests
2. Push to forge
3. Sync apps application: `argocd app sync apps`
4. Point service at branch: `argocd app set <service> --revision feature/branch`
5. Test: `argocd app sync <service>`
6. After merge, reset: `argocd app set <service> --revision main`
## Related
- [[services/argocd\|ArgoCD]] - GitOps platform details
- [[kubernetes/cluster\|Cluster]] - Kubernetes infrastructure

View file

@ -0,0 +1,73 @@
---
title: Kubernetes Cluster
tags:
- kubernetes
---
# Kubernetes Cluster
Single-node Minikube cluster running on [[infrastructure/hosts\|Indri]].
## Cluster Specifications
| Property | Value |
|----------|-------|
| **Driver** | docker |
| **Container Runtime** | docker |
| **Kubernetes Version** | v1.34.0 |
| **CPUs** | 6 |
| **Memory** | 11GB |
| **Disk** | 200GB |
| **API Server** | https://k8s.tail8d86e.ts.net |
**Prerequisites:** Docker Desktop with at least 12GB memory allocated.
## Remote Access
From gilbert:
```bash
mise run ensure-minikube-indri-kubectl-config
```
Fish abbreviations:
- `ki` -> `kubectl --context=minikube-indri`
- `k9i` -> `k9s --context=minikube-indri`
## Volume Mounting
Pods mount NFS directly from [[storage/sifaka\|Sifaka]]:
```yaml
volumes:
- name: torrents
nfs:
server: sifaka
path: /volume1/torrents
```
Docker NATs outbound traffic through indri's LAN IP (192.168.1.50), allowing access to Sifaka's NFS exports.
## Registry Mirror
Containerd uses [[services/zot\|Zot]] as a pull-through cache:
- Endpoint: `host.minikube.internal:5050`
- Config: `/etc/containerd/certs.d/<registry>/hosts.toml`
Mirrors configured: `registry.ops.eblu.me`, `docker.io`, `ghcr.io`, `quay.io`
## Useful Commands (on indri)
```bash
minikube status # Cluster status
minikube start # Start cluster
minikube stop # Stop cluster
minikube ssh # SSH into node
minikube logs # View logs
```
## Related
- [[kubernetes/apps\|Apps]] - ArgoCD applications
- [[services/argocd\|ArgoCD]] - GitOps deployment
- [[services/zot\|Zot]] - Registry mirror

View file

@ -0,0 +1,58 @@
---
title: 1Password
tags:
- service
- secrets
---
# 1Password
Root credential store for all BlumeOps secrets, synced to Kubernetes via External Secrets Operator.
## Architecture
```
1Password Cloud
|
v
1Password Connect (namespace: 1password)
|
v
External Secrets Operator (namespace: external-secrets)
|
v
Native Kubernetes Secrets
```
## Vault
The `blumeops` vault contains all infrastructure credentials.
## Kubernetes Integration
**ClusterSecretStore:** `onepassword-blumeops`
Services reference 1Password items via `ExternalSecret` manifests. Example: `argocd/manifests/devpi/external-secret.yaml`
## CLI Usage
```bash
# Get a secret field
op --vault blumeops item get <item-id> --fields <field> --reveal
# Inject into a template
op inject -i secret.yaml.tpl | kubectl apply -f -
```
## Bootstrap (Disaster Recovery)
1. Create Connect server: `op connect server create blumeops --vaults blumeops`
2. Create token: `op connect token create blumeops --server <id> --vault blumeops`
3. Store credentials in 1Password item "1Password Connect"
4. Apply bootstrap secret to k8s
5. Sync apps: 1password-connect, external-secrets-crds, external-secrets, external-secrets-config
## Related
- [[argocd\|ArgoCD]] - Uses secrets for git access
- [[postgresql\|PostgreSQL]] - Database credentials

View file

@ -0,0 +1,52 @@
---
title: Grafana Alloy
tags:
- service
- observability
---
# Grafana Alloy
Unified observability collector for metrics and logs with two deployments:
1. **Indri (host)** - System metrics and service logs from macOS host
2. **Kubernetes (DaemonSet)** - Automatic pod log collection and service health probes
## Quick Reference
| Property | Value |
|----------|-------|
| **Indri Binary** | `~/.local/bin/alloy` |
| **Indri Config** | `~/.config/grafana-alloy/config.alloy` |
| **K8s Namespace** | `alloy` |
| **K8s Image** | `grafana/alloy:v1.8.2` |
| **ArgoCD App** | `alloy-k8s` |
## Metrics Collected
### From Indri
- System metrics via `prometheus.exporter.unix`
- Textfile collector: `minikube.prom`, `borgmatic.prom`, `zot.prom`, `jellyfin.prom`
- Zot registry metrics from `http://localhost:5050/metrics`
- Pushed to [[prometheus\|Prometheus]] via remote_write
### From Kubernetes
- All pod logs via `loki.source.kubernetes`
- Service health probes: miniflux, kiwix, transmission, devpi, argocd
## Logs Collected
**Brew services:** forgejo, tailscale
**mcquack LaunchAgents:** alloy, borgmatic, zot, jellyfin
Logs pushed to [[loki\|Loki]] at `https://loki.tail8d86e.ts.net/loki/api/v1/push`.
## Why Built from Source
The Homebrew bottle uses `CGO_ENABLED=0`, which breaks Tailscale MagicDNS. Building with `CGO_ENABLED=1` uses the macOS native resolver.
## Related
- [[prometheus\|Prometheus]] - Metrics storage
- [[loki\|Loki]] - Log storage
- [[grafana\|Grafana]] - Visualization

View file

@ -0,0 +1,50 @@
---
title: ArgoCD
tags:
- service
- gitops
---
# ArgoCD
GitOps continuous delivery platform for the [[kubernetes/cluster\|Kubernetes cluster]].
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://argocd.ops.eblu.me |
| **Tailscale URL** | https://argocd.tail8d86e.ts.net |
| **Namespace** | `argocd` |
| **Git Source** | `ssh://forgejo@indri.tail8d86e.ts.net:2200/eblume/blumeops.git` |
| **Manifests Path** | `argocd/` |
## Sync Policy
| Application | Sync Policy | Rationale |
|-------------|-------------|-----------|
| `apps` | Automated | Picks up new Application manifests |
| All workloads | Manual | Explicit control over deployments |
## CLI Commands
```bash
# Login
argocd login argocd.ops.eblu.me --username admin --password "$(op ...)"
# Common operations
argocd app list
argocd app diff <app>
argocd app sync <app>
argocd app get <app>
```
## Credentials
- Admin password: 1Password (blumeops vault)
- Git deploy key (SSH): 1Password
## Related
- [[kubernetes/apps\|Apps]] - Full application registry
- [[forgejo\|Forgejo]] - Git source

View file

@ -0,0 +1,60 @@
---
title: Borgmatic
tags:
- service
- backup
---
# Borgmatic
Daily backup system using Borg backup, running on indri.
## Quick Reference
| Property | Value |
|----------|-------|
| **Install** | mise (pipx) |
| **Config** | `~/.config/borgmatic/config.yaml` |
| **Schedule** | Daily at 2:00 AM |
| **Repository** | `/Volumes/backups/borg/` on [[storage/sifaka\|Sifaka]] |
## What Gets Backed Up
**Directories:**
- `~/code/personal/zk` - Zettelkasten
- `/opt/homebrew/var/forgejo` - Git forge data
- `~/.config/borgmatic` - Borgmatic config
- `~/Documents` - Personal documents
- `~/Pictures` - Photos
**Databases:**
- `miniflux` on [[postgresql\|PostgreSQL]]
- `teslamate` on [[postgresql\|PostgreSQL]]
**Not backed up (by design):**
- ZIM archives (re-downloadable)
- Prometheus metrics (ephemeral)
- Loki logs (ephemeral)
## Retention Policy
| Period | Count |
|--------|-------|
| Daily | 7 |
| Monthly | 12 |
| Yearly | 1000 |
## Monitoring
Metrics exposed via textfile collector to [[prometheus\|Prometheus]]:
- `borgmatic_up` - Repository accessibility
- `borgmatic_last_archive_timestamp` - Last backup time
- `borgmatic_repo_deduplicated_size_bytes` - Disk usage
Dashboard: "Borgmatic Backups" in [[grafana\|Grafana]]
## Related
- [[storage/backups\|Backups]] - Full backup policy
- [[storage/sifaka\|Sifaka]] - Backup target
- [[postgresql\|PostgreSQL]] - Database backups

View file

@ -0,0 +1,58 @@
---
title: Forgejo
tags:
- service
- git
- cicd
---
# Forgejo
Git forge and CI/CD platform. **Primary source of truth for blumeops** (mirrored to GitHub).
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://forge.ops.eblu.me |
| **SSH** | `ssh://forgejo@forge.ops.eblu.me:2222` |
| **Local Ports** | 3001 (HTTP), 2200 (SSH) |
| **Config** | `ansible/roles/forgejo/templates/app.ini.j2` |
## Repositories
| Repo | Description |
|------|-------------|
| `eblume/blumeops` | Infrastructure as code (primary) |
| `eblume/alloy` | Grafana Alloy fork (CGO build) |
| `eblume/tesla_auth` | Tesla OAuth helper |
| Helm chart mirrors | cloudnative-pg-charts, grafana-helm-charts |
## CI/CD (Forgejo Actions)
**Runner:** Kubernetes pod with Docker-in-Docker sidecar
- Namespace: `forgejo-runner`
- Labels: `k8s`
- ArgoCD app: `forgejo-runner`
**Workflows:** `.forgejo/workflows/`
- `build-container.yaml` - Container image builds on tag
**Container release:**
```bash
mise run container-list # List containers
mise run container-release runner v1.0.0 # Tag and build
```
## Ansible Management
```bash
mise run provision-indri -- --tags forgejo
```
Secrets fetched from 1Password: `lfs-jwt-secret`, `internal-token`, `oauth2-jwt-secret`, `runner_reg`
## Related
- [[argocd\|ArgoCD]] - Uses Forgejo as git source
- [[zot\|Zot]] - Container registry for built images

View file

@ -0,0 +1,50 @@
---
title: Grafana
tags:
- service
- observability
---
# Grafana
Dashboards and visualization for BlumeOps observability.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://grafana.ops.eblu.me |
| **Tailscale URL** | https://grafana.tail8d86e.ts.net |
| **Namespace** | `monitoring` |
| **Helm Chart** | grafana (mirrored to forge) |
| **Values** | `argocd/manifests/grafana/values.yaml` |
## Datasources
| Name | Type | Target |
|------|------|--------|
| Prometheus | prometheus | `prometheus.monitoring.svc.cluster.local:9090` |
| Loki | loki | `loki.monitoring.svc.cluster.local:3100` |
| TeslaMate | postgres | `blumeops-pg-rw.databases.svc.cluster.local:5432` |
## Dashboard Provisioning
Dashboards are ConfigMaps with label `grafana_dashboard: "1"`.
Location: `argocd/manifests/grafana-config/dashboards/`
Optional annotation: `grafana_folder: "FolderName"`
## Key Dashboards
- macOS System - Host metrics for indri
- Minikube - Kubernetes cluster overview
- Borgmatic Backups - Backup status and trends
- Services Health - HTTP probe results
- TeslaMate (18 dashboards) - Vehicle data
## Related
- [[prometheus\|Prometheus]] - Metrics datasource
- [[loki\|Loki]] - Logs datasource
- [[alloy\|Alloy]] - Data collector

View file

@ -0,0 +1,51 @@
---
title: Jellyfin
tags:
- service
- media
---
# Jellyfin
Open-source media server running natively on indri for VideoToolbox hardware transcoding.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://jellyfin.ops.eblu.me |
| **Local Port** | 8096 |
| **Data** | `~/Library/Application Support/jellyfin` |
| **Media** | `/Volumes/allisonflix` (NFS from sifaka) |
| **LaunchAgent** | `mcquack.jellyfin` |
## Hardware Transcoding
Apple VideoToolbox on M1 Mac Mini.
| Codec | Support |
|-------|---------|
| H.264 encode/decode | Hardware |
| HEVC (H.265) encode/decode | Hardware |
| AV1 decode | Software (requires M3+) |
| HDR to SDR tone mapping | VPP (hardware) |
Concurrent 4K streams with HDR tonemapping: ~3
## Configuration
Dashboard > Playback:
1. Hardware Acceleration: Apple VideoToolbox
2. Allow hardware encoding: Enabled
3. VPP Tone mapping: Enabled
## Observability
- Metrics: `jellyfin_metrics` ansible role
- Logs: Forwarded via [[alloy\|Alloy]]
- Dashboard: "Jellyfin Media Server" in [[grafana\|Grafana]]
## Related
- [[navidrome\|Navidrome]] - Music streaming
- [[storage/sifaka\|Sifaka]] - Media storage

View file

@ -0,0 +1,52 @@
---
title: Kiwix
tags:
- service
- knowledge
---
# Kiwix
Offline Wikipedia and ZIM archive server.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://kiwix.ops.eblu.me |
| **Tailscale URL** | https://kiwix.tail8d86e.ts.net |
| **Namespace** | `kiwix` |
| **Image** | `ghcr.io/kiwix/kiwix-serve:3.8.1` |
| **Storage** | NFS from [[storage/sifaka\|Sifaka]] (`/volume1/torrents`) |
## Architecture
| Component | Purpose |
|-----------|---------|
| kiwix-serve | Serves ZIM files on port 80 |
| torrent-sync | Sidecar syncing ZIM torrents to [[transmission\|Transmission]] |
| zim-watcher | CronJob (hourly) to restart on new ZIMs |
## Configured Archives
- Wikipedia top 1M English articles with images
- Project Gutenberg (60,000+ books)
- iFixit repair guides
- Stack Exchange (SuperUser, Math, etc.)
- LibreTexts textbooks
- DevDocs developer documentation
Full list: `argocd/manifests/kiwix/configmap-zim-torrents.yaml`
## Adding Archives
1. Edit `configmap-zim-torrents.yaml`
2. Add torrent URL from https://download.kiwix.org/zim/
3. Sync: `argocd app sync kiwix`
4. Torrent-sync adds to [[transmission\|Transmission]]
5. zim-watcher restarts kiwix when download completes
## Related
- [[transmission\|Transmission]] - Downloads ZIM files
- [[storage/sifaka\|Sifaka]] - ZIM storage

View file

@ -0,0 +1,51 @@
---
title: Loki
tags:
- service
- observability
---
# Loki
Log aggregation system for BlumeOps infrastructure.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://loki.ops.eblu.me |
| **Tailscale URL** | https://loki.tail8d86e.ts.net |
| **Namespace** | `monitoring` |
| **Image** | `grafana/loki:3.4.2` |
| **Storage** | 50Gi PVC |
| **Retention** | 31 days |
## Architecture
- Single-node deployment with filesystem storage
- TSDB index with 24h period
- Logs collected by [[alloy\|Alloy]] and pushed via Loki API
- Queried via [[grafana\|Grafana]]
## Log Sources
**From Indri (via Alloy):**
- forgejo, tailscale (brew services)
- alloy, borgmatic, zot, jellyfin (LaunchAgents)
**From Kubernetes (via Alloy DaemonSet):**
- All pods in all namespaces
## Query Examples (LogQL)
```logql
{service="forgejo"} # All forgejo logs
{service="borgmatic", stream="stderr"} # Borgmatic errors
{host="indri"} |= "error" # All logs containing "error"
```
## Related
- [[alloy\|Alloy]] - Log collector
- [[grafana\|Grafana]] - Log visualization
- [[prometheus\|Prometheus]] - Metrics counterpart

View file

@ -0,0 +1,49 @@
---
title: Miniflux
tags:
- service
- rss
---
# Miniflux
Minimalist RSS/Atom feed reader.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://feed.ops.eblu.me |
| **Tailscale URL** | https://feed.tail8d86e.ts.net |
| **Namespace** | `miniflux` |
| **Image** | `ghcr.io/miniflux/miniflux:latest` |
| **Database** | [[postgresql\|PostgreSQL]] |
## Features
- Keyboard shortcuts for efficient reading
- Fever and Google Reader API compatible
- Mobile-friendly web interface
- OPML import/export
- Content scraping for full articles
## Database
Uses CloudNativePG cluster at `pg.ops.eblu.me`.
Database user password stored in `blumeops-pg-app` secret (auto-generated by CNPG).
## Backup
Feed subscriptions and read state backed up via [[borgmatic\|Borgmatic]] PostgreSQL hook.
## Health Check
```bash
curl https://feed.ops.eblu.me/healthcheck
```
## Related
- [[postgresql\|PostgreSQL]] - Database backend
- [[borgmatic\|Borgmatic]] - Data backup

View file

@ -0,0 +1,52 @@
---
title: Navidrome
tags:
- service
- media
---
# Navidrome
Self-hosted music streaming server.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://dj.ops.eblu.me |
| **Tailscale URL** | https://dj.tail8d86e.ts.net |
| **Namespace** | `navidrome` |
| **Manifests** | `argocd/manifests/navidrome/` |
## Storage
| Mount | Type | Source | Access |
|-------|------|--------|--------|
| /music | NFS PV | sifaka:/volume1/music | Read-only |
| /data | Local PVC (10Gi) | minikube storage | Read-write |
The `/data` directory contains SQLite database, configuration, and cache.
## Configuration
| Variable | Value |
|----------|-------|
| `ND_SCANSCHEDULE` | 1h |
| `ND_LOGLEVEL` | info |
| `ND_MUSICFOLDER` | /music |
| `ND_DATAFOLDER` | /data |
## Initial Setup
On first access, Navidrome prompts to create an admin user. No default credentials.
## Verify NFS Mount
```bash
kubectl --context=minikube-indri -n navidrome exec deploy/navidrome -- ls /music
```
## Related
- [[jellyfin\|Jellyfin]] - Video streaming
- [[storage/sifaka\|Sifaka]] - Music storage

View file

@ -0,0 +1,68 @@
---
title: PostgreSQL
tags:
- service
- database
---
# PostgreSQL
Database cluster via CloudNativePG operator.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | `tcp://pg.ops.eblu.me:5432` |
| **Metrics** | `http://cnpg-metrics.tail8d86e.ts.net:9187/metrics` |
| **Namespace** | `databases` |
| **Cluster** | `blumeops-pg` |
| **Operator** | CloudNativePG |
## Databases
| Database | Owner | Purpose |
|----------|-------|---------|
| miniflux | miniflux | [[miniflux\|Miniflux]] feed data |
| teslamate | teslamate | [[teslamate\|TeslaMate]] vehicle data |
## Users
| User | Role | Purpose |
|------|------|---------|
| postgres | superuser | CNPG internal |
| miniflux | app owner | Owns miniflux database |
| teslamate | superuser | TeslaMate (needs extensions) |
| eblume | superuser | Admin access |
| borgmatic | pg_read_all_data | [[borgmatic\|Backup]] access |
## Quick Connect
```bash
PGPASSWORD=$(op --vault blumeops item get <item-id> --fields password --reveal) \
psql -h pg.ops.eblu.me -U eblume -d miniflux
```
## Backup
Backed up via [[borgmatic\|Borgmatic]] `postgresql_databases` hook.
Borgmatic streams `pg_dump` directly to Borg (no intermediate files, no downtime).
## Credentials
**1Password items:**
- `guxu3j7ajhjyey6xxl2ovsl2ui` - eblume password
- `mw2bv5we7woicjza7hc6s44yvy` - borgmatic password
**CNPG-managed secrets:**
- `blumeops-pg-app` - miniflux user
- `blumeops-pg-eblume` - eblume superuser
- `blumeops-pg-borgmatic` - borgmatic backup user
- `blumeops-pg-teslamate` - teslamate user
## Related
- [[miniflux\|Miniflux]] - Feed reader database
- [[teslamate\|TeslaMate]] - Vehicle data database
- [[borgmatic\|Borgmatic]] - Database backup

View file

@ -0,0 +1,54 @@
---
title: Prometheus
tags:
- service
- observability
---
# Prometheus
Metrics storage and querying for BlumeOps infrastructure.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://prometheus.ops.eblu.me |
| **Tailscale URL** | https://prometheus.tail8d86e.ts.net |
| **Namespace** | `monitoring` |
| **Image** | `prom/prometheus:v3.2.1` |
| **Storage** | 50Gi PVC |
## Data Sources
### Remote Write (from Alloy)
- Indri system metrics via [[alloy\|Alloy]] remote_write
- Textfile metrics: minikube, borgmatic, zot, jellyfin
### Scrape Targets
| Target | Metrics |
|--------|---------|
| `sifaka:9100` | [[storage/sifaka\|Sifaka]] NAS (node_exporter) |
| `cnpg-metrics.tail8d86e.ts.net:9187` | [[postgresql\|CloudNativePG]] metrics |
| `kube-state-metrics.monitoring.svc:8080` | Kubernetes resource metrics |
## Query API
```bash
# Check targets
curl -s https://prometheus.ops.eblu.me/api/v1/targets | jq '.data.activeTargets[].scrapeUrl'
```
## ArgoCD Management
```bash
argocd app sync prometheus
```
Manifests: `argocd/manifests/prometheus/`
## Related
- [[alloy\|Alloy]] - Metrics collector
- [[grafana\|Grafana]] - Visualization
- [[loki\|Loki]] - Logs counterpart

View file

@ -0,0 +1,58 @@
---
title: TeslaMate
tags:
- service
- vehicle
---
# TeslaMate
Self-hosted Tesla data logger collecting vehicle telemetry from the Tesla Owner API.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://tesla.ops.eblu.me |
| **Tailscale URL** | https://tesla.tail8d86e.ts.net |
| **Namespace** | `teslamate` |
| **Image** | `teslamate/teslamate:2.2.0` |
| **Database** | [[postgresql\|PostgreSQL]] |
## Data Collected
- Battery level, state of charge, range estimates
- Charging sessions (location, energy, cost, duration)
- Drives (distance, efficiency, routes)
- Climate/HVAC usage
- Software update history
- Vampire drain analysis
- Vehicle states (asleep, driving, charging, online)
## Grafana Dashboards
18 dashboards in the "TeslaMate" folder:
- Overview, Charges, Drives, Efficiency, States
- Battery Health, Vampire Drain, Statistics
- Charge Level, Locations, Trip, Mileage
- Drive Stats, Charging Stats, Projected Range
- Timeline, Updates, Visited
Dashboards use PostgreSQL datasource (not Prometheus).
## Authentication
Uses Tesla Owner API via OAuth:
1. Access https://tesla.ops.eblu.me
2. Click "Sign in with Tesla"
3. Tokens encrypted with ENCRYPTION_KEY
## Credentials
**1Password:** `TeslaMate` item with `db_password` and `api_enc_key`
## Related
- [[postgresql\|PostgreSQL]] - Data storage
- [[grafana\|Grafana]] - Dashboards
- [[borgmatic\|Borgmatic]] - Database backup

View file

@ -0,0 +1,53 @@
---
title: Transmission
tags:
- service
- torrent
---
# Transmission
BitTorrent daemon, primarily for downloading ZIM archives for [[kiwix\|Kiwix]].
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://torrent.ops.eblu.me |
| **Tailscale URL** | https://torrent.tail8d86e.ts.net |
| **Namespace** | `torrent` |
| **Image** | `lscr.io/linuxserver/transmission:latest` |
| **Storage** | NFS PVC from [[storage/sifaka\|Sifaka]] |
## Storage Layout
NFS share on sifaka (`/volume1/torrents`):
| Path | Purpose |
|------|---------|
| `/downloads/` | Active downloads and metadata |
| `/downloads/complete/` | Completed downloads |
| `/config/` | Transmission configuration |
| `/watch/` | Watch directory for .torrent files |
[[kiwix\|Kiwix]] reads from `/downloads/complete/` to serve ZIM archives.
## Integration with Kiwix
The Kiwix deployment includes a torrent-sync sidecar that:
1. Reads ZIM torrent list from ConfigMap
2. Adds missing torrents via RPC
3. Runs on startup and every 30 minutes
When downloads complete, the zim-watcher CronJob detects new ZIMs and restarts Kiwix.
## Monitoring
Basic uptime via blackbox probe in [[alloy\|Alloy]] k8s (Services Health dashboard).
Web UI shows: active/seeding/paused counts, speeds, disk usage.
## Related
- [[kiwix\|Kiwix]] - ZIM archive consumer
- [[storage/sifaka\|Sifaka]] - Download storage

View file

@ -0,0 +1,66 @@
---
title: Zot
tags:
- service
- registry
---
# Zot
OCI-native container registry providing pull-through cache and private image storage.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://registry.ops.eblu.me |
| **Local Port** | 5050 |
| **Data** | `~/zot` |
| **Config** | `~/.config/zot/config.json` |
| **LaunchAgent** | mcquack |
## Namespace Convention
| Path | Source |
|------|--------|
| `registry.ops.eblu.me/docker.io/*` | Cached from Docker Hub |
| `registry.ops.eblu.me/ghcr.io/*` | Cached from GHCR |
| `registry.ops.eblu.me/quay.io/*` | Cached from Quay |
| `registry.ops.eblu.me/blumeops/*` | Private images |
## Pull-Through Cache
When [[kubernetes/cluster\|minikube]] pulls an image:
1. Containerd checks zot first (`host.minikube.internal:5050`)
2. If cached, returns immediately
3. If not, zot fetches from upstream, caches, returns
## Private Images
```bash
# Build and push from gilbert
podman build -t registry.ops.eblu.me/blumeops/myapp:v1 .
podman push registry.ops.eblu.me/blumeops/myapp:v1
# Use in k8s manifest
image: registry.ops.eblu.me/blumeops/myapp:v1
```
## Security Model
Network access only (no authentication). Defense is the Tailscale ACL boundary.
## Useful Commands
```bash
# List all images
curl -s http://indri:5050/v2/_catalog | jq
# List tags
curl -s http://indri:5050/v2/blumeops/devpi/tags/list | jq
```
## Related
- [[forgejo\|Forgejo]] - Container build CI
- [[kubernetes/cluster\|Cluster]] - Registry consumer

View file

@ -0,0 +1,84 @@
---
title: Backup Policy
tags:
- storage
- backup
---
# Backup Policy
Daily automated backups from [[infrastructure/hosts\|Indri]] to [[storage/sifaka\|Sifaka]] NAS.
## Schedule
| Time | Frequency | System |
|------|-----------|--------|
| 2:00 AM | Daily | [[services/borgmatic\|Borgmatic]] |
## What Gets Backed Up
### Directories
| Path | Description | Priority |
|------|-------------|----------|
| `~/code/personal/zk` | Zettelkasten notes | Critical |
| `/opt/homebrew/var/forgejo` | Git repositories | Critical |
| `~/.config/borgmatic` | Backup config | High |
| `~/Documents` | Personal documents | High |
| `~/Pictures` | Photos | Medium |
### Databases
| Database | Host | Method |
|----------|------|--------|
| miniflux | [[services/postgresql\|pg.ops.eblu.me]] | pg_dump stream |
| teslamate | [[services/postgresql\|pg.ops.eblu.me]] | pg_dump stream |
## What Is NOT Backed Up
| Data | Reason |
|------|--------|
| ZIM archives (`~/transmission/`) | Re-downloadable via torrent |
| Prometheus metrics | Ephemeral, in k8s PVC |
| Loki logs | Ephemeral, in k8s PVC |
| devpi cache | Re-fetchable from PyPI |
## Retention Policy
| Period | Retention |
|--------|-----------|
| Daily | 7 backups |
| Monthly | 12 backups |
| Yearly | 1000 backups |
## Backup Target
Repository: `/Volumes/backups/borg/` on [[storage/sifaka\|Sifaka]]
## Monitoring
Metrics exposed to [[services/prometheus\|Prometheus]]:
- `borgmatic_up` - Repository accessible
- `borgmatic_last_archive_timestamp` - Last backup time
- `borgmatic_repo_deduplicated_size_bytes` - Disk usage
Dashboard: "Borgmatic Backups" in [[services/grafana\|Grafana]]
## Recovery
```bash
# List archives
ssh indri 'mise x -- borgmatic list'
# Extract specific path from latest
ssh indri 'mise x -- borgmatic extract --archive latest --path /some/path'
# Check repository health
ssh indri 'mise x -- borgmatic check'
```
## Related
- [[services/borgmatic\|Borgmatic]] - Backup system details
- [[storage/sifaka\|Sifaka]] - Backup storage
- [[services/postgresql\|PostgreSQL]] - Database backups

View file

@ -0,0 +1,51 @@
---
title: Sifaka NAS
tags:
- storage
---
# Sifaka NAS
Synology NAS providing network storage and backup target.
## Quick Reference
| Property | Value |
|----------|-------|
| **Dashboard** | https://nas.ops.eblu.me |
| **Model** | Synology |
| **Storage** | 10.9TB RAID 5 |
| **Role** | Backup target, media storage |
## Network Shares
| Share | Path | Purpose | Consumers |
|-------|------|---------|-----------|
| backups | `/volume1/backups` | Borg backup repository | [[services/borgmatic\|Borgmatic]] |
| torrents | `/volume1/torrents` | ZIM downloads | [[services/kiwix\|Kiwix]], [[services/transmission\|Transmission]] |
| music | `/volume1/music` | Music library | [[services/navidrome\|Navidrome]] |
| allisonflix | `/volume1/allisonflix` | Video library | [[services/jellyfin\|Jellyfin]] |
| photos | `/volume1/photos` | Photo library | Immich |
## NFS Exports
| Export | Allowed Clients | Purpose |
|--------|-----------------|---------|
| `/volume1/torrents` | 192.168.1.0/24, 100.64.0.0/10 | k8s pods via Docker NAT |
| `/volume1/music` | 192.168.1.0/24, 100.64.0.0/10 | k8s pods via Docker NAT |
## Monitoring
Node exporter running in Docker container, scraped by [[services/prometheus\|Prometheus]] at `sifaka:9100`.
## Tailscale
- Tag: `tag:nas`
- ACL: `tag:homelab` can access for backups
## Related
- [[storage/backups\|Backups]] - Backup policy
- [[services/borgmatic\|Borgmatic]] - Backup system
- [[services/jellyfin\|Jellyfin]] - Media consumer
- [[services/navidrome\|Navidrome]] - Music consumer