eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions
blumeops/docs/zk/1767747119-YCPO.md
Erich Blume b8104d75ad Move zk cards to docs/zk/ for documentation restructuring (#84) 
## Summary
- Move all existing zettelkasten cards from `docs/` to `docs/zk/` as a temporary holding area
- Update `zk-docs` mise task to look in the new location
- Add `docs/README.md` explaining the Diataxis-based restructuring plan and target audiences

## Context
This is phase 1 of a multi-phase documentation restructuring effort. The goal is to reorganize docs to follow the Diataxis framework while serving multiple audiences:
1. Erich (owner) - knowledge graph/zk
2. Claude/AI agents - memory and context enrichment
3. New external readers - high-level overview
4. Potential operators/contributors - onboarding
5. Replicators - people wanting to duplicate the approach

## Testing
- [x] Verified `mise run zk-docs` still works with the new path
- [x] Updated obsidian.nvim config (in ~/.config/nvim) to point to new path

## Note
The obsidian.nvim config change is outside this repo but was made as part of this work.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/84
2026-02-03 09:13:50 -08:00

12 KiB
Raw Blame History

id aliases tags
1767747119-YCPO
blumeops
BlumeOps
blumeops

BlumeOps, aka Blue Mops, refers to my own personal computing operations stack.

Source code: https://forge.ops.eblu.me/eblume/blumeops (mirrored to https://github.com/eblume/blumeops)

Infrastructure

Host Description Notes
**[[indri Indri]]** Mac Mini M1, 2020
Sifaka Synology NAS 10.9TB RAID 5, backup target
Gilbert 13" MacBook Air M4, 2025 Primary workstation
Mouse 13" MacBook Air M2 Allison's laptop
UniFi UniFi Express 7 Home WiFi network (cloud)
Dwarf iPad Air Employer-provided, off tailnet

All devices are connected via Tailscale tailnet tail8d86e.ts.net.

Tailscale Access Control

ACLs are managed via Pulumi in pulumi/policy.hujson. See pulumi for deployment commands.

Important lesson learned:

  • Don't tag user-owned devices (like gilbert) - tagging converts them to "tagged devices" which lose user identity and break user-based SSH rules

Groups

Group Members Purpose
group:allisonflix , Jellyfin media access

Device Tags

Tag Devices Purpose
tag:homelab indri Server infrastructure
tag:nas sifaka Network-attached storage for backups
tag:blumeops indri, sifaka Resources managed by Pulumi IaC
tag:registry indri Container registry access
tag:k8s-api indri Kubernetes API server access

Access Matrix

Source Kiwix Forge PyPI Miniflux PostgreSQL NAS Grafana Loki
autogroup:admin Y Y Y Y Y Y Y Y
autogroup:member Y Y Y Y Y - - -
tag:homelab - - - - - Y - -

Notes:

  • Admins - full access to all services via autogroup:admin
  • Allison (member) - member services only, no Grafana/Loki/NAS

SSH Access

Source Destinations Auth
autogroup:member autogroup:self check
autogroup:admin tag:homelab check (12h)
autogroup:admin tag:nas check (12h)

Services

Services are accessible via two DNS domains:

  • *.ops.eblu.me - Caddy reverse proxy (reachable from k8s pods, docker containers, and tailnet)
  • *.tail8d86e.ts.net - Tailscale MagicDNS (tailnet clients only, not from k8s/docker)

Caddy Services (*.ops.eblu.me)

Caddy proxies to k8s services via their Tailscale endpoints (traffic stays local on indri). Both *.ops.eblu.me and *.tail8d86e.ts.net URLs work - use ops.eblu.me for access from pods/containers.

Service URL Description Management Log
Homepage https://go.ops.eblu.me Service dashboard / start page —
Forgejo https://forge.ops.eblu.me Git hosting (SSH: port 2222) forgejo
Registry https://registry.ops.eblu.me OCI container registry (Zot) zot
Sifaka NAS https://nas.ops.eblu.me Synology NAS dashboard —
Grafana https://grafana.ops.eblu.me Dashboards & observability (k8s) grafana
ArgoCD https://argocd.ops.eblu.me GitOps continuous delivery (k8s) argocd
Prometheus https://prometheus.ops.eblu.me Metrics collection (k8s) prometheus
Loki https://loki.ops.eblu.me Log aggregation (k8s) loki
Miniflux https://feed.ops.eblu.me RSS/Atom feed reader (k8s) miniflux
PyPI https://pypi.ops.eblu.me PyPI caching proxy (devpi, k8s) pypi
Kiwix https://kiwix.ops.eblu.me Offline Wikipedia & ZIM (k8s) argocd
Torrent https://torrent.ops.eblu.me BitTorrent daemon web UI (k8s) argocd
TeslaMate https://tesla.ops.eblu.me Tesla data logger (k8s) teslamate
Immich https://photos.ops.eblu.me Photo management (k8s Helm, CNPG) argocd
DJ https://dj.ops.eblu.me Music streaming server (Navidrome) navidrome
PostgreSQL pg.ops.eblu.me:5432 Database server (k8s CloudNativePG) postgresql

Tailscale-Only Services (*.tail8d86e.ts.net)

These services are only accessible via Tailscale (not from k8s pods/containers):

Service URL Description Management Log
Kubernetes https://k8s.tail8d86e.ts.net Minikube API (TCP passthrough) minikube
Jellyfin https://jellyfin.ops.eblu.me Media server (VideoToolbox HW) jellyfin

Supporting services (not directly user-facing):

Service Description Management Log
Alloy (indri) Metrics & logs collector (indri host) alloy
Alloy (k8s) Pod log collection & service probes alloy
Kube-state-metrics K8s resource metrics (pods, deployments) prometheus
Borgmatic Daily backups to Sifaka NAS (2:00 AM) borgmatic

Port Map (Indri)

Port Service Protocol Binding Notes
443 Caddy HTTPS 0.0.0.0 Reverse proxy for *.ops.eblu.me
2222 Caddy L4 TCP 0.0.0.0 SSH proxy → Forgejo (localhost:2200)
5432 Caddy L4 TCP 0.0.0.0 PostgreSQL proxy → k8s pg
2200 Forgejo SSH TCP localhost Built-in SSH server
3001 Forgejo HTTP localhost Web UI (proxied by Caddy)
5050 Zot HTTP localhost Registry API (proxied by Caddy)
8096 Jellyfin HTTP localhost Media server (proxied by Caddy)
44491 K8s API HTTPS 0.0.0.0 Minikube API server (via Tailscale k8s.*)

Service Management

Pulumi (Tailnet IaC)

Tailnet-wide configuration (ACLs, tags, DNS) is managed via Pulumi. See pulumi for details.

mise run tailnet-preview   # preview ACL changes
mise run tailnet-up        # apply ACL changes

Edit pulumi/policy.hujson to modify ACLs or add new tags.

Ansible

Services on Indri are managed via ansible. Playbooks live in the ansible/ directory of the blumeops repo:

mise run provision-indri        # runs ansible/playbooks/indri.yml
mise run indri-services-check   # checks health of all services

Run with --check --diff first to preview changes, or target specific services:

mise run provision-indri -- --check --diff          # dry run
mise run provision-indri -- --tags alloy            # only alloy
mise run provision-indri -- --tags zot,borgmatic    # multiple tags

Adding a New Service

Indri Services (via Caddy)

For services running directly on indri that need to be accessible from k8s pods:

  1. Host service locally on localhost (e.g., localhost:3000)
  2. Add service to ansible/roles/caddy/defaults/main.yml under caddy_services
  3. Run mise run provision-indri -- --tags caddy
  4. Add backup entry in borgmatic role if needed

DNS is handled by a wildcard record (*.ops.eblu.me → indri's Tailscale IP) managed via Pulumi in pulumi/gandi/.

Access via https://foo.ops.eblu.me.

K8s Services (via Tailscale Ingress)

For services running in minikube:

  1. Create Kubernetes manifests in argocd/manifests/<service>/
  2. Add ArgoCD Application in argocd/apps/<service>.yaml
  3. Add Tailscale Ingress annotation for *.tail8d86e.ts.net hostname
  4. Add Homepage annotations to the Ingress for dashboard discovery (see below)
  5. Add Caddy proxy entry in ansible/roles/caddy/defaults/main.yml
  6. Sync via ArgoCD: argocd app sync <service>

Access via https://foo.ops.eblu.me (preferred) or https://foo.tail8d86e.ts.net.

Note: K8s services using Tailscale Ingress are NOT accessible from other k8s pods or docker containers. Use Caddy (*.ops.eblu.me) if pod-to-service communication is needed.

Homepage annotations for automatic dashboard discovery:

annotations:
  gethomepage.dev/enabled: "true"
  gethomepage.dev/name: "My Service"
  gethomepage.dev/group: "Apps"
  gethomepage.dev/icon: "myservice.png"
  gethomepage.dev/description: "Short description"
  gethomepage.dev/href: "https://myservice.ops.eblu.me"
  gethomepage.dev/pod-selector: "app=myservice"

Icons use Dashboard Icons format (e.g., grafana.png, prometheus.png). The pod-selector annotation enables pod status badges on the dashboard.

Secrets Management

Kubernetes secrets are managed via external-secrets, which syncs from 1Password via 1Password Connect.

To add a secret to a k8s service:

  1. Ensure the 1Password item exists in the blumeops vault
  2. Create an ExternalSecret manifest in the service's directory
  3. Reference the onepassword-blumeops ClusterSecretStore
  4. Sync via ArgoCD

See external-secrets for detailed usage and bootstrap instructions.

Notes

Go DNS Resolution on macOS

Important lesson learned (2026-01-22): Go programs built with CGO_ENABLED=0 (pure Go) use a DNS resolver that reads /etc/resolv.conf directly and ignores macOS /etc/resolver/* files. This breaks Tailscale MagicDNS resolution.

Solution: Build Go programs with CGO_ENABLED=1 to use the macOS native resolver. This is why alloy is built from source rather than using the Homebrew bottle.

Remote Kubernetes Access (from Gilbert)

The minikube cluster on indri is accessible from gilbert via Tailscale service. Cluster was created with --apiserver-names=k8s.tail8d86e.ts.net,indri --listen-address=0.0.0.0. API server exposed at https://k8s.tail8d86e.ts.net via TCP passthrough (preserves mTLS).

Fish abbreviations (in ~/.config/fish/config.fish):

  • ki -> kubectl --context=minikube-indri
  • k9i -> k9s --context=minikube-indri
  • k9 -> k9s
# Quick access via abbreviations
ki get nodes
k9i

# Or explicitly set context
kubectl config use-context minikube-indri
kubectl get nodes

Credentials are stored in 1Password and fetched via exec credential plugin. See minikube for details.