eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions
blumeops/docs/reference/services/grafana.md

67 lines
2.2 KiB
Markdown
Raw Normal View History

Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
---
Update all docs titles to human-readable (#117) ## Summary - Updated frontmatter `title:` in all 63 doc cards from slug-case to human-readable (e.g. `borgmatic` → `Borgmatic`, `ai-assistance-guide` → `AI Assistance Guide`) - Titles now closely match file stems so `[[wiki-links]]` render naturally without alternate anchor text - Corrected titles that diverged from stems (e.g. `host-inventory` → `Hosts`, `grafana-alloy` → `Alloy`, `argocd-applications` → `Apps`) - Deleted `title-test-alpha.md` and `title-test-beta.md` test cards and removed their reference index entry ## Deployment and Testing - [x] `docs-check-links` passes — all wiki-links valid - [x] `docs-check-index` passes - [x] `docs-check-filenames` passes - [ ] Verify titles render correctly on docs site after deploy Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/117
2026-02-07 21:44:57 -08:00
title: Grafana
Review build-grafana-container docs: stamp reviewed, fix cross-links Also fix stale grafana.md reference card (Helm → Kustomize). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 07:28:06 -08:00
modified: 2026-02-28
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
tags:
- service
- observability
---
# Grafana
Dashboards and visualization for BlumeOps observability.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://grafana.ops.eblu.me |
| **Tailscale URL** | https://grafana.tail8d86e.ts.net |
| **Namespace** | `monitoring` |
Review build-grafana-container docs: stamp reviewed, fix cross-links Also fix stale grafana.md reference card (Helm → Kustomize). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 07:28:06 -08:00
| **Deployment** | Kustomize (`argocd/manifests/grafana/`) |
| **Image** | `registry.ops.eblu.me/blumeops/grafana` |
Home-build grafana-sidecar container (#281) ## Summary - Home-build the k8s-sidecar container (`grafana-sidecar`) from forge mirror, replacing upstream `quay.io/kiwigrid/k8s-sidecar:1.28.0` - Pinned to v1.28.0 — v2.x deferred due to 135% memory regression and readOnlyRootFilesystem crashloop - Adds Dockerfile, service-versions entry, docs, and changelog fragment - Manifest switch to home-built image pending container build ## Deployment and Testing - [ ] `mise run container-build-and-release grafana-sidecar` - [ ] Update kustomization.yaml with built image tag - [ ] `argocd app set grafana --revision feature/grafana-sidecar && argocd app sync grafana` - [ ] Verify sidecar logs and dashboards at https://grafana.ops.eblu.me - [ ] Post-merge: `argocd app set grafana --revision main && argocd app sync grafana` Reviewed-on: https://forge.eblu.me/eblume/blumeops/pulls/281
2026-03-03 13:48:24 -08:00
| **Sidecar Image** | `registry.ops.eblu.me/blumeops/grafana-sidecar` |
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
Document Dex OIDC and add services-check integration (#223) ## Summary - Create Dex reference card (`docs/reference/services/dex.md`) with quick reference, architecture, identity source, storage, OIDC clients, secrets, and endpoints - Write federated login explanation article (`docs/explanation/federated-login.md`) covering the Dex + Forgejo two-layer auth model, login flow, and break-glass access - Add Dex to `services-check` (HTTP health endpoint + k3s pod check) - Update Grafana docs with new Authentication section documenting SSO via Dex - Update Forgejo docs with OAuth2 Provider section documenting its role as upstream identity source - Add Dex to ringtail workloads table and reference service index - Move `adopt-oidc-provider` plan to `completed/` with final design reflecting actual implementation ## Test plan - [ ] `mise run services-check` passes (includes new Dex checks) - [ ] `docs-check-links` passes (all wiki-links resolve) - [ ] `docs-check-index` passes (new docs are indexed) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/223
2026-02-19 20:44:23 -08:00
## Authentication
Grafana supports two login methods:
Deploy Authentik identity provider (C2 Mikado) (#227) ## Summary C2 Mikado chain for deploying Authentik as the SSO identity provider, replacing Dex. This PR will evolve over multiple sessions. Each iteration adds documentation (prerequisite cards) and eventually code as leaf nodes are resolved. ## Current Mikado State - **Goal:** `deploy-authentik` (active) - **Leaf prerequisites:** - `build-authentik-container` — Build Nix container image - `provision-authentik-database` — Create PostgreSQL database on CNPG cluster - `create-authentik-secrets` — Create 1Password item with credentials ## Process refinements - Updated agent-change-process with lessons from first attempt: reset code before committing cards, open PRs early ## Test plan - [ ] `mise run docs-mikado` shows correct dependency chain - [ ] Leaf nodes can be worked independently - [ ] Container builds on ringtail - [ ] Authentik starts and reaches healthy state - [ ] Forgejo OAuth2 connector works Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/227
2026-02-20 12:55:59 -08:00
- **SSO via [[authentik]]** — OIDC login through Authentik (`auth.generic_oauth`). Users click "Sign in with Authentik", authenticate at Authentik, and are redirected back as Admin.
- **Local admin** — break-glass login using the password from 1Password ("Grafana (blumeops)"). Always available if Authentik is down.
Document Dex OIDC and add services-check integration (#223) ## Summary - Create Dex reference card (`docs/reference/services/dex.md`) with quick reference, architecture, identity source, storage, OIDC clients, secrets, and endpoints - Write federated login explanation article (`docs/explanation/federated-login.md`) covering the Dex + Forgejo two-layer auth model, login flow, and break-glass access - Add Dex to `services-check` (HTTP health endpoint + k3s pod check) - Update Grafana docs with new Authentication section documenting SSO via Dex - Update Forgejo docs with OAuth2 Provider section documenting its role as upstream identity source - Add Dex to ringtail workloads table and reference service index - Move `adopt-oidc-provider` plan to `completed/` with final design reflecting actual implementation ## Test plan - [ ] `mise run services-check` passes (includes new Dex checks) - [ ] `docs-check-links` passes (all wiki-links resolve) - [ ] `docs-check-index` passes (new docs are indexed) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/223
2026-02-19 20:44:23 -08:00
Deploy Authentik identity provider (C2 Mikado) (#227) ## Summary C2 Mikado chain for deploying Authentik as the SSO identity provider, replacing Dex. This PR will evolve over multiple sessions. Each iteration adds documentation (prerequisite cards) and eventually code as leaf nodes are resolved. ## Current Mikado State - **Goal:** `deploy-authentik` (active) - **Leaf prerequisites:** - `build-authentik-container` — Build Nix container image - `provision-authentik-database` — Create PostgreSQL database on CNPG cluster - `create-authentik-secrets` — Create 1Password item with credentials ## Process refinements - Updated agent-change-process with lessons from first attempt: reset code before committing cards, open PRs early ## Test plan - [ ] `mise run docs-mikado` shows correct dependency chain - [ ] Leaf nodes can be worked independently - [ ] Container builds on ringtail - [ ] Authentik starts and reaches healthy state - [ ] Forgejo OAuth2 connector works Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/227
2026-02-20 12:55:59 -08:00
The OIDC client secret is injected via [[external-secrets]] (`grafana-authentik-oauth` secret in monitoring namespace).
Document Dex OIDC and add services-check integration (#223) ## Summary - Create Dex reference card (`docs/reference/services/dex.md`) with quick reference, architecture, identity source, storage, OIDC clients, secrets, and endpoints - Write federated login explanation article (`docs/explanation/federated-login.md`) covering the Dex + Forgejo two-layer auth model, login flow, and break-glass access - Add Dex to `services-check` (HTTP health endpoint + k3s pod check) - Update Grafana docs with new Authentication section documenting SSO via Dex - Update Forgejo docs with OAuth2 Provider section documenting its role as upstream identity source - Add Dex to ringtail workloads table and reference service index - Move `adopt-oidc-provider` plan to `completed/` with final design reflecting actual implementation ## Test plan - [ ] `mise run services-check` passes (includes new Dex checks) - [ ] `docs-check-links` passes (all wiki-links resolve) - [ ] `docs-check-index` passes (new docs are indexed) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/223
2026-02-19 20:44:23 -08:00
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
## Datasources
| Name | Type | Target |
|------|------|--------|
| Prometheus | prometheus | `prometheus.monitoring.svc.cluster.local:9090` |
| Loki | loki | `loki.monitoring.svc.cluster.local:3100` |
| TeslaMate | postgres | `blumeops-pg-rw.databases.svc.cluster.local:5432` |
## Dashboard Provisioning
Dashboards are ConfigMaps with label `grafana_dashboard: "1"`.
Location: `argocd/manifests/grafana-config/dashboards/`
Optional annotation: `grafana_folder: "FolderName"`
## Key Dashboards
- macOS System - Host metrics for indri
- Minikube - Kubernetes cluster overview
- Borgmatic Backups - Backup status and trends
- Services Health - HTTP probe results
Add Fly.io proxy observability via embedded Alloy (#123) ## Summary - Embed Grafana Alloy in the Fly.io proxy container to collect nginx JSON access logs (→ Loki) and derive request rate, latency histogram, cache status, and bandwidth metrics (→ Prometheus) - Add nginx `stub_status` endpoint for connection-level metrics (active/reading/writing/waiting) - Create two Grafana dashboards: **Docs APM** (per-service view filtered by `host="docs.eblu.me"`) and **Fly.io Proxy Health** (aggregate proxy health across all upstream services) ## Changed Files | File | Change | |------|--------| | `fly/nginx.conf` | Add JSON `log_format` + `access_log`, add `stub_status` endpoint | | `fly/Dockerfile` | COPY Alloy binary from `grafana/alloy:v1.5.1`, COPY `alloy.river` config | | `fly/alloy.river` | **New** — Alloy config: log tailing, metric extraction, remote_write | | `fly/start.sh` | Start Alloy after Tailscale, before nginx | | `argocd/manifests/grafana-config/dashboards/configmap-docs-apm.yaml` | **New** — Docs APM dashboard | | `argocd/manifests/grafana-config/dashboards/configmap-flyio.yaml` | **New** — Fly.io Proxy Health dashboard | | `argocd/manifests/grafana-config/kustomization.yaml` | Register new dashboard configmaps | | `docs/reference/services/flyio-proxy.md` | Document observability setup | ## Deployment and Testing - [ ] `mise run fly-deploy` — rebuild container with Alloy - [ ] `curl https://docs.eblu.me/` — generate traffic - [ ] `fly logs -a blumeops-proxy` — verify Alloy startup - [ ] Query Prometheus: `flyio_nginx_http_requests_total{instance="flyio-proxy"}` - [ ] Query Loki: `{instance="flyio-proxy", job="flyio-nginx"}` - [ ] `argocd app sync grafana-config` — deploy dashboards - [ ] Verify dashboards show data in Grafana - [ ] `mise run services-check` — no regressions Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/123
2026-02-08 10:05:38 -08:00
- Docs APM - Request rate, latency, cache for docs.eblu.me
- Fly.io Proxy Health - Aggregate proxy health across all upstream services
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
- TeslaMate (18 dashboards) - Vehicle data
## Related
Review build-grafana-container docs: stamp reviewed, fix cross-links Also fix stale grafana.md reference card (Helm → Kustomize). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 07:28:06 -08:00
- [[build-grafana-container]] - Home-built container image
Home-build grafana-sidecar container (#281) ## Summary - Home-build the k8s-sidecar container (`grafana-sidecar`) from forge mirror, replacing upstream `quay.io/kiwigrid/k8s-sidecar:1.28.0` - Pinned to v1.28.0 — v2.x deferred due to 135% memory regression and readOnlyRootFilesystem crashloop - Adds Dockerfile, service-versions entry, docs, and changelog fragment - Manifest switch to home-built image pending container build ## Deployment and Testing - [ ] `mise run container-build-and-release grafana-sidecar` - [ ] Update kustomization.yaml with built image tag - [ ] `argocd app set grafana --revision feature/grafana-sidecar && argocd app sync grafana` - [ ] Verify sidecar logs and dashboards at https://grafana.ops.eblu.me - [ ] Post-merge: `argocd app set grafana --revision main && argocd app sync grafana` Reviewed-on: https://forge.eblu.me/eblume/blumeops/pulls/281
2026-03-03 13:48:24 -08:00
- [[build-grafana-sidecar]] - Home-built sidecar container
Review build-grafana-container docs: stamp reviewed, fix cross-links Also fix stale grafana.md reference card (Helm → Kustomize). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 07:28:06 -08:00
- [[kustomize-grafana-deployment]] - Kustomize manifest structure
Deploy Authentik identity provider (C2 Mikado) (#227) ## Summary C2 Mikado chain for deploying Authentik as the SSO identity provider, replacing Dex. This PR will evolve over multiple sessions. Each iteration adds documentation (prerequisite cards) and eventually code as leaf nodes are resolved. ## Current Mikado State - **Goal:** `deploy-authentik` (active) - **Leaf prerequisites:** - `build-authentik-container` — Build Nix container image - `provision-authentik-database` — Create PostgreSQL database on CNPG cluster - `create-authentik-secrets` — Create 1Password item with credentials ## Process refinements - Updated agent-change-process with lessons from first attempt: reset code before committing cards, open PRs early ## Test plan - [ ] `mise run docs-mikado` shows correct dependency chain - [ ] Leaf nodes can be worked independently - [ ] Container builds on ringtail - [ ] Authentik starts and reaches healthy state - [ ] Forgejo OAuth2 connector works Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/227
2026-02-20 12:55:59 -08:00
- [[authentik]] - OIDC identity provider for SSO
Convert wiki-link titles to lowercase slugs (#92) ## Summary - Convert all frontmatter titles to lowercase-hyphenated format (e.g., `grafana-alloy` instead of `Grafana Alloy`) - Update all wiki-links to use the new slug format - Update `doc-titles` task to validate slug format (lowercase, hyphens only) Quartz appears to require titles without spaces for wiki-link resolution. ## Deployment and Testing - [x] Pre-commit hooks pass (`doc-titles` and `doc-links`) - [ ] Build docs v1.0.8 and deploy - [ ] Verify wiki-links resolve correctly (e.g., `[[grafana-alloy]]`) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/92
2026-02-03 16:06:35 -08:00
- [[prometheus]] - Metrics datasource
- [[loki]] - Logs datasource
Enforce unique doc filenames and simple wiki-links (#109) ## Summary - Rename section index files to match their titles (tutorials.md, reference.md, how-to.md, explanation.md) so all filenames are unique - Convert all ~47 path-based wiki-links to simple filename format across 15 files - Update doc-filenames task to no longer skip index.md files - Update doc-links task to reject path-based links containing '/' This ensures all wiki-links work correctly in obsidian.nvim by making links resolvable by filename alone. ## Testing - `mise run doc-filenames` - all unique - `mise run doc-links` - no broken or path-based links - `mise run doc-titles` - no duplicates Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/109
2026-02-04 17:21:34 -08:00
- [[alloy|Alloy]] - Data collector
Reference in a new issue Copy permalink