eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions
blumeops/docs/reference/services/loki.md

56 lines
1.3 KiB
Markdown
Raw Normal View History

Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
---
Update all docs titles to human-readable (#117) ## Summary - Updated frontmatter `title:` in all 63 doc cards from slug-case to human-readable (e.g. `borgmatic` → `Borgmatic`, `ai-assistance-guide` → `AI Assistance Guide`) - Titles now closely match file stems so `[[wiki-links]]` render naturally without alternate anchor text - Corrected titles that diverged from stems (e.g. `host-inventory` → `Hosts`, `grafana-alloy` → `Alloy`, `argocd-applications` → `Apps`) - Deleted `title-test-alpha.md` and `title-test-beta.md` test cards and removed their reference index entry ## Deployment and Testing - [x] `docs-check-links` passes — all wiki-links valid - [x] `docs-check-index` passes - [x] `docs-check-filenames` passes - [ ] Verify titles render correctly on docs site after deploy Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/117
2026-02-07 21:44:57 -08:00
title: Loki
Fix frontmatter field name for Quartz date display (#158) ## Summary - Rename `date-modified` -> `modified` in all 80 docs and the `docs-check-frontmatter` task Quartz's `CreatedModifiedDate` plugin recognizes `modified`, `lastmod`, `updated`, and `last-modified` — but not `date-modified`. The wrong field name caused Quartz to ignore frontmatter dates entirely and fall through to filesystem timestamps (UTC inside Dagger), showing Feb 12 on pages built late on Feb 11 PST. ## Test plan - [x] `mise run docs-check-frontmatter` passes - [ ] Kick off docs release after merge — verify rendered dates match frontmatter values Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/158
2026-02-11 16:45:12 -08:00
modified: 2026-02-08
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
tags:
- service
- observability
---
# Loki
Log aggregation system for BlumeOps infrastructure.
## Quick Reference
| Property | Value |
|----------|-------|
| **URL** | https://loki.ops.eblu.me |
| **Tailscale URL** | https://loki.tail8d86e.ts.net |
| **Namespace** | `monitoring` |
| **Image** | `grafana/loki:3.4.2` |
| **Storage** | 50Gi PVC |
| **Retention** | 31 days |
## Architecture
- Single-node deployment with filesystem storage
- TSDB index with 24h period
Enforce unique doc filenames and simple wiki-links (#109) ## Summary - Rename section index files to match their titles (tutorials.md, reference.md, how-to.md, explanation.md) so all filenames are unique - Convert all ~47 path-based wiki-links to simple filename format across 15 files - Update doc-filenames task to no longer skip index.md files - Update doc-links task to reject path-based links containing '/' This ensures all wiki-links work correctly in obsidian.nvim by making links resolvable by filename alone. ## Testing - `mise run doc-filenames` - all unique - `mise run doc-links` - no broken or path-based links - `mise run doc-titles` - no duplicates Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/109
2026-02-04 17:21:34 -08:00
- Logs collected by [[alloy|Alloy]] and pushed via Loki API
Convert wiki-link titles to lowercase slugs (#92) ## Summary - Convert all frontmatter titles to lowercase-hyphenated format (e.g., `grafana-alloy` instead of `Grafana Alloy`) - Update all wiki-links to use the new slug format - Update `doc-titles` task to validate slug format (lowercase, hyphens only) Quartz appears to require titles without spaces for wiki-link resolution. ## Deployment and Testing - [x] Pre-commit hooks pass (`doc-titles` and `doc-links`) - [ ] Build docs v1.0.8 and deploy - [ ] Verify wiki-links resolve correctly (e.g., `[[grafana-alloy]]`) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/92
2026-02-03 16:06:35 -08:00
- Queried via [[grafana]]
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
## Log Sources
**From Indri (via Alloy):**
- forgejo, tailscale (brew services)
- alloy, borgmatic, zot, jellyfin (LaunchAgents)
**From Kubernetes (via Alloy DaemonSet):**
- All pods in all namespaces
Add Fly.io proxy observability via embedded Alloy (#123) ## Summary - Embed Grafana Alloy in the Fly.io proxy container to collect nginx JSON access logs (→ Loki) and derive request rate, latency histogram, cache status, and bandwidth metrics (→ Prometheus) - Add nginx `stub_status` endpoint for connection-level metrics (active/reading/writing/waiting) - Create two Grafana dashboards: **Docs APM** (per-service view filtered by `host="docs.eblu.me"`) and **Fly.io Proxy Health** (aggregate proxy health across all upstream services) ## Changed Files | File | Change | |------|--------| | `fly/nginx.conf` | Add JSON `log_format` + `access_log`, add `stub_status` endpoint | | `fly/Dockerfile` | COPY Alloy binary from `grafana/alloy:v1.5.1`, COPY `alloy.river` config | | `fly/alloy.river` | **New** — Alloy config: log tailing, metric extraction, remote_write | | `fly/start.sh` | Start Alloy after Tailscale, before nginx | | `argocd/manifests/grafana-config/dashboards/configmap-docs-apm.yaml` | **New** — Docs APM dashboard | | `argocd/manifests/grafana-config/dashboards/configmap-flyio.yaml` | **New** — Fly.io Proxy Health dashboard | | `argocd/manifests/grafana-config/kustomization.yaml` | Register new dashboard configmaps | | `docs/reference/services/flyio-proxy.md` | Document observability setup | ## Deployment and Testing - [ ] `mise run fly-deploy` — rebuild container with Alloy - [ ] `curl https://docs.eblu.me/` — generate traffic - [ ] `fly logs -a blumeops-proxy` — verify Alloy startup - [ ] Query Prometheus: `flyio_nginx_http_requests_total{instance="flyio-proxy"}` - [ ] Query Loki: `{instance="flyio-proxy", job="flyio-nginx"}` - [ ] `argocd app sync grafana-config` — deploy dashboards - [ ] Verify dashboards show data in Grafana - [ ] `mise run services-check` — no regressions Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/123
2026-02-08 10:05:38 -08:00
**From Fly.io proxy (via embedded Alloy):**
- nginx JSON access logs (`{instance="flyio-proxy", job="flyio-nginx"}`)
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
## Query Examples (LogQL)
```logql
{service="forgejo"} # All forgejo logs
{service="borgmatic", stream="stderr"} # Borgmatic errors
{host="indri"} |= "error" # All logs containing "error"
Add Fly.io proxy observability via embedded Alloy (#123) ## Summary - Embed Grafana Alloy in the Fly.io proxy container to collect nginx JSON access logs (→ Loki) and derive request rate, latency histogram, cache status, and bandwidth metrics (→ Prometheus) - Add nginx `stub_status` endpoint for connection-level metrics (active/reading/writing/waiting) - Create two Grafana dashboards: **Docs APM** (per-service view filtered by `host="docs.eblu.me"`) and **Fly.io Proxy Health** (aggregate proxy health across all upstream services) ## Changed Files | File | Change | |------|--------| | `fly/nginx.conf` | Add JSON `log_format` + `access_log`, add `stub_status` endpoint | | `fly/Dockerfile` | COPY Alloy binary from `grafana/alloy:v1.5.1`, COPY `alloy.river` config | | `fly/alloy.river` | **New** — Alloy config: log tailing, metric extraction, remote_write | | `fly/start.sh` | Start Alloy after Tailscale, before nginx | | `argocd/manifests/grafana-config/dashboards/configmap-docs-apm.yaml` | **New** — Docs APM dashboard | | `argocd/manifests/grafana-config/dashboards/configmap-flyio.yaml` | **New** — Fly.io Proxy Health dashboard | | `argocd/manifests/grafana-config/kustomization.yaml` | Register new dashboard configmaps | | `docs/reference/services/flyio-proxy.md` | Document observability setup | ## Deployment and Testing - [ ] `mise run fly-deploy` — rebuild container with Alloy - [ ] `curl https://docs.eblu.me/` — generate traffic - [ ] `fly logs -a blumeops-proxy` — verify Alloy startup - [ ] Query Prometheus: `flyio_nginx_http_requests_total{instance="flyio-proxy"}` - [ ] Query Loki: `{instance="flyio-proxy", job="flyio-nginx"}` - [ ] `argocd app sync grafana-config` — deploy dashboards - [ ] Verify dashboards show data in Grafana - [ ] `mise run services-check` — no regressions Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/123
2026-02-08 10:05:38 -08:00
{instance="flyio-proxy"} |= "docs.eblu.me" # Fly.io proxy access logs for docs
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
```
## Related
Enforce unique doc filenames and simple wiki-links (#109) ## Summary - Rename section index files to match their titles (tutorials.md, reference.md, how-to.md, explanation.md) so all filenames are unique - Convert all ~47 path-based wiki-links to simple filename format across 15 files - Update doc-filenames task to no longer skip index.md files - Update doc-links task to reject path-based links containing '/' This ensures all wiki-links work correctly in obsidian.nvim by making links resolvable by filename alone. ## Testing - `mise run doc-filenames` - all unique - `mise run doc-links` - no broken or path-based links - `mise run doc-titles` - no duplicates Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/109
2026-02-04 17:21:34 -08:00
- [[alloy|Alloy]] - Log collector
Convert wiki-link titles to lowercase slugs (#92) ## Summary - Convert all frontmatter titles to lowercase-hyphenated format (e.g., `grafana-alloy` instead of `Grafana Alloy`) - Update all wiki-links to use the new slug format - Update `doc-titles` task to validate slug format (lowercase, hyphens only) Quartz appears to require titles without spaces for wiki-link resolution. ## Deployment and Testing - [x] Pre-commit hooks pass (`doc-titles` and `doc-links`) - [ ] Build docs v1.0.8 and deploy - [ ] Verify wiki-links resolve correctly (e.g., `[[grafana-alloy]]`) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/92
2026-02-03 16:06:35 -08:00
- [[grafana]] - Log visualization
- [[prometheus]] - Metrics counterpart
Reference in a new issue Copy permalink