blumeops/docs/zk/1768506761-XGYX.md
Erich Blume 278f231563 Switch to title-based wiki-links (Quartz resolves via frontmatter title and aliases)
- Remove aliases from all zk cards to prevent them from capturing wiki-links
- Convert all wiki-links from [[filename|Title]] to [[Title]] format
- Replace doc-filenames task with doc-titles for duplicate detection
- Update pre-commit hook to use doc-titles

Wiki-links now resolve to reference docs by their frontmatter title,
which is more readable and maintainable than filename-based links.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 15:48:15 -08:00

2.3 KiB

id tags
1768506761-XGYX
blumeops

Loki Management Log

Loki is a log aggregation system running in Kubernetes (minikube on indri), providing log storage and querying for the 1767747119-YCPO infrastructure.

Service Details

Architecture

  • Single-node deployment with filesystem storage
  • TSDB index with 24h period
  • Logs collected by alloy and pushed via Loki API
  • Queried via Grafana using the Loki datasource

Useful Commands

# View logs
kubectl --context=minikube-indri -n monitoring logs -f loki-0

# Check if Loki is ready
curl -s https://loki.tail8d86e.ts.net/ready

# Sync from ArgoCD
argocd app sync loki

Grafana Integration

Loki is configured as a datasource in Grafana. To explore logs:

  1. Go to https://grafana.tail8d86e.ts.net/explore
  2. Select "Loki" datasource
  3. Use LogQL queries:
    • {service="forgejo"} - all forgejo logs
    • {service="borgmatic", stream="stderr"} - borgmatic errors
    • {host="indri"} |= "error" - all logs containing "error"

ArgoCD Management

Loki is deployed via ArgoCD from argocd/manifests/loki/:

  • statefulset.yaml - StatefulSet with 50Gi PVC
  • configmap.yaml - Loki configuration
  • service.yaml - ClusterIP service
  • ingress-tailscale.yaml - Tailscale Ingress

Log

Thu Jan 23 2026

  • Suppressed noisy v1 Endpoints is deprecated warning from minikube storage-provisioner (upstream issue)
  • Added JSON field extraction for zot compatibility (message vs msg)
  • Removed logfmt parsing stage - stage.match selectors don't prevent Alloy from logging internal decode errors, and most structured logs use JSON anyway
  • Fixed devpi dashboard JSON escaping

Wed Jan 22 2026

  • Migrated to Kubernetes - moved from Homebrew on indri to k8s StatefulSet
  • Exposed via Tailscale Ingress at loki.tail8d86e.ts.net
  • Alloy updated to push logs to k8s endpoint
  • Retired ansible loki role from indri

Wed Jan 15 2026

  • Initial setup with single-node filesystem storage
  • Configured 31-day retention with compactor
  • Integrated with Grafana as datasource
  • Logs collected via Alloy from all services