Fix spider trap: disable SPA mode, remove index files, relax wiki-links (#290)
All checks were successful
Build Container / detect (push) Successful in 3s
Build Container (Nix) / detect (push) Successful in 1s
Build Container (Nix) / build (quartz) (push) Successful in 1s
Build Container / build (quartz) (push) Successful in 10s

## Summary

Fixes the Facebook crawler spider trap that's been generating infinite recursive URLs like `/how-to/tutorials/tutorials/how-to/explanation/...` for several days.

**Root cause:** Quartz SPA mode + nginx `try_files` fallback to `index.html` meant any fabricated URL returned the root HTML shell with HTTP 200. Crawlers followed relative links from those fake URLs, creating infinite recursion.

**Fix:**
- Disable Quartz SPA mode (`enableSPA: false`) — all pages are now fully static HTML
- Replace nginx SPA fallback with `=404` + Quartz's static `404.html`
- Remove `robots.txt` exclusions (no longer needed)

**Docs cleanup (Obsidian.nvim compat no longer needed):**
- Delete hand-curated category index files (`tutorials.md`, `reference.md`, `how-to.md`, `explanation.md`) — Quartz auto-generates folder pages
- Delete `postgresql-storage.md` (redirect stub) and `migrate-forgejo-from-brew.md` (stale history)
- Drop `docs-check-index` and `docs-check-filenames` prek hooks
- Rewrite `docs-check-links` to allow path-based wiki-links (`[[path/to/file]]`) and only error on true ambiguity
- Add `ai-docs` doc tree listing to replace index files for AI context
- Add natural cross-links from reference cards to fix orphan docs

## Deployment and Testing

- [ ] Merge and let the build pipeline run
- [ ] Verify docs.eblu.me serves pages correctly with full page loads
- [ ] Verify non-existent URLs return 404
- [ ] Monitor crawler traffic — should drop to near zero for fabricated URLs

Reviewed-on: #290
This commit is contained in:
Erich Blume 2026-03-09 11:59:43 -07:00
commit 4f0476a851
24 changed files with 110 additions and 666 deletions

View file

@ -0,0 +1 @@
Relax wiki-link constraints: allow path-based links for disambiguation, drop global filename uniqueness requirement, remove docs-check-filenames and docs-check-index hooks.

View file

@ -0,0 +1 @@
Disable Quartz SPA mode and remove robots.txt crawler exclusions to fix the Facebook crawler spider trap. Remove hand-curated category index files in favor of Quartz auto-generated folder pages.

View file

@ -1,25 +0,0 @@
---
title: Explanation
modified: 2026-02-10
last-reviewed: 2026-02-10
tags:
- explanation
---
# Explanation
Understanding-oriented content explaining the "why" behind BlumeOps design decisions.
## Philosophy
| Article | Description |
|---------|-------------|
| [[why-gitops]] | Why infrastructure-as-code and GitOps for a homelab |
## Design
| Article | Description |
|---------|-------------|
| [[architecture]] | How all the pieces fit together |
| [[federated-login]] | How SSO works across BlumeOps (Authentik) |
| [[security-model]] | Network security, secrets, and access control |

View file

@ -1,47 +0,0 @@
---
title: Migrate Forgejo from Brew to Source Build
status: active
modified: 2026-03-04
last-reviewed: 2026-03-05
tags:
- how-to
- forgejo
---
# Migrate Forgejo from Brew to Source Build
Transition Forgejo on indri from Homebrew to a source-built binary with LaunchAgent, matching the pattern used by [[zot]], [[caddy]], and [[alloy]].
## Motivation
Forgejo was force-upgraded from v13 to v14 by `brew upgrade`, breaking version control. A source build pins versions and aligns with the established native service pattern.
## Architecture Decisions
| Decision | Choice | Rationale |
|----------|--------|-----------|
| **Source remote** | Codeberg upstream | Avoids circular dependency (Forgejo hosting its own source) |
| **Secondary remote** | `forge.eblu.me/mirrors/forgejo` | Convenience and backup |
| **Version tracking** | `indri-deployment` branch on tag | Rebase to upgrade; explicit version pinning |
| **Build deps** | Go 1.24+, Node 20+ via mise | Consistent with other mise-managed tooling |
| **Process manager** | LaunchAgent plist | Matches zot, caddy, alloy |
| **Data location** | `~/forgejo` | Migrated from `/opt/homebrew/var/forgejo` |
| **Run user** | `erichblume` | LaunchAgent session user (SSH git user stays `forgejo`) |
## Key Steps
1. Clone from Codeberg, add forge mirror remote
2. Check out target tag, create `indri-deployment` branch
3. Build with `TAGS="bindata timedzdata sqlite sqlite_unlock_notify" mise x -- make build`
4. Stop brew service, copy data to `~/forgejo`, fix ownership
5. Run Ansible (`--tags forgejo`) to deploy updated role with LaunchAgent
6. Verify (API version, SSH clone, push, Actions runners, services-check)
7. `brew uninstall forgejo`
## Reference Patterns
- `ansible/roles/zot/` — primary pattern for source-built binary roles (tasks, defaults, handlers, plist template)
## Related
- [[forgejo]] — Service reference

View file

@ -1,100 +0,0 @@
---
title: How-To
modified: 2026-03-06
last-reviewed: 2026-03-06
tags:
- how-to
---
# How-To Guides
## Deployment
- [[deploy-k8s-service]]
- [[add-ansible-role]]
- [[create-release-artifact-workflow]]
- [[build-container-image]]
## Configuration
- [[update-tailscale-acls]]
- [[gandi-operations]]
- [[use-pypi-proxy]]
- [[expose-service-publicly]]
- [[manage-forgejo-mirrors]]
- [[update-documentation]]
- [[update-tooling-dependencies]]
## Knowledge Base
- [[review-documentation]]
- [[review-services]]
- [[agent-change-process]]
## Operations
- [[connect-to-postgres]]
- [[restart-indri]]
- [[manage-flyio-proxy]]
- [[restore-1password-backup]]
- [[troubleshooting]]
## Forgejo
- [[migrate-forgejo-from-brew]]
## Ringtail
- [[manage-lockfile]]
## Zot
- [[harden-zot-registry]]
- [[register-zot-oidc-client]]
- [[wire-ci-registry-auth]]
- [[enforce-tag-immutability]]
- [[adopt-commit-based-container-tags]]
- [[add-container-version-sync-check]]
- [[install-dagger-on-nix-runner]]
- [[pin-container-versions]]
- [[add-dagger-nix-build]]
- [[fix-ntfy-nix-version]]
## Authentik
- [[deploy-authentik]]
- [[build-authentik-container]]
- [[provision-authentik-database]]
- [[create-authentik-secrets]]
- [[migrate-grafana-to-authentik]]
## Authentik Source Build
- [[build-authentik-from-source]]
- [[mirror-authentik-build-deps]]
- [[authentik-api-client-generation]]
- [[authentik-python-backend-derivation]]
- [[authentik-web-ui-derivation]]
- [[authentik-go-server-derivation]]
## Grafana
- [[upgrade-grafana]]
- [[kustomize-grafana-deployment]]
- [[build-grafana-container]]
- [[build-grafana-sidecar]]
## Dagger
- [[upgrade-dagger]]
## JobSync
- [[deploy-jobsync]]
- [[build-jobsync-container]]
## Forgejo Runner
- [[upgrade-k8s-runner]]
- [[validate-workflows-against-v12]]
- [[review-runner-config-v12]]

View file

@ -39,8 +39,8 @@ The goal of BlumeOps is threefold:
## Sections
- [[tutorials|Tutorials]] - Learning-oriented guides for getting started
- [[reference|Reference]] - Technical specifications and service details
- [[how-to|How-to]] - Task-oriented instructions for common operations
- [[explanation|Explanation]] - Understanding the "why" behind BlumeOps
- [Tutorials](/tutorials/) - Learning-oriented guides for getting started
- [Reference](/reference/) - Technical specifications and service details
- [How-to](/how-to/) - Task-oriented instructions for common operations
- [Explanation](/explanation/) - Understanding the "why" behind BlumeOps
- [[CHANGELOG]] - Release history and changes

View file

@ -9,7 +9,7 @@ const config: QuartzConfig = {
configuration: {
pageTitle: "BlumeOps Docs",
pageTitleSuffix: "",
enableSPA: true,
enableSPA: false,
enablePopovers: true,
analytics: null,
locale: "en-US",

View file

@ -1,95 +0,0 @@
---
title: Reference
modified: 2026-03-04
tags:
- reference
---
# Reference
Technical specifications, inventories, and configuration details for BlumeOps infrastructure.
## Services
Individual service reference cards with URLs and configuration details.
| Service | Description | Location |
|---------|-------------|----------|
| [[alloy|Alloy]] | Observability collector (metrics & logs) | indri + k8s |
| [[argocd]] | GitOps continuous delivery | k8s |
| [[borgmatic]] | Backup system | indri |
| [[caddy]] | Reverse proxy & TLS termination | indri |
| [[1password]] | Secrets management | cloud + k8s |
| [[forgejo]] | Git forge & CI/CD | indri |
| [[frigate]] | Network video recorder | k8s (ringtail) |
| [[grafana]] | Dashboards & visualization | k8s |
| [[immich]] | Photo management | k8s |
| [[jellyfin]] | Media server | indri |
| [[jobsync]] | Job application tracker | k8s (ringtail) |
| [[kiwix]] | Offline Wikipedia & ZIM archives | k8s |
| [[loki]] | Log aggregation | k8s |
| [[tempo]] | Distributed tracing | k8s |
| [[miniflux]] | RSS feed reader | k8s |
| [[navidrome]] | Music streaming | k8s |
| [[ntfy]] | Push notifications | k8s (ringtail) |
| [[postgresql]] | Database cluster | k8s |
| [[prometheus]] | Metrics collection | k8s |
| [[teslamate]] | Tesla data logger | k8s |
| [[transmission]] | BitTorrent daemon | k8s |
| [[zot]] | Container registry | indri |
| [[devpi]] | PyPI caching proxy | k8s |
| [[cv]] | Resume / CV site | k8s |
| [[authentik]] | OIDC identity provider | k8s (ringtail) |
| [[docs]] | Documentation site (Quartz) | k8s |
| [[flyio-proxy]] | Public reverse proxy (Fly.io + Tailscale) | Fly.io |
| [[ollama]] | LLM inference server | k8s (ringtail) |
| [[automounter]] | SMB share automounter | indri |
## Infrastructure
Host inventory and network configuration.
- [[hosts|Hosts]] - Device inventory
- [[indri]] - Primary server
- [[ringtail]] - Service host & gaming PC
- [[gilbert]] - Development workstation
- [[tailscale]] - ACLs, groups, tags
- [[gandi]] - DNS hosting for `eblu.me`
- [[unifi]] - Home WiFi router (UniFi Express 7)
- [[routing|Routing]] - DNS domains, port mappings
- [[power]] - Battery-backed power chain
## Tools
Build, deployment, and IaC tool reference.
- [[mise-tasks]] - Operational task runner (all `mise run` tasks)
- [[dagger]] - CI/CD build engine (Python SDK)
- [[argocd-cli]] - ArgoCD CLI workflows
- [[ansible]] - Configuration management for indri
- [[pulumi]] - Infrastructure-as-Code (DNS, Tailscale ACLs)
## Kubernetes
Cluster configuration and application registry.
- [[cluster|Cluster]] - Minikube specs, storage, networking
- [[apps|Apps]] - ArgoCD application registry
- [[tailscale-operator]] - Tailscale ingress for k8s services
- [[external-secrets]] - Secrets management
## Storage
Network storage and backup configuration.
- [[sifaka|Sifaka]] - Synology NAS configuration
- [[postgresql-storage]] - Database cluster
- [[backups|Backups]] - Backup policy and schedule
## Operations
Operational concerns and their components.
- [[observability]] - Metrics, logs, dashboards
- [[backup]] - Data protection
- [[disaster-recovery]] - Recovery procedures (TBD)

View file

@ -60,7 +60,7 @@ Future clients: [[argocd]], [[miniflux]], [[zot]]
## Secrets
Injected via [[external-secrets]] from the "Authentik (blumeops)" 1Password item.
Injected via [[external-secrets]] from the "Authentik (blumeops)" 1Password item (see [[create-authentik-secrets]] for setup).
| 1Password Field | Purpose |
|-----------------|---------|
@ -79,4 +79,7 @@ Nix-built via `dockerTools.buildLayeredImage`. The entrypoint wrapper symlinks b
- [[federated-login]] - How authentication works across BlumeOps
- [[grafana]] - First OIDC client
- [[deploy-authentik]] - Deployment how-to
- [[migrate-grafana-to-authentik]] - Grafana SSO migration from Dex
- [[build-authentik-from-source]] - Nix-based container build
- [[mirror-authentik-build-deps]] - Supply chain mirrors for the build
- [[external-secrets]] - Secrets injection from 1Password

View file

@ -120,6 +120,10 @@ The UI shows `forge.eblu.me` for HTTPS clone URLs and `forge.ops.eblu.me` for SS
`mise run fly-shutoff` stops all public traffic immediately. forge.ops.eblu.me continues to work from the tailnet. See [[expose-service-publicly#Break-glass shutoff]].
## Mirrors
Forgejo hosts pull mirrors of external repositories (GitHub, etc.) for supply chain control. Mirrors live in the `mirrors/` org and sync on a configurable interval. See [[manage-forgejo-mirrors]] for operations.
## Related
- [[argocd]] - Uses Forgejo as git source

View file

@ -63,6 +63,7 @@ Optional annotation: `grafana_folder: "FolderName"`
- [[build-grafana-sidecar]] - Home-built sidecar container
- [[kustomize-grafana-deployment]] - Kustomize manifest structure
- [[authentik]] - OIDC identity provider for SSO
- [[migrate-grafana-to-authentik]] - How SSO was migrated from Dex to Authentik
- [[prometheus]] - Metrics datasource
- [[loki]] - Logs datasource
- [[tempo]] - Traces datasource

View file

@ -65,3 +65,5 @@ The `zot-ci` API key expires every **90 days**. To rotate:
- [[forgejo]] - Container build CI
- [[cluster|Cluster]] - Registry consumer
- [[authentik]] - OIDC identity provider
- [[harden-zot-registry]] - Security hardening guide
- [[install-dagger-on-nix-runner]] - Why Dagger can't run on the Nix builder

View file

@ -1,11 +0,0 @@
---
title: PostgreSQL Storage
modified: 2026-02-07
tags:
- storage
- database
---
# PostgreSQL Storage
See [[postgresql]] in Services.

View file

@ -17,11 +17,9 @@ Run `mise tasks --sort name` for the live list with descriptions.
| Task | Description |
|------|-------------|
| `ai-docs` | Prime AI context with key documentation |
| `docs-check-filenames` | Detect duplicate filenames in documentation |
| `ai-docs` | Prime AI context with key documentation and doc tree |
| `docs-check-frontmatter` | Check required frontmatter fields |
| `docs-check-index` | Check every doc is referenced in its category index |
| `docs-check-links` | Validate wiki-links point to existing filenames |
| `docs-check-links` | Validate wiki-links resolve correctly (supports path-based links) |
| `docs-mikado` | View active Mikado dependency chains (C2 changes) |
| `docs-review` | Review the most stale doc by `last-reviewed` date |
| `docs-review-stale` | Report docs by last-modified date |

View file

@ -91,7 +91,7 @@ BlumeOps operations are driven by mise tasks. Run `mise tasks` to list all avail
| Task | When to Use |
|------|-------------|
| `ai-docs` | At session start - review infrastructure documentation |
| `ai-docs` | At session start - review infrastructure documentation (see [[mise-tasks]]) |
| `docs-mikado` | View active Mikado dependency chains for C2 changes |
| `docs-mikado --resume` | Resume a C2 chain: detect branch, show state and next steps |
| `provision-indri` | Deploy changes to [[indri]]-hosted services via Ansible |
@ -104,9 +104,7 @@ BlumeOps operations are driven by mise tasks. Run `mise tasks` to list all avail
| `dns-up` | Apply DNS changes via Pulumi |
| `tailnet-preview` | Preview Tailscale ACL changes |
| `tailnet-up` | Apply Tailscale ACL changes via Pulumi |
| `docs-check-links` | Validate wiki-links in documentation (includes orphan detection) |
| `docs-check-index` | Check every doc is referenced in its category index |
| `docs-check-filenames` | Check for duplicate doc filenames |
| `docs-check-links` | Validate wiki-links resolve correctly (supports path-based links, orphan detection) |
| `docs-review-stale` | Report docs by last-modified date, highlight stale ones |
| `docs-review-tags` | Print frontmatter tag inventory across all docs |
| `docs-review` | Review the most stale doc by last-reviewed date |
@ -120,7 +118,7 @@ For ArgoCD operations, use the `argocd` CLI directly:
For AI agents building context:
- [[reference|Reference]] - Entry point for technical details
- [Reference](/reference/) - Entry point for technical details
- [[hosts|Host Inventory]] - What hardware exists
- [[apps|ArgoCD Apps]] - What's deployed in Kubernetes
- [[routing|Routing]] - How services are exposed

View file

@ -18,18 +18,18 @@ The docs follow the [Diataxis](https://diataxis.fr/) framework:
| Section | Purpose | When to Use |
|---------|---------|-------------|
| **[[tutorials|Tutorials]]** | Learning-oriented | "I'm new and want to understand" |
| **[[reference|Reference]]** | Information-oriented | "I need specific technical details" |
| **[[how-to|How-to]]** | Task-oriented | "I need to do X" |
| **[[explanation|Explanation]]** | Understanding-oriented | "I want to understand why" |
| **[Tutorials](/tutorials/)** | Learning-oriented | "I'm new and want to understand" |
| **[Reference](/reference/)** | Information-oriented | "I need specific technical details" |
| **[How-to](/how-to/)** | Task-oriented | "I need to do X" |
| **[Explanation](/explanation/)** | Understanding-oriented | "I want to understand why" |
## Quick Paths by Audience
### For Erich (Owner)
You probably want quick access to operational details:
- [[how-to]] guides for common operations (deploy, troubleshoot, update ACLs)
- [[reference]] has service URLs, commands, and config locations
- [How-to](/how-to/) guides for common operations (deploy, troubleshoot, update ACLs)
- [Reference](/reference/) has service URLs, commands, and config locations
- [[ai-assistance-guide]] explains how to work effectively with Claude
- Run `mise run ai-docs` to prime AI context with key documentation
@ -37,40 +37,41 @@ You probably want quick access to operational details:
Context for effective assistance:
- Read [[ai-assistance-guide]] for operational conventions
- [[reference]] has the technical specifics you'll need
- [Reference](/reference/) has the technical specifics you'll need
- The repo's `CLAUDE.md` has critical rules (especially the kubectl context requirement)
### For External Readers
Understanding what this is:
- [[explanation]] covers the "why" behind design decisions
- [[reference]] shows what's actually running
- [Explanation](/explanation/) covers the "why" behind design decisions
- [Reference](/reference/) shows what's actually running
- Browse service pages to see specific implementations
### For Contributors
Getting started with changes:
- [[contributing]] walks through the workflow
- [[how-to]] guides for specific tasks (deploy services, add roles)
- [[reference]] tells you where things live
- [How-to](/how-to/) guides for specific tasks (deploy services, add roles)
- [Reference](/reference/) tells you where things live
### For Replicators
Replicators are people who want to build their own similar homelab GitOps setup, using BlumeOps as inspiration.
- [[replicating-blumeops]] provides the overview, with linked tutorials that go deep on individual components
- [[explanation]] covers architecture and design rationale
- [Explanation](/explanation/) covers architecture and design rationale
- Reference pages show specific configuration choices
## Using Wiki Links
Documentation uses `[[wiki-links]]` for cross-references:
- `[[service-name]]` links to a reference page
- `[[service-name]]` links by filename stem (must be unambiguous)
- `[[path/to/file]]` links by path from docs root (for disambiguation)
- `[[page|Display Text]]` customizes the link text
When reading on the web (docs.eblu.me), these render as clickable links. The backlinks panel shows what references each page.
Prek hooks automatically validate that all wiki-links point to existing files and that link targets are unambiguous.
Prek hooks validate that all wiki-links resolve to existing files and flag ambiguous bare-name links.
## AI Context Priming
@ -80,10 +81,9 @@ The `ai-docs` mise task concatenates key documentation files for AI context:
mise run ai-docs
```
This outputs the AI assistance guide, reference index, how-to index, architecture overview, and tutorials index in plain text with file headers - providing Claude with essential context for BlumeOps operations.
This outputs key documentation files and a full tree listing of all docs, providing Claude with essential context for BlumeOps operations.
## Related
- [[tutorials]] - Parent index of all tutorials
- [[update-documentation]] - How to publish doc changes
- [[review-documentation]] - Periodic doc review process

View file

@ -136,5 +136,5 @@ Begin with [[tailscale-setup]] - networking is the foundation everything else bu
## Related
- [[reference]] - See BlumeOps' specific configurations
- [Reference](/reference/) - See BlumeOps' specific configurations
- [[contributing]] - Help improve BlumeOps instead

View file

@ -1,49 +0,0 @@
---
title: Tutorials
modified: 2026-02-07
tags:
- tutorials
---
# Tutorials
Learning-oriented guides for understanding and working with BlumeOps.
## Audience Guide
Each tutorial indicates which audiences it serves:
| Icon | Audience | Description |
|------|----------|-------------|
| **Owner** | Erich | Quick recall and operational refreshers |
| **AI** | Claude/AI agents | Context for AI-assisted operations |
| **Reader** | External readers | Understanding what BlumeOps is |
| **Contributor** | Operators/contributors | Helping with BlumeOps development |
| **Replicator** | Replicators | Building your own similar setup |
## Getting Started
| Tutorial | Audiences | Description |
|----------|-----------|-------------|
| [[exploring-the-docs]] | All | How to navigate and use this documentation |
| [[ai-assistance-guide]] | AI, Owner | Context for effective AI-assisted operations |
## Contributing
| Tutorial | Audiences | Description |
|----------|-----------|-------------|
| [[contributing]] | Contributor | Your first contribution to BlumeOps |
| [[adding-a-service]] | Contributor, Replicator | Deploy a new service via ArgoCD |
## Replication
For those building their own homelab GitOps setup.
| Tutorial | Audiences | Description |
|----------|-----------|-------------|
| [[replicating-blumeops]] | Replicator | Overview: building a similar environment |
| [[tailscale-setup|Tailscale Setup]] | Replicator | Setting up Tailscale networking |
| [[core-services|Core Services]] | Replicator | Forgejo and container registry |
| [[kubernetes-bootstrap|Kubernetes Bootstrap]] | Replicator | Bootstrapping a Kubernetes cluster |
| [[argocd-config|ArgoCD Config]] | Replicator | Configuring GitOps with ArgoCD |
| [[observability-stack|Observability Stack]] | Replicator | Metrics, logs, and dashboards |