Fix spider trap: disable SPA mode, remove index files, relax wiki-links (#290)
All checks were successful
Build Container / detect (push) Successful in 3s
Build Container (Nix) / detect (push) Successful in 1s
Build Container (Nix) / build (quartz) (push) Successful in 1s
Build Container / build (quartz) (push) Successful in 10s

## Summary

Fixes the Facebook crawler spider trap that's been generating infinite recursive URLs like `/how-to/tutorials/tutorials/how-to/explanation/...` for several days.

**Root cause:** Quartz SPA mode + nginx `try_files` fallback to `index.html` meant any fabricated URL returned the root HTML shell with HTTP 200. Crawlers followed relative links from those fake URLs, creating infinite recursion.

**Fix:**
- Disable Quartz SPA mode (`enableSPA: false`) — all pages are now fully static HTML
- Replace nginx SPA fallback with `=404` + Quartz's static `404.html`
- Remove `robots.txt` exclusions (no longer needed)

**Docs cleanup (Obsidian.nvim compat no longer needed):**
- Delete hand-curated category index files (`tutorials.md`, `reference.md`, `how-to.md`, `explanation.md`) — Quartz auto-generates folder pages
- Delete `postgresql-storage.md` (redirect stub) and `migrate-forgejo-from-brew.md` (stale history)
- Drop `docs-check-index` and `docs-check-filenames` prek hooks
- Rewrite `docs-check-links` to allow path-based wiki-links (`[[path/to/file]]`) and only error on true ambiguity
- Add `ai-docs` doc tree listing to replace index files for AI context
- Add natural cross-links from reference cards to fix orphan docs

## Deployment and Testing

- [ ] Merge and let the build pipeline run
- [ ] Verify docs.eblu.me serves pages correctly with full page loads
- [ ] Verify non-existent URLs return 404
- [ ] Monitor crawler traffic — should drop to near zero for fabricated URLs

Reviewed-on: #290
This commit is contained in:
Erich Blume 2026-03-09 11:59:43 -07:00
commit 4f0476a851
24 changed files with 110 additions and 666 deletions

View file

@ -60,7 +60,7 @@ Future clients: [[argocd]], [[miniflux]], [[zot]]
## Secrets
Injected via [[external-secrets]] from the "Authentik (blumeops)" 1Password item.
Injected via [[external-secrets]] from the "Authentik (blumeops)" 1Password item (see [[create-authentik-secrets]] for setup).
| 1Password Field | Purpose |
|-----------------|---------|
@ -79,4 +79,7 @@ Nix-built via `dockerTools.buildLayeredImage`. The entrypoint wrapper symlinks b
- [[federated-login]] - How authentication works across BlumeOps
- [[grafana]] - First OIDC client
- [[deploy-authentik]] - Deployment how-to
- [[migrate-grafana-to-authentik]] - Grafana SSO migration from Dex
- [[build-authentik-from-source]] - Nix-based container build
- [[mirror-authentik-build-deps]] - Supply chain mirrors for the build
- [[external-secrets]] - Secrets injection from 1Password

View file

@ -120,6 +120,10 @@ The UI shows `forge.eblu.me` for HTTPS clone URLs and `forge.ops.eblu.me` for SS
`mise run fly-shutoff` stops all public traffic immediately. forge.ops.eblu.me continues to work from the tailnet. See [[expose-service-publicly#Break-glass shutoff]].
## Mirrors
Forgejo hosts pull mirrors of external repositories (GitHub, etc.) for supply chain control. Mirrors live in the `mirrors/` org and sync on a configurable interval. See [[manage-forgejo-mirrors]] for operations.
## Related
- [[argocd]] - Uses Forgejo as git source

View file

@ -63,6 +63,7 @@ Optional annotation: `grafana_folder: "FolderName"`
- [[build-grafana-sidecar]] - Home-built sidecar container
- [[kustomize-grafana-deployment]] - Kustomize manifest structure
- [[authentik]] - OIDC identity provider for SSO
- [[migrate-grafana-to-authentik]] - How SSO was migrated from Dex to Authentik
- [[prometheus]] - Metrics datasource
- [[loki]] - Logs datasource
- [[tempo]] - Traces datasource

View file

@ -65,3 +65,5 @@ The `zot-ci` API key expires every **90 days**. To rotate:
- [[forgejo]] - Container build CI
- [[cluster|Cluster]] - Registry consumer
- [[authentik]] - OIDC identity provider
- [[harden-zot-registry]] - Security hardening guide
- [[install-dagger-on-nix-runner]] - Why Dagger can't run on the Nix builder