blumeops/mise-tasks
Erich Blume 4f0476a851
All checks were successful
Build Container / detect (push) Successful in 3s
Build Container (Nix) / detect (push) Successful in 1s
Build Container (Nix) / build (quartz) (push) Successful in 1s
Build Container / build (quartz) (push) Successful in 10s
Fix spider trap: disable SPA mode, remove index files, relax wiki-links (#290)
## Summary

Fixes the Facebook crawler spider trap that's been generating infinite recursive URLs like `/how-to/tutorials/tutorials/how-to/explanation/...` for several days.

**Root cause:** Quartz SPA mode + nginx `try_files` fallback to `index.html` meant any fabricated URL returned the root HTML shell with HTTP 200. Crawlers followed relative links from those fake URLs, creating infinite recursion.

**Fix:**
- Disable Quartz SPA mode (`enableSPA: false`) — all pages are now fully static HTML
- Replace nginx SPA fallback with `=404` + Quartz's static `404.html`
- Remove `robots.txt` exclusions (no longer needed)

**Docs cleanup (Obsidian.nvim compat no longer needed):**
- Delete hand-curated category index files (`tutorials.md`, `reference.md`, `how-to.md`, `explanation.md`) — Quartz auto-generates folder pages
- Delete `postgresql-storage.md` (redirect stub) and `migrate-forgejo-from-brew.md` (stale history)
- Drop `docs-check-index` and `docs-check-filenames` prek hooks
- Rewrite `docs-check-links` to allow path-based wiki-links (`[[path/to/file]]`) and only error on true ambiguity
- Add `ai-docs` doc tree listing to replace index files for AI context
- Add natural cross-links from reference cards to fix orphan docs

## Deployment and Testing

- [ ] Merge and let the build pipeline run
- [ ] Verify docs.eblu.me serves pages correctly with full page loads
- [ ] Verify non-existent URLs return 404
- [ ] Monitor crawler traffic — should drop to near zero for fabricated URLs

Reviewed-on: #290
2026-03-09 11:59:43 -07:00
..
ai-docs Fix spider trap: disable SPA mode, remove index files, relax wiki-links (#290) 2026-03-09 11:59:43 -07:00
blumeops-tasks Update tooling dependencies (Feb 2026 cycle) (#254) 2026-02-23 13:08:41 -08:00
branch-cleanup Expose Forgejo publicly at forge.eblu.me (#278) 2026-03-03 08:40:41 -08:00
changelog-check Add pre-commit check for changelog fragment placement 2026-03-03 10:49:01 -08:00
container-build-and-release Expose Forgejo publicly at forge.eblu.me (#278) 2026-03-03 08:40:41 -08:00
container-list Document container tag provenance and enhance container-list (#263) 2026-02-24 09:54:58 -08:00
container-version-check Harden zot registry, pt 1 (#231) 2026-02-20 22:50:01 -08:00
dns-preview Add plan and reference card for UniFi Express 7 Pulumi stack (#145) 2026-02-10 15:36:13 -08:00
dns-up Add plan and reference card for UniFi Express 7 Pulumi stack (#145) 2026-02-10 15:36:13 -08:00
docs-check-frontmatter Add agent change process (C0/C1/C2) and docs-mikado tool (#225) 2026-02-20 08:15:20 -08:00
docs-check-links Fix spider trap: disable SPA mode, remove index files, relax wiki-links (#290) 2026-03-09 11:59:43 -07:00
docs-mikado Expose Forgejo publicly at forge.eblu.me (#278) 2026-03-03 08:40:41 -08:00
docs-review docs-review: print file path instead of content for LLM usage 2026-02-26 07:24:37 -08:00
docs-review-stale Update tooling dependencies (Feb 2026 cycle) (#254) 2026-02-23 13:08:41 -08:00
docs-review-tags Rename doc-* mise tasks to docs-check-* / docs-review-* (#113) 2026-02-06 07:08:46 -08:00
ensure-k3s-ringtail-kubectl-config Add k3s, 1Password Connect, and systemd nix-container-builder to ringtail (#209) 2026-02-18 21:15:30 -08:00
ensure-minikube-indri-kubectl-config P5.1: Migrate minikube from podman to QEMU2 driver (#38) 2026-01-21 16:03:37 -08:00
fly-deploy Add Fly.io public reverse proxy for docs.eblu.me (#120) 2026-02-08 02:36:19 -08:00
fly-setup Expose Forgejo publicly at forge.eblu.me (#278) 2026-03-03 08:40:41 -08:00
fly-shutoff Add Fly.io public reverse proxy for docs.eblu.me (#120) 2026-02-08 02:36:19 -08:00
frigate-export-model Fix dagger call hanging in mise tasks on interactive terminals (#256) 2026-02-23 14:15:58 -08:00
mikado-branch-invariant-check Fix mikado invariant check rejecting close without impl 2026-03-07 20:41:03 -08:00
mirror-create Expose Forgejo publicly at forge.eblu.me (#278) 2026-03-03 08:40:41 -08:00
mirror-update-pats Fix mirror-update-pats corrupting all GitHub mirror URLs 2026-03-03 11:46:41 -08:00
op-backup Add how-to guide for restoring 1Password backup from borgmatic (#141) 2026-02-10 10:55:00 -08:00
pr-comments Expose Forgejo publicly at forge.eblu.me (#278) 2026-03-03 08:40:41 -08:00
provision-indri Set MISE_TASK_OUTPUT=interleave in provision-indri 2026-01-14 14:15:11 -08:00
provision-ringtail Fix dagger call hanging in mise tasks on interactive terminals (#256) 2026-02-23 14:15:58 -08:00
provision-sifaka Operations and observability for sifaka NAS (#135) 2026-02-09 17:44:05 -08:00
runner-logs Expose Forgejo publicly at forge.eblu.me (#278) 2026-03-03 08:40:41 -08:00
service-review Update tooling dependencies (Feb 2026 cycle) (#254) 2026-02-23 13:08:41 -08:00
services-check Add Forge (public) to services-check 2026-03-03 08:48:15 -08:00
tailnet-preview Add plan and reference card for UniFi Express 7 Pulumi stack (#145) 2026-02-10 15:36:13 -08:00
tailnet-up Add plan and reference card for UniFi Express 7 Pulumi stack (#145) 2026-02-10 15:36:13 -08:00
validate-workflows Fix dagger call hanging in mise tasks on interactive terminals (#256) 2026-02-23 14:15:58 -08:00