Fix spider trap: disable SPA mode, remove index files, relax wiki-links (#290)
## Summary Fixes the Facebook crawler spider trap that's been generating infinite recursive URLs like `/how-to/tutorials/tutorials/how-to/explanation/...` for several days. **Root cause:** Quartz SPA mode + nginx `try_files` fallback to `index.html` meant any fabricated URL returned the root HTML shell with HTTP 200. Crawlers followed relative links from those fake URLs, creating infinite recursion. **Fix:** - Disable Quartz SPA mode (`enableSPA: false`) — all pages are now fully static HTML - Replace nginx SPA fallback with `=404` + Quartz's static `404.html` - Remove `robots.txt` exclusions (no longer needed) **Docs cleanup (Obsidian.nvim compat no longer needed):** - Delete hand-curated category index files (`tutorials.md`, `reference.md`, `how-to.md`, `explanation.md`) — Quartz auto-generates folder pages - Delete `postgresql-storage.md` (redirect stub) and `migrate-forgejo-from-brew.md` (stale history) - Drop `docs-check-index` and `docs-check-filenames` prek hooks - Rewrite `docs-check-links` to allow path-based wiki-links (`[[path/to/file]]`) and only error on true ambiguity - Add `ai-docs` doc tree listing to replace index files for AI context - Add natural cross-links from reference cards to fix orphan docs ## Deployment and Testing - [ ] Merge and let the build pipeline run - [ ] Verify docs.eblu.me serves pages correctly with full page loads - [ ] Verify non-existent URLs return 404 - [ ] Monitor crawler traffic — should drop to near zero for fabricated URLs Reviewed-on: #290
This commit is contained in:
parent
953640d2b7
commit
4f0476a851
24 changed files with 110 additions and 666 deletions
|
|
@ -17,11 +17,9 @@ Run `mise tasks --sort name` for the live list with descriptions.
|
|||
|
||||
| Task | Description |
|
||||
|------|-------------|
|
||||
| `ai-docs` | Prime AI context with key documentation |
|
||||
| `docs-check-filenames` | Detect duplicate filenames in documentation |
|
||||
| `ai-docs` | Prime AI context with key documentation and doc tree |
|
||||
| `docs-check-frontmatter` | Check required frontmatter fields |
|
||||
| `docs-check-index` | Check every doc is referenced in its category index |
|
||||
| `docs-check-links` | Validate wiki-links point to existing filenames |
|
||||
| `docs-check-links` | Validate wiki-links resolve correctly (supports path-based links) |
|
||||
| `docs-mikado` | View active Mikado dependency chains (C2 changes) |
|
||||
| `docs-review` | Review the most stale doc by `last-reviewed` date |
|
||||
| `docs-review-stale` | Report docs by last-modified date |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue