Fix spider trap: disable SPA mode, remove index files, relax wiki-links (#290)
## Summary Fixes the Facebook crawler spider trap that's been generating infinite recursive URLs like `/how-to/tutorials/tutorials/how-to/explanation/...` for several days. **Root cause:** Quartz SPA mode + nginx `try_files` fallback to `index.html` meant any fabricated URL returned the root HTML shell with HTTP 200. Crawlers followed relative links from those fake URLs, creating infinite recursion. **Fix:** - Disable Quartz SPA mode (`enableSPA: false`) — all pages are now fully static HTML - Replace nginx SPA fallback with `=404` + Quartz's static `404.html` - Remove `robots.txt` exclusions (no longer needed) **Docs cleanup (Obsidian.nvim compat no longer needed):** - Delete hand-curated category index files (`tutorials.md`, `reference.md`, `how-to.md`, `explanation.md`) — Quartz auto-generates folder pages - Delete `postgresql-storage.md` (redirect stub) and `migrate-forgejo-from-brew.md` (stale history) - Drop `docs-check-index` and `docs-check-filenames` prek hooks - Rewrite `docs-check-links` to allow path-based wiki-links (`[[path/to/file]]`) and only error on true ambiguity - Add `ai-docs` doc tree listing to replace index files for AI context - Add natural cross-links from reference cards to fix orphan docs ## Deployment and Testing - [ ] Merge and let the build pipeline run - [ ] Verify docs.eblu.me serves pages correctly with full page loads - [ ] Verify non-existent URLs return 404 - [ ] Monitor crawler traffic — should drop to near zero for fabricated URLs Reviewed-on: #290
This commit is contained in:
parent
953640d2b7
commit
4f0476a851
24 changed files with 110 additions and 666 deletions
|
|
@ -1,95 +0,0 @@
|
|||
---
|
||||
title: Reference
|
||||
modified: 2026-03-04
|
||||
tags:
|
||||
- reference
|
||||
---
|
||||
|
||||
# Reference
|
||||
|
||||
Technical specifications, inventories, and configuration details for BlumeOps infrastructure.
|
||||
|
||||
## Services
|
||||
|
||||
Individual service reference cards with URLs and configuration details.
|
||||
|
||||
| Service | Description | Location |
|
||||
|---------|-------------|----------|
|
||||
| [[alloy|Alloy]] | Observability collector (metrics & logs) | indri + k8s |
|
||||
| [[argocd]] | GitOps continuous delivery | k8s |
|
||||
| [[borgmatic]] | Backup system | indri |
|
||||
| [[caddy]] | Reverse proxy & TLS termination | indri |
|
||||
| [[1password]] | Secrets management | cloud + k8s |
|
||||
| [[forgejo]] | Git forge & CI/CD | indri |
|
||||
| [[frigate]] | Network video recorder | k8s (ringtail) |
|
||||
| [[grafana]] | Dashboards & visualization | k8s |
|
||||
| [[immich]] | Photo management | k8s |
|
||||
| [[jellyfin]] | Media server | indri |
|
||||
| [[jobsync]] | Job application tracker | k8s (ringtail) |
|
||||
| [[kiwix]] | Offline Wikipedia & ZIM archives | k8s |
|
||||
| [[loki]] | Log aggregation | k8s |
|
||||
| [[tempo]] | Distributed tracing | k8s |
|
||||
| [[miniflux]] | RSS feed reader | k8s |
|
||||
| [[navidrome]] | Music streaming | k8s |
|
||||
| [[ntfy]] | Push notifications | k8s (ringtail) |
|
||||
| [[postgresql]] | Database cluster | k8s |
|
||||
| [[prometheus]] | Metrics collection | k8s |
|
||||
| [[teslamate]] | Tesla data logger | k8s |
|
||||
| [[transmission]] | BitTorrent daemon | k8s |
|
||||
| [[zot]] | Container registry | indri |
|
||||
| [[devpi]] | PyPI caching proxy | k8s |
|
||||
| [[cv]] | Resume / CV site | k8s |
|
||||
| [[authentik]] | OIDC identity provider | k8s (ringtail) |
|
||||
| [[docs]] | Documentation site (Quartz) | k8s |
|
||||
| [[flyio-proxy]] | Public reverse proxy (Fly.io + Tailscale) | Fly.io |
|
||||
| [[ollama]] | LLM inference server | k8s (ringtail) |
|
||||
| [[automounter]] | SMB share automounter | indri |
|
||||
|
||||
## Infrastructure
|
||||
|
||||
Host inventory and network configuration.
|
||||
|
||||
- [[hosts|Hosts]] - Device inventory
|
||||
- [[indri]] - Primary server
|
||||
- [[ringtail]] - Service host & gaming PC
|
||||
- [[gilbert]] - Development workstation
|
||||
- [[tailscale]] - ACLs, groups, tags
|
||||
- [[gandi]] - DNS hosting for `eblu.me`
|
||||
- [[unifi]] - Home WiFi router (UniFi Express 7)
|
||||
- [[routing|Routing]] - DNS domains, port mappings
|
||||
- [[power]] - Battery-backed power chain
|
||||
|
||||
## Tools
|
||||
|
||||
Build, deployment, and IaC tool reference.
|
||||
|
||||
- [[mise-tasks]] - Operational task runner (all `mise run` tasks)
|
||||
- [[dagger]] - CI/CD build engine (Python SDK)
|
||||
- [[argocd-cli]] - ArgoCD CLI workflows
|
||||
- [[ansible]] - Configuration management for indri
|
||||
- [[pulumi]] - Infrastructure-as-Code (DNS, Tailscale ACLs)
|
||||
|
||||
## Kubernetes
|
||||
|
||||
Cluster configuration and application registry.
|
||||
|
||||
- [[cluster|Cluster]] - Minikube specs, storage, networking
|
||||
- [[apps|Apps]] - ArgoCD application registry
|
||||
- [[tailscale-operator]] - Tailscale ingress for k8s services
|
||||
- [[external-secrets]] - Secrets management
|
||||
|
||||
## Storage
|
||||
|
||||
Network storage and backup configuration.
|
||||
|
||||
- [[sifaka|Sifaka]] - Synology NAS configuration
|
||||
- [[postgresql-storage]] - Database cluster
|
||||
- [[backups|Backups]] - Backup policy and schedule
|
||||
|
||||
## Operations
|
||||
|
||||
Operational concerns and their components.
|
||||
|
||||
- [[observability]] - Metrics, logs, dashboards
|
||||
- [[backup]] - Data protection
|
||||
- [[disaster-recovery]] - Recovery procedures (TBD)
|
||||
|
|
@ -60,7 +60,7 @@ Future clients: [[argocd]], [[miniflux]], [[zot]]
|
|||
|
||||
## Secrets
|
||||
|
||||
Injected via [[external-secrets]] from the "Authentik (blumeops)" 1Password item.
|
||||
Injected via [[external-secrets]] from the "Authentik (blumeops)" 1Password item (see [[create-authentik-secrets]] for setup).
|
||||
|
||||
| 1Password Field | Purpose |
|
||||
|-----------------|---------|
|
||||
|
|
@ -79,4 +79,7 @@ Nix-built via `dockerTools.buildLayeredImage`. The entrypoint wrapper symlinks b
|
|||
- [[federated-login]] - How authentication works across BlumeOps
|
||||
- [[grafana]] - First OIDC client
|
||||
- [[deploy-authentik]] - Deployment how-to
|
||||
- [[migrate-grafana-to-authentik]] - Grafana SSO migration from Dex
|
||||
- [[build-authentik-from-source]] - Nix-based container build
|
||||
- [[mirror-authentik-build-deps]] - Supply chain mirrors for the build
|
||||
- [[external-secrets]] - Secrets injection from 1Password
|
||||
|
|
|
|||
|
|
@ -120,6 +120,10 @@ The UI shows `forge.eblu.me` for HTTPS clone URLs and `forge.ops.eblu.me` for SS
|
|||
|
||||
`mise run fly-shutoff` stops all public traffic immediately. forge.ops.eblu.me continues to work from the tailnet. See [[expose-service-publicly#Break-glass shutoff]].
|
||||
|
||||
## Mirrors
|
||||
|
||||
Forgejo hosts pull mirrors of external repositories (GitHub, etc.) for supply chain control. Mirrors live in the `mirrors/` org and sync on a configurable interval. See [[manage-forgejo-mirrors]] for operations.
|
||||
|
||||
## Related
|
||||
|
||||
- [[argocd]] - Uses Forgejo as git source
|
||||
|
|
|
|||
|
|
@ -63,6 +63,7 @@ Optional annotation: `grafana_folder: "FolderName"`
|
|||
- [[build-grafana-sidecar]] - Home-built sidecar container
|
||||
- [[kustomize-grafana-deployment]] - Kustomize manifest structure
|
||||
- [[authentik]] - OIDC identity provider for SSO
|
||||
- [[migrate-grafana-to-authentik]] - How SSO was migrated from Dex to Authentik
|
||||
- [[prometheus]] - Metrics datasource
|
||||
- [[loki]] - Logs datasource
|
||||
- [[tempo]] - Traces datasource
|
||||
|
|
|
|||
|
|
@ -65,3 +65,5 @@ The `zot-ci` API key expires every **90 days**. To rotate:
|
|||
- [[forgejo]] - Container build CI
|
||||
- [[cluster|Cluster]] - Registry consumer
|
||||
- [[authentik]] - OIDC identity provider
|
||||
- [[harden-zot-registry]] - Security hardening guide
|
||||
- [[install-dagger-on-nix-runner]] - Why Dagger can't run on the Nix builder
|
||||
|
|
|
|||
|
|
@ -1,11 +0,0 @@
|
|||
---
|
||||
title: PostgreSQL Storage
|
||||
modified: 2026-02-07
|
||||
tags:
|
||||
- storage
|
||||
- database
|
||||
---
|
||||
|
||||
# PostgreSQL Storage
|
||||
|
||||
See [[postgresql]] in Services.
|
||||
|
|
@ -17,11 +17,9 @@ Run `mise tasks --sort name` for the live list with descriptions.
|
|||
|
||||
| Task | Description |
|
||||
|------|-------------|
|
||||
| `ai-docs` | Prime AI context with key documentation |
|
||||
| `docs-check-filenames` | Detect duplicate filenames in documentation |
|
||||
| `ai-docs` | Prime AI context with key documentation and doc tree |
|
||||
| `docs-check-frontmatter` | Check required frontmatter fields |
|
||||
| `docs-check-index` | Check every doc is referenced in its category index |
|
||||
| `docs-check-links` | Validate wiki-links point to existing filenames |
|
||||
| `docs-check-links` | Validate wiki-links resolve correctly (supports path-based links) |
|
||||
| `docs-mikado` | View active Mikado dependency chains (C2 changes) |
|
||||
| `docs-review` | Review the most stale doc by `last-reviewed` date |
|
||||
| `docs-review-stale` | Report docs by last-modified date |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue