eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions
blumeops/docs/changelog.d/+robots-txt-explorer.infra.md
Erich Blume b64010b3c7
All checks were successful
Build Container (Nix) / detect (push) Successful in 3s
Details
Build Container / detect (push) Successful in 3s
Details
Build Container (Nix) / build (quartz) (push) Successful in 2s
Details
Build Container / build (quartz) (push) Successful in 9s
Details
Replace spider-trap nginx 404s with robots.txt disallowing /explorer/ 
The /explorer/ SPA endpoints were the source of all spider-trap traffic.
A robots.txt Disallow is a better fix than serving 404s — it prevents
crawlers from entering the infinite URL tree in the first place, avoids
serving large numbers of 404s that hurt SEO, and doesn't break legitimate
deep links.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 18:34:37 -08:00

123 B
Raw Blame History

Replace nginx spider-trap 404 guards with robots.txt disallowing /explorer/ to prevent crawler-induced infinite URL trees.