eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions

Fix robots.txt: block /explore/ and /tags/ (was /explorer/)
All checks were successful
Build Container (Nix) / detect (push) Successful in 2s
Details
Build Container / detect (push) Successful in 3s
Details
Build Container (Nix) / build (quartz) (push) Successful in 1s
Details
Build Container / build (quartz) (push) Successful in 16s
Details

Browse source 
The previous robots.txt had a typo blocking /explorer/ instead of
/explore/, allowing Facebook's crawler to hit the spider trap.
Also block /tags/ which has the same infinite relative-link issue.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Erich Blume 2026-03-08 19:57:45 -07:00
commit ede9a51394
1 changed files with 2 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

4
containers/quartz/default.conf
View file

@ -14,12 +14,12 @@ server {
add_header Cache-Control "public, immutable"; add_header Cache-Control "public, immutable";
} }
# Serve robots.txt inline to prevent crawlers from entering /explorer/, # Serve robots.txt inline to prevent crawlers from entering /explore/ and /tags/,
# which is an SPA feature that generates infinite relative-link trees # which is an SPA feature that generates infinite relative-link trees
# when crawled (the March 2026 spider-trap incident). # when crawled (the March 2026 spider-trap incident).
location = /robots.txt { location = /robots.txt {
default_type text/plain; default_type text/plain;
return 200 "User-agent: *\nDisallow: /explorer/\n"; return 200 "User-agent: *\nDisallow: /explore/\nDisallow: /tags/\n";
} }
# SPA fallback - serve index.html for client-side routing # SPA fallback - serve index.html for client-side routing