eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions

Add /user/ to forge robots.txt exclusion
All checks were successful
Deploy Fly.io Proxy / deploy (push) Successful in 1m47s
Details

Browse source 
Crawlers follow auth redirects to /user/login which is pointless for them.
Saves round-trips for both sides.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Erich Blume 2026-04-17 14:34:24 -07:00
commit 1631e11137
1 changed files with 1 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
fly/nginx.conf
View file

@ -158,7 +158,7 @@ http {
# Serve robots.txt directly — block crawlers from expensive endpoints # Serve robots.txt directly — block crawlers from expensive endpoints
location = /robots.txt { location = /robots.txt {
default_type text/plain; default_type text/plain;
return 200 "User-agent: *\nDisallow: /mirrors/\nDisallow: /users/\nDisallow: /*/archive/\nDisallow: /*/releases/download/\n"; return 200 "User-agent: *\nDisallow: /mirrors/\nDisallow: /user/\nDisallow: /users/\nDisallow: /*/archive/\nDisallow: /*/releases/download/\n";
} }
# Block swagger API docs — use forge.ops.eblu.me from tailnet # Block swagger API docs — use forge.ops.eblu.me from tailnet