eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions

Add robots.txt to forge.eblu.me blocking crawlers from /mirrors/
All checks were successful
Deploy Fly.io Proxy / deploy (push) Successful in 2m19s
Details

Browse source 
Facebook has been scraping forge mirror repos at ~3-4 req/s, slowing
down the Forgejo instance. Serve robots.txt directly from nginx to
disallow /mirrors/ while leaving eblume/* accessible to crawlers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Erich Blume 2026-04-16 15:39:48 -07:00
commit 7f6bbdc82c
1 changed files with 6 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

6
fly/nginx.conf
View file

@ -155,6 +155,12 @@ http {
internal; internal;
} }
# Serve robots.txt directly — block crawlers from mirror repos
location = /robots.txt {
default_type text/plain;
return 200 "User-agent: *\nDisallow: /mirrors/\n";
}
# Block swagger API docs — use forge.ops.eblu.me from tailnet # Block swagger API docs — use forge.ops.eblu.me from tailnet
location /swagger { location /swagger {
return 403 "API documentation is only available at forge.ops.eblu.me (tailnet).\n"; return 403 "API documentation is only available at forge.ops.eblu.me (tailnet).\n";