eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions

Update docs for Caddy routing and direct WireGuard peering

Browse source 
Comprehensive docs pass reflecting the new Fly proxy architecture:
- Fly proxy routes through Caddy on indri (not per-service TS Ingress)
- Direct WireGuard peering via --port=41641 pinning
- DERP relay performance lesson in Tailscale docs
- Caddy now in public traffic path
- indri tagged as flyio-target
- Removed fly-reload references
- Updated architecture diagrams and per-service setup guide
- Added changelog fragment

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Erich Blume 2026-04-18 09:57:30 -07:00
commit d26a6ae3b2
8 changed files with 81 additions and 108 deletions
Download patch file Download diff file Expand all files Collapse all files

1
docs/changelog.d/+fly-caddy-direct-peering.infra.md Normal file
View file

@ -0,0 +1 @@
Route Fly.io proxy through Caddy on indri with direct WireGuard peering, reducing public-facing latency from 20+ seconds (DERP relay) to sub-second. Fixed Beyla eBPF tracing on ringtail (memlock rlimit + BPF permissions). Restored trace collection to Tempo.

4
docs/explanation/architecture.md
View file

@ -59,9 +59,9 @@ Three layers of reverse proxying expose services at different scopes:
**Tailscale** is the base layer — every service gets a MagicDNS hostname. The [[tailscale-operator]] gives Kubernetes services their own Tailscale Ingress endpoints.
**[[caddy]]** runs natively on [[indri]] and provides a unified `*.ops.eblu.me` wildcard with TLS (Let's Encrypt via DNS-01/Gandi). It proxies to both local services (Forgejo, Zot, Jellyfin) and Kubernetes services (via their Tailscale Ingress endpoints). Access is restricted by Tailscale ACLs — only `tag:homelab` and `autogroup:admin` can reach Caddy.
**[[caddy]]** runs natively on [[indri]] and provides a unified `*.ops.eblu.me` wildcard with TLS (Let's Encrypt via DNS-01/Gandi). It proxies to both local services (Forgejo, Zot, Jellyfin) and Kubernetes services (via their Tailscale Ingress endpoints). Caddy serves both tailnet clients and public traffic (via the Fly proxy).
**[[flyio-proxy]]** runs on Fly.io for select services that need public internet access. Traffic hits Fly.io's Anycast edge, terminates TLS, and tunnels back to the homelab over Tailscale. Only services explicitly tagged `tag:flyio-target` are reachable — a compromised proxy cannot route to arbitrary services on the tailnet.
**[[flyio-proxy]]** runs on Fly.io for select services that need public internet access. Traffic hits Fly.io's Anycast edge, terminates TLS, and tunnels back to Caddy on indri over a direct Tailscale WireGuard connection. The proxy uses `tag:flyio-target` ACLs — indri carries this tag so the proxy can reach Caddy, but cannot route to arbitrary services on the tailnet.
See [[routing]] for the full service URL table and port map.

18
docs/how-to/operations/manage-flyio-proxy.md
View file

@ -1,7 +1,7 @@
---
title: Manage Fly.io Proxy
modified: 2026-04-17
last-reviewed: 2026-04-17
modified: 2026-04-18
last-reviewed: 2026-04-18
tags:
- how-to
- fly-io
@ -23,16 +23,6 @@ mise run fly-deploy
Pushes to `fly/` on main also trigger automatic deployment via the Forgejo CI workflow.
## Reload Nginx (Re-resolve Upstream DNS)
Nginx uses `upstream` blocks with keepalive connection pools. DNS is resolved at config load. If Tailscale Ingress pods get new IPs (restart, reschedule, minikube restart), reload nginx to re-resolve without a full redeploy:
```bash
mise run fly-reload
```
A Grafana alert fires when upstreams are unreachable, prompting this action. A full `fly-deploy` also re-resolves DNS (it replaces the container).
## Add a New Public Service
See [[expose-service-publicly#Per-service setup]] for the full walkthrough. In short:
@ -88,15 +78,13 @@ The auth key expires every 90 days. To rotate:
## Troubleshooting
**502 Bad Gateway after Tailscale Ingress restart**: Upstream DNS is stale. Run `mise run fly-reload` to re-resolve. This is the most common cause of 502s.
**502 Bad Gateway on fresh deploy**: MagicDNS may not be ready when nginx starts. The `start.sh` script polls `nslookup` before launching nginx, but if it still fails, check that `tailscale status` is healthy inside the container.
**Health check failing**: `fly ssh console -a blumeops-proxy` then `curl localhost:8080/healthz` to test locally.
**TLS errors on custom domain**: Check cert status with `fly certs show <domain> -a blumeops-proxy`. Certs auto-provision via Let's Encrypt and may take a few minutes.
**High latency (>1s p50)**: Likely lost keepalive — redeploy with `mise run fly-deploy`. Before the keepalive change (April 2026), per-request TLS handshakes through the WireGuard tunnel caused 35s+ p50 at >1 req/s.
**High latency (>1s p50)**: Check if direct WireGuard peering is established: `fly ssh console -a blumeops-proxy -C "tailscale ping indri"`. If it shows `via DERP`, the tunnel is relayed and latency will be 10-30s. See [[tailscale#Direct Peering vs DERP Relay]] for diagnosis.
## Related

4
docs/reference/infrastructure/routing.md
View file

@ -1,6 +1,6 @@
---
title: Routing
modified: 2026-04-17
modified: 2026-04-18
tags:
- infrastructure
- networking
@ -46,7 +46,7 @@ DNS points to [[indri]]'s Tailscale IP. TLS via Let's Encrypt (ACME DNS-01 with
## Public Services (`*.eblu.me`)
DNS CNAMEs point to `blumeops-proxy.fly.dev`. TLS via Fly.io-managed Let's Encrypt. Traffic tunnels back to the homelab over Tailscale. Only services tagged `tag:flyio-target` are reachable by the proxy — see [[flyio-proxy]] for details.
DNS CNAMEs point to `blumeops-proxy.fly.dev`. TLS via Fly.io-managed Let's Encrypt. Traffic tunnels back to [[caddy]] on [[indri]] over a direct Tailscale WireGuard connection, then Caddy routes to the service. See [[flyio-proxy]] for details.
| Service | URL | Description |
|---------|-----|-------------|

19
docs/reference/infrastructure/tailscale.md
View file

@ -1,7 +1,7 @@
---
title: Tailscale
modified: 2026-03-22
last-reviewed: 2026-03-22
modified: 2026-04-18
last-reviewed: 2026-04-18
tags:
- infrastructure
- networking
@ -36,7 +36,7 @@ ACLs managed via Pulumi in `pulumi/tailscale/policy.hujson`.
| `tag:k8s` | (Ingress proxy pods) | Kubernetes Tailscale Ingress nodes; each also carries a per-service tag (`tag:grafana`, `tag:kiwix`, `tag:devpi`, `tag:feed`, `tag:pg`) |
| `tag:ci-gateway` | (ephemeral CI containers) | CI containers pushing images to registry |
| `tag:flyio-proxy` | (Fly.io proxy container) | Public reverse proxy |
| `tag:flyio-target` | (designated Ingress endpoints) | Endpoints reachable by the Fly.io proxy |
| `tag:flyio-target` | indri, designated Ingress endpoints | Endpoints reachable by the Fly.io proxy (indri for Caddy routing, Ingress pods for Alloy metrics/logs) |
**Important:** Don't tag user-owned devices (like gilbert) via Pulumi. Tagging converts them to "tagged devices" which lose user identity and break user-based SSH rules. Gilbert is referenced as `tag:workstation` in tagOwners for ownership purposes but remains user-owned so `blume.erich@gmail.com` identity is preserved.
@ -81,6 +81,19 @@ Pulumi uses OAuth client from 1Password (blumeops vault):
- Scopes: acl, dns, devices, services
- Auto-applies `tag:blumeops` to IaC-managed resources
## Direct Peering vs DERP Relay
Just because Tailscale can route traffic does not mean it routes it efficiently. DERP relay servers are a fallback for when direct WireGuard connections cannot be established — they add significant latency (20+ seconds observed under load) because every packet bounces through a relay server.
**Direct peering is critical for any production-like traffic path.** Check with `tailscale ping <host>` — it should say `via <ip>:<port>`, not `via DERP(<region>)`.
Common reasons direct peering fails:
- **k8s pods**: Tailscale Ingress pods behind pod-network NAT cannot hole-punch. Route through a host-level Tailscale node (e.g., Caddy on indri) instead.
- **Cloud VMs**: Some cloud providers block incoming UDP. Pin the WireGuard port (`tailscaled --port=41641`) and expose it as a UDP service if possible.
- **Double NAT / CGNAT**: Multiple NAT layers make hole punching unreliable.
The [[flyio-proxy]] uses `--port=41641` pinning to enable direct peering with indri, and routes through [[caddy]] (host-level Tailscale) to avoid the DERP bottleneck of k8s-hosted Tailscale Ingress pods.
## Related
- [[routing|Routing]] - Service URLs

6
docs/reference/services/caddy.md
View file

@ -1,6 +1,6 @@
---
title: Caddy
modified: 2026-03-15
modified: 2026-04-18
tags:
- service
- networking
@ -83,7 +83,9 @@ The token is written to `~/.config/caddy/gandi-token` (chmod 0600) and sourced b
## Security Considerations
Caddy has no authentication layer — it is a plain reverse proxy. Access control relies entirely on Tailscale ACLs restricting which devices can reach indri on port 443. Currently `tag:homelab` and `autogroup:admin` can reach Caddy. The [[flyio-proxy]] no longer routes through Caddy — it pushes logs and metrics directly to [[loki]] and [[prometheus]] via their Tailscale Ingress endpoints.
Caddy has no authentication layer — it is a plain reverse proxy. Access control relies entirely on Tailscale ACLs restricting which devices can reach indri on port 443. Currently `tag:homelab`, `autogroup:admin`, and `tag:flyio-proxy` (via `tag:flyio-target` on indri) can reach Caddy.
The [[flyio-proxy]] routes all public traffic through Caddy. This is the path for `*.eblu.me` requests from the public internet. Caddy sees these as requests from the Fly VM with `Host: *.ops.eblu.me` headers — the same routes used by tailnet clients.
## Custom Build

34
docs/reference/services/flyio-proxy.md
View file

@ -1,6 +1,6 @@
---
title: Fly.io Proxy
modified: 2026-04-17
modified: 2026-04-18
tags:
- service
- networking
@ -23,23 +23,27 @@ Public reverse proxy on [Fly.io](https://fly.io) that exposes selected BlumeOps
## Exposed Services
| Public domain | Backend | Service |
|---------------|---------|---------|
| `docs.eblu.me` | `docs.tail8d86e.ts.net` | [[docs]] |
| `cv.eblu.me` | `cv.tail8d86e.ts.net` | [[cv]] |
| `forge.eblu.me` | `forge.tail8d86e.ts.net` | [[forgejo]] |
| Public domain | Backend (via Caddy) | Service |
|---------------|---------------------|---------|
| `docs.eblu.me` | `docs.ops.eblu.me` | [[docs]] |
| `cv.eblu.me` | `cv.ops.eblu.me` | [[cv]] |
| `forge.eblu.me` | `forge.ops.eblu.me` | [[forgejo]] |
## Architecture
Internet traffic hits Fly.io's Anycast edge, terminates TLS with a Let's Encrypt certificate, and is proxied by nginx to the backend service over a Tailscale WireGuard tunnel. See [[expose-service-publicly]] for the full architecture diagram.
Internet traffic hits Fly.io's Anycast edge, terminates TLS with a Let's Encrypt certificate, and is proxied by nginx to [[caddy]] on [[indri]] over a direct Tailscale WireGuard tunnel. Caddy then routes to the actual service. See [[expose-service-publicly]] for the full architecture diagram.
### Upstream Keepalive
### Why Caddy, not per-service Tailscale Ingress?
Nginx uses `upstream` blocks with `keepalive` connection pools to reuse TLS connections through the WireGuard tunnel. This avoids a per-request TLS handshake, which was previously the dominant source of latency (35s+ p50 before keepalive, sub-second after).
Previously, nginx connected directly to each service's `*.tail8d86e.ts.net` Tailscale Ingress endpoint. This caused **20+ second latency** because the Tailscale Ingress pods (running inside k8s) are behind pod-network NAT and can only reach the Fly VM via Tailscale DERP relay servers — not direct WireGuard peering.
**Trade-off:** DNS for upstream hostnames is resolved once at config load, not per-request. If Tailscale Ingress pods get new IPs (restart, reschedule, minikube restart), run `mise run fly-reload` to re-resolve without a full redeploy. A Grafana alert fires when upstreams are unreachable.
Routing through Caddy on indri solves this because indri's host-level Tailscale can establish direct WireGuard connections with the Fly VM (45ms round trip). This generalizes to all services regardless of where they run (native on indri, minikube, or ringtail k3s), since Caddy already routes to everything.
Each upstream requires `proxy_ssl_name` set to the actual Tailscale hostname — nginx sends the upstream block name as SNI by default, which the Tailscale Ingress proxy won't recognize.
### Direct WireGuard Peering
The Fly VM pins its Tailscale WireGuard listener to port 41641 (`tailscaled --port=41641`). Combined with well-behaved NAT on both sides (`MappingVariesByDestIP: false`), this allows Tailscale to establish direct peer-to-peer connections via UDP hole punching — no dedicated IPv4 required.
If direct peering fails (observable via `tailscale ping indri` showing "via DERP"), allocate a dedicated IPv4 ($2/month) with `fly ips allocate-v4` to provide a guaranteed inbound UDP path.
## Key Files
@ -58,6 +62,8 @@ Each upstream requires `proxy_ssl_name` set to the actual Tailscale hostname —
Fly.io runs Firecracker microVMs which support TUN devices natively. Tailscale runs with a real TUN interface (not userspace networking), so MagicDNS and direct Tailscale IP routing work normally.
The `tailscaled` process is started with `--port=41641` to pin the WireGuard listener to a fixed port. This is critical for direct peering — without it, hole punching is unreliable. A `[[services]]` block in `fly.toml` exposes this port as UDP, though it is only active when a dedicated IPv4 is allocated.
The Tailscale auth key is `preauthorized=True` to avoid device approval hangs on container restarts.
## Observability
@ -83,9 +89,7 @@ Alloy listens on `127.0.0.1:12345` for self-scraping its `/metrics` endpoint. Al
## Security Considerations
The `tag:flyio-proxy` ACL grants access only to `tag:flyio-target:443`. Services must explicitly opt in by adding a `tailscale.com/tags: "tag:k8s,tag:flyio-target"` annotation to their Tailscale Ingress. This means the proxy can only reach endpoints that have been individually tagged — a compromised nginx config cannot route to arbitrary services on the tailnet.
Currently tagged as `tag:flyio-target`: [[docs]], [[cv]], [[forgejo]], [[loki]], [[prometheus]]. Loki and Prometheus are tagged so that [[alloy|Alloy]] (running inside the container) can push logs and metrics directly via their Tailscale Ingress endpoints — the restricted ACL means Caddy on indri (`tag:homelab`) is not reachable from the proxy.
The `tag:flyio-proxy` ACL grants access only to `tag:flyio-target:443`. Indri carries this tag (for Caddy), and the k8s Tailscale Ingress pods for Loki and Prometheus also carry it so [[alloy|Alloy]] can push logs and metrics directly. A compromised proxy cannot route to arbitrary services on the tailnet — only `tag:flyio-target` endpoints on port 443.
### Crawler Mitigation
@ -101,7 +105,7 @@ Archive requests (`/<owner>/<repo>/archive/*`) are 302-redirected to `forge.ops.
Release downloads are cached at the proxy layer (7-day TTL, keyed by URI) to absorb repeated downloads of the same artifact.
To expose an additional service through the proxy, add the `tag:flyio-target` annotation to its Tailscale Ingress. See [[expose-service-publicly]] for the full workflow.
To expose an additional service through the proxy, add a Caddy route for it and an nginx `server` block. See [[expose-service-publicly]] for the full workflow.
## Spider Trap Mitigation

103
docs/tutorials/expose-service-publicly.md
View file

@ -1,7 +1,7 @@
---
title: Expose a Service Publicly
modified: 2026-04-17
last-reviewed: 2026-04-17
modified: 2026-04-18
last-reviewed: 2026-04-18
tags:
- tutorials
- fly-io
@ -27,24 +27,21 @@ Internet → <service>.eblu.me
Fly.io edge (Anycast, TLS via Let's Encrypt)
│
Fly.io VM (nginx reverse proxy + Tailscale)
│ (WireGuard tunnel)
tailnet (tail8d86e.ts.net)
│ (direct WireGuard tunnel to indri)
Caddy on indri (*.ops.eblu.me routing)
│
<service>.tail8d86e.ts.net (Tailscale ingress)
│
k8s Service → pod
backend service (k8s, native, or remote)
```
(The approach works similarly for non-k8s services via `tailscale serve`
service definitions, eg. [[forgejo]] and [[zot]])
A single Fly.io container serves as the public-facing proxy for all exposed
services. Each service gets a `server` block in the nginx config and a DNS
CNAME. The container joins the tailnet via an ephemeral auth key and reaches
backend services through Tailscale ingress endpoints.
services. Nginx routes all traffic through [[caddy]] on [[indri]] via a
direct Tailscale WireGuard connection. Caddy already knows how to route
to every service (native, minikube, or ringtail k3s), so adding a new
public service only requires an nginx `server` block and a DNS CNAME.
Existing `*.ops.eblu.me` services remain private behind Tailscale — this
approach does not touch [[caddy]], [[gandi]] DNS-01, or any other existing
infrastructure. They can continue to operate in parallel for private access.
The `*.ops.eblu.me` routes continue to work in parallel for private tailnet
access — the Fly proxy sends `Host: <service>.ops.eblu.me` headers that
match the same Caddy routes.
## Key decisions
@ -61,21 +58,18 @@ infrastructure. They can continue to operate in parallel for private access.
## TLS in this architecture
There are three independent TLS segments — none involve Caddy:
There are three independent TLS segments:
1. **Browser → Fly.io edge**: Fly.io auto-provisions a Let's Encrypt
certificate for each custom domain (e.g., `docs.eblu.me`). Validated via
TLS-ALPN challenge — no DNS API needed.
2. **nginx → Tailscale ingress**: nginx proxies to
`https://<service>.tail8d86e.ts.net`. The Tailscale ingress serves a
Tailscale-issued cert. nginx uses `proxy_ssl_verify off` since the
underlying tunnel is already encrypted.
2. **nginx → Caddy on indri**: nginx proxies to `https://indri.tail8d86e.ts.net`
with `Host: <service>.ops.eblu.me`. Caddy serves its `*.ops.eblu.me`
Let's Encrypt wildcard cert. nginx uses `proxy_ssl_verify off` since the
underlying WireGuard tunnel is already encrypted.
3. **WireGuard tunnel**: All Tailscale traffic is encrypted at the network
layer regardless of application-level TLS.
Caddy continues to serve `*.ops.eblu.me` with its existing Gandi DNS-01
certificates. The two TLS domains are completely independent.
## External references
- [Tailscale on Fly.io](https://tailscale.com/kb/1132/flydotio) — official guide for running Tailscale in a Fly.io container
@ -116,8 +110,8 @@ See the actual files in `fly/` for current configuration. Key design points:
- **`fly.toml`** — uses bluegreen deploys so the old machine serves traffic until the new one passes health checks. `auto_stop_machines = "off"` keeps the proxy always-on.
- **`Dockerfile`** — multi-stage build pulling nginx, Tailscale, and [[alloy]] binaries. Alloy runs as a sidecar inside the container for observability (see below).
- **`start.sh`** — starts `tailscaled` first, waits for MagicDNS readiness (polls `nslookup` against `100.100.100.100`), then starts nginx, fail2ban, and Alloy, and blocks on the nginx process. The MagicDNS check is required because `upstream` blocks resolve DNS at config load.
- **`nginx.conf`** — uses `upstream` blocks with `keepalive` connection pools for each backend service. DNS is resolved at config load via MagicDNS (`resolver 100.100.100.100`). Each upstream requires `proxy_ssl_name` set explicitly to the Tailscale hostname (nginx sends the block name as SNI by default). A `map` directive conditionally sets the `Connection` header — empty string for keepalive on normal requests, `upgrade` only for WebSocket requests. Includes a JSON access log format that Alloy tails for log collection and metric extraction. A catch-all server block serves `/healthz` and rejects unknown hosts.
- **`start.sh`** — starts `tailscaled --port=41641` first (pinned port enables direct WireGuard peering), waits for MagicDNS readiness (polls `nslookup` against `100.100.100.100`), then starts nginx, fail2ban, and Alloy, and blocks on the nginx process. The MagicDNS check is required because the `upstream` block resolves DNS at config load.
- **`nginx.conf`** — uses a single `upstream` block with `keepalive` pointing at Caddy on indri (`indri.tail8d86e.ts.net:443`). All services route through this upstream with `Host: <service>.ops.eblu.me` headers for Caddy routing. Includes a JSON access log format that Alloy tails for log collection and metric extraction. A catch-all server block serves `/healthz` and rejects unknown hosts.
- **`error.html`** — shown via `proxy_intercept_errors` when upstreams are unreachable (indri offline, tunnel down, etc.). Cached responses still take priority via `proxy_cache_use_stale`.
#### Observability sidecar
@ -174,11 +168,11 @@ ACL test:
{
"src": "tag:flyio-proxy",
"accept": ["tag:flyio-target:443"],
"deny": ["tag:k8s:443", "tag:homelab:443", "tag:homelab:22", "tag:nas:445", "tag:registry:443"],
"deny": ["tag:k8s:443", "tag:homelab:22", "tag:nas:445", "tag:registry:443"],
},
```
Each service's Tailscale Ingress must be annotated with `tag:flyio-target` to be reachable by the proxy — see [[#7. Tag the Tailscale Ingress with tag:flyio-target]].
Indri carries `tag:flyio-target` so the Fly proxy can reach Caddy. No per-service tagging is needed — Caddy handles routing to all services.
Deploy: `mise run tailnet-preview` then `mise run tailnet-up`.
@ -214,20 +208,17 @@ The `FLY_DEPLOY_TOKEN` Forgejo Actions secret must be set via the [[forgejo]] AP
To expose an additional service (example: `wiki.eblu.me`):
### 1. Add nginx server block
### 1. Ensure the service has a Caddy route
Edit `fly/nginx.conf` — two changes needed:
The service must be accessible via `<service>.ops.eblu.me` through [[caddy]].
Most services already have this. If not, add it to `ansible/roles/caddy/defaults/main.yml`
and deploy with `mise run provision-indri -- --tags caddy`.
1. **Add an `upstream` block** (in the `http` context, alongside the existing ones):
### 2. Add nginx server block
```nginx
upstream wiki_backend {
server wiki.tail8d86e.ts.net:443;
keepalive 4;
}
```
2. **Add a `server` block.** The configuration differs significantly between static and dynamic services. See the existing blocks in `fly/nginx.conf` for the current pattern.
Edit `fly/nginx.conf` — add a `server` block. All services use the shared
`indri_backend` upstream (Caddy on indri). Set `Host` and `proxy_ssl_name`
to the service's `*.ops.eblu.me` hostname so Caddy routes correctly.
**Static site template** (simplified — adapt from existing blocks):
@ -246,11 +237,11 @@ server {
}
location / {
proxy_pass https://wiki_backend$request_uri;
proxy_pass https://indri_backend$request_uri;
proxy_ssl_verify off;
proxy_ssl_server_name on;
proxy_ssl_name wiki.tail8d86e.ts.net;
proxy_set_header Host wiki.tail8d86e.ts.net;
proxy_ssl_name wiki.ops.eblu.me;
proxy_set_header Host wiki.ops.eblu.me;
proxy_intercept_errors on;
proxy_http_version 1.1;
@ -270,25 +261,8 @@ server {
}
```
**Key points for all upstream blocks:**
- `proxy_ssl_name` must be set explicitly — nginx sends the upstream block name as SNI by default, which the Tailscale Ingress won't recognize
- `proxy_http_version 1.1` + `Connection $connection_upgrade` enables keepalive (empty string for normal requests, "upgrade" for WebSocket)
- `keepalive` pool size: 4 for low-traffic static sites, 8 for higher-traffic dynamic services
**Dynamic service template** — see `fly/nginx.conf` for the live Forgejo configuration, which includes rate-limited auth endpoints, cached static assets and release downloads, archive endpoint redirects, robots.txt, and WebSocket support.
Key differences for dynamic services:
- **No blanket caching** — only static assets (CSS, JS, images) are cached
- **Respect `Set-Cookie`** — do not ignore session headers
- **Include query strings** in non-cached requests (default behavior when
`proxy_cache_key` is not overridden)
- **Higher rate limits** — legitimate usage patterns are burstier
- **Proxy headers** — pass `X-Real-IP`, `X-Forwarded-For`, `X-Forwarded-Proto`
so the backend sees the real client IP (important for Forgejo's audit logs
and its own rate limiting)
- **WebSocket support** — many modern web apps use WebSockets
- **Larger body size** — git pushes and file uploads need more than the default 1MB
### 2. Add Fly.io certificate
```bash
@ -345,18 +319,9 @@ curl -I https://wiki.eblu.me
# Should return 200 with X-Cache-Status header
```
### 7. Tag the Tailscale Ingress with `tag:flyio-target`
### 7. Verify routing
The fly.io proxy can only reach endpoints tagged with `tag:flyio-target`. Add the annotation to the service's Tailscale Ingress:
```yaml
annotations:
tailscale.com/tags: "tag:k8s,tag:flyio-target"
```
Include `tag:k8s` to preserve existing access rules for the Ingress proxy node. The `tag:flyio-target` tag opts this specific endpoint into being reachable by the fly.io proxy — no broad ACL changes needed.
For non-k8s services (e.g., Forgejo on indri), create a k8s ExternalName Service pointing to the host, then a Tailscale Ingress with the same annotation.
Since all traffic routes through Caddy on indri, no per-service Tailscale Ingress tagging is needed. As long as the service has a Caddy route (step 1), the Fly proxy can reach it.
---