C0: split gandi-operations docs; add dns-acme-cleanup mise task
Splits the nebulous gandi-operations how-to into two single-topic cards (manage-eblu-me-dns, rotate-gandi-pat) and adds a mise task for the recurring _acme-challenge TXT cleanup needed due to a value-comparison bug in libdns/gandi v1.1.0 that prevents certmagic's cleanup phase from removing presented TXT values. The gandi reference card is updated to drop the false "different credential from Pulumi PAT" claim — verified during the 2026-04-27 incident that Caddy and Pulumi share a single PAT. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
72b27b7fd2
commit
005e2a03ed
10 changed files with 315 additions and 159 deletions
|
|
@ -1,90 +0,0 @@
|
|||
---
|
||||
title: Gandi Operations
|
||||
modified: 2026-02-17
|
||||
last-reviewed: 2026-02-17
|
||||
tags:
|
||||
- how-to
|
||||
- dns
|
||||
- pulumi
|
||||
---
|
||||
|
||||
# Gandi Operations
|
||||
|
||||
How to manage DNS records and cycle the Gandi API token.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Pulumi CLI installed (`brew install pulumi`)
|
||||
- Access to 1Password blumeops vault (for PAT)
|
||||
- On the tailnet (Pulumi resolves indri's IP via MagicDNS)
|
||||
|
||||
## Preview and Apply DNS Changes
|
||||
|
||||
```bash
|
||||
# Preview changes (always do this first)
|
||||
mise run dns-preview
|
||||
|
||||
# Apply changes
|
||||
mise run dns-up
|
||||
```
|
||||
|
||||
Both tasks fetch the Gandi PAT from 1Password automatically.
|
||||
|
||||
To run Pulumi directly:
|
||||
|
||||
```bash
|
||||
export GANDI_PERSONAL_ACCESS_TOKEN=$(op read "op://vg6xf6vvfmoh5hqjjhlhbeoaie/mco6ka3dc3rmw7zkg2dhia5d2m/pat")
|
||||
cd pulumi/gandi
|
||||
pulumi preview
|
||||
pulumi up --yes
|
||||
```
|
||||
|
||||
## Cycle the Gandi PAT
|
||||
|
||||
The Gandi Personal Access Token has a maximum lifetime of 90 days. Currently set to 30 days as a security compromise, though shorter may be appropriate given infrequent use.
|
||||
|
||||
### 1. Create a new PAT
|
||||
|
||||
Go to the [Gandi admin console](https://admin.gandi.net/organizations/1db8d76a-f729-11ed-b8d1-00163e94b645/account/pat) and create a new token:
|
||||
|
||||
- **Name:** `blumeops-pulumi` (or similar)
|
||||
- **Expiration:** 30 days (max 90; shorter is fine if you run this rarely)
|
||||
- **Required permission:** Manage domain name technical configurations
|
||||
- **Also enable:** See and renew domain names
|
||||
|
||||
Copy the new PAT to your clipboard.
|
||||
|
||||
### 2. Update 1Password
|
||||
|
||||
With the new PAT on your clipboard:
|
||||
|
||||
```bash
|
||||
op item edit mco6ka3dc3rmw7zkg2dhia5d2m pat="$(pbpaste)" --vault vg6xf6vvfmoh5hqjjhlhbeoaie
|
||||
```
|
||||
|
||||
### 3. Delete the old PAT
|
||||
|
||||
Return to the Gandi admin console and delete the previous token.
|
||||
|
||||
### 4. Verify
|
||||
|
||||
```bash
|
||||
mise run dns-preview
|
||||
```
|
||||
|
||||
A successful preview confirms the new PAT is working.
|
||||
|
||||
## Break-Glass Override
|
||||
|
||||
If MagicDNS is unavailable and Pulumi can't resolve indri's IP, set the target IP manually. Find indri's current Tailscale IP via `tailscale status` or the admin console:
|
||||
|
||||
```bash
|
||||
export BLUMEOPS_REVERSE_PROXY_IP=<indri-tailscale-ip>
|
||||
mise run dns-up
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- [[gandi]] - DNS configuration reference
|
||||
- [[caddy]] - Reverse proxy (also uses a Gandi token for TLS)
|
||||
- [[update-tailscale-acls]] - Similar Pulumi workflow for Tailscale
|
||||
52
docs/how-to/configuration/manage-eblu-me-dns.md
Normal file
52
docs/how-to/configuration/manage-eblu-me-dns.md
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
---
|
||||
title: Manage eblu.me DNS Records
|
||||
modified: 2026-04-27
|
||||
last-reviewed: 2026-04-27
|
||||
tags:
|
||||
- how-to
|
||||
- dns
|
||||
- pulumi
|
||||
---
|
||||
|
||||
# Manage eblu.me DNS Records
|
||||
|
||||
How to add, change, and apply DNS records for `eblu.me` via [[pulumi]].
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Pulumi CLI installed (`brew install pulumi`)
|
||||
- 1Password access (`blumeops` vault) — Pulumi reads the Gandi PAT from there
|
||||
- On the tailnet — Pulumi resolves [[indri]]'s IP via MagicDNS at apply time
|
||||
|
||||
## Preview and apply
|
||||
|
||||
```bash
|
||||
mise run dns-preview # always do this first
|
||||
mise run dns-up # apply
|
||||
```
|
||||
|
||||
Both fetch the PAT from 1Password automatically. The Pulumi program is in `pulumi/gandi/`; stack is `eblu-me`.
|
||||
|
||||
## Adding a record
|
||||
|
||||
Edit `pulumi/gandi/__main__.py` and add a `gandi.livedns.Record(...)`. The stack config (`Pulumi.eblu-me.yaml`) only holds `domain` and `subdomain`; everything else is in the program.
|
||||
|
||||
After editing, preview, then apply.
|
||||
|
||||
## Break-glass: override the indri target IP
|
||||
|
||||
The wildcard `*.ops.eblu.me` is computed from `indri.tail8d86e.ts.net` via MagicDNS at apply time. If MagicDNS is unavailable:
|
||||
|
||||
```bash
|
||||
export BLUMEOPS_REVERSE_PROXY_IP=<indri-tailscale-ip>
|
||||
mise run dns-up
|
||||
```
|
||||
|
||||
Find the IP via `tailscale status` or the Tailscale admin console.
|
||||
|
||||
## Related
|
||||
|
||||
- [[gandi]] — Gandi reference card
|
||||
- [[rotate-gandi-pat]] — Rotate the PAT shared with [[caddy]]
|
||||
- [[pulumi]] — Pulumi tooling reference
|
||||
- [[routing]] — Service URLs and routing architecture
|
||||
|
|
@ -144,6 +144,6 @@ Trigger a manual sync on one mirror to confirm the new PAT works:
|
|||
## Related
|
||||
|
||||
- [[forgejo]] — Forgejo service reference
|
||||
- [[gandi-operations]] — Similar PAT rotation workflow for Gandi DNS
|
||||
- [[rotate-gandi-pat]] — Similar PAT rotation workflow for Gandi DNS
|
||||
- [[spork-strategy]] — floating-branch soft-fork strategy explanation
|
||||
- [[create-a-spork]] — create a spork on top of a mirror
|
||||
|
|
|
|||
125
docs/how-to/configuration/rotate-gandi-pat.md
Normal file
125
docs/how-to/configuration/rotate-gandi-pat.md
Normal file
|
|
@ -0,0 +1,125 @@
|
|||
---
|
||||
title: Rotate the Gandi PAT
|
||||
modified: 2026-04-27
|
||||
last-reviewed: 2026-04-27
|
||||
tags:
|
||||
- how-to
|
||||
- dns
|
||||
- secrets
|
||||
---
|
||||
|
||||
# Rotate the Gandi PAT
|
||||
|
||||
How to rotate the Gandi Personal Access Token. **One PAT** is shared by [[caddy]] (TLS via ACME DNS-01) and Pulumi (DNS records). It lives in 1Password at `op://blumeops/gandi - blumeops/pat`.
|
||||
|
||||
## When to rotate
|
||||
|
||||
- Every 60 days (Todoist recurring task)
|
||||
- After any compromise / accidental disclosure
|
||||
- Whenever Gandi starts rejecting the PAT (see [Debugging](#debugging))
|
||||
|
||||
Gandi caps PAT lifetime at 90 days; rotating at 60 leaves a 30-day buffer.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Access to the [Gandi PAT admin console](https://admin.gandi.net/organizations/1db8d76a-f729-11ed-b8d1-00163e94b645/account/pat)
|
||||
- 1Password (`blumeops` vault)
|
||||
- Ability to run `mise run provision-indri` (ssh to [[indri]] + 1Password biometric)
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Create a new PAT in Gandi
|
||||
|
||||
In the [Gandi PAT console](https://admin.gandi.net/organizations/1db8d76a-f729-11ed-b8d1-00163e94b645/account/pat), create a token:
|
||||
|
||||
- **Name:** `blumeops`
|
||||
- **Expiration:** **90 days** (the max — paired with the 60-day rotation cadence)
|
||||
- **Permissions:**
|
||||
- Manage domain name technical configurations *(required — DNS records and ACME TXT writes)*
|
||||
- See and renew domain names
|
||||
|
||||
Other permissions are not used.
|
||||
|
||||
Copy the new PAT to your clipboard.
|
||||
|
||||
### 2. Update 1Password
|
||||
|
||||
```bash
|
||||
op item edit mco6ka3dc3rmw7zkg2dhia5d2m pat="$(pbpaste)" --vault vg6xf6vvfmoh5hqjjhlhbeoaie
|
||||
```
|
||||
|
||||
### 3. Push to indri
|
||||
|
||||
The PAT lives in two places: 1Password (read by Pulumi at runtime) and `~/.config/caddy/gandi-token` on indri (read by Caddy at startup). The 1Password edit only updates the first.
|
||||
|
||||
```bash
|
||||
mise run provision-indri --tags caddy
|
||||
```
|
||||
|
||||
This re-fetches the PAT from 1Password, writes it to indri, and restarts Caddy. Caddy will renew any due certificates within minutes.
|
||||
|
||||
### 4. Verify
|
||||
|
||||
```bash
|
||||
mise run dns-preview
|
||||
```
|
||||
|
||||
A successful preview confirms Pulumi can use the PAT.
|
||||
|
||||
```bash
|
||||
ssh indri 'tail -50 ~/Library/Logs/mcquack.caddy.err.log' \
|
||||
| grep -E "obtained|renew|error"
|
||||
```
|
||||
|
||||
Expect to see no `LiveDNS returned a 403` lines, and either no renewal activity (if no certs were due) or `certificate obtained successfully`.
|
||||
|
||||
### 5. Delete the old PAT in Gandi
|
||||
|
||||
Return to the Gandi PAT console and delete the previous token.
|
||||
|
||||
### 6. Clean up orphan ACME records
|
||||
|
||||
Each successful Caddy renewal leaves orphan `_acme-challenge.ops` TXT records in the zone (a bug in `libdns/gandi` v1.1.0 — see the script docstring). Cadence aligns with rotation:
|
||||
|
||||
```bash
|
||||
mise run dns-acme-cleanup --dry-run
|
||||
mise run dns-acme-cleanup
|
||||
```
|
||||
|
||||
## Debugging
|
||||
|
||||
### Caddy logs `LiveDNS returned a 403`
|
||||
|
||||
The PAT is invalid (expired, revoked, or insufficient scope). **Gandi returns 403 — not 401 — for an expired PAT**, which can read as a permissions issue. The most common cause is plain expiry. Rotate.
|
||||
|
||||
### `mise run dns-preview` returns 403
|
||||
|
||||
Same root cause — Pulumi and Caddy share this PAT.
|
||||
|
||||
### After a fresh PAT, Caddy still fails
|
||||
|
||||
Check that the value on indri matches 1Password:
|
||||
|
||||
```bash
|
||||
diff <(ssh indri 'cat ~/.config/caddy/gandi-token') \
|
||||
<(op read 'op://blumeops/gandi - blumeops/pat')
|
||||
```
|
||||
|
||||
If they differ, `mise run provision-indri --tags caddy` was skipped or failed.
|
||||
|
||||
Confirm the new PAT works against Gandi directly:
|
||||
|
||||
```bash
|
||||
curl -s -o /dev/null -w "HTTP %{http_code}\n" \
|
||||
-H "Authorization: Bearer $(op read 'op://blumeops/gandi - blumeops/pat')" \
|
||||
https://api.gandi.net/v5/livedns/domains/eblu.me
|
||||
```
|
||||
|
||||
`200` = healthy. `403` = scope or expiry. `401` = malformed token.
|
||||
|
||||
## Related
|
||||
|
||||
- [[gandi]] — Gandi reference card
|
||||
- [[manage-eblu-me-dns]] — DNS records workflow (separate operation, same PAT)
|
||||
- [[caddy]] — Reverse proxy that uses the PAT for TLS
|
||||
- [[mise-tasks]] — `dns-acme-cleanup`, `provision-indri`, `dns-preview` reference
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
title: Gandi
|
||||
modified: 2026-04-09
|
||||
last-reviewed: 2026-04-09
|
||||
modified: 2026-04-27
|
||||
last-reviewed: 2026-04-27
|
||||
tags:
|
||||
- infrastructure
|
||||
- networking
|
||||
|
|
@ -20,12 +20,11 @@ DNS hosting provider for the `eblu.me` domain, managed via Pulumi IaC.
|
|||
| **Provider** | Gandi LiveDNS |
|
||||
| **IaC** | `pulumi/gandi/` |
|
||||
| **Stack** | `eblu-me` |
|
||||
| **PAT** | `op://blumeops/gandi - blumeops/pat` |
|
||||
|
||||
## What It Does
|
||||
|
||||
Gandi hosts the DNS records that make `*.ops.eblu.me` resolve to [[indri]]'s Tailscale IP (`indri.tail8d86e.ts.net`). Since Tailscale IPs are not publicly routable, this gives services real DNS names while keeping them private to the tailnet.
|
||||
|
||||
The target IP is resolved dynamically from `indri.tail8d86e.ts.net` at deploy time, so if indri's Tailscale IP changes, re-running the deployment is sufficient.
|
||||
Gandi hosts the DNS records that make `*.ops.eblu.me` resolve to [[indri]]'s Tailscale IP. Since Tailscale IPs are not publicly routable, this gives services real DNS names while keeping them private to the tailnet. The target IP is resolved dynamically from `indri.tail8d86e.ts.net` at deploy time.
|
||||
|
||||
## DNS Records
|
||||
|
||||
|
|
@ -46,38 +45,25 @@ Both records point to [[indri]], which runs [[caddy]] as the reverse proxy for a
|
|||
| `cv.eblu.me` | CNAME | `blumeops-proxy.fly.dev` | 300s |
|
||||
| `forge.eblu.me` | CNAME | `blumeops-proxy.fly.dev` | 300s |
|
||||
|
||||
Public CNAMEs point to [[flyio-proxy]] on Fly.io. See [[expose-service-publicly]] for adding new public services.
|
||||
|
||||
See [[routing]] for the full service URL map.
|
||||
|
||||
## Pulumi Configuration
|
||||
|
||||
The Pulumi program lives in `pulumi/gandi/`:
|
||||
|
||||
- `__main__.py` - Creates A and CNAME records via `pulumiverse_gandi`
|
||||
- `Pulumi.eblu-me.yaml` - Stack config (domain, subdomain)
|
||||
|
||||
Stack config values:
|
||||
|
||||
| Key | Value |
|
||||
|-----|-------|
|
||||
| `blumeops-dns:domain` | `eblu.me` |
|
||||
| `blumeops-dns:subdomain` | `ops` |
|
||||
|
||||
A break-glass override is available via the `BLUMEOPS_REVERSE_PROXY_IP` environment variable, which bypasses dynamic IP resolution.
|
||||
Public CNAMEs point to [[flyio-proxy]] on Fly.io. See [[expose-service-publicly]] for adding new public services. See [[routing]] for the full service URL map.
|
||||
|
||||
## TLS Integration
|
||||
|
||||
[[caddy]] uses Gandi's API separately (via `GANDI_BEARER_TOKEN`) for ACME DNS-01 challenges to obtain a wildcard Let's Encrypt certificate for `*.ops.eblu.me`. This is a different credential from the Pulumi PAT.
|
||||
[[caddy]] uses this same Gandi PAT for ACME DNS-01 challenges to obtain a wildcard Let's Encrypt certificate for `*.ops.eblu.me`. Caddy reads the PAT from `~/.config/caddy/gandi-token` on [[indri]], populated by ansible from 1Password.
|
||||
|
||||
## Authentication
|
||||
|
||||
Gandi requires a Personal Access Token (PAT) for API access. PATs have a maximum lifetime of 90 days (currently set to 30). See [[gandi-operations]] for deployment and PAT cycling instructions.
|
||||
One Gandi Personal Access Token, shared by Pulumi and Caddy. Gandi caps PATs at 90 days; rotate every 60 days via [[rotate-gandi-pat]].
|
||||
|
||||
## ACME Challenge Cleanup
|
||||
|
||||
Caddy's renewal flow leaves `_acme-challenge.ops` TXT orphans in the zone — a value-comparison bug in `libdns/gandi` v1.1.0 makes the cleanup phase a no-op. Run `mise run dns-acme-cleanup` periodically (alongside PAT rotation works well).
|
||||
|
||||
## Related
|
||||
|
||||
- [[gandi-operations]] - PAT cycling and deployment how-to
|
||||
- [[routing]] - Service URLs and routing architecture
|
||||
- [[caddy]] - Reverse proxy using Gandi for TLS
|
||||
- [[tailscale]] - Tailnet networking
|
||||
- [[indri]] - Server hosting Caddy (DNS target)
|
||||
- [[manage-eblu-me-dns]] — Add/change DNS records via Pulumi
|
||||
- [[rotate-gandi-pat]] — Rotate the shared Gandi PAT
|
||||
- [[routing]] — Service URLs and routing architecture
|
||||
- [[caddy]] — Reverse proxy using this PAT for TLS
|
||||
- [[tailscale]] — Tailnet networking
|
||||
- [[indri]] — Server hosting Caddy (DNS target)
|
||||
|
|
|
|||
|
|
@ -39,6 +39,7 @@ Run `mise tasks --sort name` for the live list with descriptions.
|
|||
| `fly-shutoff` | Emergency shutoff: stop all Fly.io proxy machines |
|
||||
| `dns-preview` | Preview DNS changes with [[pulumi]] |
|
||||
| `dns-up` | Apply DNS changes with [[pulumi]] |
|
||||
| `dns-acme-cleanup` | Delete orphaned `_acme-challenge.ops` TXT records (libdns/gandi v1.1.0 workaround) |
|
||||
| `tailnet-preview` | Preview Tailscale ACL changes with [[pulumi]] |
|
||||
| `tailnet-up` | Apply Tailscale ACL changes with [[pulumi]] |
|
||||
|
||||
|
|
|
|||
|
|
@ -49,7 +49,8 @@ mise run tailnet-up # Apply ACL/tag changes
|
|||
|
||||
## Related
|
||||
|
||||
- [[gandi-operations]] — DNS PAT rotation and Pulumi workflow
|
||||
- [[manage-eblu-me-dns]] — DNS records workflow
|
||||
- [[rotate-gandi-pat]] — Rotate the Gandi PAT
|
||||
- [[update-tailscale-acls]] — ACL editing and Pulumi workflow
|
||||
- [[gandi]] — DNS hosting
|
||||
- [[tailscale]] — Tailnet configuration
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue