Add authenticated GitHub PAT for Forgejo mirror sync (#269)
## Summary - **mirror-create**: Auto-includes GitHub PAT from 1Password for authenticated upstream fetches at mirror creation time - **mirror-update-pats**: New mise task that SSHes into indri and rewrites the git remote URL in every GitHub mirror's bare repo config to embed the PAT. Idempotent, supports `--dry-run` - **app.ini.j2**: Explicit `[mirror]` section with `DEFAULT_INTERVAL = 8h` and `MIN_INTERVAL = 10m` (bakes in the defaults for visibility) - **manage-forgejo-mirrors**: New how-to doc covering mirror creation, PAT storage, the `mirror-update-pats` task, and the full 20-day PAT rotation procedure ## Context GitHub tightened unauthenticated rate limits for git clone/fetch in May 2025. With 23 GitHub mirrors syncing every 8 hours, authenticated fetches avoid throttling. The PAT is stored in 1Password (`Forgejo Secrets` → `github-mirror-pat`) and has been applied to all existing mirrors. ## Deployment and Testing - [x] `mirror-update-pats` dry-run verified (23 mirrors detected) - [x] `mirror-update-pats` applied to all 23 GitHub mirrors on indri - [x] Idempotency confirmed (re-run shows 0 updated, 23 skipped) - [ ] Provision indri with `--tags forgejo` to apply `[mirror]` config - [ ] Trigger a manual mirror sync and verify success in Forgejo UI Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/269
This commit is contained in:
parent
23dc79058e
commit
84338c32c2
7 changed files with 234 additions and 2 deletions
147
docs/how-to/configuration/manage-forgejo-mirrors.md
Normal file
147
docs/how-to/configuration/manage-forgejo-mirrors.md
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
---
|
||||
title: Manage Forgejo Mirrors
|
||||
modified: 2026-02-25
|
||||
tags:
|
||||
- how-to
|
||||
- forgejo
|
||||
- git
|
||||
---
|
||||
|
||||
# Manage Forgejo Mirrors
|
||||
|
||||
How Forgejo upstream mirrors work, how to create new mirrors, and how to rotate the GitHub PAT used for authenticated sync.
|
||||
|
||||
## Overview
|
||||
|
||||
BlumeOps mirrors upstream repositories (mostly from GitHub) into the `mirrors/` organization on forge. These are **pull mirrors** — Forgejo periodically fetches from the upstream URL and updates the local copy. ArgoCD and other consumers then read from forge instead of hitting upstream directly.
|
||||
|
||||
### Why Authenticate
|
||||
|
||||
GitHub rate-limits unauthenticated git fetch/clone over HTTPS. As of May 2025, these limits were tightened significantly. All mirrors should use an authenticated `clone_addr` (via a GitHub fine-grained PAT) to avoid throttling.
|
||||
|
||||
The GitHub PAT is stored in 1Password:
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Vault** | blumeops (`vg6xf6vvfmoh5hqjjhlhbeoaie`) |
|
||||
| **Item** | Forgejo Secrets (`w3663ffnvkewbftncqxtcpeavy`) |
|
||||
| **Field** | `github-mirror-pat` |
|
||||
| **op ref** | `op://blumeops/w3663ffnvkewbftncqxtcpeavy/github-mirror-pat` |
|
||||
|
||||
### Sync Interval
|
||||
|
||||
Mirror sync frequency is controlled by two settings in `app.ini`:
|
||||
|
||||
| Setting | Section | Default | Purpose |
|
||||
|---------|---------|---------|---------|
|
||||
| `DEFAULT_INTERVAL` | `[mirror]` | `8h` | How often each mirror checks for upstream changes |
|
||||
| `MIN_INTERVAL` | `[mirror]` | `10m` | Floor for per-repo interval overrides |
|
||||
| `SCHEDULE` | `[cron.update_mirrors]` | `@every 10m` | How often the cron scans for due mirrors |
|
||||
|
||||
With 10–30 mirrors at 8h intervals, expect ~1–4 fetches/hour — well within any rate limit when authenticated.
|
||||
|
||||
The explicit configuration lives in `ansible/roles/forgejo/templates/app.ini.j2`.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Access to 1Password blumeops vault
|
||||
- Forgejo admin account on forge.ops.eblu.me
|
||||
- `op` CLI authenticated
|
||||
- For new mirrors: `mise run mirror-create`
|
||||
|
||||
## Create a New Mirror
|
||||
|
||||
```fish
|
||||
mise run mirror-create https://github.com/org/repo.git
|
||||
```
|
||||
|
||||
Options:
|
||||
- `--name <name>` — override the repo name on forge (default: derived from URL)
|
||||
- `--description <text>` — set the repo description
|
||||
- `--dry-run` — preview without creating
|
||||
|
||||
For GitHub upstreams, the script automatically includes the GitHub PAT from 1Password so the mirror authenticates from the start. Non-GitHub upstreams (Codeberg, etc.) are created without upstream auth.
|
||||
|
||||
## Update All Mirror PATs
|
||||
|
||||
To update the GitHub PAT on all existing mirrors at once:
|
||||
|
||||
```fish
|
||||
mise run mirror-update-pats
|
||||
```
|
||||
|
||||
This SSHs into indri and rewrites the git remote URL in each mirror's bare repository to embed `eblume:<PAT>@` in the upstream URL. It reads the PAT from 1Password and skips mirrors that already have the current PAT.
|
||||
|
||||
Use `--dry-run` to preview:
|
||||
|
||||
```fish
|
||||
mise run mirror-update-pats --dry-run
|
||||
```
|
||||
|
||||
### How It Works
|
||||
|
||||
Forgejo stores mirror credentials directly in the bare repo's git config on disk (not in the database). The `remote_address` in SQLite stays as the clean URL; the actual fetch URL in `<repo>.git/config` contains the embedded credentials:
|
||||
|
||||
```
|
||||
# Unauthenticated
|
||||
url = https://github.com/org/repo.git
|
||||
|
||||
# Authenticated
|
||||
url = https://eblume:<pat>@github.com/org/repo.git
|
||||
```
|
||||
|
||||
The Forgejo API has no endpoint for updating pull mirror credentials, so the script updates the git config directly via SSH.
|
||||
|
||||
## Rotate the GitHub PAT
|
||||
|
||||
The GitHub fine-grained PAT has a 30-day expiry. Set a recurring reminder (every 20 days) to rotate it before it expires.
|
||||
|
||||
### 1. Create a New PAT on GitHub
|
||||
|
||||
Go to [GitHub fine-grained token settings](https://github.com/settings/personal-access-tokens/new) and create a new token:
|
||||
|
||||
- **Name:** `forgejo-mirror-sync` (or similar, include the date for tracking)
|
||||
- **Expiration:** 30 days
|
||||
- **Repository access:** Public repositories (read-only)
|
||||
- **Permissions:** None required — fine-grained PATs automatically include read-only access to all public repos
|
||||
|
||||
Copy the new PAT to your clipboard.
|
||||
|
||||
### 2. Update 1Password
|
||||
|
||||
With the new PAT on your clipboard:
|
||||
|
||||
```fish
|
||||
op item edit w3663ffnvkewbftncqxtcpeavy github-mirror-pat=(pbpaste) --vault blumeops
|
||||
```
|
||||
|
||||
Verify the update:
|
||||
|
||||
```fish
|
||||
op read "op://blumeops/w3663ffnvkewbftncqxtcpeavy/github-mirror-pat" | head -c 12
|
||||
# Should print the first 12 chars of the new PAT (github_pat_...)
|
||||
```
|
||||
|
||||
### 3. Push the PAT to All Mirrors
|
||||
|
||||
```fish
|
||||
mise run mirror-update-pats
|
||||
```
|
||||
|
||||
### 4. Delete the Old PAT on GitHub
|
||||
|
||||
Return to [GitHub token settings](https://github.com/settings/tokens?type=beta) and delete the previous token.
|
||||
|
||||
### 5. Verify
|
||||
|
||||
Trigger a manual sync on one mirror to confirm the new PAT works:
|
||||
|
||||
1. Go to any mirror repo on forge (e.g., `mirrors/cloudnative-pg`)
|
||||
2. Click the sync button (circular arrows icon) next to the mirror status
|
||||
3. Confirm the sync completes without errors
|
||||
|
||||
## Related
|
||||
|
||||
- [[forgejo]] — Forgejo service reference
|
||||
- [[upstream-fork-strategy]] — Stacked-branch forking for repos with local modifications
|
||||
- [[gandi-operations]] — Similar PAT rotation workflow for Gandi DNS
|
||||
|
|
@ -26,6 +26,7 @@ Task-oriented instructions for common BlumeOps operations. These guides assume y
|
|||
| [[gandi-operations]] | Manage DNS records and cycle the Gandi API token |
|
||||
| [[use-pypi-proxy]] | Configure pip and publish packages to devpi |
|
||||
| [[expose-service-publicly]] | Expose a service to the public internet via Fly.io + Tailscale |
|
||||
| [[manage-forgejo-mirrors]] | Create mirrors, update PATs, and rotate GitHub credentials |
|
||||
| [[update-documentation]] | Publish docs via build-blumeops workflow |
|
||||
| [[update-tooling-dependencies]] | Monthly update cycle for pre-commit, Fly, mise, and workflow deps |
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue