Add tag:k8s-operator and update Phase 1 plan

ACL changes:
- Added tag:k8s-operator for the Tailscale K8s Operator
- Made tag:k8s-operator an owner of tag:k8s so the operator can
  assign that tag to resources it creates

Phase 1 plan updates:
- Added Kubernetes Tags Overview section explaining all three tags
- Expanded OAuth client creation instructions
- Added 1Password storage instructions
- Added verification and rollback sections

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

plans/k8s-migration/P1_k8s_infrastructure.md

@@ -8,15 +8,31 @@
 
 ---
 
+## Kubernetes Tags Overview
+
+Phase 1 introduces three Tailscale tags for Kubernetes:
+
+| Tag | Purpose | Applied To |
+|-----|---------|------------|
+| `tag:k8s-api` | Controls access to the K8s API server | indri (Phase 0.14) |
+| `tag:k8s-operator` | Identifies the Tailscale K8s Operator | OAuth client for operator |
+| `tag:k8s` | Default tag for operator-managed resources | Proxies, services, ingresses created by operator |
+
+**Ownership chain**: `tag:k8s-operator` must own `tag:k8s` so the operator can assign that tag to devices it creates.
+
+---
+
 ## Steps
 
 ### 1. Update Pulumi ACLs for k8s workloads
 
-Add `tag:k8s` to `pulumi/policy.hujson` - this tag is for k8s workloads that need to access other services (e.g., Woodpecker CI pushing to registry).
+Add the operator and workload tags to `pulumi/policy.hujson`.
 
 **Changes to tagOwners:**
 ```hujson
-"tag:k8s": ["autogroup:admin", "tag:blumeops"],
+// Tailscale K8s Operator tags (Phase 1)
+"tag:k8s-operator": ["autogroup:admin", "tag:blumeops"],
+"tag:k8s": ["autogroup:admin", "tag:blumeops", "tag:k8s-operator"],
 ```
 
 **Add grant for k8s→registry access:**
@@ -37,30 +53,61 @@ Add `tag:k8s` to `pulumi/policy.hujson` - this tag is for k8s workloads that nee
 },
 ```
 
+**Deploy:**
 ```bash
-mise run tailnet-preview && mise run tailnet-up
+mise run tailnet-preview   # Review changes
+mise run tailnet-up        # Apply changes
 ```
 
 ---
 
-### 2. Create Tailscale OAuth client
+### 2. Create Tailscale OAuth client (MANUAL)
 
-- Scopes: Devices Core, Auth Keys, Services write
-- Tag: `tag:k8s-operator`
-- Store in 1Password
+Go to https://login.tailscale.com/admin/settings/oauth and create an OAuth client:
+
+**Configuration:**
+- **Description**: `k8s-operator`
+- **Tags**: `tag:k8s-operator`
+- **Scopes**:
+  - Devices: Core (Read & Write)
+  - Auth Keys: Read & Write
+  - Services: Write
+
+**After creation:**
+1. Copy the Client ID and Client Secret
+2. Store in 1Password (vault: `vg6xf6vvfmoh5hqjjhlhbeoaie`)
+   - Item name: `Tailscale K8s Operator OAuth`
+   - Fields: `client-id`, `client-secret`
 
 ---
 
 ### 3. Deploy Tailscale Kubernetes Operator
 
 ```bash
+# Add helm repo
 helm repo add tailscale https://pkgs.tailscale.com/helmcharts
+helm repo update
+
+# Get credentials from 1Password
+CLIENT_ID=$(op --vault vg6xf6vvfmoh5hqjjhlhbeoaie item get "Tailscale K8s Operator OAuth" --fields client-id --reveal)
+CLIENT_SECRET=$(op --vault vg6xf6vvfmoh5hqjjhlhbeoaie item get "Tailscale K8s Operator OAuth" --fields client-secret --reveal)
+
+# Install operator
 helm install tailscale-operator tailscale/tailscale-operator \
   --namespace tailscale-system --create-namespace \
   --set oauth.clientId=$CLIENT_ID \
   --set oauth.clientSecret=$CLIENT_SECRET
 ```
 
+**Verification:**
+```bash
+kubectl get pods -n tailscale-system
+# Expected: tailscale-operator pod Running
+
+# Check operator logs
+kubectl logs -n tailscale-system -l app.kubernetes.io/name=tailscale-operator
+```
+
 ---
 
 ### 4. Deploy CloudNativePG operator
@@ -69,11 +116,24 @@ helm install tailscale-operator tailscale/tailscale-operator \
 kubectl apply -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.24/releases/cnpg-1.24.0.yaml
 ```
 
+**Verification:**
+```bash
+kubectl get pods -n cnpg-system
+# Expected: cnpg-controller-manager pod Running
+```
+
 ---
 
 ### 5. Create PostgreSQL cluster
 
+Create namespace and cluster manifest:
+
+```bash
+kubectl create namespace databases
+```
+
 ```yaml
+# ansible/k8s/databases/blumeops-pg.yaml
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
@@ -88,26 +148,78 @@ spec:
     enablePodMonitor: true
 ```
 
+```bash
+kubectl apply -f ansible/k8s/databases/blumeops-pg.yaml
+```
+
+**Verification:**
+```bash
+kubectl get cluster -n databases
+# Expected: blumeops-pg with STATUS "Cluster in healthy state"
+
+kubectl get pods -n databases
+# Expected: blumeops-pg-1 pod Running
+```
+
 ---
 
 ### 6. Update Alloy config
 
-- Add kubernetes_sd_configs for k8s metrics
-- Scrape operator metrics
+Add kubernetes_sd_configs for k8s metrics scraping.
+
+**Files to modify:**
+- `ansible/roles/alloy/templates/config.alloy.j2`
+
+**Changes:**
+- Add scrape config for CloudNativePG metrics
+- Add scrape config for Tailscale operator metrics (if exposed)
 
 ---
 
 ## New Files
 
-- `ansible/k8s/operators/` - Operator manifests
-- `ansible/k8s/databases/` - PostgreSQL cluster
+| File | Purpose |
+|------|---------|
+| `ansible/k8s/operators/` | Operator deployment notes/scripts |
+| `ansible/k8s/databases/blumeops-pg.yaml` | PostgreSQL cluster manifest |
 
 ---
 
-## Verification
+## Verification Checklist
 
 ```bash
+# 1. Tailscale operator running
 kubectl get pods -n tailscale-system
+
+# 2. CloudNativePG operator running
 kubectl get pods -n cnpg-system
+
+# 3. PostgreSQL cluster healthy
 kubectl get cluster -n databases
+kubectl get pods -n databases
+
+# 4. Test database connection (from indri)
+kubectl -n databases get secret blumeops-pg-app -o jsonpath='{.data.uri}' | base64 -d
+# Use the URI to connect via psql
+```
+
+---
+
+## Rollback
+
+```bash
+# Remove PostgreSQL cluster
+kubectl delete cluster -n databases blumeops-pg
+kubectl delete namespace databases
+
+# Remove CloudNativePG operator
+kubectl delete -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.24/releases/cnpg-1.24.0.yaml
+
+# Remove Tailscale operator
+helm uninstall tailscale-operator -n tailscale-system
+kubectl delete namespace tailscale-system
+
+# Revert ACL changes
+git checkout pulumi/policy.hujson
+mise run tailnet-up
 ```

pulumi/policy.hujson

@@ -111,7 +111,8 @@
 		"tag:feed": ["autogroup:admin", "tag:blumeops"],
 		"tag:registry": ["autogroup:admin", "tag:blumeops"],
 		"tag:k8s-api": ["autogroup:admin", "tag:blumeops"],
-		"tag:k8s": ["autogroup:admin", "tag:blumeops"],
+		"tag:k8s-operator": ["autogroup:admin", "tag:blumeops"],
+		"tag:k8s": ["autogroup:admin", "tag:blumeops", "tag:k8s-operator"],
 	},
 
 	// ============== ACL Tests ==============