Add External Secrets Operator with 1Password Connect (#66) (#66)

## Summary
- Add 1Password Connect server for secrets automation API
- Add External Secrets Operator (ESO) to sync secrets from 1Password to K8s
- Add ClusterSecretStore connecting ESO to 1Password Connect
- Convert devpi secret to ExternalSecret as proof of concept

## Architecture
```
1Password Cloud → 1Password Connect (k8s) → ESO → Native K8s Secrets
```

## Deployment and Testing
- [ ] Mirror Helm charts to forge (connect-helm-charts, external-secrets) - DONE
- [ ] Create 1Password Connect credentials (`op connect server create`)
- [ ] Store credentials in 1Password item "1Password Connect"
- [ ] Bootstrap secret: `op inject -i argocd/manifests/1password-connect/secret-credentials.yaml.tpl | kubectl apply -f -`
- [ ] Deploy 1password-connect: `argocd app sync 1password-connect`
- [ ] Deploy external-secrets: `argocd app sync external-secrets`
- [ ] Deploy external-secrets-config: `argocd app sync external-secrets-config`
- [ ] Test devpi ExternalSecret: `argocd app sync devpi`
- [ ] Verify secret synced: `kubectl get externalsecret -n devpi`

## Future Work
After PoC validated, migrate remaining 12 secret templates to ExternalSecrets:
- databases (3), tailscale-operator (1), grafana-config (2), teslamate (2)
- forgejo-runner (1), argocd (1), immich (1), 1password-connect (1 - self-bootstrap)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/66

argocd/apps/1password-connect.yaml

@@ -0,0 +1,38 @@
+# 1Password Connect - Secrets Automation Server
+# Provides REST API access to 1Password vault items for External Secrets Operator
+#
+# Chart mirrored from https://github.com/1Password/connect-helm-charts
+#
+# Prerequisites (one-time setup):
+# 1. Create Connect server: op connect server create blumeops --vaults blumeops
+# 2. Create token: op connect token create blumeops --server <server-id> --vault blumeops
+# 3. Store credentials in 1Password item "1Password Connect" in blumeops vault
+# 4. Bootstrap secret:
+#    kubectl --context=minikube-indri create namespace 1password
+#    op inject -i argocd/manifests/1password-connect/secret-credentials.yaml.tpl | \
+#      kubectl --context=minikube-indri apply -f -
+#
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: 1password-connect
+  namespace: argocd
+spec:
+  project: default
+  sources:
+    - repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/connect-helm-charts.git
+      targetRevision: connect-2.2.1
+      path: charts/connect
+      helm:
+        releaseName: onepassword-connect
+        valueFiles:
+          - $values/argocd/manifests/1password-connect/values.yaml
+    - repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git
+      targetRevision: main
+      ref: values
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: 1password
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true

argocd/apps/external-secrets-config.yaml

@@ -0,0 +1,26 @@
+# External Secrets Configuration - ClusterSecretStore for 1Password
+#
+# Deploys the ClusterSecretStore that connects ESO to 1Password Connect.
+# Must be synced AFTER external-secrets operator is running.
+#
+# Prerequisites:
+# - 1password-connect is deployed and healthy
+# - external-secrets operator is deployed and CRDs are installed
+#
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: external-secrets-config
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git
+    targetRevision: main
+    path: argocd/manifests/external-secrets
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: external-secrets
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true

argocd/apps/external-secrets-crds.yaml

@@ -0,0 +1,28 @@
+# External Secrets Operator CRDs
+#
+# CRDs are installed separately because:
+# 1. They need ServerSideApply due to large annotation sizes
+# 2. The Helm chart's CRDs are auto-generated during packaging (not in raw git)
+# 3. CRDs should exist before the operator starts
+#
+# Must be synced BEFORE external-secrets operator app.
+#
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: external-secrets-crds
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/external-secrets.git
+    targetRevision: helm-chart-1.3.1
+    path: config/crds/bases
+    directory:
+      exclude: 'kustomization.yaml'
+  destination:
+    server: https://kubernetes.default.svc
+  syncPolicy:
+    syncOptions:
+      - ServerSideApply=true
+      - CreateNamespace=false

argocd/apps/external-secrets.yaml

@@ -0,0 +1,33 @@
+# External Secrets Operator - Kubernetes secret sync from external providers
+# Syncs secrets from 1Password Connect to native Kubernetes Secrets
+#
+# Chart mirrored from https://github.com/external-secrets/external-secrets
+#
+# Prerequisites:
+# - 1password-connect must be deployed and healthy
+#
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: external-secrets
+  namespace: argocd
+spec:
+  project: default
+  sources:
+    - repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/external-secrets.git
+      targetRevision: helm-chart-1.3.1
+      path: deploy/charts/external-secrets
+      helm:
+        releaseName: external-secrets
+        valueFiles:
+          - $values/argocd/manifests/external-secrets/values.yaml
+    - repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git
+      targetRevision: main
+      ref: values
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: external-secrets
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true
+      - ServerSideApply=true

argocd/manifests/1password-connect/README.md

@@ -0,0 +1,90 @@
+# 1Password Connect
+
+1Password Connect provides REST API access to 1Password vault items for External Secrets Operator.
+
+## Architecture
+
+```
+1Password Cloud
+      |
+      v
+1Password Connect (this service)
+      |
+      v
+External Secrets Operator
+      |
+      v
+Native Kubernetes Secrets
+```
+
+## Prerequisites (One-Time Setup)
+
+Run these steps on the workstation (gilbert) before deploying:
+
+### 1. Create Connect Server Credentials
+
+```bash
+# This creates the credentials file and outputs a server ID
+op connect server create blumeops --vaults blumeops
+
+# Save the 1password-credentials.json file contents
+```
+
+### 2. Create Access Token
+
+```bash
+# Replace <server-id> with the ID from step 1
+op connect token create blumeops --server <server-id> --vault blumeops
+
+# Save the token
+```
+
+### 3. Store Credentials in 1Password
+
+Create a new item "1Password Connect" in the blumeops vault with:
+- `credentials-file` field: Paste the contents of `1password-credentials.json` (NOT base64 encoded)
+- `token` field: Paste the access token
+
+### 4. Create Bootstrap Secret
+
+```bash
+kubectl --context=minikube-indri create namespace 1password
+op inject -i argocd/manifests/1password-connect/secret-credentials.yaml.tpl | \
+  kubectl --context=minikube-indri apply -f -
+```
+
+## Deployment
+
+```bash
+argocd app sync apps
+argocd app sync 1password-connect
+```
+
+## Verification
+
+```bash
+# Check pods are running
+kubectl --context=minikube-indri -n 1password get pods
+
+# Check logs
+kubectl --context=minikube-indri -n 1password logs -l app=onepassword-connect
+
+# Test API health (port-forward first)
+kubectl --context=minikube-indri -n 1password port-forward svc/onepassword-connect 8080:8080 &
+curl http://localhost:8080/health
+```
+
+## Troubleshooting
+
+### Pods not starting
+- Check the bootstrap secret exists: `kubectl --context=minikube-indri -n 1password get secret op-credentials`
+- Verify credentials format in 1Password item
+
+### API returning 401
+- Check the token secret: `kubectl --context=minikube-indri -n 1password get secret onepassword-token`
+- Verify the token has access to the blumeops vault
+
+## Related
+
+- [1Password Connect Documentation](https://developer.1password.com/docs/connect/)
+- [External Secrets Operator](../external-secrets/README.md)

argocd/manifests/1password-connect/secret-credentials.yaml.tpl

@@ -0,0 +1,42 @@
+# 1Password Connect bootstrap credentials
+#
+# This template is processed ONCE manually to bootstrap the system.
+# After External Secrets is operational, this could be converted to an
+# ExternalSecret for self-management (chicken-and-egg bootstrap).
+#
+# Prerequisites:
+# 1. Create Connect server: op connect server create blumeops --vaults blumeops
+# 2. Create token: op connect token create blumeops --server <server-id> --vault blumeops
+# 3. Create 1Password item "1Password Connect" in blumeops vault with:
+#    - credentials-file: contents of 1password-credentials.json (raw JSON)
+#    - credentials-base64: base64-encoded contents of 1password-credentials.json
+#    - token: the access token
+#
+#    To add credentials-base64 to existing item:
+#      CREDS=$(op item get "1Password Connect" --vault blumeops --format json | \
+#        jq -r '.fields[] | select(.label == "credentials-file") | .value' | base64)
+#      op item edit "1Password Connect" --vault blumeops "credentials-base64=$CREDS"
+#
+# Usage:
+#   kubectl --context=minikube-indri create namespace 1password
+#   op inject -i argocd/manifests/1password-connect/secret-credentials.yaml.tpl | \
+#     kubectl --context=minikube-indri apply -f -
+#
+apiVersion: v1
+kind: Secret
+metadata:
+  name: op-credentials
+  namespace: 1password
+type: Opaque
+stringData:
+  # OP_SESSION env var expects base64-encoded credentials
+  1password-credentials.json: "{{ op://blumeops/1Password Connect/credentials-base64 }}"
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: onepassword-token
+  namespace: 1password
+type: Opaque
+stringData:
+  token: "{{ op://blumeops/1Password Connect/token }}"

argocd/manifests/1password-connect/values.yaml

@@ -0,0 +1,33 @@
+# 1Password Connect Helm values for blumeops
+# Chart: https://github.com/1Password/connect-helm-charts
+#
+# The credentials are bootstrapped manually via secret-credentials.yaml.tpl
+# before deploying this chart.
+
+connect:
+  # Use pre-created credentials secret (from bootstrap)
+  credentialsKey: 1password-credentials.json
+  credentialsName: op-credentials
+
+  # Resource limits for minikube
+  api:
+    resources:
+      requests:
+        memory: "64Mi"
+        cpu: "50m"
+      limits:
+        memory: "256Mi"
+        cpu: "200m"
+
+  sync:
+    resources:
+      requests:
+        memory: "64Mi"
+        cpu: "50m"
+      limits:
+        memory: "256Mi"
+        cpu: "200m"
+
+# We don't use the 1Password Operator (using External Secrets instead)
+operator:
+  create: false

argocd/manifests/devpi/external-secret.yaml

@@ -0,0 +1,25 @@
+# ExternalSecret for devpi root password
+#
+# Replaces the manual op inject workflow from secret-root.yaml.tpl
+#
+# 1Password item: "devpi" in blumeops vault
+# Field: "root password"
+#
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: devpi-root
+  namespace: devpi
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: onepassword-blumeops
+  target:
+    name: devpi-root
+    creationPolicy: Owner
+  data:
+  - secretKey: password
+    remoteRef:
+      key: devpi
+      property: root password

argocd/manifests/devpi/kustomization.yaml

@@ -7,3 +7,4 @@ resources:
   - statefulset.yaml
   - service.yaml
   - ingress-tailscale.yaml
+  - external-secret.yaml

argocd/manifests/external-secrets/README.md

@@ -0,0 +1,83 @@
+# External Secrets Operator
+
+External Secrets Operator (ESO) syncs secrets from 1Password Connect to native Kubernetes Secrets.
+
+## Architecture
+
+- **ClusterSecretStore** (`onepassword-blumeops`): Cluster-wide access to 1Password via Connect
+- **ExternalSecret** (per-namespace): Defines which secrets to sync from 1Password
+
+## Prerequisites
+
+1Password Connect must be deployed and healthy before syncing ESO.
+
+## Deployment
+
+```bash
+argocd app sync external-secrets
+```
+
+## Verification
+
+```bash
+# Check operator pods
+kubectl --context=minikube-indri -n external-secrets get pods
+
+# Check ClusterSecretStore status
+kubectl --context=minikube-indri get clustersecretstore onepassword-blumeops
+
+# Check all ExternalSecrets across namespaces
+kubectl --context=minikube-indri get externalsecret -A
+```
+
+## Creating ExternalSecrets
+
+To sync a secret from 1Password, create an ExternalSecret in the target namespace:
+
+```yaml
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: my-secret
+  namespace: my-namespace
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: onepassword-blumeops
+  target:
+    name: my-secret           # Name of K8s Secret to create
+    creationPolicy: Owner     # ESO owns and manages the Secret
+  data:
+  - secretKey: password       # Key in the K8s Secret
+    remoteRef:
+      key: My 1Password Item  # Title of item in 1Password
+      property: password      # Field label in 1Password item
+```
+
+### Finding 1Password Item Details
+
+```bash
+# List items in blumeops vault
+op item list --vault blumeops
+
+# Get field names for an item
+op item get <item-id> --vault blumeops --format json | jq -r '.fields[] | .label'
+```
+
+## Troubleshooting
+
+### ClusterSecretStore not ready
+- Check 1Password Connect is running: `kubectl --context=minikube-indri -n 1password get pods`
+- Verify token secret exists: `kubectl --context=minikube-indri -n 1password get secret onepassword-token`
+
+### ExternalSecret not syncing
+- Check the ExternalSecret status: `kubectl --context=minikube-indri describe externalsecret <name> -n <namespace>`
+- Verify the 1Password item title and field names match exactly
+- Check ESO controller logs: `kubectl --context=minikube-indri -n external-secrets logs -l app.kubernetes.io/name=external-secrets`
+
+## Related
+
+- [External Secrets Operator Docs](https://external-secrets.io/)
+- [1Password Provider](https://external-secrets.io/latest/provider/1password-automation/)
+- [1Password Connect](../1password-connect/README.md)

argocd/manifests/external-secrets/cluster-secret-store.yaml

@@ -0,0 +1,21 @@
+# ClusterSecretStore for 1Password Connect
+#
+# Provides cluster-wide access to the blumeops vault via 1Password Connect.
+# ExternalSecret resources in any namespace can reference this store.
+#
+apiVersion: external-secrets.io/v1
+kind: ClusterSecretStore
+metadata:
+  name: onepassword-blumeops
+spec:
+  provider:
+    onepassword:
+      connectHost: http://onepassword-connect.1password.svc.cluster.local:8080
+      vaults:
+        blumeops: 1
+      auth:
+        secretRef:
+          connectTokenSecretRef:
+            name: onepassword-token
+            namespace: 1password
+            key: token

argocd/manifests/external-secrets/kustomization.yaml

@@ -0,0 +1,5 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - cluster-secret-store.yaml

argocd/manifests/external-secrets/values.yaml

@@ -0,0 +1,31 @@
+# External Secrets Operator Helm values for blumeops
+# Chart: https://github.com/external-secrets/external-secrets
+
+installCRDs: true
+
+# Resource limits for minikube
+resources:
+  requests:
+    memory: "64Mi"
+    cpu: "50m"
+  limits:
+    memory: "256Mi"
+    cpu: "200m"
+
+webhook:
+  resources:
+    requests:
+      memory: "32Mi"
+      cpu: "25m"
+    limits:
+      memory: "128Mi"
+      cpu: "100m"
+
+certController:
+  resources:
+    requests:
+      memory: "32Mi"
+      cpu: "25m"
+    limits:
+      memory: "128Mi"
+      cpu: "100m"