Add Phase 3 tutorials with audience targeting (#94)

## Summary
- Create tutorials directory structure with index page
- Add 5 main tutorials targeting different audiences:
  - **what-is-blumeops** (Reader, AI) - High-level orientation
  - **exploring-the-docs** (All) - Navigation guide
  - **ai-assistance-guide** (AI, Owner) - Context for AI-assisted operations
  - **contributing** (Contributor) - First contribution workflow
  - **replicating-blumeops** (Replicator) - Overview for building similar setup
- Add 4 replication sub-tutorials:
  - tailscale-setup, kubernetes-bootstrap, argocd-config, observability-stack
- Update README.md to mark Phase 3 complete
- Add changelog fragment

Each tutorial explicitly identifies its target audiences and links to reference material rather than re-explaining concepts.

## Deployment and Testing
- [x] All pre-commit hooks pass (doc-links validates wiki links)
- [ ] Build docs via workflow to verify rendering
- [ ] Review content for accuracy

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/94
This commit is contained in:
Erich Blume 2026-02-03 18:51:57 -08:00
commit 7ebac4aef6
20 changed files with 1864 additions and 10 deletions

View file

@ -0,0 +1,221 @@
---
title: argocd-config
tags:
- tutorials
- replication
- argocd
---
# Configuring ArgoCD
> **Audiences:** Replicator
This tutorial walks through installing ArgoCD and establishing GitOps-driven deployments for your homelab.
## What is GitOps?
GitOps means your git repository is the source of truth for infrastructure:
- Infrastructure state is defined in git
- Changes happen through commits and pull requests
- A controller (ArgoCD) syncs git state to the cluster
- Drift is detected and can be corrected automatically
For BlumeOps specifics, see [[argocd|ArgoCD Reference]].
## Step 1: Install ArgoCD
```bash
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
```
Wait for pods to be ready:
```bash
kubectl -n argocd get pods -w
```
## Step 2: Access the UI
### Get the Initial Password
```bash
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
```
### Expose the Service
For Tailscale access:
```bash
tailscale serve --bg --https 8443 https+insecure://localhost:$(kubectl -n argocd get svc argocd-server -o jsonpath='{.spec.ports[?(@.name=="https")].port}')
```
Or create a Tailscale Ingress in Kubernetes (see [[tailscale-operator]]).
Access at `https://your-server.tailnet.ts.net:8443`
### Install the CLI
BlumeOps includes `argocd` in its Brewfile (`brew bundle`), or install it however you prefer.
Login:
```bash
argocd login your-server.tailnet.ts.net:8443
```
## Step 3: Connect Your Git Repository
Create a repository credential:
```bash
# For SSH
argocd repo add git@github.com:you/your-repo.git \
--ssh-private-key-path ~/.ssh/id_ed25519
# For HTTPS
argocd repo add https://github.com/you/your-repo.git \
--username you \
--password your-token
```
## Step 4: Create Your First Application
Create a directory in your repo:
```
your-repo/
└── apps/
└── hello-world/
├── deployment.yaml
└── service.yaml
```
With a simple deployment:
```yaml
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world
spec:
replicas: 1
selector:
matchLabels:
app: hello-world
template:
metadata:
labels:
app: hello-world
spec:
containers:
- name: hello
image: nginx:alpine
ports:
- containerPort: 80
```
Create the ArgoCD Application:
```bash
argocd app create hello-world \
--repo git@github.com:you/your-repo.git \
--path apps/hello-world \
--dest-server https://kubernetes.default.svc \
--dest-namespace default
```
## Step 5: Sync and Verify
```bash
# See what will be deployed
argocd app diff hello-world
# Deploy it
argocd app sync hello-world
# Check status
argocd app get hello-world
```
The pods should now be running:
```bash
kubectl get pods -l app=hello-world
```
## Step 6: App of Apps Pattern
For managing multiple applications, use the "app of apps" pattern:
```
your-repo/
├── argocd/
│ ├── apps/ # Application definitions
│ │ ├── hello-world.yaml
│ │ └── another-app.yaml
│ └── manifests/ # Actual Kubernetes manifests
│ ├── hello-world/
│ └── another-app/
```
Create a root Application that manages other Applications:
```yaml
# argocd/apps/apps.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: apps
namespace: argocd
spec:
project: default
source:
repoURL: git@github.com:you/your-repo.git
targetRevision: main
path: argocd/apps
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
```
Now adding a new application is just creating a YAML file.
## Step 7: Configure Sync Policies
| Policy | When to Use |
|--------|-------------|
| Manual sync | Production, explicit control |
| Auto sync | Development, or trusted workloads |
| Auto prune | Remove resources deleted from git |
| Self heal | Revert manual kubectl changes |
BlumeOps uses manual sync for workloads, auto sync only for the `apps` Application itself.
## What You Now Have
- GitOps workflow for deployments
- UI for visualizing application state
- Automatic drift detection
- Declarative application management
## Next Steps
- [[tutorials/replication/observability-stack | Build observability]] - Monitor your deployments
- Add more applications to your repo
- Set up notifications for sync failures
## BluemeOps Specifics
BlumeOps' ArgoCD configuration includes:
- SSH connection to [[forgejo]] git server
- Manual sync policy for all workloads
- Separate manifests and apps directories
See [[argocd|ArgoCD Reference]] and [[apps|Apps Reference]] for full details.
## Troubleshooting
| Problem | Solution |
|---------|----------|
| Sync failed | Check `argocd app get <app>` for error details |
| Can't connect to repo | Verify credentials, check SSH key permissions |
| Resources not appearing | Ensure path in Application matches repo structure |
| Out of sync but no diff | Check for ignored differences in app config |

View file

@ -0,0 +1,113 @@
---
title: core-services
tags:
- tutorials
- replication
- forgejo
---
# Core Services Setup
> **Audiences:** Replicator
This tutorial walks through setting up the foundational services that your GitOps infrastructure depends on: a git forge and optionally a container registry.
## Why Core Services First?
Before Kubernetes and ArgoCD, you need somewhere to store your infrastructure definitions. [[forgejo]] provides:
- Git hosting for your GitOps repository
- CI/CD workflows for building and deploying
- A web interface for code review and PRs
The [[zot]] container registry is optional but useful for hosting your own container images.
## Step 1: Install Forgejo
Forgejo runs directly on your server (not in Kubernetes) because Kubernetes depends on it.
### Using Ansible (BlumeOps Approach)
BlumeOps manages Forgejo via an Ansible role. See [[reference/ansible/roles | Ansible Roles]].
### Manual Installation
1. Download Forgejo from [forgejo.org](https://forgejo.org/download/)
2. Create a service user and directories
3. Configure with `app.ini`
4. Set up as a system service
Key configuration points:
- SSH on a non-standard port (e.g., 2222) to avoid conflicts
- Database (SQLite works fine for personal use)
- Domain and URL settings for your Tailscale hostname
## Step 2: Configure SSH Access
Set up SSH for git operations:
```bash
# Add your SSH key to Forgejo via the web UI
# Then test access:
ssh -T git@your-server.tailnet.ts.net -p 2222
```
## Step 3: Create Your GitOps Repository
1. Create a new repository in Forgejo (e.g., `infrastructure` or `homelab`)
2. Initialize the standard directory structure:
```
your-repo/
├── ansible/ # Host configuration
│ ├── playbooks/
│ └── roles/
├── argocd/ # Kubernetes GitOps
│ ├── apps/ # ArgoCD Applications
│ └── manifests/ # K8s manifests per service
├── pulumi/ # IaC for Tailscale, DNS
└── docs/ # Documentation
```
3. Push your initial commit
## Step 4: Set Up CI/CD Runner (Optional)
Forgejo Actions runs workflows defined in `.forgejo/workflows/`. To use it:
1. Register a runner on your server
2. Configure runner to access your build tools
3. Create workflow files for builds and deployments
BlumeOps runs a Forgejo runner in Kubernetes - see [[forgejo]] for details.
## Step 5: Container Registry (Optional)
If you'll build custom container images, set up [[zot]]:
1. Install Zot on your server
2. Configure authentication
3. Set up TLS (via Caddy or similar)
For getting started, you can skip this and use public registries.
## What You Now Have
- Git hosting for infrastructure code
- SSH access for git operations
- Foundation for CI/CD workflows
- Optionally, a private container registry
## Next Steps
- [[tutorials/replication/kubernetes-bootstrap | Bootstrap Kubernetes]] - Now that you have a git repo, set up your cluster
- Configure Forgejo webhooks for ArgoCD (after ArgoCD is running)
## BlumeOps Specifics
BlumeOps' Forgejo setup includes:
- Ansible role for installation and updates
- SSH on port 2222, proxied via Caddy
- Integration with ArgoCD via deploy keys
- Forgejo runner in Kubernetes for CI/CD
See [[forgejo]] and [[zot]] for full details.

View file

@ -0,0 +1,170 @@
---
title: kubernetes-bootstrap
tags:
- tutorials
- replication
- kubernetes
---
# Bootstrapping Kubernetes
> **Audiences:** Replicator
This tutorial walks through setting up a Kubernetes cluster for your homelab, making it accessible via Tailscale.
## Choosing a Distribution
For homelab use, lightweight distributions work well:
| Distribution | Best For | BlumeOps Uses |
|--------------|----------|---------------|
| **Minikube** | Single-node, macOS | Yes |
| **k3s** | Single-node, Linux | - |
| **kind** | Local development | - |
| **kubeadm** | Multi-node clusters | - |
This tutorial uses minikube, but principles apply broadly.
For BlumeOps specifics, see [[cluster|Cluster Reference]].
## Step 1: Install Minikube
### macOS
```bash
brew install minikube
```
### Linux
```bash
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube
```
## Step 2: Create the Cluster
```bash
minikube start \
--driver=docker \
--cpus=4 \
--memory=8g \
--disk-size=100g \
--apiserver-names=k8s.your-tailnet.ts.net,$(hostname) \
--listen-address=0.0.0.0
```
Key flags:
- `--apiserver-names` - Include your Tailscale hostname for remote access
- `--listen-address=0.0.0.0` - Allow connections from other machines
## Step 3: Verify the Cluster
```bash
kubectl get nodes
# Should show your node as Ready
kubectl get pods -A
# Should show system pods running
```
## Step 4: Expose via Tailscale
To access the cluster from other Tailscale devices, expose the API server:
### Option A: Tailscale Serve (Simple)
```bash
tailscale serve --bg --tcp 6443 tcp://localhost:$(minikube ip --format '{{.Port}}')
```
### Option B: Tailscale Kubernetes Operator (Advanced)
For production-like setup, install the Tailscale operator which manages ingress automatically.
BlumeOps uses TCP passthrough via Caddy - see [[routing|Routing Reference]].
## Step 5: Configure Remote Access
On your workstation, add a context for the remote cluster:
```bash
# Copy the CA cert from the server
scp server:~/.minikube/ca.crt ~/.kube/minikube-ca.crt
# Add the cluster
kubectl config set-cluster minikube-remote \
--server=https://k8s.your-tailnet.ts.net:6443 \
--certificate-authority=$HOME/.kube/minikube-ca.crt
# Add credentials (copy from server's ~/.kube/config)
kubectl config set-credentials minikube-remote \
--client-certificate=... \
--client-key=...
# Add context
kubectl config set-context minikube-remote \
--cluster=minikube-remote \
--user=minikube-remote
# Test
kubectl --context=minikube-remote get nodes
```
## Step 6: Storage Configuration
For persistent workloads, configure storage:
### Local Path Provisioner (Simple)
```bash
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
```
### NFS for Shared Storage
If you have a NAS:
```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-share
spec:
capacity:
storage: 1Ti
accessModes:
- ReadWriteMany
nfs:
server: nas.your-tailnet.ts.net
path: /volume1/k8s
```
## What You Now Have
- A Kubernetes cluster running on your server
- Remote access via Tailscale
- Storage for persistent workloads
## Next Steps
- [[tutorials/replication/argocd-config | Configure ArgoCD]] - GitOps deployments
- Install essential addons (ingress controller, cert-manager)
## BluemeOps Specifics
BlumeOps' cluster configuration includes:
- Tailscale operator for automatic ingress
- NFS mounts from [[sifaka]] for media storage
- CloudNativePG for PostgreSQL databases
See [[cluster|Cluster Reference]] and [[apps|Apps Reference]] for full details.
## Troubleshooting
| Problem | Solution |
|---------|----------|
| Can't connect remotely | Check `--apiserver-names` includes Tailscale hostname |
| Pods stuck pending | Check storage class is available |
| Connection refused | Verify `--listen-address=0.0.0.0` was set |
| Certificate errors | Ensure CA cert matches server's |

View file

@ -0,0 +1,231 @@
---
title: observability-stack
tags:
- tutorials
- replication
- observability
---
# Building the Observability Stack
> **Audiences:** Replicator
This tutorial walks through deploying metrics, logs, and dashboards for your homelab - because you can't fix what you can't see.
## The Stack
A complete observability solution has three pillars:
| Component | Purpose | BlumeOps Uses |
|-----------|---------|---------------|
| **Metrics** | Numeric measurements over time | [[prometheus]] |
| **Logs** | Text output from applications | [[loki]] |
| **Dashboards** | Visualization and alerting | [[grafana]] |
| **Collection** | Gathering and forwarding data | [[alloy]] |
For BlumeOps specifics, see [[observability|Observability Reference]].
## Step 1: Create Monitoring Namespace
```bash
kubectl create namespace monitoring
```
## Step 2: Deploy Prometheus
Prometheus collects and stores metrics.
### Using Helm
```bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/prometheus \
--namespace monitoring \
--set server.persistentVolume.size=10Gi
```
### Or via ArgoCD
Create an Application pointing to a values file in your repo:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: prometheus
namespace: argocd
spec:
project: default
source:
repoURL: https://prometheus-community.github.io/helm-charts
chart: prometheus
targetRevision: 25.0.0
helm:
values: |
server:
persistentVolume:
size: 10Gi
destination:
server: https://kubernetes.default.svc
namespace: monitoring
```
### Verify
```bash
kubectl -n monitoring get pods -l app.kubernetes.io/name=prometheus
```
## Step 3: Deploy Loki
Loki aggregates logs (like Prometheus but for logs).
```bash
helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki-stack \
--namespace monitoring \
--set loki.persistence.enabled=true \
--set loki.persistence.size=10Gi
```
This also installs Promtail for log collection from pods.
## Step 4: Deploy Grafana
Grafana provides dashboards and visualization.
```bash
helm install grafana grafana/grafana \
--namespace monitoring \
--set persistence.enabled=true \
--set persistence.size=1Gi \
--set adminPassword=admin # Change this!
```
### Configure Data Sources
After installation, add data sources in Grafana UI or via ConfigMap:
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-datasources
namespace: monitoring
labels:
grafana_datasource: "1"
data:
datasources.yaml: |
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
url: http://prometheus-server.monitoring.svc:80
isDefault: true
- name: Loki
type: loki
url: http://loki.monitoring.svc:3100
```
## Step 5: Access Grafana
Expose via Tailscale:
```bash
kubectl -n monitoring port-forward svc/grafana 3000:80 &
tailscale serve --bg --https 3000 http://localhost:3000
```
Or create an Ingress.
Default credentials: `admin` / (password you set or retrieve from secret)
## Step 6: Add Dashboards
Import community dashboards from [grafana.com/grafana/dashboards](https://grafana.com/grafana/dashboards/):
| Dashboard | ID | Shows |
|-----------|-----|-------|
| Node Exporter Full | 1860 | Host metrics |
| Kubernetes Cluster | 7249 | Cluster overview |
| Loki Logs | 13639 | Log exploration |
In Grafana: Dashboards > Import > Enter ID
## Step 7: Deploy Alloy (Optional)
Grafana Alloy is a unified collector that replaces multiple agents (Promtail, node_exporter, etc.).
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: alloy
namespace: argocd
spec:
project: default
source:
repoURL: https://grafana.github.io/helm-charts
chart: alloy
targetRevision: 0.1.0
helm:
values: |
alloy:
configMap:
content: |
// Alloy configuration here
destination:
server: https://kubernetes.default.svc
namespace: monitoring
```
BluemeOps uses Alloy on both [[indri]] (for host metrics, via [[reference/ansible/roles | Ansible role]]) and in the [[cluster]] (for pod logs and service probes).
## What You Now Have
- Metrics collection and storage (Prometheus)
- Log aggregation (Loki)
- Dashboards and visualization (Grafana)
- Foundation for alerting
## Adding Alerts
Configure alerting rules in Prometheus:
```yaml
groups:
- name: example
rules:
- alert: HighMemoryUsage
expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1
for: 5m
labels:
severity: warning
annotations:
summary: "High memory usage detected"
```
And notification channels in Grafana (email, Slack, PagerDuty, etc.).
## Next Steps
- Create custom dashboards for your services
- Set up alerting for critical conditions
- Add service-specific metrics exporters
## BluemeOps Specifics
BlumeOps' observability setup includes:
- Prometheus scraping all services via annotations
- Loki collecting logs from all pods and [[indri]] services
- Custom dashboards for [[jellyfin]], [[teslamate]], and cluster health
- [[alloy]] running on both host and in-cluster
See [[observability|Observability Reference]] for full details.
## Troubleshooting
| Problem | Solution |
|---------|----------|
| No metrics appearing | Check Prometheus targets (`/targets` endpoint) |
| No logs in Loki | Verify Promtail/Alloy is collecting (`/ready` endpoint) |
| Dashboard shows no data | Check data source configuration and time range |
| High storage usage | Adjust retention settings in Prometheus/Loki |

View file

@ -0,0 +1,134 @@
---
title: tailscale-setup
tags:
- tutorials
- replication
- tailscale
---
# Setting Up Tailscale
> **Audiences:** Replicator
This tutorial walks through establishing a Tailscale mesh network as the foundation for your homelab infrastructure.
## Why Tailscale?
Tailscale solves several problems at once:
- **Secure connectivity** - WireGuard-encrypted traffic between all devices
- **No port forwarding** - Devices connect directly through NATs and firewalls
- **MagicDNS** - Human-readable names like `server.tailnet.ts.net`
- **ACLs** - Fine-grained access control between devices
For BlumeOps context, see [[tailscale|Tailscale Reference]].
## Step 1: Create Your Tailnet
1. Sign up at [tailscale.com](https://tailscale.com)
2. Choose your identity provider (Google, Microsoft, GitHub, etc.)
3. Note your tailnet name (e.g., `yourname.ts.net`)
## Step 2: Install on Your Devices
### macOS
```bash
brew install tailscale
sudo tailscaled &
tailscale up
```
### Linux
```bash
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
```
### Other Platforms
See [Tailscale Downloads](https://tailscale.com/download) for iOS, Android, Windows, etc.
## Step 3: Verify Connectivity
After installing on two devices:
```bash
tailscale status
# Shows all connected devices
ping <other-device>.yourname.ts.net
# Should work immediately
```
## Step 4: Configure ACLs
Default Tailscale allows all-to-all connectivity. For a homelab, you'll want restrictions.
Create `policy.hujson` (or use the web admin):
```json
{
"groups": {
"group:admin": ["your-email@example.com"]
},
"tagOwners": {
"tag:homelab": ["group:admin"]
},
"acls": [
// Admins can access everything
{"action": "accept", "src": ["group:admin"], "dst": ["*:*"]},
// Homelab servers can reach NAS
{"action": "accept", "src": ["tag:homelab"], "dst": ["tag:nas:*"]}
]
}
```
BlumeOps manages ACLs via Pulumi - see [[tailscale|Tailscale Reference]] for the actual configuration.
## Step 5: Enable MagicDNS
In the Tailscale admin console:
1. Go to DNS settings
2. Enable MagicDNS
3. Optionally add a search domain
Now `ssh server` works instead of `ssh 100.x.y.z`.
## Step 6: Tag Your Devices
Tags enable role-based access control:
```bash
# On your server
sudo tailscale up --advertise-tags=tag:homelab
```
Tags must be defined in ACLs before use.
## What You Now Have
- Encrypted mesh network between all your devices
- DNS names for each device
- Foundation for exposing services securely
## Next Steps
With networking established:
- [[tutorials/replication/kubernetes-bootstrap | Bootstrap Kubernetes]] - Your cluster will join the tailnet
- Set up your server and storage devices
## BlumeOps Specifics
BluemeOps' Tailscale configuration includes:
- Multiple device tags (`homelab`, `nas`, `registry`, `k8s-api`)
- Group-based access for family members
- SSH access rules with authentication requirements
See [[tailscale|Tailscale Reference]] for full details.
## Troubleshooting
| Problem | Solution |
|---------|----------|
| Device won't connect | Check firewall allows UDP 41641 |
| Can't reach other devices | Verify ACLs don't block traffic |
| DNS not resolving | Enable MagicDNS in admin console |
| Tags not applying | Ensure tags defined in ACL policy |