eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions

Add Phase 3 tutorials with audience targeting

Browse source 
Create tutorials directory with learning-oriented content:
- what-is-blumeops: High-level orientation (Reader, AI)
- exploring-the-docs: Navigation guide (All audiences)
- ai-assistance-guide: Context for AI assistance (AI, Owner)
- contributing: First contribution workflow (Contributor)
- replicating-blumeops: Overview for replicators

Add replication sub-tutorials:
- tailscale-setup: Networking foundation
- kubernetes-bootstrap: Cluster setup
- argocd-config: GitOps configuration
- observability-stack: Metrics, logs, dashboards

Each tutorial explicitly identifies target audiences and links
heavily to reference material rather than re-explaining.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Erich Blume 2026-02-03 17:51:32 -08:00
commit 50046f42f8
12 changed files with 1361 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

21
docs/README.md
View file

@ -79,13 +79,22 @@ Information-oriented technical descriptions. Built first so other docs can link
**Reference URL:** https://docs.ops.eblu.me/reference/
### Phase 3: Tutorials
Learning-oriented content for getting started.
### Phase 3: Tutorials (Complete)
Learning-oriented content for getting started. Each tutorial explicitly identifies its target audiences.
- [ ] Create `tutorials/` directory
- [ ] "Getting Started with BlumeOps" - What this is and how to explore it
- [ ] "Setting Up a Similar Environment" - For replicators
- [ ] "Your First Contribution" - For potential contributors
- [x] Create `tutorials/` directory with index
- [x] "What is BlumeOps?" - High-level orientation (Reader, AI)
- [x] "Exploring the Docs" - How to navigate documentation (All)
- [x] "AI Assistance Guide" - Context for AI-assisted operations (AI, Owner)
- [x] "Contributing" - Your first contribution (Contributor)
- [x] "Replicating BlumeOps" - Overview for building similar setup (Replicator)
- [x] Replication sub-tutorials:
- [x] Tailscale Setup
- [x] Kubernetes Bootstrap
- [x] ArgoCD Config
- [x] Observability Stack
**Tutorials URL:** https://docs.ops.eblu.me/tutorials/
### Phase 4: How-to Guides
Task-oriented instructions for specific operations.

1
docs/changelog.d/phase3-tutorials.doc.md Normal file
View file

@ -0,0 +1 @@
Add Phase 3 tutorials: "What is BlumeOps?", "Exploring the Docs", "AI Assistance Guide", "Contributing", and "Replicating BlumeOps" with sub-tutorials for Tailscale, Kubernetes, ArgoCD, and Observability. Each tutorial explicitly identifies its target audiences.

116
docs/tutorials/ai-assistance-guide.md Normal file
View file

@ -0,0 +1,116 @@
---
title: ai-assistance-guide
tags:
- tutorials
- ai
---
# AI Assistance Guide
> **Audiences:** AI, Owner
This guide provides context for AI agents (like Claude Code) assisting with BlumeOps operations, and helps Erich understand how to work effectively with AI assistance.
## Critical Rules
These are non-negotiable for AI agents working in this repo:
1. **Always use `--context=minikube-indri` with kubectl** - Work contexts exist that must never be touched
2. **Run `mise run zk-docs` at session start** - Review current infrastructure state
3. **Never commit secrets** - The repo is public at github.com/eblume/blumeops
4. **Wait for user review before deploying** - Create PRs, don't auto-deploy
5. **Never merge PRs without explicit request** - The user merges after review
Full rules are in the repo's `CLAUDE.md`.
## Workflow Conventions
### Feature Branches
All work happens on feature branches:
```bash
git checkout main && git pull
git checkout -b feature/descriptive-name
# ... make changes ...
git commit -m "Description"
```
### Pull Requests
Use the forge's `tea` CLI:
```bash
tea pr create --title "Title" --description "$(cat <<'EOF'
## Summary
- Change 1
- Change 2
## Deployment and Testing
- [ ] Test step
EOF
)"
```
### Changelog Fragments
Add a fragment for user-visible changes:
```bash
echo "Description" > docs/changelog.d/branch-name.feature.md
```
Types: `feature`, `bugfix`, `infra`, `doc`, `misc`
## Service Locations
Understanding where services run helps target changes correctly:
| Location | Services | Management |
|----------|----------|------------|
| [[indri]] (native) | Forgejo, Zot, Jellyfin, Caddy | Ansible |
| [[cluster | Kubernetes]] | Everything else | ArgoCD |
## Common Operations
### Kubernetes Service Changes
1. Modify manifests in `argocd/manifests/<service>/`
2. Preview with `argocd app diff <service>`
3. Deploy with `argocd app sync <service>`
### Indri Service Changes
1. Modify ansible role in `ansible/roles/<service>/`
2. Dry run: `mise run provision-indri -- --check --diff --tags <service>`
3. Deploy: `mise run provision-indri -- --tags <service>`
### Health Checks
After changes:
```bash
mise run indri-services-check
```
## Reference Navigation
For AI agents building context:
- [[reference/index|Reference Index]] - Entry point for technical details
- [[hosts|Host Inventory]] - What hardware exists
- [[apps|ArgoCD Apps]] - What's deployed in Kubernetes
- [[routing|Routing]] - How services are exposed
## Credential Access
Credentials live in 1Password. Never retrieve them directly - use existing patterns:
- Ansible `pre_tasks` gather secrets at playbook start
- [[external-secrets|External Secrets]] syncs to Kubernetes
- Scripts use `op` CLI with user biometric prompts
## Common Pitfalls
| Pitfall | Correct Approach |
|---------|------------------|
| Missing kubectl context | Always add `--context=minikube-indri` |
| Deploying without review | Create PR first, wait for user approval |
| Re-explaining reference material | Link to reference cards instead |
| Committing to main | Use feature branches |
| Guessing at credentials | Ask user or check 1Password patterns |

149
docs/tutorials/contributing.md Normal file
View file

@ -0,0 +1,149 @@
---
title: contributing
tags:
- tutorials
- contributing
---
# Your First Contribution
> **Audiences:** Contributor
This tutorial walks through making your first contribution to BluemeOps - from understanding the codebase to submitting a pull request.
## Prerequisites
Before contributing, you'll need:
- Access to the [[tailscale|Tailscale]] network (request from Erich)
- SSH key added to [[forgejo|Forgejo]] (https://forge.ops.eblu.me)
- `tea` CLI installed for PR creation
## Understanding the Codebase
BlumeOps manages infrastructure through three main systems:
| System | Directory | What It Manages |
|--------|-----------|-----------------|
| **Ansible** | `ansible/` | Services running directly on [[indri]] |
| **ArgoCD** | `argocd/` | Kubernetes services in the [[cluster]] |
| **Pulumi** | `pulumi/` | [[tailscale|Tailscale]] ACLs and DNS |
Most contributions involve either Ansible roles or ArgoCD manifests.
## The Contribution Workflow
### 1. Clone and Branch
```bash
git clone ssh://git@forge.ops.eblu.me:2222/eblume/blumeops.git
cd blumeops
git checkout -b feature/your-change-name
```
### 2. Make Your Changes
Depending on what you're changing:
**For Kubernetes services:**
- Edit manifests in `argocd/manifests/<service>/`
- Or create new Application in `argocd/apps/`
**For Indri services:**
- Edit or create roles in `ansible/roles/`
- Update `ansible/playbooks/indri.yml` if adding a role
**For documentation:**
- Edit files in `docs/`
- Add changelog fragment (see below)
### 3. Add a Changelog Fragment
For user-visible changes:
```bash
echo "Description of your change" > docs/changelog.d/your-branch.feature.md
```
Fragment types:
- `.feature.md` - New functionality
- `.bugfix.md` - Bug fixes
- `.infra.md` - Infrastructure changes
- `.doc.md` - Documentation
- `.misc.md` - Other
### 4. Test Your Changes
**Before pushing, always test:**
For Kubernetes changes:
```bash
# Preview what will change
argocd app diff <service>
```
For Ansible changes:
```bash
# Dry run
mise run provision-indri -- --check --diff --tags <role>
```
### 5. Commit and Push
```bash
git add <files>
git commit -m "Brief description of change"
git push -u origin feature/your-change-name
```
### 6. Create a Pull Request
```bash
tea pr create --title "Your PR Title" --description "$(cat <<'EOF'
## Summary
- What you changed
- Why you changed it
## Deployment and Testing
- [ ] Tested locally / dry run
- [ ] Ready for ArgoCD sync / Ansible apply
EOF
)"
```
### 7. Wait for Review
Erich will review your PR and may leave comments. Check for feedback:
```bash
mise run pr-comments <pr_number>
```
Address each comment, then Erich will:
1. Approve the changes
2. Deploy them (you don't need to do this)
3. Merge the PR
## Example: Adding a Homepage Link
A simple first contribution - adding a service to the Homepage dashboard (go.ops.eblu.me):
1. Find the service's Ingress in `argocd/manifests/<service>/`
2. Add homepage annotations:
```yaml
annotations:
gethomepage.dev/enabled: "true"
gethomepage.dev/name: "Service Name"
gethomepage.dev/group: "Apps"
gethomepage.dev/icon: "service.png"
```
3. Create PR and wait for sync
## Getting Help
- Browse [[reference/index|Reference]] for technical details
- Check `CLAUDE.md` in the repo for rules and conventions
- Ask Erich directly (he's friendly)
## Related
- [[ai-assistance-guide]] - If using AI assistance for contributions
- [[replicating-blumeops]] - If you want to build your own instead

78
docs/tutorials/exploring-the-docs.md Normal file
View file

@ -0,0 +1,78 @@
---
title: exploring-the-docs
tags:
- tutorials
- getting-started
---
# Exploring the Documentation
> **Audiences:** All (Owner, AI, Reader, Contributor, Replicator)
This guide explains how the BlumeOps documentation is organized and how to find what you need.
## Documentation Structure
The docs follow the [Diataxis](https://diataxis.fr/) framework:
| Section | Purpose | When to Use |
|---------|---------|-------------|
| **[[tutorials/index | Tutorials]]** | Learning-oriented | "I'm new and want to understand" |
| **[[reference/index | Reference]]** | Information-oriented | "I need specific technical details" |
| **How-to** (planned) | Task-oriented | "I need to do X" |
| **Explanation** (planned) | Understanding-oriented | "I want to understand why" |
## Quick Paths by Audience
### For Erich (Owner)
You probably want quick access to operational details:
- [[reference/index|Reference]] has service URLs, commands, and config locations
- The `zk-docs` mise task still works for legacy zettelkasten access
- [[ai-assistance-guide]] explains how to work effectively with Claude
### For Claude/AI Agents
Context for effective assistance:
- Read [[ai-assistance-guide]] for operational conventions
- [[reference/index|Reference]] has the technical specifics you'll need
- The repo's `CLAUDE.md` has critical rules (especially the kubectl context requirement)
### For External Readers
Understanding what this is:
- [[what-is-blumeops]] gives the high-level overview
- [[reference/index|Reference]] shows what's actually running
- Browse service pages to see specific implementations
### For Contributors
Getting started with changes:
- [[contributing]] walks through the workflow
- [[reference/index|Reference]] tells you where things live
- The `CLAUDE.md` in the repo root has contribution rules
### For Replicators
Building your own:
- [[replicating-blumeops]] provides the overview
- The `replication/` tutorials go deep on components
- Reference pages show specific configuration choices
## Using Wiki Links
Documentation uses `[[wiki-links]]` for cross-references:
- `[[service-name]]` links to a reference page
- `[[folder/page]]` links to nested pages
- `[[page|Display Text]]` customizes the link text
When reading on the web (docs.ops.eblu.me), these render as clickable links. The backlinks panel shows what references each page.
## Legacy Content
The `docs/zk/` directory contains zettelkasten cards from before the restructuring. These are read-only reference - new content goes in the structured sections. The cards will eventually be migrated or archived.
To view legacy cards:
```bash
mise run zk-docs
```

47
docs/tutorials/index.md Normal file
View file

@ -0,0 +1,47 @@
---
title: tutorials
tags:
- tutorials
---
# Tutorials
Learning-oriented guides for understanding and working with BlumeOps.
## Audience Guide
Each tutorial indicates which audiences it serves:
| Icon | Audience | Description |
|------|----------|-------------|
| **Owner** | Erich | Quick recall and operational refreshers |
| **AI** | Claude/AI agents | Context for AI-assisted operations |
| **Reader** | External readers | Understanding what BlumeOps is |
| **Contributor** | Operators/contributors | Helping with BlumeOps development |
| **Replicator** | Replicators | Building your own similar setup |
## Getting Started
| Tutorial | Audiences | Description |
|----------|-----------|-------------|
| [[what-is-blumeops]] | Reader, AI | High-level orientation to the project |
| [[exploring-the-docs]] | All | How to navigate and use this documentation |
| [[ai-assistance-guide]] | AI, Owner | Context for effective AI-assisted operations |
## Contributing
| Tutorial | Audiences | Description |
|----------|-----------|-------------|
| [[contributing]] | Contributor | Your first contribution to BlumeOps |
## Replication
For those building their own homelab GitOps setup.
| Tutorial | Audiences | Description |
|----------|-----------|-------------|
| [[replicating-blumeops]] | Replicator | Overview: building a similar environment |
| [[tutorials/replication/tailscale-setup | Tailscale Setup]] | Replicator | Setting up Tailscale networking |
| [[tutorials/replication/kubernetes-bootstrap | Kubernetes Bootstrap]] | Replicator | Bootstrapping a Kubernetes cluster |
| [[tutorials/replication/argocd-config | ArgoCD Config]] | Replicator | Configuring GitOps with ArgoCD |
| [[tutorials/replication/observability-stack | Observability Stack]] | Replicator | Metrics, logs, and dashboards |

128
docs/tutorials/replicating-blumeops.md Normal file
View file

@ -0,0 +1,128 @@
---
title: replicating-blumeops
tags:
- tutorials
- replication
---
# Replicating BlumeOps
> **Audiences:** Replicator
This tutorial provides a roadmap for building your own homelab GitOps environment inspired by BluemeOps. It links to detailed component tutorials for each major piece.
## What You'll Build
By following this guide, you'll have:
- A secure mesh network connecting your devices
- A Kubernetes cluster for running containerized services
- GitOps-driven deployments via ArgoCD
- Observability with metrics, logs, and dashboards
- Backup and disaster recovery capabilities
## Hardware Requirements
BluemeOps runs on modest hardware. At minimum:
| Component | BlumeOps Uses | Minimum Alternative |
|-----------|---------------|---------------------|
| **Server** | Mac Mini M1 | Any machine with 16GB RAM |
| **NAS** | Synology DS920+ | USB drive or second machine |
| **Workstation** | MacBook Air M4 | Whatever you use daily |
You can start with a single machine and add storage later.
## The Journey
### Phase 1: Networking Foundation
Before deploying services, establish secure connectivity.
**[[tutorials/replication/tailscale-setup|Setting Up Tailscale]]**
- Create a tailnet and connect your devices
- Configure ACLs for service access
- Set up MagicDNS for convenient naming
This replaces: traditional VPNs, port forwarding, dynamic DNS
### Phase 2: Kubernetes Cluster
A cluster for running containerized workloads.
**[[tutorials/replication/kubernetes-bootstrap|Bootstrapping Kubernetes]]**
- Install minikube (or k3s, kind, etc.)
- Configure persistent storage
- Expose the API securely via Tailscale
BlumeOps uses minikube for simplicity, but the patterns apply to any distribution.
### Phase 3: GitOps with ArgoCD
Declarative, git-driven deployments.
**[[tutorials/replication/argocd-config|Configuring ArgoCD]]**
- Install ArgoCD in your cluster
- Connect to your git repository
- Deploy your first application
- Set up the app-of-apps pattern
This is the heart of GitOps - changes in git automatically sync to your cluster.
### Phase 4: Observability Stack
Know what's happening in your infrastructure.
**[[tutorials/replication/observability-stack|Building the Observability Stack]]**
- Deploy Prometheus for metrics
- Deploy Loki for logs
- Deploy Grafana for dashboards
- Configure Alloy for collection
Without observability, you're flying blind.
### Phase 5: Your First Services
With the foundation in place, deploy actual workloads. BluemeOps runs:
- [[miniflux]] - RSS reader
- [[jellyfin]] - Media server
- [[immich]] - Photo management
- [[navidrome]] - Music streaming
Pick what matters to you. Each service follows similar patterns:
1. Create Kubernetes manifests
2. Create ArgoCD Application
3. Configure ingress routing
4. Sync and verify
### Phase 6: Backups and Resilience
Protect your data.
- Set up [[borgmatic]] for backup automation
- Configure NAS as backup target
- Test restore procedures
- Document disaster recovery
## Alternative Approaches
BluemeOps makes specific choices that may not suit everyone:
| BlumeOps Choice | Alternative |
|-----------------|-------------|
| macOS server | Linux server (more common) |
| Minikube | k3s, kind, or managed K8s |
| Tailscale | WireGuard, Nebula |
| ArgoCD | Flux, manual kubectl |
| Ansible | NixOS, Docker Compose |
The principles (GitOps, IaC, observability) matter more than specific tools.
## Getting Started
Begin with [[tutorials/replication/tailscale-setup]] - networking is the foundation everything else builds on.
## Related
- [[what-is-blumeops]] - Understand what you're replicating
- [[reference/index]] - See BlumeOps' specific configurations
- [[contributing]] - Help improve BlumeOps instead

224
docs/tutorials/replication/argocd-config.md Normal file
View file

@ -0,0 +1,224 @@
---
title: argocd-config
tags:
- tutorials
- replication
- argocd
---
# Configuring ArgoCD
> **Audiences:** Replicator
This tutorial walks through installing ArgoCD and establishing GitOps-driven deployments for your homelab.
## What is GitOps?
GitOps means your git repository is the source of truth for infrastructure:
- Infrastructure state is defined in git
- Changes happen through commits and pull requests
- A controller (ArgoCD) syncs git state to the cluster
- Drift is detected and can be corrected automatically
For BlumeOps specifics, see [[argocd|ArgoCD Reference]].
## Step 1: Install ArgoCD
```bash
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
```
Wait for pods to be ready:
```bash
kubectl -n argocd get pods -w
```
## Step 2: Access the UI
### Get the Initial Password
```bash
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
```
### Expose the Service
For Tailscale access:
```bash
tailscale serve --bg --https 8443 https+insecure://localhost:$(kubectl -n argocd get svc argocd-server -o jsonpath='{.spec.ports[?(@.name=="https")].port}')
```
Or create a Tailscale Ingress in Kubernetes.
Access at `https://your-server.tailnet.ts.net:8443`
### Install the CLI
```bash
brew install argocd # macOS
# or download from GitHub releases
```
Login:
```bash
argocd login your-server.tailnet.ts.net:8443
```
## Step 3: Connect Your Git Repository
Create a repository credential:
```bash
# For SSH
argocd repo add git@github.com:you/your-repo.git \
--ssh-private-key-path ~/.ssh/id_ed25519
# For HTTPS
argocd repo add https://github.com/you/your-repo.git \
--username you \
--password your-token
```
## Step 4: Create Your First Application
Create a directory in your repo:
```
your-repo/
└── apps/
└── hello-world/
├── deployment.yaml
└── service.yaml
```
With a simple deployment:
```yaml
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world
spec:
replicas: 1
selector:
matchLabels:
app: hello-world
template:
metadata:
labels:
app: hello-world
spec:
containers:
- name: hello
image: nginx:alpine
ports:
- containerPort: 80
```
Create the ArgoCD Application:
```bash
argocd app create hello-world \
--repo git@github.com:you/your-repo.git \
--path apps/hello-world \
--dest-server https://kubernetes.default.svc \
--dest-namespace default
```
## Step 5: Sync and Verify
```bash
# See what will be deployed
argocd app diff hello-world
# Deploy it
argocd app sync hello-world
# Check status
argocd app get hello-world
```
The pods should now be running:
```bash
kubectl get pods -l app=hello-world
```
## Step 6: App of Apps Pattern
For managing multiple applications, use the "app of apps" pattern:
```
your-repo/
├── argocd/
│ ├── apps/ # Application definitions
│ │ ├── hello-world.yaml
│ │ └── another-app.yaml
│ └── manifests/ # Actual Kubernetes manifests
│ ├── hello-world/
│ └── another-app/
```
Create a root Application that manages other Applications:
```yaml
# argocd/apps/apps.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: apps
namespace: argocd
spec:
project: default
source:
repoURL: git@github.com:you/your-repo.git
targetRevision: main
path: argocd/apps
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
```
Now adding a new application is just creating a YAML file.
## Step 7: Configure Sync Policies
| Policy | When to Use |
|--------|-------------|
| Manual sync | Production, explicit control |
| Auto sync | Development, or trusted workloads |
| Auto prune | Remove resources deleted from git |
| Self heal | Revert manual kubectl changes |
BlumeOps uses manual sync for workloads, auto sync only for the `apps` Application itself.
## What You Now Have
- GitOps workflow for deployments
- UI for visualizing application state
- Automatic drift detection
- Declarative application management
## Next Steps
- [[tutorials/replication/observability-stack | Build observability]] - Monitor your deployments
- Add more applications to your repo
- Set up notifications for sync failures
## BluemeOps Specifics
BlumeOps' ArgoCD configuration includes:
- SSH connection to [[forgejo]] git server
- Manual sync policy for all workloads
- Separate manifests and apps directories
See [[argocd|ArgoCD Reference]] and [[apps|Apps Reference]] for full details.
## Troubleshooting
| Problem | Solution |
|---------|----------|
| Sync failed | Check `argocd app get <app>` for error details |
| Can't connect to repo | Verify credentials, check SSH key permissions |
| Resources not appearing | Ensure path in Application matches repo structure |
| Out of sync but no diff | Check for ignored differences in app config |

170
docs/tutorials/replication/kubernetes-bootstrap.md Normal file
View file

@ -0,0 +1,170 @@
---
title: kubernetes-bootstrap
tags:
- tutorials
- replication
- kubernetes
---
# Bootstrapping Kubernetes
> **Audiences:** Replicator
This tutorial walks through setting up a Kubernetes cluster for your homelab, making it accessible via Tailscale.
## Choosing a Distribution
For homelab use, lightweight distributions work well:
| Distribution | Best For | BlumeOps Uses |
|--------------|----------|---------------|
| **Minikube** | Single-node, macOS | Yes |
| **k3s** | Single-node, Linux | - |
| **kind** | Local development | - |
| **kubeadm** | Multi-node clusters | - |
This tutorial uses minikube, but principles apply broadly.
For BlumeOps specifics, see [[cluster|Cluster Reference]].
## Step 1: Install Minikube
### macOS
```bash
brew install minikube
```
### Linux
```bash
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube
```
## Step 2: Create the Cluster
```bash
minikube start \
--driver=docker \
--cpus=4 \
--memory=8g \
--disk-size=100g \
--apiserver-names=k8s.your-tailnet.ts.net,$(hostname) \
--listen-address=0.0.0.0
```
Key flags:
- `--apiserver-names` - Include your Tailscale hostname for remote access
- `--listen-address=0.0.0.0` - Allow connections from other machines
## Step 3: Verify the Cluster
```bash
kubectl get nodes
# Should show your node as Ready
kubectl get pods -A
# Should show system pods running
```
## Step 4: Expose via Tailscale
To access the cluster from other Tailscale devices, expose the API server:
### Option A: Tailscale Serve (Simple)
```bash
tailscale serve --bg --tcp 6443 tcp://localhost:$(minikube ip --format '{{.Port}}')
```
### Option B: Tailscale Kubernetes Operator (Advanced)
For production-like setup, install the Tailscale operator which manages ingress automatically.
BlumeOps uses TCP passthrough via Caddy - see [[routing|Routing Reference]].
## Step 5: Configure Remote Access
On your workstation, add a context for the remote cluster:
```bash
# Copy the CA cert from the server
scp server:~/.minikube/ca.crt ~/.kube/minikube-ca.crt
# Add the cluster
kubectl config set-cluster minikube-remote \
--server=https://k8s.your-tailnet.ts.net:6443 \
--certificate-authority=$HOME/.kube/minikube-ca.crt
# Add credentials (copy from server's ~/.kube/config)
kubectl config set-credentials minikube-remote \
--client-certificate=... \
--client-key=...
# Add context
kubectl config set-context minikube-remote \
--cluster=minikube-remote \
--user=minikube-remote
# Test
kubectl --context=minikube-remote get nodes
```
## Step 6: Storage Configuration
For persistent workloads, configure storage:
### Local Path Provisioner (Simple)
```bash
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
```
### NFS for Shared Storage
If you have a NAS:
```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-share
spec:
capacity:
storage: 1Ti
accessModes:
- ReadWriteMany
nfs:
server: nas.your-tailnet.ts.net
path: /volume1/k8s
```
## What You Now Have
- A Kubernetes cluster running on your server
- Remote access via Tailscale
- Storage for persistent workloads
## Next Steps
- [[tutorials/replication/argocd-config | Configure ArgoCD]] - GitOps deployments
- Install essential addons (ingress controller, cert-manager)
## BluemeOps Specifics
BlumeOps' cluster configuration includes:
- Tailscale operator for automatic ingress
- NFS mounts from [[sifaka]] for media storage
- CloudNativePG for PostgreSQL databases
See [[cluster|Cluster Reference]] and [[apps|Apps Reference]] for full details.
## Troubleshooting
| Problem | Solution |
|---------|----------|
| Can't connect remotely | Check `--apiserver-names` includes Tailscale hostname |
| Pods stuck pending | Check storage class is available |
| Connection refused | Verify `--listen-address=0.0.0.0` was set |
| Certificate errors | Ensure CA cert matches server's |

231
docs/tutorials/replication/observability-stack.md Normal file
View file

@ -0,0 +1,231 @@
---
title: observability-stack
tags:
- tutorials
- replication
- observability
---
# Building the Observability Stack
> **Audiences:** Replicator
This tutorial walks through deploying metrics, logs, and dashboards for your homelab - because you can't fix what you can't see.
## The Stack
A complete observability solution has three pillars:
| Component | Purpose | BlumeOps Uses |
|-----------|---------|---------------|
| **Metrics** | Numeric measurements over time | [[prometheus]] |
| **Logs** | Text output from applications | [[loki]] |
| **Dashboards** | Visualization and alerting | [[grafana]] |
| **Collection** | Gathering and forwarding data | [[alloy]] |
For BlumeOps specifics, see [[observability|Observability Reference]].
## Step 1: Create Monitoring Namespace
```bash
kubectl create namespace monitoring
```
## Step 2: Deploy Prometheus
Prometheus collects and stores metrics.
### Using Helm
```bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/prometheus \
--namespace monitoring \
--set server.persistentVolume.size=10Gi
```
### Or via ArgoCD
Create an Application pointing to a values file in your repo:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: prometheus
namespace: argocd
spec:
project: default
source:
repoURL: https://prometheus-community.github.io/helm-charts
chart: prometheus
targetRevision: 25.0.0
helm:
values: |
server:
persistentVolume:
size: 10Gi
destination:
server: https://kubernetes.default.svc
namespace: monitoring
```
### Verify
```bash
kubectl -n monitoring get pods -l app.kubernetes.io/name=prometheus
```
## Step 3: Deploy Loki
Loki aggregates logs (like Prometheus but for logs).
```bash
helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki-stack \
--namespace monitoring \
--set loki.persistence.enabled=true \
--set loki.persistence.size=10Gi
```
This also installs Promtail for log collection from pods.
## Step 4: Deploy Grafana
Grafana provides dashboards and visualization.
```bash
helm install grafana grafana/grafana \
--namespace monitoring \
--set persistence.enabled=true \
--set persistence.size=1Gi \
--set adminPassword=admin # Change this!
```
### Configure Data Sources
After installation, add data sources in Grafana UI or via ConfigMap:
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-datasources
namespace: monitoring
labels:
grafana_datasource: "1"
data:
datasources.yaml: |
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
url: http://prometheus-server.monitoring.svc:80
isDefault: true
- name: Loki
type: loki
url: http://loki.monitoring.svc:3100
```
## Step 5: Access Grafana
Expose via Tailscale:
```bash
kubectl -n monitoring port-forward svc/grafana 3000:80 &
tailscale serve --bg --https 3000 http://localhost:3000
```
Or create an Ingress.
Default credentials: `admin` / (password you set or retrieve from secret)
## Step 6: Add Dashboards
Import community dashboards from [grafana.com/grafana/dashboards](https://grafana.com/grafana/dashboards/):
| Dashboard | ID | Shows |
|-----------|-----|-------|
| Node Exporter Full | 1860 | Host metrics |
| Kubernetes Cluster | 7249 | Cluster overview |
| Loki Logs | 13639 | Log exploration |
In Grafana: Dashboards > Import > Enter ID
## Step 7: Deploy Alloy (Optional)
Grafana Alloy is a unified collector that replaces multiple agents (Promtail, node_exporter, etc.).
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: alloy
namespace: argocd
spec:
project: default
source:
repoURL: https://grafana.github.io/helm-charts
chart: alloy
targetRevision: 0.1.0
helm:
values: |
alloy:
configMap:
content: |
// Alloy configuration here
destination:
server: https://kubernetes.default.svc
namespace: monitoring
```
BluemeOps uses Alloy on both [[indri]] (for host metrics) and in the [[cluster]] (for pod logs and service probes).
## What You Now Have
- Metrics collection and storage (Prometheus)
- Log aggregation (Loki)
- Dashboards and visualization (Grafana)
- Foundation for alerting
## Adding Alerts
Configure alerting rules in Prometheus:
```yaml
groups:
- name: example
rules:
- alert: HighMemoryUsage
expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1
for: 5m
labels:
severity: warning
annotations:
summary: "High memory usage detected"
```
And notification channels in Grafana (email, Slack, PagerDuty, etc.).
## Next Steps
- Create custom dashboards for your services
- Set up alerting for critical conditions
- Add service-specific metrics exporters
## BluemeOps Specifics
BlumeOps' observability setup includes:
- Prometheus scraping all services via annotations
- Loki collecting logs from all pods and [[indri]] services
- Custom dashboards for [[jellyfin]], [[teslamate]], and cluster health
- [[alloy]] running on both host and in-cluster
See [[observability|Observability Reference]] for full details.
## Troubleshooting
| Problem | Solution |
|---------|----------|
| No metrics appearing | Check Prometheus targets (`/targets` endpoint) |
| No logs in Loki | Verify Promtail/Alloy is collecting (`/ready` endpoint) |
| Dashboard shows no data | Check data source configuration and time range |
| High storage usage | Adjust retention settings in Prometheus/Loki |

134
docs/tutorials/replication/tailscale-setup.md Normal file
View file

@ -0,0 +1,134 @@
---
title: tailscale-setup
tags:
- tutorials
- replication
- tailscale
---
# Setting Up Tailscale
> **Audiences:** Replicator
This tutorial walks through establishing a Tailscale mesh network as the foundation for your homelab infrastructure.
## Why Tailscale?
Tailscale solves several problems at once:
- **Secure connectivity** - WireGuard-encrypted traffic between all devices
- **No port forwarding** - Devices connect directly through NATs and firewalls
- **MagicDNS** - Human-readable names like `server.tailnet.ts.net`
- **ACLs** - Fine-grained access control between devices
For BlumeOps context, see [[tailscale|Tailscale Reference]].
## Step 1: Create Your Tailnet
1. Sign up at [tailscale.com](https://tailscale.com)
2. Choose your identity provider (Google, Microsoft, GitHub, etc.)
3. Note your tailnet name (e.g., `yourname.ts.net`)
## Step 2: Install on Your Devices
### macOS
```bash
brew install tailscale
sudo tailscaled &
tailscale up
```
### Linux
```bash
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
```
### Other Platforms
See [Tailscale Downloads](https://tailscale.com/download) for iOS, Android, Windows, etc.
## Step 3: Verify Connectivity
After installing on two devices:
```bash
tailscale status
# Shows all connected devices
ping <other-device>.yourname.ts.net
# Should work immediately
```
## Step 4: Configure ACLs
Default Tailscale allows all-to-all connectivity. For a homelab, you'll want restrictions.
Create `policy.hujson` (or use the web admin):
```json
{
"groups": {
"group:admin": ["your-email@example.com"]
},
"tagOwners": {
"tag:homelab": ["group:admin"]
},
"acls": [
// Admins can access everything
{"action": "accept", "src": ["group:admin"], "dst": ["*:*"]},
// Homelab servers can reach NAS
{"action": "accept", "src": ["tag:homelab"], "dst": ["tag:nas:*"]}
]
}
```
BlumeOps manages ACLs via Pulumi - see [[tailscale|Tailscale Reference]] for the actual configuration.
## Step 5: Enable MagicDNS
In the Tailscale admin console:
1. Go to DNS settings
2. Enable MagicDNS
3. Optionally add a search domain
Now `ssh server` works instead of `ssh 100.x.y.z`.
## Step 6: Tag Your Devices
Tags enable role-based access control:
```bash
# On your server
sudo tailscale up --advertise-tags=tag:homelab
```
Tags must be defined in ACLs before use.
## What You Now Have
- Encrypted mesh network between all your devices
- DNS names for each device
- Foundation for exposing services securely
## Next Steps
With networking established:
- [[tutorials/replication/kubernetes-bootstrap | Bootstrap Kubernetes]] - Your cluster will join the tailnet
- Set up your server and storage devices
## BlumeOps Specifics
BluemeOps' Tailscale configuration includes:
- Multiple device tags (`homelab`, `nas`, `registry`, `k8s-api`)
- Group-based access for family members
- SSH access rules with authentication requirements
See [[tailscale|Tailscale Reference]] for full details.
## Troubleshooting
| Problem | Solution |
|---------|----------|
| Device won't connect | Check firewall allows UDP 41641 |
| Can't reach other devices | Verify ACLs don't block traffic |
| DNS not resolving | Enable MagicDNS in admin console |
| Tags not applying | Ensure tags defined in ACL policy |

67
docs/tutorials/what-is-blumeops.md Normal file
View file

@ -0,0 +1,67 @@
---
title: what-is-blumeops
tags:
- tutorials
- getting-started
---
# What is BlumeOps?
> **Audiences:** Reader, AI
BlumeOps is Erich Blume's personal infrastructure GitOps repository - a system for managing homelab services and infrastructure through version-controlled configuration.
## The Short Version
BlumeOps runs a collection of self-hosted services (media streaming, git hosting, photo management, RSS feeds, etc.) on home hardware, managed through code rather than manual configuration. Everything is tracked in a git repository, meaning changes are reviewable, reversible, and reproducible.
## Why Does This Exist?
Three motivations:
1. **Learning** - A playground for exploring DevOps, Kubernetes, and infrastructure automation
2. **Privacy** - Self-hosting services keeps data under personal control
3. **Resilience** - Less dependence on cloud providers and their service changes
## What's Running?
BlumeOps consists of:
- **A Mac Mini server** ([[indri]]) running Kubernetes and native services
- **A NAS** ([[sifaka]]) for storage and backups
- **~16 services** ranging from [[jellyfin|media streaming]] to [[grafana|observability dashboards]]
- **A Tailscale network** connecting everything securely
See [[reference/index|Reference]] for the complete service inventory.
## How Is It Organized?
```
blumeops/
├── ansible/ # Configuration for services on indri
├── argocd/ # Kubernetes manifests and ArgoCD apps
├── pulumi/ # Tailscale ACLs and DNS
└── docs/ # This documentation
```
Changes follow a GitOps workflow:
1. Modify configuration in a feature branch
2. Create a pull request for review
3. Deploy via ArgoCD (Kubernetes) or Ansible (indri)
4. Merge to main after verification
## Key Concepts
| Concept | What It Means in BlumeOps |
|---------|---------------------------|
| **GitOps** | All infrastructure defined in git; deploys happen by syncing git state |
| **IaC** | Infrastructure as Code - servers, networks, services defined in files |
| **Homelab** | Personal server infrastructure at home, not in a datacenter |
| **Tailscale** | VPN mesh network connecting all devices securely |
## Where to Go Next
- **Want to understand the architecture?** See [[reference/index|Reference]]
- **Want to help with BlumeOps?** See [[contributing]]
- **Want to build something similar?** See [[replicating-blumeops]]
- **Working with AI assistance?** See [[ai-assistance-guide]]