From 247a079cd8b427429c03bb026b626bf003808bdd Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Sun, 31 May 2026 10:05:15 -0700 Subject: [PATCH] Reframe v1 around a targetable storage backend MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace "distributed from day one" with a targetable Store backend that supports the full spectrum from local-only to distributed by configuration: - Store trait with LocalStore (direct SQLite, exclusive lock) and RemoteStore (RPC to a server) - three hephd runtime modes: local / server / client - exclusive-lock handoff so the same SQLite file can pass between local and server mode; client mode is thin and online-only - offline remains a property of local-backed replicas syncing via the hub Local-only is now a first-class configuration. Build order starts at local mode. Updates docs/reference/tech-spec.md (§1, §3.1, §11) and docs/explanation/design.md (§9, §11) to match. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/explanation/design.md | 18 ++++++++--------- docs/reference/tech-spec.md | 40 +++++++++++++++++++++++++------------ 2 files changed, 36 insertions(+), 22 deletions(-) diff --git a/docs/explanation/design.md b/docs/explanation/design.md index 9a06a09..abf2068 100644 --- a/docs/explanation/design.md +++ b/docs/explanation/design.md @@ -332,13 +332,13 @@ Reuse the established blumeops patterns (🔒 confirmed by repo conventions): - ✅ Source of truth: **SQLite-primary**, markdown as the body payload, with a directory `export` mode. Clean break from Obsidian-the-app. - ✅ Note/task model: **typed node graph** — thin `task` nodes, rich `doc` nodes, connected by typed links. -- ✅ Front-end: **per-device `hephd` daemon** with thin clients; **nvim plugin** as primary editing UX, **CLI** as everyday driver, **web UI** on the hub. -- ✅ Sync (shape): **hub-and-spoke, op-log + CRDT**, HLC ordering, OIDC/Authentik auth, conflict queue for the ambiguous remainder. (Details below still open.) +- ✅ Front-end: **`hephd` daemon** with thin clients over a local socket; **nvim plugin** (`heph.nvim`) as the primary surface, **CLI** secondary; **web UI** on the hub (later). +- ✅ Sync (shape): **hub-and-spoke, op-log + CRDT**, HLC ordering, OIDC/Authentik auth, conflict queue for the ambiguous remainder, over a **targetable backend** (tech-spec §3.1). ### Decided since (B/C/D) - ✅ Context surfacing v1: **explicit links only** + auto-created canonical context doc per task + nvim fuzzy search. Inferred/semantic deferred. -- ✅ **v1 is distributed, not local-only:** client/server + **offline-first sync with automatic merge + conflict queue** + **full OIDC/Authentik auth with per-user isolation** are all IN v1 (tech-spec §12/§13). Web UI and k3s deployment are later (hub binary is built deployable). +- ✅ **v1 supports local-only *and* distributed via a targetable backend** (tech-spec §3.1): `Store` trait + three modes (`local`/`server`/`client`) + exclusive-lock handoff over a shared SQLite file. **Offline-first sync with automatic merge + conflict queue** + **full OIDC/Authentik auth with per-user isolation** are all IN v1 (tech-spec §12/§13). Local-only is a first-class config. Web UI and k3s deployment are later (hub binary is built deployable). - ✅ Security v1: **OIDC/Authentik auth + per-user isolation** is the boundary; plain SQLite at rest (no encryption). May revisit. - ✅ Migration v1: **clean break**, no ZK import / no Todoist bridge. heph is system of record day one. - ✅ **License: All Rights Reserved / private.** Open-sourcing considered later. @@ -380,7 +380,7 @@ Reuse the established blumeops patterns (🔒 confirmed by repo conventions): ## 10. Roadmap (provisional) - **Phase 0 — Design** (this document + [[tech-spec]]): done enough to build. -- **Phase 1 — v1 prototype (one C1 effort, built in TDD slices):** `heph-core` (model, schema, extraction, recurrence, "what is next", op-log/HLC/CRDT merge) → `hephd` client mode → **hub mode + offline sync + conflict queue** → **OIDC/Authentik auth + per-user isolation** → `heph.nvim` + `heph` CLI. Runnable client/server, offline-capable, on the tailnet. +- **Phase 1 — v1 prototype (one C1 effort, built in TDD slices):** `heph-core` (model, schema, extraction, recurrence, "what is next", `Store` trait, op-log/HLC/CRDT merge) → `hephd` **local mode** → **server + client modes (+ lock handoff)** → **offline sync + conflict queue** → **OIDC/Authentik auth + per-user isolation** → `heph.nvim` + `heph` CLI. Local-only works standalone; runnable client/server + offline sync on the tailnet. - **Phase 2 — k3s deployment:** Dagger→Zot image, ArgoCD app + Kustomize manifests, external-secrets; hub on blumeops. - **Phase 3 — Web UI** on the hub. - **Phase 4 (later, optional)** — calendar integration (careful CalDAV); migration from ZK / Todoist; iOS / Apple Watch voice capture; Hermes-style planning mode. @@ -391,14 +391,14 @@ Reuse the established blumeops patterns (🔒 confirmed by repo conventions): > **The canonical, implementation-ready scope/stack/schema/API now live in [[tech-spec]].** This section keeps the *intent* and the kickoff process; defer to the spec for details (and update the spec when decisions change). -### 11.1 Scope — *distributed from day one* (ratified) +### 11.1 Scope — *local-to-distributed via a targetable backend* (ratified) -v1 is **not** local-only. It is a **client/server, offline-first** system: +v1 **supports the full spectrum** — pure local-only through full distributed — chosen by configuration, not a rebuild. A `Store` abstraction backs three runtime modes (`local` / `server` / `client`) with an exclusive-lock handoff over a shared SQLite file (tech-spec §3.1). -- **In:** the full data model (§3, §6) incl. **recurrence + recurring checklists** (§3.3); the "what is next?" engine; **client/server architecture** (per-device client `hephd` + a runnable/deployable hub `hephd`); **offline-first op-log + CRDT sync with automatic merge and a conflict queue** (tech-spec §12); **OIDC/Authentik auth with per-user isolation** (tech-spec §13); the `heph.nvim` + `heph` CLI surfaces. +- **In:** the full data model (§3, §6) incl. **recurrence + recurring checklists** (§3.3); the "what is next?" engine; the **targetable backend + all three modes** with the same-file local↔server lock handoff; **offline-first op-log + CRDT sync with automatic merge and a conflict queue** for local-backed replicas (tech-spec §12); **OIDC/Authentik auth with per-user isolation** on the server endpoint (tech-spec §13); the `heph.nvim` + `heph` CLI surfaces. - **Out (later, scaffolded):** the **web UI** (hub serves sync only in v1); **actual k3s deployment** to blumeops (hub binary is built deployable; ArgoCD/Kustomize/Dagger is a fast-follow); calendar; iOS/Watch; inferred context; P2P-over-tailnet fallback; persistent goal stack (an in-plugin session stack suffices). -**Why this bigger v1:** the distributed/offline/merge behavior and the auth model are *architecturally load-bearing* — bolting them on later would mean reworking the core. The owner explicitly wants them proven from the start. (Recurrence-with-checklists is in for the same reason — see §3.3.) +**Why this v1:** the backend abstraction, offline/merge behavior, and auth model are *architecturally load-bearing* — bolting them on later means reworking the core. The owner wants them designed in from the start, while keeping **local-only a first-class, friction-free configuration**. (Recurrence-with-checklists is in for the same reason — see §3.3.) Natural build order: `local` mode first (simplest), then `server`/`client`, then sync, then auth. ### 11.2 Stack — ✅ RATIFIED @@ -409,7 +409,7 @@ Core: `rusqlite` (bundled) + migrations · `tokio` + JSON-RPC/unix-socket (local 1. `mise run ai-docs`; read **[[tech-spec]]** (the build spec) and this doc's §3/§6 for rationale. 2. **Classify as C1** — greenfield, no existing system to Mikado-untangle. Single long-lived feature branch + early draft PR, docs-first, push as you go ([[agent-change-process]]). 3. Scaffold the cargo workspace + `heph.nvim` skeleton; **fill the AGENTS.md Project Structure section** (last template TODO). -4. Build outward in testable slices (TDD, tech-spec §2/§9): `heph-core` (schema, model, extraction, recurrence, "what is next", **op-log/HLC/CRDT merge**) → `hephd` client mode (local RPC) → **hub mode + sync + conflict queue** → **OIDC auth** → `heph.nvim` + `heph` CLI. +4. Build outward in testable slices (TDD, tech-spec §2/§9): `heph-core` (schema, model, extraction, recurrence, "what is next", `Store` trait, **op-log/HLC/CRDT merge**) → `hephd` **local mode** (LocalStore + lock + local RPC) → **server + client modes** (network endpoint, RemoteStore, lock handoff) → **sync + conflict queue** → **OIDC auth** → `heph.nvim` + `heph` CLI. 5. Confirm the kickoff-open picks: recurrence model (a vs b, §3.3), CRDT lib (`yrs` vs `automerge`), hub transport, context-item editing surface (A vs B, §6.3). ## Related diff --git a/docs/reference/tech-spec.md b/docs/reference/tech-spec.md index 194b155..65f41b0 100644 --- a/docs/reference/tech-spec.md +++ b/docs/reference/tech-spec.md @@ -12,12 +12,10 @@ tags: ## 1. Overview -Hephaestus (heph) is a self-hosted personal context-management system that unifies a markdown knowledge base with task management in one database. **v1 is a distributed client/server system**: each device runs a local replica that works fully **offline**, and syncs to a central **hub** when reachable, with **automatic merge + conflict resolution** for concurrent offline edits. Access is authenticated via **OIDC (Authentik)** with per-user data isolation. The web UI and the actual k3s deployment are later phases; the hub ships in v1 as a runnable/deployable binary. Components: +Hephaestus (heph) is a self-hosted personal context-management system that unifies a markdown knowledge base with task management in one database. **v1 supports the full spectrum from local-only to distributed**, selected by configuration via a **targetable storage backend** (§3.1): run purely local against a SQLite file, or run a server that fronts that file for remote clients and acts as the sync hub. Multi-device use is **offline-first** — local-backed replicas reconcile through the hub with **automatic merge + conflict resolution** — and access is authenticated via **OIDC (Authentik)** with per-user isolation. The web UI and the actual k3s deployment are later phases; the hub ships in v1 as a runnable/deployable binary. Components: -- **`heph-core`** — Rust library: data model, SQLite store, query engine, markdown parsing/extraction, recurrence, and the sync engine (op-log, HLC, CRDT merge, conflict detection). -- **`hephd`** — Rust daemon with two modes: - - **client mode** (per device): owns the local SQLite replica; serves a JSON-RPC API over a unix socket to local surfaces; runs background sync to the hub. - - **server/hub mode**: owns the central SQLite; serves the authenticated sync endpoint (and, later, the web UI) over the network. +- **`heph-core`** — Rust library: data model, the `Store` abstraction + local SQLite store, query engine, markdown parsing/extraction, recurrence, and the sync engine (op-log, HLC, CRDT merge, conflict detection). +- **`hephd`** — Rust daemon; one binary, three runtime modes (`local` / `server` / `client`, §3.1). Always serves a JSON-RPC API over a local unix socket to local surfaces; in `server` mode it additionally exposes an authenticated network endpoint and runs as the sync hub. - **`heph`** — Rust CLI: utility/admin surface (export, scripting, smoke tests, `heph conflicts`). - **`heph.nvim`** — Lua Neovim plugin: the primary user surface ("org-mode"-style); a thin client of the local `hephd`. @@ -27,12 +25,28 @@ Hephaestus (heph) is a self-hosted personal context-management system that unifi ## 3. Architecture -- Surfaces (`heph.nvim`, `heph` CLI) are thin clients; they never touch SQLite directly. All state operations go through the local `hephd` over the unix socket. Socket path default: `$XDG_RUNTIME_DIR/heph/hephd.sock`. -- Each device's `hephd` (client mode) is the single writer/owner of that device's local SQLite replica (WAL mode), and works **fully offline**. It records every local mutation as an op in an append-only **op-log** and runs **background sync** to the hub. -- The **hub** `hephd` (server mode) is the **hub-and-spoke rendezvous**: devices push/pull ops to it over the network (authenticated, §13). It is *not* required to be online — devices keep working and reconcile when it returns. -- Merge is automatic where possible; the unresolvable remainder goes to a **conflict queue** surfaced to the user (§12). Sync never blocks local work. +- Surfaces (`heph.nvim`, `heph` CLI) never touch SQLite directly. They connect to the local `hephd` over a unix socket (default `$XDG_RUNTIME_DIR/heph/hephd.sock`) regardless of mode. - `heph-core` is synchronous and side-effect-light (incl. deterministic merge logic); `hephd` wraps it with async I/O, transport, and auth (`tokio`). DB calls run on a blocking pool. -- See **§12 (Sync & Conflict Resolution)** and **§13 (Authentication)** for the detailed models. +- See **§3.1 (Storage backends & runtime modes)**, **§12 (Sync & Conflict Resolution)**, **§13 (Authentication)**. + +### 3.1 Storage backends & runtime modes + +`heph-core` exposes a **`Store` trait** with two implementations, so what a runtime points at is configuration: + +- **`LocalStore`** — a SQLite file opened **directly**, acquiring an **exclusive lock** on open (advisory `flock` on the DB file / a sidecar `.lock`). Refuses to start if the file is already locked. +- **`RemoteStore`** — no local file; proxies every operation to a `server` over the network. + +`hephd` runs in one of **three modes**: + +| Mode | Backend | Network listener | Offline | Notes | +|---|---|---|---|---| +| **local** | `LocalStore` | none | yes | standalone single-host; the simplest deployment | +| **server** | `LocalStore` | yes (authenticated) | yes | fronts the file for remote clients; **the sync hub** | +| **client** | `RemoteStore` → a server | n/a | **no** | thin; online-only convenience | + +**Lock handoff (the key flexibility):** `local` and `server` both take the file's exclusive lock, so **only one can own a given DB file at a time**. Kill the server → lock releases → a `local` process can open *the exact same file* and just work. Stop it → relaunch in `server` mode → remote clients reconnect. A `client` process never opens the file, so it never contends. + +**Where offline + sync live:** offline capability comes from a **local-backed** instance (`local` or `server`) — a full replica with its own op-log. Two *different machines* each run a local-backed instance and **sync** op-logs through a hub (a `server`-mode instance), which is where automatic merge + the **conflict queue** (§12) happen; sync never blocks local work, and the hub need not always be online. A `client`-mode instance is **online-only** (no replica, no offline) — the convenience option (e.g. on the hub host, or an always-connected client). Same-file local↔server is single-host graceful-degradation; device↔device sync uses *separate replicas* that reconcile. ## 4. Data model @@ -258,9 +272,9 @@ All layers are required; CI runs them on every push/PR (extend `.forgejo/scripts **In:** - The full data model, markdown handling, "what is next?" ranking, and **recurrence + recurring checklists** (§4–§8). -- **Client/server architecture:** per-device client `hephd` + a central hub `hephd` (runnable/deployable binary). -- **Offline-first** operation with **op-log + CRDT sync** and **automatic merge + a conflict queue** (§12). -- **OIDC/Authentik authentication** with **per-user data isolation** (§13). +- **Targetable storage backend + all three runtime modes** — `local`, `server`, `client` — with the **exclusive-lock handoff** over a shared SQLite file (§3.1). Local-only is a first-class, fully-supported configuration. +- **Offline-first** operation for local-backed instances, with **op-log + CRDT sync** and **automatic merge + a conflict queue** (§12). +- **OIDC/Authentik authentication** with **per-user data isolation** (§13), enforced on the `server` network endpoint. - `heph.nvim` + `heph` CLI surfaces (incl. `heph conflicts`). **Out (later phases, scaffolded so as not to block):**