Reframe v1 around a targetable storage backend
Some checks failed
Build / validate (push) Failing after 3s

Replace "distributed from day one" with a targetable Store backend that
supports the full spectrum from local-only to distributed by configuration:

- Store trait with LocalStore (direct SQLite, exclusive lock) and
  RemoteStore (RPC to a server)
- three hephd runtime modes: local / server / client
- exclusive-lock handoff so the same SQLite file can pass between local and
  server mode; client mode is thin and online-only
- offline remains a property of local-backed replicas syncing via the hub

Local-only is now a first-class configuration. Build order starts at local
mode. Updates docs/reference/tech-spec.md (§1, §3.1, §11) and
docs/explanation/design.md (§9, §11) to match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Erich Blume 2026-05-31 10:05:15 -07:00
commit 247a079cd8
2 changed files with 36 additions and 22 deletions

View file

@ -12,12 +12,10 @@ tags:
## 1. Overview
Hephaestus (heph) is a self-hosted personal context-management system that unifies a markdown knowledge base with task management in one database. **v1 is a distributed client/server system**: each device runs a local replica that works fully **offline**, and syncs to a central **hub** when reachable, with **automatic merge + conflict resolution** for concurrent offline edits. Access is authenticated via **OIDC (Authentik)** with per-user data isolation. The web UI and the actual k3s deployment are later phases; the hub ships in v1 as a runnable/deployable binary. Components:
Hephaestus (heph) is a self-hosted personal context-management system that unifies a markdown knowledge base with task management in one database. **v1 supports the full spectrum from local-only to distributed**, selected by configuration via a **targetable storage backend** (§3.1): run purely local against a SQLite file, or run a server that fronts that file for remote clients and acts as the sync hub. Multi-device use is **offline-first** — local-backed replicas reconcile through the hub with **automatic merge + conflict resolution** — and access is authenticated via **OIDC (Authentik)** with per-user isolation. The web UI and the actual k3s deployment are later phases; the hub ships in v1 as a runnable/deployable binary. Components:
- **`heph-core`** — Rust library: data model, SQLite store, query engine, markdown parsing/extraction, recurrence, and the sync engine (op-log, HLC, CRDT merge, conflict detection).
- **`hephd`** — Rust daemon with two modes:
- **client mode** (per device): owns the local SQLite replica; serves a JSON-RPC API over a unix socket to local surfaces; runs background sync to the hub.
- **server/hub mode**: owns the central SQLite; serves the authenticated sync endpoint (and, later, the web UI) over the network.
- **`heph-core`** — Rust library: data model, the `Store` abstraction + local SQLite store, query engine, markdown parsing/extraction, recurrence, and the sync engine (op-log, HLC, CRDT merge, conflict detection).
- **`hephd`** — Rust daemon; one binary, three runtime modes (`local` / `server` / `client`, §3.1). Always serves a JSON-RPC API over a local unix socket to local surfaces; in `server` mode it additionally exposes an authenticated network endpoint and runs as the sync hub.
- **`heph`** — Rust CLI: utility/admin surface (export, scripting, smoke tests, `heph conflicts`).
- **`heph.nvim`** — Lua Neovim plugin: the primary user surface ("org-mode"-style); a thin client of the local `hephd`.
@ -27,12 +25,28 @@ Hephaestus (heph) is a self-hosted personal context-management system that unifi
## 3. Architecture
- Surfaces (`heph.nvim`, `heph` CLI) are thin clients; they never touch SQLite directly. All state operations go through the local `hephd` over the unix socket. Socket path default: `$XDG_RUNTIME_DIR/heph/hephd.sock`.
- Each device's `hephd` (client mode) is the single writer/owner of that device's local SQLite replica (WAL mode), and works **fully offline**. It records every local mutation as an op in an append-only **op-log** and runs **background sync** to the hub.
- The **hub** `hephd` (server mode) is the **hub-and-spoke rendezvous**: devices push/pull ops to it over the network (authenticated, §13). It is *not* required to be online — devices keep working and reconcile when it returns.
- Merge is automatic where possible; the unresolvable remainder goes to a **conflict queue** surfaced to the user (§12). Sync never blocks local work.
- Surfaces (`heph.nvim`, `heph` CLI) never touch SQLite directly. They connect to the local `hephd` over a unix socket (default `$XDG_RUNTIME_DIR/heph/hephd.sock`) regardless of mode.
- `heph-core` is synchronous and side-effect-light (incl. deterministic merge logic); `hephd` wraps it with async I/O, transport, and auth (`tokio`). DB calls run on a blocking pool.
- See **§12 (Sync & Conflict Resolution)** and **§13 (Authentication)** for the detailed models.
- See **§3.1 (Storage backends & runtime modes)**, **§12 (Sync & Conflict Resolution)**, **§13 (Authentication)**.
### 3.1 Storage backends & runtime modes
`heph-core` exposes a **`Store` trait** with two implementations, so what a runtime points at is configuration:
- **`LocalStore`** — a SQLite file opened **directly**, acquiring an **exclusive lock** on open (advisory `flock` on the DB file / a sidecar `.lock`). Refuses to start if the file is already locked.
- **`RemoteStore`** — no local file; proxies every operation to a `server` over the network.
`hephd` runs in one of **three modes**:
| Mode | Backend | Network listener | Offline | Notes |
|---|---|---|---|---|
| **local** | `LocalStore` | none | yes | standalone single-host; the simplest deployment |
| **server** | `LocalStore` | yes (authenticated) | yes | fronts the file for remote clients; **the sync hub** |
| **client** | `RemoteStore` → a server | n/a | **no** | thin; online-only convenience |
**Lock handoff (the key flexibility):** `local` and `server` both take the file's exclusive lock, so **only one can own a given DB file at a time**. Kill the server → lock releases → a `local` process can open *the exact same file* and just work. Stop it → relaunch in `server` mode → remote clients reconnect. A `client` process never opens the file, so it never contends.
**Where offline + sync live:** offline capability comes from a **local-backed** instance (`local` or `server`) — a full replica with its own op-log. Two *different machines* each run a local-backed instance and **sync** op-logs through a hub (a `server`-mode instance), which is where automatic merge + the **conflict queue** (§12) happen; sync never blocks local work, and the hub need not always be online. A `client`-mode instance is **online-only** (no replica, no offline) — the convenience option (e.g. on the hub host, or an always-connected client). Same-file local↔server is single-host graceful-degradation; device↔device sync uses *separate replicas* that reconcile.
## 4. Data model
@ -258,9 +272,9 @@ All layers are required; CI runs them on every push/PR (extend `.forgejo/scripts
**In:**
- The full data model, markdown handling, "what is next?" ranking, and **recurrence + recurring checklists** (§4§8).
- **Client/server architecture:** per-device client `hephd` + a central hub `hephd` (runnable/deployable binary).
- **Offline-first** operation with **op-log + CRDT sync** and **automatic merge + a conflict queue** (§12).
- **OIDC/Authentik authentication** with **per-user data isolation** (§13).
- **Targetable storage backend + all three runtime modes**`local`, `server`, `client` — with the **exclusive-lock handoff** over a shared SQLite file (§3.1). Local-only is a first-class, fully-supported configuration.
- **Offline-first** operation for local-backed instances, with **op-log + CRDT sync** and **automatic merge + a conflict queue** (§12).
- **OIDC/Authentik authentication** with **per-user data isolation** (§13), enforced on the `server` network endpoint.
- `heph.nvim` + `heph` CLI surfaces (incl. `heph conflicts`).
**Out (later phases, scaffolded so as not to block):**