generated from eblume/project-template
Reframe v1 around a targetable storage backend
Some checks failed
Build / validate (push) Failing after 3s
Some checks failed
Build / validate (push) Failing after 3s
Replace "distributed from day one" with a targetable Store backend that supports the full spectrum from local-only to distributed by configuration: - Store trait with LocalStore (direct SQLite, exclusive lock) and RemoteStore (RPC to a server) - three hephd runtime modes: local / server / client - exclusive-lock handoff so the same SQLite file can pass between local and server mode; client mode is thin and online-only - offline remains a property of local-backed replicas syncing via the hub Local-only is now a first-class configuration. Build order starts at local mode. Updates docs/reference/tech-spec.md (§1, §3.1, §11) and docs/explanation/design.md (§9, §11) to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
cbf859b2d7
commit
247a079cd8
2 changed files with 36 additions and 22 deletions
|
|
@ -332,13 +332,13 @@ Reuse the established blumeops patterns (🔒 confirmed by repo conventions):
|
|||
|
||||
- ✅ Source of truth: **SQLite-primary**, markdown as the body payload, with a directory `export` mode. Clean break from Obsidian-the-app.
|
||||
- ✅ Note/task model: **typed node graph** — thin `task` nodes, rich `doc` nodes, connected by typed links.
|
||||
- ✅ Front-end: **per-device `hephd` daemon** with thin clients; **nvim plugin** as primary editing UX, **CLI** as everyday driver, **web UI** on the hub.
|
||||
- ✅ Sync (shape): **hub-and-spoke, op-log + CRDT**, HLC ordering, OIDC/Authentik auth, conflict queue for the ambiguous remainder. (Details below still open.)
|
||||
- ✅ Front-end: **`hephd` daemon** with thin clients over a local socket; **nvim plugin** (`heph.nvim`) as the primary surface, **CLI** secondary; **web UI** on the hub (later).
|
||||
- ✅ Sync (shape): **hub-and-spoke, op-log + CRDT**, HLC ordering, OIDC/Authentik auth, conflict queue for the ambiguous remainder, over a **targetable backend** (tech-spec §3.1).
|
||||
|
||||
### Decided since (B/C/D)
|
||||
|
||||
- ✅ Context surfacing v1: **explicit links only** + auto-created canonical context doc per task + nvim fuzzy search. Inferred/semantic deferred.
|
||||
- ✅ **v1 is distributed, not local-only:** client/server + **offline-first sync with automatic merge + conflict queue** + **full OIDC/Authentik auth with per-user isolation** are all IN v1 (tech-spec §12/§13). Web UI and k3s deployment are later (hub binary is built deployable).
|
||||
- ✅ **v1 supports local-only *and* distributed via a targetable backend** (tech-spec §3.1): `Store` trait + three modes (`local`/`server`/`client`) + exclusive-lock handoff over a shared SQLite file. **Offline-first sync with automatic merge + conflict queue** + **full OIDC/Authentik auth with per-user isolation** are all IN v1 (tech-spec §12/§13). Local-only is a first-class config. Web UI and k3s deployment are later (hub binary is built deployable).
|
||||
- ✅ Security v1: **OIDC/Authentik auth + per-user isolation** is the boundary; plain SQLite at rest (no encryption). May revisit.
|
||||
- ✅ Migration v1: **clean break**, no ZK import / no Todoist bridge. heph is system of record day one.
|
||||
- ✅ **License: All Rights Reserved / private.** Open-sourcing considered later.
|
||||
|
|
@ -380,7 +380,7 @@ Reuse the established blumeops patterns (🔒 confirmed by repo conventions):
|
|||
## 10. Roadmap (provisional)
|
||||
|
||||
- **Phase 0 — Design** (this document + [[tech-spec]]): done enough to build.
|
||||
- **Phase 1 — v1 prototype (one C1 effort, built in TDD slices):** `heph-core` (model, schema, extraction, recurrence, "what is next", op-log/HLC/CRDT merge) → `hephd` client mode → **hub mode + offline sync + conflict queue** → **OIDC/Authentik auth + per-user isolation** → `heph.nvim` + `heph` CLI. Runnable client/server, offline-capable, on the tailnet.
|
||||
- **Phase 1 — v1 prototype (one C1 effort, built in TDD slices):** `heph-core` (model, schema, extraction, recurrence, "what is next", `Store` trait, op-log/HLC/CRDT merge) → `hephd` **local mode** → **server + client modes (+ lock handoff)** → **offline sync + conflict queue** → **OIDC/Authentik auth + per-user isolation** → `heph.nvim` + `heph` CLI. Local-only works standalone; runnable client/server + offline sync on the tailnet.
|
||||
- **Phase 2 — k3s deployment:** Dagger→Zot image, ArgoCD app + Kustomize manifests, external-secrets; hub on blumeops.
|
||||
- **Phase 3 — Web UI** on the hub.
|
||||
- **Phase 4 (later, optional)** — calendar integration (careful CalDAV); migration from ZK / Todoist; iOS / Apple Watch voice capture; Hermes-style planning mode.
|
||||
|
|
@ -391,14 +391,14 @@ Reuse the established blumeops patterns (🔒 confirmed by repo conventions):
|
|||
|
||||
> **The canonical, implementation-ready scope/stack/schema/API now live in [[tech-spec]].** This section keeps the *intent* and the kickoff process; defer to the spec for details (and update the spec when decisions change).
|
||||
|
||||
### 11.1 Scope — *distributed from day one* (ratified)
|
||||
### 11.1 Scope — *local-to-distributed via a targetable backend* (ratified)
|
||||
|
||||
v1 is **not** local-only. It is a **client/server, offline-first** system:
|
||||
v1 **supports the full spectrum** — pure local-only through full distributed — chosen by configuration, not a rebuild. A `Store` abstraction backs three runtime modes (`local` / `server` / `client`) with an exclusive-lock handoff over a shared SQLite file (tech-spec §3.1).
|
||||
|
||||
- **In:** the full data model (§3, §6) incl. **recurrence + recurring checklists** (§3.3); the "what is next?" engine; **client/server architecture** (per-device client `hephd` + a runnable/deployable hub `hephd`); **offline-first op-log + CRDT sync with automatic merge and a conflict queue** (tech-spec §12); **OIDC/Authentik auth with per-user isolation** (tech-spec §13); the `heph.nvim` + `heph` CLI surfaces.
|
||||
- **In:** the full data model (§3, §6) incl. **recurrence + recurring checklists** (§3.3); the "what is next?" engine; the **targetable backend + all three modes** with the same-file local↔server lock handoff; **offline-first op-log + CRDT sync with automatic merge and a conflict queue** for local-backed replicas (tech-spec §12); **OIDC/Authentik auth with per-user isolation** on the server endpoint (tech-spec §13); the `heph.nvim` + `heph` CLI surfaces.
|
||||
- **Out (later, scaffolded):** the **web UI** (hub serves sync only in v1); **actual k3s deployment** to blumeops (hub binary is built deployable; ArgoCD/Kustomize/Dagger is a fast-follow); calendar; iOS/Watch; inferred context; P2P-over-tailnet fallback; persistent goal stack (an in-plugin session stack suffices).
|
||||
|
||||
**Why this bigger v1:** the distributed/offline/merge behavior and the auth model are *architecturally load-bearing* — bolting them on later would mean reworking the core. The owner explicitly wants them proven from the start. (Recurrence-with-checklists is in for the same reason — see §3.3.)
|
||||
**Why this v1:** the backend abstraction, offline/merge behavior, and auth model are *architecturally load-bearing* — bolting them on later means reworking the core. The owner wants them designed in from the start, while keeping **local-only a first-class, friction-free configuration**. (Recurrence-with-checklists is in for the same reason — see §3.3.) Natural build order: `local` mode first (simplest), then `server`/`client`, then sync, then auth.
|
||||
|
||||
### 11.2 Stack — ✅ RATIFIED
|
||||
|
||||
|
|
@ -409,7 +409,7 @@ Core: `rusqlite` (bundled) + migrations · `tokio` + JSON-RPC/unix-socket (local
|
|||
1. `mise run ai-docs`; read **[[tech-spec]]** (the build spec) and this doc's §3/§6 for rationale.
|
||||
2. **Classify as C1** — greenfield, no existing system to Mikado-untangle. Single long-lived feature branch + early draft PR, docs-first, push as you go ([[agent-change-process]]).
|
||||
3. Scaffold the cargo workspace + `heph.nvim` skeleton; **fill the AGENTS.md Project Structure section** (last template TODO).
|
||||
4. Build outward in testable slices (TDD, tech-spec §2/§9): `heph-core` (schema, model, extraction, recurrence, "what is next", **op-log/HLC/CRDT merge**) → `hephd` client mode (local RPC) → **hub mode + sync + conflict queue** → **OIDC auth** → `heph.nvim` + `heph` CLI.
|
||||
4. Build outward in testable slices (TDD, tech-spec §2/§9): `heph-core` (schema, model, extraction, recurrence, "what is next", `Store` trait, **op-log/HLC/CRDT merge**) → `hephd` **local mode** (LocalStore + lock + local RPC) → **server + client modes** (network endpoint, RemoteStore, lock handoff) → **sync + conflict queue** → **OIDC auth** → `heph.nvim` + `heph` CLI.
|
||||
5. Confirm the kickoff-open picks: recurrence model (a vs b, §3.3), CRDT lib (`yrs` vs `automerge`), hub transport, context-item editing surface (A vs B, §6.3).
|
||||
|
||||
## Related
|
||||
|
|
|
|||
|
|
@ -12,12 +12,10 @@ tags:
|
|||
|
||||
## 1. Overview
|
||||
|
||||
Hephaestus (heph) is a self-hosted personal context-management system that unifies a markdown knowledge base with task management in one database. **v1 is a distributed client/server system**: each device runs a local replica that works fully **offline**, and syncs to a central **hub** when reachable, with **automatic merge + conflict resolution** for concurrent offline edits. Access is authenticated via **OIDC (Authentik)** with per-user data isolation. The web UI and the actual k3s deployment are later phases; the hub ships in v1 as a runnable/deployable binary. Components:
|
||||
Hephaestus (heph) is a self-hosted personal context-management system that unifies a markdown knowledge base with task management in one database. **v1 supports the full spectrum from local-only to distributed**, selected by configuration via a **targetable storage backend** (§3.1): run purely local against a SQLite file, or run a server that fronts that file for remote clients and acts as the sync hub. Multi-device use is **offline-first** — local-backed replicas reconcile through the hub with **automatic merge + conflict resolution** — and access is authenticated via **OIDC (Authentik)** with per-user isolation. The web UI and the actual k3s deployment are later phases; the hub ships in v1 as a runnable/deployable binary. Components:
|
||||
|
||||
- **`heph-core`** — Rust library: data model, SQLite store, query engine, markdown parsing/extraction, recurrence, and the sync engine (op-log, HLC, CRDT merge, conflict detection).
|
||||
- **`hephd`** — Rust daemon with two modes:
|
||||
- **client mode** (per device): owns the local SQLite replica; serves a JSON-RPC API over a unix socket to local surfaces; runs background sync to the hub.
|
||||
- **server/hub mode**: owns the central SQLite; serves the authenticated sync endpoint (and, later, the web UI) over the network.
|
||||
- **`heph-core`** — Rust library: data model, the `Store` abstraction + local SQLite store, query engine, markdown parsing/extraction, recurrence, and the sync engine (op-log, HLC, CRDT merge, conflict detection).
|
||||
- **`hephd`** — Rust daemon; one binary, three runtime modes (`local` / `server` / `client`, §3.1). Always serves a JSON-RPC API over a local unix socket to local surfaces; in `server` mode it additionally exposes an authenticated network endpoint and runs as the sync hub.
|
||||
- **`heph`** — Rust CLI: utility/admin surface (export, scripting, smoke tests, `heph conflicts`).
|
||||
- **`heph.nvim`** — Lua Neovim plugin: the primary user surface ("org-mode"-style); a thin client of the local `hephd`.
|
||||
|
||||
|
|
@ -27,12 +25,28 @@ Hephaestus (heph) is a self-hosted personal context-management system that unifi
|
|||
|
||||
## 3. Architecture
|
||||
|
||||
- Surfaces (`heph.nvim`, `heph` CLI) are thin clients; they never touch SQLite directly. All state operations go through the local `hephd` over the unix socket. Socket path default: `$XDG_RUNTIME_DIR/heph/hephd.sock`.
|
||||
- Each device's `hephd` (client mode) is the single writer/owner of that device's local SQLite replica (WAL mode), and works **fully offline**. It records every local mutation as an op in an append-only **op-log** and runs **background sync** to the hub.
|
||||
- The **hub** `hephd` (server mode) is the **hub-and-spoke rendezvous**: devices push/pull ops to it over the network (authenticated, §13). It is *not* required to be online — devices keep working and reconcile when it returns.
|
||||
- Merge is automatic where possible; the unresolvable remainder goes to a **conflict queue** surfaced to the user (§12). Sync never blocks local work.
|
||||
- Surfaces (`heph.nvim`, `heph` CLI) never touch SQLite directly. They connect to the local `hephd` over a unix socket (default `$XDG_RUNTIME_DIR/heph/hephd.sock`) regardless of mode.
|
||||
- `heph-core` is synchronous and side-effect-light (incl. deterministic merge logic); `hephd` wraps it with async I/O, transport, and auth (`tokio`). DB calls run on a blocking pool.
|
||||
- See **§12 (Sync & Conflict Resolution)** and **§13 (Authentication)** for the detailed models.
|
||||
- See **§3.1 (Storage backends & runtime modes)**, **§12 (Sync & Conflict Resolution)**, **§13 (Authentication)**.
|
||||
|
||||
### 3.1 Storage backends & runtime modes
|
||||
|
||||
`heph-core` exposes a **`Store` trait** with two implementations, so what a runtime points at is configuration:
|
||||
|
||||
- **`LocalStore`** — a SQLite file opened **directly**, acquiring an **exclusive lock** on open (advisory `flock` on the DB file / a sidecar `.lock`). Refuses to start if the file is already locked.
|
||||
- **`RemoteStore`** — no local file; proxies every operation to a `server` over the network.
|
||||
|
||||
`hephd` runs in one of **three modes**:
|
||||
|
||||
| Mode | Backend | Network listener | Offline | Notes |
|
||||
|---|---|---|---|---|
|
||||
| **local** | `LocalStore` | none | yes | standalone single-host; the simplest deployment |
|
||||
| **server** | `LocalStore` | yes (authenticated) | yes | fronts the file for remote clients; **the sync hub** |
|
||||
| **client** | `RemoteStore` → a server | n/a | **no** | thin; online-only convenience |
|
||||
|
||||
**Lock handoff (the key flexibility):** `local` and `server` both take the file's exclusive lock, so **only one can own a given DB file at a time**. Kill the server → lock releases → a `local` process can open *the exact same file* and just work. Stop it → relaunch in `server` mode → remote clients reconnect. A `client` process never opens the file, so it never contends.
|
||||
|
||||
**Where offline + sync live:** offline capability comes from a **local-backed** instance (`local` or `server`) — a full replica with its own op-log. Two *different machines* each run a local-backed instance and **sync** op-logs through a hub (a `server`-mode instance), which is where automatic merge + the **conflict queue** (§12) happen; sync never blocks local work, and the hub need not always be online. A `client`-mode instance is **online-only** (no replica, no offline) — the convenience option (e.g. on the hub host, or an always-connected client). Same-file local↔server is single-host graceful-degradation; device↔device sync uses *separate replicas* that reconcile.
|
||||
|
||||
## 4. Data model
|
||||
|
||||
|
|
@ -258,9 +272,9 @@ All layers are required; CI runs them on every push/PR (extend `.forgejo/scripts
|
|||
**In:**
|
||||
|
||||
- The full data model, markdown handling, "what is next?" ranking, and **recurrence + recurring checklists** (§4–§8).
|
||||
- **Client/server architecture:** per-device client `hephd` + a central hub `hephd` (runnable/deployable binary).
|
||||
- **Offline-first** operation with **op-log + CRDT sync** and **automatic merge + a conflict queue** (§12).
|
||||
- **OIDC/Authentik authentication** with **per-user data isolation** (§13).
|
||||
- **Targetable storage backend + all three runtime modes** — `local`, `server`, `client` — with the **exclusive-lock handoff** over a shared SQLite file (§3.1). Local-only is a first-class, fully-supported configuration.
|
||||
- **Offline-first** operation for local-backed instances, with **op-log + CRDT sync** and **automatic merge + a conflict queue** (§12).
|
||||
- **OIDC/Authentik authentication** with **per-user data isolation** (§13), enforced on the `server` network endpoint.
|
||||
- `heph.nvim` + `heph` CLI surfaces (incl. `heph conflicts`).
|
||||
|
||||
**Out (later phases, scaffolded so as not to block):**
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue