From 2fcc5c0f224bce5ab332324091ae23ad4376c7f7 Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Sun, 31 May 2026 18:08:47 -0700 Subject: [PATCH] Fold second-pass design review into v1 spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolve the open tensions surfaced in the pre-Phase-1 second pass over design.md and tech-spec.md: - Context items: Fork A index model β€” body markdown is the source of truth; context items are a locally-derived, non-synced index; identity is pinned at promotion. Dissolves the body-CRDT vs. extraction convergence problem. - Recurrence: roll-forward in place; drop task_occurrences and is_template; advance to next RRULE instance after now (skip misses). - Identity: deterministic ids for journal/tag (offline-convergent); ULID for content nodes and project. - Mode/sync: orthogonal hub_url spoke capability; everyday device is local + hub_url, not server. - Auth/owner: nullable oidc_sub, friction-free local user, hub- authoritative identity, one-time pre-first-sync adoption rewrite. - Ranking: do_date is a boolean candidacy filter only; late_on is the sole urgency signal (global tier); FIFO tiebreak; order expressed as a reorderable named-dimension list. - Modes are plugin-side compositions; add list() and log.tail(). - Frame v1 as a single deliberate C1; misc cleanups (export, health, CI nvim runner, README license). Co-Authored-By: Claude Opus 4.8 (1M context) --- AGENTS.md | 2 +- docs/changelog.d/+v1-spec-second-pass.doc.md | 1 + docs/explanation/design.md | 48 ++++--- docs/reference/tech-spec.md | 128 ++++++++++--------- 4 files changed, 100 insertions(+), 79 deletions(-) create mode 100644 docs/changelog.d/+v1-spec-second-pass.doc.md diff --git a/AGENTS.md b/AGENTS.md index 6188bef..4e0d4cc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,7 +13,7 @@ This repository was instantiated from the `project-template` Forgejo template. S - [x] `baseUrl` in `docs/quartz.config.ts` set to `localhost` (update once docs are hosted) - [x] Dagger module renamed to `.dagger/src/hephaestus_ci/` (`HephaestusCi` class) - [ ] Fill in the Project Structure section below once the Rust workspace is scaffolded -- [ ] Fill in license info in `README.md` +- [x] Fill in license info in `README.md` (All Rights Reserved / private) Delete this section once the remaining items are resolved. diff --git a/docs/changelog.d/+v1-spec-second-pass.doc.md b/docs/changelog.d/+v1-spec-second-pass.doc.md new file mode 100644 index 0000000..2348847 --- /dev/null +++ b/docs/changelog.d/+v1-spec-second-pass.doc.md @@ -0,0 +1 @@ +Second-pass review of the v1 design + tech spec before Phase 1: resolved context-item storage (Fork A index model), recurrence (roll-forward in place), key-unique node identity (deterministic ids for journal/tag), mode/sync orthogonality (`hub_url` spoke), the local-only owner / OIDC adoption path, the "what is next?" ranking (do-date as candidacy filter only, late-on as the sole urgency signal), and the plugin-side mode model (added `list` + `log.tail`). diff --git a/docs/explanation/design.md b/docs/explanation/design.md index abf2068..c8cb070 100644 --- a/docs/explanation/design.md +++ b/docs/explanation/design.md @@ -62,7 +62,8 @@ Today these are loosely coupled by fragile cross-links (`todoist://` in ### 3.1 Identity -- ❓ **Leaning ULID** (sortable, sync-safe, collision-free across offline devices) as the internal primary key, with a human-facing `slug`/`alias` layer on top so `[[wiki-links]]` can be written by name. The ZK's timestamp-IDs (`1722897441-MWFE`) are ULID-like already; a migration can map them. +- πŸ”’ **DECIDED β€” ULID for content nodes** (`doc`/`task`/`project`), sortable and sync-safe, with a human-facing `slug`/`alias` layer so `[[wiki-links]]` can be written by name. The ZK's timestamp-IDs (`1722897441-MWFE`) are ULID-like already; a future migration can map them. +- πŸ”’ **DECIDED β€” deterministic ids for key-unique kinds.** Random ULIDs *diverge* when two offline replicas independently create the *same logical singleton* (today's `journal`, a `tag` by name) β€” duplicates that never merge. So `journal` ids derive deterministically from `(owner, ISO-date)` and `tag` ids from `(owner, normalized-name)`; independent creation then yields the *same* id and the CRDT/op-log coalesce them (two partial daily notes even merge). `project` stays a random ULID (deliberately created, renameable, tiny stable set). Tag rename = retag (no in-place key change); the normalization function is a fixed, versioned constant. See [[tech-spec]] Β§4.5. ### 3.2 Scalar task attributes vs. links @@ -74,18 +75,13 @@ A **recurring task** carries an **RFC-5545 RRULE** and acts as a recurring **def > ❌ **The anti-pattern we must avoid (Todoist's mistake):** modeling a recurring task's checklist sub-items as standalone, non-recurring tasks that **carry their completion state forward** when the parent recurs. This is the corner; designing around it now is *why* recurring checklists are in v1 rather than deferred. -**heph model (proposed β€” one point to ratify):** +**heph model β€” πŸ”’ DECIDED (roll-forward in place):** -- The recurring definition holds a **checklist template** (the ordered set of sub-item definitions). -- Each **occurrence** materializes a **fresh checklist instance** β€” brand-new context items, all `outstanding` β€” scoped to that occurrence. **Completion state is per-occurrence and never carries forward.** -- Occurrences are retained as history (no hard deletes; ties naturally to the per-task **log**, Β§6.4). The "what is next?" engine surfaces the **current/active occurrence**; completing it advances to the next per the RRULE. +- The recurring definition's **checklist template** is just the `- [ ]` lines in its canonical context `doc` body (under Fork A, Β§6.3, the body *is* the template β€” no separate template entity). +- A recurring task is a **single node**. On completion: (1) append the finished occurrence to the per-task **log** (Β§6.4), (2) reset the body's checkboxes to unchecked (a body-CRDT edit), (3) advance the do-date to **the next RRULE instance strictly after `now`**, *skipping* missed occurrences. So a fresh, all-`outstanding` checklist each occurrence; **completion never carries forward**; a missed daily routine is **one** gently-overdue item, not a pile. +- History is narrative (the log), matching "narrative > list" (Β§6.1). RRULE expansion stays **lazy** β€” only "next instance after now" is ever computed, never the series (Β§6.6). -**The one fork to confirm:** - -- **(a) Occurrence instances (recommended):** the definition spawns a lightweight **instance** per occurrence (its own do-date + its own fresh checklist items). Full per-occurrence history; unambiguously "a fresh instance per recurrence"; heavier (more rows). -- **(b) Roll-forward in place:** a single task node; on completion, **log the finished occurrence** (to the per-task log) and **reset the checklist to outstanding** + advance the do-date. Lighter; history is narrative (in the log) rather than queryable per-occurrence. - -Both satisfy the correctness requirement (fresh checklist, no carried state). (a) is recommended for clean history and zero ambiguity; (b) is the pragmatic-lite option. +**Why not occurrence-instances?** The rejected alternative (a fresh node per occurrence) was attractive for queryable per-occurrence history, but under Fork A each occurrence would need its own body β€” hence its own node β€” turning every daily recurrence into a new node-pair: exactly the explosion Β§6.6 forbids. No v1 query needs occurrence rows (the ranking reads the single node's do-date), and adherence/streak stats can be reconstructed from the log later. See [[tech-spec]] Β§4.4. ## 4. Architecture @@ -94,7 +90,7 @@ Both satisfy the correctness requirement (fresh checklist, no carried state). (a Layers, top to bottom: - **Surfaces (thin clients):** the **nvim plugin (`heph.nvim`)** is the **primary surface for v1** β€” a full "org-mode"-style experience (markdown buffers backed by `doc` nodes; agenda / "what is next" / capture / linking / journaling as plugin commands). It is the explicit **successor to obsidian.nvim**, which the owner currently drives the ZK with (telescope picker, `o*` verbs, `` to follow `[[wiki-links]]`, dailies, multi-state checkboxes) and rarely uses Obsidian-proper alongside. heph.nvim must reach that feature parity (see Β§6.5) and replace it. The **CLI** (`heph`) is a secondary/utility surface (export, scripting, admin) β€” explicitly *not* a focus, though it shares the same command surface. The **web UI** is the occasional hub-served surface. A later iOS/Watch client talks to the hub directly. - - ❓ **Dual-mode binary:** a single Rust binary that can run as the **server/daemon** *or* expose the same operations as **CLI subcommands** *or* act as the **backend the nvim plugin drives**. Appealing (one codebase, one command surface) β€” to confirm as the packaging model. + - πŸ”’ **Single binary, three modes.** One Rust binary runs as `local` / `server` / `client` ([[tech-spec]] Β§3.1); the `heph` CLI shares the same command surface. Mode is two orthogonal axes (backend + inbound listener) plus an optional `hub_url` that makes any `local` instance a syncing **spoke** β€” the everyday device is `local` + `hub_url`, the hub is `server`, `client` is the online-only convenience. - **Per-device daemon (`hephd`):** owns the local SQLite handle, the CRDT/op-log state, and background sync. All local surfaces connect to it over a local socket. This is what makes multi-surface access concurrent and safe on one SQLite file, and gives one place to run background sync. - **Core crate (Rust lib):** data model, query engine, markdown parsing + wiki-link extraction, and sync logic. Linked by both `hephd` and the hub server. - **Hub:** `hephd` running in "server" mode on blumeops k3s β€” central SQLite, the sync rendezvous point, and the web UI host. May be offline for long periods; devices keep working and reconcile when it returns. @@ -251,6 +247,8 @@ The owner's old rule β€” "avoid ncurses and interactive UIs; write atomic code a **heph's default "what is next?" (Tactical blank-slate)** therefore β‰ˆ Top of Mind (Red first β€” by *consequence*, then Orange) **+** White items whose do-date has arrived, scoped to the current project/context, **with Blue hidden**. Concise, honest, light. +> πŸ”’ **DECIDED ranking mechanics ([[tech-spec]] Β§7):** `do_date` is a *boolean candidacy filter only* (null β‡’ "now"), **never** an urgency input; **`late_on` is the sole urgency signal** (a global "now a problem" top tier); within a band, FIFO by `created_at` β€” **age never becomes urgency**. The order is expressed as a reorderable named-dimension list so it can be retuned without touching the engine. + ### 6.3 Two kinds of task: commitments vs. context items > πŸ”’ **DECIDED (shape).** A **commitment axis** orthogonal to the Β§6.2 attention-states. @@ -263,10 +261,10 @@ The owner's old rule β€” "avoid ncurses and interactive UIs; write atomic code a #### Editing surface for context items β€” *deliberately deferred to the prototype* -> πŸ”‘ **Key realization: this is NOT a data-model fork.** In both options below the **stored representation is identical** β€” context items are outstanding nodes linked to their container; the daemon owns them; markdown `export` renders them as checkboxes regardless. The choice is *purely the nvim editing affordance*, so the backend is shared and we **prototype both and pick by feel** (ergonomics can't be judged on paper). +> πŸ”‘ **DECIDED β€” Fork A "index" backend (the affordance is still trialed both ways).** The stored source of truth is the `doc` **body markdown**: context items are `- [ ]` lines in it, and a context item's checked state *is* the `[ ]`/`[x]`. They are **not** synced nodes β€” each replica derives a **local index** from its converged CRDT body, so they converge for free (no divergent ULIDs across offline edits β€” the original sync trap). Identity is pinned only at **promotion**, which mints a real committed `task` node and rewrites the source line into a link to it. This collapses the hardest sync problem into "the body CRDT already converges" and matches this section's "ephemeral, low-stakes identity" intent. **What's still trialed both ways is purely the nvim *affordance*** (Option A checkboxes-in-body vs. Option B command-driven capture) β€” the backend is shared, so ergonomics can be judged in the prototype. See [[tech-spec]] Β§4.3, Β§5. - **Option A β€” checkboxes in the body, derived on save.** Edit the task's canonical context doc as plain markdown; `- [ ]` lines are materialized into context-item state on `:w` (`BufWritePost`), reusing the wiki-link extraction machinery. **No real-time scanning needed** (debounced live updates are optional polish). Remote CRDT edits are *pushed* from the daemon to reconcile an open buffer. Item identity is low-stakes (reword = tombstone-old + add-new) since items are ephemeral; identity is pinned only at **promotion**. Most "personal org-mode"-native; millisecond capture = "type a line." -- **Option B β€” structured items, rendered.** Items are nodes from the start; the body stays pure prose. Render inline via **virtual text/extmarks** (no pane) or a transient **floating window** (avoids the disliked persistent cutaway); capture via a leader chord + `vim.ui.input`. Cleaner separation and trivial promotion; heavier capture; more UI to build. +- **Option B β€” command-driven capture, rendered.** Capture via a leader chord + `vim.ui.input` (no raw checkbox typing); the new item is still written as a `- [ ]` line into the body-backed store (Fork A), then **rendered** inline via virtual text/extmarks or a transient **floating window** (avoids the disliked persistent cutaway). Heavier capture; more UI to build; promotion identical. - Lean: **A** for capture flow, but **build the shared node backend first** and trial both affordances in the plugin. > **Prior-art note (multi-state checkboxes):** the owner's obsidian.nvim config uses checkbox states `" ", "x", ">", "~", "!"` (todo / done / forwarded / ~ / important). heph's task/context-item state model should accommodate richer-than-binary states (at least the done/dropped/outstanding set in Β§6.1; possibly forwarded/deferred). @@ -364,10 +362,18 @@ Reuse the established blumeops patterns (πŸ”’ confirmed by repo conventions): *Prototype-blocking (resolve at kickoff β€” see Β§11 / tech-spec):* -1. **Recurrence model (Β§3.3): (a) occurrence-instances vs (b) roll-forward-in-place** β€” both avoid the corner; leaning (a). -2. **CRDT library** for body merge: `yrs` (leaning) vs `automerge`; bespoke op-log/HLC for scalar fields either way. -3. **Hub network transport** (tech-spec Β§6.1): `axum` HTTP/JSON (leaning) vs gRPC; sync propagation cadence (push vs periodic pull). -4. **Context-item editing surface (Β§6.3): Option A vs B** β€” *decided inside the prototype by feel* (shared backend). +1. **CRDT library** for body merge: `yrs` (leaning) vs `automerge`; bespoke op-log/HLC for scalar fields either way. +2. **Hub network transport** (tech-spec Β§6.1): `axum` HTTP/JSON (leaning) vs gRPC; sync propagation cadence (push vs periodic pull); **device-id provisioning** (how a spoke mints its `origin` id and registers with the hub). + +*Resolved in the second-pass review (2026-05-31), now folded into [[tech-spec]]:* + +3. βœ… **Recurrence β†’ roll-forward in place** (Β§3.3) β€” occurrence-instances rejected (would explode into a node-per-occurrence under Fork A); advance to the next RRULE instance after `now`, skipping misses. +4. βœ… **Context items β†’ Fork A "index" backend** (Β§6.3) β€” body markdown is the source of truth; context items are a local derived index, not synced nodes; identity at promotion. The nvim *affordance* (A vs B) is still trialed in-prototype. +5. βœ… **Identity β†’ deterministic ids for `journal`/`tag`** (Β§3.1); random ULID for content nodes/`project`. +6. βœ… **Mode/sync orthogonality** (Β§4): backend + inbound-listener axes plus optional `hub_url`; everyday device = `local` + `hub_url`. +7. βœ… **Auth/owner** (tech-spec Β§13): nullable `oidc_sub`, local user for local-only, hub-authoritative identity, one-time pre-first-sync adoption rewrite. +8. βœ… **Ranking** (tech-spec Β§7): `do_date` boolean candidacy filter only; `late_on` sole urgency signal (global tier); FIFO tiebreak; order as a reorderable named-dimension list. +9. βœ… **Modes** (Β§6.1) are plugin-side compositions of mode-agnostic daemon primitives; added `list` (Organizational) and `log.tail` (resumption breadcrumb). *Not prototype-blocking (later phases):* @@ -380,7 +386,7 @@ Reuse the established blumeops patterns (πŸ”’ confirmed by repo conventions): ## 10. Roadmap (provisional) - **Phase 0 β€” Design** (this document + [[tech-spec]]): done enough to build. -- **Phase 1 β€” v1 prototype (one C1 effort, built in TDD slices):** `heph-core` (model, schema, extraction, recurrence, "what is next", `Store` trait, op-log/HLC/CRDT merge) β†’ `hephd` **local mode** β†’ **server + client modes (+ lock handoff)** β†’ **offline sync + conflict queue** β†’ **OIDC/Authentik auth + per-user isolation** β†’ `heph.nvim` + `heph` CLI. Local-only works standalone; runnable client/server + offline sync on the tailnet. +- **Phase 1 β€” v1 prototype (a single C1 effort, deliberately β€” a one-shot test of delivering a high-complexity prototype from the spec; built in TDD slices):** `heph-core` (model, schema, extraction, recurrence, "what is next", `Store` trait, op-log/HLC/CRDT merge) β†’ `hephd` **local mode** β†’ **server + client modes (+ lock handoff)** β†’ **offline sync + conflict queue** β†’ **OIDC/Authentik auth + per-user isolation** β†’ `heph.nvim` + `heph` CLI. Local-only works standalone; runnable client/server + offline sync on the tailnet. The build order doubles as the cross-session resume tracker (next un-green slice = where to resume). C2/Mikado is *not* used: it sequences prerequisites against existing code under test, and this is greenfield delivery from a complete spec; follow-up C1s or a C2 refactor come later as needed. - **Phase 2 β€” k3s deployment:** Daggerβ†’Zot image, ArgoCD app + Kustomize manifests, external-secrets; hub on blumeops. - **Phase 3 β€” Web UI** on the hub. - **Phase 4 (later, optional)** β€” calendar integration (careful CalDAV); migration from ZK / Todoist; iOS / Apple Watch voice capture; Hermes-style planning mode. @@ -407,10 +413,10 @@ Core: `rusqlite` (bundled) + migrations Β· `tokio` + JSON-RPC/unix-socket (local ### 11.3 First-session kickoff checklist 1. `mise run ai-docs`; read **[[tech-spec]]** (the build spec) and this doc's Β§3/Β§6 for rationale. -2. **Classify as C1** β€” greenfield, no existing system to Mikado-untangle. Single long-lived feature branch + early draft PR, docs-first, push as you go ([[agent-change-process]]). +2. **Classify as a single C1** (deliberate β€” a one-shot test of delivering this high-complexity prototype from the spec; C2/Mikado is for sequencing prerequisites against *existing* code, not greenfield delivery). Single long-lived feature branch + early draft PR, docs-first, push as you go ([[agent-change-process]]). The Β§11.1 build order is the resume tracker; the hairy slices (sync/CRDT) may spin off follow-up C1s rather than blocking the prototype. 3. Scaffold the cargo workspace + `heph.nvim` skeleton; **fill the AGENTS.md Project Structure section** (last template TODO). 4. Build outward in testable slices (TDD, tech-spec Β§2/Β§9): `heph-core` (schema, model, extraction, recurrence, "what is next", `Store` trait, **op-log/HLC/CRDT merge**) β†’ `hephd` **local mode** (LocalStore + lock + local RPC) β†’ **server + client modes** (network endpoint, RemoteStore, lock handoff) β†’ **sync + conflict queue** β†’ **OIDC auth** β†’ `heph.nvim` + `heph` CLI. -5. Confirm the kickoff-open picks: recurrence model (a vs b, Β§3.3), CRDT lib (`yrs` vs `automerge`), hub transport, context-item editing surface (A vs B, Β§6.3). +5. Confirm the kickoff-open picks: CRDT lib (`yrs` vs `automerge`), hub transport (`axum` vs gRPC) + propagation cadence + device-id provisioning. (Recurrence = roll-forward and the context-item backend = Fork A are now decided; only the nvim *affordance* A vs B is trialed in-prototype.) ## Related diff --git a/docs/reference/tech-spec.md b/docs/reference/tech-spec.md index 65f41b0..24736bf 100644 --- a/docs/reference/tech-spec.md +++ b/docs/reference/tech-spec.md @@ -36,17 +36,23 @@ Hephaestus (heph) is a self-hosted personal context-management system that unifi - **`LocalStore`** β€” a SQLite file opened **directly**, acquiring an **exclusive lock** on open (advisory `flock` on the DB file / a sidecar `.lock`). Refuses to start if the file is already locked. - **`RemoteStore`** β€” no local file; proxies every operation to a `server` over the network. -`hephd` runs in one of **three modes**: +**Two orthogonal axes, not one knob.** A runtime is configured along two independent axes plus one optional capability: -| Mode | Backend | Network listener | Offline | Notes | -|---|---|---|---|---| -| **local** | `LocalStore` | none | yes | standalone single-host; the simplest deployment | -| **server** | `LocalStore` | yes (authenticated) | yes | fronts the file for remote clients; **the sync hub** | -| **client** | `RemoteStore` β†’ a server | n/a | **no** | thin; online-only convenience | +1. **Backend** β€” `LocalStore` (own replica + op-log) vs. `RemoteStore` (proxy to a server). +2. **Inbound listener** β€” does it expose an authenticated network endpoint others connect to (i.e. is it a hub)? +3. **Outbound sync** *(optional, `LocalStore` only)* β€” an optional **`hub_url`**; when set, the instance is a **spoke** that background-syncs its op-log to that hub. Independent of whether it also listens. -**Lock handoff (the key flexibility):** `local` and `server` both take the file's exclusive lock, so **only one can own a given DB file at a time**. Kill the server β†’ lock releases β†’ a `local` process can open *the exact same file* and just work. Stop it β†’ relaunch in `server` mode β†’ remote clients reconnect. A `client` process never opens the file, so it never contends. +The named **modes** are presets over those axes: -**Where offline + sync live:** offline capability comes from a **local-backed** instance (`local` or `server`) β€” a full replica with its own op-log. Two *different machines* each run a local-backed instance and **sync** op-logs through a hub (a `server`-mode instance), which is where automatic merge + the **conflict queue** (Β§12) happen; sync never blocks local work, and the hub need not always be online. A `client`-mode instance is **online-only** (no replica, no offline) β€” the convenience option (e.g. on the hub host, or an always-connected client). Same-file local↔server is single-host graceful-degradation; device↔device sync uses *separate replicas* that reconcile. +| Mode | Backend | Inbound listener | `hub_url` | Offline | Role | +|---|---|---|---|---|---| +| **local** | `LocalStore` | none | optional | yes | everyday device β€” standalone if `hub_url` unset, a **syncing spoke** if set | +| **server** | `LocalStore` | yes (authenticated) | n/a (it *is* the hub) | yes | the sync **hub**; fronts the file and serves remote clients | +| **client** | `RemoteStore` β†’ a server | n/a | n/a | **no** | thin online-only convenience (no replica) | + +"Inbound listener: none" on `local` means *no inbound endpoint* β€” it does **not** preclude **outbound** sync. The everyday device (Gilbert, ringtail) runs **`local` + `hub_url`**: a full offline-capable replica that syncs to the blumeops hub. A `local` instance with no `hub_url` is the first-class, fully-offline standalone config. `client` is the escape hatch for "don't keep a replica here" (a borrowed box, CI, the future web-UI backend). + +**Lock handoff (the key flexibility):** `local` and `server` both take the file's exclusive lock, so **only one can own a given DB file at a time**. Kill the server β†’ lock releases β†’ a `local` process can open *the exact same file* and just work. Stop it β†’ relaunch in `server` mode β†’ remote clients reconnect. A `client` process never opens the file, so it never contends. Each machine takes the lock on its *own* file; **device↔device sync uses separate replicas that reconcile through the hub** (Β§12), never a shared file. ## 4. Data model @@ -69,24 +75,28 @@ All first-class entities are **nodes**; relationships are **links**. Markdown bo ### 4.3 Task semantics - **Attention-state** (required on committed tasks): `white` (do once do-date arrives), `orange` (top of mind), `red` (top of mind + a consequence exists if late β€” *consequence, not severity*), `blue` (on-deck/backlog). -- **do-date** = *earliest actionable date* ("do date"), **not a deadline**. Optional **late-on** marks when lateness becomes a problem. -- **Commitment axis:** `committed = 1` tasks participate in scheduling/"what is next"; `committed = 0` are **ephemeral context items** scoped to a container (`container_id`), with only `outstanding`/done states, never surfaced globally. Context items may be **promoted** to committed tasks. +- **do-date** = *earliest actionable date* ("do date"), **not a deadline** and **not an urgency signal** β€” a boolean candidacy gate only (Β§7). Optional **late-on** marks when lateness becomes a problem; it is the *sole* urgency signal. +- **Commitment axis (Fork A β€” [[design]] Β§6.3):** a **committed task** is a `task` node (a row in `tasks`) and participates in "what is next". A **context item** is *not* a synced node β€” it is a `- [ ]` line in a container `doc` body (its `[ ]`/`[x]` is its only state), a **locally-derived index** over the converged body (Β§5) that converges for free under sync. Identity is pinned only at **promotion**, which mints a committed `task` node and rewrites the source line into a link to it. - **States:** `outstanding`, `done`, `dropped` (done and dropped are both "not outstanding"; the distinction is retained). - **No hard deletes:** everything uses `tombstoned`; physical deletion only in an explicit cleanup mode. ### 4.4 Recurrence (Β§3.3 of [[design]]) -A `task` with a non-null `recurrence` (RFC-5545 RRULE) is a **recurring definition**. Sub-items flagged `is_template = 1` form its **checklist template**. **Each occurrence produces a fresh checklist instance** (new `outstanding` context items copied from the template); **completion never carries forward across occurrences** β€” this is a hard requirement. +A `task` with a non-null `recurrence` (RFC-5545 RRULE) is a **recurring definition**. Its checklist is the set of `- [ ]` lines in its canonical context `doc` body β€” under Fork A the body *is* the template (no separate template flag). **Each occurrence presents a fresh, all-unchecked checklist**, and **completion never carries forward across occurrences** β€” a hard requirement. -Two candidate implementations (pick at kickoff; (a) is the lean): -- **(a) Occurrence instances:** definition spawns a `task_occurrences` row per occurrence, each with its own do-date and fresh checklist items. Full history. -- **(b) Roll-forward in place:** single node; on completion, log the occurrence, reset the checklist to `outstanding`, advance the do-date. +**Model: roll-forward in place** (decided; the occurrence-instances alternative was rejected because, under Fork A, each occurrence would need its own body and thus its own node β€” the explosion Β§6.6 forbids). A recurring task is a **single node**. On `task.set_state(done)`: + +1. append the completed occurrence to the task's log (`log-of`); +2. reset the context doc's checkboxes to unchecked (a body-CRDT edit); +3. advance `do_date` to **the next RRULE instance strictly after `now`** β€” *skipping* missed occurrences rather than enqueuing them. + +So missing a daily routine for three days leaves **one** gently-overdue item, never a pile; the gap is implied by the hole in the log, not by overdue rows. RRULE expansion is **lazy** β€” only "the next instance after now" is ever computed, never the series (Β§6.6). A `skipped` state advances the do-date the same way without logging a completion. History is narrative (the log), per "narrative > list". ### 4.5 SQLite schema (starting point) ``` nodes( - id TEXT PRIMARY KEY, -- ULID + id TEXT PRIMARY KEY, -- ULID (content nodes); deterministic fn(owner,key) for journal/tag (Β§3.1 of [[design]]) owner_id TEXT NOT NULL REFERENCES users(id), -- per-user isolation kind TEXT NOT NULL, -- doc|task|project|tag|journal title TEXT NOT NULL, @@ -98,26 +108,13 @@ nodes( tombstoned INTEGER NOT NULL DEFAULT 0 ) -tasks( +tasks( -- committed tasks only; context items live in doc bodies (Β§4.3) node_id TEXT PRIMARY KEY REFERENCES nodes(id), - attention TEXT, -- white|orange|red|blue (committed tasks) - do_date INTEGER, -- epoch ms, nullable - late_on INTEGER, -- epoch ms, nullable + attention TEXT, -- white|orange|red|blue + do_date INTEGER, -- epoch ms, nullable; boolean candidacy gate only (Β§7) + late_on INTEGER, -- epoch ms, nullable; the sole urgency signal (Β§7) state TEXT NOT NULL, -- outstanding|done|dropped - committed INTEGER NOT NULL, -- 1 committed task, 0 context item - container_id TEXT REFERENCES nodes(id), -- context item β†’ container - recurrence TEXT, -- RRULE; present = recurring definition - is_template INTEGER NOT NULL DEFAULT 0 -- checklist-template item -) - --- recurrence model (a) only: -task_occurrences( - id TEXT PRIMARY KEY, -- ULID - def_id TEXT NOT NULL REFERENCES nodes(id), - occurrence_date INTEGER NOT NULL, - state TEXT NOT NULL, -- outstanding|done|dropped|skipped - created_at INTEGER NOT NULL, - tombstoned INTEGER NOT NULL DEFAULT 0 + recurrence TEXT -- RRULE; present = recurring definition (roll-forward, Β§4.4) ) links( @@ -135,7 +132,7 @@ nodes_fts -- FTS5 over title, bod -- identity & sync -- users( id TEXT PRIMARY KEY, -- ULID - oidc_sub TEXT UNIQUE NOT NULL, -- OIDC subject (Authentik) + oidc_sub TEXT UNIQUE, -- OIDC subject (Authentik); NULL = local/unlinked user (Β§13) name TEXT, created_at INTEGER NOT NULL ) @@ -144,7 +141,7 @@ oplog( -- append-only operation log (the sync unit id TEXT PRIMARY KEY, -- ULID owner_id TEXT NOT NULL REFERENCES users(id), hlc TEXT NOT NULL, -- hybrid logical clock (causal order) - origin TEXT NOT NULL, -- originating device id + origin TEXT NOT NULL, -- originating device id (provisioning TBD at sync slice, Β§12) op_type TEXT NOT NULL, -- node.create|node.body_delta|task.set_field|link.add|link.remove|... target_id TEXT NOT NULL, payload TEXT NOT NULL, -- JSON (e.g. CRDT delta, field+value, OR-Set add/remove) @@ -152,7 +149,7 @@ oplog( -- append-only operation log (the sync unit ) sync_state( -- per-peer cursor (device ↔ hub) - peer TEXT PRIMARY KEY, -- 'hub' on a client; device id on the hub + peer TEXT PRIMARY KEY, -- 'hub' on a spoke; device id on the hub last_pushed_hlc TEXT, last_pulled_hlc TEXT, updated_at INTEGER NOT NULL @@ -170,15 +167,17 @@ conflicts( -- ambiguous merges surfaced to the user ) ``` -Projects/tags are `nodes`; membership is `links` (`in-project`, `tagged`). All `tasks`/`task_occurrences`/`links` rows inherit ownership via their node(s). +Projects/tags are `nodes`; membership is `links` (`in-project`, `tagged`). All `tasks`/`links` rows inherit ownership via their node(s). **Context items are not in this schema** β€” they are `- [ ]` lines in `doc` bodies (Β§4.3, Β§5); a daemon may keep a *local, non-synced* derived index of them for the plugin, rebuilt from bodies and never written to the op-log. **Deterministic ids** for `journal`/`tag` let two offline replicas that independently create the same day's journal or the same-named tag converge automatically; the tag-name normalization function is a fixed, versioned constant so every device derives byte-identical ids (tag rename = retag, not in-place key change). ## 5. Markdown handling -- Bodies are stored verbatim. On write (node create/update), `heph-core` extracts: +- **The body is the source of truth** and merges as a **text CRDT** (`body_crdt`); the `body` TEXT column is its materialized view. A full-body write (`node.update({body})`) is **diffed into the CRDT** (yrs computes the delta), so surfaces can send whole-buffer text without knowing about deltas. +- On write, `heph-core` derives from the body: - `[[wiki-links]]` β†’ `wiki` links (resolved via `aliases`/title; unresolved links are allowed and recorded). - - GFM task-list items (`- [ ]` / `- [x]`) β†’ context-item state under the node (Option A editing model β€” see [[design]] Β§6.3; the alternative is command-driven items, decided in-prototype). -- Extraction is idempotent and diff-based: re-writing an unchanged body is a no-op; reworded checklist lines tombstone-old + add-new (context items are cheap). -- `export` materializes all non-tombstoned nodes to a directory tree of `.md` files (frontmatter + body), reproducing the corpus portably. + - GFM task-list items (`- [ ]` / `- [x]`) β†’ the **local context-item index** (Fork A β€” see [[design]] Β§6.3). This index is **derived, not synced**: each replica rebuilds it from its converged CRDT body, so context items converge for free and never get divergent ULIDs. Context-item identity is pinned only at **promotion** (Β§6). +- Derivation is idempotent and diff-based: re-writing an unchanged body is a no-op. +- **Tombstoned nodes are excluded** from the derived index, `nodes_fts`, `next`/`list`, and `export`. +- `export` materializes all non-tombstoned nodes to a directory tree of `.md` files (**frontmatter β€” id, kind, task scalars, aliases, links β€” plus body**), a faithful one-way portable snapshot (no import in v1). ## 6. Daemon RPC API (JSON-RPC over unix socket) @@ -191,51 +190,62 @@ Methods (request β†’ response; errors are JSON-RPC errors). Signatures are indic - `task.create({title, project?, attention?, do_date?, late_on?, recurrence?, committed?}) β†’ Task` (auto-creates the canonical context `doc` + `canonical-context` link) - `task.set_state({id, state}) β†’ Task` (recurring: advances per Β§4.4) - `task.set_attention({id, attention}) β†’ Task` -- `task.promote({context_item_id, attention?, project?}) β†’ Task` -- `next({scope?, limit?}) β†’ [RankedTask]` (the "what is next?" query, Β§7) +- `task.promote({container_id, item_ref, attention?, project?}) β†’ Task` (mints a committed task from a context-item line and rewrites that line into a link to it, Β§4.3) +- `next({scope?, limit?}) β†’ [RankedTask]` (the Tactical blank-slate "what is next?" ranking, Β§7) +- `list({scope?, attention?, include_blue?, include_future?, group_by?}) β†’ [Task]` (enumeration for the Organizational view β€” the whole set incl. backlog) - `search({query, filters?}) β†’ [Node]` (FTS) - `links.outgoing(id) β†’ [Link]` / `links.backlinks(id) β†’ [Link]` - `journal.open_or_create(date) β†’ Node` - `log.append({task_id, text}) β†’ ok` (append to the task's `log-of` node) +- `log.tail({task_id, n?}) β†’ [Entry]` (read the task's latest log entries β€” the resumption breadcrumb, [[design]] Β§6.1) - `export({path}) β†’ {count}` - `health() β†’ {orange_count, active_count, on_deck_count, conflict_count, sync_status, ...}` (working-set + sync indicators) - **auth:** `auth.login() β†’ {device_code_flow...}` / `auth.status() β†’ {user, logged_in}` / `auth.logout()` (Β§13) - **sync:** `sync.now() β†’ {pushed, pulled, conflicts}` (force a sync cycle; background sync runs automatically) / `sync.status() β†’ {last_pushed, last_pulled, pending_ops, online}` - **conflicts:** `conflicts.list() β†’ [Conflict]` / `conflicts.resolve({id, choice}) β†’ ok` -Every local mutation method records an op in the **op-log** for sync. The local daemon supports **server-push** notifications (e.g. a node changed by an incoming sync) so an open buffer can reconcile in real time. +The daemon is **mode-agnostic** β€” Tactical/Strategic/Organizational are plugin-side compositions of these primitives (Β§8), not daemon concepts. Every local mutation method records an op in the **op-log** for sync. The local daemon supports **server-push** notifications (e.g. a node changed by an incoming sync) so an open buffer can reconcile in real time. ### 6.1 Hub (server-mode) sync endpoint Separate from the local unix-socket RPC, the hub exposes an **authenticated network endpoint** (HTTP/JSON or gRPC β€” pick at kickoff) for op exchange: clients present an OIDC bearer token (Β§13); the hub validates, scopes by `owner_id`, accepts pushed ops, and returns ops the client hasn't seen (by HLC cursor). The hub applies the same `heph-core` merge logic. -## 7. "What is next?" ranking +## 7. "What is next?" ranking (Tactical blank-slate) -Given an optional `scope` (project/context) and `limit` (default 5): +This is the **Tactical** ranking only; Strategic/Organizational are other plugin-side views over different primitives (Β§6, Β§8). Given an optional `scope` and `limit` (default 5), the engine is **two stages β€” a filter, then an order expressed as data** so the sort is trivially reorderable later. -1. **Candidates:** committed tasks where `state = outstanding`, not tombstoned, `attention β‰  blue`, and actionable now: `do_date IS NULL OR do_date ≀ now`. For recurring definitions, evaluate the current active occurrence's do-date. Apply `scope` if given. -2. **Order:** - 1. attention: `red` β†’ `orange` β†’ `white`; - 2. urgency: tasks past `late_on` first, then most-overdue (smallest `do_date`) first; - 3. tie-break: earlier `do_date`, then `created_at`. +1. **Filter (candidacy)** β€” a pure predicate: `committed` ∧ `state = outstanding` ∧ Β¬`tombstoned` ∧ `attention β‰  blue` ∧ **actionable** (`do_date IS NULL OR do_date ≀ now`) ∧ in `scope`. **`do_date` is used *only here*** β€” a boolean "can this be done now?" gate, never an urgency input. For a recurring task, the single rolled-forward node's `do_date` is the gate (Β§4.4). +2. **Order** β€” an **ordered list of named sort dimensions**, applied lexicographically; reordering this list (one place) reshuffles the ranking: + + ``` + RANKING = [ PastLateOn(desc), LateOverdueAmount(desc), Attention(red>orange>white), CreatedAt(asc) ] + ``` + + - **`late_on` is the sole urgency signal** and forms a global top tier: items past `late_on` float above everything (incl. `red`), most-past first. This is the only legitimate "overdue" β€” `late_on` is a deliberately-set problem marker, not an actionability date. *(Global-tier vs. within-band placement is expected to be revisited; it's just a reordering of `RANKING`.)* + - then **attention band** `red β†’ orange β†’ white`; + - tie-break **`created_at` ascending (FIFO)**. **Age is never urgency** β€” an item is not hotter for having been actionable longer; raise its attention or set a `late_on` to escalate it deliberately. 3. **Output:** concise rows β€” title, project, attention, do/late, link to canonical context. `red` items always appear regardless of `limit`. -`blue` (on-deck) is hidden from `next` by design; surfaced only by an explicit on-deck view. `health()` exposes the working-set tensions (orange ≀ 6, active ≀ ~30, on-deck count) honestly β€” never masking overload nor manufacturing calm. +`blue` (on-deck) is hidden from `next` by design; surfaced only by `list` (Β§6). `health()` exposes the working-set tensions (orange vs. 6, active vs. ~30, on-deck count) **with over-threshold deltas**, honestly β€” never masking overload nor manufacturing calm. ## 8. heph.nvim surface (v1) -Replaces obsidian.nvim. Telescope-backed. Core commands/gestures: +Replaces obsidian.nvim. Telescope-backed. **Tactical / Strategic / Organizational are named plugin-side views** composed from the mode-agnostic daemon primitives (Β§6) β€” the daemon never infers a mode. The session **goal stack** is plugin state (no persistent stack in v1; the durable re-seed is the per-task breadcrumb, `log.tail`). Core commands/gestures: - Follow `[[wiki-link]]` under cursor on ``. - Search / quick-switch / tags / backlinks / outgoing links (pickers). - Daily journal picker (create/open dated `journal` nodes). -- Task capture; show "what is next" (`:Heph next`); set attention; mark done/dropped. -- Open a task's canonical context doc; edit context-item checkboxes (Option A) in the buffer (extracted on `:w`). +- Task capture; show "what is next" (`:Heph next`, Tactical); `list` views (Organizational); set attention; mark done/dropped. +- Open a task's canonical context doc; edit context-item checkboxes (Fork A) in the buffer (derived on `:w`). - Per-task log quick-append without leaving the current buffer. +**Deferred (fast-follow, scaffolded):** guided working-set rituals β€” the Blue keep/drop review and Orange daily reconfirm β€” are pure compositions of `list` + `set_attention` + `set_state(dropped)`; v1 ships `health()` reporting and manual review. + +**Known-hard:** reconciling an incoming CRDT body delta into a *dirty* buffer (unsaved local edits, cursor position) β€” the Β§9 "update arrives while a buffer is open" case β€” is genuinely fiddly under Fork A; expect to iterate. + ## 9. Testing strategy (TDD, layered) -All layers are required; CI runs them on every push/PR (extend `.forgejo/scripts/build` to run `cargo test` and the nvim e2e suite; `prek` already runs in `build.yaml`). +All layers are required; CI runs them on every push/PR (extend `.forgejo/scripts/build` to run `cargo test` and the nvim e2e suite; `prek` already runs in `build.yaml`). **The Forgejo runner image must provide `neovim` + `plenary.nvim`** for the headless e2e suite. - **Unit (`heph-core`):** model invariants; markdown extraction (wiki-links, checkboxes); **RRULE expansion and the fresh-checklist-per-occurrence rule** (assert completion never carries forward); the "what is next?" ranking (table-driven cases); migration up/down. - **Property tests (`proptest`):** ranking yields a total order; extraction is idempotent; recurrence never leaks completion state across occurrences; tombstones are never resurrected. @@ -300,11 +310,15 @@ See [[design]] Β§5–§7 for the constraints later phases impose on present choi **Determinism:** HLCs are clock-injected; op application is idempotent and order-independent given HLC. These are the core invariants the sync tests assert. -**Open at kickoff:** CRDT lib confirmation (`yrs` vs `automerge`); hub transport (`axum` HTTP/JSON vs gRPC); propagation cadence (push vs. periodic pull); exactly which fields are "meaningful" enough to enqueue vs. silently LWW. +**Open at kickoff:** CRDT lib confirmation (`yrs` vs `automerge`); hub transport (`axum` HTTP/JSON vs gRPC); propagation cadence (push vs. periodic pull); exactly which fields are "meaningful" enough to enqueue vs. silently LWW; **device-id provisioning** (how a spoke mints its `origin` id and registers with the hub). ## 13. Authentication - **OIDC against Authentik.** Clients authenticate via the **OAuth 2.0 device-code flow** (`auth.login`); the resulting tokens are cached in the OS keychain (`keyring`) / 1Password and refreshed automatically. Offline devices operate on cached credentials. +- **Auth is a network-boundary concern; ops are authenticated at *exchange*, not creation.** Creating ops needs only local trust, so losing the IdP/hub never blocks local work β€” ops queue in the op-log and the bearer token gates only the push/pull session. The one degraded case: the refresh token expiring during a long offline stretch β†’ re-run the device-code flow once on reconnect, then the queued op-log drains. +- **Local-only needs no auth.** A `local` instance with no `hub_url` runs against a single **local user** (`users` row, generated `id`, `oidc_sub = NULL`) β€” friction-free, no IdP. `oidc_sub` is therefore nullable. +- **The hub is authoritative for user identity.** On first authentication it maps `sub` β†’ a canonical `users.id` (creating it if new) and hands the *same* id to every device that authenticates as that human β€” no device-vs-device id reconciliation. +- **Adoption (local-only β†’ authed)** is a **one-time, pre-first-sync** rewrite: on a previously-local replica's first authenticated sync, fetch the canonical `users.id`, then in one transaction rewrite `owner_id` (and the owner-embedded deterministic ids of Β§4.5 + the links referencing them) from the local placeholder to the canonical id. Safe precisely because it precedes any sync (no remote refs, no concurrent writers); afterward `owner_id` is immutable. Born-authed replicas stamp the canonical id from the start and never rewrite. - **Hub enforcement:** the sync endpoint requires a valid **OIDC bearer token**; the hub maps the token's `sub` to a `users` row and **scopes every op by `owner_id`**. No cross-user reads/writes. - **Per-user isolation:** all nodes (and their dependent rows) carry `owner_id`; queries and sync are always user-scoped. In practice a single user (`eblume`), but the isolation is real from day one. - **Local trust:** the local unix-socket RPC trusts the OS user (file-permission-scoped socket); app-level auth is for the **network** boundary (device ↔ hub).