From 2166b932b638b7fa6d1dabe68a033dadc2ceefa2 Mon Sep 17 00:00:00 2001 From: Julien Calixte Date: Mon, 13 Jul 2026 00:53:50 +0200 Subject: [PATCH] docs(sync): update the wiring section to how-it-landed The merged plan section now records the shipped design: single path set (working tree as truth) instead of {changed, deleted}, the dirty journal, stranded-commit recovery, radio-free up-to-date, and the soft-reset reconcile with its carried-files side win. --- docs/tradeoff-curves/sync-commit-staging.md | 157 ++++++++++---------- 1 file changed, 75 insertions(+), 82 deletions(-) diff --git a/docs/tradeoff-curves/sync-commit-staging.md b/docs/tradeoff-curves/sync-commit-staging.md index 1be3692..dffe927 100644 --- a/docs/tradeoff-curves/sync-commit-staging.md +++ b/docs/tradeoff-curves/sync-commit-staging.md @@ -16,10 +16,10 @@ > window cache was **removed entirely** (0 hits across four instrumented runs — > `mwindow` absorbs any true repetition above `p_mmap`). The sub-second bar > failed, but ~2–2.8 s against 611 s/OOM for every alternative ships: a full -> cold real-repo `:sync` lands at ~9–10 s. **Decision: wire the splice in** — -> the [implementation plan](#the-fix--wiring-the-odepth-splice-into-the-firmware) -> below (merged from the retired `notes/sync-commit-handoff.md`) is the working -> checklist. Shrinking the repo is **not** an option +> cold real-repo `:sync` lands at ~9–10 s. **Decision: wire the splice in — +> done 2026-07-13** ([how it landed](#the-fix--wiring-the-odepth-splice-into-the-firmware), +> merged from the retired `notes/sync-commit-handoff.md`; on-device `:sync` +> against the real repo still pending). Shrinking the repo is **not** an option > ([`../notes/git-sync-images-and-repo-size.md`](../notes/git-sync-images-and-repo-size.md): > the images are load-bearing for another app), which is exactly why the O(depth) > mechanism is the only lever left. This note records the cost model and the full @@ -491,8 +491,10 @@ threading, FD budget) is the > Merged from `docs/notes/sync-commit-handoff.md` (written 2026-07-12, retired > 2026-07-13 once the bench phase closed). The handoff's bench half is done — -> the splice op lives in `git_bench` and the numbers above are its output. What -> follows is the still-open firmware plumbing, updated to the run-5 final state. +> the splice op lives in `git_bench` and the numbers above are its output. +> **The firmware plumbing below SHIPPED 2026-07-13** (compile-verified both +> feature flavors; on-device `:sync` against the real repo still pending) — +> each item now records how it landed rather than what to do. ### The splice walk @@ -503,90 +505,81 @@ carries every unchanged entry (all 260 images, the other 1176 files) forward untouched — which means **the device doesn't even need the images in its working tree.** -Sketch (git2 0.20 API — all present: `Repository::treebuilder(Option<&Tree>)`, -`TreeBuilder::{insert,remove,write}`, `Tree::{get_name,get_path}`, -`TreeEntry::{to_object,filemode}`): +Shipped as `git_sync::splice` (git2 0.20: `Repository::treebuilder(Option<&Tree>)` ++ `TreeBuilder::{insert,remove,write}`). Signature: ```rust -/// Return a new tree OID = `base` with `path` set to `new` (Some(blob_oid) to -/// add/replace, None to delete). Reads ~depth subtree objects, writes ~depth trees. -fn splice(repo: &Repository, base: &Tree, path: &[&str], new: Option) -> Result { - let (head, rest) = path.split_first().unwrap(); - if rest.is_empty() { - // leaf level: patch this tree directly - let mut tb = repo.treebuilder(Some(base))?; - match new { - Some(oid) => tb.insert(head, oid, 0o100644)?, // FileMode::Blob - None => { tb.remove(head).ok(); } // delete - }; - return Ok(tb.write()?); - } - // descend: find (or synthesize) the subtree, recurse, re-insert its new OID - let sub = base.get_name(head) - .and_then(|e| e.to_object(repo).ok()) - .and_then(|o| o.peel_to_tree().ok()); - let empty; let sub_ref = match &sub { Some(t) => t, None => { empty = repo.treebuilder(None)?; /* ... */ unreachable!() } }; - let new_sub = splice(repo, sub_ref, rest, new)?; - let mut tb = repo.treebuilder(Some(base))?; - // if new_sub is the empty tree and this was a delete, remove the dir entry instead - tb.insert(head, new_sub, 0o040000)?; // FileMode::Tree - Ok(tb.write()?) -} +fn splice(repo: &Repository, base: Option<&Tree>, path: &[&str], blob: Option) -> Result ``` -Then, for the dirty set, fold each change through `splice` (thread the running -root tree), and `commit(Some("HEAD"), …, &final_tree, &[&parent])`. Edge cases -to handle: new files where an intermediate dir doesn't exist yet (build subtree -from `None`), deletes that empty a directory (remove the dir entry rather than -insert an empty tree), and path splitting on `/` (paths are repo-relative -POSIX). The benched reference is `git_bench`'s `splice stage→tree` op. +`Some(blob)` inserts/replaces, `None` removes; `base: None` synthesizes a +missing intermediate directory on the way down, and a directory emptied by a +remove is pruned on the way up (the empty tree is never re-inserted). +`stage_and_commit` folds every dirty path through it (threading the running +root tree), then `commit(Some("HEAD"), …)` exactly as before — the +`tree unchanged → nothing to publish` check and the `commit split —` timing +log survive. The benched reference is `git_bench`'s `splice stage→tree` op. -### `firmware/src/git_sync.rs` +### `firmware/src/git_sync.rs` — how it landed -- **Set the mwindow options at service start** (before the first - `Repository::open`): `git2::opts::set_mwindow_size(256 * 1024)` + - `set_mwindow_mapped_limit(4 * 1024 * 1024)`. Today only `git_bench.rs` sets - them — the shipping service runs libgit2's 32-bit defaults (**32 MB window / - 256 MB mapped limit**, mwindow.c:16), so the first pack access on the 570 MB - clone would try to `git__malloc` a 32 MB window and die on the 8 MB PSRAM - heap before the walk even runs. Run 5 sharpened the stakes: the ~1.85 MB - "resident" in the final bench **is** mwindow's live open-window set, so these - opts are the only thing bounding it. -- Rewrite `stage_and_commit` (currently ~L271–332) to the `splice`-walk above. - Drop `add_all`/`update_all`/`index.write`/`index.write_tree`. Keep the - `commit split —` timing log, the `tree unchanged → nothing to publish` check, - and the signature/message code. -- `PublishRequest` (~L79) is currently an **empty struct** — it must carry the - dirty set: `{ changed: Vec<(String /*repo-rel path*/, ...)>, deleted: Vec }`. - The commit needs the blob content or a way to read it; simplest is to pass the - paths and let the git thread `repo.blob_path(abs)` / read `/sd/repo/`. -- `reconcile_onto_origin` (~L377/L394) uses `repo.reset(Mixed)` — with an - index-free commit there's no index to reset; switch to `ResetType::Soft` (move - HEAD only) or drop the reset and just re-`splice` onto the new origin tip. -- The macOS-cruft filter (`skip_macos_cruft`) is no longer needed — the walk only - ever touches paths the editor explicitly hands it, so `._*`/`.DS_Store` can't - sneak in. (Keep a note; don't silently lose the Spike-14 lesson.) -- **Deliberate behavior change to record:** the walk commits *only* the - editor's dirty set. Files changed on the card outside the editor (e.g. the - card mounted on a Mac) were swept in by `add_all` before; they will now never - be committed, and the working tree will show a permanent diff against HEAD if - inspected on a desktop. Correct for the appliance (it's also what makes the - cruft filter unnecessary), but it must be intentional, not accidental. +- **mwindow options set at service start** (top of `run_git_service`, before + any `Repository::open`): `set_mwindow_size(256 KB)` + + `set_mwindow_mapped_limit(4 MB)`. Without them the 32-bit defaults (32 MB + window / 256 MB mapped limit, mwindow.c:16) would `git__malloc` a 32 MB + window on the first pack access and die on the 8 MB PSRAM heap. Run 5 + sharpened the stakes: the ~1.85 MB bench "resident" **is** mwindow's live + open-window set, so these opts are the only thing bounding it. +- `stage_and_commit(repo, paths)` is the splice walk; `add_all`/`update_all`/ + `index.write`/`write_tree` are gone. **The request carries one path set, not + `{changed, deleted}`:** the working tree is the source of truth, so at commit + time a recorded path that exists on the card is spliced in from disk and a + missing one is spliced out. An unchanged path is a no-op — over-reporting is + free, which is what makes the retry/journal semantics below simple. +- **Stranded-commit recovery (new):** when the splice yields the parent's tree + (nothing to commit), the service now compares HEAD against + `refs/remotes/origin/` and still pushes if origin lacks HEAD — a + previous cycle that committed and then failed mid-push used to strand that + commit forever (the old path reported "up to date" and never retried). +- **Radio-free up-to-date (new):** an empty dirty set + origin already at HEAD + short-circuits before Wi-Fi even comes up — `:sync` with nothing to do + answers in ~150 ms instead of a Wi-Fi/TLS round. +- `reconcile_onto_origin` now `ResetType::Soft` (ref move only) — there is no + index to reset, and a Mixed reset's index write is exactly the racy-clean + wall the splice avoids. Side win: a remote-only added file is now *carried* + by the replay (origin's tree is the splice base) where the old `add --all` + replay dropped it. +- The macOS-cruft filter (`skip_macos_cruft`) is gone with the walk — the + splice only touches paths the editor recorded, so `._*`/`.DS_Store` can't + sneak in the way they once did (07d87772, the Spike-14-era lesson). +- **Deliberate behavior change (now in effect):** only paths the editor + saved/deleted are ever committed. Files changed on the card outside the + editor (card mounted on a Mac) were swept in by `add_all` before; they will + now never be committed, and the working tree will show a permanent diff + against HEAD if inspected on a desktop. Correct for the appliance; recorded + here so it reads as intent, not accident. -### Dirty-set source — `firmware/src/persistence.rs` + `main.rs` +### Dirty-set source — `firmware/src/persistence.rs` + `main.rs` (how it landed) -- Writes funnel through `Storage::save_path` (~L316) and deletes through - `Storage::delete_path` (~L352), both `&self`. Accumulate a dirty/deleted set - (needs `RefCell` interior mutability, or move the set up to `main.rs`). -- `Effect::Publish` handler in `main.rs` (~L222) builds the `PublishRequest` from - that set and clears it on a successful `Pushed`/`UpToDate` outcome. -- **FD budget: `main.rs` (~L413) mounts with `Storage::mount()` = 4 open - files, and the git thread shares that mount.** libgit2 keeps the pack + - `.idx` (+ commit-graph) descriptors open and opens loose objects on top — - that's why `git_bench` needed `mount_for_git` (16). On the real repo the - shipping `:sync` will fail with "no free file descriptors" long before any - latency question. Either mount with the 16-file budget in `main.rs` (the - editor's 2-FD peak coexists fine) or split the budgets some other way. +- `Storage` owns the record: `save_path`/`delete_path` note their repo-relative + path in a `RefCell` (paths outside `/sd/repo` are skipped), and the + set is **journaled to `/sd/.typoena-dirty`** — atomic write, rewritten only + when the set actually grows. Without the journal a power pull would strand + every file saved-but-not-published that session: nothing walks the tree + anymore, so an unrecorded change would never reach the remote. The journal + is loaded at mount and its paths ride the next `:sync`. +- Lifecycle: `take_dirty()` snapshots pending → in-flight for one publish + (journal keeps carrying the union); the outcome settles it — + `publish_succeeded()` forgets the snapshot and shrinks the journal, + `publish_failed()` returns it to pending for the next `:sync`. A save landing + *while* a publish runs re-enters pending and rides the next one. Recording + happens *before* the file write, so a crash between the two only + over-approximates (a no-op splice), never under-records. +- `Effect::Publish` in `main.rs` sends `PublishRequest { paths: take_dirty() }`; + the outcome handler in the idle branch calls the matching settle method. +- **FD budget:** a git build now mounts `Storage::mount_for_git()` (16 FDs) in + `boot_storage` — libgit2 keeps the pack + `.idx` descriptors open and opens + loose objects on top, which overruns the editor's 4-FD budget. The light + build keeps the editor's own budget. ### `esp_map.c` — nothing left to do