From 5f50cf1b4500318f4b060bf5b7bcaf7002aebdc5 Mon Sep 17 00:00:00 2001 From: Julien Calixte Date: Mon, 13 Jul 2026 11:16:34 +0200 Subject: [PATCH] chore: remove bar --- docs/kaizen/real-repo-sync.md | 99 +++++++++++++++++------------------ 1 file changed, 49 insertions(+), 50 deletions(-) diff --git a/docs/kaizen/real-repo-sync.md b/docs/kaizen/real-repo-sync.md index b1b595a..e517f31 100644 --- a/docs/kaizen/real-repo-sync.md +++ b/docs/kaizen/real-repo-sync.md @@ -10,11 +10,11 @@ device — only the toy test repo has. Full measurement trail: **Value model for the writer (Julien on the appliance)** -| Want more of | Do less of | -|---|---| -| + words safely on the remote with one keystroke | – waiting on a frozen device | -| + trust that `:sync` completes, every time | – power-cycling mid-sync and losing the session's trust | -| + keep writing while it publishes | – babysitting git from a desktop to rescue the card | +| Want more of | Do less of | +| ----------------------------------------------- | ------------------------------------------------------- | +| + words safely on the remote with one keystroke | – waiting on a frozen device | +| + trust that `:sync` completes, every time | – power-cycling mid-sync and losing the session's trust | +| + keep writing while it publishes | – babysitting git from a desktop to rescue the card | **Measurement**: seconds from `:sync` to the snackbar, on the real `jcalixte/notes` clone on the device. @@ -30,13 +30,12 @@ xychart-beta title "Improvement potential — :sync → snackbar" x-axis ["Before kaizen (~611 s, never completes)", "After kaizen (target ≤ ~10 s)"] y-axis "seconds" 0 --> 650 - bar [611, 0] + bar [611] line [10, 10] ``` The before bar understates the reality: a reset re-enters the freeze, so the -true value is ∞ — 0 successful syncs ever. The line is the ≤ ~10 s target; the -after column has no bar until the step-6 measurement lands. +true value is ∞ — 0 successful syncs ever. The line is the ≤ ~10 s target; ## 2) Current method analysis @@ -44,13 +43,13 @@ after column has no bar until the step-6 measurement lands. :sync (current method, real repo) │ ▼ -┌─ local commit ────────────────────────────────────┐ ┌─ network push ──────┐ -│ stage: add_all(["*"]) + update_all(["*"]) │ │ TLS + push │ -│ → stat()s 1179 files / 158 dirs over 10 MHz SPI │──▶│ ~6.5 s floor │ -│ index.write ⚡ ~611 s │ │ (separate curve) │ -│ → FAT's 2 s mtime marks ~every entry "racy" │ └─────────────────────┘ -│ → re-hashes ~170 MB (150 MB of images) │ -│ write_tree + commit-obj ⚡ seconds each │ +┌─ local commit ─────────────────────────────────────┐ ┌─ network push ──────┐ +│ stage: add_all(["*"]) + update_all(["*"]) │ │ TLS + push │ +│ → stat()s 1179 files / 158 dirs over 10 MHz SPI │──▶│ ~6.5 s floor │ +│ index.write ⚡ ~611 s │ │ (separate curve) │ +│ → FAT's 2 s mtime marks ~every entry "racy" │ └─────────────────────┘ +│ → re-hashes ~170 MB (150 MB of images) │ +│ write_tree + commit-obj ⚡ seconds each │ │ → loose writes trigger pack reads through │ │ emulated mmap; each far lseek walks the │ │ FAT cluster chain (~190 ms) │ @@ -62,16 +61,16 @@ Every factor was benched on the device (`sd_bench` for raw FAT, `git_bench` for git2 primitives on the real clone) — details and full tables in [`../tradeoff-curves/sync-commit-staging.md`](../tradeoff-curves/sync-commit-staging.md): -| Factor # | Factor | Hypothesis | Test method | Test result | True or false? | -|---|---|---|---|---|---| -| 1 | SD card raw I/O | The card itself is slow: a loose-object write costs ~700–900 ms | `sd_bench`: raw FAT create/write/rename composite, same card, same 10 MHz | complete loose-object composite **86 ms** — 8× under libgit2's number | **FALSE** | -| 2 | `index.write()` racy-clean | FAT's 2 s mtime granularity makes ~all 1179 entries look racy → full ~170 MB re-hash over SPI | `git_bench` on the real clone, 3 consecutive `index.write()` | **611 s → 12.8 s → 360 ms** — the decay is the racy set shrinking as the index mtime advances | **TRUE** | -| 3 | Index-free alternative | Skipping `index.write` (fresh in-memory index) escapes the wall | `git_bench`: `Index::new` + `read_tree(HEAD)` + `write_tree_to` | seed `read_tree` **77 s cold** + mmap grows to 7.4 MB → **zlib OOM crash** — still O(N_tree) | **FALSE** (as a fix) | -| 4 | FatFS `lseek` | Long/backward seeks walk the FAT cluster chain over SPI (~67 KB of FAT reads for a 263 MB file) | `sd_bench`: seek+read 4 KB @offset 0 vs @end of the pack; A/B with `CONFIG_FATFS_USE_FASTSEEK` | @0 = 5.8 ms, @end = **198.7 ms**; fast-seek → **20.4 ms** | **TRUE** | -| 5 | Free-cluster scan | A ~740 MB-full card slows FAT allocation | `sd_bench` re-run on the full card | composite 77 ms — unchanged | **FALSE** | -| 6 | Strict object creation | OID validation on commit/treebuilder causes the ~1.3 s commit premium | strict-off A/B in `git_bench` | 1.80 s vs 1.93 s — no premium removed | **FALSE** | -| 7 | Repeated small mmap windows | A window cache in `esp_map.c` would absorb the ~8 small pack reads per loose write | instrumented cache (v2), 4 real-repo runs | **0 hits / 313 misses** — every read hits a unique (offset, len); `mwindow` already absorbs true repetition | **FALSE** | -| 8 | Cache memory discipline | The OOM in factor 3 is our own cache holding buffers past `p_munmap`, defeating `MWINDOW_MAPPED_LIMIT` | run-4 memory monitor after evict-on-munmap fix, then full removal (run 5) | resident flat at 1833 KB, heap ≥ 6.2 MB, no OOM; removal I/O-neutral | **TRUE** | +| Factor # | Factor | Hypothesis | Test method | Test result | True or false? | +| -------- | --------------------------- | ------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | -------------------- | +| 1 | SD card raw I/O | The card itself is slow: a loose-object write costs ~700–900 ms | `sd_bench`: raw FAT create/write/rename composite, same card, same 10 MHz | complete loose-object composite **86 ms** — 8× under libgit2's number | **FALSE** | +| 2 | `index.write()` racy-clean | FAT's 2 s mtime granularity makes ~all 1179 entries look racy → full ~170 MB re-hash over SPI | `git_bench` on the real clone, 3 consecutive `index.write()` | **611 s → 12.8 s → 360 ms** — the decay is the racy set shrinking as the index mtime advances | **TRUE** | +| 3 | Index-free alternative | Skipping `index.write` (fresh in-memory index) escapes the wall | `git_bench`: `Index::new` + `read_tree(HEAD)` + `write_tree_to` | seed `read_tree` **77 s cold** + mmap grows to 7.4 MB → **zlib OOM crash** — still O(N_tree) | **FALSE** (as a fix) | +| 4 | FatFS `lseek` | Long/backward seeks walk the FAT cluster chain over SPI (~67 KB of FAT reads for a 263 MB file) | `sd_bench`: seek+read 4 KB @offset 0 vs @end of the pack; A/B with `CONFIG_FATFS_USE_FASTSEEK` | @0 = 5.8 ms, @end = **198.7 ms**; fast-seek → **20.4 ms** | **TRUE** | +| 5 | Free-cluster scan | A ~740 MB-full card slows FAT allocation | `sd_bench` re-run on the full card | composite 77 ms — unchanged | **FALSE** | +| 6 | Strict object creation | OID validation on commit/treebuilder causes the ~1.3 s commit premium | strict-off A/B in `git_bench` | 1.80 s vs 1.93 s — no premium removed | **FALSE** | +| 7 | Repeated small mmap windows | A window cache in `esp_map.c` would absorb the ~8 small pack reads per loose write | instrumented cache (v2), 4 real-repo runs | **0 hits / 313 misses** — every read hits a unique (offset, len); `mwindow` already absorbs true repetition | **FALSE** | +| 8 | Cache memory discipline | The OOM in factor 3 is our own cache holding buffers past `p_munmap`, defeating `MWINDOW_MAPPED_LIMIT` | run-4 memory monitor after evict-on-munmap fix, then full removal (run 5) | resident flat at 1833 KB, heap ≥ 6.2 MB, no OOM; removal I/O-neutral | **TRUE** | ### Details on hypothesis #2 (the freeze itself) @@ -98,15 +97,15 @@ the device.) ### New ideas -| Name | Estimated gain | Estimated lead time | Cause addressed | Comments | -|---|---|---|---|---| -| **O(depth) TreeBuilder splice** — patch only the edited path's ancestor chain onto HEAD's tree | commit 611 s → ~2–3 s; **flat in repo size forever** | ~2 days (bench op, then plumbing) | #2 + #3 — never opens the index, never materialises the tree | **Chosen.** Bonus: carries the 150 MB of images forward by OID, so the device doesn't even need them in its working tree | -| `CONFIG_FATFS_USE_FASTSEEK` + 256-word CLMT buffer | 2.3× on the splice (6.5 → 2.8 s); far seek 198.7 → 20.4 ms | hours — config, not code | #4 | Shipped as the companion fix; write-mode files fall back transparently | -| Explicit-path index staging (`add_path` over the editor's dirty set) | removes the O(N) walk term only | ~1 day | #2 partially | Retired — still calls `index.write()`, so it hits the same racy-clean wall | -| Index-free `write_tree_to` on a fresh in-memory index | — | ~1 day | #2 | Refuted by factor 3: seeding `read_tree(HEAD)` is 77 s + OOM — trades one O(N) for another | -| `esp_map.c` window cache (v2: size-keyed admission, evict-on-munmap) | hoped ~150–250 ms per loose write | ~1 day | #7 | Built and instrumented → **0 hits in 4 runs** → removed entirely. The one build made on an unverified cause; see learnings | -| Faster card / SD at 20 MHz | none for the commit | PCB respin budget | #1 | Rejected — factor 1 refuted; the card was exonerated twice (86 ms then 77 ms composites) | -| Shrink the repo (images off-card / LFS-style pointers) | shrinks N and the 570 MB clone | unknown | N itself | Rejected for now — the images are load-bearing for another app ([`../notes/git-sync-images-and-repo-size.md`](../notes/git-sync-images-and-repo-size.md)). Composes with the splice later | +| Name | Estimated gain | Estimated lead time | Cause addressed | Comments | +| ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | --------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **O(depth) TreeBuilder splice** — patch only the edited path's ancestor chain onto HEAD's tree | commit 611 s → ~2–3 s; **flat in repo size forever** | ~2 days (bench op, then plumbing) | #2 + #3 — never opens the index, never materialises the tree | **Chosen.** Bonus: carries the 150 MB of images forward by OID, so the device doesn't even need them in its working tree | +| `CONFIG_FATFS_USE_FASTSEEK` + 256-word CLMT buffer | 2.3× on the splice (6.5 → 2.8 s); far seek 198.7 → 20.4 ms | hours — config, not code | #4 | Shipped as the companion fix; write-mode files fall back transparently | +| Explicit-path index staging (`add_path` over the editor's dirty set) | removes the O(N) walk term only | ~1 day | #2 partially | Retired — still calls `index.write()`, so it hits the same racy-clean wall | +| Index-free `write_tree_to` on a fresh in-memory index | — | ~1 day | #2 | Refuted by factor 3: seeding `read_tree(HEAD)` is 77 s + OOM — trades one O(N) for another | +| `esp_map.c` window cache (v2: size-keyed admission, evict-on-munmap) | hoped ~150–250 ms per loose write | ~1 day | #7 | Built and instrumented → **0 hits in 4 runs** → removed entirely. The one build made on an unverified cause; see learnings | +| Faster card / SD at 20 MHz | none for the commit | PCB respin budget | #1 | Rejected — factor 1 refuted; the card was exonerated twice (86 ms then 77 ms composites) | +| Shrink the repo (images off-card / LFS-style pointers) | shrinks N and the 570 MB clone | unknown | N itself | Rejected for now — the images are load-bearing for another app ([`../notes/git-sync-images-and-repo-size.md`](../notes/git-sync-images-and-repo-size.md)). Composes with the splice later | **Chosen idea**: the O(depth) TreeBuilder splice, with fast-seek as its config companion — it is the only candidate whose cost is independent of repo size, so @@ -117,14 +116,14 @@ stays O(N_tree) and merely moves the wall. **What could go wrong?** -| Lens | Anticipated consequence | Mitigation | -|---|---|---| -| Stable | libgit2's 32-bit mwindow defaults `malloc` a 32 MB window on first pack access → instant OOM on the 8 MB PSRAM heap | `GIT_OPT_SET_MWINDOW_SIZE` 256 KB / `MAPPED_LIMIT` 4 MB set at service start, before any `Repository::open` | -| Stable | Power pull between a save and the next `:sync` loses the dirty set — nothing walks the tree anymore, so an unrecorded file would **never** sync | Dirty set journaled to `/sd/.typoena-dirty` (atomic write), recorded *before* the file write; over-reporting is a free no-op splice, so the semantics stay simple | -| Stable | Commit lands, push fails → the commit strands forever behind "up to date" | Stranded-commit recovery: compare HEAD against `refs/remotes/origin/` and push even when there is nothing new to commit | -| Method | Reconcile's `Mixed` reset writes the index → re-enters the racy-clean wall through the back door | `ResetType::Soft` — ref move only; there is no index to reset anymore | -| Method | **Deliberate behavior change:** files edited on the card from a desktop are never committed anymore (`add_all` used to sweep them in) | Accepted and documented as intent — the appliance's editor is the only writer that counts | -| Machine | libgit2 holds the pack + `.idx` descriptors open and opens loose objects on top → blows the editor's 4-FD mount | git builds mount with 16 FDs (`Storage::mount_for_git()`) | +| Lens | Anticipated consequence | Mitigation | +| ------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Stable | libgit2's 32-bit mwindow defaults `malloc` a 32 MB window on first pack access → instant OOM on the 8 MB PSRAM heap | `GIT_OPT_SET_MWINDOW_SIZE` 256 KB / `MAPPED_LIMIT` 4 MB set at service start, before any `Repository::open` | +| Stable | Power pull between a save and the next `:sync` loses the dirty set — nothing walks the tree anymore, so an unrecorded file would **never** sync | Dirty set journaled to `/sd/.typoena-dirty` (atomic write), recorded _before_ the file write; over-reporting is a free no-op splice, so the semantics stay simple | +| Stable | Commit lands, push fails → the commit strands forever behind "up to date" | Stranded-commit recovery: compare HEAD against `refs/remotes/origin/` and push even when there is nothing new to commit | +| Method | Reconcile's `Mixed` reset writes the index → re-enters the racy-clean wall through the back door | `ResetType::Soft` — ref move only; there is no index to reset anymore | +| Method | **Deliberate behavior change:** files edited on the card from a desktop are never committed anymore (`add_all` used to sweep them in) | Accepted and documented as intent — the appliance's editor is the only writer that counts | +| Machine | libgit2 holds the pack + `.idx` descriptors open and opens loose objects on top → blows the editor's 4-FD mount | git builds mount with 16 FDs (`Storage::mount_for_git()`) | The mechanism was stress-tested as a prototype first, never in production: the splice ran as a `git_bench` op against a full real-repo clone through four @@ -194,14 +193,14 @@ The same pipeline, redrawn with the change applied: :sync (new method, real repo) │ ▼ -┌─ local commit ────────────────────────────────────┐ ┌─ network push ──────┐ -│ splice: for each journaled dirty path (≈1) │ │ TLS + push │ -│ read file → blob → rebuild its ancestor │──▶│ ~6.5 s floor │ -│ subtree chain (~depth tree objects) │ │ (untouched — next │ -│ commit-obj │ │ curve to attack) │ -│ no index · no walk · no re-hash · images │ └─────────────────────┘ -│ carried by OID · lseek O(1) via fast-seek │ -└────────────────────────────────────────────────────┘ +┌─ local commit ───────────────────────────────┐ ┌─ network push ──────┐ +│ splice: for each journaled dirty path (≈1) │ │ TLS + push │ +│ read file → blob → rebuild its ancestor │──▶│ ~6.5 s floor │ +│ subtree chain (~depth tree objects) │ │ (untouched — next │ +│ commit-obj │ │ curve to attack) │ +│ no index · no walk · no re-hash · images │ └─────────────────────┘ +│ carried by OID · lseek O(1) via fast-seek │ +└──────────────────────────────────────────────┘ ``` Around the mechanism, the plumbing that makes it safe day-to-day: the