typewriter/docs/kaizen/real-repo-sync.md

Kaizen — :sync never completes on the real notes repo

Status (2026-07-13): closed. First end-to-end :sync on the real repo — ∞ → 24.1 s snackbar-to-snackbar. Getting there took two loops: the commit half (steps 1–5 below) and a second localization loop on the push half, surfaced by the step-6 verification runs and documented inside step 6.

The writer presses :sync on the Typoena appliance and the device freezes for ten minutes; a power-cycle re-enters the same freeze forever. The real jcalixte/notes clone (1179 files, 263 MB pack) has never been synced from the device — only the toy test repo has. Full measurement trail: ../tradeoff-curves/sync-commit-staging.md.

1) Improvement potential

Value model for the writer (Julien on the appliance)

Want more of	Do less of
+ words safely on the remote with one keystroke	– waiting on a frozen device
+ trust that :sync completes, every time	– power-cycling mid-sync and losing the session's trust
+ keep writing while it publishes	– babysitting git from a desktop to rescue the card

Measurement: seconds from :sync to the snackbar, on the real jcalixte/notes clone on the device.

---
config:
  xyChart:
    width: 900
    height: 500
---
xychart-beta
    title "Improvement potential — :sync → snackbar"
    x-axis ["Before kaizen (~611 s, never completes)", "After kaizen (target ≤ ~10 s)"]
    y-axis "seconds" 0 --> 650
    bar [611]
    line [10, 10]

The before bar understates the reality: a reset re-enters the freeze, so the true value is ∞ — 0 successful syncs ever. The line is the ≤ ~10 s target; the after-kaizen slot stays empty here — the measured bar lands in step 6.

2) Current method analysis

:sync (current method, real repo)
        │
        ▼
┌─ local commit ─────────────────────────────────────┐   ┌─ network push ──────┐
│ stage: add_all(["*"]) + update_all(["*"])          │   │ TLS + push          │
│   → stat()s 1179 files / 158 dirs over 10 MHz SPI  │──▶│ ~6.5 s floor        │
│ index.write ⚡ ~611 s                               │   │ (separate curve)    │
│   → FAT's 2 s mtime marks ~every entry "racy"      │   └─────────────────────┘
│     → re-hashes ~170 MB (150 MB of images)         │
│ write_tree + commit-obj ⚡ seconds each             │
│   → loose writes trigger pack reads through        │
│     emulated mmap; each far lseek walks the        │
│     FAT cluster chain (~190 ms)                    │
└────────────────────────────────────────────────────┘
   ⚡ = weak points, measured on device

Every factor was benched on the device (sd_bench for raw FAT, git_bench for git2 primitives on the real clone) — details and full tables in ../tradeoff-curves/sync-commit-staging.md:

Factor #	Factor	Hypothesis	Test method	Test result	True or false?
1	SD card raw I/O	The card itself is slow: a loose-object write costs ~700–900 ms	sd_bench: raw FAT create/write/rename composite, same card, same 10 MHz	complete loose-object composite 86 ms — 8× under libgit2's number	FALSE
2	index.write() racy-clean	FAT's 2 s mtime granularity makes ~all 1179 entries look racy → full ~170 MB re-hash over SPI	git_bench on the real clone, 3 consecutive index.write()	611 s → 12.8 s → 360 ms — the decay is the racy set shrinking as the index mtime advances	TRUE
3	Index-free alternative	Skipping index.write (fresh in-memory index) escapes the wall	git_bench: Index::new + read_tree(HEAD) + write_tree_to	seed read_tree 77 s cold + mmap grows to 7.4 MB → zlib OOM crash — still O(N_tree)	FALSE (as a fix)
4	FatFS lseek	Long/backward seeks walk the FAT cluster chain over SPI (~67 KB of FAT reads for a 263 MB file)	sd_bench: seek+read 4 KB @offset 0 vs @end of the pack; A/B with CONFIG_FATFS_USE_FASTSEEK	@0 = 5.8 ms, @end = 198.7 ms; fast-seek → 20.4 ms	TRUE
5	Free-cluster scan	A ~740 MB-full card slows FAT allocation	sd_bench re-run on the full card	composite 77 ms — unchanged	FALSE
6	Strict object creation	OID validation on commit/treebuilder causes the ~1.3 s commit premium	strict-off A/B in git_bench	1.80 s vs 1.93 s — no premium removed	FALSE
7	Repeated small mmap windows	A window cache in esp_map.c would absorb the ~8 small pack reads per loose write	instrumented cache (v2), 4 real-repo runs	0 hits / 313 misses — every read hits a unique (offset, len); mwindow already absorbs true repetition	FALSE
8	Cache memory discipline	The OOM in factor 3 is our own cache holding buffers past p_munmap, defeating MWINDOW_MAPPED_LIMIT	run-4 memory monitor after evict-on-munmap fix, then full removal (run 5)	resident flat at 1833 KB, heap ≥ 6.2 MB, no OOM; removal I/O-neutral	TRUE

Details on hypothesis #2 (the freeze itself)

git_index_write unconditionally runs truncate_racily_clean, which re-hashes every entry whose mtime is not strictly older than the index stamp. On a fresh FAT clone the 2 s mtime granularity makes the whole tree racy, so a one-line note edit re-hashes ~170 MB — mostly the ~260 images a text edit never touches — over the 10 MHz SPI bus. This is why the device freezes ~10 minutes and why a reset never helps: the index mtime doesn't advance, so it re-hashes forever.

Selected weak point: the local commit stage — its staging strategy is O(N_tree) in files and bytes, on a device where N_tree ≈ 1179 and the bus is 10 MHz SPI. (The ~6.5 s network half is real but is a separate curve, ../notes/sync-latency.md, and it does not brick the device.)

3) Ideas

Bibliography

• FatFS f_lseek docs (elm-chan.org) — the cluster-chain walk and the CLMT fast-seek mechanism.
• ESP-IDF FatFS docs — CONFIG_FATFS_USE_FASTSEEK, recommended for read-heavy workloads with long backward seeks.
• Vendored libgit2 v1.9.4 source — index.c truncate_racily_clean (the re-hash wall), odb.c git_odb__freshen/git_odb_refresh (the per-write pack probes), mwindow.c (32-bit defaults: 32 MB window / 256 MB mapped limit).

New ideas

Name	Estimated gain	Estimated lead time	Cause addressed	Comments
O(depth) TreeBuilder splice — patch only the edited path's ancestor chain onto HEAD's tree	commit 611 s → ~2–3 s; flat in repo size forever	~2 days (bench op, then plumbing)	#2 + #3 — never opens the index, never materialises the tree	Chosen. Bonus: carries the 150 MB of images forward by OID, so the device doesn't even need them in its working tree
CONFIG_FATFS_USE_FASTSEEK + 256-word CLMT buffer	2.3× on the splice (6.5 → 2.8 s); far seek 198.7 → 20.4 ms	hours — config, not code	#4	Shipped as the companion fix; write-mode files fall back transparently
Explicit-path index staging (add_path over the editor's dirty set)	removes the O(N) walk term only	~1 day	#2 partially	Retired — still calls index.write(), so it hits the same racy-clean wall
Index-free write_tree_to on a fresh in-memory index	—	~1 day	#2	Refuted by factor 3: seeding read_tree(HEAD) is 77 s + OOM — trades one O(N) for another
esp_map.c window cache (v2: size-keyed admission, evict-on-munmap)	hoped ~150–250 ms per loose write	~1 day	#7	Built and instrumented → 0 hits in 4 runs → removed entirely. The one build made on an unverified cause; see learnings
Faster card / SD at 20 MHz	none for the commit	PCB respin budget	#1	Rejected — factor 1 refuted; the card was exonerated twice (86 ms then 77 ms composites)
Shrink the repo (images off-card / LFS-style pointers)	shrinks N and the 570 MB clone	unknown	N itself	Rejected for now — the images are load-bearing for another app (../notes/git-sync-images-and-repo-size.md). Composes with the splice later

Chosen idea: the O(depth) TreeBuilder splice, with fast-seek as its config companion — it is the only candidate whose cost is independent of repo size, so the problem cannot come back as the notes tree grows. Everything index-shaped stays O(N_tree) and merely moves the wall.

4) Test plan

What could go wrong?

Lens	Anticipated consequence	Mitigation
Stable	libgit2's 32-bit mwindow defaults malloc a 32 MB window on first pack access → instant OOM on the 8 MB PSRAM heap	GIT_OPT_SET_MWINDOW_SIZE 256 KB / MAPPED_LIMIT 4 MB set at service start, before any Repository::open
Stable	Power pull between a save and the next :sync loses the dirty set — nothing walks the tree anymore, so an unrecorded file would never sync	Dirty set journaled to /sd/.typoena-dirty (atomic write), recorded before the file write; over-reporting is a free no-op splice, so the semantics stay simple
Stable	Commit lands, push fails → the commit strands forever behind "up to date"	Stranded-commit recovery: compare HEAD against refs/remotes/origin/<branch> and push even when there is nothing new to commit
Method	Reconcile's Mixed reset writes the index → re-enters the racy-clean wall through the back door	ResetType::Soft — ref move only; there is no index to reset anymore
Method	Deliberate behavior change: files edited on the card from a desktop are never committed anymore (add_all used to sweep them in)	Accepted and documented as intent — the appliance's editor is the only writer that counts
Machine	libgit2 holds the pack + .idx descriptors open and opens loose objects on top → blows the editor's 4-FD mount	git builds mount with 16 FDs (Storage::mount_for_git())

The mechanism was stress-tested as a prototype first, never in production: the splice ran as a git_bench op against a full real-repo clone through four localization rounds before any firmware plumbing.

Who must we convince?

• Upstream libgit2 maintainers — the tlsf double-free in the mbedTLS stream error path (found during rollout; wrap's error path frees the caller's socket stream, new() frees it again). Needs an upstream report; we ship a whole-file override meanwhile.
• Desktop-me (and any future contributor) — the card's working tree now shows a permanent diff against HEAD when inspected on a Mac. That is intent, not a bug.

Rollback: the git feature flag gates the whole sync stack (the light build never links it), and the plumbing is a small commit range to revert. The journal file is additive — an old build simply ignores it.

Measurement protocol for step 6: full :sync on the device against the real clone — commit split — log lines plus snackbar-to-snackbar wall time, cold and steady-state.

5) Implementation

Before — stage_and_commit walked and hashed the world through the index (condensed; firmware/src/git_sync.rs at a5edaed~1):

fn stage_and_commit(repo: &Repository) -> Result<Option<Oid>> {
    let mut index = repo.index()?;
    index.add_all(["*"], IndexAddOption::DEFAULT, Some(&mut skip_macos_cruft))?;
    index.update_all(["*"], Some(&mut skip_macos_cruft))?;  // stat 1179 files
    index.write()?;                    // ⚡ racy-clean re-hash: ~611 s on FAT
    let tree = repo.find_tree(index.write_tree()?)?;
    // … commit(Some("HEAD"), …, &tree, &parents)
}

After — the editor's journaled dirty paths are spliced onto HEAD's tree; the index never exists (condensed; firmware/src/git_sync.rs:345):

fn stage_and_commit(repo: &Repository, paths: &BTreeSet<String>) -> Result<Option<Oid>> {
    let parent = repo.head().ok().and_then(|h| h.peel_to_commit().ok());
    let mut tree = parent.as_ref().map(|c| c.tree()).transpose()?;
    for path in paths {
        let blob = match fs::read(format!("{REPO_DIR}/{path}")) {
            Ok(bytes) => Some(repo.blob(&bytes)?),       // present → splice in
            Err(e) if e.kind() == NotFound => None,      // deleted → splice out
            Err(e) => return Err(e.into()),
        };
        let spliced = splice(repo, tree.as_ref(), &parts(path), blob)?;
        tree = Some(repo.find_tree(spliced)?);
    }
    // … same "tree unchanged → nothing to publish" check, same commit call
}

/// Reads ~depth tree objects, writes ~depth new ones; every sibling entry is
/// carried by OID — never opened.
fn splice(repo: &Repository, base: Option<&Tree>, path: &[&str], blob: Option<Oid>) -> Result<Oid>

The same pipeline, redrawn with the change applied:

:sync (new method, real repo)
        │
        ▼
┌─ local commit ───────────────────────────────┐   ┌─ network push ──────┐
│ splice: for each journaled dirty path (≈1)   │   │ TLS + push          │
│   read file → blob → rebuild its ancestor    │──▶│ ~6.5 s floor        │
│   subtree chain (~depth tree objects)        │   │ (untouched — next   │
│ commit-obj                                   │   │  curve to attack)   │
│ no index · no walk · no re-hash · images     │   └─────────────────────┘
│ carried by OID · lseek O(1) via fast-seek    │
└──────────────────────────────────────────────┘

Around the mechanism, the plumbing that makes it safe day-to-day: the /sd/.typoena-dirty journal with its pending → in-flight → settled lifecycle, stranded-commit recovery, a radio-free "up to date" answer (~150 ms instead of a Wi-Fi/TLS round), soft-reset reconcile, the 16-FD git mount, and the mwindow options at service start. Details: the fix — wiring.

6) Evaluation

Measurement redone (seconds, :sync → snackbar on the real repo): ∞ (never completed) → 24.1 s measured end-to-end on device, 2026-07-13 (commit 6713ea6e pushed and accepted by origin).

---
config:
  xyChart:
    width: 900
    height: 500
---
xychart-beta
    title "Measurement redone — :sync → snackbar"
    x-axis ["Before kaizen (~611 s, never completes)", "After kaizen (24.1 s measured)"]
    y-axis "seconds" 0 --> 650
    bar [611, 24.1]
    line [10, 10]

Split: Wi-Fi bring-up 3.7 s + clock/TLS 0.6 s + publish 19.8 s (splice 5.0 s, commit-obj 1.2 s, push ≈ 6.1 s, repo open/journal/refs the rest). The ≤ ~10 s target line is not yet met — but the failure mode changed category: from "bricks the device, forever" to "a finite number to shave".

Getting from the working commit half to an accepted push took a second localization loop: five verification runs, each killing one hypothesis with on-device telemetry built for that run.

The push half — the second loop (runs 4–9, all 2026-07-13)

The step-4 mitigation "mwindow 256 KB / 4 MB" turned out to be necessary but nowhere near sufficient. The verification runs surfaced, in order:

Factor	Hypothesis	Test method	Test result	True or false?
P1 — UI death by heap	The editor's per-draw vec![0xFF; 26928] framebuffer aborts the firmware when the push exhausts the heap (run 4: 66 s silent push, then abort on a keystroke)	backtrace + allocator fallback math (internal 23 KB < 27 KB ⇒ PSRAM had no 27 KB block either)	TRUE — a C malloc failure inside libgit2 returns an error; a Rust Vec failure aborts the process	fix: two boot-time frames, draw_into + mem::swap — steady-state repaints never allocate
P2 — odb cache is the ~6 MB eater	libgit2's odb object cache (default max 256 MB — unbounded here) caches every tree/commit read during pack building	cap it at 1 MB + read GIT_OPT_GET_CACHED_MEMORY in a per-run heap line	cache used 59 KB of its 1 MB cap at the moment of death	FALSE (cap kept as insurance)
P3 — progress callbacks would show the slope	pack_progress throttled by object count will trace the silent stretch	run 6: zero lines in 65 s; then source read	count gate can't fire (AddingObjects reports total = 0, a 2-commit push inserts dozens of objects) — and deeper: the whole 65 s is mark_edges_uninteresting reading origin tip's ~1200-object tree before the first insert, a phase with no callbacks at all	TRUE (as a bug in our telemetry) — fix: 5 s sibling heartbeat thread through remote.push()
P4 — libgit2's mmaps are the eater	every "mmap" is a real PSRAM malloc in esp_map.c; its live-bytes counter should track the drop	wire esp_map_stats() into the heap line + heartbeat	run 7: mmap-live tracked the PSRAM drop 1:1, plateau 7151 KB of the 8 MB pool, min-ever 472 B, 19.9 MB read off the card to push two commits	TRUE
P5 — the budget itself is impossible	fixed whole-file index maps live outside the mwindow budget	inspect the source clone	5 packs, 562 MB: 1730 KB multi-pack-index + ~1737 KB of .idx = ~3.4 MB fixed, plus windows at the (working, soft) 4 MB limit	TRUE — fix: mwindow 64 KB windows / 1.5 MB budget
P6 — keep-alive dies during the marking gap	with memory fixed (run 8: min-ever 2.05 MB, marking 32 s), the push died at the first write of the upload: SSL error: Generic error after ~31 s of wire silence between the negotiation GET and the pack POST	timeline correlation (pack had built fine — 11 objects, deltafication complete)	GitHub/router closes the idle connection ~30 s; libgit2 treats the dead socket as fatal, no reconnect	TRUE
P7 — pack layout is a device-performance variable	git repack -ad (5 packs → 1, midx gone) shrinks the fixed maps and clusters trees for the marking walk	repack the source clone, reload the card, run 9	marking 3.5 s (was 32 s) — the whole boundary walk hit ONE 1740 KB idx + two 64 KB windows; push accepted	TRUE

Run 9's heap line, for the record: min-ever PSRAM 4.5 MB free (was 472 B three runs earlier), mmap live 1868 KB, odb cache 70 KB.

Learnings:

• Bench the real thing. The toy repo understated everything by ~2 orders of magnitude; "works on the toy" hid a total brick. The real clone is now the only valid bench target.
• Refute before building. Four theories died to measurement (card speed, free-cluster scan, strict creation, repeated windows). The one build made on an unverified cause — the esp_map.c window cache — was the one wasted build: 0 hits in 4 instrumented runs, removed entirely.
• FAT breaks POSIX assumptions. 2 s mtime granularity, no inode, cluster-chain seeks: any POSIX-shaped library (libgit2's racy-clean check, the mmap emulation) needs its cost model audited on FAT before trusting it.
• Prefer mechanisms flat in N. The splice cannot regress as the notes tree grows — the fix that prevents the problem from ever coming back, which is the kind kaizen prefers.
• The UI must not share a fate with git. A C allocation failure inside libgit2 degrades to an error snackbar; a Rust Vec allocation failure aborts the firmware. Steady-state repaints now render into pre-allocated frames so a heap-starved :sync can fail without taking the editor down (run 4 → 5).
• Telemetry needs the right axis. A count-based progress gate logged nothing because libgit2 reports total = 0 during insertion — and the longest phase fires no callbacks at all. Time-gated logging plus a sibling heartbeat thread is what finally produced the ~6 MB consumer's name in one run.
• Budget the whole emulation, not the knob you can see. The mwindow mapped-limit worked exactly as configured — and was irrelevant, because whole-file .idx/multi-pack-index maps (~3.4 MB) live outside it. On a malloc-backed mmap, the budget is esp_map live bytes, not the mwindow limit.
• Repo pack layout is a device-performance variable. The same clone as 5 packs vs 1 pack is the difference between a 32 s marking walk (95 windows, 9.2 MB read) and a 3.5 s one (2 windows) — and between losing and winning the keep-alive race. git repack -ad on the source clone is card-prep, not an optimization.

Standard to update:

• Bench and verify against the real repo clone, never the toy (the toy is ~2 orders of magnitude too kind) — recorded in the tradeoff doc's how to bench.
• Any new libgit2 entry point must set the mwindow options before its first Repository::open — the 32-bit defaults OOM the 8 MB heap. Values revised by the push loop: 64 KB windows / 1.5 MB mapped limit (the step-4 values, 256 KB / 4 MB, left no room for the ~3.4 MB of fixed index maps).
• Cards are loaded from a single-pack clone: git repack -ad on the source before just load (candidate: fold into the _load-repo recipe).
• CONFIG_FATFS_USE_FASTSEEK=y + 256-word CLMT buffer stay pinned in sdkconfig.defaults.
• Firmware repaint paths never allocate the framebuffer per draw — render into the two persistent frames via Editor::draw_into.

Share with:

• Upstream libgit2 — file the tlsf double-free report (mbedTLS stream error path: wrap's error path frees the caller's socket stream, new() frees it again; v1.9.4).
• A write-up for the notes/blog — the four-round localization story (611 s → ~2.8 s commit on a microcontroller, four theories refuted on the way) stands on its own.

Next steps

• Close the gap to the ≤ ~10 s target: publish is 19.8 s; the splice's 5.0 s and the ~7 s of repo-open/negotiation overhead are the next curves.
• The keep-alive race is avoided, not fixed (3.5 s ≪ ~30 s) — a future multi-pack or cold-cache state could still lose it. Durable fix: reconnect-on-stale-connection in the http/stream layer.
• Internal RAM min-ever hit 2099 bytes during the TLS send (the diagnostic heartbeat's 8 KB stack is part of it) — shrink or retire the heartbeat now that it has served its purpose.
• Fold git repack -ad into the _load-repo recipe so single-pack card prep is enforced, not remembered.
• File the libgit2 upstream bug report (tlsf double-free in the mbedTLS stream error path).
• Instrument the residual ~360 ms/loose-write (suspect: FAT directory-op cost in the freshen/refresh path) — one sd_bench + p_mmap-miss logging pass.
• The images-off-card lever (../notes/git-sync-images-and-repo-size.md) composes with the splice.