docs(sync): record the esp_map v2 verdict and cache removal

Run 4: memory discipline verified (1833 KB flat, no OOM) but 0 hits
with the small maps demonstrably retained — window-repetition theory
refuted, sub-second bar failed (splice 2.83 s cold / ~2 s steady).
Decision: wire the splice in anyway (~9-10 s cold real-repo :sync vs
611 s/OOM for every alternative). Run 5: cache removal confirmed
free; the ~1.85 MB "resident" was mwindow's live window set, which
makes the mwindow opts in shipping git_sync.rs load-bearing.

docs/notes/sync-commit-handoff.md

@@ -43,10 +43,27 @@ mechanism**, not a tuning knob.
 > excluded (`odb.read_header(packed)` = 470 ms; the strict-object-creation
 > theory was refuted — strict-off changed nothing). **esp_map.c v2 built**:
 > cache admission keyed on FILE size ≥ 1 MB so the hot small windows cache, and
-> evict-on-`p_munmap` to a 2 MB low-water mark fixes the 7.4 MB OOM. **Final
-> `git_bench` verdict pending — if splice lands sub-second, proceed with the
-> firmware plumbing below.** Full trail: the splice-bench, root-cause and
-> second-localization sections of the tradeoff curve.
+> evict-on-`p_munmap` to a 2 MB low-water mark fixes the 7.4 MB OOM.
+>
+> **Final `git_bench` verdict (run 4, same evening): the memory fix is
+> VERIFIED (resident 1833 KB flat, 6.4 MB heap free all run — no OOM), the
+> window-cache theory is REFUTED (v2 retains the small maps and still scored
+> 0 hits in 313 misses — the small reads hit unique offsets every time;
+> `mwindow` already absorbs true repetition), and the sub-second bar FAILED:
+> splice 2.83 s cold / ~2 s steady-state, commit(None) 713 ms. Decision:
+> WIRE IT IN ANYWAY** — ~2–2.8 s on the real repo vs 611 s/OOM for every
+> alternative puts a full cold `:sync` at ~9–10 s, which ships. The residual
+> ~360 ms/loose-write ≈ 8 unique small SD round-trips; next suspect is FAT
+> directory-op cost in freshen/refresh (instrumentation pass for later, not a
+> prerequisite). **The cache was then REMOVED entirely (run 5 confirmation):
+> esp_map.c is now the plain malloc-read/free-at-munmap emulation** — run 5's
+> read pattern was byte-identical to run 4 (even 15 reads better; v2's
+> eviction fought mwindow), warm splice identical at 1.95 s, and the ~1.85 MB
+> "resident" turned out to be mwindow's live open-window set, not cache
+> retention — which is one more reason the mwindow opts below are mandatory
+> in shipping `git_sync.rs`. **Proceed with the firmware plumbing below.**
+> Full trail: the splice-bench, root-cause, second-localization and
+> final-bench sections of the tradeoff curve.
 
 ## The fix — O(depth) TreeBuilder walk
 

docs/tradeoff-curves/sync-commit-staging.md

@@ -354,7 +354,7 @@ fast-seek runs — a re-provision resets it), and the mmap cache STILL scored 0
 hits — because its 64 KB map-length floor excluded exactly these hot small
 maps.
 
-**Fix built (esp_map.c v2, pending re-bench):** cache admission re-keyed from
+**Fix built (esp_map.c v2):** cache admission re-keyed from
 map length to **file size ≥ 1 MB** — the pack/idx's small repeated windows now
 cache (RAM hits after first touch) while small mutable working-tree files stay
 excluded — plus **evict-on-`p_munmap` down to a 2 MB low-water mark**, fixing
@@ -363,6 +363,71 @@ returned to `git__malloc`, so `MWINDOW_MAPPED_LIMIT` stays honest). Expected:
 `read_header` collapses toward CPU-only, `odb.write` toward ~150–250 ms,
 splice at or under the sub-second bar, and no end-of-run zlib OOM.
 
+### Final bench (run 4, esp_map v2) — memory fix works, cache theory dead, bar failed
+
+Run 4 (2026-07-12 evening, same card state as run 3b plus its orphans):
+
+| op | run 3b (fast-seek) | run 4 (+ esp_map v2) |
+| --- | ---: | ---: |
+| `splice stage→tree` (cold, first op) | 2.81 s | **2.83 s — unchanged** |
+| `splice` again (warm, strict-off phase) | 3.21 s | 1.95 s |
+| `commit(None)` | 1.72 s | 713 ms |
+| `odb.write(blob)` | 416 ms | 366 ms |
+| `odb.read_header(packed)` | 470 ms | 412 ms |
+| `odb.exists(missing)` | 968 ms | 852 ms |
+| mmap-cache hits | 0 | **0** (313 misses) |
+| cache resident / heap free | grew to 7.4 MB → zlib OOM | **1833 KB flat / 6.4 MB free all run** |
+
+Three findings:
+
+1. **The memory discipline is verified.** Resident sits at 1833 KB through
+   every phase (under the 2 MB low-water, so nothing is being churned) and
+   heap never drops below 6.2 MB. The one uncaptured datum is the index-free
+   `read_tree` tail (the section that OOM'd runs 1–3) — the monitor was cut
+   before it ran. Not blocking: the shipping splice path never calls
+   `read_tree`; the tail would only re-confirm eviction under burst.
+2. **The repeated-small-window theory is REFUTED — theory #3 down** (after
+   strict-creation and free-cluster-scan). v2 demonstrably admits and retains
+   the small maps now — the 1833 KB resident *is* them, held below low-water so
+   nothing is evicted before reuse — and still scored 0 hits in 313 misses.
+   So the ~8 small reads per loose write hit **unique (offset, len) every
+   time**: `mwindow` was already absorbing any true repetition above `p_mmap`,
+   and what reaches the emulation layer is distinct data (different objects,
+   different delta bases). A window cache cannot help. The residual
+   ~360 ms/loose-write ≈ 8 distinct small SD round-trips × ~45 ms each
+   (post-fast-seek) — I/O count, not I/O size or seek cost.
+3. **Within-run drift cuts both ways, so cross-run tables are mushy.** In this
+   single run `commit(None)` degraded 713 ms → 1.79 s between the early and
+   late (strict-off) phases, while splice *improved* 2.83 → 1.95 s. Two
+   competing effects: first-touch warm-up fading (CLMT build, first pack
+   reads — helps later ops) and orphan loose objects accumulating in
+   `.git/objects/xx/` slowing every freshen existence check (FAT directory
+   lookups are linear scans — hurts later ops). Steady-state on a clean
+   objects dir ≈ **~2 s per splice+commit**.
+
+**Run 5 (confirmation, cache removed entirely):** esp_map.c stripped back to
+plain malloc-read/free-at-munmap (stats counters kept). Read pattern
+**byte-identical** to run 4 at every checkpoint (118 maps / 2314 KB after
+splice, 148/2434, 163/2494, 208/2674) — except the strict-off phase did **15
+fewer reads (~60 KB)** than v2: the low-water eviction had been kicking out
+buffers mwindow still wanted, forcing re-reads. The cache was marginally
+worse than nothing. Warm splice identical (1953 vs 1949 ms); the cold-op
++10–15 % drift (splice 2.83 → 3.26 s) is the known orphan-creep signature,
+not the removal. Run 5 also reframes run 4's "resident": **1854 KB `live` is
+mwindow's open-window working set** (pinned mappings under the bench's 4 MB
+mapped limit), not retained cache buffers — the cache had been retaining
+essentially nothing. Removal CONFIRMED free; simpler emulation ships.
+
+**Verdict: the sub-second bar FAILED — wire the splice in anyway.** The bar
+was aspirational; measured reality is ~2–2.8 s to commit on the real
+263 MB-pack repo versus 611 s (or a hard OOM) for every alternative benched.
+That puts a full real-repo `:sync` at roughly **9–10 s cold**, which ships.
+The remaining ~2 s has survived four localization rounds; the next suspect —
+FAT *directory-op* cost in the freshen/refresh path (open/stat/rename by path
+walk FAT directories linearly; consistent with the orphan-creep signal) — is
+one instrumentation pass for later (log each `p_mmap` miss + bench dir ops in
+`sd_bench`), not a prerequisite for the plumbing.
+
 ### The walk is ~1.4 s even at N ≈ 2
 
 Mostly fixed cost — the worktree-diff setup and the second (`update_all`) pass —
@@ -387,15 +452,21 @@ work, ranked:
    flat in repo size, small heap, images carried forward untouched. Replaces
    `stage_and_commit`'s `add_all`/`update_all`/`index.write`/`write_tree`. Needs the
    editor's dirty set (+ deleted set) plumbed to the git service — the editor
-   already knows both. **Benched 2026-07-12: 6.5 s — the shape is right but it
-   fails the sub-second bar**; wiring it in is blocked on localizing the ~1.5 s
-   loose-object write (see the splice-bench section above).
-2. **Fix the `esp_map.c` cache so it can't OOM (do it alongside #1).** It grew to
-   7.4 MB resident past its 4 MB cap and starved zlib. Evict on `p_munmap` (not
-   only lazily on the next `p_mmap`) down to a low-water mark, and lower the cap, so
-   a released window's memory is actually returned to `git__malloc`. The cache's
-   `odb.write` win (862 → 142 ms) is real and worth keeping — this is a
-   memory-discipline fix, not a removal.
+   already knows both. **Benched to completion 2026-07-12: 6.5 s → 2.8 s cold /
+   ~2 s steady-state after the fast-seek fix (run 4). The sub-second bar failed
+   but the block is lifted — wire it in** (see the final-bench section above:
+   the residual is unique small SD round-trips, not something a cache or seek
+   fix can remove).
+2. **Fix the `esp_map.c` cache so it can't OOM — RESOLVED BY REMOVAL (run 5).**
+   The cache never scored a hit in four instrumented real-repo runs
+   (`mwindow` absorbs true repetition above `p_mmap`; only new ranges reach
+   the emulation), and the 7.4 MB OOM it was patched to avoid was caused by
+   the cache itself holding buffers past `p_munmap`. esp_map.c is now the
+   plain malloc-read/free-at-munmap emulation: honest with
+   `MWINDOW_MAPPED_LIMIT` by construction, ~120 lines lighter, and run 5
+   confirmed removal is I/O-neutral (even 15 reads *better* than v2, whose
+   low-water eviction fought mwindow). Stats counters kept to spot any future
+   workload that does repeat ranges.
 3. **Retired: `add_all`/explicit-path *index* staging.** Explicit-path `add_path`
    still goes through the index and `index.write` → `truncate_racily_clean`, so it
    hits Wall 1 just the same. The TreeBuilder walk supersedes it entirely; the

firmware/components/libgit2/esp_map.c

@@ -7,40 +7,19 @@
  * Allocations go through git__malloc, so with PSRAM in the heap they land in
  * the 8 MB external RAM rather than the ~340 KB internal DRAM.
  *
- * CACHE (2026-07-12): the emulation reads the range from SD on every call, so
- * libgit2's repeated pack access — `git_odb_write` → `git_odb__freshen` →
- * `git_odb_refresh` re-reads the pack idx/windows on *every* object write —
- * turned a 45 ms object write into 500 ms–1.1 s on a small repo, and would be far
- * worse on the real 570 MB-pack clone (see
- * docs/tradeoff-curves/sync-commit-staging.md). We cache the read buffers so a
- * given file region is read from the card once and reused.
+ * There is deliberately NO cache here. A window cache was built and removed
+ * on 2026-07-12: across four instrumented real-repo bench runs it scored
+ * exactly 0 hits, because libgit2's mwindow layer reuses its open windows and
+ * only genuinely new (offset, len) ranges ever reach p_mmap — and the one
+ * memory bug it "fixed" (7.4 MB resident starving zlib) was caused by the
+ * cache itself holding buffers past p_munmap. Free-at-munmap keeps
+ * GIT_OPT_SET_MWINDOW_MAPPED_LIMIT honest by construction: when libgit2
+ * releases a window, the memory really is back in git__malloc's pool. Full
+ * trail: docs/tradeoff-curves/sync-commit-staging.md (final-bench section).
+ * The stats counters below are kept so a future workload that *does* repeat
+ * ranges (push? reconcile?) can be spotted before anyone rebuilds a cache.
  *
- * Correctness: cache ONLY read-only mappings of files >= ESP_MAP_CACHE_MIN_FILE
- * (the FILE size, not the map length — 2026-07-12b). The hot set turned out to
- * be the SMALL maps: pack trailer probes, idx fanout reads and delta-base
- * windows repeat at the same offsets on every freshen/refresh, and a map-length
- * floor excluded exactly those (0 cache hits over three full real-repo bench
- * runs). Keying on file size caches them while still excluding the small
- * mutable working-tree files diff_file.c maps (notes.md etc.) — only the pack,
- * its idx, a midx and the commit-graph are ever that large, and all are
- * immutable on this device (only loose objects/refs/index are written, none
- * via mmap). The writable mapping the pack indexer uses (fetch/clone) is
- * excluded by the prot check. Identity is (dev, ino, size, mtime, offset, len);
- * for an immutable pack any of size/mtime already differs if the file is ever
- * replaced. NOTE: on esp-idf's FAT VFS st_dev/st_ino are constant and mtime has
- * 2 s granularity, so identity is effectively (size, mtime₂ₛ, offset, len) —
- * fine for packs (a replaced pack changes size), weak for small mutable files,
- * which is exactly why those must never be cacheable.
- *
- * Memory discipline (2026-07-12b): the first version freed released buffers
- * only lazily on the next p_mmap, so a burst (read_tree of 158 trees) pinned
- * 7.4 MB and starved zlib's git__malloc (crash at 508 KB heap) — it defeated
- * GIT_OPT_SET_MWINDOW_MAPPED_LIMIT because libgit2 believed the memory was
- * returned. Now p_munmap evicts unreferenced entries down to
- * ESP_MAP_CACHE_LOW_WATER, so a released window's memory really is returned
- * under pressure while a warm working set stays resident.
- *
- * Limitation: writable/shared mappings are not written back (and not cached).
+ * Limitation: writable/shared mappings are not written back.
  */
 
 #include "git2_util.h"
@@ -49,7 +28,6 @@
 #include <unistd.h>
 #include <string.h>
 #include <errno.h>
-#include <sys/stat.h>
 #include <stdint.h>
 
 int git__page_size(size_t *page_size)
@@ -64,48 +42,19 @@ int git__mmap_alignment(size_t *alignment)
 	return 0;
 }
 
-/* Only cache mappings of files at least this large (FILE size, not map length):
- * covers the pack, its idx, midx and commit-graph — and thus the hot small
- * windows within them — while excluding the small mutable working-tree files
- * diff_file.c maps. */
-#define ESP_MAP_CACHE_MIN_FILE (1024 * 1024)
-/* Cached-buffer budget in PSRAM. Entries with no live ref are LRU-evicted to
- * stay under this; pinned entries may exceed it transiently. */
-#define ESP_MAP_CACHE_CAP (4 * 1024 * 1024)
-/* On p_munmap, unreferenced entries are evicted down to this — the resident
- * working set a burst leaves behind. Must leave git__malloc (zlib, mwindow)
- * ample heap; the 2026-07-12 OOM run crashed with 7.4 MB resident. */
-#define ESP_MAP_CACHE_LOW_WATER (2 * 1024 * 1024)
-/* Small maps are numerous (trailer/fanout/delta-base probes), so more slots
- * than the window-only design needed; a slot is ~40 B. */
-#define ESP_MAP_CACHE_SLOTS 48
-
-struct map_entry {
-	unsigned char *data;
-	size_t len;
-	off64_t offset;
-	dev_t dev;
-	ino_t ino;
-	off_t size;
-	time_t mtime;
-	int refcount;
-	uint32_t used; /* LRU stamp; 0 = empty slot */
-};
-
-static struct map_entry g_cache[ESP_MAP_CACHE_SLOTS];
-static uint32_t g_clock;
-static size_t g_cached_bytes;
-
-/* Diagnostics, read from the bench via esp_map_stats(). */
-static uint32_t g_hits, g_misses;
+/* Diagnostics, read from the bench via esp_map_stats(). The signature predates
+ * the cache removal: `hits` is always 0, `misses` counts every mapping, and
+ * `cached_kb` now reports the LIVE mapped bytes (the mwindow working set). */
+static uint32_t g_maps;
 static uint64_t g_read_bytes;
+static size_t g_live_bytes;
 
 void esp_map_stats(uint32_t *hits, uint32_t *misses, uint32_t *read_kb, uint32_t *cached_kb)
 {
-	if (hits) *hits = g_hits;
-	if (misses) *misses = g_misses;
+	if (hits) *hits = 0;
+	if (misses) *misses = g_maps;
 	if (read_kb) *read_kb = (uint32_t)(g_read_bytes / 1024);
-	if (cached_kb) *cached_kb = (uint32_t)(g_cached_bytes / 1024);
+	if (cached_kb) *cached_kb = (uint32_t)(g_live_bytes / 1024);
 }
 
 static int read_range(int fd, off64_t offset, size_t len, unsigned char *data)
@@ -133,97 +82,26 @@ static int read_range(int fd, off64_t offset, size_t len, unsigned char *data)
 	return 0;
 }
 
-/* Best-effort: evict unreferenced LRU entries until total cached bytes fit
- * `budget`. Pinned entries (refcount > 0) can't be evicted, so budgets are
- * soft. */
-static void evict_to(size_t budget)
-{
-	while (g_cached_bytes > budget) {
-		int victim = -1;
-		uint32_t oldest = 0xFFFFFFFFu;
-		int i;
-
-		for (i = 0; i < ESP_MAP_CACHE_SLOTS; i++) {
-			if (g_cache[i].used && g_cache[i].refcount == 0 &&
-			    g_cache[i].used < oldest) {
-				oldest = g_cache[i].used;
-				victim = i;
-			}
-		}
-		if (victim < 0)
-			break;
-		git__free(g_cache[victim].data);
-		g_cached_bytes -= g_cache[victim].len;
-		memset(&g_cache[victim], 0, sizeof(g_cache[victim]));
-	}
-}
-
 int p_mmap(git_map *out, size_t len, int prot, int flags, int fd, off64_t offset)
 {
 	unsigned char *data;
-	struct stat st;
-	int cacheable, i;
 
+	GIT_UNUSED(prot);
 	GIT_UNUSED(flags);
 	GIT_MMAP_VALIDATE(out, len, prot, flags);
 
 	out->data = NULL;
 	out->len = 0;
 
-	/* Cache only read-only mappings of large files (pack/idx/...) that we can
-	 * identify. The map itself may be small — small repeated maps ARE the hot
-	 * set (see header comment). */
-	cacheable = !(prot & GIT_PROT_WRITE);
-	if (cacheable && (fstat(fd, &st) != 0 || st.st_size < ESP_MAP_CACHE_MIN_FILE))
-		cacheable = 0;
-
-	if (cacheable) {
-		for (i = 0; i < ESP_MAP_CACHE_SLOTS; i++) {
-			struct map_entry *e = &g_cache[i];
-			if (e->used && e->len == len && e->offset == offset &&
-			    e->dev == st.st_dev && e->ino == st.st_ino &&
-			    e->size == st.st_size && e->mtime == st.st_mtime) {
-				e->refcount++;
-				e->used = ++g_clock;
-				g_hits++;
-				out->data = e->data;
-				out->len = len;
-				return 0;
-			}
-		}
-	}
-
 	data = git__malloc(len);
 	GIT_ERROR_CHECK_ALLOC(data);
 	if (read_range(fd, offset, len, data) < 0) {
 		git__free(data);
 		return -1;
 	}
-	g_misses++;
+	g_maps++;
 	g_read_bytes += len;
-
-	if (cacheable) {
-		if (len < ESP_MAP_CACHE_CAP)
-			evict_to(ESP_MAP_CACHE_CAP - len);
-		for (i = 0; i < ESP_MAP_CACHE_SLOTS; i++) {
-			if (!g_cache[i].used) {
-				g_cache[i].data = data;
-				g_cache[i].len = len;
-				g_cache[i].offset = offset;
-				g_cache[i].dev = st.st_dev;
-				g_cache[i].ino = st.st_ino;
-				g_cache[i].size = st.st_size;
-				g_cache[i].mtime = st.st_mtime;
-				g_cache[i].refcount = 1;
-				g_cache[i].used = ++g_clock;
-				g_cached_bytes += len;
-				out->data = data;
-				out->len = len;
-				return 0;
-			}
-		}
-		/* All slots pinned: return uncached (freed directly at munmap). */
-	}
+	g_live_bytes += len;
 
 	out->data = data;
 	out->len = len;
@@ -232,26 +110,13 @@ int p_mmap(git_map *out, size_t len, int prot, int flags, int fd, off64_t offset
 
 int p_munmap(git_map *map)
 {
-	int i;
-
 	GIT_ASSERT_ARG(map);
 
-	/* Cached buffer: drop a ref and keep it for reuse — but return memory to
-	 * git__malloc under pressure (down to the low-water mark), so a released
-	 * window is really released and MWINDOW_MAPPED_LIMIT stays honest. */
-	for (i = 0; i < ESP_MAP_CACHE_SLOTS; i++) {
-		if (g_cache[i].used && g_cache[i].data == map->data) {
-			if (g_cache[i].refcount > 0)
-				g_cache[i].refcount--;
-			if (g_cache[i].refcount == 0 &&
-			    g_cached_bytes > ESP_MAP_CACHE_LOW_WATER)
-				evict_to(ESP_MAP_CACHE_LOW_WATER);
-			map->data = NULL;
-			map->len = 0;
-			return 0;
-		}
-	}
 	git__free(map->data);
+	if (g_live_bytes >= map->len)
+		g_live_bytes -= map->len;
+	else
+		g_live_bytes = 0;
 	map->data = NULL;
 	map->len = 0;
 	return 0;

firmware/src/bin/git_bench.rs

@@ -293,22 +293,25 @@ fn find_edit_path(repo: &Repository, root: &Tree) -> Result<Vec<String>> {
 }
 
 unsafe extern "C" {
-    /// Counters from the p_mmap cache in `components/libgit2/esp_map.c`.
+    /// Counters from the p_mmap emulation in `components/libgit2/esp_map.c`.
+    /// Post cache-removal: `hits` is always 0, `misses` counts every mapping,
+    /// `cached_kb` reports the LIVE mapped bytes (the mwindow working set).
     fn esp_map_stats(hits: *mut u32, misses: *mut u32, read_kb: *mut u32, cached_kb: *mut u32);
 }
 
-/// Log the p_mmap cache counters — hits vs misses (SD reads avoided), total KB
-/// read from the card, and KB currently resident. If the pack-read hypothesis is
-/// right, hits climb and `KB read` stops growing across the write ops.
+/// Log the p_mmap counters — mappings performed, total KB read from the card,
+/// and KB currently live-mapped (should track mwindow's open windows and stay
+/// well under MWINDOW_MAPPED_LIMIT now that munmap frees immediately).
 fn log_map_stats(label: &str) {
     let (mut hits, mut misses, mut read_kb, mut cached_kb) = (0u32, 0u32, 0u32, 0u32);
     unsafe { esp_map_stats(&mut hits, &mut misses, &mut read_kb, &mut cached_kb) };
+    let _ = hits; // always 0 since the cache removal; slot kept for ABI stability
     // Free heap spans PSRAM here; a drop toward 0 during write_tree/commit on the
     // real repo would point at mwindow/idx allocation pressure (or thrash) as the
     // cause of an apparent hang, not CPU.
     let free_kb = unsafe { esp_idf_svc::sys::esp_get_free_heap_size() } / 1024;
     log::info!(
-        "mmap cache @ {label:<11} {hits} hit / {misses} miss, {read_kb} KB read, {cached_kb} KB resident, {free_kb} KB heap free"
+        "mmap @ {label:<11} {misses} maps, {read_kb} KB read, {cached_kb} KB live, {free_kb} KB heap free"
     );
 }