docs(sync): record the esp_map v2 verdict and cache removal

Run 4: memory discipline verified (1833 KB flat, no OOM) but 0 hits
with the small maps demonstrably retained — window-repetition theory
refuted, sub-second bar failed (splice 2.83 s cold / ~2 s steady).
Decision: wire the splice in anyway (~9-10 s cold real-repo :sync vs
611 s/OOM for every alternative). Run 5: cache removal confirmed
free; the ~1.85 MB "resident" was mwindow's live window set, which
makes the mwindow opts in shipping git_sync.rs load-bearing.
This commit is contained in:
Julien Calixte
2026-07-13 00:20:12 +02:00
parent 1a5e1736f5
commit 9309f3f239
2 changed files with 102 additions and 14 deletions

View File

@@ -43,10 +43,27 @@ mechanism**, not a tuning knob.
> excluded (`odb.read_header(packed)` = 470 ms; the strict-object-creation
> theory was refuted — strict-off changed nothing). **esp_map.c v2 built**:
> cache admission keyed on FILE size ≥ 1 MB so the hot small windows cache, and
> evict-on-`p_munmap` to a 2 MB low-water mark fixes the 7.4 MB OOM. **Final
> `git_bench` verdict pending — if splice lands sub-second, proceed with the
> firmware plumbing below.** Full trail: the splice-bench, root-cause and
> second-localization sections of the tradeoff curve.
> evict-on-`p_munmap` to a 2 MB low-water mark fixes the 7.4 MB OOM.
>
> **Final `git_bench` verdict (run 4, same evening): the memory fix is
> VERIFIED (resident 1833 KB flat, 6.4 MB heap free all run — no OOM), the
> window-cache theory is REFUTED (v2 retains the small maps and still scored
> 0 hits in 313 misses — the small reads hit unique offsets every time;
> `mwindow` already absorbs true repetition), and the sub-second bar FAILED:
> splice 2.83 s cold / ~2 s steady-state, commit(None) 713 ms. Decision:
> WIRE IT IN ANYWAY** — ~22.8 s on the real repo vs 611 s/OOM for every
> alternative puts a full cold `:sync` at ~910 s, which ships. The residual
> ~360 ms/loose-write ≈ 8 unique small SD round-trips; next suspect is FAT
> directory-op cost in freshen/refresh (instrumentation pass for later, not a
> prerequisite). **The cache was then REMOVED entirely (run 5 confirmation):
> esp_map.c is now the plain malloc-read/free-at-munmap emulation** — run 5's
> read pattern was byte-identical to run 4 (even 15 reads better; v2's
> eviction fought mwindow), warm splice identical at 1.95 s, and the ~1.85 MB
> "resident" turned out to be mwindow's live open-window set, not cache
> retention — which is one more reason the mwindow opts below are mandatory
> in shipping `git_sync.rs`. **Proceed with the firmware plumbing below.**
> Full trail: the splice-bench, root-cause, second-localization and
> final-bench sections of the tradeoff curve.
## The fix — O(depth) TreeBuilder walk

View File

@@ -354,7 +354,7 @@ fast-seek runs — a re-provision resets it), and the mmap cache STILL scored 0
hits — because its 64 KB map-length floor excluded exactly these hot small
maps.
**Fix built (esp_map.c v2, pending re-bench):** cache admission re-keyed from
**Fix built (esp_map.c v2):** cache admission re-keyed from
map length to **file size ≥ 1 MB** — the pack/idx's small repeated windows now
cache (RAM hits after first touch) while small mutable working-tree files stay
excluded — plus **evict-on-`p_munmap` down to a 2 MB low-water mark**, fixing
@@ -363,6 +363,71 @@ returned to `git__malloc`, so `MWINDOW_MAPPED_LIMIT` stays honest). Expected:
`read_header` collapses toward CPU-only, `odb.write` toward ~150250 ms,
splice at or under the sub-second bar, and no end-of-run zlib OOM.
### Final bench (run 4, esp_map v2) — memory fix works, cache theory dead, bar failed
Run 4 (2026-07-12 evening, same card state as run 3b plus its orphans):
| op | run 3b (fast-seek) | run 4 (+ esp_map v2) |
| --- | ---: | ---: |
| `splice stage→tree` (cold, first op) | 2.81 s | **2.83 s — unchanged** |
| `splice` again (warm, strict-off phase) | 3.21 s | 1.95 s |
| `commit(None)` | 1.72 s | 713 ms |
| `odb.write(blob)` | 416 ms | 366 ms |
| `odb.read_header(packed)` | 470 ms | 412 ms |
| `odb.exists(missing)` | 968 ms | 852 ms |
| mmap-cache hits | 0 | **0** (313 misses) |
| cache resident / heap free | grew to 7.4 MB → zlib OOM | **1833 KB flat / 6.4 MB free all run** |
Three findings:
1. **The memory discipline is verified.** Resident sits at 1833 KB through
every phase (under the 2 MB low-water, so nothing is being churned) and
heap never drops below 6.2 MB. The one uncaptured datum is the index-free
`read_tree` tail (the section that OOM'd runs 13) — the monitor was cut
before it ran. Not blocking: the shipping splice path never calls
`read_tree`; the tail would only re-confirm eviction under burst.
2. **The repeated-small-window theory is REFUTED — theory #3 down** (after
strict-creation and free-cluster-scan). v2 demonstrably admits and retains
the small maps now — the 1833 KB resident *is* them, held below low-water so
nothing is evicted before reuse — and still scored 0 hits in 313 misses.
So the ~8 small reads per loose write hit **unique (offset, len) every
time**: `mwindow` was already absorbing any true repetition above `p_mmap`,
and what reaches the emulation layer is distinct data (different objects,
different delta bases). A window cache cannot help. The residual
~360 ms/loose-write ≈ 8 distinct small SD round-trips × ~45 ms each
(post-fast-seek) — I/O count, not I/O size or seek cost.
3. **Within-run drift cuts both ways, so cross-run tables are mushy.** In this
single run `commit(None)` degraded 713 ms → 1.79 s between the early and
late (strict-off) phases, while splice *improved* 2.83 → 1.95 s. Two
competing effects: first-touch warm-up fading (CLMT build, first pack
reads — helps later ops) and orphan loose objects accumulating in
`.git/objects/xx/` slowing every freshen existence check (FAT directory
lookups are linear scans — hurts later ops). Steady-state on a clean
objects dir ≈ **~2 s per splice+commit**.
**Run 5 (confirmation, cache removed entirely):** esp_map.c stripped back to
plain malloc-read/free-at-munmap (stats counters kept). Read pattern
**byte-identical** to run 4 at every checkpoint (118 maps / 2314 KB after
splice, 148/2434, 163/2494, 208/2674) — except the strict-off phase did **15
fewer reads (~60 KB)** than v2: the low-water eviction had been kicking out
buffers mwindow still wanted, forcing re-reads. The cache was marginally
worse than nothing. Warm splice identical (1953 vs 1949 ms); the cold-op
+1015 % drift (splice 2.83 → 3.26 s) is the known orphan-creep signature,
not the removal. Run 5 also reframes run 4's "resident": **1854 KB `live` is
mwindow's open-window working set** (pinned mappings under the bench's 4 MB
mapped limit), not retained cache buffers — the cache had been retaining
essentially nothing. Removal CONFIRMED free; simpler emulation ships.
**Verdict: the sub-second bar FAILED — wire the splice in anyway.** The bar
was aspirational; measured reality is ~22.8 s to commit on the real
263 MB-pack repo versus 611 s (or a hard OOM) for every alternative benched.
That puts a full real-repo `:sync` at roughly **910 s cold**, which ships.
The remaining ~2 s has survived four localization rounds; the next suspect —
FAT *directory-op* cost in the freshen/refresh path (open/stat/rename by path
walk FAT directories linearly; consistent with the orphan-creep signal) — is
one instrumentation pass for later (log each `p_mmap` miss + bench dir ops in
`sd_bench`), not a prerequisite for the plumbing.
### The walk is ~1.4 s even at N ≈ 2
Mostly fixed cost — the worktree-diff setup and the second (`update_all`) pass —
@@ -387,15 +452,21 @@ work, ranked:
flat in repo size, small heap, images carried forward untouched. Replaces
`stage_and_commit`'s `add_all`/`update_all`/`index.write`/`write_tree`. Needs the
editor's dirty set (+ deleted set) plumbed to the git service — the editor
already knows both. **Benched 2026-07-12: 6.5 s — the shape is right but it
fails the sub-second bar**; wiring it in is blocked on localizing the ~1.5 s
loose-object write (see the splice-bench section above).
2. **Fix the `esp_map.c` cache so it can't OOM (do it alongside #1).** It grew to
7.4 MB resident past its 4 MB cap and starved zlib. Evict on `p_munmap` (not
only lazily on the next `p_mmap`) down to a low-water mark, and lower the cap, so
a released window's memory is actually returned to `git__malloc`. The cache's
`odb.write` win (862 → 142 ms) is real and worth keeping — this is a
memory-discipline fix, not a removal.
already knows both. **Benched to completion 2026-07-12: 6.5 s → 2.8 s cold /
~2 s steady-state after the fast-seek fix (run 4). The sub-second bar failed
but the block is lifted — wire it in** (see the final-bench section above:
the residual is unique small SD round-trips, not something a cache or seek
fix can remove).
2. **Fix the `esp_map.c` cache so it can't OOM — RESOLVED BY REMOVAL (run 5).**
The cache never scored a hit in four instrumented real-repo runs
(`mwindow` absorbs true repetition above `p_mmap`; only new ranges reach
the emulation), and the 7.4 MB OOM it was patched to avoid was caused by
the cache itself holding buffers past `p_munmap`. esp_map.c is now the
plain malloc-read/free-at-munmap emulation: honest with
`MWINDOW_MAPPED_LIMIT` by construction, ~120 lines lighter, and run 5
confirmed removal is I/O-neutral (even 15 reads *better* than v2, whose
low-water eviction fought mwindow). Stats counters kept to spot any future
workload that does repeat ranges.
3. **Retired: `add_all`/explicit-path *index* staging.** Explicit-path `add_path`
still goes through the index and `index.write``truncate_racily_clean`, so it
hits Wall 1 just the same. The TreeBuilder walk supersedes it entirely; the