docs(sync): record the esp_map v2 verdict and cache removal
Run 4: memory discipline verified (1833 KB flat, no OOM) but 0 hits with the small maps demonstrably retained — window-repetition theory refuted, sub-second bar failed (splice 2.83 s cold / ~2 s steady). Decision: wire the splice in anyway (~9-10 s cold real-repo :sync vs 611 s/OOM for every alternative). Run 5: cache removal confirmed free; the ~1.85 MB "resident" was mwindow's live window set, which makes the mwindow opts in shipping git_sync.rs load-bearing.
This commit is contained in:
@@ -43,10 +43,27 @@ mechanism**, not a tuning knob.
|
||||
> excluded (`odb.read_header(packed)` = 470 ms; the strict-object-creation
|
||||
> theory was refuted — strict-off changed nothing). **esp_map.c v2 built**:
|
||||
> cache admission keyed on FILE size ≥ 1 MB so the hot small windows cache, and
|
||||
> evict-on-`p_munmap` to a 2 MB low-water mark fixes the 7.4 MB OOM. **Final
|
||||
> `git_bench` verdict pending — if splice lands sub-second, proceed with the
|
||||
> firmware plumbing below.** Full trail: the splice-bench, root-cause and
|
||||
> second-localization sections of the tradeoff curve.
|
||||
> evict-on-`p_munmap` to a 2 MB low-water mark fixes the 7.4 MB OOM.
|
||||
>
|
||||
> **Final `git_bench` verdict (run 4, same evening): the memory fix is
|
||||
> VERIFIED (resident 1833 KB flat, 6.4 MB heap free all run — no OOM), the
|
||||
> window-cache theory is REFUTED (v2 retains the small maps and still scored
|
||||
> 0 hits in 313 misses — the small reads hit unique offsets every time;
|
||||
> `mwindow` already absorbs true repetition), and the sub-second bar FAILED:
|
||||
> splice 2.83 s cold / ~2 s steady-state, commit(None) 713 ms. Decision:
|
||||
> WIRE IT IN ANYWAY** — ~2–2.8 s on the real repo vs 611 s/OOM for every
|
||||
> alternative puts a full cold `:sync` at ~9–10 s, which ships. The residual
|
||||
> ~360 ms/loose-write ≈ 8 unique small SD round-trips; next suspect is FAT
|
||||
> directory-op cost in freshen/refresh (instrumentation pass for later, not a
|
||||
> prerequisite). **The cache was then REMOVED entirely (run 5 confirmation):
|
||||
> esp_map.c is now the plain malloc-read/free-at-munmap emulation** — run 5's
|
||||
> read pattern was byte-identical to run 4 (even 15 reads better; v2's
|
||||
> eviction fought mwindow), warm splice identical at 1.95 s, and the ~1.85 MB
|
||||
> "resident" turned out to be mwindow's live open-window set, not cache
|
||||
> retention — which is one more reason the mwindow opts below are mandatory
|
||||
> in shipping `git_sync.rs`. **Proceed with the firmware plumbing below.**
|
||||
> Full trail: the splice-bench, root-cause, second-localization and
|
||||
> final-bench sections of the tradeoff curve.
|
||||
|
||||
## The fix — O(depth) TreeBuilder walk
|
||||
|
||||
|
||||
@@ -354,7 +354,7 @@ fast-seek runs — a re-provision resets it), and the mmap cache STILL scored 0
|
||||
hits — because its 64 KB map-length floor excluded exactly these hot small
|
||||
maps.
|
||||
|
||||
**Fix built (esp_map.c v2, pending re-bench):** cache admission re-keyed from
|
||||
**Fix built (esp_map.c v2):** cache admission re-keyed from
|
||||
map length to **file size ≥ 1 MB** — the pack/idx's small repeated windows now
|
||||
cache (RAM hits after first touch) while small mutable working-tree files stay
|
||||
excluded — plus **evict-on-`p_munmap` down to a 2 MB low-water mark**, fixing
|
||||
@@ -363,6 +363,71 @@ returned to `git__malloc`, so `MWINDOW_MAPPED_LIMIT` stays honest). Expected:
|
||||
`read_header` collapses toward CPU-only, `odb.write` toward ~150–250 ms,
|
||||
splice at or under the sub-second bar, and no end-of-run zlib OOM.
|
||||
|
||||
### Final bench (run 4, esp_map v2) — memory fix works, cache theory dead, bar failed
|
||||
|
||||
Run 4 (2026-07-12 evening, same card state as run 3b plus its orphans):
|
||||
|
||||
| op | run 3b (fast-seek) | run 4 (+ esp_map v2) |
|
||||
| --- | ---: | ---: |
|
||||
| `splice stage→tree` (cold, first op) | 2.81 s | **2.83 s — unchanged** |
|
||||
| `splice` again (warm, strict-off phase) | 3.21 s | 1.95 s |
|
||||
| `commit(None)` | 1.72 s | 713 ms |
|
||||
| `odb.write(blob)` | 416 ms | 366 ms |
|
||||
| `odb.read_header(packed)` | 470 ms | 412 ms |
|
||||
| `odb.exists(missing)` | 968 ms | 852 ms |
|
||||
| mmap-cache hits | 0 | **0** (313 misses) |
|
||||
| cache resident / heap free | grew to 7.4 MB → zlib OOM | **1833 KB flat / 6.4 MB free all run** |
|
||||
|
||||
Three findings:
|
||||
|
||||
1. **The memory discipline is verified.** Resident sits at 1833 KB through
|
||||
every phase (under the 2 MB low-water, so nothing is being churned) and
|
||||
heap never drops below 6.2 MB. The one uncaptured datum is the index-free
|
||||
`read_tree` tail (the section that OOM'd runs 1–3) — the monitor was cut
|
||||
before it ran. Not blocking: the shipping splice path never calls
|
||||
`read_tree`; the tail would only re-confirm eviction under burst.
|
||||
2. **The repeated-small-window theory is REFUTED — theory #3 down** (after
|
||||
strict-creation and free-cluster-scan). v2 demonstrably admits and retains
|
||||
the small maps now — the 1833 KB resident *is* them, held below low-water so
|
||||
nothing is evicted before reuse — and still scored 0 hits in 313 misses.
|
||||
So the ~8 small reads per loose write hit **unique (offset, len) every
|
||||
time**: `mwindow` was already absorbing any true repetition above `p_mmap`,
|
||||
and what reaches the emulation layer is distinct data (different objects,
|
||||
different delta bases). A window cache cannot help. The residual
|
||||
~360 ms/loose-write ≈ 8 distinct small SD round-trips × ~45 ms each
|
||||
(post-fast-seek) — I/O count, not I/O size or seek cost.
|
||||
3. **Within-run drift cuts both ways, so cross-run tables are mushy.** In this
|
||||
single run `commit(None)` degraded 713 ms → 1.79 s between the early and
|
||||
late (strict-off) phases, while splice *improved* 2.83 → 1.95 s. Two
|
||||
competing effects: first-touch warm-up fading (CLMT build, first pack
|
||||
reads — helps later ops) and orphan loose objects accumulating in
|
||||
`.git/objects/xx/` slowing every freshen existence check (FAT directory
|
||||
lookups are linear scans — hurts later ops). Steady-state on a clean
|
||||
objects dir ≈ **~2 s per splice+commit**.
|
||||
|
||||
**Run 5 (confirmation, cache removed entirely):** esp_map.c stripped back to
|
||||
plain malloc-read/free-at-munmap (stats counters kept). Read pattern
|
||||
**byte-identical** to run 4 at every checkpoint (118 maps / 2314 KB after
|
||||
splice, 148/2434, 163/2494, 208/2674) — except the strict-off phase did **15
|
||||
fewer reads (~60 KB)** than v2: the low-water eviction had been kicking out
|
||||
buffers mwindow still wanted, forcing re-reads. The cache was marginally
|
||||
worse than nothing. Warm splice identical (1953 vs 1949 ms); the cold-op
|
||||
+10–15 % drift (splice 2.83 → 3.26 s) is the known orphan-creep signature,
|
||||
not the removal. Run 5 also reframes run 4's "resident": **1854 KB `live` is
|
||||
mwindow's open-window working set** (pinned mappings under the bench's 4 MB
|
||||
mapped limit), not retained cache buffers — the cache had been retaining
|
||||
essentially nothing. Removal CONFIRMED free; simpler emulation ships.
|
||||
|
||||
**Verdict: the sub-second bar FAILED — wire the splice in anyway.** The bar
|
||||
was aspirational; measured reality is ~2–2.8 s to commit on the real
|
||||
263 MB-pack repo versus 611 s (or a hard OOM) for every alternative benched.
|
||||
That puts a full real-repo `:sync` at roughly **9–10 s cold**, which ships.
|
||||
The remaining ~2 s has survived four localization rounds; the next suspect —
|
||||
FAT *directory-op* cost in the freshen/refresh path (open/stat/rename by path
|
||||
walk FAT directories linearly; consistent with the orphan-creep signal) — is
|
||||
one instrumentation pass for later (log each `p_mmap` miss + bench dir ops in
|
||||
`sd_bench`), not a prerequisite for the plumbing.
|
||||
|
||||
### The walk is ~1.4 s even at N ≈ 2
|
||||
|
||||
Mostly fixed cost — the worktree-diff setup and the second (`update_all`) pass —
|
||||
@@ -387,15 +452,21 @@ work, ranked:
|
||||
flat in repo size, small heap, images carried forward untouched. Replaces
|
||||
`stage_and_commit`'s `add_all`/`update_all`/`index.write`/`write_tree`. Needs the
|
||||
editor's dirty set (+ deleted set) plumbed to the git service — the editor
|
||||
already knows both. **Benched 2026-07-12: 6.5 s — the shape is right but it
|
||||
fails the sub-second bar**; wiring it in is blocked on localizing the ~1.5 s
|
||||
loose-object write (see the splice-bench section above).
|
||||
2. **Fix the `esp_map.c` cache so it can't OOM (do it alongside #1).** It grew to
|
||||
7.4 MB resident past its 4 MB cap and starved zlib. Evict on `p_munmap` (not
|
||||
only lazily on the next `p_mmap`) down to a low-water mark, and lower the cap, so
|
||||
a released window's memory is actually returned to `git__malloc`. The cache's
|
||||
`odb.write` win (862 → 142 ms) is real and worth keeping — this is a
|
||||
memory-discipline fix, not a removal.
|
||||
already knows both. **Benched to completion 2026-07-12: 6.5 s → 2.8 s cold /
|
||||
~2 s steady-state after the fast-seek fix (run 4). The sub-second bar failed
|
||||
but the block is lifted — wire it in** (see the final-bench section above:
|
||||
the residual is unique small SD round-trips, not something a cache or seek
|
||||
fix can remove).
|
||||
2. **Fix the `esp_map.c` cache so it can't OOM — RESOLVED BY REMOVAL (run 5).**
|
||||
The cache never scored a hit in four instrumented real-repo runs
|
||||
(`mwindow` absorbs true repetition above `p_mmap`; only new ranges reach
|
||||
the emulation), and the 7.4 MB OOM it was patched to avoid was caused by
|
||||
the cache itself holding buffers past `p_munmap`. esp_map.c is now the
|
||||
plain malloc-read/free-at-munmap emulation: honest with
|
||||
`MWINDOW_MAPPED_LIMIT` by construction, ~120 lines lighter, and run 5
|
||||
confirmed removal is I/O-neutral (even 15 reads *better* than v2, whose
|
||||
low-water eviction fought mwindow). Stats counters kept to spot any future
|
||||
workload that does repeat ranges.
|
||||
3. **Retired: `add_all`/explicit-path *index* staging.** Explicit-path `add_path`
|
||||
still goes through the index and `index.write` → `truncate_racily_clean`, so it
|
||||
hits Wall 1 just the same. The TreeBuilder walk supersedes it entirely; the
|
||||
|
||||
Reference in New Issue
Block a user