Compare commits

...

5 Commits

Author SHA1 Message Date
Julien Calixte
ce3204a350 docs(sync): record fast-seek root cause and splice bench trail
Splice bench (6.5 s, O(depth) shape confirmed), FatFS cluster-chain
root cause with elm-chan/ESP-IDF references, on-device fast-seek
verification, second localization round (strict-creation refuted,
read_header 470 ms), esp_map v2 design, and the two firmware plumbing
gaps (mwindow opts + 16-FD mount).
2026-07-13 00:02:55 +02:00
Julien Calixte
dd3d70cbec perf(sync): cache small pack windows and evict mmap cache on munmap
Admission keyed on file size (>= 1 MB) instead of map length: the hot
set is the small repeated maps (pack trailer, idx fanout, delta bases)
that the 64 KB floor excluded — 0 hits across three real-repo runs.
Small mutable working-tree files stay uncacheable. p_munmap now evicts
unreferenced entries to a 2 MB low-water mark so released windows are
actually returned to git__malloc (fixes the 7.4 MB resident / 508 KB
heap zlib OOM and keeps MWINDOW_MAPPED_LIMIT honest).
2026-07-13 00:02:54 +02:00
Julien Calixte
2c1c590c9e perf(sync): enable FatFS fast seek for pack reads
Without CONFIG_FATFS_USE_FASTSEEK every long/backward lseek in the
263 MB pack walks the FAT cluster chain over SPI (~190 ms), and libgit2
pays ~8 such seeks per loose-object write. The CLMT makes lseek O(1)
for read-mode files; verified on device: far seek 198.7 -> 20.4 ms,
splice commit 6.5 -> 2.8 s.
2026-07-13 00:02:42 +02:00
Julien Calixte
2c24ece3a5 feat(bench): add packfile long-seek op to sd_bench
Reads 4 KB at the start vs the end of the repo's largest pack (skipping
macOS ._ sidecars). Proved the ~1.5 s/loose-object cost was FatFS
walking the FAT cluster chain on every long/backward lseek: 5.8 ms at
offset 0 vs 198.7 ms at the end of the 263 MB pack.
2026-07-13 00:02:42 +02:00
Julien Calixte
da8ca7d8d2 feat(bench): bench the O(depth) TreeBuilder splice and odb probes
Adds the splice prototype (patch one path onto HEAD's tree, O(depth))
as git_bench's headline op, run first so its cold number survives the
index ops' OOM, plus odb.read_header/odb.exists probes and strict-off
re-benches that localized the residual cost and refuted the
strict-object-creation theory.
2026-07-13 00:02:29 +02:00
6 changed files with 543 additions and 97 deletions

View File

@@ -22,6 +22,32 @@ the toy `typoena-test` (`notes.md`) has. The repo cannot be shrunk (the 150 MB o
images serve another app — see the images note). So the fix is a **new commit
mechanism**, not a tuning knob.
> **Splice bench result (2026-07-12, later the same day): 6.5 s — do NOT wire it
> in yet.** The walk is in `git_bench` (`splice stage→tree`) and its O(depth)
> shape is confirmed on the real repo (flat pack reads, heap healthy, no OOM),
> but each of its 4 loose-object writes costs **~1.5 s** — isolated `odb.write`
> regressed 142 ms → 1.5 s vs the previous run, with **0 mmap-cache hits**.
> Projected full commit ≈ 89 s.
>
> **Root cause found the same evening (sd_bench seek op): FatFS lseek walks the
> FAT cluster chain** — seek+read(4 KB) into the 263 MB pack costs **198.7 ms at
> the end vs 5.8 ms at offset 0**, and each loose write pays ~8 such walks via
> the freshen path's small `p_mmap`s → the ~1.5 s. Fix landed in
> `sdkconfig.defaults`: `CONFIG_FATFS_USE_FASTSEEK=y` +
> `FAST_SEEK_BUFFER_SIZE=256` (O(1) lseek for read-mode files, i.e. the pack —
> see [FatFS f_lseek/CLMT](http://elm-chan.org/fsw/ff/doc/lseek.html) and the
> [ESP-IDF FatFS docs](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/fatfs.html)).
> **Verified on device: far seek 198.7 → 20.4 ms.** The A/B (splice 6.5 → 2.8 s,
> odb.write 1.5 s → 416 ms) plus new probes then localized the residual to ~78
> repeated small (~4 KB) pack reads per op that the mmap cache's 64 KB floor
> excluded (`odb.read_header(packed)` = 470 ms; the strict-object-creation
> theory was refuted — strict-off changed nothing). **esp_map.c v2 built**:
> cache admission keyed on FILE size ≥ 1 MB so the hot small windows cache, and
> evict-on-`p_munmap` to a 2 MB low-water mark fixes the 7.4 MB OOM. **Final
> `git_bench` verdict pending — if splice lands sub-second, proceed with the
> firmware plumbing below.** Full trail: the splice-bench, root-cause and
> second-localization sections of the tradeoff curve.
## The fix — O(depth) TreeBuilder walk
Rebuild only the edited file's ancestor subtree chain onto HEAD's tree. Never
@@ -70,11 +96,24 @@ tree), and path splitting on `/` (paths are repo-relative POSIX).
**Bench it first** (per the established discipline): add a `treebuilder splice→tree`
op to `firmware/src/bin/git_bench.rs` alongside the existing ones and confirm it's
sub-second cold on the real repo, with heap staying healthy (no OOM). Only then wire
it into the firmware.
it into the firmware. While in there, two bench-hygiene fixes: (a) the `edit_path`
seed block does the cold `read_tree(HEAD)` **untimed** (the 77 s number came from
log timestamps), so the `Index::new + read_tree` bench only ever measures warm —
wrap the seed in a timer so a re-run prints the cold number; (b) the "3) THE
PROPOSED FIX — index-free commit tree" comment is now stale — the real-repo run
refuted that path (77 s + OOM) and this handoff supersedes it — retitle it as the
refuted alternative.
## Firmware plumbing (after the bench validates)
1. **`firmware/src/git_sync.rs`**
- **Set the mwindow options at service start** (before the first
`Repository::open`): `git2::opts::set_mwindow_size(256 * 1024)` +
`set_mwindow_mapped_limit(4 * 1024 * 1024)`. Today only `git_bench.rs` sets
them — the shipping service runs libgit2's 32-bit defaults (**32 MB window /
256 MB mapped limit**, mwindow.c:16), so the first pack access on the 570 MB
clone would try to `git__malloc` a 32 MB window and die on the 8 MB PSRAM
heap before the walk even runs.
- Rewrite `stage_and_commit` (currently ~L271332) to the `splice`-walk above.
Drop `add_all`/`update_all`/`index.write`/`index.write_tree`. Keep the
`commit split —` timing log, the `tree unchanged → nothing to publish` check,
@@ -89,13 +128,26 @@ it into the firmware.
- The macOS-cruft filter (`skip_macos_cruft`) is no longer needed — the walk only
ever touches paths the editor explicitly hands it, so `._*`/`.DS_Store` can't
sneak in. (Keep a note; don't silently lose the Spike-14 lesson.)
- **Deliberate behavior change to record:** the walk commits *only* the
editor's dirty set. Files changed on the card outside the editor (e.g. the
card mounted on a Mac) were swept in by `add_all` before; they will now never
be committed, and the working tree will show a permanent diff against HEAD if
inspected on a desktop. Correct for the appliance (it's also what makes the
cruft filter unnecessary), but it must be intentional, not accidental.
2. **Dirty-set source — `firmware/src/persistence.rs` + `main.rs`**
- Writes funnel through `Storage::save_path` (~L296) and deletes through
`Storage::delete_path` (~L332), both `&self`. Accumulate a dirty/deleted set
- Writes funnel through `Storage::save_path` (~L316) and deletes through
`Storage::delete_path` (~L352), both `&self`. Accumulate a dirty/deleted set
(needs `RefCell` interior mutability, or move the set up to `main.rs`).
- `Effect::Publish` handler in `main.rs` (~L222) builds the `PublishRequest` from
that set and clears it on a successful `Pushed`/`UpToDate` outcome.
- **FD budget: `main.rs` (~L413) mounts with `Storage::mount()` = 4 open
files, and the git thread shares that mount.** libgit2 keeps the pack +
`.idx` (+ commit-graph) descriptors open and opens loose objects on top —
that's why `git_bench` needed `mount_for_git` (16). On the real repo the
shipping `:sync` will fail with "no free file descriptors" long before any
latency question. Either mount with the 16-file budget in `main.rs` (the
editor's 2-FD peak coexists fine) or split the budgets some other way.
3. **`esp_map.c` cache fix (same pass) — `firmware/components/libgit2/esp_map.c`**
- Bug: cached buffers are freed only lazily in `p_mmap`'s `evict_for`, so a
@@ -128,11 +180,22 @@ the toy pack understates everything by ~2 orders of magnitude.
## What's proven vs open
**Proven (2026-07-12, real repo):** `odb.write` 142 ms (mmap cache holds);
`index.write` 611 s (whole-tree re-hash via `truncate_racily_clean`, index.c:822 /
index.h:117); index-free `read_tree` 77 s cold; mmap cache OOM at 7.4 MB → zlib
crash. `Repository::open` 88 ms, odb-open ~6 s cold (maps 1.7 MB `.idx`).
**Proven (2026-07-12, real repo):** `index.write` 611 s (whole-tree re-hash via
`truncate_racily_clean`, index.c:822 / index.h:117); index-free `read_tree`
7782 s cold (reproduced across both runs); mmap cache OOM at 7.4 MB → zlib crash
(reproduced). `Repository::open` ~9099 ms, odb-open ~68 s cold (maps 1.7 MB
`.idx`). **Splice walk benched (second run): 6.5 s p50, warm ≈ cold** — O(depth)
shape confirmed (flat reads ~40 KB/write, 6.4 MB heap free), cost is 4 loose
writes × ~1.6 s. `commit(None)` 1.7 s.
**Open:** the TreeBuilder walk is **designed but not yet benched or built.** Confirm
its cold-real-repo latency and heap before wiring. Push (network half, ~6.5 s) is a
separate floor, untouched here.
**Open — the new gating question: why does one loose-object write cost ~1.5 s?**
`odb.write(blob)` measured 142 ms in the first real-repo run but 1.5 s in the
second (same `esp_map.c`, same card, **0 cache hits** the whole second run) — the
two runs are unreconciled. Suspects, cheapest first: (a) FAT free-cluster scan on
the ~740 MB-full card → re-run `sd_bench` as-is on this card; (b) loose-write
internals (filebuf tmp + rename, per-write `git_odb_refresh` readdir) under the
accumulating orphan objects from bench runs → re-provision a fresh clone and A/B;
(c) the ~10 small 4 KB `p_mmap`s per write (sub-64 KB, uncacheable) — bounded
~100 ms, secondary. Also open: whether the esp_map cache earns its keep at all
(0 hits this run), the ref/reflog-update cost on the real repo, and the push
(network ~6.5 s) floor — all untouched here.

View File

@@ -4,10 +4,19 @@
> nor an index-free `read_tree`+`write_tree` is viable on the real
> `jcalixte/notes` clone — **both are O(N_tree)** and blow up on the 570 MB pack /
> 1179-file tree (611 s hash one end, 77 s tree-read + OOM the other; see
> [Real-repo run](#real-repo-run-2026-07-12-jcalixtenotes-570 mb-pack--the-index-is-the-wrong-primitive)
> [Real-repo run](#real-repo-run-2026-07-12-jcalixtenotes-570-mb-pack--the-index-is-the-wrong-primitive)
> below). The commit must be rebuilt with an **O(depth) TreeBuilder walk** —
> patch only the edited file's ancestor subtree chain onto HEAD's tree, never
> materialise all 1179 entries. Shrinking the repo is **not** an option
> materialise all 1179 entries.
>
> **Splice-bench update (2026-07-12, later the same day):** the walk was built
> into `git_bench` and measured on the real repo — **6.5 s, failing the
> sub-second bar.** The O(depth) *shape* holds (flat pack reads, healthy heap),
> but each of its 4 loose-object writes costs **~1.5 s** (isolated `odb.write`
> regressed from the 142 ms above; the mmap cache scored **0 hits** this run).
> Localizing the loose-write cost is now the gating work — see
> [Splice bench](#splice-bench-2026-07-12-second-real-repo-run--the-walk-is-right-the-loose-object-write-is-the-new-wall).
> Shrinking the repo is **not** an option
> ([`../notes/git-sync-images-and-repo-size.md`](../notes/git-sync-images-and-repo-size.md):
> the images are load-bearing for another app), which is exactly why the O(depth)
> mechanism is the only lever left. This note records the cost model and the full
@@ -221,6 +230,139 @@ needs an evict-on-`munmap` fix (drop the cap, free past a low-water mark) so it
never again starve a downstream `git__malloc`, but with the TreeBuilder walk the
pressure it was under largely disappears.
### Splice bench (2026-07-12, second real-repo run) — the walk is right, the loose-object write is the new wall
The O(depth) splice op was added to `git_bench` (1 blob + 3 tree writes onto the
depth-3 path `.claude/commands/bsky.md`, run FIRST so its first iteration is
cold; the index ops moved last so their OOM can't cost the new data — it did
crash again, after everything was logged):
| op | result | reading |
| --- | ---: | --- |
| `splice stage→tree` (1 blob + 3 trees) | **6.5 s p50, warm ≈ cold** | O(depth) confirmed — cost is 4 loose writes × ~1.6 s |
| `commit(None)` orphan obj | 1.7 s p50 | one more loose write |
| `odb.write(blob)` | **1.5 s p50** | ⚠️ was 142 ms in the previous run |
| `repo.index()` load | 524 ms max | matches previous run |
| seed `read_tree(HEAD)` cold (now timed) | 81.6 s | reproduces the 77 s |
| `index-free stage→tree` | 💥 crash, 508 KB heap | reproduces the zlib OOM exactly |
Three readings:
1. **The splice mechanism is validated as a mechanism.** Pack reads stayed flat
(~40 KB per write; 6.4 MB heap free through splice + commit + odb.write), so
it really is O(depth) and it cannot OOM. The 6.5 s is not tree-walk cost.
2. **The wall moved to the loose-object write: ~1.5 s each, ×4 per splice.**
The isolated `odb.write(blob)` — one tiny orphan blob — took 1.5 s where the
raw FAT composite is 86 ms. Projected full commit (splice 6.5 s + commit-obj
1.7 s + ref/reflog update) ≈ **89 s**: enormously better than 611 s, still
far off the bar.
3. **The mmap cache scored 0 hits over the entire run** — the documented
862→142 ms `odb.write` win did **not reproduce** (same `esp_map.c`, same
card). Either the earlier run's conditions differed (orphan-object
population? FAT allocation state?) or the win was misattributed. Whatever the
1.5 s is, it is *not* SD data volume: each write moves ~40 KB read + ~1 KB
written.
### ROOT CAUSE FOUND (2026-07-12, `sd_bench` seek op): FatFS lseek walks the cluster chain
Two `sd_bench` re-runs on the ~740 MB-full card settled it:
1. **Free-cluster-scan hypothesis: refuted.** Raw FAT write ops are unchanged on
the full card — loose-object composite **77 ms p50** (was 86 ms), create
20 ms, rename 10 ms. The card is exonerated a second time.
2. **Long seeks are the cost.** A new op opens the repo's largest packfile
(263 MB — the "570 MB pack" was actually the whole `.git`) read-only and does
seek+read(4 KB): **@offset 0 = 5.8 ms; @end = 198.7 ms** — dead constant
across 20 iters. Without `CONFIG_FATFS_USE_FASTSEEK`, FatFS resolves lseek by
walking the file's FAT cluster chain over SPI: forward from the current
position, **from the chain head on any backward seek**. 263 MB ≈ 16.8k
clusters ≈ ~67 KB of FAT reads ≈ ~190 ms per long walk.
**Why FAT behaves this way:** FAT has no extent map or inode — a file is a
singly-linked list of clusters, and the only way to find "byte 260,000,000" is
to follow that list entry by entry through the allocation table. FatFS walks
forward from the current position when it can, but a backward seek restarts
from the chain head ([FatFS `f_lseek` docs, elm-chan.org](http://elm-chan.org/fsw/ff/doc/lseek.html)).
The fast-seek feature fixes exactly this: a pre-computed **cluster link map
table (CLMT)** per file object, "(fragments + 1) × 2" words, after which "no
FAT access is occured in subsequent f_read/f_write/f_lseek" (same page). On
esp-idf it's `CONFIG_FATFS_USE_FASTSEEK` — the official docs recommend it "for
read-heavy workloads with long backward seeks" and note it does not apply to
files opened in write mode
([ESP-IDF FatFS docs](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/fatfs.html)).
The budget closes: each loose write does ~810 small (~4 KB) `p_mmap`s (freshen
→ trailer/idx probes) interleaved with low-offset reads, so ~8 of them pay a
fresh ~190 ms walk → **~1.5 s per object**. It also explains everything the
cache couldn't: warm ≈ cold (the walk is paid inside `lseek` before any data
moves, and the maps are below the 64 KB cache floor), the 142 ms vs 1.5 s
run-1/run-2 discrepancy (run 1's `odb.write` bench ran first and hammered only
the trailer — the file position stayed there, so its seeks were forward/no-ops),
and a large slice of the 81.6 s `read_tree` (133 windows × backward seeks ≈ 25 s
of walking on top of the 25 MB of data).
**Fix (config, not code): `CONFIG_FATFS_USE_FASTSEEK=y` +
`CONFIG_FATFS_FAST_SEEK_BUFFER_SIZE=256`** (landed in `sdkconfig.defaults`
2026-07-12). Fast seek builds an in-memory cluster-link map per read-mode file —
exactly how the pack is opened — making lseek O(1); write-mode files fall back
to the walk transparently. 256 words = 1 KB per open read-only file, covering
~127 fragments (default 64 covers ~31; a fragmented pack would silently fall
back to slow seeks, so the headroom matters).
**A/B measured (same evening): a 2.3× partial win, not a full one.**
| op | fast-seek off | fast-seek on |
| --- | ---: | ---: |
| `splice stage→tree` | 6.5 s | **2.81 s** |
| `odb.write(blob)` | 1.5 s | **416 ms** |
| `commit(None)` | 1.7 s | **1.72 s — unchanged** |
`odb.write` dropped by almost exactly the ~6 chain walks the model predicted —
the seek theory holds — but two residuals remain: **~400 ms per loose write**
(vs the 77 ms raw-FAT floor) and **`commit(None)`'s ~1.3 s premium over a plain
write, which was never seek-bound at all**. Prime suspect for the commit
premium: strict object creation makes `git_commit_create` validate its parent +
tree OIDs with pack header resolves, and `git_treebuilder_insert` does the same
per inserted entry — `git_bench` grew `odb.read_header(packed)` /
`odb.exists(missing)` probes and strict-off re-benches to test it.
### Second localization round (2026-07-12, run 3b + sd_bench re-run)
**Fast-seek verified on the metal:** re-running the sd_bench seek op with
`CONFIG_FATFS_USE_FASTSEEK=y` dropped `pack seek+read 4KB @end` from
**198.7 ms → 20.4 ms** (the CLMT fits the pack in the 256-word buffer — the
pack is not too fragmented). A far seek is now ~15 ms, i.e. effectively fixed.
**The strict-creation theory is refuted; the probes found the real unit cost:**
| op | p50 | reading |
| --- | ---: | --- |
| `odb.read_header(packed)` | **470 ms** | ONE pack header resolve costs ~½ s |
| `odb.exists(missing)` | **968 ms** (±0.1 ms) | miss path (scan → refresh → rescan) ≈ 2× |
| `commit(None)` strict OFF | 1.80 s | vs 1.93 s strict on — validation is NOT the premium |
| `splice` strict OFF | 5.7 s | noise-worse; also not validation |
The ±0.1 ms constancy of `exists(missing)` = a fixed, deterministic SD-op
sequence. The map counters identify it: **~78 small (~4 KB) `p_mmap` reads per
op** — pack trailer probes, idx fanout reads and delta-base windows, repeated
at the *same offsets* on every freshen/refresh. Post-fast-seek those cost
~20 ms each (~150 ms/op); the rest of `read_header`'s 470 ms is CPU-side
delta-chain inflation on the 160 MHz core plus repeated re-reads. Two other
observations from run 3b: the loose-object orphan population from bench runs
is creeping costs upward (splice 2.81 s → 3.21 s between consecutive
fast-seek runs — a re-provision resets it), and the mmap cache STILL scored 0
hits — because its 64 KB map-length floor excluded exactly these hot small
maps.
**Fix built (esp_map.c v2, pending re-bench):** cache admission re-keyed from
map length to **file size ≥ 1 MB** — the pack/idx's small repeated windows now
cache (RAM hits after first touch) while small mutable working-tree files stay
excluded — plus **evict-on-`p_munmap` down to a 2 MB low-water mark**, fixing
the 7.4 MB OOM from the first real-repo run (released windows are actually
returned to `git__malloc`, so `MWINDOW_MAPPED_LIMIT` stays honest). Expected:
`read_header` collapses toward CPU-only, `odb.write` toward ~150250 ms,
splice at or under the sub-second bar, and no end-of-run zlib OOM.
### The walk is ~1.4 s even at N ≈ 2
Mostly fixed cost — the worktree-diff setup and the second (`update_all`) pass —
@@ -245,7 +387,9 @@ work, ranked:
flat in repo size, small heap, images carried forward untouched. Replaces
`stage_and_commit`'s `add_all`/`update_all`/`index.write`/`write_tree`. Needs the
editor's dirty set (+ deleted set) plumbed to the git service — the editor
already knows both. **Bench the walk on the real repo before wiring it in.**
already knows both. **Benched 2026-07-12: 6.5 s — the shape is right but it
fails the sub-second bar**; wiring it in is blocked on localizing the ~1.5 s
loose-object write (see the splice-bench section above).
2. **Fix the `esp_map.c` cache so it can't OOM (do it alongside #1).** It grew to
7.4 MB resident past its 4 MB cap and starved zlib. Evict on `p_munmap` (not
only lazily on the next `p_mmap`) down to a low-water mark, and lower the cap, so

View File

@@ -15,14 +15,30 @@
* docs/tradeoff-curves/sync-commit-staging.md). We cache the read buffers so a
* given file region is read from the card once and reused.
*
* Correctness: cache ONLY read-only mappings >= ESP_MAP_CACHE_MIN. libgit2 maps
* pack idx/data, the commit-graph, midx and packed-refs — all immutable on this
* device (only loose objects/refs/index are written, none via mmap). The one
* mutable mmap is diff_file.c on small working-tree files (notes.md), which the
* size floor excludes, so a mutable file is never served stale. The writable
* mapping the pack indexer uses (fetch/clone) is excluded by the prot check.
* Identity is (dev, ino, size, mtime, offset, len); for an immutable pack any of
* size/mtime already differs if the file is ever replaced.
* Correctness: cache ONLY read-only mappings of files >= ESP_MAP_CACHE_MIN_FILE
* (the FILE size, not the map length — 2026-07-12b). The hot set turned out to
* be the SMALL maps: pack trailer probes, idx fanout reads and delta-base
* windows repeat at the same offsets on every freshen/refresh, and a map-length
* floor excluded exactly those (0 cache hits over three full real-repo bench
* runs). Keying on file size caches them while still excluding the small
* mutable working-tree files diff_file.c maps (notes.md etc.) — only the pack,
* its idx, a midx and the commit-graph are ever that large, and all are
* immutable on this device (only loose objects/refs/index are written, none
* via mmap). The writable mapping the pack indexer uses (fetch/clone) is
* excluded by the prot check. Identity is (dev, ino, size, mtime, offset, len);
* for an immutable pack any of size/mtime already differs if the file is ever
* replaced. NOTE: on esp-idf's FAT VFS st_dev/st_ino are constant and mtime has
* 2 s granularity, so identity is effectively (size, mtime₂ₛ, offset, len) —
* fine for packs (a replaced pack changes size), weak for small mutable files,
* which is exactly why those must never be cacheable.
*
* Memory discipline (2026-07-12b): the first version freed released buffers
* only lazily on the next p_mmap, so a burst (read_tree of 158 trees) pinned
* 7.4 MB and starved zlib's git__malloc (crash at 508 KB heap) — it defeated
* GIT_OPT_SET_MWINDOW_MAPPED_LIMIT because libgit2 believed the memory was
* returned. Now p_munmap evicts unreferenced entries down to
* ESP_MAP_CACHE_LOW_WATER, so a released window's memory really is returned
* under pressure while a warm working set stays resident.
*
* Limitation: writable/shared mappings are not written back (and not cached).
*/
@@ -48,13 +64,21 @@ int git__mmap_alignment(size_t *alignment)
return 0;
}
/* Only cache mappings at least this large: covers pack idx/windows, excludes the
* small mutable working-tree files diff_file.c maps. */
#define ESP_MAP_CACHE_MIN (64 * 1024)
/* Only cache mappings of files at least this large (FILE size, not map length):
* covers the pack, its idx, midx and commit-graph — and thus the hot small
* windows within them — while excluding the small mutable working-tree files
* diff_file.c maps. */
#define ESP_MAP_CACHE_MIN_FILE (1024 * 1024)
/* Cached-buffer budget in PSRAM. Entries with no live ref are LRU-evicted to
* stay under this; pinned entries may exceed it transiently. */
#define ESP_MAP_CACHE_CAP (4 * 1024 * 1024)
#define ESP_MAP_CACHE_SLOTS 24
/* On p_munmap, unreferenced entries are evicted down to this — the resident
* working set a burst leaves behind. Must leave git__malloc (zlib, mwindow)
* ample heap; the 2026-07-12 OOM run crashed with 7.4 MB resident. */
#define ESP_MAP_CACHE_LOW_WATER (2 * 1024 * 1024)
/* Small maps are numerous (trailer/fanout/delta-base probes), so more slots
* than the window-only design needed; a slot is ~40 B. */
#define ESP_MAP_CACHE_SLOTS 48
struct map_entry {
unsigned char *data;
@@ -109,11 +133,12 @@ static int read_range(int fd, off64_t offset, size_t len, unsigned char *data)
return 0;
}
/* Best-effort: evict unreferenced LRU entries until `need` more bytes fit the
* cap. Pinned entries (refcount > 0) can't be evicted, so the cap is soft. */
static void evict_for(size_t need)
/* Best-effort: evict unreferenced LRU entries until total cached bytes fit
* `budget`. Pinned entries (refcount > 0) can't be evicted, so budgets are
* soft. */
static void evict_to(size_t budget)
{
while (g_cached_bytes + need > ESP_MAP_CACHE_CAP) {
while (g_cached_bytes > budget) {
int victim = -1;
uint32_t oldest = 0xFFFFFFFFu;
int i;
@@ -145,9 +170,11 @@ int p_mmap(git_map *out, size_t len, int prot, int flags, int fd, off64_t offset
out->data = NULL;
out->len = 0;
/* Cache only large, read-only mappings whose file we can identify. */
cacheable = (len >= ESP_MAP_CACHE_MIN) && !(prot & GIT_PROT_WRITE);
if (cacheable && fstat(fd, &st) != 0)
/* Cache only read-only mappings of large files (pack/idx/...) that we can
* identify. The map itself may be small — small repeated maps ARE the hot
* set (see header comment). */
cacheable = !(prot & GIT_PROT_WRITE);
if (cacheable && (fstat(fd, &st) != 0 || st.st_size < ESP_MAP_CACHE_MIN_FILE))
cacheable = 0;
if (cacheable) {
@@ -176,7 +203,8 @@ int p_mmap(git_map *out, size_t len, int prot, int flags, int fd, off64_t offset
g_read_bytes += len;
if (cacheable) {
evict_for(len);
if (len < ESP_MAP_CACHE_CAP)
evict_to(ESP_MAP_CACHE_CAP - len);
for (i = 0; i < ESP_MAP_CACHE_SLOTS; i++) {
if (!g_cache[i].used) {
g_cache[i].data = data;
@@ -208,11 +236,16 @@ int p_munmap(git_map *map)
GIT_ASSERT_ARG(map);
/* Cached buffer: drop a ref, keep it for reuse. Otherwise free it. */
/* Cached buffer: drop a ref and keep it for reuse — but return memory to
* git__malloc under pressure (down to the low-water mark), so a released
* window is really released and MWINDOW_MAPPED_LIMIT stays honest. */
for (i = 0; i < ESP_MAP_CACHE_SLOTS; i++) {
if (g_cache[i].used && g_cache[i].data == map->data) {
if (g_cache[i].refcount > 0)
g_cache[i].refcount--;
if (g_cache[i].refcount == 0 &&
g_cached_bytes > ESP_MAP_CACHE_LOW_WATER)
evict_to(ESP_MAP_CACHE_LOW_WATER);
map->data = NULL;
map->len = 0;
return 0;

View File

@@ -44,6 +44,21 @@ CONFIG_SPIRAM=y
CONFIG_SPIRAM_MODE_OCT=y
CONFIG_SPIRAM_USE_MALLOC=y
# FatFS fast seek (sync-latency root cause, 2026-07-12). Without it, lseek
# resolves by walking the file's FAT cluster chain over SPI — forward from the
# current position, from the CHAIN HEAD on any backward seek. sd_bench measured
# ~190 ms per long seek into the 263 MB packfile (5.8 ms at offset 0), and
# libgit2 pays ~8 such seeks per loose-object write via p_mmap's lseek+read →
# the ~1.5 s/object commit cost (docs/tradeoff-curves/sync-commit-staging.md).
# Fast seek builds an in-memory cluster-link map (CLMT) per READ-mode file,
# making lseek O(1); write-mode files transparently fall back to the walk. The
# buffer is per-open-file, 4 B × size: 256 words = 1 KB, covering ~127 fragments
# (the default 64 covers ~31 — a freshly-provisioned pack is near-contiguous,
# but don't let card aging silently disable the fix; vfs_fat falls back to slow
# seeks when the map doesn't fit).
CONFIG_FATFS_USE_FASTSEEK=y
CONFIG_FATFS_FAST_SEEK_BUFFER_SIZE=256
# Silence the legacy-I2C boot warning. esp-idf-hal's i2c.rs is always compiled
# and binds the old `driver/i2c.h` API, so the legacy driver's TU (and its
# `__attribute__((constructor))` deprecation warning) gets linked into every

View File

@@ -4,9 +4,17 @@
//! The ~8× gap between that and `write_tree`'s 710 ms lives inside libgit2, not
//! FAT — this bench times the git2 ODB/index primitives in isolation to find it.
//!
//! HEADLINE OP (since the 2026-07-12 real-repo run): `splice stage→tree` — the
//! O(depth) TreeBuilder walk that replaces the index-based commit entirely
//! (docs/notes/sync-commit-handoff.md). It runs FIRST so its first iteration is
//! the cold number; acceptance bar: **sub-second cold on the real 570 MB-pack
//! clone, heap staying healthy**. The index paths it supersedes run LAST, for
//! regression tracking — they previously OOM'd, and a late crash can't cost the
//! splice data.
//!
//! Read-mostly on `/sd/repo`: the only writes are unreferenced ("orphan") loose
//! blobs — never reachable from a ref, so never pushed, and gc-able — plus
//! rewrites of the existing index/tree (idempotent). Safe on the test card.
//! blobs/trees/commits — never reachable from a ref, so never pushed, and
//! gc-able. Safe on the test card.
//!
//! Flash with `just flash-gitbench` (needs the `git` feature; env in the recipe).
@@ -14,7 +22,7 @@ use std::time::Instant;
use anyhow::{Context, Result};
use esp_idf_svc::hal::delay::FreeRtos;
use git2::{IndexEntry, IndexTime, ObjectType, Oid, Repository, Signature};
use git2::{IndexEntry, IndexTime, ObjectType, Oid, Repository, Signature, Tree};
use firmware::git_sync::GIT_STACK;
use firmware::persistence::{Storage, REPO_DIR};
@@ -22,8 +30,9 @@ use firmware::persistence::{Storage, REPO_DIR};
const BUILD_TAG: &str = concat!("build ", env!("BUILD_TIME"), " @", env!("BUILD_GIT"));
/// Iterations per op. Small — some ops write to the card, and the first vs rest
/// spread (min vs max) is itself the signal (e.g. write vs freshen-skip). Kept
/// low (3) on the real 570 MB-pack clone so a slow op still finishes in seconds.
/// spread (min vs max) is itself the signal (e.g. cold vs warm, write vs
/// freshen-skip). Kept low (3) on the real 570 MB-pack clone so a slow op still
/// finishes in seconds.
const N: usize = 3;
fn main() -> Result<()> {
@@ -71,10 +80,58 @@ fn run() -> Result<()> {
log::info!("Repository::open {:.1} ms", t.elapsed().as_micros() as f64 / 1000.0);
log_map_stats("open");
// 1) odb.write(blob) in isolation — unique content each iter forces a real
// write (no freshen-skip). This is the single number that localizes it: if
// ~86 ms the ODB write path is fine and the cost is in the tree/ref layer;
// if ~700 ms the cost is inside the ODB write itself (deflate/sha/freshen).
// 1) THE FIX — `splice stage→tree`, the O(depth) TreeBuilder walk
// (docs/notes/sync-commit-handoff.md): patch the edited file's ancestor
// subtree chain onto HEAD's tree; never materialise the 1179-entry index,
// never index.write(), never read_tree the whole tree. Runs FIRST so
// iteration #1 is genuinely cold (only `open` has touched the pack).
let head_tree = repo
.head()?
.peel_to_commit()
.context("HEAD → commit")?
.tree()
.context("HEAD tree")?;
// A nested path that already exists in HEAD's tree, found by an O(depth)
// descent — NOT read_tree, which is itself the 77 s op — so the splice
// REPLACES a real file and rebuilds a real ancestor chain, not just the root.
let edit_path = find_edit_path(&repo, &head_tree)?;
log::info!(
"splice: editing {} (depth {})",
edit_path.join("/"),
edit_path.len()
);
// The blob write is inside the timing: the real commit pays blob + trees, and
// it keeps the number comparable to `index-free stage→tree` below. Not
// measured here: the ref/reflog update (commit(Some("HEAD"))) — flat FAT
// writes, ~350 ms on the toy repo.
bench("splice stage→tree", |i| {
let data = format!("typoena splice bench edit #{i}\n");
let oid = repo.blob(data.as_bytes()).context("write blob")?;
let parts: Vec<&str> = edit_path.iter().map(String::as_str).collect();
splice(&repo, Some(&head_tree), &parts, Some(oid)).map(|_| ())
})?;
log_map_stats("splice");
// 2) commit(None, …) — create a commit OBJECT without moving HEAD or writing a
// reflog (update_ref = None → an orphan commit, gc-able). Isolates commit-
// object creation from the ref-update + reflog cost; splice + this projects
// the full real-repo commit. Reuses the parent's tree (no new tree needed);
// unique message each iter forces a real write.
let parent = repo.head()?.peel_to_commit().context("HEAD → commit")?;
let sig = Signature::now("typoena-bench", "bench@typoena.local").context("sig")?;
bench("commit(None) orphan obj", |i| {
let msg = format!("typoena git_bench orphan commit #{i}");
repo.commit(None, &sig, &sig, &msg, &head_tree, &[&parent])
.map(|_| ())
.context("commit(None)")
})?;
log_map_stats("commit");
// 3) odb.write(blob) in isolation — unique content each iter forces a real
// write (no freshen-skip). If ~100 ms the ODB write path is fine and any
// slow op above is in the tree/ref layer; if ~1 s the cost is inside the
// ODB write itself (deflate/sha/freshen) and the mmap cache regressed.
let odb = repo.odb().context("opening odb")?;
bench("odb.write(blob)", |i| {
let data = format!("typoena git_bench orphan blob #{i} — unique so the write is real\n");
@@ -84,13 +141,51 @@ fn run() -> Result<()> {
})?;
log_map_stats("odb.write");
// 2) on-disk index LOAD (no write). Times loading all ~1179 entries from the
// card and prints the count. We deliberately do NOT bench index.write() any
// more: it calls truncate_racily_clean, which diffs the whole working tree
// 3b) LOCALIZE the commit-vs-blob gap. The fast-seek A/B (2026-07-12) left
// `commit(None)` at 1.7 s while `odb.write` dropped to ~0.4 s — commit
// additionally VALIDATES its parent + tree OIDs against the odb (strict
// object creation → pack header resolves) and freshens the packed tree.
// Price the two suspects, then re-bench commit + splice with strict
// creation OFF. If commit collapses toward odb.write, validation was the
// gap — and git_sync can ship with strict off (every OID it inserts
// comes from HEAD's tree or a blob it just wrote).
let parent_id = parent.id();
let tree_id = head_tree.id();
bench("odb.read_header(packed)", |i| {
let id = if i % 2 == 0 { tree_id } else { parent_id };
odb.read_header(id).map(|_| ()).context("read_header")
})?;
bench("odb.exists(missing)", |i| {
let id = Oid::from_str(&format!("{:040x}", 0xdead_beef_u64 + i as u64))?;
let _ = odb.exists(id); // miss → freshen fails → git_odb_refresh path
Ok(())
})?;
log_map_stats("probes");
// Process-global libgit2 flag; this bench owns the process.
git2::opts::strict_object_creation(false);
bench("commit(None) [strict off]", |i| {
let msg = format!("typoena git_bench strict-off commit #{i}");
repo.commit(None, &sig, &sig, &msg, &head_tree, &[&parent])
.map(|_| ())
.context("commit strict-off")
})?;
bench("splice [strict off]", |i| {
let data = format!("typoena splice strict-off edit #{i}\n");
let oid = repo.blob(data.as_bytes()).context("write blob")?;
let parts: Vec<&str> = edit_path.iter().map(String::as_str).collect();
splice(&repo, Some(&head_tree), &parts, Some(oid)).map(|_| ())
})?;
git2::opts::strict_object_creation(true);
log_map_stats("strict-off");
// 4) on-disk index LOAD (no write). Times loading all ~1179 entries from the
// card and prints the count. We deliberately do NOT bench index.write():
// it calls truncate_racily_clean, which diffs the whole working tree
// against the index and — because a fresh FAT clone makes every entry look
// "racy" (2 s mtime granularity) — re-hashes ~170 MB over SPI, up to ~10 min
// on this repo (proven 2026-07-12, index.write max 611 s). The fix below
// never writes the on-disk index, so that path never runs.
// on this repo (proven 2026-07-12, index.write max 611 s). The splice
// never touches the on-disk index, so that path never runs.
bench("repo.index() load", |_| {
repo.index().map(|_| ()).context("index open")
})?;
@@ -98,73 +193,105 @@ fn run() -> Result<()> {
log::info!("on-disk index has {n_entries} entries");
log_map_stats("index load");
// 3) THE PROPOSED FIX — index-free commit tree (what git_sync::stage_and_commit
// will do). Build the new tree from HEAD + one changed file in a FRESH
// in-memory index: read_tree(HEAD) leaves stamp=0 so truncate_racily_clean
// can NEVER fire; the changed file is written as a blob and added by OID;
// write_tree_to writes to the odb WITHOUT touching the on-disk index. Because
// read_tree seeds the tree cache and add invalidates only the changed path,
// write_tree rebuilds just that path's subtrees — O(changed), not O(1179).
// If this is sub-second on the real repo, the fix is validated.
let head_tree = repo
.head()?
.peel_to_commit()
.context("HEAD → commit")?
.tree()
.context("HEAD tree")?;
// A real path already in the tree, so the add REPLACES it (a realistic edit) and
// write_tree rebuilds its ancestor subtrees — not just the cheap root case.
let edit_path: Vec<u8> = {
let mut seed = git2::Index::new().context("seed index")?;
seed.read_tree(&head_tree).context("seed read_tree")?;
seed.get(0)
.map(|e| e.path)
.unwrap_or_else(|| b"notes.md".to_vec())
};
// 5) REFUTED ALTERNATIVE — the index-free in-memory-index commit
// (read_tree(HEAD) + add + write_tree_to). It dodges truncate_racily_clean
// but is still O(N_tree): the 2026-07-12 real-repo run measured ~77 s for
// the cold read_tree and drove the mmap cache to 7.4 MB (zlib OOM). Kept
// for regression tracking, run LAST so a crash here can't cost the splice
// data above. The cold read_tree is now timed explicitly (the 77 s was
// previously visible only via log timestamps); the ops above warmed only
// ~depth of the ~158 tree windows, so this is still ~cold.
let t = Instant::now();
{
let mut idx = git2::Index::new().context("Index::new")?;
idx.read_tree(&head_tree).context("seed read_tree")?;
}
log::info!(
"index-free: editing existing path {}",
String::from_utf8_lossy(&edit_path)
"seed read_tree(HEAD) cold {:.1} ms",
t.elapsed().as_micros() as f64 / 1000.0
);
log_map_stats("read_tree");
// read_tree alone: populating the in-memory index from HEAD's tree (reads tree
// objects through the mmap cache; NO working-file hashing).
// Warm repeats: windows resident → pure CPU + cache lookups.
bench("Index::new + read_tree", |_| {
let mut idx = git2::Index::new().context("Index::new")?;
idx.read_tree(&head_tree).context("read_tree")?;
Ok(())
})?;
log_map_stats("read_tree");
// Full index-free staging → tree — this REPLACES add_all + index.write +
// write_tree (the ~10-min hang) with an O(changed) path.
let edit_path_bytes = edit_path.join("/").into_bytes();
bench("index-free stage→tree", |i| {
let mut idx = git2::Index::new().context("Index::new")?;
idx.read_tree(&head_tree).context("read_tree")?;
let data = format!("typoena index-free bench edit #{i}\n");
let oid = repo.blob(data.as_bytes()).context("write blob")?;
idx.add(&blob_entry(&edit_path, oid)).context("index.add")?;
idx.add(&blob_entry(&edit_path_bytes, oid)).context("index.add")?;
idx.write_tree_to(&repo).map(|_| ()).context("write_tree_to")
})?;
log_map_stats("index-free");
// 6) commit(None, …) — create a commit OBJECT without moving HEAD or writing a
// reflog (update_ref = None → an orphan commit, gc-able). Isolates commit-
// object creation from the ref-update + reflog cost. Reuses the parent's
// tree (no new tree needed); unique message each iter forces a real write.
let parent = repo.head()?.peel_to_commit().context("HEAD → commit")?;
let tree = parent.tree().context("parent tree")?;
let sig = Signature::now("typoena-bench", "bench@typoena.local").context("sig")?;
bench("commit(None) orphan obj", |i| {
let msg = format!("typoena git_bench orphan commit #{i}");
repo.commit(None, &sig, &sig, &msg, &tree, &[&parent])
.map(|_| ())
.context("commit(None)")
})?;
log_map_stats("commit");
Ok(())
}
/// PROTOTYPE of the real fix (destined for `git_sync::stage_and_commit`): return
/// a new tree OID equal to `base` with `path` set to `new` — `Some(blob)` to
/// add/replace, `None` to delete. Reads ~depth subtree objects, writes ~depth
/// trees; every other entry (all 1179 files, the 150 MB of images) is carried
/// forward by OID without ever being read. `base = None` builds a fresh subtree
/// chain (new file in a new directory). The git_sync version must additionally
/// drop a directory entry when a delete empties its subtree; the bench only
/// exercises replace.
fn splice(repo: &Repository, base: Option<&Tree>, path: &[&str], new: Option<Oid>) -> Result<Oid> {
let (head, rest) = path.split_first().context("splice: empty path")?;
let mut tb = repo.treebuilder(base).context("treebuilder")?;
if rest.is_empty() {
match new {
Some(oid) => {
tb.insert(*head, oid, 0o100644).context("insert blob")?;
}
None => {
let _ = tb.remove(*head); // already absent ⇒ nothing to delete
}
}
} else {
let sub = match base.and_then(|b| b.get_name(head)) {
Some(e) if e.kind() == Some(ObjectType::Tree) => {
Some(repo.find_tree(e.id()).context("loading subtree")?)
}
_ => None, // no such dir yet (or a blob in the way): build from empty
};
let new_sub = splice(repo, sub.as_ref(), rest, new)?;
tb.insert(*head, new_sub, 0o040000).context("insert subtree")?;
}
tb.write().context("treebuilder write")
}
/// Find a real file to "edit": descend the first subtree at each level (capped),
/// then take the first blob of the deepest tree reached. Reads O(depth) tree
/// objects — never `read_tree`/materialise the whole tree (that's the 77 s op
/// this bench exists to retire).
fn find_edit_path(repo: &Repository, root: &Tree) -> Result<Vec<String>> {
let mut path = Vec::new();
let mut cur_id = root.id();
for _ in 0..6 {
let cur = repo.find_tree(cur_id).context("descending tree")?;
match cur.iter().find(|e| e.kind() == Some(ObjectType::Tree)) {
Some(sub) => {
path.push(sub.name().context("non-utf8 tree name")?.to_string());
cur_id = sub.id();
}
None => break,
}
}
let cur = repo.find_tree(cur_id).context("leaf tree")?;
let blob = cur
.iter()
.find(|e| e.kind() == Some(ObjectType::Blob))
.context("no blob along the first-subtree chain — pick an edit path manually")?;
path.push(blob.name().context("non-utf8 blob name")?.to_string());
Ok(path)
}
unsafe extern "C" {
/// Counters from the p_mmap cache in `components/libgit2/esp_map.c`.
fn esp_map_stats(hits: *mut u32, misses: *mut u32, read_kb: *mut u32, cached_kb: *mut u32);

View File

@@ -15,13 +15,14 @@
//! i.e. **two directory-mutating writes** (temp create + rename) per object. This
//! bench times each FAT primitive in isolation, then a composite that mirrors the
//! sequence above, so we can attribute the ~700 ms to specific ops and get a
//! baseline to compare an A1/A2 card or a 20 MHz bus against. It never touches
//! `/sd/repo` — all work is in `/sd/sdbench`, cleaned up at the end.
//! baseline to compare an A1/A2 card or a 20 MHz bus against. All writes go to
//! `/sd/sdbench` (cleaned up at the end); the pack-seek op additionally opens
//! `/sd/repo`'s packfile READ-ONLY — it never writes there.
//!
//! Flash with `just flash-bench`. Needs no `.env`, no `git` feature (pure SD).
use std::fs::{self, File};
use std::io::Write;
use std::io::{Read, Seek, SeekFrom, Write};
use std::time::Instant;
use anyhow::{Context, Result};
@@ -118,9 +119,72 @@ fn run() -> Result<()> {
// Clean up so the card is left as we found it.
fs::remove_dir_all(BENCH_DIR).with_context(|| format!("removing {BENCH_DIR}"))?;
// 7) THE ~1.5 s LOOSE-WRITE SUSPECT (git_bench, 2026-07-12 second real-repo
// run): lseek inside a huge file. Without CONFIG_FATFS_USE_FASTSEEK,
// FatFS resolves lseek by walking the file's FAT cluster chain — forward
// from the current position, from the CHAIN HEAD on any backward seek.
// The 570 MB pack is ~36k clusters ≈ ~146 KB of FAT reads over SPI per
// long walk. `p_mmap` (esp_map.c) does lseek+read per window, and
// libgit2's freshen path probes the pack TRAILER (near the end) while
// tree windows sit at low offsets — so each loose write pays ~one full
// walk. Prediction: "@start" stays ~ms; "@end" costs ~1.5 s per iter.
// If so, the fix is CONFIG_FATFS_USE_FASTSEEK=y (fast-seek applies to
// read-mode files only — exactly how the pack is opened).
match find_pack()? {
Some(pack) => {
let len = fs::metadata(&pack)?.len();
log::info!("pack seek bench: {pack} ({} MB)", len / (1024 * 1024));
if len < 1024 * 1024 {
log::info!("pack too small to show chain-walk cost — skipping (toy card?)");
} else {
let mut f = File::open(&pack).with_context(|| format!("opening {pack}"))?;
let mut buf = vec![0u8; 4096];
// Baseline: rewind + read at the chain head — no walk to resolve.
summarize("pack seek+read 4KB @start", time_each(|_| {
f.seek(SeekFrom::Start(0))?;
f.read_exact(&mut buf)?;
Ok(())
})?);
// Rewind (cheap, measured above), then seek near the end — pays
// one full cluster-chain walk per iteration if fast-seek is off.
let high = len - 4096;
summarize("pack seek+read 4KB @end", time_each(|_| {
f.seek(SeekFrom::Start(0))?;
f.read_exact(&mut buf)?;
f.seek(SeekFrom::Start(high))?;
f.read_exact(&mut buf)?;
Ok(())
})?);
}
}
None => log::info!("no packfile under /sd/repo/.git/objects/pack — skipping seek bench"),
}
Ok(())
}
/// Largest `*.pack` under the repo's pack dir, if the card carries a clone.
/// Skips macOS AppleDouble sidecars (`._pack-*.pack`, 4 KB of Finder metadata) —
/// the Spike-14 cruft in its latest disguise.
fn find_pack() -> Result<Option<String>> {
let Ok(entries) = fs::read_dir("/sd/repo/.git/objects/pack") else {
return Ok(None);
};
let mut best: Option<(u64, String)> = None;
for e in entries.flatten() {
let p = e.path();
let name = p.file_name().and_then(|n| n.to_str()).unwrap_or("");
if name.starts_with("._") || !name.ends_with(".pack") {
continue;
}
let len = fs::metadata(&p).map(|m| m.len()).unwrap_or(0);
if best.as_ref().is_none_or(|(l, _)| len > *l) {
best = Some((len, p.to_string_lossy().into_owned()));
}
}
Ok(best.map(|(_, p)| p))
}
/// Run `op(i)` for `i in 0..N`, returning each call's wall time in microseconds.
fn time_each<F: FnMut(usize) -> Result<()>>(mut op: F) -> Result<Vec<u64>> {
let mut times = Vec::with_capacity(N);