Compare commits
5 Commits
dade5e6bb3
...
ce3204a350
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ce3204a350 | ||
|
|
dd3d70cbec | ||
|
|
2c1c590c9e | ||
|
|
2c24ece3a5 | ||
|
|
da8ca7d8d2 |
@@ -22,6 +22,32 @@ the toy `typoena-test` (`notes.md`) has. The repo cannot be shrunk (the 150 MB o
|
||||
images serve another app — see the images note). So the fix is a **new commit
|
||||
mechanism**, not a tuning knob.
|
||||
|
||||
> **Splice bench result (2026-07-12, later the same day): 6.5 s — do NOT wire it
|
||||
> in yet.** The walk is in `git_bench` (`splice stage→tree`) and its O(depth)
|
||||
> shape is confirmed on the real repo (flat pack reads, heap healthy, no OOM),
|
||||
> but each of its 4 loose-object writes costs **~1.5 s** — isolated `odb.write`
|
||||
> regressed 142 ms → 1.5 s vs the previous run, with **0 mmap-cache hits**.
|
||||
> Projected full commit ≈ 8–9 s.
|
||||
>
|
||||
> **Root cause found the same evening (sd_bench seek op): FatFS lseek walks the
|
||||
> FAT cluster chain** — seek+read(4 KB) into the 263 MB pack costs **198.7 ms at
|
||||
> the end vs 5.8 ms at offset 0**, and each loose write pays ~8 such walks via
|
||||
> the freshen path's small `p_mmap`s → the ~1.5 s. Fix landed in
|
||||
> `sdkconfig.defaults`: `CONFIG_FATFS_USE_FASTSEEK=y` +
|
||||
> `FAST_SEEK_BUFFER_SIZE=256` (O(1) lseek for read-mode files, i.e. the pack —
|
||||
> see [FatFS f_lseek/CLMT](http://elm-chan.org/fsw/ff/doc/lseek.html) and the
|
||||
> [ESP-IDF FatFS docs](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/fatfs.html)).
|
||||
> **Verified on device: far seek 198.7 → 20.4 ms.** The A/B (splice 6.5 → 2.8 s,
|
||||
> odb.write 1.5 s → 416 ms) plus new probes then localized the residual to ~7–8
|
||||
> repeated small (~4 KB) pack reads per op that the mmap cache's 64 KB floor
|
||||
> excluded (`odb.read_header(packed)` = 470 ms; the strict-object-creation
|
||||
> theory was refuted — strict-off changed nothing). **esp_map.c v2 built**:
|
||||
> cache admission keyed on FILE size ≥ 1 MB so the hot small windows cache, and
|
||||
> evict-on-`p_munmap` to a 2 MB low-water mark fixes the 7.4 MB OOM. **Final
|
||||
> `git_bench` verdict pending — if splice lands sub-second, proceed with the
|
||||
> firmware plumbing below.** Full trail: the splice-bench, root-cause and
|
||||
> second-localization sections of the tradeoff curve.
|
||||
|
||||
## The fix — O(depth) TreeBuilder walk
|
||||
|
||||
Rebuild only the edited file's ancestor subtree chain onto HEAD's tree. Never
|
||||
@@ -70,11 +96,24 @@ tree), and path splitting on `/` (paths are repo-relative POSIX).
|
||||
**Bench it first** (per the established discipline): add a `treebuilder splice→tree`
|
||||
op to `firmware/src/bin/git_bench.rs` alongside the existing ones and confirm it's
|
||||
sub-second cold on the real repo, with heap staying healthy (no OOM). Only then wire
|
||||
it into the firmware.
|
||||
it into the firmware. While in there, two bench-hygiene fixes: (a) the `edit_path`
|
||||
seed block does the cold `read_tree(HEAD)` **untimed** (the 77 s number came from
|
||||
log timestamps), so the `Index::new + read_tree` bench only ever measures warm —
|
||||
wrap the seed in a timer so a re-run prints the cold number; (b) the "3) THE
|
||||
PROPOSED FIX — index-free commit tree" comment is now stale — the real-repo run
|
||||
refuted that path (77 s + OOM) and this handoff supersedes it — retitle it as the
|
||||
refuted alternative.
|
||||
|
||||
## Firmware plumbing (after the bench validates)
|
||||
|
||||
1. **`firmware/src/git_sync.rs`**
|
||||
- **Set the mwindow options at service start** (before the first
|
||||
`Repository::open`): `git2::opts::set_mwindow_size(256 * 1024)` +
|
||||
`set_mwindow_mapped_limit(4 * 1024 * 1024)`. Today only `git_bench.rs` sets
|
||||
them — the shipping service runs libgit2's 32-bit defaults (**32 MB window /
|
||||
256 MB mapped limit**, mwindow.c:16), so the first pack access on the 570 MB
|
||||
clone would try to `git__malloc` a 32 MB window and die on the 8 MB PSRAM
|
||||
heap before the walk even runs.
|
||||
- Rewrite `stage_and_commit` (currently ~L271–332) to the `splice`-walk above.
|
||||
Drop `add_all`/`update_all`/`index.write`/`index.write_tree`. Keep the
|
||||
`commit split —` timing log, the `tree unchanged → nothing to publish` check,
|
||||
@@ -89,13 +128,26 @@ it into the firmware.
|
||||
- The macOS-cruft filter (`skip_macos_cruft`) is no longer needed — the walk only
|
||||
ever touches paths the editor explicitly hands it, so `._*`/`.DS_Store` can't
|
||||
sneak in. (Keep a note; don't silently lose the Spike-14 lesson.)
|
||||
- **Deliberate behavior change to record:** the walk commits *only* the
|
||||
editor's dirty set. Files changed on the card outside the editor (e.g. the
|
||||
card mounted on a Mac) were swept in by `add_all` before; they will now never
|
||||
be committed, and the working tree will show a permanent diff against HEAD if
|
||||
inspected on a desktop. Correct for the appliance (it's also what makes the
|
||||
cruft filter unnecessary), but it must be intentional, not accidental.
|
||||
|
||||
2. **Dirty-set source — `firmware/src/persistence.rs` + `main.rs`**
|
||||
- Writes funnel through `Storage::save_path` (~L296) and deletes through
|
||||
`Storage::delete_path` (~L332), both `&self`. Accumulate a dirty/deleted set
|
||||
- Writes funnel through `Storage::save_path` (~L316) and deletes through
|
||||
`Storage::delete_path` (~L352), both `&self`. Accumulate a dirty/deleted set
|
||||
(needs `RefCell` interior mutability, or move the set up to `main.rs`).
|
||||
- `Effect::Publish` handler in `main.rs` (~L222) builds the `PublishRequest` from
|
||||
that set and clears it on a successful `Pushed`/`UpToDate` outcome.
|
||||
- **FD budget: `main.rs` (~L413) mounts with `Storage::mount()` = 4 open
|
||||
files, and the git thread shares that mount.** libgit2 keeps the pack +
|
||||
`.idx` (+ commit-graph) descriptors open and opens loose objects on top —
|
||||
that's why `git_bench` needed `mount_for_git` (16). On the real repo the
|
||||
shipping `:sync` will fail with "no free file descriptors" long before any
|
||||
latency question. Either mount with the 16-file budget in `main.rs` (the
|
||||
editor's 2-FD peak coexists fine) or split the budgets some other way.
|
||||
|
||||
3. **`esp_map.c` cache fix (same pass) — `firmware/components/libgit2/esp_map.c`**
|
||||
- Bug: cached buffers are freed only lazily in `p_mmap`'s `evict_for`, so a
|
||||
@@ -128,11 +180,22 @@ the toy pack understates everything by ~2 orders of magnitude.
|
||||
|
||||
## What's proven vs open
|
||||
|
||||
**Proven (2026-07-12, real repo):** `odb.write` 142 ms (mmap cache holds);
|
||||
`index.write` 611 s (whole-tree re-hash via `truncate_racily_clean`, index.c:822 /
|
||||
index.h:117); index-free `read_tree` 77 s cold; mmap cache OOM at 7.4 MB → zlib
|
||||
crash. `Repository::open` 88 ms, odb-open ~6 s cold (maps 1.7 MB `.idx`).
|
||||
**Proven (2026-07-12, real repo):** `index.write` 611 s (whole-tree re-hash via
|
||||
`truncate_racily_clean`, index.c:822 / index.h:117); index-free `read_tree`
|
||||
77–82 s cold (reproduced across both runs); mmap cache OOM at 7.4 MB → zlib crash
|
||||
(reproduced). `Repository::open` ~90–99 ms, odb-open ~6–8 s cold (maps 1.7 MB
|
||||
`.idx`). **Splice walk benched (second run): 6.5 s p50, warm ≈ cold** — O(depth)
|
||||
shape confirmed (flat reads ~40 KB/write, 6.4 MB heap free), cost is 4 loose
|
||||
writes × ~1.6 s. `commit(None)` 1.7 s.
|
||||
|
||||
**Open:** the TreeBuilder walk is **designed but not yet benched or built.** Confirm
|
||||
its cold-real-repo latency and heap before wiring. Push (network half, ~6.5 s) is a
|
||||
separate floor, untouched here.
|
||||
**Open — the new gating question: why does one loose-object write cost ~1.5 s?**
|
||||
`odb.write(blob)` measured 142 ms in the first real-repo run but 1.5 s in the
|
||||
second (same `esp_map.c`, same card, **0 cache hits** the whole second run) — the
|
||||
two runs are unreconciled. Suspects, cheapest first: (a) FAT free-cluster scan on
|
||||
the ~740 MB-full card → re-run `sd_bench` as-is on this card; (b) loose-write
|
||||
internals (filebuf tmp + rename, per-write `git_odb_refresh` readdir) under the
|
||||
accumulating orphan objects from bench runs → re-provision a fresh clone and A/B;
|
||||
(c) the ~10 small 4 KB `p_mmap`s per write (sub-64 KB, uncacheable) — bounded
|
||||
~100 ms, secondary. Also open: whether the esp_map cache earns its keep at all
|
||||
(0 hits this run), the ref/reflog-update cost on the real repo, and the push
|
||||
(network ~6.5 s) floor — all untouched here.
|
||||
|
||||
@@ -4,10 +4,19 @@
|
||||
> nor an index-free `read_tree`+`write_tree` is viable on the real
|
||||
> `jcalixte/notes` clone — **both are O(N_tree)** and blow up on the 570 MB pack /
|
||||
> 1179-file tree (611 s hash one end, 77 s tree-read + OOM the other; see
|
||||
> [Real-repo run](#real-repo-run-2026-07-12-jcalixtenotes-570 mb-pack--the-index-is-the-wrong-primitive)
|
||||
> [Real-repo run](#real-repo-run-2026-07-12-jcalixtenotes-570-mb-pack--the-index-is-the-wrong-primitive)
|
||||
> below). The commit must be rebuilt with an **O(depth) TreeBuilder walk** —
|
||||
> patch only the edited file's ancestor subtree chain onto HEAD's tree, never
|
||||
> materialise all 1179 entries. Shrinking the repo is **not** an option
|
||||
> materialise all 1179 entries.
|
||||
>
|
||||
> **Splice-bench update (2026-07-12, later the same day):** the walk was built
|
||||
> into `git_bench` and measured on the real repo — **6.5 s, failing the
|
||||
> sub-second bar.** The O(depth) *shape* holds (flat pack reads, healthy heap),
|
||||
> but each of its 4 loose-object writes costs **~1.5 s** (isolated `odb.write`
|
||||
> regressed from the 142 ms above; the mmap cache scored **0 hits** this run).
|
||||
> Localizing the loose-write cost is now the gating work — see
|
||||
> [Splice bench](#splice-bench-2026-07-12-second-real-repo-run--the-walk-is-right-the-loose-object-write-is-the-new-wall).
|
||||
> Shrinking the repo is **not** an option
|
||||
> ([`../notes/git-sync-images-and-repo-size.md`](../notes/git-sync-images-and-repo-size.md):
|
||||
> the images are load-bearing for another app), which is exactly why the O(depth)
|
||||
> mechanism is the only lever left. This note records the cost model and the full
|
||||
@@ -221,6 +230,139 @@ needs an evict-on-`munmap` fix (drop the cap, free past a low-water mark) so it
|
||||
never again starve a downstream `git__malloc`, but with the TreeBuilder walk the
|
||||
pressure it was under largely disappears.
|
||||
|
||||
### Splice bench (2026-07-12, second real-repo run) — the walk is right, the loose-object write is the new wall
|
||||
|
||||
The O(depth) splice op was added to `git_bench` (1 blob + 3 tree writes onto the
|
||||
depth-3 path `.claude/commands/bsky.md`, run FIRST so its first iteration is
|
||||
cold; the index ops moved last so their OOM can't cost the new data — it did
|
||||
crash again, after everything was logged):
|
||||
|
||||
| op | result | reading |
|
||||
| --- | ---: | --- |
|
||||
| `splice stage→tree` (1 blob + 3 trees) | **6.5 s p50, warm ≈ cold** | O(depth) confirmed — cost is 4 loose writes × ~1.6 s |
|
||||
| `commit(None)` orphan obj | 1.7 s p50 | one more loose write |
|
||||
| `odb.write(blob)` | **1.5 s p50** | ⚠️ was 142 ms in the previous run |
|
||||
| `repo.index()` load | 524 ms max | matches previous run |
|
||||
| seed `read_tree(HEAD)` cold (now timed) | 81.6 s | reproduces the 77 s |
|
||||
| `index-free stage→tree` | 💥 crash, 508 KB heap | reproduces the zlib OOM exactly |
|
||||
|
||||
Three readings:
|
||||
|
||||
1. **The splice mechanism is validated as a mechanism.** Pack reads stayed flat
|
||||
(~40 KB per write; 6.4 MB heap free through splice + commit + odb.write), so
|
||||
it really is O(depth) and it cannot OOM. The 6.5 s is not tree-walk cost.
|
||||
2. **The wall moved to the loose-object write: ~1.5 s each, ×4 per splice.**
|
||||
The isolated `odb.write(blob)` — one tiny orphan blob — took 1.5 s where the
|
||||
raw FAT composite is 86 ms. Projected full commit (splice 6.5 s + commit-obj
|
||||
1.7 s + ref/reflog update) ≈ **8–9 s**: enormously better than 611 s, still
|
||||
far off the bar.
|
||||
3. **The mmap cache scored 0 hits over the entire run** — the documented
|
||||
862→142 ms `odb.write` win did **not reproduce** (same `esp_map.c`, same
|
||||
card). Either the earlier run's conditions differed (orphan-object
|
||||
population? FAT allocation state?) or the win was misattributed. Whatever the
|
||||
1.5 s is, it is *not* SD data volume: each write moves ~40 KB read + ~1 KB
|
||||
written.
|
||||
|
||||
### ROOT CAUSE FOUND (2026-07-12, `sd_bench` seek op): FatFS lseek walks the cluster chain
|
||||
|
||||
Two `sd_bench` re-runs on the ~740 MB-full card settled it:
|
||||
|
||||
1. **Free-cluster-scan hypothesis: refuted.** Raw FAT write ops are unchanged on
|
||||
the full card — loose-object composite **77 ms p50** (was 86 ms), create
|
||||
20 ms, rename 10 ms. The card is exonerated a second time.
|
||||
2. **Long seeks are the cost.** A new op opens the repo's largest packfile
|
||||
(263 MB — the "570 MB pack" was actually the whole `.git`) read-only and does
|
||||
seek+read(4 KB): **@offset 0 = 5.8 ms; @end = 198.7 ms** — dead constant
|
||||
across 20 iters. Without `CONFIG_FATFS_USE_FASTSEEK`, FatFS resolves lseek by
|
||||
walking the file's FAT cluster chain over SPI: forward from the current
|
||||
position, **from the chain head on any backward seek**. 263 MB ≈ 16.8k
|
||||
clusters ≈ ~67 KB of FAT reads ≈ ~190 ms per long walk.
|
||||
|
||||
**Why FAT behaves this way:** FAT has no extent map or inode — a file is a
|
||||
singly-linked list of clusters, and the only way to find "byte 260,000,000" is
|
||||
to follow that list entry by entry through the allocation table. FatFS walks
|
||||
forward from the current position when it can, but a backward seek restarts
|
||||
from the chain head ([FatFS `f_lseek` docs, elm-chan.org](http://elm-chan.org/fsw/ff/doc/lseek.html)).
|
||||
The fast-seek feature fixes exactly this: a pre-computed **cluster link map
|
||||
table (CLMT)** per file object, "(fragments + 1) × 2" words, after which "no
|
||||
FAT access is occured in subsequent f_read/f_write/f_lseek" (same page). On
|
||||
esp-idf it's `CONFIG_FATFS_USE_FASTSEEK` — the official docs recommend it "for
|
||||
read-heavy workloads with long backward seeks" and note it does not apply to
|
||||
files opened in write mode
|
||||
([ESP-IDF FatFS docs](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/fatfs.html)).
|
||||
|
||||
The budget closes: each loose write does ~8–10 small (~4 KB) `p_mmap`s (freshen
|
||||
→ trailer/idx probes) interleaved with low-offset reads, so ~8 of them pay a
|
||||
fresh ~190 ms walk → **~1.5 s per object**. It also explains everything the
|
||||
cache couldn't: warm ≈ cold (the walk is paid inside `lseek` before any data
|
||||
moves, and the maps are below the 64 KB cache floor), the 142 ms vs 1.5 s
|
||||
run-1/run-2 discrepancy (run 1's `odb.write` bench ran first and hammered only
|
||||
the trailer — the file position stayed there, so its seeks were forward/no-ops),
|
||||
and a large slice of the 81.6 s `read_tree` (133 windows × backward seeks ≈ 25 s
|
||||
of walking on top of the 25 MB of data).
|
||||
|
||||
**Fix (config, not code): `CONFIG_FATFS_USE_FASTSEEK=y` +
|
||||
`CONFIG_FATFS_FAST_SEEK_BUFFER_SIZE=256`** (landed in `sdkconfig.defaults`
|
||||
2026-07-12). Fast seek builds an in-memory cluster-link map per read-mode file —
|
||||
exactly how the pack is opened — making lseek O(1); write-mode files fall back
|
||||
to the walk transparently. 256 words = 1 KB per open read-only file, covering
|
||||
~127 fragments (default 64 covers ~31; a fragmented pack would silently fall
|
||||
back to slow seeks, so the headroom matters).
|
||||
|
||||
**A/B measured (same evening): a 2.3× partial win, not a full one.**
|
||||
|
||||
| op | fast-seek off | fast-seek on |
|
||||
| --- | ---: | ---: |
|
||||
| `splice stage→tree` | 6.5 s | **2.81 s** |
|
||||
| `odb.write(blob)` | 1.5 s | **416 ms** |
|
||||
| `commit(None)` | 1.7 s | **1.72 s — unchanged** |
|
||||
|
||||
`odb.write` dropped by almost exactly the ~6 chain walks the model predicted —
|
||||
the seek theory holds — but two residuals remain: **~400 ms per loose write**
|
||||
(vs the 77 ms raw-FAT floor) and **`commit(None)`'s ~1.3 s premium over a plain
|
||||
write, which was never seek-bound at all**. Prime suspect for the commit
|
||||
premium: strict object creation makes `git_commit_create` validate its parent +
|
||||
tree OIDs with pack header resolves, and `git_treebuilder_insert` does the same
|
||||
per inserted entry — `git_bench` grew `odb.read_header(packed)` /
|
||||
`odb.exists(missing)` probes and strict-off re-benches to test it.
|
||||
|
||||
### Second localization round (2026-07-12, run 3b + sd_bench re-run)
|
||||
|
||||
**Fast-seek verified on the metal:** re-running the sd_bench seek op with
|
||||
`CONFIG_FATFS_USE_FASTSEEK=y` dropped `pack seek+read 4KB @end` from
|
||||
**198.7 ms → 20.4 ms** (the CLMT fits the pack in the 256-word buffer — the
|
||||
pack is not too fragmented). A far seek is now ~15 ms, i.e. effectively fixed.
|
||||
|
||||
**The strict-creation theory is refuted; the probes found the real unit cost:**
|
||||
|
||||
| op | p50 | reading |
|
||||
| --- | ---: | --- |
|
||||
| `odb.read_header(packed)` | **470 ms** | ONE pack header resolve costs ~½ s |
|
||||
| `odb.exists(missing)` | **968 ms** (±0.1 ms) | miss path (scan → refresh → rescan) ≈ 2× |
|
||||
| `commit(None)` strict OFF | 1.80 s | vs 1.93 s strict on — validation is NOT the premium |
|
||||
| `splice` strict OFF | 5.7 s | noise-worse; also not validation |
|
||||
|
||||
The ±0.1 ms constancy of `exists(missing)` = a fixed, deterministic SD-op
|
||||
sequence. The map counters identify it: **~7–8 small (~4 KB) `p_mmap` reads per
|
||||
op** — pack trailer probes, idx fanout reads and delta-base windows, repeated
|
||||
at the *same offsets* on every freshen/refresh. Post-fast-seek those cost
|
||||
~20 ms each (~150 ms/op); the rest of `read_header`'s 470 ms is CPU-side
|
||||
delta-chain inflation on the 160 MHz core plus repeated re-reads. Two other
|
||||
observations from run 3b: the loose-object orphan population from bench runs
|
||||
is creeping costs upward (splice 2.81 s → 3.21 s between consecutive
|
||||
fast-seek runs — a re-provision resets it), and the mmap cache STILL scored 0
|
||||
hits — because its 64 KB map-length floor excluded exactly these hot small
|
||||
maps.
|
||||
|
||||
**Fix built (esp_map.c v2, pending re-bench):** cache admission re-keyed from
|
||||
map length to **file size ≥ 1 MB** — the pack/idx's small repeated windows now
|
||||
cache (RAM hits after first touch) while small mutable working-tree files stay
|
||||
excluded — plus **evict-on-`p_munmap` down to a 2 MB low-water mark**, fixing
|
||||
the 7.4 MB OOM from the first real-repo run (released windows are actually
|
||||
returned to `git__malloc`, so `MWINDOW_MAPPED_LIMIT` stays honest). Expected:
|
||||
`read_header` collapses toward CPU-only, `odb.write` toward ~150–250 ms,
|
||||
splice at or under the sub-second bar, and no end-of-run zlib OOM.
|
||||
|
||||
### The walk is ~1.4 s even at N ≈ 2
|
||||
|
||||
Mostly fixed cost — the worktree-diff setup and the second (`update_all`) pass —
|
||||
@@ -245,7 +387,9 @@ work, ranked:
|
||||
flat in repo size, small heap, images carried forward untouched. Replaces
|
||||
`stage_and_commit`'s `add_all`/`update_all`/`index.write`/`write_tree`. Needs the
|
||||
editor's dirty set (+ deleted set) plumbed to the git service — the editor
|
||||
already knows both. **Bench the walk on the real repo before wiring it in.**
|
||||
already knows both. **Benched 2026-07-12: 6.5 s — the shape is right but it
|
||||
fails the sub-second bar**; wiring it in is blocked on localizing the ~1.5 s
|
||||
loose-object write (see the splice-bench section above).
|
||||
2. **Fix the `esp_map.c` cache so it can't OOM (do it alongside #1).** It grew to
|
||||
7.4 MB resident past its 4 MB cap and starved zlib. Evict on `p_munmap` (not
|
||||
only lazily on the next `p_mmap`) down to a low-water mark, and lower the cap, so
|
||||
|
||||
@@ -15,14 +15,30 @@
|
||||
* docs/tradeoff-curves/sync-commit-staging.md). We cache the read buffers so a
|
||||
* given file region is read from the card once and reused.
|
||||
*
|
||||
* Correctness: cache ONLY read-only mappings >= ESP_MAP_CACHE_MIN. libgit2 maps
|
||||
* pack idx/data, the commit-graph, midx and packed-refs — all immutable on this
|
||||
* device (only loose objects/refs/index are written, none via mmap). The one
|
||||
* mutable mmap is diff_file.c on small working-tree files (notes.md), which the
|
||||
* size floor excludes, so a mutable file is never served stale. The writable
|
||||
* mapping the pack indexer uses (fetch/clone) is excluded by the prot check.
|
||||
* Identity is (dev, ino, size, mtime, offset, len); for an immutable pack any of
|
||||
* size/mtime already differs if the file is ever replaced.
|
||||
* Correctness: cache ONLY read-only mappings of files >= ESP_MAP_CACHE_MIN_FILE
|
||||
* (the FILE size, not the map length — 2026-07-12b). The hot set turned out to
|
||||
* be the SMALL maps: pack trailer probes, idx fanout reads and delta-base
|
||||
* windows repeat at the same offsets on every freshen/refresh, and a map-length
|
||||
* floor excluded exactly those (0 cache hits over three full real-repo bench
|
||||
* runs). Keying on file size caches them while still excluding the small
|
||||
* mutable working-tree files diff_file.c maps (notes.md etc.) — only the pack,
|
||||
* its idx, a midx and the commit-graph are ever that large, and all are
|
||||
* immutable on this device (only loose objects/refs/index are written, none
|
||||
* via mmap). The writable mapping the pack indexer uses (fetch/clone) is
|
||||
* excluded by the prot check. Identity is (dev, ino, size, mtime, offset, len);
|
||||
* for an immutable pack any of size/mtime already differs if the file is ever
|
||||
* replaced. NOTE: on esp-idf's FAT VFS st_dev/st_ino are constant and mtime has
|
||||
* 2 s granularity, so identity is effectively (size, mtime₂ₛ, offset, len) —
|
||||
* fine for packs (a replaced pack changes size), weak for small mutable files,
|
||||
* which is exactly why those must never be cacheable.
|
||||
*
|
||||
* Memory discipline (2026-07-12b): the first version freed released buffers
|
||||
* only lazily on the next p_mmap, so a burst (read_tree of 158 trees) pinned
|
||||
* 7.4 MB and starved zlib's git__malloc (crash at 508 KB heap) — it defeated
|
||||
* GIT_OPT_SET_MWINDOW_MAPPED_LIMIT because libgit2 believed the memory was
|
||||
* returned. Now p_munmap evicts unreferenced entries down to
|
||||
* ESP_MAP_CACHE_LOW_WATER, so a released window's memory really is returned
|
||||
* under pressure while a warm working set stays resident.
|
||||
*
|
||||
* Limitation: writable/shared mappings are not written back (and not cached).
|
||||
*/
|
||||
@@ -48,13 +64,21 @@ int git__mmap_alignment(size_t *alignment)
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Only cache mappings at least this large: covers pack idx/windows, excludes the
|
||||
* small mutable working-tree files diff_file.c maps. */
|
||||
#define ESP_MAP_CACHE_MIN (64 * 1024)
|
||||
/* Only cache mappings of files at least this large (FILE size, not map length):
|
||||
* covers the pack, its idx, midx and commit-graph — and thus the hot small
|
||||
* windows within them — while excluding the small mutable working-tree files
|
||||
* diff_file.c maps. */
|
||||
#define ESP_MAP_CACHE_MIN_FILE (1024 * 1024)
|
||||
/* Cached-buffer budget in PSRAM. Entries with no live ref are LRU-evicted to
|
||||
* stay under this; pinned entries may exceed it transiently. */
|
||||
#define ESP_MAP_CACHE_CAP (4 * 1024 * 1024)
|
||||
#define ESP_MAP_CACHE_SLOTS 24
|
||||
/* On p_munmap, unreferenced entries are evicted down to this — the resident
|
||||
* working set a burst leaves behind. Must leave git__malloc (zlib, mwindow)
|
||||
* ample heap; the 2026-07-12 OOM run crashed with 7.4 MB resident. */
|
||||
#define ESP_MAP_CACHE_LOW_WATER (2 * 1024 * 1024)
|
||||
/* Small maps are numerous (trailer/fanout/delta-base probes), so more slots
|
||||
* than the window-only design needed; a slot is ~40 B. */
|
||||
#define ESP_MAP_CACHE_SLOTS 48
|
||||
|
||||
struct map_entry {
|
||||
unsigned char *data;
|
||||
@@ -109,11 +133,12 @@ static int read_range(int fd, off64_t offset, size_t len, unsigned char *data)
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Best-effort: evict unreferenced LRU entries until `need` more bytes fit the
|
||||
* cap. Pinned entries (refcount > 0) can't be evicted, so the cap is soft. */
|
||||
static void evict_for(size_t need)
|
||||
/* Best-effort: evict unreferenced LRU entries until total cached bytes fit
|
||||
* `budget`. Pinned entries (refcount > 0) can't be evicted, so budgets are
|
||||
* soft. */
|
||||
static void evict_to(size_t budget)
|
||||
{
|
||||
while (g_cached_bytes + need > ESP_MAP_CACHE_CAP) {
|
||||
while (g_cached_bytes > budget) {
|
||||
int victim = -1;
|
||||
uint32_t oldest = 0xFFFFFFFFu;
|
||||
int i;
|
||||
@@ -145,9 +170,11 @@ int p_mmap(git_map *out, size_t len, int prot, int flags, int fd, off64_t offset
|
||||
out->data = NULL;
|
||||
out->len = 0;
|
||||
|
||||
/* Cache only large, read-only mappings whose file we can identify. */
|
||||
cacheable = (len >= ESP_MAP_CACHE_MIN) && !(prot & GIT_PROT_WRITE);
|
||||
if (cacheable && fstat(fd, &st) != 0)
|
||||
/* Cache only read-only mappings of large files (pack/idx/...) that we can
|
||||
* identify. The map itself may be small — small repeated maps ARE the hot
|
||||
* set (see header comment). */
|
||||
cacheable = !(prot & GIT_PROT_WRITE);
|
||||
if (cacheable && (fstat(fd, &st) != 0 || st.st_size < ESP_MAP_CACHE_MIN_FILE))
|
||||
cacheable = 0;
|
||||
|
||||
if (cacheable) {
|
||||
@@ -176,7 +203,8 @@ int p_mmap(git_map *out, size_t len, int prot, int flags, int fd, off64_t offset
|
||||
g_read_bytes += len;
|
||||
|
||||
if (cacheable) {
|
||||
evict_for(len);
|
||||
if (len < ESP_MAP_CACHE_CAP)
|
||||
evict_to(ESP_MAP_CACHE_CAP - len);
|
||||
for (i = 0; i < ESP_MAP_CACHE_SLOTS; i++) {
|
||||
if (!g_cache[i].used) {
|
||||
g_cache[i].data = data;
|
||||
@@ -208,11 +236,16 @@ int p_munmap(git_map *map)
|
||||
|
||||
GIT_ASSERT_ARG(map);
|
||||
|
||||
/* Cached buffer: drop a ref, keep it for reuse. Otherwise free it. */
|
||||
/* Cached buffer: drop a ref and keep it for reuse — but return memory to
|
||||
* git__malloc under pressure (down to the low-water mark), so a released
|
||||
* window is really released and MWINDOW_MAPPED_LIMIT stays honest. */
|
||||
for (i = 0; i < ESP_MAP_CACHE_SLOTS; i++) {
|
||||
if (g_cache[i].used && g_cache[i].data == map->data) {
|
||||
if (g_cache[i].refcount > 0)
|
||||
g_cache[i].refcount--;
|
||||
if (g_cache[i].refcount == 0 &&
|
||||
g_cached_bytes > ESP_MAP_CACHE_LOW_WATER)
|
||||
evict_to(ESP_MAP_CACHE_LOW_WATER);
|
||||
map->data = NULL;
|
||||
map->len = 0;
|
||||
return 0;
|
||||
|
||||
@@ -44,6 +44,21 @@ CONFIG_SPIRAM=y
|
||||
CONFIG_SPIRAM_MODE_OCT=y
|
||||
CONFIG_SPIRAM_USE_MALLOC=y
|
||||
|
||||
# FatFS fast seek (sync-latency root cause, 2026-07-12). Without it, lseek
|
||||
# resolves by walking the file's FAT cluster chain over SPI — forward from the
|
||||
# current position, from the CHAIN HEAD on any backward seek. sd_bench measured
|
||||
# ~190 ms per long seek into the 263 MB packfile (5.8 ms at offset 0), and
|
||||
# libgit2 pays ~8 such seeks per loose-object write via p_mmap's lseek+read →
|
||||
# the ~1.5 s/object commit cost (docs/tradeoff-curves/sync-commit-staging.md).
|
||||
# Fast seek builds an in-memory cluster-link map (CLMT) per READ-mode file,
|
||||
# making lseek O(1); write-mode files transparently fall back to the walk. The
|
||||
# buffer is per-open-file, 4 B × size: 256 words = 1 KB, covering ~127 fragments
|
||||
# (the default 64 covers ~31 — a freshly-provisioned pack is near-contiguous,
|
||||
# but don't let card aging silently disable the fix; vfs_fat falls back to slow
|
||||
# seeks when the map doesn't fit).
|
||||
CONFIG_FATFS_USE_FASTSEEK=y
|
||||
CONFIG_FATFS_FAST_SEEK_BUFFER_SIZE=256
|
||||
|
||||
# Silence the legacy-I2C boot warning. esp-idf-hal's i2c.rs is always compiled
|
||||
# and binds the old `driver/i2c.h` API, so the legacy driver's TU (and its
|
||||
# `__attribute__((constructor))` deprecation warning) gets linked into every
|
||||
|
||||
@@ -4,9 +4,17 @@
|
||||
//! The ~8× gap between that and `write_tree`'s 710 ms lives inside libgit2, not
|
||||
//! FAT — this bench times the git2 ODB/index primitives in isolation to find it.
|
||||
//!
|
||||
//! HEADLINE OP (since the 2026-07-12 real-repo run): `splice stage→tree` — the
|
||||
//! O(depth) TreeBuilder walk that replaces the index-based commit entirely
|
||||
//! (docs/notes/sync-commit-handoff.md). It runs FIRST so its first iteration is
|
||||
//! the cold number; acceptance bar: **sub-second cold on the real 570 MB-pack
|
||||
//! clone, heap staying healthy**. The index paths it supersedes run LAST, for
|
||||
//! regression tracking — they previously OOM'd, and a late crash can't cost the
|
||||
//! splice data.
|
||||
//!
|
||||
//! Read-mostly on `/sd/repo`: the only writes are unreferenced ("orphan") loose
|
||||
//! blobs — never reachable from a ref, so never pushed, and gc-able — plus
|
||||
//! rewrites of the existing index/tree (idempotent). Safe on the test card.
|
||||
//! blobs/trees/commits — never reachable from a ref, so never pushed, and
|
||||
//! gc-able. Safe on the test card.
|
||||
//!
|
||||
//! Flash with `just flash-gitbench` (needs the `git` feature; env in the recipe).
|
||||
|
||||
@@ -14,7 +22,7 @@ use std::time::Instant;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use esp_idf_svc::hal::delay::FreeRtos;
|
||||
use git2::{IndexEntry, IndexTime, ObjectType, Oid, Repository, Signature};
|
||||
use git2::{IndexEntry, IndexTime, ObjectType, Oid, Repository, Signature, Tree};
|
||||
|
||||
use firmware::git_sync::GIT_STACK;
|
||||
use firmware::persistence::{Storage, REPO_DIR};
|
||||
@@ -22,8 +30,9 @@ use firmware::persistence::{Storage, REPO_DIR};
|
||||
const BUILD_TAG: &str = concat!("build ", env!("BUILD_TIME"), " @", env!("BUILD_GIT"));
|
||||
|
||||
/// Iterations per op. Small — some ops write to the card, and the first vs rest
|
||||
/// spread (min vs max) is itself the signal (e.g. write vs freshen-skip). Kept
|
||||
/// low (3) on the real 570 MB-pack clone so a slow op still finishes in seconds.
|
||||
/// spread (min vs max) is itself the signal (e.g. cold vs warm, write vs
|
||||
/// freshen-skip). Kept low (3) on the real 570 MB-pack clone so a slow op still
|
||||
/// finishes in seconds.
|
||||
const N: usize = 3;
|
||||
|
||||
fn main() -> Result<()> {
|
||||
@@ -71,10 +80,58 @@ fn run() -> Result<()> {
|
||||
log::info!("Repository::open {:.1} ms", t.elapsed().as_micros() as f64 / 1000.0);
|
||||
log_map_stats("open");
|
||||
|
||||
// 1) odb.write(blob) in isolation — unique content each iter forces a real
|
||||
// write (no freshen-skip). This is the single number that localizes it: if
|
||||
// ~86 ms the ODB write path is fine and the cost is in the tree/ref layer;
|
||||
// if ~700 ms the cost is inside the ODB write itself (deflate/sha/freshen).
|
||||
// 1) THE FIX — `splice stage→tree`, the O(depth) TreeBuilder walk
|
||||
// (docs/notes/sync-commit-handoff.md): patch the edited file's ancestor
|
||||
// subtree chain onto HEAD's tree; never materialise the 1179-entry index,
|
||||
// never index.write(), never read_tree the whole tree. Runs FIRST so
|
||||
// iteration #1 is genuinely cold (only `open` has touched the pack).
|
||||
let head_tree = repo
|
||||
.head()?
|
||||
.peel_to_commit()
|
||||
.context("HEAD → commit")?
|
||||
.tree()
|
||||
.context("HEAD tree")?;
|
||||
// A nested path that already exists in HEAD's tree, found by an O(depth)
|
||||
// descent — NOT read_tree, which is itself the 77 s op — so the splice
|
||||
// REPLACES a real file and rebuilds a real ancestor chain, not just the root.
|
||||
let edit_path = find_edit_path(&repo, &head_tree)?;
|
||||
log::info!(
|
||||
"splice: editing {} (depth {})",
|
||||
edit_path.join("/"),
|
||||
edit_path.len()
|
||||
);
|
||||
|
||||
// The blob write is inside the timing: the real commit pays blob + trees, and
|
||||
// it keeps the number comparable to `index-free stage→tree` below. Not
|
||||
// measured here: the ref/reflog update (commit(Some("HEAD"))) — flat FAT
|
||||
// writes, ~350 ms on the toy repo.
|
||||
bench("splice stage→tree", |i| {
|
||||
let data = format!("typoena splice bench edit #{i}\n");
|
||||
let oid = repo.blob(data.as_bytes()).context("write blob")?;
|
||||
let parts: Vec<&str> = edit_path.iter().map(String::as_str).collect();
|
||||
splice(&repo, Some(&head_tree), &parts, Some(oid)).map(|_| ())
|
||||
})?;
|
||||
log_map_stats("splice");
|
||||
|
||||
// 2) commit(None, …) — create a commit OBJECT without moving HEAD or writing a
|
||||
// reflog (update_ref = None → an orphan commit, gc-able). Isolates commit-
|
||||
// object creation from the ref-update + reflog cost; splice + this projects
|
||||
// the full real-repo commit. Reuses the parent's tree (no new tree needed);
|
||||
// unique message each iter forces a real write.
|
||||
let parent = repo.head()?.peel_to_commit().context("HEAD → commit")?;
|
||||
let sig = Signature::now("typoena-bench", "bench@typoena.local").context("sig")?;
|
||||
bench("commit(None) orphan obj", |i| {
|
||||
let msg = format!("typoena git_bench orphan commit #{i}");
|
||||
repo.commit(None, &sig, &sig, &msg, &head_tree, &[&parent])
|
||||
.map(|_| ())
|
||||
.context("commit(None)")
|
||||
})?;
|
||||
log_map_stats("commit");
|
||||
|
||||
// 3) odb.write(blob) in isolation — unique content each iter forces a real
|
||||
// write (no freshen-skip). If ~100 ms the ODB write path is fine and any
|
||||
// slow op above is in the tree/ref layer; if ~1 s the cost is inside the
|
||||
// ODB write itself (deflate/sha/freshen) and the mmap cache regressed.
|
||||
let odb = repo.odb().context("opening odb")?;
|
||||
bench("odb.write(blob)", |i| {
|
||||
let data = format!("typoena git_bench orphan blob #{i} — unique so the write is real\n");
|
||||
@@ -84,13 +141,51 @@ fn run() -> Result<()> {
|
||||
})?;
|
||||
log_map_stats("odb.write");
|
||||
|
||||
// 2) on-disk index LOAD (no write). Times loading all ~1179 entries from the
|
||||
// card and prints the count. We deliberately do NOT bench index.write() any
|
||||
// more: it calls truncate_racily_clean, which diffs the whole working tree
|
||||
// 3b) LOCALIZE the commit-vs-blob gap. The fast-seek A/B (2026-07-12) left
|
||||
// `commit(None)` at 1.7 s while `odb.write` dropped to ~0.4 s — commit
|
||||
// additionally VALIDATES its parent + tree OIDs against the odb (strict
|
||||
// object creation → pack header resolves) and freshens the packed tree.
|
||||
// Price the two suspects, then re-bench commit + splice with strict
|
||||
// creation OFF. If commit collapses toward odb.write, validation was the
|
||||
// gap — and git_sync can ship with strict off (every OID it inserts
|
||||
// comes from HEAD's tree or a blob it just wrote).
|
||||
let parent_id = parent.id();
|
||||
let tree_id = head_tree.id();
|
||||
bench("odb.read_header(packed)", |i| {
|
||||
let id = if i % 2 == 0 { tree_id } else { parent_id };
|
||||
odb.read_header(id).map(|_| ()).context("read_header")
|
||||
})?;
|
||||
bench("odb.exists(missing)", |i| {
|
||||
let id = Oid::from_str(&format!("{:040x}", 0xdead_beef_u64 + i as u64))?;
|
||||
let _ = odb.exists(id); // miss → freshen fails → git_odb_refresh path
|
||||
Ok(())
|
||||
})?;
|
||||
log_map_stats("probes");
|
||||
|
||||
// Process-global libgit2 flag; this bench owns the process.
|
||||
git2::opts::strict_object_creation(false);
|
||||
bench("commit(None) [strict off]", |i| {
|
||||
let msg = format!("typoena git_bench strict-off commit #{i}");
|
||||
repo.commit(None, &sig, &sig, &msg, &head_tree, &[&parent])
|
||||
.map(|_| ())
|
||||
.context("commit strict-off")
|
||||
})?;
|
||||
bench("splice [strict off]", |i| {
|
||||
let data = format!("typoena splice strict-off edit #{i}\n");
|
||||
let oid = repo.blob(data.as_bytes()).context("write blob")?;
|
||||
let parts: Vec<&str> = edit_path.iter().map(String::as_str).collect();
|
||||
splice(&repo, Some(&head_tree), &parts, Some(oid)).map(|_| ())
|
||||
})?;
|
||||
git2::opts::strict_object_creation(true);
|
||||
log_map_stats("strict-off");
|
||||
|
||||
// 4) on-disk index LOAD (no write). Times loading all ~1179 entries from the
|
||||
// card and prints the count. We deliberately do NOT bench index.write():
|
||||
// it calls truncate_racily_clean, which diffs the whole working tree
|
||||
// against the index and — because a fresh FAT clone makes every entry look
|
||||
// "racy" (2 s mtime granularity) — re-hashes ~170 MB over SPI, up to ~10 min
|
||||
// on this repo (proven 2026-07-12, index.write max 611 s). The fix below
|
||||
// never writes the on-disk index, so that path never runs.
|
||||
// on this repo (proven 2026-07-12, index.write max 611 s). The splice
|
||||
// never touches the on-disk index, so that path never runs.
|
||||
bench("repo.index() load", |_| {
|
||||
repo.index().map(|_| ()).context("index open")
|
||||
})?;
|
||||
@@ -98,73 +193,105 @@ fn run() -> Result<()> {
|
||||
log::info!("on-disk index has {n_entries} entries");
|
||||
log_map_stats("index load");
|
||||
|
||||
// 3) THE PROPOSED FIX — index-free commit tree (what git_sync::stage_and_commit
|
||||
// will do). Build the new tree from HEAD + one changed file in a FRESH
|
||||
// in-memory index: read_tree(HEAD) leaves stamp=0 so truncate_racily_clean
|
||||
// can NEVER fire; the changed file is written as a blob and added by OID;
|
||||
// write_tree_to writes to the odb WITHOUT touching the on-disk index. Because
|
||||
// read_tree seeds the tree cache and add invalidates only the changed path,
|
||||
// write_tree rebuilds just that path's subtrees — O(changed), not O(1179).
|
||||
// If this is sub-second on the real repo, the fix is validated.
|
||||
let head_tree = repo
|
||||
.head()?
|
||||
.peel_to_commit()
|
||||
.context("HEAD → commit")?
|
||||
.tree()
|
||||
.context("HEAD tree")?;
|
||||
// A real path already in the tree, so the add REPLACES it (a realistic edit) and
|
||||
// write_tree rebuilds its ancestor subtrees — not just the cheap root case.
|
||||
let edit_path: Vec<u8> = {
|
||||
let mut seed = git2::Index::new().context("seed index")?;
|
||||
seed.read_tree(&head_tree).context("seed read_tree")?;
|
||||
seed.get(0)
|
||||
.map(|e| e.path)
|
||||
.unwrap_or_else(|| b"notes.md".to_vec())
|
||||
};
|
||||
// 5) REFUTED ALTERNATIVE — the index-free in-memory-index commit
|
||||
// (read_tree(HEAD) + add + write_tree_to). It dodges truncate_racily_clean
|
||||
// but is still O(N_tree): the 2026-07-12 real-repo run measured ~77 s for
|
||||
// the cold read_tree and drove the mmap cache to 7.4 MB (zlib OOM). Kept
|
||||
// for regression tracking, run LAST so a crash here can't cost the splice
|
||||
// data above. The cold read_tree is now timed explicitly (the 77 s was
|
||||
// previously visible only via log timestamps); the ops above warmed only
|
||||
// ~depth of the ~158 tree windows, so this is still ~cold.
|
||||
let t = Instant::now();
|
||||
{
|
||||
let mut idx = git2::Index::new().context("Index::new")?;
|
||||
idx.read_tree(&head_tree).context("seed read_tree")?;
|
||||
}
|
||||
log::info!(
|
||||
"index-free: editing existing path {}",
|
||||
String::from_utf8_lossy(&edit_path)
|
||||
"seed read_tree(HEAD) cold {:.1} ms",
|
||||
t.elapsed().as_micros() as f64 / 1000.0
|
||||
);
|
||||
log_map_stats("read_tree");
|
||||
|
||||
// read_tree alone: populating the in-memory index from HEAD's tree (reads tree
|
||||
// objects through the mmap cache; NO working-file hashing).
|
||||
// Warm repeats: windows resident → pure CPU + cache lookups.
|
||||
bench("Index::new + read_tree", |_| {
|
||||
let mut idx = git2::Index::new().context("Index::new")?;
|
||||
idx.read_tree(&head_tree).context("read_tree")?;
|
||||
Ok(())
|
||||
})?;
|
||||
log_map_stats("read_tree");
|
||||
|
||||
// Full index-free staging → tree — this REPLACES add_all + index.write +
|
||||
// write_tree (the ~10-min hang) with an O(changed) path.
|
||||
let edit_path_bytes = edit_path.join("/").into_bytes();
|
||||
bench("index-free stage→tree", |i| {
|
||||
let mut idx = git2::Index::new().context("Index::new")?;
|
||||
idx.read_tree(&head_tree).context("read_tree")?;
|
||||
let data = format!("typoena index-free bench edit #{i}\n");
|
||||
let oid = repo.blob(data.as_bytes()).context("write blob")?;
|
||||
idx.add(&blob_entry(&edit_path, oid)).context("index.add")?;
|
||||
idx.add(&blob_entry(&edit_path_bytes, oid)).context("index.add")?;
|
||||
idx.write_tree_to(&repo).map(|_| ()).context("write_tree_to")
|
||||
})?;
|
||||
log_map_stats("index-free");
|
||||
|
||||
// 6) commit(None, …) — create a commit OBJECT without moving HEAD or writing a
|
||||
// reflog (update_ref = None → an orphan commit, gc-able). Isolates commit-
|
||||
// object creation from the ref-update + reflog cost. Reuses the parent's
|
||||
// tree (no new tree needed); unique message each iter forces a real write.
|
||||
let parent = repo.head()?.peel_to_commit().context("HEAD → commit")?;
|
||||
let tree = parent.tree().context("parent tree")?;
|
||||
let sig = Signature::now("typoena-bench", "bench@typoena.local").context("sig")?;
|
||||
bench("commit(None) orphan obj", |i| {
|
||||
let msg = format!("typoena git_bench orphan commit #{i}");
|
||||
repo.commit(None, &sig, &sig, &msg, &tree, &[&parent])
|
||||
.map(|_| ())
|
||||
.context("commit(None)")
|
||||
})?;
|
||||
log_map_stats("commit");
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// PROTOTYPE of the real fix (destined for `git_sync::stage_and_commit`): return
|
||||
/// a new tree OID equal to `base` with `path` set to `new` — `Some(blob)` to
|
||||
/// add/replace, `None` to delete. Reads ~depth subtree objects, writes ~depth
|
||||
/// trees; every other entry (all 1179 files, the 150 MB of images) is carried
|
||||
/// forward by OID without ever being read. `base = None` builds a fresh subtree
|
||||
/// chain (new file in a new directory). The git_sync version must additionally
|
||||
/// drop a directory entry when a delete empties its subtree; the bench only
|
||||
/// exercises replace.
|
||||
fn splice(repo: &Repository, base: Option<&Tree>, path: &[&str], new: Option<Oid>) -> Result<Oid> {
|
||||
let (head, rest) = path.split_first().context("splice: empty path")?;
|
||||
let mut tb = repo.treebuilder(base).context("treebuilder")?;
|
||||
if rest.is_empty() {
|
||||
match new {
|
||||
Some(oid) => {
|
||||
tb.insert(*head, oid, 0o100644).context("insert blob")?;
|
||||
}
|
||||
None => {
|
||||
let _ = tb.remove(*head); // already absent ⇒ nothing to delete
|
||||
}
|
||||
}
|
||||
} else {
|
||||
let sub = match base.and_then(|b| b.get_name(head)) {
|
||||
Some(e) if e.kind() == Some(ObjectType::Tree) => {
|
||||
Some(repo.find_tree(e.id()).context("loading subtree")?)
|
||||
}
|
||||
_ => None, // no such dir yet (or a blob in the way): build from empty
|
||||
};
|
||||
let new_sub = splice(repo, sub.as_ref(), rest, new)?;
|
||||
tb.insert(*head, new_sub, 0o040000).context("insert subtree")?;
|
||||
}
|
||||
tb.write().context("treebuilder write")
|
||||
}
|
||||
|
||||
/// Find a real file to "edit": descend the first subtree at each level (capped),
|
||||
/// then take the first blob of the deepest tree reached. Reads O(depth) tree
|
||||
/// objects — never `read_tree`/materialise the whole tree (that's the 77 s op
|
||||
/// this bench exists to retire).
|
||||
fn find_edit_path(repo: &Repository, root: &Tree) -> Result<Vec<String>> {
|
||||
let mut path = Vec::new();
|
||||
let mut cur_id = root.id();
|
||||
for _ in 0..6 {
|
||||
let cur = repo.find_tree(cur_id).context("descending tree")?;
|
||||
match cur.iter().find(|e| e.kind() == Some(ObjectType::Tree)) {
|
||||
Some(sub) => {
|
||||
path.push(sub.name().context("non-utf8 tree name")?.to_string());
|
||||
cur_id = sub.id();
|
||||
}
|
||||
None => break,
|
||||
}
|
||||
}
|
||||
let cur = repo.find_tree(cur_id).context("leaf tree")?;
|
||||
let blob = cur
|
||||
.iter()
|
||||
.find(|e| e.kind() == Some(ObjectType::Blob))
|
||||
.context("no blob along the first-subtree chain — pick an edit path manually")?;
|
||||
path.push(blob.name().context("non-utf8 blob name")?.to_string());
|
||||
Ok(path)
|
||||
}
|
||||
|
||||
unsafe extern "C" {
|
||||
/// Counters from the p_mmap cache in `components/libgit2/esp_map.c`.
|
||||
fn esp_map_stats(hits: *mut u32, misses: *mut u32, read_kb: *mut u32, cached_kb: *mut u32);
|
||||
|
||||
@@ -15,13 +15,14 @@
|
||||
//! i.e. **two directory-mutating writes** (temp create + rename) per object. This
|
||||
//! bench times each FAT primitive in isolation, then a composite that mirrors the
|
||||
//! sequence above, so we can attribute the ~700 ms to specific ops and get a
|
||||
//! baseline to compare an A1/A2 card or a 20 MHz bus against. It never touches
|
||||
//! `/sd/repo` — all work is in `/sd/sdbench`, cleaned up at the end.
|
||||
//! baseline to compare an A1/A2 card or a 20 MHz bus against. All writes go to
|
||||
//! `/sd/sdbench` (cleaned up at the end); the pack-seek op additionally opens
|
||||
//! `/sd/repo`'s packfile READ-ONLY — it never writes there.
|
||||
//!
|
||||
//! Flash with `just flash-bench`. Needs no `.env`, no `git` feature (pure SD).
|
||||
|
||||
use std::fs::{self, File};
|
||||
use std::io::Write;
|
||||
use std::io::{Read, Seek, SeekFrom, Write};
|
||||
use std::time::Instant;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
@@ -118,9 +119,72 @@ fn run() -> Result<()> {
|
||||
|
||||
// Clean up so the card is left as we found it.
|
||||
fs::remove_dir_all(BENCH_DIR).with_context(|| format!("removing {BENCH_DIR}"))?;
|
||||
|
||||
// 7) THE ~1.5 s LOOSE-WRITE SUSPECT (git_bench, 2026-07-12 second real-repo
|
||||
// run): lseek inside a huge file. Without CONFIG_FATFS_USE_FASTSEEK,
|
||||
// FatFS resolves lseek by walking the file's FAT cluster chain — forward
|
||||
// from the current position, from the CHAIN HEAD on any backward seek.
|
||||
// The 570 MB pack is ~36k clusters ≈ ~146 KB of FAT reads over SPI per
|
||||
// long walk. `p_mmap` (esp_map.c) does lseek+read per window, and
|
||||
// libgit2's freshen path probes the pack TRAILER (near the end) while
|
||||
// tree windows sit at low offsets — so each loose write pays ~one full
|
||||
// walk. Prediction: "@start" stays ~ms; "@end" costs ~1.5 s per iter.
|
||||
// If so, the fix is CONFIG_FATFS_USE_FASTSEEK=y (fast-seek applies to
|
||||
// read-mode files only — exactly how the pack is opened).
|
||||
match find_pack()? {
|
||||
Some(pack) => {
|
||||
let len = fs::metadata(&pack)?.len();
|
||||
log::info!("pack seek bench: {pack} ({} MB)", len / (1024 * 1024));
|
||||
if len < 1024 * 1024 {
|
||||
log::info!("pack too small to show chain-walk cost — skipping (toy card?)");
|
||||
} else {
|
||||
let mut f = File::open(&pack).with_context(|| format!("opening {pack}"))?;
|
||||
let mut buf = vec![0u8; 4096];
|
||||
// Baseline: rewind + read at the chain head — no walk to resolve.
|
||||
summarize("pack seek+read 4KB @start", time_each(|_| {
|
||||
f.seek(SeekFrom::Start(0))?;
|
||||
f.read_exact(&mut buf)?;
|
||||
Ok(())
|
||||
})?);
|
||||
// Rewind (cheap, measured above), then seek near the end — pays
|
||||
// one full cluster-chain walk per iteration if fast-seek is off.
|
||||
let high = len - 4096;
|
||||
summarize("pack seek+read 4KB @end", time_each(|_| {
|
||||
f.seek(SeekFrom::Start(0))?;
|
||||
f.read_exact(&mut buf)?;
|
||||
f.seek(SeekFrom::Start(high))?;
|
||||
f.read_exact(&mut buf)?;
|
||||
Ok(())
|
||||
})?);
|
||||
}
|
||||
}
|
||||
None => log::info!("no packfile under /sd/repo/.git/objects/pack — skipping seek bench"),
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Largest `*.pack` under the repo's pack dir, if the card carries a clone.
|
||||
/// Skips macOS AppleDouble sidecars (`._pack-*.pack`, 4 KB of Finder metadata) —
|
||||
/// the Spike-14 cruft in its latest disguise.
|
||||
fn find_pack() -> Result<Option<String>> {
|
||||
let Ok(entries) = fs::read_dir("/sd/repo/.git/objects/pack") else {
|
||||
return Ok(None);
|
||||
};
|
||||
let mut best: Option<(u64, String)> = None;
|
||||
for e in entries.flatten() {
|
||||
let p = e.path();
|
||||
let name = p.file_name().and_then(|n| n.to_str()).unwrap_or("");
|
||||
if name.starts_with("._") || !name.ends_with(".pack") {
|
||||
continue;
|
||||
}
|
||||
let len = fs::metadata(&p).map(|m| m.len()).unwrap_or(0);
|
||||
if best.as_ref().is_none_or(|(l, _)| len > *l) {
|
||||
best = Some((len, p.to_string_lossy().into_owned()));
|
||||
}
|
||||
}
|
||||
Ok(best.map(|(_, p)| p))
|
||||
}
|
||||
|
||||
/// Run `op(i)` for `i in 0..N`, returning each call's wall time in microseconds.
|
||||
fn time_each<F: FnMut(usize) -> Result<()>>(mut op: F) -> Result<Vec<u64>> {
|
||||
let mut times = Vec::with_capacity(N);
|
||||
|
||||
Reference in New Issue
Block a user