Compare commits
10 Commits
ce3204a350
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
2660a3e9dd | ||
|
|
79fad4689c | ||
|
|
98fc817b3f | ||
|
|
d306caacf7 | ||
|
|
2166b932b6 | ||
|
|
02b7ed88b6 | ||
|
|
a5edaed810 | ||
|
|
e86a3b8254 | ||
|
|
9309f3f239 | ||
|
|
1a5e1736f5 |
@@ -12,4 +12,3 @@
|
||||
| [`git-sync-images-and-repo-size.md`](git-sync-images-and-repo-size.md) | Why we don't shrink the notes repo — its 153 MB of media is remanso's image CDN, so rewriting history to slim the on-device clone breaks the web app. |
|
||||
| [`boot-time-budget.md`](boot-time-budget.md) | Where the ~4.3 s to cursor goes — and why the ≤ 3 s v1.0 target is hard: one ~1.9 s full refresh is unavoidable at cold boot, so the splash is nearly free. |
|
||||
| [`sync-latency.md`](sync-latency.md) | Where the ~16 s cold `:sync` goes — Wi-Fi + SNTP + one TLS push; why optimistic-retry cut a whole handshake, and why the rest is close to the protocol floor. |
|
||||
| [`sync-commit-handoff.md`](sync-commit-handoff.md) | **Handoff:** the on-device commit must become an O(depth) TreeBuilder walk — `add_all`/`index.write` re-hash the whole 570 MB-pack tree (611 s / OOM) and the real repo has never synced. Design + firmware plumbing + bench steps for the next session. |
|
||||
|
||||
@@ -1,201 +0,0 @@
|
||||
# Handoff — the on-device commit must become an O(depth) TreeBuilder walk
|
||||
|
||||
> **Start here if you're picking up `:sync` performance.** Written 2026-07-12 after
|
||||
> benching `git_bench` on a full clone of the real `jcalixte/notes` repo. Full
|
||||
> analysis + numbers: [`../tradeoff-curves/sync-commit-staging.md`](../tradeoff-curves/sync-commit-staging.md).
|
||||
> Latency memory: `sync-timing`. Why the repo can't shrink:
|
||||
> [`git-sync-images-and-repo-size.md`](git-sync-images-and-repo-size.md).
|
||||
|
||||
## TL;DR
|
||||
|
||||
The current `firmware::git_sync::stage_and_commit` (`add_all` → `index.write` →
|
||||
`write_tree`) **cannot commit the real repo** — it is O(N_tree) and the real repo is
|
||||
1179 files / 158 dirs / 570 MB pack / 150 MB of images. Measured on device:
|
||||
|
||||
- `index.write()` re-hashes the whole working tree → **up to 611 s** (10-min freeze).
|
||||
- The index-free alternative (`Index::new` + `read_tree(HEAD)` + `write_tree_to`)
|
||||
still reads the whole tree cold → **77 s**, and drove the `esp_map.c` mmap cache
|
||||
to 7.4 MB, starving zlib so `repo.blob()` crashed (`zlib (5)`, 508 KB heap left).
|
||||
|
||||
**The real repo has almost certainly never completed a `:sync` on device** — only
|
||||
the toy `typoena-test` (`notes.md`) has. The repo cannot be shrunk (the 150 MB of
|
||||
images serve another app — see the images note). So the fix is a **new commit
|
||||
mechanism**, not a tuning knob.
|
||||
|
||||
> **Splice bench result (2026-07-12, later the same day): 6.5 s — do NOT wire it
|
||||
> in yet.** The walk is in `git_bench` (`splice stage→tree`) and its O(depth)
|
||||
> shape is confirmed on the real repo (flat pack reads, heap healthy, no OOM),
|
||||
> but each of its 4 loose-object writes costs **~1.5 s** — isolated `odb.write`
|
||||
> regressed 142 ms → 1.5 s vs the previous run, with **0 mmap-cache hits**.
|
||||
> Projected full commit ≈ 8–9 s.
|
||||
>
|
||||
> **Root cause found the same evening (sd_bench seek op): FatFS lseek walks the
|
||||
> FAT cluster chain** — seek+read(4 KB) into the 263 MB pack costs **198.7 ms at
|
||||
> the end vs 5.8 ms at offset 0**, and each loose write pays ~8 such walks via
|
||||
> the freshen path's small `p_mmap`s → the ~1.5 s. Fix landed in
|
||||
> `sdkconfig.defaults`: `CONFIG_FATFS_USE_FASTSEEK=y` +
|
||||
> `FAST_SEEK_BUFFER_SIZE=256` (O(1) lseek for read-mode files, i.e. the pack —
|
||||
> see [FatFS f_lseek/CLMT](http://elm-chan.org/fsw/ff/doc/lseek.html) and the
|
||||
> [ESP-IDF FatFS docs](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/fatfs.html)).
|
||||
> **Verified on device: far seek 198.7 → 20.4 ms.** The A/B (splice 6.5 → 2.8 s,
|
||||
> odb.write 1.5 s → 416 ms) plus new probes then localized the residual to ~7–8
|
||||
> repeated small (~4 KB) pack reads per op that the mmap cache's 64 KB floor
|
||||
> excluded (`odb.read_header(packed)` = 470 ms; the strict-object-creation
|
||||
> theory was refuted — strict-off changed nothing). **esp_map.c v2 built**:
|
||||
> cache admission keyed on FILE size ≥ 1 MB so the hot small windows cache, and
|
||||
> evict-on-`p_munmap` to a 2 MB low-water mark fixes the 7.4 MB OOM. **Final
|
||||
> `git_bench` verdict pending — if splice lands sub-second, proceed with the
|
||||
> firmware plumbing below.** Full trail: the splice-bench, root-cause and
|
||||
> second-localization sections of the tradeoff curve.
|
||||
|
||||
## The fix — O(depth) TreeBuilder walk
|
||||
|
||||
Rebuild only the edited file's ancestor subtree chain onto HEAD's tree. Never
|
||||
materialise the 1179-entry index; never `index.write()`; never `read_tree` the whole
|
||||
tree. Cost is O(depth × dirty_files), flat in repo size, tiny heap, and it carries
|
||||
every unchanged entry (all 260 images, the other 1176 files) forward untouched —
|
||||
which means **the device doesn't even need the images in its working tree.**
|
||||
|
||||
Sketch (git2 0.20 API — all present: `Repository::treebuilder(Option<&Tree>)`,
|
||||
`TreeBuilder::{insert,remove,write}`, `Tree::{get_name,get_path}`,
|
||||
`TreeEntry::{to_object,filemode}`):
|
||||
|
||||
```rust
|
||||
/// Return a new tree OID = `base` with `path` set to `new` (Some(blob_oid) to
|
||||
/// add/replace, None to delete). Reads ~depth subtree objects, writes ~depth trees.
|
||||
fn splice(repo: &Repository, base: &Tree, path: &[&str], new: Option<Oid>) -> Result<Oid> {
|
||||
let (head, rest) = path.split_first().unwrap();
|
||||
if rest.is_empty() {
|
||||
// leaf level: patch this tree directly
|
||||
let mut tb = repo.treebuilder(Some(base))?;
|
||||
match new {
|
||||
Some(oid) => tb.insert(head, oid, 0o100644)?, // FileMode::Blob
|
||||
None => { tb.remove(head).ok(); } // delete
|
||||
};
|
||||
return Ok(tb.write()?);
|
||||
}
|
||||
// descend: find (or synthesize) the subtree, recurse, re-insert its new OID
|
||||
let sub = base.get_name(head)
|
||||
.and_then(|e| e.to_object(repo).ok())
|
||||
.and_then(|o| o.peel_to_tree().ok());
|
||||
let empty; let sub_ref = match &sub { Some(t) => t, None => { empty = repo.treebuilder(None)?; /* ... */ unreachable!() } };
|
||||
let new_sub = splice(repo, sub_ref, rest, new)?;
|
||||
let mut tb = repo.treebuilder(Some(base))?;
|
||||
// if new_sub is the empty tree and this was a delete, remove the dir entry instead
|
||||
tb.insert(head, new_sub, 0o040000)?; // FileMode::Tree
|
||||
Ok(tb.write()?)
|
||||
}
|
||||
```
|
||||
|
||||
Then, for the dirty set, fold each change through `splice` (thread the running root
|
||||
tree), and `commit(Some("HEAD"), …, &final_tree, &[&parent])`. Edge cases to handle:
|
||||
new files where an intermediate dir doesn't exist yet (build subtree from `None`),
|
||||
deletes that empty a directory (remove the dir entry rather than insert an empty
|
||||
tree), and path splitting on `/` (paths are repo-relative POSIX).
|
||||
|
||||
**Bench it first** (per the established discipline): add a `treebuilder splice→tree`
|
||||
op to `firmware/src/bin/git_bench.rs` alongside the existing ones and confirm it's
|
||||
sub-second cold on the real repo, with heap staying healthy (no OOM). Only then wire
|
||||
it into the firmware. While in there, two bench-hygiene fixes: (a) the `edit_path`
|
||||
seed block does the cold `read_tree(HEAD)` **untimed** (the 77 s number came from
|
||||
log timestamps), so the `Index::new + read_tree` bench only ever measures warm —
|
||||
wrap the seed in a timer so a re-run prints the cold number; (b) the "3) THE
|
||||
PROPOSED FIX — index-free commit tree" comment is now stale — the real-repo run
|
||||
refuted that path (77 s + OOM) and this handoff supersedes it — retitle it as the
|
||||
refuted alternative.
|
||||
|
||||
## Firmware plumbing (after the bench validates)
|
||||
|
||||
1. **`firmware/src/git_sync.rs`**
|
||||
- **Set the mwindow options at service start** (before the first
|
||||
`Repository::open`): `git2::opts::set_mwindow_size(256 * 1024)` +
|
||||
`set_mwindow_mapped_limit(4 * 1024 * 1024)`. Today only `git_bench.rs` sets
|
||||
them — the shipping service runs libgit2's 32-bit defaults (**32 MB window /
|
||||
256 MB mapped limit**, mwindow.c:16), so the first pack access on the 570 MB
|
||||
clone would try to `git__malloc` a 32 MB window and die on the 8 MB PSRAM
|
||||
heap before the walk even runs.
|
||||
- Rewrite `stage_and_commit` (currently ~L271–332) to the `splice`-walk above.
|
||||
Drop `add_all`/`update_all`/`index.write`/`index.write_tree`. Keep the
|
||||
`commit split —` timing log, the `tree unchanged → nothing to publish` check,
|
||||
and the signature/message code.
|
||||
- `PublishRequest` (~L79) is currently an **empty struct** — it must carry the
|
||||
dirty set: `{ changed: Vec<(String /*repo-rel path*/, ...)>, deleted: Vec<String> }`.
|
||||
The commit needs the blob content or a way to read it; simplest is to pass the
|
||||
paths and let the git thread `repo.blob_path(abs)` / read `/sd/repo/<path>`.
|
||||
- `reconcile_onto_origin` (~L377/L394) uses `repo.reset(Mixed)` — with an
|
||||
index-free commit there's no index to reset; switch to `ResetType::Soft` (move
|
||||
HEAD only) or drop the reset and just re-`splice` onto the new origin tip.
|
||||
- The macOS-cruft filter (`skip_macos_cruft`) is no longer needed — the walk only
|
||||
ever touches paths the editor explicitly hands it, so `._*`/`.DS_Store` can't
|
||||
sneak in. (Keep a note; don't silently lose the Spike-14 lesson.)
|
||||
- **Deliberate behavior change to record:** the walk commits *only* the
|
||||
editor's dirty set. Files changed on the card outside the editor (e.g. the
|
||||
card mounted on a Mac) were swept in by `add_all` before; they will now never
|
||||
be committed, and the working tree will show a permanent diff against HEAD if
|
||||
inspected on a desktop. Correct for the appliance (it's also what makes the
|
||||
cruft filter unnecessary), but it must be intentional, not accidental.
|
||||
|
||||
2. **Dirty-set source — `firmware/src/persistence.rs` + `main.rs`**
|
||||
- Writes funnel through `Storage::save_path` (~L316) and deletes through
|
||||
`Storage::delete_path` (~L352), both `&self`. Accumulate a dirty/deleted set
|
||||
(needs `RefCell` interior mutability, or move the set up to `main.rs`).
|
||||
- `Effect::Publish` handler in `main.rs` (~L222) builds the `PublishRequest` from
|
||||
that set and clears it on a successful `Pushed`/`UpToDate` outcome.
|
||||
- **FD budget: `main.rs` (~L413) mounts with `Storage::mount()` = 4 open
|
||||
files, and the git thread shares that mount.** libgit2 keeps the pack +
|
||||
`.idx` (+ commit-graph) descriptors open and opens loose objects on top —
|
||||
that's why `git_bench` needed `mount_for_git` (16). On the real repo the
|
||||
shipping `:sync` will fail with "no free file descriptors" long before any
|
||||
latency question. Either mount with the 16-file budget in `main.rs` (the
|
||||
editor's 2-FD peak coexists fine) or split the budgets some other way.
|
||||
|
||||
3. **`esp_map.c` cache fix (same pass) — `firmware/components/libgit2/esp_map.c`**
|
||||
- Bug: cached buffers are freed only lazily in `p_mmap`'s `evict_for`, so a
|
||||
window libgit2 has `p_munmap`'d stays resident until the *next* map — defeating
|
||||
`MWINDOW_MAPPED_LIMIT` and starving non-mmap `git__malloc` (zlib).
|
||||
- Fix: on `p_munmap` when refcount hits 0, evict down to a low-water mark (or free
|
||||
outright); lower `ESP_MAP_CACHE_CAP` (4 MB → ~1.5–2 MB) and/or `SLOTS`.
|
||||
- ⚠️ Editing this `.c` needs the fingerprint dance or the change won't rebuild —
|
||||
see the `esp-idf-component-rebuild` memory:
|
||||
`rm -rf firmware/target/xtensa-esp32s3-espidf/release/.fingerprint/esp-idf-sys-*`
|
||||
then rebuild.
|
||||
|
||||
## How to bench / flash
|
||||
|
||||
`git_bench.rs` runs git ops on the 96 KB `GIT_STACK` thread (the main task stack
|
||||
overflows on these ops — that's why the real service has a dedicated thread). It's
|
||||
Rust-only, so a plain rebuild picks it up (no fingerprint dance unless you also
|
||||
touched `esp_map.c`).
|
||||
|
||||
```
|
||||
just flash-gitbench
|
||||
# = . ~/export-esp.sh && LIBGIT2_SRC=<repo>/firmware/components/libgit2/vendor \
|
||||
# LIBGIT2_NO_VENDOR=1 PKG_CONFIG_ALLOW_CROSS=1 \
|
||||
# PKG_CONFIG_LIBDIR=<repo>/firmware/pkgconfig \
|
||||
# cargo run --release --bin git_bench --features git
|
||||
```
|
||||
|
||||
Bench on the **real repo** clone (`/sd/repo` = full `jcalixte/notes`), not the toy —
|
||||
the toy pack understates everything by ~2 orders of magnitude.
|
||||
|
||||
## What's proven vs open
|
||||
|
||||
**Proven (2026-07-12, real repo):** `index.write` 611 s (whole-tree re-hash via
|
||||
`truncate_racily_clean`, index.c:822 / index.h:117); index-free `read_tree`
|
||||
77–82 s cold (reproduced across both runs); mmap cache OOM at 7.4 MB → zlib crash
|
||||
(reproduced). `Repository::open` ~90–99 ms, odb-open ~6–8 s cold (maps 1.7 MB
|
||||
`.idx`). **Splice walk benched (second run): 6.5 s p50, warm ≈ cold** — O(depth)
|
||||
shape confirmed (flat reads ~40 KB/write, 6.4 MB heap free), cost is 4 loose
|
||||
writes × ~1.6 s. `commit(None)` 1.7 s.
|
||||
|
||||
**Open — the new gating question: why does one loose-object write cost ~1.5 s?**
|
||||
`odb.write(blob)` measured 142 ms in the first real-repo run but 1.5 s in the
|
||||
second (same `esp_map.c`, same card, **0 cache hits** the whole second run) — the
|
||||
two runs are unreconciled. Suspects, cheapest first: (a) FAT free-cluster scan on
|
||||
the ~740 MB-full card → re-run `sd_bench` as-is on this card; (b) loose-write
|
||||
internals (filebuf tmp + rename, per-write `git_odb_refresh` readdir) under the
|
||||
accumulating orphan objects from bench runs → re-provision a fresh clone and A/B;
|
||||
(c) the ~10 small 4 KB `p_mmap`s per write (sub-64 KB, uncacheable) — bounded
|
||||
~100 ms, secondary. Also open: whether the esp_map cache earns its keep at all
|
||||
(0 hits this run), the ref/reflog-update cost on the real repo, and the push
|
||||
(network ~6.5 s) floor — all untouched here.
|
||||
@@ -11,4 +11,4 @@
|
||||
| --- | --- |
|
||||
| [`wifi-auto-sync.md`](wifi-auto-sync.md) | `auto_sync` interval vs Wi-Fi energy (a `1/T` hyperbola) — why the default is 10 min and opportunistic, not a wall-clock timer. |
|
||||
| [`epd-refresh-latency.md`](epd-refresh-latency.md) | E-ink refresh latency vs rows driven — the full / full-area-partial / windowed-Y cost model behind typing responsiveness and the boot splash→editor swap. |
|
||||
| [`sync-commit-staging.md`](sync-commit-staging.md) | Commit-staging strategy vs working-tree size — `add_all(["*"])` (O(tree) FAT walk) vs explicit-path (O(churn)); the walk-vs-writes split that decides whether explicit staging is worth it. |
|
||||
| [`sync-commit-staging.md`](sync-commit-staging.md) | Commit-staging strategy vs working-tree size — RESOLVED: every index-based path is O(N_tree) and fails on the real repo (611 s / OOM); the O(depth) TreeBuilder splice is benched at ~2–2.8 s and ships. Holds the full measurement trail plus the firmware plumbing plan (merged from the retired handoff note). |
|
||||
|
||||
@@ -9,14 +9,17 @@
|
||||
> patch only the edited file's ancestor subtree chain onto HEAD's tree, never
|
||||
> materialise all 1179 entries.
|
||||
>
|
||||
> **Splice-bench update (2026-07-12, later the same day):** the walk was built
|
||||
> into `git_bench` and measured on the real repo — **6.5 s, failing the
|
||||
> sub-second bar.** The O(depth) *shape* holds (flat pack reads, healthy heap),
|
||||
> but each of its 4 loose-object writes costs **~1.5 s** (isolated `odb.write`
|
||||
> regressed from the 142 ms above; the mmap cache scored **0 hits** this run).
|
||||
> Localizing the loose-write cost is now the gating work — see
|
||||
> [Splice bench](#splice-bench-2026-07-12-second-real-repo-run--the-walk-is-right-the-loose-object-write-is-the-new-wall).
|
||||
> Shrinking the repo is **not** an option
|
||||
> **Final state (2026-07-13, after four localization rounds):** the splice walk
|
||||
> is benched and the block is lifted. It first measured 6.5 s (each loose-object
|
||||
> write cost ~1.5 s); the root cause was FatFS's lseek cluster-chain walk, and
|
||||
> the fast-seek fix cut it to **2.8 s cold / ~2 s steady-state**. The `esp_map.c`
|
||||
> window cache was **removed entirely** (0 hits across four instrumented runs —
|
||||
> `mwindow` absorbs any true repetition above `p_mmap`). The sub-second bar
|
||||
> failed, but ~2–2.8 s against 611 s/OOM for every alternative ships: a full
|
||||
> cold real-repo `:sync` lands at ~9–10 s. **Decision: wire the splice in —
|
||||
> done 2026-07-13** ([how it landed](#the-fix--wiring-the-odepth-splice-into-the-firmware),
|
||||
> merged from the retired `notes/sync-commit-handoff.md`; on-device `:sync`
|
||||
> against the real repo still pending). Shrinking the repo is **not** an option
|
||||
> ([`../notes/git-sync-images-and-repo-size.md`](../notes/git-sync-images-and-repo-size.md):
|
||||
> the images are load-bearing for another app), which is exactly why the O(depth)
|
||||
> mechanism is the only lever left. This note records the cost model and the full
|
||||
@@ -354,7 +357,7 @@ fast-seek runs — a re-provision resets it), and the mmap cache STILL scored 0
|
||||
hits — because its 64 KB map-length floor excluded exactly these hot small
|
||||
maps.
|
||||
|
||||
**Fix built (esp_map.c v2, pending re-bench):** cache admission re-keyed from
|
||||
**Fix built (esp_map.c v2):** cache admission re-keyed from
|
||||
map length to **file size ≥ 1 MB** — the pack/idx's small repeated windows now
|
||||
cache (RAM hits after first touch) while small mutable working-tree files stay
|
||||
excluded — plus **evict-on-`p_munmap` down to a 2 MB low-water mark**, fixing
|
||||
@@ -363,6 +366,71 @@ returned to `git__malloc`, so `MWINDOW_MAPPED_LIMIT` stays honest). Expected:
|
||||
`read_header` collapses toward CPU-only, `odb.write` toward ~150–250 ms,
|
||||
splice at or under the sub-second bar, and no end-of-run zlib OOM.
|
||||
|
||||
### Final bench (run 4, esp_map v2) — memory fix works, cache theory dead, bar failed
|
||||
|
||||
Run 4 (2026-07-12 evening, same card state as run 3b plus its orphans):
|
||||
|
||||
| op | run 3b (fast-seek) | run 4 (+ esp_map v2) |
|
||||
| --- | ---: | ---: |
|
||||
| `splice stage→tree` (cold, first op) | 2.81 s | **2.83 s — unchanged** |
|
||||
| `splice` again (warm, strict-off phase) | 3.21 s | 1.95 s |
|
||||
| `commit(None)` | 1.72 s | 713 ms |
|
||||
| `odb.write(blob)` | 416 ms | 366 ms |
|
||||
| `odb.read_header(packed)` | 470 ms | 412 ms |
|
||||
| `odb.exists(missing)` | 968 ms | 852 ms |
|
||||
| mmap-cache hits | 0 | **0** (313 misses) |
|
||||
| cache resident / heap free | grew to 7.4 MB → zlib OOM | **1833 KB flat / 6.4 MB free all run** |
|
||||
|
||||
Three findings:
|
||||
|
||||
1. **The memory discipline is verified.** Resident sits at 1833 KB through
|
||||
every phase (under the 2 MB low-water, so nothing is being churned) and
|
||||
heap never drops below 6.2 MB. The one uncaptured datum is the index-free
|
||||
`read_tree` tail (the section that OOM'd runs 1–3) — the monitor was cut
|
||||
before it ran. Not blocking: the shipping splice path never calls
|
||||
`read_tree`; the tail would only re-confirm eviction under burst.
|
||||
2. **The repeated-small-window theory is REFUTED — theory #3 down** (after
|
||||
strict-creation and free-cluster-scan). v2 demonstrably admits and retains
|
||||
the small maps now — the 1833 KB resident *is* them, held below low-water so
|
||||
nothing is evicted before reuse — and still scored 0 hits in 313 misses.
|
||||
So the ~8 small reads per loose write hit **unique (offset, len) every
|
||||
time**: `mwindow` was already absorbing any true repetition above `p_mmap`,
|
||||
and what reaches the emulation layer is distinct data (different objects,
|
||||
different delta bases). A window cache cannot help. The residual
|
||||
~360 ms/loose-write ≈ 8 distinct small SD round-trips × ~45 ms each
|
||||
(post-fast-seek) — I/O count, not I/O size or seek cost.
|
||||
3. **Within-run drift cuts both ways, so cross-run tables are mushy.** In this
|
||||
single run `commit(None)` degraded 713 ms → 1.79 s between the early and
|
||||
late (strict-off) phases, while splice *improved* 2.83 → 1.95 s. Two
|
||||
competing effects: first-touch warm-up fading (CLMT build, first pack
|
||||
reads — helps later ops) and orphan loose objects accumulating in
|
||||
`.git/objects/xx/` slowing every freshen existence check (FAT directory
|
||||
lookups are linear scans — hurts later ops). Steady-state on a clean
|
||||
objects dir ≈ **~2 s per splice+commit**.
|
||||
|
||||
**Run 5 (confirmation, cache removed entirely):** esp_map.c stripped back to
|
||||
plain malloc-read/free-at-munmap (stats counters kept). Read pattern
|
||||
**byte-identical** to run 4 at every checkpoint (118 maps / 2314 KB after
|
||||
splice, 148/2434, 163/2494, 208/2674) — except the strict-off phase did **15
|
||||
fewer reads (~60 KB)** than v2: the low-water eviction had been kicking out
|
||||
buffers mwindow still wanted, forcing re-reads. The cache was marginally
|
||||
worse than nothing. Warm splice identical (1953 vs 1949 ms); the cold-op
|
||||
+10–15 % drift (splice 2.83 → 3.26 s) is the known orphan-creep signature,
|
||||
not the removal. Run 5 also reframes run 4's "resident": **1854 KB `live` is
|
||||
mwindow's open-window working set** (pinned mappings under the bench's 4 MB
|
||||
mapped limit), not retained cache buffers — the cache had been retaining
|
||||
essentially nothing. Removal CONFIRMED free; simpler emulation ships.
|
||||
|
||||
**Verdict: the sub-second bar FAILED — wire the splice in anyway.** The bar
|
||||
was aspirational; measured reality is ~2–2.8 s to commit on the real
|
||||
263 MB-pack repo versus 611 s (or a hard OOM) for every alternative benched.
|
||||
That puts a full real-repo `:sync` at roughly **9–10 s cold**, which ships.
|
||||
The remaining ~2 s has survived four localization rounds; the next suspect —
|
||||
FAT *directory-op* cost in the freshen/refresh path (open/stat/rename by path
|
||||
walk FAT directories linearly; consistent with the orphan-creep signal) — is
|
||||
one instrumentation pass for later (log each `p_mmap` miss + bench dir ops in
|
||||
`sd_bench`), not a prerequisite for the plumbing.
|
||||
|
||||
### The walk is ~1.4 s even at N ≈ 2
|
||||
|
||||
Mostly fixed cost — the worktree-diff setup and the second (`update_all`) pass —
|
||||
@@ -387,15 +455,21 @@ work, ranked:
|
||||
flat in repo size, small heap, images carried forward untouched. Replaces
|
||||
`stage_and_commit`'s `add_all`/`update_all`/`index.write`/`write_tree`. Needs the
|
||||
editor's dirty set (+ deleted set) plumbed to the git service — the editor
|
||||
already knows both. **Benched 2026-07-12: 6.5 s — the shape is right but it
|
||||
fails the sub-second bar**; wiring it in is blocked on localizing the ~1.5 s
|
||||
loose-object write (see the splice-bench section above).
|
||||
2. **Fix the `esp_map.c` cache so it can't OOM (do it alongside #1).** It grew to
|
||||
7.4 MB resident past its 4 MB cap and starved zlib. Evict on `p_munmap` (not
|
||||
only lazily on the next `p_mmap`) down to a low-water mark, and lower the cap, so
|
||||
a released window's memory is actually returned to `git__malloc`. The cache's
|
||||
`odb.write` win (862 → 142 ms) is real and worth keeping — this is a
|
||||
memory-discipline fix, not a removal.
|
||||
already knows both. **Benched to completion 2026-07-12: 6.5 s → 2.8 s cold /
|
||||
~2 s steady-state after the fast-seek fix (run 4). The sub-second bar failed
|
||||
but the block is lifted — wire it in** (see the final-bench section above:
|
||||
the residual is unique small SD round-trips, not something a cache or seek
|
||||
fix can remove).
|
||||
2. **Fix the `esp_map.c` cache so it can't OOM — RESOLVED BY REMOVAL (run 5).**
|
||||
The cache never scored a hit in four instrumented real-repo runs
|
||||
(`mwindow` absorbs true repetition above `p_mmap`; only new ranges reach
|
||||
the emulation), and the 7.4 MB OOM it was patched to avoid was caused by
|
||||
the cache itself holding buffers past `p_munmap`. esp_map.c is now the
|
||||
plain malloc-read/free-at-munmap emulation: honest with
|
||||
`MWINDOW_MAPPED_LIMIT` by construction, ~120 lines lighter, and run 5
|
||||
confirmed removal is I/O-neutral (even 15 reads *better* than v2, whose
|
||||
low-water eviction fought mwindow). Stats counters kept to spot any future
|
||||
workload that does repeat ranges.
|
||||
3. **Retired: `add_all`/explicit-path *index* staging.** Explicit-path `add_path`
|
||||
still goes through the index and `index.write` → `truncate_racily_clean`, so it
|
||||
hits Wall 1 just the same. The TreeBuilder walk supersedes it entirely; the
|
||||
@@ -408,10 +482,143 @@ work, ranked:
|
||||
window / 4 MB mapped limit). It fixed `odb.write` and the push read path; #2 just
|
||||
makes it well-behaved under memory pressure.
|
||||
|
||||
**Recommendation:** build #1 (O(depth) TreeBuilder walk) with #2 (cache
|
||||
evict-on-munmap) in the same pass. See
|
||||
[`../notes/sync-commit-handoff.md`](../notes/sync-commit-handoff.md) for the
|
||||
concrete next-session plan (bench design, firmware plumbing, exact call sites).
|
||||
**Recommendation:** build #1 (the O(depth) TreeBuilder walk) — #2 resolved
|
||||
itself by removal. The concrete plumbing plan (exact call sites, dirty-set
|
||||
threading, FD budget) is the
|
||||
[next section](#the-fix--wiring-the-odepth-splice-into-the-firmware).
|
||||
|
||||
## The fix — wiring the O(depth) splice into the firmware
|
||||
|
||||
> Merged from `docs/notes/sync-commit-handoff.md` (written 2026-07-12, retired
|
||||
> 2026-07-13 once the bench phase closed). The handoff's bench half is done —
|
||||
> the splice op lives in `git_bench` and the numbers above are its output.
|
||||
> **The firmware plumbing below SHIPPED 2026-07-13** (compile-verified both
|
||||
> feature flavors; on-device `:sync` against the real repo still pending) —
|
||||
> each item now records how it landed rather than what to do.
|
||||
|
||||
### The splice walk
|
||||
|
||||
Rebuild only the edited file's ancestor subtree chain onto HEAD's tree. Never
|
||||
materialise the 1179-entry index; never `index.write()`; never `read_tree` the
|
||||
whole tree. Cost is O(depth × dirty_files), flat in repo size, tiny heap, and it
|
||||
carries every unchanged entry (all 260 images, the other 1176 files) forward
|
||||
untouched — which means **the device doesn't even need the images in its
|
||||
working tree.**
|
||||
|
||||
Shipped as `git_sync::splice` (git2 0.20: `Repository::treebuilder(Option<&Tree>)`
|
||||
+ `TreeBuilder::{insert,remove,write}`). Signature:
|
||||
|
||||
```rust
|
||||
fn splice(repo: &Repository, base: Option<&Tree>, path: &[&str], blob: Option<Oid>) -> Result<Oid>
|
||||
```
|
||||
|
||||
`Some(blob)` inserts/replaces, `None` removes; `base: None` synthesizes a
|
||||
missing intermediate directory on the way down, and a directory emptied by a
|
||||
remove is pruned on the way up (the empty tree is never re-inserted).
|
||||
`stage_and_commit` folds every dirty path through it (threading the running
|
||||
root tree), then `commit(Some("HEAD"), …)` exactly as before — the
|
||||
`tree unchanged → nothing to publish` check and the `commit split —` timing
|
||||
log survive. The benched reference is `git_bench`'s `splice stage→tree` op.
|
||||
|
||||
### `firmware/src/git_sync.rs` — how it landed
|
||||
|
||||
- **mwindow options set at service start** (top of `run_git_service`, before
|
||||
any `Repository::open`): `set_mwindow_size(256 KB)` +
|
||||
`set_mwindow_mapped_limit(4 MB)`. Without them the 32-bit defaults (32 MB
|
||||
window / 256 MB mapped limit, mwindow.c:16) would `git__malloc` a 32 MB
|
||||
window on the first pack access and die on the 8 MB PSRAM heap. Run 5
|
||||
sharpened the stakes: the ~1.85 MB bench "resident" **is** mwindow's live
|
||||
open-window set, so these opts are the only thing bounding it.
|
||||
- `stage_and_commit(repo, paths)` is the splice walk; `add_all`/`update_all`/
|
||||
`index.write`/`write_tree` are gone. **The request carries one path set, not
|
||||
`{changed, deleted}`:** the working tree is the source of truth, so at commit
|
||||
time a recorded path that exists on the card is spliced in from disk and a
|
||||
missing one is spliced out. An unchanged path is a no-op — over-reporting is
|
||||
free, which is what makes the retry/journal semantics below simple.
|
||||
- **Stranded-commit recovery (new):** when the splice yields the parent's tree
|
||||
(nothing to commit), the service now compares HEAD against
|
||||
`refs/remotes/origin/<branch>` and still pushes if origin lacks HEAD — a
|
||||
previous cycle that committed and then failed mid-push used to strand that
|
||||
commit forever (the old path reported "up to date" and never retried).
|
||||
- **Radio-free up-to-date (new):** an empty dirty set + origin already at HEAD
|
||||
short-circuits before Wi-Fi even comes up — `:sync` with nothing to do
|
||||
answers in ~150 ms instead of a Wi-Fi/TLS round.
|
||||
- `reconcile_onto_origin` now `ResetType::Soft` (ref move only) — there is no
|
||||
index to reset, and a Mixed reset's index write is exactly the racy-clean
|
||||
wall the splice avoids. Side win: a remote-only added file is now *carried*
|
||||
by the replay (origin's tree is the splice base) where the old `add --all`
|
||||
replay dropped it.
|
||||
- The macOS-cruft filter (`skip_macos_cruft`) is gone with the walk — the
|
||||
splice only touches paths the editor recorded, so `._*`/`.DS_Store` can't
|
||||
sneak in the way they once did (07d87772, the Spike-14-era lesson).
|
||||
- **Deliberate behavior change (now in effect):** only paths the editor
|
||||
saved/deleted are ever committed. Files changed on the card outside the
|
||||
editor (card mounted on a Mac) were swept in by `add_all` before; they will
|
||||
now never be committed, and the working tree will show a permanent diff
|
||||
against HEAD if inspected on a desktop. Correct for the appliance; recorded
|
||||
here so it reads as intent, not accident.
|
||||
|
||||
### Dirty-set source — `firmware/src/persistence.rs` + `main.rs` (how it landed)
|
||||
|
||||
- `Storage` owns the record: `save_path`/`delete_path` note their repo-relative
|
||||
path in a `RefCell<Dirty>` (paths outside `/sd/repo` are skipped), and the
|
||||
set is **journaled to `/sd/.typoena-dirty`** — atomic write, rewritten only
|
||||
when the set actually grows. Without the journal a power pull would strand
|
||||
every file saved-but-not-published that session: nothing walks the tree
|
||||
anymore, so an unrecorded change would never reach the remote. The journal
|
||||
is loaded at mount and its paths ride the next `:sync`.
|
||||
- Lifecycle: `take_dirty()` snapshots pending → in-flight for one publish
|
||||
(journal keeps carrying the union); the outcome settles it —
|
||||
`publish_succeeded()` forgets the snapshot and shrinks the journal,
|
||||
`publish_failed()` returns it to pending for the next `:sync`. A save landing
|
||||
*while* a publish runs re-enters pending and rides the next one. Recording
|
||||
happens *before* the file write, so a crash between the two only
|
||||
over-approximates (a no-op splice), never under-records.
|
||||
- `Effect::Publish` in `main.rs` sends `PublishRequest { paths: take_dirty() }`;
|
||||
the outcome handler in the idle branch calls the matching settle method.
|
||||
- **FD budget:** a git build now mounts `Storage::mount_for_git()` (16 FDs) in
|
||||
`boot_storage` — libgit2 keeps the pack + `.idx` descriptors open and opens
|
||||
loose objects on top, which overruns the editor's 4-FD budget. The light
|
||||
build keeps the editor's own budget.
|
||||
|
||||
### `esp_map.c` — nothing left to do
|
||||
|
||||
The handoff's third work item (an evict-on-`p_munmap` cache fix) is superseded:
|
||||
run 5 removed the cache outright and `esp_map.c` is already the plain
|
||||
malloc-read/free-at-munmap emulation (see verdict item 2). One surviving
|
||||
operational note: editing `components/libgit2/*.c` won't rebuild via plain
|
||||
`cargo build` — first
|
||||
`rm -rf firmware/target/xtensa-esp32s3-espidf/release/.fingerprint/esp-idf-sys-*`
|
||||
(the `esp-idf-component-rebuild` lesson).
|
||||
|
||||
### How to bench / flash
|
||||
|
||||
`git_bench.rs` runs git ops on the 96 KB `GIT_STACK` thread (the main task stack
|
||||
overflows on these ops — that's why the real service has a dedicated thread). It's
|
||||
Rust-only, so a plain rebuild picks it up (no fingerprint dance unless you also
|
||||
touched `esp_map.c`).
|
||||
|
||||
```
|
||||
just flash-gitbench
|
||||
# = . ~/export-esp.sh && LIBGIT2_SRC=<repo>/firmware/components/libgit2/vendor \
|
||||
# LIBGIT2_NO_VENDOR=1 PKG_CONFIG_ALLOW_CROSS=1 \
|
||||
# PKG_CONFIG_LIBDIR=<repo>/firmware/pkgconfig \
|
||||
# cargo run --release --bin git_bench --features git
|
||||
```
|
||||
|
||||
Bench on the **real repo** clone (`/sd/repo` = full `jcalixte/notes`), not the
|
||||
toy — the toy pack understates everything by ~2 orders of magnitude.
|
||||
|
||||
### Still open (none block the plumbing)
|
||||
|
||||
- The residual ~360 ms/loose-write ≈ 8 unique small SD round-trips; next
|
||||
suspect is FAT **directory-op** cost in the freshen/refresh path. One
|
||||
instrumentation pass for later (log each `p_mmap` miss + bench dir ops in
|
||||
`sd_bench`).
|
||||
- Ref/reflog-update cost on the real repo — the bench's `commit(None)` writes
|
||||
no ref, so the shipping commit's last leg is unmeasured.
|
||||
- The push's ~6.5 s network floor
|
||||
([`../notes/sync-latency.md`](../notes/sync-latency.md)) — a separate curve.
|
||||
|
||||
## Adjacent lever: should the images be on the card at all?
|
||||
|
||||
|
||||
@@ -149,6 +149,24 @@ shown). On-device check: reflash → open a note (trailing empty line visible)
|
||||
newline for the same reason; the guarded save leaves the prefs file with exactly
|
||||
one.
|
||||
|
||||
**Amendment 2026-07-13 — recursive enumeration + a 2-char search threshold.**
|
||||
Loading a real repo (`jcalixte/notes`) exposed that `enumerate_files` listed
|
||||
only the **top-level** files of `/sd/repo` and `/sd/local` — a nested notes tree
|
||||
showed a single file in the palette (subpaths always *opened* fine via
|
||||
`:e repo/sub/x.md`; only the listing was flat). The enumeration is now a
|
||||
recursive walk: dot entries are skipped at every level (so `.git` is never
|
||||
descended into), each directory is read fully before recursing (one FatFS dir
|
||||
handle open at a time — the `remove_dir_recursive` pattern, kind to the
|
||||
FD-bounded mount), depth is capped at 8, and the boot-time walk logs its file
|
||||
count and duration (`file walk: N files in Xms`) so the FAT dir-IO cost on a
|
||||
big repo is measurable, not assumed. With the list now card-sized, the palette
|
||||
gained a **search threshold**: below 2 typed chars the result list is the
|
||||
**recents (MRU) only** — quick-switch (`Cmd-P`, `Enter`) stays one keystroke
|
||||
away — and the full fuzzy-ranked list appears from 2 chars on
|
||||
(`PALETTE_MIN_QUERY`). A fresh boot with no opens yet shows `(type to search)`.
|
||||
`>` commands and `$` snippets are short curated lists; the threshold does not
|
||||
apply to them.
|
||||
|
||||
- [x] `Cmd-P` opens fuzzy file palette over **both** `/sd/repo/` and
|
||||
`/sd/local/` — **landed and CONFIRMED ON DEVICE 2026-07-12** (Spike 11: no
|
||||
ghosting on the transient panel); scope shows as the inline
|
||||
|
||||
@@ -743,8 +743,9 @@ pub struct Editor {
|
||||
/// key batch. See [`Effect`].
|
||||
requests: Vec<Effect>,
|
||||
/// Every openable file, as absolute paths, fed by the host at boot via
|
||||
/// [`set_file_list`](Self::set_file_list) (an enumeration of `/sd/repo` and
|
||||
/// `/sd/local`). The palette fuzzy-filters this; empty until the host feeds it.
|
||||
/// [`set_file_list`](Self::set_file_list) (a recursive walk of `/sd/repo`
|
||||
/// and `/sd/local`). The palette fuzzy-filters this once the query reaches
|
||||
/// [`PALETTE_MIN_QUERY`] chars; empty until the host feeds it.
|
||||
files: Vec<String>,
|
||||
/// Recently-opened files, most-recent-first (an MRU), deduped and bounded to
|
||||
/// [`MRU_MAX`]. Every `:e`/palette open pushes to the front
|
||||
@@ -798,12 +799,20 @@ struct Buffer {
|
||||
/// evicted; it is saved first if dirty, so an evicted buffer is never lost.
|
||||
const MAX_RESIDENT: usize = 3;
|
||||
|
||||
/// Recent-files (MRU) list length — how many opens the palette remembers to
|
||||
/// float to the top on an empty query. Far more than [`MAX_RESIDENT`] (recency
|
||||
/// Recent-files (MRU) list length — how many opens the palette remembers; they
|
||||
/// are the whole result list below [`PALETTE_MIN_QUERY`] chars and float to the
|
||||
/// top above it. Far more than [`MAX_RESIDENT`] (recency
|
||||
/// outlives residency: a file evicted from memory is still recently *used*), but
|
||||
/// bounded so the list can't grow without limit over a long session.
|
||||
const MRU_MAX: usize = 16;
|
||||
|
||||
/// Query length (chars) at which the file palette searches the full file list.
|
||||
/// Shorter queries show only the recents ([`MRU_MAX`]) — the list is a
|
||||
/// recursive walk of the card, and one char can't rank hundreds of paths
|
||||
/// usefully. `>` commands and `$` snippets are short curated lists, so the
|
||||
/// threshold does not apply to them.
|
||||
const PALETTE_MIN_QUERY: usize = 2;
|
||||
|
||||
/// Maximum undo depth (change-groups). A full-buffer snapshot per group means
|
||||
/// worst-case memory is `UNDO_DEPTH × buffer size`; for note-sized files on the
|
||||
/// 8 MB PSRAM this is negligible, and prose editing rarely nears 100 groups
|
||||
@@ -1924,6 +1933,11 @@ impl Editor {
|
||||
/// Base order is MRU-first (recents in use order, then the rest as sorted). A
|
||||
/// non-empty query keeps only fuzzy matches and stable-sorts them by score, so
|
||||
/// equal scores keep their MRU/base position. See [`fuzzy_score`].
|
||||
///
|
||||
/// Below [`PALETTE_MIN_QUERY`] chars the candidate set is the recents only:
|
||||
/// the file list is a recursive walk of the whole card, too long to page
|
||||
/// through unranked, but the MRU keeps quick-switch (`Cmd-P`, `Enter`) one
|
||||
/// keystroke away. Two typed chars reveal the full list.
|
||||
fn palette_matches(&self) -> Vec<usize> {
|
||||
let mut order: Vec<usize> = Vec::with_capacity(self.files.len());
|
||||
for r in &self.recent {
|
||||
@@ -1931,9 +1945,11 @@ impl Editor {
|
||||
order.push(i);
|
||||
}
|
||||
}
|
||||
for i in 0..self.files.len() {
|
||||
if !order.contains(&i) {
|
||||
order.push(i);
|
||||
if self.palette_query.chars().count() >= PALETTE_MIN_QUERY {
|
||||
for i in 0..self.files.len() {
|
||||
if !order.contains(&i) {
|
||||
order.push(i);
|
||||
}
|
||||
}
|
||||
}
|
||||
if self.palette_query.is_empty() {
|
||||
@@ -3562,6 +3578,10 @@ impl Editor {
|
||||
"(no command)"
|
||||
} else if self.files.is_empty() {
|
||||
"(no files on card)"
|
||||
} else if self.palette_query.chars().count() < PALETTE_MIN_QUERY {
|
||||
// No recents yet and the query is below the search threshold —
|
||||
// the full list needs 2+ chars.
|
||||
"(type to search)"
|
||||
} else {
|
||||
"(no match)"
|
||||
};
|
||||
@@ -5124,6 +5144,9 @@ mod tests {
|
||||
e.take_effects();
|
||||
assert!(!e.files.contains(&"/sd/repo/notes.md".to_string()));
|
||||
e.handle(Key::Palette);
|
||||
for c in "md".chars() {
|
||||
e.handle(Key::Char(c)); // reach the search threshold
|
||||
}
|
||||
assert_eq!(palette_labels(&e), vec!["repo/todo.md"]); // only the survivor
|
||||
}
|
||||
|
||||
@@ -5272,6 +5295,9 @@ mod tests {
|
||||
fn half_page_keys_move_the_selection_clamped() {
|
||||
let mut e = palette_editor(&["/sd/repo/a.md", "/sd/repo/b.md", "/sd/repo/c.md"]);
|
||||
e.handle(Key::Palette);
|
||||
for ch in "md".chars() {
|
||||
e.handle(Key::Char(ch)); // reach the search threshold: all three match
|
||||
}
|
||||
assert_eq!(e.palette_sel, 0);
|
||||
e.handle(Key::HalfPageDown);
|
||||
assert_eq!(e.palette_sel, 1);
|
||||
@@ -5286,6 +5312,9 @@ mod tests {
|
||||
fn ctrl_n_p_navigate_the_palette() {
|
||||
let mut e = palette_editor(&["/sd/repo/a.md", "/sd/repo/b.md", "/sd/repo/c.md"]);
|
||||
e.handle(Key::Palette);
|
||||
for ch in "md".chars() {
|
||||
e.handle(Key::Char(ch)); // reach the search threshold: all three match
|
||||
}
|
||||
e.handle(Key::Down); // Ctrl-n
|
||||
assert_eq!(e.palette_sel, 1);
|
||||
e.handle(Key::Down);
|
||||
@@ -5337,6 +5366,9 @@ mod tests {
|
||||
fn editing_the_query_resets_the_selection_to_the_top() {
|
||||
let mut e = palette_editor(&["/sd/repo/a.md", "/sd/repo/b.md"]);
|
||||
e.handle(Key::Palette);
|
||||
for ch in "md".chars() {
|
||||
e.handle(Key::Char(ch)); // reach the search threshold: both match
|
||||
}
|
||||
e.handle(Key::HalfPageDown);
|
||||
assert_eq!(e.palette_sel, 1);
|
||||
e.handle(Key::Char('a')); // a query edit resets the selection
|
||||
@@ -5352,17 +5384,46 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_query_orders_recents_first_then_sorted() {
|
||||
fn short_query_lists_recents_only() {
|
||||
let mut e = palette_editor(&["/sd/repo/b.md", "/sd/repo/a.md", "/sd/repo/c.md"]);
|
||||
// No opens yet: pure sorted order.
|
||||
assert_eq!(palette_labels(&e), vec!["repo/a.md", "repo/b.md", "repo/c.md"]);
|
||||
// Open c.md through the palette; it should float to the front next time.
|
||||
// No opens yet: below the search threshold there is nothing to show.
|
||||
assert!(palette_labels(&e).is_empty());
|
||||
// Open c.md through the palette; it becomes the recents-only result.
|
||||
e.handle(Key::Palette);
|
||||
for ch in "c.md".chars() {
|
||||
e.handle(Key::Char(ch));
|
||||
}
|
||||
e.handle(Key::Enter);
|
||||
e.take_effects(); // drop the queued Load; we only care about the MRU
|
||||
assert_eq!(palette_labels(&e), vec!["repo/c.md"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn two_char_query_reveals_the_full_file_list() {
|
||||
let mut e = palette_editor(&["/sd/repo/b.md", "/sd/repo/a.md", "/sd/repo/c.md"]);
|
||||
e.handle(Key::Palette);
|
||||
e.handle(Key::Char('m')); // one char: still recents-only (none yet)
|
||||
assert!(e.palette_matches().is_empty());
|
||||
e.handle(Key::Char('d')); // "md": the full list, fuzzy-ranked
|
||||
assert_eq!(palette_labels(&e), vec!["repo/a.md", "repo/b.md", "repo/c.md"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recents_float_above_the_full_list_on_a_matching_query() {
|
||||
let mut e = palette_editor(&["/sd/repo/b.md", "/sd/repo/a.md", "/sd/repo/c.md"]);
|
||||
// Open c.md so it is the MRU head.
|
||||
e.handle(Key::Palette);
|
||||
for ch in "c.md".chars() {
|
||||
e.handle(Key::Char(ch));
|
||||
}
|
||||
e.handle(Key::Enter);
|
||||
e.take_effects();
|
||||
// "md" scores the three labels equally; the stable sort keeps the
|
||||
// recently-opened c.md in front of the sorted rest.
|
||||
e.handle(Key::Palette);
|
||||
for ch in "md".chars() {
|
||||
e.handle(Key::Char(ch));
|
||||
}
|
||||
assert_eq!(palette_labels(&e), vec!["repo/c.md", "repo/a.md", "repo/b.md"]);
|
||||
}
|
||||
|
||||
@@ -5370,6 +5431,10 @@ mod tests {
|
||||
fn draw_in_palette_mode_does_not_panic() {
|
||||
let mut e = palette_editor(&["/sd/repo/a.md", "/sd/local/j.md"]);
|
||||
e.handle(Key::Palette);
|
||||
let _ = e.draw(true); // empty query, no recents: "(type to search)"
|
||||
e.handle(Key::Char('j')); // one char, still below the threshold
|
||||
let _ = e.draw(true);
|
||||
e.handle(Key::Char('m')); // at the threshold: the ranked list
|
||||
let _ = e.draw(true);
|
||||
// Empty file list: the "(no files on card)" path must also be safe.
|
||||
let mut empty = Editor::new();
|
||||
|
||||
@@ -7,40 +7,19 @@
|
||||
* Allocations go through git__malloc, so with PSRAM in the heap they land in
|
||||
* the 8 MB external RAM rather than the ~340 KB internal DRAM.
|
||||
*
|
||||
* CACHE (2026-07-12): the emulation reads the range from SD on every call, so
|
||||
* libgit2's repeated pack access — `git_odb_write` → `git_odb__freshen` →
|
||||
* `git_odb_refresh` re-reads the pack idx/windows on *every* object write —
|
||||
* turned a 45 ms object write into 500 ms–1.1 s on a small repo, and would be far
|
||||
* worse on the real 570 MB-pack clone (see
|
||||
* docs/tradeoff-curves/sync-commit-staging.md). We cache the read buffers so a
|
||||
* given file region is read from the card once and reused.
|
||||
* There is deliberately NO cache here. A window cache was built and removed
|
||||
* on 2026-07-12: across four instrumented real-repo bench runs it scored
|
||||
* exactly 0 hits, because libgit2's mwindow layer reuses its open windows and
|
||||
* only genuinely new (offset, len) ranges ever reach p_mmap — and the one
|
||||
* memory bug it "fixed" (7.4 MB resident starving zlib) was caused by the
|
||||
* cache itself holding buffers past p_munmap. Free-at-munmap keeps
|
||||
* GIT_OPT_SET_MWINDOW_MAPPED_LIMIT honest by construction: when libgit2
|
||||
* releases a window, the memory really is back in git__malloc's pool. Full
|
||||
* trail: docs/tradeoff-curves/sync-commit-staging.md (final-bench section).
|
||||
* The stats counters below are kept so a future workload that *does* repeat
|
||||
* ranges (push? reconcile?) can be spotted before anyone rebuilds a cache.
|
||||
*
|
||||
* Correctness: cache ONLY read-only mappings of files >= ESP_MAP_CACHE_MIN_FILE
|
||||
* (the FILE size, not the map length — 2026-07-12b). The hot set turned out to
|
||||
* be the SMALL maps: pack trailer probes, idx fanout reads and delta-base
|
||||
* windows repeat at the same offsets on every freshen/refresh, and a map-length
|
||||
* floor excluded exactly those (0 cache hits over three full real-repo bench
|
||||
* runs). Keying on file size caches them while still excluding the small
|
||||
* mutable working-tree files diff_file.c maps (notes.md etc.) — only the pack,
|
||||
* its idx, a midx and the commit-graph are ever that large, and all are
|
||||
* immutable on this device (only loose objects/refs/index are written, none
|
||||
* via mmap). The writable mapping the pack indexer uses (fetch/clone) is
|
||||
* excluded by the prot check. Identity is (dev, ino, size, mtime, offset, len);
|
||||
* for an immutable pack any of size/mtime already differs if the file is ever
|
||||
* replaced. NOTE: on esp-idf's FAT VFS st_dev/st_ino are constant and mtime has
|
||||
* 2 s granularity, so identity is effectively (size, mtime₂ₛ, offset, len) —
|
||||
* fine for packs (a replaced pack changes size), weak for small mutable files,
|
||||
* which is exactly why those must never be cacheable.
|
||||
*
|
||||
* Memory discipline (2026-07-12b): the first version freed released buffers
|
||||
* only lazily on the next p_mmap, so a burst (read_tree of 158 trees) pinned
|
||||
* 7.4 MB and starved zlib's git__malloc (crash at 508 KB heap) — it defeated
|
||||
* GIT_OPT_SET_MWINDOW_MAPPED_LIMIT because libgit2 believed the memory was
|
||||
* returned. Now p_munmap evicts unreferenced entries down to
|
||||
* ESP_MAP_CACHE_LOW_WATER, so a released window's memory really is returned
|
||||
* under pressure while a warm working set stays resident.
|
||||
*
|
||||
* Limitation: writable/shared mappings are not written back (and not cached).
|
||||
* Limitation: writable/shared mappings are not written back.
|
||||
*/
|
||||
|
||||
#include "git2_util.h"
|
||||
@@ -49,7 +28,6 @@
|
||||
#include <unistd.h>
|
||||
#include <string.h>
|
||||
#include <errno.h>
|
||||
#include <sys/stat.h>
|
||||
#include <stdint.h>
|
||||
|
||||
int git__page_size(size_t *page_size)
|
||||
@@ -64,48 +42,19 @@ int git__mmap_alignment(size_t *alignment)
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Only cache mappings of files at least this large (FILE size, not map length):
|
||||
* covers the pack, its idx, midx and commit-graph — and thus the hot small
|
||||
* windows within them — while excluding the small mutable working-tree files
|
||||
* diff_file.c maps. */
|
||||
#define ESP_MAP_CACHE_MIN_FILE (1024 * 1024)
|
||||
/* Cached-buffer budget in PSRAM. Entries with no live ref are LRU-evicted to
|
||||
* stay under this; pinned entries may exceed it transiently. */
|
||||
#define ESP_MAP_CACHE_CAP (4 * 1024 * 1024)
|
||||
/* On p_munmap, unreferenced entries are evicted down to this — the resident
|
||||
* working set a burst leaves behind. Must leave git__malloc (zlib, mwindow)
|
||||
* ample heap; the 2026-07-12 OOM run crashed with 7.4 MB resident. */
|
||||
#define ESP_MAP_CACHE_LOW_WATER (2 * 1024 * 1024)
|
||||
/* Small maps are numerous (trailer/fanout/delta-base probes), so more slots
|
||||
* than the window-only design needed; a slot is ~40 B. */
|
||||
#define ESP_MAP_CACHE_SLOTS 48
|
||||
|
||||
struct map_entry {
|
||||
unsigned char *data;
|
||||
size_t len;
|
||||
off64_t offset;
|
||||
dev_t dev;
|
||||
ino_t ino;
|
||||
off_t size;
|
||||
time_t mtime;
|
||||
int refcount;
|
||||
uint32_t used; /* LRU stamp; 0 = empty slot */
|
||||
};
|
||||
|
||||
static struct map_entry g_cache[ESP_MAP_CACHE_SLOTS];
|
||||
static uint32_t g_clock;
|
||||
static size_t g_cached_bytes;
|
||||
|
||||
/* Diagnostics, read from the bench via esp_map_stats(). */
|
||||
static uint32_t g_hits, g_misses;
|
||||
/* Diagnostics, read from the bench via esp_map_stats(). The signature predates
|
||||
* the cache removal: `hits` is always 0, `misses` counts every mapping, and
|
||||
* `cached_kb` now reports the LIVE mapped bytes (the mwindow working set). */
|
||||
static uint32_t g_maps;
|
||||
static uint64_t g_read_bytes;
|
||||
static size_t g_live_bytes;
|
||||
|
||||
void esp_map_stats(uint32_t *hits, uint32_t *misses, uint32_t *read_kb, uint32_t *cached_kb)
|
||||
{
|
||||
if (hits) *hits = g_hits;
|
||||
if (misses) *misses = g_misses;
|
||||
if (hits) *hits = 0;
|
||||
if (misses) *misses = g_maps;
|
||||
if (read_kb) *read_kb = (uint32_t)(g_read_bytes / 1024);
|
||||
if (cached_kb) *cached_kb = (uint32_t)(g_cached_bytes / 1024);
|
||||
if (cached_kb) *cached_kb = (uint32_t)(g_live_bytes / 1024);
|
||||
}
|
||||
|
||||
static int read_range(int fd, off64_t offset, size_t len, unsigned char *data)
|
||||
@@ -133,97 +82,26 @@ static int read_range(int fd, off64_t offset, size_t len, unsigned char *data)
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Best-effort: evict unreferenced LRU entries until total cached bytes fit
|
||||
* `budget`. Pinned entries (refcount > 0) can't be evicted, so budgets are
|
||||
* soft. */
|
||||
static void evict_to(size_t budget)
|
||||
{
|
||||
while (g_cached_bytes > budget) {
|
||||
int victim = -1;
|
||||
uint32_t oldest = 0xFFFFFFFFu;
|
||||
int i;
|
||||
|
||||
for (i = 0; i < ESP_MAP_CACHE_SLOTS; i++) {
|
||||
if (g_cache[i].used && g_cache[i].refcount == 0 &&
|
||||
g_cache[i].used < oldest) {
|
||||
oldest = g_cache[i].used;
|
||||
victim = i;
|
||||
}
|
||||
}
|
||||
if (victim < 0)
|
||||
break;
|
||||
git__free(g_cache[victim].data);
|
||||
g_cached_bytes -= g_cache[victim].len;
|
||||
memset(&g_cache[victim], 0, sizeof(g_cache[victim]));
|
||||
}
|
||||
}
|
||||
|
||||
int p_mmap(git_map *out, size_t len, int prot, int flags, int fd, off64_t offset)
|
||||
{
|
||||
unsigned char *data;
|
||||
struct stat st;
|
||||
int cacheable, i;
|
||||
|
||||
GIT_UNUSED(prot);
|
||||
GIT_UNUSED(flags);
|
||||
GIT_MMAP_VALIDATE(out, len, prot, flags);
|
||||
|
||||
out->data = NULL;
|
||||
out->len = 0;
|
||||
|
||||
/* Cache only read-only mappings of large files (pack/idx/...) that we can
|
||||
* identify. The map itself may be small — small repeated maps ARE the hot
|
||||
* set (see header comment). */
|
||||
cacheable = !(prot & GIT_PROT_WRITE);
|
||||
if (cacheable && (fstat(fd, &st) != 0 || st.st_size < ESP_MAP_CACHE_MIN_FILE))
|
||||
cacheable = 0;
|
||||
|
||||
if (cacheable) {
|
||||
for (i = 0; i < ESP_MAP_CACHE_SLOTS; i++) {
|
||||
struct map_entry *e = &g_cache[i];
|
||||
if (e->used && e->len == len && e->offset == offset &&
|
||||
e->dev == st.st_dev && e->ino == st.st_ino &&
|
||||
e->size == st.st_size && e->mtime == st.st_mtime) {
|
||||
e->refcount++;
|
||||
e->used = ++g_clock;
|
||||
g_hits++;
|
||||
out->data = e->data;
|
||||
out->len = len;
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
data = git__malloc(len);
|
||||
GIT_ERROR_CHECK_ALLOC(data);
|
||||
if (read_range(fd, offset, len, data) < 0) {
|
||||
git__free(data);
|
||||
return -1;
|
||||
}
|
||||
g_misses++;
|
||||
g_maps++;
|
||||
g_read_bytes += len;
|
||||
|
||||
if (cacheable) {
|
||||
if (len < ESP_MAP_CACHE_CAP)
|
||||
evict_to(ESP_MAP_CACHE_CAP - len);
|
||||
for (i = 0; i < ESP_MAP_CACHE_SLOTS; i++) {
|
||||
if (!g_cache[i].used) {
|
||||
g_cache[i].data = data;
|
||||
g_cache[i].len = len;
|
||||
g_cache[i].offset = offset;
|
||||
g_cache[i].dev = st.st_dev;
|
||||
g_cache[i].ino = st.st_ino;
|
||||
g_cache[i].size = st.st_size;
|
||||
g_cache[i].mtime = st.st_mtime;
|
||||
g_cache[i].refcount = 1;
|
||||
g_cache[i].used = ++g_clock;
|
||||
g_cached_bytes += len;
|
||||
out->data = data;
|
||||
out->len = len;
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
/* All slots pinned: return uncached (freed directly at munmap). */
|
||||
}
|
||||
g_live_bytes += len;
|
||||
|
||||
out->data = data;
|
||||
out->len = len;
|
||||
@@ -232,26 +110,13 @@ int p_mmap(git_map *out, size_t len, int prot, int flags, int fd, off64_t offset
|
||||
|
||||
int p_munmap(git_map *map)
|
||||
{
|
||||
int i;
|
||||
|
||||
GIT_ASSERT_ARG(map);
|
||||
|
||||
/* Cached buffer: drop a ref and keep it for reuse — but return memory to
|
||||
* git__malloc under pressure (down to the low-water mark), so a released
|
||||
* window is really released and MWINDOW_MAPPED_LIMIT stays honest. */
|
||||
for (i = 0; i < ESP_MAP_CACHE_SLOTS; i++) {
|
||||
if (g_cache[i].used && g_cache[i].data == map->data) {
|
||||
if (g_cache[i].refcount > 0)
|
||||
g_cache[i].refcount--;
|
||||
if (g_cache[i].refcount == 0 &&
|
||||
g_cached_bytes > ESP_MAP_CACHE_LOW_WATER)
|
||||
evict_to(ESP_MAP_CACHE_LOW_WATER);
|
||||
map->data = NULL;
|
||||
map->len = 0;
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
git__free(map->data);
|
||||
if (g_live_bytes >= map->len)
|
||||
g_live_bytes -= map->len;
|
||||
else
|
||||
g_live_bytes = 0;
|
||||
map->data = NULL;
|
||||
map->len = 0;
|
||||
return 0;
|
||||
|
||||
@@ -270,7 +270,10 @@ _card sd_volume="":
|
||||
# .gitignore ignores is excluded (node_modules is 3.9 GB, gitignored, never in
|
||||
# .git) — the device needs .git + the checkout (~720 MB), not the JS deps or
|
||||
# secrets like firmware/.env. Copying (not a fresh `git clone` of the local
|
||||
# path) preserves origin → GitHub so on-device fetch/push still work.
|
||||
# path) preserves origin → GitHub so on-device fetch/push still work — except
|
||||
# the URL scheme: the card copy's origin is rewritten to HTTPS at the end,
|
||||
# because the device's libgit2 has no SSH transport (HTTPS+PAT over mbedTLS
|
||||
# only). The source clone is never touched.
|
||||
_load-repo repo_src vol:
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
@@ -332,6 +335,40 @@ _load-repo repo_src vol:
|
||||
# -P for progress on the ~700 MB copy. Scoped to repo/, never the card root.
|
||||
rsync -rtP --delete --modify-window=1 --exclude-from="$ignore_list" "$src/" "$dest/"
|
||||
|
||||
# The device's libgit2 speaks HTTPS+PAT only (mbedTLS — no SSH transport),
|
||||
# so an SSH origin copied verbatim fails every on-device push/fetch with
|
||||
# "unsupported URL protocol" (found the hard way, 2026-07-13). Rewrite the
|
||||
# CARD copy's origin to the HTTPS equivalent; the source clone keeps its
|
||||
# own URL. Its embedded trust store carries GitHub's roots, hence the
|
||||
# non-github warning.
|
||||
case "$origin" in
|
||||
git@*:*)
|
||||
host="${origin#git@}"; host="${host%%:*}"
|
||||
card_origin="https://$host/${origin#git@*:}"
|
||||
;;
|
||||
ssh://git@*)
|
||||
rest="${origin#ssh://git@}"
|
||||
card_origin="https://${rest%%[:/]*}/${rest#*/}"
|
||||
;;
|
||||
https://*) card_origin="$origin" ;;
|
||||
*) card_origin="" ;;
|
||||
esac
|
||||
if [ -n "$card_origin" ]; then
|
||||
if [ "$card_origin" != "$origin" ]; then
|
||||
echo "rewriting card origin for the device: $origin -> $card_origin"
|
||||
fi
|
||||
git -C "$dest" remote set-url origin "$card_origin"
|
||||
case "$card_origin" in
|
||||
https://github.com/*) ;;
|
||||
*) echo "warning: origin host isn't github.com — the device's embedded trust store" \
|
||||
"only carries GitHub roots, so on-device TLS will fail without its CA" ;;
|
||||
esac
|
||||
elif [ -n "$origin" ]; then
|
||||
echo "warning: origin '$origin' has no HTTPS equivalent I can derive —"
|
||||
echo " on-device push/fetch will fail; set it by hand:"
|
||||
echo " git -C '$dest' remote set-url origin https://github.com/<owner>/<repo>.git"
|
||||
fi
|
||||
|
||||
# First-time setup: seed the two git-tracked config files into the card's repo/
|
||||
# so they commit + sync on the device's first `:sync`. Writes ONLY a file that is
|
||||
# absent — a card whose clone already carries `.typoena.toml` /
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
//!
|
||||
//! HEADLINE OP (since the 2026-07-12 real-repo run): `splice stage→tree` — the
|
||||
//! O(depth) TreeBuilder walk that replaces the index-based commit entirely
|
||||
//! (docs/notes/sync-commit-handoff.md). It runs FIRST so its first iteration is
|
||||
//! (docs/tradeoff-curves/sync-commit-staging.md). It runs FIRST so its first iteration is
|
||||
//! the cold number; acceptance bar: **sub-second cold on the real 570 MB-pack
|
||||
//! clone, heap staying healthy**. The index paths it supersedes run LAST, for
|
||||
//! regression tracking — they previously OOM'd, and a late crash can't cost the
|
||||
@@ -81,7 +81,7 @@ fn run() -> Result<()> {
|
||||
log_map_stats("open");
|
||||
|
||||
// 1) THE FIX — `splice stage→tree`, the O(depth) TreeBuilder walk
|
||||
// (docs/notes/sync-commit-handoff.md): patch the edited file's ancestor
|
||||
// (docs/tradeoff-curves/sync-commit-staging.md): patch the edited file's ancestor
|
||||
// subtree chain onto HEAD's tree; never materialise the 1179-entry index,
|
||||
// never index.write(), never read_tree the whole tree. Runs FIRST so
|
||||
// iteration #1 is genuinely cold (only `open` has touched the pack).
|
||||
@@ -293,22 +293,25 @@ fn find_edit_path(repo: &Repository, root: &Tree) -> Result<Vec<String>> {
|
||||
}
|
||||
|
||||
unsafe extern "C" {
|
||||
/// Counters from the p_mmap cache in `components/libgit2/esp_map.c`.
|
||||
/// Counters from the p_mmap emulation in `components/libgit2/esp_map.c`.
|
||||
/// Post cache-removal: `hits` is always 0, `misses` counts every mapping,
|
||||
/// `cached_kb` reports the LIVE mapped bytes (the mwindow working set).
|
||||
fn esp_map_stats(hits: *mut u32, misses: *mut u32, read_kb: *mut u32, cached_kb: *mut u32);
|
||||
}
|
||||
|
||||
/// Log the p_mmap cache counters — hits vs misses (SD reads avoided), total KB
|
||||
/// read from the card, and KB currently resident. If the pack-read hypothesis is
|
||||
/// right, hits climb and `KB read` stops growing across the write ops.
|
||||
/// Log the p_mmap counters — mappings performed, total KB read from the card,
|
||||
/// and KB currently live-mapped (should track mwindow's open windows and stay
|
||||
/// well under MWINDOW_MAPPED_LIMIT now that munmap frees immediately).
|
||||
fn log_map_stats(label: &str) {
|
||||
let (mut hits, mut misses, mut read_kb, mut cached_kb) = (0u32, 0u32, 0u32, 0u32);
|
||||
unsafe { esp_map_stats(&mut hits, &mut misses, &mut read_kb, &mut cached_kb) };
|
||||
let _ = hits; // always 0 since the cache removal; slot kept for ABI stability
|
||||
// Free heap spans PSRAM here; a drop toward 0 during write_tree/commit on the
|
||||
// real repo would point at mwindow/idx allocation pressure (or thrash) as the
|
||||
// cause of an apparent hang, not CPU.
|
||||
let free_kb = unsafe { esp_idf_svc::sys::esp_get_free_heap_size() } / 1024;
|
||||
log::info!(
|
||||
"mmap cache @ {label:<11} {hits} hit / {misses} miss, {read_kb} KB read, {cached_kb} KB resident, {free_kb} KB heap free"
|
||||
"mmap @ {label:<11} {misses} maps, {read_kb} KB read, {cached_kb} KB live, {free_kb} KB heap free"
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -17,15 +17,23 @@
|
||||
//! notes. A `/sd/repo` that isn't a valid repo is a provisioning error
|
||||
//! (`just init`), surfaced as such, not papered over.
|
||||
//! 3. **No synthetic content.** The spike appended a marker line; here the
|
||||
//! editor has already saved the user's `notes.md` before `:sync` signals us,
|
||||
//! so we just stage + commit + push what's on disk.
|
||||
//! editor has already saved the user's buffers before `:sync` signals us,
|
||||
//! so we just commit + push what's on disk.
|
||||
//! 4. **The commit is an O(depth) TreeBuilder splice, not an index pass.**
|
||||
//! The request carries the repo-relative paths saved/deleted since the last
|
||||
//! confirmed publish (`Storage`'s journaled dirty set); `stage_and_commit`
|
||||
//! patches exactly those onto HEAD's tree. The index pipeline it replaced
|
||||
//! (`add_all` → `index.write` → `write_tree`) is O(N_tree) and measured up
|
||||
//! to 611 s on the real 1179-file / 570 MB-pack clone — see
|
||||
//! docs/tradeoff-curves/sync-commit-staging.md for the whole trail.
|
||||
//!
|
||||
//! Runs on a dedicated 96 KB thread (libgit2's init→push chain nests ~67 KB of
|
||||
//! `GIT_PATH_MAX` stack buffers — see git_push.rs / postmortem #3). Config is
|
||||
//! baked at build time (`TW_*`, ADR-007: v0.1 device config is compiled in).
|
||||
|
||||
use std::cell::RefCell;
|
||||
use std::path::Path;
|
||||
use std::collections::BTreeSet;
|
||||
use std::fs;
|
||||
use std::rc::Rc;
|
||||
use std::sync::mpsc::{Receiver, Sender};
|
||||
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
|
||||
@@ -39,8 +47,8 @@ use esp_idf_svc::sntp::{EspSntp, SyncStatus};
|
||||
use esp_idf_svc::sys;
|
||||
use esp_idf_svc::wifi::{BlockingWifi, EspWifi};
|
||||
use git2::{
|
||||
CertificateCheckStatus, Commit, Cred, CredentialType, FetchOptions, IndexAddOption,
|
||||
PushOptions, RemoteCallbacks, Repository, Signature,
|
||||
CertificateCheckStatus, Commit, Cred, CredentialType, FetchOptions, ObjectType, Oid,
|
||||
PushOptions, RemoteCallbacks, Repository, Signature, Tree,
|
||||
};
|
||||
|
||||
use crate::net::connect_wifi;
|
||||
@@ -73,10 +81,15 @@ const SNTP_TIMEOUT: Duration = Duration::from_secs(20);
|
||||
/// now also runs here, but it's shallow next to libgit2's path-buffer nesting.
|
||||
pub const GIT_STACK: usize = 96 * 1024;
|
||||
|
||||
/// A request to publish. The note is already saved to `/sd/repo/notes.md` by the
|
||||
/// UI task before this is sent, so the request carries no payload (a future
|
||||
/// multi-file publish can grow one).
|
||||
pub struct PublishRequest;
|
||||
/// A request to publish. The UI task has already saved every dirty buffer to
|
||||
/// the card before sending this; `paths` is `Storage::take_dirty`'s snapshot —
|
||||
/// the repo-relative paths saved or `:delete`d since the last confirmed
|
||||
/// publish. The working tree stays the source of truth: at commit time a path
|
||||
/// that exists on the card is spliced into the tree from disk, a missing one
|
||||
/// is spliced out. An unchanged path is a no-op, so over-reporting is safe.
|
||||
pub struct PublishRequest {
|
||||
pub paths: BTreeSet<String>,
|
||||
}
|
||||
|
||||
/// Result of a publish attempt, sent back to the UI task for the snackbar. The
|
||||
/// detailed error always goes to the serial log; the panel gets a short line.
|
||||
@@ -103,6 +116,22 @@ pub fn run_git_service(
|
||||
rx: Receiver<PublishRequest>,
|
||||
tx: Sender<PublishOutcome>,
|
||||
) {
|
||||
// Process-global libgit2 tuning, once, before any repo work. The 32-bit
|
||||
// defaults (32 MB window / 256 MB mapped budget, mwindow.c) would
|
||||
// git__malloc past PSRAM on the first pack access of the real 570 MB-pack
|
||||
// clone; these are the bench-proven values (git_bench), and ~1.9 MB of
|
||||
// windows stays live during git ops — the p_mmap emulation (esp_map.c)
|
||||
// relies on this mapped limit being real.
|
||||
// SAFETY: set on the git thread before any Repository is opened.
|
||||
unsafe {
|
||||
if let Err(e) = git2::opts::set_mwindow_size(256 * 1024) {
|
||||
log::error!("set_mwindow_size failed ({e}); first pack access may OOM");
|
||||
}
|
||||
if let Err(e) = git2::opts::set_mwindow_mapped_limit(4 * 1024 * 1024) {
|
||||
log::error!("set_mwindow_mapped_limit failed ({e}); first pack access may OOM");
|
||||
}
|
||||
}
|
||||
|
||||
// Lazily initialised on the first request, then reused across publishes.
|
||||
let mut wifi: Option<BlockingWifi<EspWifi<'static>>> = None;
|
||||
let mut modem = Some(modem);
|
||||
@@ -110,7 +139,7 @@ pub fn run_git_service(
|
||||
let mut clock_synced = false;
|
||||
let mut tls_ready = false;
|
||||
|
||||
while rx.recv().is_ok() {
|
||||
while let Ok(req) = rx.recv() {
|
||||
let outcome = publish_cycle(
|
||||
&sys_loop,
|
||||
&mut wifi,
|
||||
@@ -118,6 +147,7 @@ pub fn run_git_service(
|
||||
&mut nvs,
|
||||
&mut clock_synced,
|
||||
&mut tls_ready,
|
||||
&req.paths,
|
||||
);
|
||||
let msg = match outcome {
|
||||
Ok(o) => o,
|
||||
@@ -143,11 +173,22 @@ fn publish_cycle(
|
||||
nvs: &mut Option<EspDefaultNvsPartition>,
|
||||
clock_synced: &mut bool,
|
||||
tls_ready: &mut bool,
|
||||
paths: &BTreeSet<String>,
|
||||
) -> Result<PublishOutcome> {
|
||||
if REMOTE_URL.is_empty() || GH_USER.is_empty() || PAT.is_empty() || WIFI_SSID.is_empty() {
|
||||
bail!("git config missing — set TW_WIFI_SSID / TW_REMOTE_URL / TW_GH_USER / TW_PAT in firmware/.env and rebuild");
|
||||
}
|
||||
|
||||
// Nothing recorded dirty and origin's tracking ref already has HEAD: this
|
||||
// `:sync` has nothing to do — say so without touching the radio (~150 ms
|
||||
// instead of a Wi-Fi + TLS round). A stranded local commit (committed but
|
||||
// never pushed, e.g. a push that failed mid-air) makes the check false and
|
||||
// takes the full path below, where publish_once pushes it.
|
||||
if paths.is_empty() && remote_current().unwrap_or(false) {
|
||||
log::info!(":sync — no dirty paths and origin has HEAD; up to date, radio untouched");
|
||||
return Ok(PublishOutcome::UpToDate);
|
||||
}
|
||||
|
||||
// Phases are timed so a cold :sync reports where the seconds go. Wi-Fi, clock
|
||||
// and TLS run only on the first sync of a session; a warm sync skips them, so
|
||||
// they read 0 ms and the total collapses to just publish(fetch+commit+push).
|
||||
@@ -186,7 +227,7 @@ fn publish_cycle(
|
||||
}
|
||||
|
||||
let t_publish = Instant::now();
|
||||
let outcome = publish_once()?;
|
||||
let outcome = publish_once(paths)?;
|
||||
log::info!(
|
||||
":sync timing — wifi {wifi_ms}ms, clock {clock_ms}ms, tls {tls_ms}ms, publish(commit+push) {}ms, total {}ms",
|
||||
t_publish.elapsed().as_millis(),
|
||||
@@ -205,15 +246,16 @@ fn publish_cycle(
|
||||
///
|
||||
/// Never clones or wipes: a `/sd/repo` that isn't a valid repo is a provisioning
|
||||
/// error, surfaced as such.
|
||||
fn publish_once() -> Result<PublishOutcome> {
|
||||
log::info!("publish started — free heap {}", free_heap());
|
||||
fn publish_once(paths: &BTreeSet<String>) -> Result<PublishOutcome> {
|
||||
log::info!(
|
||||
"publish started — {} dirty path(s), free heap {}",
|
||||
paths.len(),
|
||||
free_heap()
|
||||
);
|
||||
let repo = Repository::open(REPO_DIR).with_context(|| {
|
||||
format!("opening git repo at {REPO_DIR} — provision the card with a clone (just init) whose origin is your remote")
|
||||
})?;
|
||||
|
||||
let Some(mut oid) = stage_and_commit(&repo)? else {
|
||||
return Ok(PublishOutcome::UpToDate);
|
||||
};
|
||||
let branch = repo
|
||||
.head()?
|
||||
.shorthand()
|
||||
@@ -221,17 +263,46 @@ fn publish_once() -> Result<PublishOutcome> {
|
||||
.to_string();
|
||||
let refspec = format!("refs/heads/{branch}:refs/heads/{branch}");
|
||||
|
||||
// Optimistic push. A non-fast-forward rejection means the remote moved under
|
||||
// us: reconcile onto origin and replay the note on the new tip, then retry
|
||||
// once. reconcile_onto_origin mixed-resets, so the just-saved note survives in
|
||||
// the working tree and stage_and_commit lands it on top of origin.
|
||||
if let Err(first) = try_push(&repo, &refspec) {
|
||||
log::warn!("push rejected ({first}); reconciling onto origin and replaying the note");
|
||||
let mut oid = match stage_and_commit(&repo, paths)? {
|
||||
Some(oid) => oid,
|
||||
None => {
|
||||
// Nothing new to commit. Usually genuinely up to date — but a
|
||||
// previous cycle may have committed and then failed to push,
|
||||
// stranding a local-only commit (the old add_all path silently
|
||||
// never retried those). Push whenever origin's tracking ref
|
||||
// doesn't already have HEAD.
|
||||
let head = repo.head()?.peel_to_commit()?.id();
|
||||
if tracking_tip(&repo, &branch) == Some(head) {
|
||||
return Ok(PublishOutcome::UpToDate);
|
||||
}
|
||||
log::info!(
|
||||
"tree unchanged but origin/{branch} lacks HEAD {} — pushing the stranded commit",
|
||||
short(head)
|
||||
);
|
||||
head
|
||||
}
|
||||
};
|
||||
|
||||
// Optimistic push. A non-fast-forward *rejection* means the remote moved
|
||||
// under us: reconcile onto origin and replay the dirty paths on the new
|
||||
// tip, then retry once (reconcile_onto_origin soft-resets — ref move only —
|
||||
// so the notes stay on the card and stage_and_commit splices them on top of
|
||||
// origin). A transport-level failure is surfaced as-is: its fetch would die
|
||||
// the same way, and the commit is safe locally — the stranded-commit check
|
||||
// above pushes it once the transport works again.
|
||||
if let Err(failure) = try_push(&repo, &refspec) {
|
||||
let rejection = match failure {
|
||||
PushFailure::Rejected(msg) => msg,
|
||||
PushFailure::Other(e) => return Err(e),
|
||||
};
|
||||
log::warn!("push rejected ({rejection}); reconciling onto origin and replaying the note");
|
||||
reconcile_onto_origin(&repo, &branch).context("reconciling after a rejected push")?;
|
||||
match stage_and_commit(&repo)? {
|
||||
match stage_and_commit(&repo, paths)? {
|
||||
Some(replayed) => {
|
||||
oid = replayed;
|
||||
try_push(&repo, &refspec).context("push after reconcile")?;
|
||||
try_push(&repo, &refspec)
|
||||
.map_err(PushFailure::into_error)
|
||||
.context("push after reconcile")?;
|
||||
}
|
||||
// The note was already on origin (nothing to replay) — treat as done.
|
||||
None => {
|
||||
@@ -249,65 +320,59 @@ fn publish_once() -> Result<PublishOutcome> {
|
||||
Ok(PublishOutcome::Pushed(short(oid)))
|
||||
}
|
||||
|
||||
/// Stage the working tree and commit it on top of the current branch tip.
|
||||
/// Returns the new commit id, or `None` when the tree already matches the parent
|
||||
/// (nothing to publish). Called on the first attempt and again to replay the note
|
||||
/// after a reconcile.
|
||||
/// Build the commit for `paths` as an O(depth) TreeBuilder splice onto HEAD's
|
||||
/// tree and return the new commit id — or `None` when the result matches the
|
||||
/// parent (nothing to publish). Called on the first attempt and again to
|
||||
/// replay the dirty paths after a reconcile.
|
||||
///
|
||||
/// Staging is `add --all` **plus** `add -u`, which together equal `git add -A`.
|
||||
/// `add_all` stages new + modified files; `update_all` re-syncs already-tracked
|
||||
/// entries to the working tree, which is what actually removes an index entry
|
||||
/// whose file was deleted. Spike 14 found `add_all` alone did **not** stage a
|
||||
/// `:delete`d file's removal on this libgit2 build (the tree came back unchanged,
|
||||
/// so the publish was a silent no-op), so the `update_all` pass is load-bearing,
|
||||
/// not belt-and-braces — do not drop it.
|
||||
/// This replaces the index pipeline (`add_all` → `index.write` → `write_tree`),
|
||||
/// which is O(N_tree) and cannot run on the real 1179-file / 570 MB-pack clone:
|
||||
/// `index.write`'s racy-clean pass re-hashes ~every entry on FAT's 2 s mtimes
|
||||
/// (measured up to **611 s**), and even the index-free `read_tree` walk was
|
||||
/// 77 s. The splice reads and writes only the dirty paths' ancestor chains —
|
||||
/// O(depth × dirty), flat in repo size, **~2–2.8 s measured on the real
|
||||
/// clone** — and carries every untouched entry (including the ~150 MB of
|
||||
/// images) forward by OID without ever opening it. Trail + bench numbers:
|
||||
/// docs/tradeoff-curves/sync-commit-staging.md.
|
||||
///
|
||||
/// Both run a per-path filter that drops macOS AppleDouble sidecars (`._name`)
|
||||
/// and `.DS_Store` that Finder/Spotlight sprinkle onto the FAT card whenever it's
|
||||
/// mounted on a Mac — without it, a blind add --all sweeps them into the commit
|
||||
/// (it did once: 07d87772 shipped `._.git`, `._README.md`, `._notes.md`).
|
||||
/// Filtering here fixes it for *every* repo at the device level, so no per-repo
|
||||
/// `.gitignore` is needed.
|
||||
fn stage_and_commit(repo: &Repository) -> Result<Option<git2::Oid>> {
|
||||
let mut index = repo.index().context("opening index")?;
|
||||
let mut skip_macos_cruft = |path: &Path, _matched: &[u8]| -> i32 {
|
||||
match path.file_name().and_then(|n| n.to_str()) {
|
||||
Some(name) if name.starts_with("._") || name == ".DS_Store" => 1, // skip
|
||||
_ => 0, // add
|
||||
}
|
||||
};
|
||||
// Split the commit window into its sub-phases so we can tell the FAT
|
||||
// working-tree *walk* (`add_all`/`update_all` stat every file over SPI) apart
|
||||
// from the FAT object *writes* (index/tree/commit). This decides whether
|
||||
// explicit-path staging is worth it: the walk is O(tree size) and avoidable
|
||||
// (the editor knows the dirty paths), the writes are O(churn) and a floor.
|
||||
// See docs/tradeoff-curves/sync-commit-staging.md.
|
||||
let t_walk = Instant::now();
|
||||
index
|
||||
.add_all(["*"], IndexAddOption::DEFAULT, Some(&mut skip_macos_cruft))
|
||||
.context("staging new/modified (add --all)")?;
|
||||
// Stage deletions: update_all removes index entries whose working-tree file is
|
||||
// gone. add_all does not do this reliably here (Spike 14), so this is required.
|
||||
index
|
||||
.update_all(["*"], Some(&mut skip_macos_cruft))
|
||||
.context("staging deletions (add -u)")?;
|
||||
let walk_ms = t_walk.elapsed().as_millis();
|
||||
|
||||
let t_index = Instant::now();
|
||||
index.write().context("writing index")?;
|
||||
let index_ms = t_index.elapsed().as_millis();
|
||||
|
||||
let t_tree = Instant::now();
|
||||
let tree = repo.find_tree(index.write_tree().context("writing tree")?)?;
|
||||
let tree_ms = t_tree.elapsed().as_millis();
|
||||
|
||||
// Commit on top of the current branch tip (None on an empty/unborn remote).
|
||||
let t_parent = Instant::now();
|
||||
/// The working tree is the source of truth: a recorded path that exists on the
|
||||
/// card is spliced in from disk, a missing one is spliced out (a `:delete`).
|
||||
/// Unrecorded paths are never visited — so Finder cruft (`._*`, `.DS_Store`)
|
||||
/// on the FAT card can no longer ride into a commit the way it once did with
|
||||
/// `add_all` (07d87772), and the old cruft filter is gone with the walk.
|
||||
fn stage_and_commit(repo: &Repository, paths: &BTreeSet<String>) -> Result<Option<Oid>> {
|
||||
// Commit on top of the current branch tip (None on an empty/unborn remote,
|
||||
// where the splice starts from an empty base and makes a parentless commit).
|
||||
let parent = repo.head().ok().and_then(|h| h.peel_to_commit().ok());
|
||||
let parent_ms = t_parent.elapsed().as_millis();
|
||||
log::info!(
|
||||
"commit split — walk(add_all+update_all) {walk_ms}ms, index.write {index_ms}ms, write_tree {tree_ms}ms, parent-load {parent_ms}ms"
|
||||
);
|
||||
let base = match &parent {
|
||||
Some(c) => Some(c.tree().context("loading HEAD tree")?),
|
||||
None => None,
|
||||
};
|
||||
|
||||
let t_splice = Instant::now();
|
||||
let mut tree = base;
|
||||
for path in paths {
|
||||
let parts: Vec<&str> = path.split('/').filter(|p| !p.is_empty()).collect();
|
||||
if parts.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let blob = match fs::read(format!("{REPO_DIR}/{path}")) {
|
||||
Ok(bytes) => Some(
|
||||
repo.blob(&bytes)
|
||||
.with_context(|| format!("writing blob for {path}"))?,
|
||||
),
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => None, // deleted → splice out
|
||||
Err(e) => return Err(e).with_context(|| format!("reading {path}")),
|
||||
};
|
||||
let spliced = splice(repo, tree.as_ref(), &parts, blob)
|
||||
.with_context(|| format!("splicing {path}"))?;
|
||||
tree = Some(repo.find_tree(spliced).context("loading spliced tree")?);
|
||||
}
|
||||
let Some(tree) = tree else {
|
||||
return Ok(None); // unborn branch and nothing dirty — nothing to commit
|
||||
};
|
||||
let splice_ms = t_splice.elapsed().as_millis();
|
||||
|
||||
if let Some(p) = &parent {
|
||||
if p.tree_id() == tree.id() {
|
||||
log::info!("nothing to publish — tree unchanged @ {}", short(p.id()));
|
||||
@@ -322,20 +387,111 @@ fn stage_and_commit(repo: &Repository) -> Result<Option<git2::Oid>> {
|
||||
let oid = repo
|
||||
.commit(Some("HEAD"), &sig, &sig, &message, &tree, &parents)
|
||||
.context("creating commit")?;
|
||||
let commit_ms = t_commit.elapsed().as_millis();
|
||||
log::info!(
|
||||
"commit split — commit-obj {commit_ms}ms; committed {} — free heap {}",
|
||||
"commit split — splice {splice_ms}ms ({} path(s)), commit-obj {}ms; committed {} — free heap {}",
|
||||
paths.len(),
|
||||
t_commit.elapsed().as_millis(),
|
||||
short(oid),
|
||||
free_heap()
|
||||
);
|
||||
Ok(Some(oid))
|
||||
}
|
||||
|
||||
/// Return a new tree = `base` with `path` set to `blob` (`Some` inserts or
|
||||
/// replaces, `None` removes). Recurses down the path's subtree chain: reads
|
||||
/// ~depth tree objects and writes ~depth new ones, leaving every sibling entry
|
||||
/// untouched (carried by OID — never opened). A missing intermediate directory
|
||||
/// is synthesized on the way down; a directory emptied by a remove is pruned
|
||||
/// on the way up rather than left behind as an empty tree entry.
|
||||
fn splice(repo: &Repository, base: Option<&Tree>, path: &[&str], blob: Option<Oid>) -> Result<Oid> {
|
||||
let (head, rest) = path.split_first().context("splice: empty path")?;
|
||||
let mut tb = repo.treebuilder(base).context("treebuilder")?;
|
||||
if rest.is_empty() {
|
||||
match blob {
|
||||
Some(oid) => {
|
||||
tb.insert(*head, oid, 0o100644)
|
||||
.context("inserting blob entry")?;
|
||||
}
|
||||
// Removing a never-committed path is a no-op, not an error (a note
|
||||
// created and deleted between two syncs).
|
||||
None => {
|
||||
let _ = tb.remove(*head);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
let sub = match base.and_then(|b| b.get_name(head)) {
|
||||
Some(e) if e.kind() == Some(ObjectType::Tree) => {
|
||||
Some(repo.find_tree(e.id()).context("loading subtree")?)
|
||||
}
|
||||
// Absent (a new directory) or a non-tree shadowing the name —
|
||||
// build the subtree from scratch either way.
|
||||
_ => None,
|
||||
};
|
||||
let new_sub = splice(repo, sub.as_ref(), rest, blob)?;
|
||||
if repo.find_tree(new_sub)?.len() == 0 {
|
||||
let _ = tb.remove(*head); // the remove emptied this directory — prune it
|
||||
} else {
|
||||
tb.insert(*head, new_sub, 0o040000)
|
||||
.context("inserting subtree entry")?;
|
||||
}
|
||||
}
|
||||
tb.write().context("writing spliced tree")
|
||||
}
|
||||
|
||||
/// Origin's remote-tracking tip for `branch`, if the ref exists. libgit2
|
||||
/// updates it after a successful push/fetch, so it is "the newest commit we
|
||||
/// know origin has" — without touching the network.
|
||||
fn tracking_tip(repo: &Repository, branch: &str) -> Option<Oid> {
|
||||
repo.find_reference(&format!("refs/remotes/origin/{branch}"))
|
||||
.ok()?
|
||||
.peel_to_commit()
|
||||
.ok()
|
||||
.map(|c| c.id())
|
||||
}
|
||||
|
||||
/// Whether origin is known to already have HEAD (local refs only, no network).
|
||||
/// Errors read as "not current", so the caller falls through to the full
|
||||
/// publish path where the real failure surfaces with context.
|
||||
fn remote_current() -> Result<bool> {
|
||||
let repo = Repository::open(REPO_DIR)?;
|
||||
let head = repo.head()?.peel_to_commit()?.id();
|
||||
let branch = repo
|
||||
.head()?
|
||||
.shorthand()
|
||||
.context("HEAD has no branch shorthand")?
|
||||
.to_string();
|
||||
Ok(tracking_tip(&repo, &branch) == Some(head))
|
||||
}
|
||||
|
||||
/// How a push attempt failed — this decides whether reconciling can help.
|
||||
enum PushFailure {
|
||||
/// The server processed the push but refused the ref update (arrives via
|
||||
/// the `push_update_reference` callback — e.g. non-fast-forward): the
|
||||
/// remote moved under us, and reconcile + replay is the right response.
|
||||
Rejected(String),
|
||||
/// Transport / TLS / auth / URL — the push never reached a ref decision,
|
||||
/// so a reconcile (whose fetch needs the same transport) cannot help.
|
||||
/// Surfaced directly; the 2026-07-13 on-device run burned a doomed
|
||||
/// reconcile on an "unsupported URL protocol" because this wasn't split.
|
||||
Other(anyhow::Error),
|
||||
}
|
||||
|
||||
impl PushFailure {
|
||||
fn into_error(self) -> anyhow::Error {
|
||||
match self {
|
||||
Self::Rejected(msg) => anyhow::anyhow!("remote rejected ref: {msg}"),
|
||||
Self::Other(e) => e,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One push attempt over HTTPS. Binds the PAT credential + the cert-verify
|
||||
/// callback, and surfaces a server-side ref rejection (e.g. non-fast-forward) as
|
||||
/// an error (it arrives via `push_update_reference`, not as a `push()` error).
|
||||
fn try_push(repo: &Repository, refspec: &str) -> Result<()> {
|
||||
let mut remote = repo.find_remote("origin")?;
|
||||
/// callback, and separates a server-side ref rejection (reconcilable) from a
|
||||
/// transport-level failure (not).
|
||||
fn try_push(repo: &Repository, refspec: &str) -> Result<(), PushFailure> {
|
||||
let mut remote = repo
|
||||
.find_remote("origin")
|
||||
.map_err(|e| PushFailure::Other(anyhow::Error::new(e).context("finding remote origin")))?;
|
||||
let rejection: Rc<RefCell<Option<String>>> = Rc::new(RefCell::new(None));
|
||||
|
||||
let mut cbs = auth_callbacks();
|
||||
@@ -353,27 +509,30 @@ fn try_push(repo: &Repository, refspec: &str) -> Result<()> {
|
||||
opts.remote_callbacks(cbs);
|
||||
remote
|
||||
.push(&[refspec], Some(&mut opts))
|
||||
.context("push transport")?;
|
||||
.map_err(|e| PushFailure::Other(anyhow::Error::new(e).context("push transport")))?;
|
||||
|
||||
if let Some(msg) = rejection.borrow().clone() {
|
||||
bail!("remote rejected ref: {msg}");
|
||||
return Err(PushFailure::Rejected(msg));
|
||||
}
|
||||
log::info!("push accepted by remote");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Fetch origin and mixed-reset the local branch onto it, so our just-made commit
|
||||
/// can be replayed on the current tip. Only runs after a non-fast-forward push
|
||||
/// Fetch origin and *soft*-reset the local branch onto it, so our changes can
|
||||
/// be replayed on the current tip. Only runs after a non-fast-forward push
|
||||
/// rejection — i.e. the remote moved under us.
|
||||
///
|
||||
/// **MIXED**, deliberately not a force checkout: the note we're publishing lives
|
||||
/// in the working tree, and a force checkout would clobber it. Mixed moves the
|
||||
/// branch ref + index onto origin but leaves the working tree, so the note
|
||||
/// survives and `stage_and_commit` replays it on top. For a single-writer
|
||||
/// appliance this resolves last-writer-wins — a concurrent remote *edit* to the
|
||||
/// same note loses to ours, and a remote-only *added* file the card doesn't have
|
||||
/// is dropped by the replay's add --all. Both need a real merge (increment B) and
|
||||
/// don't arise from this device's own use.
|
||||
/// **SOFT**, deliberately: it moves only the branch ref. The previous Mixed
|
||||
/// reset also rewrote the index — pure waste now that the splice commit never
|
||||
/// reads the index, and on the real repo an index write is exactly the
|
||||
/// racy-clean wall the splice exists to avoid. Neither flavor touches the
|
||||
/// working tree, so the notes being published survive on the card and the
|
||||
/// replay splices them onto the new tip. For a single-writer appliance this
|
||||
/// resolves last-writer-wins: a concurrent remote *edit* to a note we're
|
||||
/// publishing loses to ours, while a remote-only added/changed file is simply
|
||||
/// carried forward — origin's tree is now the splice base, so the replay
|
||||
/// keeps it (an improvement over the old `add --all` replay, which dropped
|
||||
/// files the card didn't have). A real merge stays increment-B work.
|
||||
fn reconcile_onto_origin(repo: &Repository, branch: &str) -> Result<()> {
|
||||
let mut remote = repo.find_remote("origin")?;
|
||||
let mut fo = FetchOptions::new();
|
||||
@@ -387,12 +546,12 @@ fn reconcile_onto_origin(repo: &Repository, branch: &str) -> Result<()> {
|
||||
.context("no FETCH_HEAD after fetch")?;
|
||||
let theirs = repo.reference_to_annotated_commit(&fetch_head)?;
|
||||
log::info!(
|
||||
"reconcile: resetting local {branch} onto origin @ {} (mixed, keeps the note)",
|
||||
"reconcile: resetting local {branch} onto origin @ {} (soft — ref move only, notes stay on the card)",
|
||||
short(theirs.id())
|
||||
);
|
||||
let their_obj = repo.find_object(theirs.id(), None)?;
|
||||
repo.reset(&their_obj, git2::ResetType::Mixed, None)
|
||||
.context("mixed reset onto origin")?;
|
||||
repo.reset(&their_obj, git2::ResetType::Soft, None)
|
||||
.context("soft reset onto origin")?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
||||
@@ -224,11 +224,21 @@ fn main() -> anyhow::Result<()> {
|
||||
// The outcome returns on `git_rx` and updates the snackbar
|
||||
// (see the idle branch below). The Save that preceded this
|
||||
// in the batch already persisted the buffer, so this is a
|
||||
// pure git push.
|
||||
// pure git publish of the recorded dirty paths — the
|
||||
// outcome decides whether the snapshot is forgotten
|
||||
// (publish_succeeded) or retried (publish_failed).
|
||||
#[cfg(feature = "git")]
|
||||
match git_tx.send(firmware::git_sync::PublishRequest) {
|
||||
Ok(()) => ed.set_notice("syncing..."),
|
||||
Err(_) => ed.set_notice("sync: git thread down"),
|
||||
{
|
||||
let paths = storage.take_dirty();
|
||||
match git_tx.send(firmware::git_sync::PublishRequest { paths }) {
|
||||
Ok(()) => ed.set_notice("syncing..."),
|
||||
Err(_) => {
|
||||
// Thread gone — nothing will report back, so
|
||||
// return the snapshot to pending ourselves.
|
||||
storage.publish_failed();
|
||||
ed.set_notice("sync: git thread down");
|
||||
}
|
||||
}
|
||||
}
|
||||
#[cfg(not(feature = "git"))]
|
||||
log::info!(":sync — saved; light build (no `git` feature) — push skipped");
|
||||
@@ -259,6 +269,13 @@ fn main() -> anyhow::Result<()> {
|
||||
#[cfg(feature = "git")]
|
||||
if let Ok(outcome) = git_rx.try_recv() {
|
||||
use firmware::git_sync::PublishOutcome::*;
|
||||
// Settle the dirty snapshot this publish took: confirmed
|
||||
// published (or up to date) → forget it; failed → back to
|
||||
// pending so the next :sync retries the same paths.
|
||||
match &outcome {
|
||||
Pushed(_) | UpToDate => storage.publish_succeeded(),
|
||||
Failed(_) => storage.publish_failed(),
|
||||
}
|
||||
ed.set_notice(match outcome {
|
||||
Pushed(oid) => format!("synced {oid}"),
|
||||
UpToDate => "up to date".to_string(),
|
||||
@@ -410,7 +427,15 @@ fn main() -> anyhow::Result<()> {
|
||||
/// `main`): the note is the whole point of the appliance, so we refuse to run
|
||||
/// in a state where the next save could destroy it.
|
||||
fn boot_storage(epd: &mut Epd) -> (Storage, String) {
|
||||
let storage = match Storage::mount() {
|
||||
// A git build shares this mount with the git thread, and libgit2 keeps the
|
||||
// pack + idx descriptors open across a publish — that overruns the
|
||||
// editor's tight 4-FD budget, so mount with the 16-FD one (persistence.rs,
|
||||
// MAX_FILES_GIT). The light build keeps the editor's own budget.
|
||||
#[cfg(feature = "git")]
|
||||
let mounted = Storage::mount_for_git();
|
||||
#[cfg(not(feature = "git"))]
|
||||
let mounted = Storage::mount();
|
||||
let storage = match mounted {
|
||||
Ok(s) => s,
|
||||
Err(e) => boot_halt(epd, "SD card not ready", &format!("{e:#}")),
|
||||
};
|
||||
@@ -526,36 +551,64 @@ fn delete_buffer(storage: &Storage, ed: &mut Editor, path: String, scope: Scope)
|
||||
}
|
||||
}
|
||||
|
||||
/// Enumerate the palette's openable files: the top-level regular files in
|
||||
/// `/sd/repo` and `/sd/local`, as absolute paths. Skips dotfiles (so `.git`,
|
||||
/// `.typoena.toml`, and the like never show) and anything that isn't a plain
|
||||
/// file. Best-effort: an unreadable directory (e.g. no `/sd/local` yet)
|
||||
/// contributes nothing rather than failing. The editor sorts and dedupes.
|
||||
/// Enumerate the palette's openable files: the regular files under `/sd/repo`
|
||||
/// and `/sd/local`, recursively, as absolute paths. Skips dot entries at every
|
||||
/// level (so `.git` and its thousands of object files, `.typoena.toml`, and the
|
||||
/// like never show or get walked). Best-effort: an unreadable directory (e.g.
|
||||
/// no `/sd/local` yet) contributes nothing rather than failing. The editor
|
||||
/// sorts and dedupes. Runs once at boot, so the walk time is logged — on a big
|
||||
/// repo the FAT directory IO is the cost to watch.
|
||||
fn enumerate_files() -> Vec<String> {
|
||||
let start = std::time::Instant::now();
|
||||
let mut out = Vec::new();
|
||||
for dir in [REPO_DIR, LOCAL_DIR] {
|
||||
let Ok(entries) = std::fs::read_dir(dir) else {
|
||||
walk_files(std::path::Path::new(dir), 0, &mut out);
|
||||
}
|
||||
log::info!("file walk: {} files in {}ms", out.len(), start.elapsed().as_millis());
|
||||
out
|
||||
}
|
||||
|
||||
/// Depth bound for [`walk_files`] — belt-and-braces against pathological
|
||||
/// nesting on a hand-edited card; notes trees are a couple of levels deep.
|
||||
const WALK_MAX_DEPTH: usize = 8;
|
||||
|
||||
/// Recursive helper for [`enumerate_files`]: push `dir`'s files onto `out`,
|
||||
/// then descend into its subdirectories. Reads each directory fully before
|
||||
/// recursing (the `remove_dir_recursive` pattern in `git_sync`), so only one
|
||||
/// FatFS directory handle is open at a time regardless of depth — relevant on
|
||||
/// the FD-bounded SD mount.
|
||||
fn walk_files(dir: &std::path::Path, depth: usize, out: &mut Vec<String>) {
|
||||
if depth > WALK_MAX_DEPTH {
|
||||
log::warn!("file walk: {} exceeds depth {WALK_MAX_DEPTH}, skipped", dir.display());
|
||||
return;
|
||||
}
|
||||
let Ok(entries) = std::fs::read_dir(dir) else {
|
||||
return;
|
||||
};
|
||||
// Keep the dirent's own file type: esp-idf's FAT VFS always fills d_type
|
||||
// (DT_DIR/DT_REG, straight from the FILINFO readdir already holds), so
|
||||
// `file_type()` is free. A per-entry `metadata()` stat instead re-walks
|
||||
// the directory by path every time — measured at ~32ms/file on the SD
|
||||
// card, it turned a 1098-file walk into 35s.
|
||||
let children: Vec<_> = entries
|
||||
.flatten()
|
||||
.filter_map(|e| e.file_type().ok().map(|t| (e.path(), t)))
|
||||
.collect();
|
||||
for (path, ftype) in children {
|
||||
let Some(name) = path.file_name().and_then(|n| n.to_str()) else {
|
||||
continue;
|
||||
};
|
||||
for entry in entries.flatten() {
|
||||
let path = entry.path();
|
||||
let Some(name) = path.file_name().and_then(|n| n.to_str()) else {
|
||||
continue;
|
||||
};
|
||||
if name.starts_with('.') {
|
||||
continue;
|
||||
}
|
||||
// Stat rather than trust d_type — FatFS's dirent type can read back
|
||||
// as unknown; a plain metadata call is reliable here.
|
||||
if !std::fs::metadata(&path).map(|m| m.is_file()).unwrap_or(false) {
|
||||
continue;
|
||||
}
|
||||
if name.starts_with('.') {
|
||||
continue;
|
||||
}
|
||||
if ftype.is_file() {
|
||||
if let Some(p) = path.to_str() {
|
||||
out.push(p.to_string());
|
||||
}
|
||||
} else if ftype.is_dir() {
|
||||
walk_files(&path, depth + 1, out);
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// A file's display name — its basename without extension (`/sd/repo/notes.md`
|
||||
|
||||
@@ -34,6 +34,8 @@
|
||||
//! sits in the tmp. [`Storage::recover`] closes the loop at boot — see its docs
|
||||
//! for the exact case analysis, which is subtler than "promote the tmp."
|
||||
|
||||
use std::cell::RefCell;
|
||||
use std::collections::BTreeSet;
|
||||
use std::fs;
|
||||
use std::io::Write as _;
|
||||
use std::mem::MaybeUninit;
|
||||
@@ -79,6 +81,14 @@ pub const MAX_FILE_BYTES: u64 = 256 * 1024;
|
||||
/// The C mount point (`/sd\0`) for the esp-idf FFI calls.
|
||||
const MOUNT_C: &std::ffi::CStr = c"/sd";
|
||||
|
||||
/// Dirty-path journal — one repo-relative path per line, mirroring the in-RAM
|
||||
/// dirty set (see [`Storage::take_dirty`]). At the card root, *outside*
|
||||
/// `/sd/repo`, so it can never itself be committed. Without it a power pull
|
||||
/// would strand every file saved-but-not-yet-published in that session: the
|
||||
/// splice commit only visits recorded paths (nothing walks the tree anymore),
|
||||
/// so an unrecorded change would never reach the remote.
|
||||
const DIRTY_JOURNAL: &str = "/sd/.typoena-dirty";
|
||||
|
||||
/// VFS open-file budget for the editor path: it opens only a note and its
|
||||
/// `*.tmp`, so a tight budget keeps FatFS's per-file buffers off the heap.
|
||||
const MAX_FILES_EDITOR: i32 = 4;
|
||||
@@ -95,6 +105,23 @@ const MAX_FILES_GIT: i32 = 16;
|
||||
/// lock serialises the two, so no extra mutex is needed here.
|
||||
pub struct Storage {
|
||||
card: *mut sys::sdmmc_card_t,
|
||||
/// Repo-relative paths saved or `:delete`d since the last confirmed
|
||||
/// publish — the editor-side half of the O(depth) splice commit
|
||||
/// (`git_sync::stage_and_commit` visits exactly these paths and nothing
|
||||
/// else). Mirrored to [`DIRTY_JOURNAL`] whenever it changes, so the record
|
||||
/// survives a power pull. `RefCell` because recording happens inside
|
||||
/// `&self` save/delete calls; `Storage` already lives on one task only.
|
||||
dirty: RefCell<Dirty>,
|
||||
}
|
||||
|
||||
/// The two halves of the dirty record: `pending` accumulates between syncs;
|
||||
/// `take_dirty` moves it to `in_flight` for the duration of a publish so a
|
||||
/// failure can put it back (and a save landing *during* the publish re-enters
|
||||
/// `pending`, riding the next one). The journal always carries the union.
|
||||
#[derive(Default)]
|
||||
struct Dirty {
|
||||
pending: BTreeSet<String>,
|
||||
in_flight: BTreeSet<String>,
|
||||
}
|
||||
|
||||
/// What [`Storage::recover`] did with a leftover `*.tmp` at boot.
|
||||
@@ -223,7 +250,10 @@ impl Storage {
|
||||
}
|
||||
esp!(rc).context("esp_vfs_fat_sdspi_mount (card present? inserted? FAT-formatted?)")?;
|
||||
|
||||
let storage = Storage { card };
|
||||
let storage = Storage {
|
||||
card,
|
||||
dirty: RefCell::new(Dirty::default()),
|
||||
};
|
||||
let (max_khz, real_khz) = storage.negotiated_khz();
|
||||
log::info!("SD mounted at {MOUNT} — max {max_khz} kHz, negotiated {real_khz} kHz");
|
||||
|
||||
@@ -238,6 +268,13 @@ impl Storage {
|
||||
newest complete copy)"
|
||||
),
|
||||
}
|
||||
let carried = storage.load_dirty_journal();
|
||||
if carried > 0 {
|
||||
log::info!(
|
||||
"dirty journal: {carried} unpublished path(s) carried over from a previous \
|
||||
session — the next :sync will commit them"
|
||||
);
|
||||
}
|
||||
Ok(storage)
|
||||
}
|
||||
|
||||
@@ -314,6 +351,16 @@ impl Storage {
|
||||
/// buffers is deferred to the v0.9 crash-safety work — the atomic swap here
|
||||
/// already protects each individual save.
|
||||
pub fn save_path(&self, path: &str, contents: &str) -> Result<()> {
|
||||
// Record BEFORE writing: a crash in between leaves an over-approximate
|
||||
// journal (the splice of an unchanged path is a no-op), whereas the
|
||||
// reverse order could leave a changed file no record ever points at.
|
||||
self.record_dirty(path);
|
||||
Self::atomic_write(path, contents)
|
||||
}
|
||||
|
||||
/// The atomic write primitive behind [`Storage::save_path`] and the dirty
|
||||
/// journal: write `{path}.tmp`, fsync, unlink the target, rename over it.
|
||||
fn atomic_write(path: &str, contents: &str) -> Result<()> {
|
||||
let tmp = format!("{path}.tmp");
|
||||
{
|
||||
let mut f = fs::File::create(&tmp)
|
||||
@@ -350,6 +397,9 @@ impl Storage {
|
||||
/// file half-present after a delete. For a Tracked file this leaves the
|
||||
/// working copy short one file; the next publish's `add --all` stages it.
|
||||
pub fn delete_path(&self, path: &str) -> Result<()> {
|
||||
// Same record-first rule as `save_path`: the splice treats a recorded
|
||||
// path with no file behind it as "remove from the tree".
|
||||
self.record_dirty(path);
|
||||
let _ = fs::remove_file(format!("{path}.tmp"));
|
||||
match fs::remove_file(path) {
|
||||
Ok(()) => Ok(()),
|
||||
@@ -358,6 +408,88 @@ impl Storage {
|
||||
}
|
||||
}
|
||||
|
||||
/// Note a working-copy file as (possibly) differing from HEAD. Paths
|
||||
/// outside `/sd/repo` (`/sd/local`, `/sd/ca.pem`, the journal itself) are
|
||||
/// not git's business and are skipped. The journal is rewritten only when
|
||||
/// the set actually grows, so re-saving the same note between syncs costs
|
||||
/// no extra card I/O.
|
||||
fn record_dirty(&self, abs_path: &str) {
|
||||
let Some(rel) = abs_path
|
||||
.strip_prefix(REPO_DIR)
|
||||
.and_then(|r| r.strip_prefix('/'))
|
||||
else {
|
||||
return;
|
||||
};
|
||||
if rel.is_empty() {
|
||||
return;
|
||||
}
|
||||
let grew = self.dirty.borrow_mut().pending.insert(rel.to_string());
|
||||
if grew {
|
||||
self.persist_dirty();
|
||||
}
|
||||
}
|
||||
|
||||
/// Snapshot the dirty paths for a publish (repo-relative). The snapshot
|
||||
/// moves to `in_flight` — the journal keeps carrying it — until the UI
|
||||
/// task reports the outcome: [`Storage::publish_succeeded`] forgets it,
|
||||
/// [`Storage::publish_failed`] returns it to pending for the next `:sync`.
|
||||
pub fn take_dirty(&self) -> BTreeSet<String> {
|
||||
let mut d = self.dirty.borrow_mut();
|
||||
let taken = std::mem::take(&mut d.pending);
|
||||
d.in_flight.extend(taken.iter().cloned());
|
||||
taken
|
||||
}
|
||||
|
||||
/// The publish that took the last snapshot committed (or confirmed
|
||||
/// up-to-date): drop its paths and shrink the journal. Anything saved
|
||||
/// while it ran is still in `pending` and rides the next sync.
|
||||
pub fn publish_succeeded(&self) {
|
||||
self.dirty.borrow_mut().in_flight.clear();
|
||||
self.persist_dirty();
|
||||
}
|
||||
|
||||
/// The publish failed: return its snapshot to pending so the next `:sync`
|
||||
/// retries it (the splice is idempotent, so a retry of an already-clean
|
||||
/// path is free). The journal already carries these paths — no rewrite.
|
||||
pub fn publish_failed(&self) {
|
||||
let mut d = self.dirty.borrow_mut();
|
||||
let inflight = std::mem::take(&mut d.in_flight);
|
||||
d.pending.extend(inflight);
|
||||
}
|
||||
|
||||
/// Mirror `pending ∪ in_flight` to [`DIRTY_JOURNAL`], atomically.
|
||||
/// Best-effort: a failed journal write must not fail the save that
|
||||
/// triggered it — the set stays correct in RAM and the journal heals on
|
||||
/// the next change.
|
||||
fn persist_dirty(&self) {
|
||||
let contents = {
|
||||
let d = self.dirty.borrow();
|
||||
let mut out = String::new();
|
||||
for p in d.pending.union(&d.in_flight) {
|
||||
out.push_str(p);
|
||||
out.push('\n');
|
||||
}
|
||||
out
|
||||
};
|
||||
if let Err(e) = Self::atomic_write(DIRTY_JOURNAL, &contents) {
|
||||
log::warn!("dirty journal write FAILED ({e:#}); set kept in RAM only");
|
||||
}
|
||||
}
|
||||
|
||||
/// Seed the dirty set from the journal at mount — the paths a previous
|
||||
/// session saved but never got confirmed as published (power pull, failed
|
||||
/// sync, or simply no `:sync` before shutdown). Returns how many.
|
||||
fn load_dirty_journal(&self) -> usize {
|
||||
let Ok(text) = fs::read_to_string(DIRTY_JOURNAL) else {
|
||||
return 0; // no journal yet — nothing carried over
|
||||
};
|
||||
let mut d = self.dirty.borrow_mut();
|
||||
for line in text.lines().map(str::trim).filter(|l| !l.is_empty()) {
|
||||
d.pending.insert(line.to_string());
|
||||
}
|
||||
d.pending.len()
|
||||
}
|
||||
|
||||
/// Reconcile a leftover `notes.md.tmp` at boot. The save sequence is
|
||||
/// write-tmp → fsync → unlink-target → rename, so a lingering tmp means the
|
||||
/// last save was interrupted. Which way to recover depends on whether the
|
||||
|
||||
Reference in New Issue
Block a user