sd_bench dir-entry scaling: stat-miss is ~0.1 ms/entry and the
loose-object composite hits p50 402.9 ms at 256 siblings, reproducing
the field ~360-400 ms/loose-write from primitives. The objects/
fan-out is ~256 dirs, so every path resolution under it pays the scan.
Bounded and accepted; levers (fewer resolutions, pack-not-loose) go to
a future perf pass.
std's remove_dir_all reads directories as char devices through the
prebuilt libc's generic-unix DT table, unlinks them, and FatFS refuses
with FR_DENIED — the 6b fan-out subdirs were the first to trigger it.
Recurse with the same both-tables decoding as the palette walk.
Pairs with :gl as the push half of the git pair. Same behavior
(fmt → save → commit → push); notices, logs, and current READMEs
follow, historical docs keep :sync as a record of their time.
Pulled shape passed on device: apply_tree_diff 1271ms for 1 file, no
crash, interning confirmed (0 KB internal on re-walk). Records run-3
timings, the reconcile 30s→5s session-resumption win, and the
1-byte-load/0-byte-save trailing-newline watch item.
Supersedes the plain case-insensitive matcher from earlier today (user
decision): an all-lowercase pattern searches case-insensitively, one
capital makes it exact (vim smartcase), and diacritics always fold both
ways — /ete finds été, /été finds ete. fold() strips the Latin-1
accent set (ligatures œ/æ excluded: they fold to two chars); n/N
recompute the smartcase verdict from the remembered pattern so repeats
behave like the original search.
Two halves of the connection-cost lever (first on-device pull: 9.7s
fetch to learn 'up to date'; a rejected-push :sync runs three full
handshakes):
- esp_mbedtls_stream.c caches the mbedTLS session at stream close and
offers it on the next connect to the same host — the server then
skips the certificate exchange and most of the key exchange. Second
vendor delta, single git thread, best-effort (any refusal falls back
to a full handshake).
- pull_once now ls-refs first: the ref advertisement alone answers
'anything new?', so up-to-date and local-ahead return without ever
entering pack negotiation, and a needed download rides the SAME open
connection instead of a second handshake. Tracking-ref updates share
one update_tracking helper with the reconcile fetch.
1099 paths as individual Strings measured 182 KB of internal DRAM
resident (each alloc is under the 16 KB SPIRAM-malloc threshold), a
main contributor to the DMA-pool exhaustion behind the :gl crash. The
walk now appends into one newline-joined String seeded at 64 KB so it
and its growth land in PSRAM; the editor stores the blob plus sorted
byte spans (set_file_list_joined) and the palette reads slices. :enew
appends to the blob; :delete drops only the span. Also log the :gl
dirty-journal gate, which looked like a silent no-op on the serial log.
First on-device ff attempt (2026-07-14) crashed: libgit2's SAFE
checkout iterates the whole working directory (readdir over SPI on all
~1100 files) to find dirty files — the same O(tree) wall the splice
commit avoids — and with fetch memory still resident it exhausted
internal DRAM; esp-idf's spi_master then NULL-derefs on its own
failed-DMA-alloc path instead of erroring.
apply_tree_diff mirrors the splice: diff_tree_to_tree skips identical
subtree OIDs in memory, then writes/deletes exactly the changed paths
(unlink+tmp+rename — FAT rename won't overwrite). Belt kept O(changed):
any to-be-clobbered file whose content no longer hashes to the old blob
aborts the pull before the first write. Half-applied + unmoved ref
self-heals on the next :gl (idempotent re-apply).
Also raise the internal DMA reserve 32→64 KB so the SPI driver never
sees a failed alloc (its error path is the crash), and log the fetch
completion so the fetch/apply phases are separable in the serial log.
Unicode lowercase-fold comparison at char boundaries (find_ci/rfind_ci)
instead of str::find — no lowercased buffer copy, whose byte lengths
could shift the returned offsets. Prose search shouldn't care about
capitalization; deliberately not smartcase (a capital silently flipping
to exact-match is a surprise here, not a feature).
The splash's ~2.2s full-refresh waveform now runs while the SD mounts
and the note/prefs load (Epd::display_frame_async + wait_ready guard on
every public display call), and the 4.3s palette walk moves to a 16KB
background thread whose result lands in the idle branch — the first
editor frame no longer waits on non-mandatory work. The post-pull
re-walk rides the same channel, removing a 4.3s UI stall after :gl.
Also: rustfmt pass on sd_bench, drop the spike-9 log line in splash.
sd_bench grows a directory-entry-scaling section (stat hit/miss and
the loose-object composite at 8/64/256 siblings): FAT resolves every
path component by linear scan, so cost climbing with N convicts
directory scans as the ~360ms residual; flat cost exonerates them.
stage_and_commit brackets the splice with the p_mmap counters and logs
'N mmaps / M KB read' per commit, splitting the residual into pack-read
I/O vs everything else on the next real :sync.
The git channel now carries GitRequest::{Publish,Pull} with a shared
ensure_online preamble (Wi-Fi/clock/TLS once per session). pull_once
fetches origin — refreshing the tracking ref so the radio-free
up-to-date check stays honest — and maps four shapes: up to date,
LocalAhead (stranded commit, :sync's job), clean fast-forward,
Diverged (refused; no merge on the device). The fast-forward is a SAFE
checkout then ref move: it refuses to overwrite content that differs
from HEAD, the belt under the UI gate that refuses :gl while the dirty
journal is non-empty. A RAM-dirty buffer doesn't gate — its edits win
(last-writer-wins, like the reconcile).
After a pull the UI drops clean parked buffers, re-reads the clean
active buffer in place (Editor::refresh_active), and re-walks the
palette list. Firmware 0.6.0 -> 0.7.0. On-device verification pending.
Literal case-sensitive substring search sharing the : command line's
editing (cmd_prompt discriminates the prompt; Enter dispatches). Jumps
on Enter only — incremental caret-chasing loses to the e-ink refresh
cost, same call as the snippet no-popup decision. Wraps with a notice,
keeps the caret on a miss, bare / repeats the editor-global last
pattern. n/N live in move_by so counts and Visual extension come free;
they are not operator targets.
Kaizen/postmortem next-steps kept accumulating with no owner. Each item
now lives in one version doc (v0.7 sync perf, v0.9 robustness, v1.0
boot) with the source docs linking there; the editor-freeze safety net
is marked hardware-verified and the v0.5 walk re-measure closed at 4.3s.
Single-pack card prep is now enforced, not remembered: each pack's
whole .idx (plus any midx) is mmap'd outside the device's window
budget, and a multi-pack marking walk lost the HTTP keep-alive (run 8:
31s vs 3.5s repacked). Skipped when the clone is already one pack.
It served its purpose (localized the mmap plateau, runs 7-9); its 8KB
internal stack was live during the TLS send, where internal min-ever
hit 2099 bytes. The time-gated pack/transfer callbacks stay.
Step 6 evaluated: infinity (never completes) to 24.1s measured
snackbar-to-snackbar, with the measured after-bar charted in step 6 (the
step-1 chart stays potential-only, per the format). Documents the second
localization loop the verification runs forced on the push half: UI death
by shared heap fate, odb cache exonerated, telemetry axis bug, mmaps named
as the eater, the impossible fixed-maps budget, the keep-alive race, and
pack layout as a device-performance variable.
Runs 7-9 on the real repo (2026-07-13): esp_map live-bytes wired into
log_push_heap plus a 5s sibling heartbeat thread proved the push OOM was
libgit2's mmaps — ~3.4MB of whole-file .idx/multi-pack-index maps living
OUTSIDE the mwindow budget, plus windows at the (working) 4MB soft limit,
an impossible budget on the 8MB pool. Shrinking to 64KB windows / 1.5MB
mapped limit left 4.5MB min-ever free and cut marking reads 4x; with the
card repacked to a single pack, the first-ever real-repo push landed
(6713ea6e, :sync 24.1s end-to-end).
The heartbeat stays for now (the marking phase fires no libgit2 callbacks,
so it is the only heap trace through it); its 8KB internal stack is part
of the min-ever-2099B-internal tightness flagged for follow-up.
Run 4 (2026-07-13): remote.push() consumed ~6 MB of PSRAM over 66 s and
the UI thread aborted on Frame::new_white's per-draw vec![0xFF; 26928].
Run 5 confirmed the UI now survives, but the push still exhausts both
pools (a ~7 KB inflateInit fails inside the pack build) and the heap
telemetry never fired — hence the telemetry rework.
- display/editor: Editor::draw_into() renders into a caller-owned
Frame; main.rs keeps two boot-time frames (shown/back) and mem::swaps
on success, so steady-state repaints never allocate.
- git_sync: odb cache capped at 1 MB via raw libgit2-sys opts (git2
0.20 doesn't wrap the total cap); log_push_heap at pre-push,
post-push and push-failure with largest-PSRAM-block and per-pool
min-evers; pack_progress logging is now time-gated (2 s) — the
count gate (+256 objects) never fired because AddingObjects reports
total=0 and a small push inserts only dozens of objects.
- editor: empty/whitespace snippets file parses as 0 snippets instead
of a JSON error at boot.
- build.rs: refuse a git-feature build with unset TW_* publish vars —
a bare `cargo build --features git` baked empty creds and produced a
firmware whose :sync could never work.
- Palette Ctrl-N/P (and half-page keys) wrap around the result list
instead of clamping at the ends; tests updated to the new semantics.
- try_push maps libgit2's negotiation-time NotFastForward error to
PushFailure::Rejected so the reconcile path runs instead of failing
as a transport error.
- walk_files decodes esp-idf d_type values (DT_REG=1/DT_DIR=2) that
libc 0.2.178's generic unix table misreads as fifo/chardev — the
palette walk was silently dropping every file on the card.
- `just load` refuses to mirror over unpublished device edits listed
in the card's .typoena-dirty journal (backup+discard via
TW_DISCARD_UNPUBLISHED=1), gains --force for a from-scratch copy,
and a flash-only recipe flashes the already-built ELF.
The tradeoff-curves doc stays the chronological data layer; the kaizen
is the story layer — value model, 8-factor hypothesis table, ideas
ranked by verified cause, and an evaluation left PENDING until the
end-to-end on-device :sync lands.
"free heap 5.8MB" in the crash log was PSRAM and masked the actual
internal-RAM exhaustion. git_sync's publish/commit/push lines now
carry the MALLOC_CAP_INTERNAL reading, and boot brackets the palette
file-list build with it — small Strings are forced internal by the
16 KB SPIRAM malloc threshold, so the 1098-path list is a suspected
~60-70 KB DRAM consumer competing with Wi-Fi/TLS; the log decides
whether interning into one PSRAM buffer is worth it.
esp-idf's default is internal-RAM-only, and a TLS connection needs
~33 KB of it (two ~17 KB I/O buffers + contexts) at the exact moment
:sync pushes — with Wi-Fi, USB host, the editor and libgit2 resident,
mbedtls_ssl_setup failed there on the first real-repo push (and its
failure path then hit the stream double-free fixed in the previous
commit). TLS buffers are CPU-only data, so PSRAM is safe; the
handshake is network-bound.
Upstream libgit2 v1.9.4 bug: mbedtls_stream_wrap's ssl_setup error
path closes and frees the caller's socket stream, then
git_mbedtls_stream_new frees it again — and wrap's other error paths
don't free it at all, so callers can't compensate either way. When
ssl_setup failed on the device (internal-RAM exhaustion during the
first real-repo push), the double git__free tripped tlsf ("block
already marked as free") and reset the chip mid-:sync.
Fixed the way esp_map.c replaces map.c: the vendored streams/mbedtls.c
is excluded from the component and esp_mbedtls_stream.c (verbatim copy
+ one-hunk fix: out_err leaves st->io to the caller, and frees the
leaked ssl struct) takes its place. Keep the copy in lockstep on
submodule bumps; worth reporting upstream.
The per-entry metadata() call makes FatFS re-walk the directory by
path every time — ~32ms/file on the card, 35s for a 1098-file tree.
esp-idf's FAT VFS always fills d_type (DT_DIR/DT_REG from the FILINFO
readdir already holds, never unknown) and Rust std maps file_type()
onto it stat-free, so the walk is now one readdir pass per directory.
With the file list now a recursive walk of the whole card, an unranked
short query pages through too much. Below PALETTE_MIN_QUERY (2) the
result list is the MRU only, keeping Cmd-P + Enter quick-switch one
keystroke away; two chars reveal the full fuzzy-ranked list. Commands
(>) and snippets ($) are short curated lists and keep matching from
the first char.
A nested repo (jcalixte/notes) showed only its top-level files in the
palette. Dot entries are skipped at every level so .git is never
descended; each directory is read fully before recursing so a single
FatFS dir handle is open at a time; depth capped at 8. The boot walk
logs its file count and duration to keep the FAT dir-IO cost visible.
The merged plan section now records the shipped design: single path
set (working tree as truth) instead of {changed, deleted}, the dirty
journal, stranded-commit recovery, radio-free up-to-date, and the
soft-reset reconcile with its carried-files side win.
The device's libgit2 has no SSH transport (HTTPS+PAT over mbedTLS
only), but _load-repo copied the desktop clone's origin verbatim — an
SSH-shaped URL fails every on-device push/fetch with "unsupported URL
protocol" (first real-repo :sync, 2026-07-13). Derive the HTTPS
equivalent for the card copy (git@host:path and ssh://git@host/path);
the source clone keeps its own URL. Warns on non-github hosts, since
the embedded trust store carries GitHub roots only.
The index pipeline (add_all → index.write → write_tree) is O(N_tree)
and cannot commit the real 1179-file / 570 MB-pack clone (index.write
measured up to 611 s on FAT's racy-clean re-hash). stage_and_commit is
now an O(depth) TreeBuilder splice of exactly the paths the editor
saved or deleted — ~2-2.8 s on the real clone — with the working tree
as source of truth (existing file → insert, missing → remove).
Storage records those repo-relative paths on every save/delete and
journals them to /sd/.typoena-dirty (atomic, only on growth), so a
power pull can't strand a saved-but-unpublished note now that nothing
walks the tree. take_dirty → publish_succeeded/publish_failed settles
each publish's snapshot from the UI outcome handler.
Also required by / discovered with the splice:
- mwindow opts at git-service start (32-bit defaults would OOM PSRAM
on the first real-pack access; bench-proven 256 KB / 4 MB).
- 16-FD mount for git builds (libgit2 holds pack+idx descriptors open;
the editor's 4-FD budget overruns).
- reconcile is a soft reset (no index to reset anymore); side win: a
remote-only added file is carried by the replay instead of dropped.
- stranded-commit recovery: tree-unchanged now pushes anyway when
origin/<branch> lacks HEAD (a commit whose push failed used to be
silently never retried).
- radio-free up-to-date: empty dirty set + origin at HEAD answers
without bringing Wi-Fi up.
- try_push splits ref rejection (reconcilable) from transport failure
(surfaced directly) — the first on-device run burned a doomed
reconcile on "unsupported URL protocol" and hid the cause.
Deliberate behavior change: files changed on the card outside the
editor are never committed anymore (also retires the macOS-cruft
filter). Trail: docs/tradeoff-curves/sync-commit-staging.md.
The bench phase the handoff was written for is closed (splice benched,
fast-seek landed, cache removed, decision made), so the note's live half —
the firmware plumbing plan — moves into sync-commit-staging.md as "The fix —
wiring the O(depth) splice into the firmware", reconciled to the run-5 state.
The duplicated TL;DR/measurement summaries are dropped in favor of the
curve's own trail; inbound references (notes index, git_bench comments)
now point at the curve.
Run 4: memory discipline verified (1833 KB flat, no OOM) but 0 hits
with the small maps demonstrably retained — window-repetition theory
refuted, sub-second bar failed (splice 2.83 s cold / ~2 s steady).
Decision: wire the splice in anyway (~9-10 s cold real-repo :sync vs
611 s/OOM for every alternative). Run 5: cache removal confirmed
free; the ~1.85 MB "resident" was mwindow's live window set, which
makes the mwindow opts in shipping git_sync.rs load-bearing.
Four instrumented real-repo bench runs scored 0 cache hits: libgit2's
mwindow layer reuses its open windows, so only genuinely new
(offset, len) ranges ever reach p_mmap — there is no repetition to
cache. The 7.4 MB OOM the v2 discipline fixed was caused by the cache
itself holding buffers past p_munmap; plain free-at-munmap is honest
with MWINDOW_MAPPED_LIMIT by construction. Run 5 confirmed removal is
I/O-neutral on device (byte-identical read pattern, warm splice 1953
vs 1949 ms) and even 15 reads better — v2's low-water eviction was
fighting mwindow. Stats counters kept to spot any future workload
that does repeat ranges.
Admission keyed on file size (>= 1 MB) instead of map length: the hot
set is the small repeated maps (pack trailer, idx fanout, delta bases)
that the 64 KB floor excluded — 0 hits across three real-repo runs.
Small mutable working-tree files stay uncacheable. p_munmap now evicts
unreferenced entries to a 2 MB low-water mark so released windows are
actually returned to git__malloc (fixes the 7.4 MB resident / 508 KB
heap zlib OOM and keeps MWINDOW_MAPPED_LIMIT honest).
Without CONFIG_FATFS_USE_FASTSEEK every long/backward lseek in the
263 MB pack walks the FAT cluster chain over SPI (~190 ms), and libgit2
pays ~8 such seeks per loose-object write. The CLMT makes lseek O(1)
for read-mode files; verified on device: far seek 198.7 -> 20.4 ms,
splice commit 6.5 -> 2.8 s.
Reads 4 KB at the start vs the end of the repo's largest pack (skipping
macOS ._ sidecars). Proved the ~1.5 s/loose-object cost was FatFS
walking the FAT cluster chain on every long/backward lseek: 5.8 ms at
offset 0 vs 198.7 ms at the end of the 263 MB pack.
Adds the splice prototype (patch one path onto HEAD's tree, O(depth))
as git_bench's headline op, run first so its cold number survives the
index ops' OOM, plus odb.read_header/odb.exists probes and strict-off
re-benches that localized the residual cost and refuted the
strict-object-creation theory.
The real jcalixte/notes clone (570 MB pack, 1179 files) proved both index
strategies are O(N_tree) and unshippable: index.write re-hashes the tree
(611 s), and the index-free read_tree is 77 s cold and OOMs the mmap cache
(zlib crash). Record the run in the tradeoff curve, revise the verdict to an
O(depth) TreeBuilder walk, and add a handoff note with the design, firmware
call sites, and bench steps for the next session.
Replace the index.write()/write_tree ops (which re-hash the whole working
tree on a fresh FAT clone -- up to 611 s on the real repo) with the
index-free candidate: Index::new + read_tree(HEAD) + blob + write_tree_to.
Add per-op mmap-cache + free-heap logging (via esp_map_stats), announce each
op before it runs so a hang reveals the culprit, mount_for_git for the FD
budget, tune the mwindow limits, and drop N 10 -> 3 for the slow real clone.
libgit2 keeps the pack, .idx and commit-graph descriptors open for the
repo's lifetime and opens loose objects on top, so a read_tree walk overruns
the editor's tight 4-FD budget with "no free file descriptors". Add
mount_for_git (16 FDs, matching the flash-FAT git binaries) alongside the
editor's mount; both route through a shared mount_with_max_files.
p_mmap emulated a mapping by malloc+read()ing the range from SD on every
call. libgit2 re-hits the pack idx/windows on every object write (via
git_odb__freshen -> git_odb_refresh), so a commit re-read pack bytes over
SPI repeatedly -- ~500ms-1.1s/op on a small repo, far worse on the real
570 MB pack.
Cache read-only mappings >= 64 KB (pack idx/windows, commit-graph, midx,
packed-refs -- all immutable on this device) in a refcounted, PSRAM-backed
slot table keyed by (dev, ino, size, mtime, offset, len), LRU-evicted under
a soft cap. Small mutable working-tree maps (diff_file.c) fall below the
floor and are never served stale.
Break the stage+commit window into sub-phases (FAT working-tree walk vs
object writes) via `commit split —` log lines, and add two micro-benchmarks
(sd_bench for SD/FAT primitive ops, git_bench for libgit2 object overhead)
with justfile recipes. Documents the walk-vs-writes cost model in
tradeoff-curves/sync-commit-staging.md to decide whether explicit-path
staging over the editor's dirty set is worth replacing add_all(["*"]).
