typewriter/docs/v0.7-search-and-git.md

v0.7 — Search + better git

Part of the Typoena macro plan. Requirements and targets: qfd.md. Load-bearing decisions: adr.md.

Status: CLOSED 2026-07-14. / search is done and panel-verified (smartcase + accent-folded; n/N confirmed on-device). :gl is verified on-device in all four shapes: the fast-forward (Pulled) closing gate passed on run 3 — main 17840720 → e738a6a1, apply_tree_diff wrote the one changed file in 1271 ms, internal DRAM never dropped below ~48 KB, and the palette re-walk picked the new file up in the background (0 KB internal consumed — interning confirmed). Run 2 had crashed there in libgit2's checkout_tree (O(tree) workdir walk + internal-DRAM exhaustion + an esp-idf spi_master NULL-deref on its own failed-alloc path); the rebuild as an O(changed) tree-diff apply plus three memory fixes (file-list interning, 64 KB DMA reserve, TLS session reuse) is what run 3 verified. Run-1 bonus: the first on-device rejected-push → reconcile → replay → push success; run 3 repeated that cycle at 24 s total (was 59 s — the reconcile fetch fell from ~30 s to ~5 s with session resumption). Residual watch item from run 3: boot loaded 1 byte from notes.md but :w saved 0 — possible trailing-newline round-trip asymmetry on an empty-note buffer (host-test it; harmless here, the tree was unchanged).

• / forward search, n N — core 2026-07-13, panel-verified 2026-07-14 (including n/N). Literal substring (no regex on-device), matched smartcase + accent-folded (2026-07-14, user decision — supersedes the same-day plain-insensitive version): an all-lowercase pattern is case-insensitive, one capital makes it exact (vim smartcase), and diacritics always fold both ways — /ete finds été, /été finds ete (Latin-1 repertoire; ligatures œ/æ excluded — they fold to two chars). n/N recompute smartcase from the remembered pattern, so repeats behave like the original search. / reuses the : command line's editing with a / prompt (one command-line mode, as in vim), and the jump happens on Enter only — no incremental caret-chasing, for the same e-ink-refresh reason snippets have no completion popup. Wraps around with a "wrapped" notice; "not found: pat" leaves the caret alone; a bare /+Enter repeats the last pattern, which is editor-global (survives buffer switches, like vim's search register). n/N are motions (counts work, Visual extends over them); deliberately not operator targets (dn out of scope).
• :gl — pull: fetch + fast-forward only, refuse on divergence and surface it (renamed from the planned :Gpull). On-device 2026-07-14: UpToDate shape, dirty-journal gate, and the reconcile/replay path verified (runs 1 and 3); Pulled (fast-forward) verified run 3 (ls-refs 2792 ms, download 5136 ms, apply 1271 ms / 1 file, :gl total 16.0 s — ~3 s of the total is still unaccounted connect/negotiation, tracked with the publish gap below). How it works:
  • The git thread takes a GitRequest::Pull on the same channel as publish (shared ensure_online Wi-Fi/clock/TLS preamble). pull_once does ls-refs first: the ref advertisement alone answers "anything new?", so up to date and LocalAhead (stranded commit — :sync's job) return without entering pack negotiation (run 1 paid a 9.7 s fetch to learn "up to date"); a needed download rides the same open connection. Divergence is refused — no merge on the device. Tracking-ref updates keep publish's radio-free up-to-date check honest.
  • The fast-forward is an O(changed) tree-diff apply, not checkout_tree (apply_tree_diff, the splice philosophy on the read side): diff_tree_to_tree skips identical subtree OIDs in memory, then exactly the changed paths are written (unlink+tmp+rename — FAT f_rename won't overwrite) or unlinked, and the ref moves last. libgit2's SAFE checkout instead iterates the whole working directory (readdir over SPI on all ~1100 files) — run 2 died there: internal DRAM was exhausted mid-walk and esp-idf's spi_master NULL-derefs on its own failed-DMA-alloc path (LoadProhibited in setup/uninstall_priv_desc). Belt kept O(changed): any to-be-clobbered file whose content no longer hashes to the old tree's blob aborts the pull before the first write (edits made behind git's back — e.g. desktop edits directly on the card). A half-applied working copy with the ref unmoved self-heals: the next :gl re-applies idempotently.
  • UI gate: :gl is refused while the dirty journal is non-empty ("unsynced changes - :sync first", also logged) — an unpublished save would fight the apply, and sync-then-pull is the single-writer appliance's natural order. A RAM-dirty (never-saved) buffer does not gate: its edits simply win over the pulled state (last-writer-wins, like the reconcile).
  • After a Pulled outcome the UI drops clean parked buffers (next switch re-reads the disk), re-reads the clean active buffer in place (Editor::refresh_active — boot posture, undo cleared), keeps a RAM-dirty active buffer untouched, and re-walks the palette file list in the background. Snackbar: pulled <oid> / up to date / ahead - :sync to publish / diverged - resolve on a computer / pull: <reason>.

Memory + connection work landed here (2026-07-14, forced by the run-2 crash; all three compose):

• File-list interning — 1099 palette paths as individual Strings measured 182 KB of internal DRAM (each alloc under the 16 KB SPIRAM-malloc threshold), a main contributor to the DMA-pool exhaustion. The walk now emits one newline-joined blob (seeded at 64 KB → PSRAM); the editor stores blob + sorted spans (set_file_list_joined) and the palette reads slices. Confirmed run 3: the post-pull re-walk logged 0 KB consumed, blob 51 KB. (The boot walk's "125 KB consumed" is a measurement artifact — that window runs concurrently with USB attach and other boot allocations; the quiescent re-walk is the clean reading.)
• DMA reserve 32 → 64 KB (CONFIG_SPIRAM_MALLOC_RESERVE_INTERNAL) — the esp-idf SPI driver crashes rather than errors on a failed DMA alloc, so the floor must be generous enough that it never sees one.
• TLS session resumption + one-connection pull — every git op paid a full multi-second handshake (a rejected-push :sync runs three). esp_mbedtls_stream.c (second vendor delta) caches the mbedTLS session at stream close and offers it on the next connect to the same host. First same-session evidence (run 3): the reconcile fetch — always the second connection of a rejected-push :sync — fell from ~30 s (run 1) to ~5 s; a controlled first-op vs second-op ls-refs comparison is still worth logging when convenient.

Sync performance — inherited from the real-repo-sync kaizen (closed 2026-07-13 at 24.1 s end-to-end; target ≤ ~10 s):

• [~] Close the publish gap: 19.8 s at the kaizen close — the splice's 5.0 s and the ~7 s of repo-open/negotiation overhead are the next curves (tradeoff analysis). TLS session resumption (above) attacks the negotiation half; re-measure a warm :sync before hunting further. Run-3 data point: a rejected-push :sync (worst case, three connections) totalled 24.0 s vs 59 s in run 1; a clean warm publish is still unmeasured.
• [~] Instrument the residual ~360 ms/loose-write (suspect: FAT directory-op cost in the freshen/refresh path) — instrumentation BUILT 2026-07-14, measurement run pending: sd_bench grew a directory-entry-scaling section (stat hit/miss + the loose-object composite at 8/64/256 siblings — FAT resolves paths by linear scan, so cost climbing with N convicts dir scans; flash with just flash-bench), and stage_and_commit now brackets the splice with the p_mmap counters, logging N mmaps / M KB read per commit. First data point (run 1, 2026-07-14): both real-repo splices logged 0 mmaps / 0 KB read — the residual is FAT ops, not pack reads.
• The images-off-card lever (notes/git-sync-images-and-repo-size.md) composes with the splice — decide whether to take it here (user's call).