typewriter/docs/v0.7-search-and-git.md

v0.7 — Search + better git

Part of the Typoena macro plan. Requirements and targets: qfd.md. Load-bearing decisions: adr.md.

Status: feature-complete, one gate left. / search is done and panel-verified (case-insensitive since 2026-07-14; n/N confirmed on-device). :gl is built end-to-end and three of its four shapes are on-device verified; the fast-forward (Pulled) shape is the closing gate — its first attempt (2026-07-14, run 2) crashed in libgit2's checkout_tree (O(tree) workdir walk + internal-DRAM exhaustion + an esp-idf spi_master NULL-deref on its own failed-alloc path), and the checkout has been rebuilt as an O(changed) tree-diff apply plus three memory fixes (file-list interning, 64 KB DMA reserve, TLS session reuse). Re-run the Mac-push → device :gl test to close the version. Run-1 bonus: the first on-device rejected-push → reconcile → replay → push success (replay commit 59444a94).

• / forward search, n N — core 2026-07-13, panel-verified 2026-07-14 (including n/N). Literal substring (no regex on-device), matched smartcase + accent-folded (2026-07-14, user decision — supersedes the same-day plain-insensitive version): an all-lowercase pattern is case-insensitive, one capital makes it exact (vim smartcase), and diacritics always fold both ways — /ete finds été, /été finds ete (Latin-1 repertoire; ligatures œ/æ excluded — they fold to two chars). n/N recompute smartcase from the remembered pattern, so repeats behave like the original search. / reuses the : command line's editing with a / prompt (one command-line mode, as in vim), and the jump happens on Enter only — no incremental caret-chasing, for the same e-ink-refresh reason snippets have no completion popup. Wraps around with a "wrapped" notice; "not found: pat" leaves the caret alone; a bare /+Enter repeats the last pattern, which is editor-global (survives buffer switches, like vim's search register). n/N are motions (counts work, Visual extends over them); deliberately not operator targets (dn out of scope).
• [~] :gl — pull: fetch + fast-forward only, refuse on divergence and surface it (renamed from the planned :Gpull). On-device 2026-07-14: UpToDate shape, dirty-journal gate, and the reconcile/replay path verified; the Pulled (fast-forward) shape is rebuilt after the run-2 crash and awaits its on-device pass — the closing gate. How it works:
  • The git thread takes a GitRequest::Pull on the same channel as publish (shared ensure_online Wi-Fi/clock/TLS preamble). pull_once does ls-refs first: the ref advertisement alone answers "anything new?", so up to date and LocalAhead (stranded commit — :sync's job) return without entering pack negotiation (run 1 paid a 9.7 s fetch to learn "up to date"); a needed download rides the same open connection. Divergence is refused — no merge on the device. Tracking-ref updates keep publish's radio-free up-to-date check honest.
  • The fast-forward is an O(changed) tree-diff apply, not checkout_tree (apply_tree_diff, the splice philosophy on the read side): diff_tree_to_tree skips identical subtree OIDs in memory, then exactly the changed paths are written (unlink+tmp+rename — FAT f_rename won't overwrite) or unlinked, and the ref moves last. libgit2's SAFE checkout instead iterates the whole working directory (readdir over SPI on all ~1100 files) — run 2 died there: internal DRAM was exhausted mid-walk and esp-idf's spi_master NULL-derefs on its own failed-DMA-alloc path (LoadProhibited in setup/uninstall_priv_desc). Belt kept O(changed): any to-be-clobbered file whose content no longer hashes to the old tree's blob aborts the pull before the first write (edits made behind git's back — e.g. desktop edits directly on the card). A half-applied working copy with the ref unmoved self-heals: the next :gl re-applies idempotently.
  • UI gate: :gl is refused while the dirty journal is non-empty ("unsynced changes - :sync first", also logged) — an unpublished save would fight the apply, and sync-then-pull is the single-writer appliance's natural order. A RAM-dirty (never-saved) buffer does not gate: its edits simply win over the pulled state (last-writer-wins, like the reconcile).
  • After a Pulled outcome the UI drops clean parked buffers (next switch re-reads the disk), re-reads the clean active buffer in place (Editor::refresh_active — boot posture, undo cleared), keeps a RAM-dirty active buffer untouched, and re-walks the palette file list in the background. Snackbar: pulled <oid> / up to date / ahead - :sync to publish / diverged - resolve on a computer / pull: <reason>.

Memory + connection work landed here (2026-07-14, forced by the run-2 crash; all three compose):

• File-list interning — 1099 palette paths as individual Strings measured 182 KB of internal DRAM (each alloc under the 16 KB SPIRAM-malloc threshold), a main contributor to the DMA-pool exhaustion. The walk now emits one newline-joined blob (seeded at 64 KB → PSRAM); the editor stores blob + sorted spans (set_file_list_joined) and the palette reads slices. Expected walk log line now: ~0 KB internal consumed — confirm on the next boot.
• DMA reserve 32 → 64 KB (CONFIG_SPIRAM_MALLOC_RESERVE_INTERNAL) — the esp-idf SPI driver crashes rather than errors on a failed DMA alloc, so the floor must be generous enough that it never sees one.
• TLS session resumption + one-connection pull — every git op paid a full multi-second handshake (a rejected-push :sync runs three). esp_mbedtls_stream.c (second vendor delta) caches the mbedTLS session at stream close and offers it on the next connect to the same host; measure as first-op vs second-op ls-refs/handshake time in a session.

Sync performance — inherited from the real-repo-sync kaizen (closed 2026-07-13 at 24.1 s end-to-end; target ≤ ~10 s):

• [~] Close the publish gap: 19.8 s at the kaizen close — the splice's 5.0 s and the ~7 s of repo-open/negotiation overhead are the next curves (tradeoff analysis). TLS session resumption (above) attacks the negotiation half; re-measure a warm :sync before hunting further.
• [~] Instrument the residual ~360 ms/loose-write (suspect: FAT directory-op cost in the freshen/refresh path) — instrumentation BUILT 2026-07-14, measurement run pending: sd_bench grew a directory-entry-scaling section (stat hit/miss + the loose-object composite at 8/64/256 siblings — FAT resolves paths by linear scan, so cost climbing with N convicts dir scans; flash with just flash-bench), and stage_and_commit now brackets the splice with the p_mmap counters, logging N mmaps / M KB read per commit. First data point (run 1, 2026-07-14): both real-repo splices logged 0 mmaps / 0 KB read — the residual is FAT ops, not pack reads.
• The images-off-card lever (notes/git-sync-images-and-repo-size.md) composes with the splice — decide whether to take it here (user's call).