typewriter/docs/v0.7-search-and-git.md

v0.7 — Search + better git

Part of the Typoena macro plan. Requirements and targets: qfd.md. Load-bearing decisions: adr.md.

Status: CLOSED 2026-07-14. / search is done and panel-verified (smartcase + accent-folded; n/N confirmed on-device). :gl is verified on-device in all four shapes: the fast-forward (Pulled) closing gate passed on run 3 — main 17840720 → e738a6a1, apply_tree_diff wrote the one changed file in 1271 ms, internal DRAM never dropped below ~48 KB, and the palette re-walk picked the new file up in the background (0 KB internal consumed — interning confirmed). Run 2 had crashed there in libgit2's checkout_tree (O(tree) workdir walk + internal-DRAM exhaustion + an esp-idf spi_master NULL-deref on its own failed-alloc path); the rebuild as an O(changed) tree-diff apply plus three memory fixes (file-list interning, 64 KB DMA reserve, TLS session reuse) is what run 3 verified. Run-1 bonus: the first on-device rejected-push → reconcile → replay → push success; run 3 repeated that cycle at 24 s total (was 59 s — the reconcile fetch fell from ~30 s to ~5 s with session resumption). Residual watch item from run 3: boot loaded 1 byte from notes.md but :w saved 0 — possible trailing-newline round-trip asymmetry on an empty-note buffer (host-test it; harmless here, the tree was unchanged).

• / forward search, n N — core 2026-07-13, panel-verified 2026-07-14 (including n/N). Literal substring (no regex on-device), matched smartcase + accent-folded (2026-07-14, user decision — supersedes the same-day plain-insensitive version): an all-lowercase pattern is case-insensitive, one capital makes it exact (vim smartcase), and diacritics always fold both ways — /ete finds été, /été finds ete (Latin-1 repertoire; ligatures œ/æ excluded — they fold to two chars). n/N recompute smartcase from the remembered pattern, so repeats behave like the original search. / reuses the : command line's editing with a / prompt (one command-line mode, as in vim), and the jump happens on Enter only — no incremental caret-chasing, for the same e-ink-refresh reason snippets have no completion popup. Wraps around with a "wrapped" notice; "not found: pat" leaves the caret alone; a bare /+Enter repeats the last pattern, which is editor-global (survives buffer switches, like vim's search register). n/N are motions (counts work, Visual extends over them); deliberately not operator targets (dn out of scope).
• :gl — pull: fetch + fast-forward only, refuse on divergence and surface it (renamed from the planned :Gpull). On-device 2026-07-14: UpToDate shape, dirty-journal gate, and the reconcile/replay path verified (runs 1 and 3); Pulled (fast-forward) verified run 3 (ls-refs 2792 ms, download 5136 ms, apply 1271 ms / 1 file, :gl total 16.0 s — ~3 s of the total is still unaccounted connect/negotiation, tracked with the publish gap below). How it works:
  • The git thread takes a GitRequest::Pull on the same channel as publish (shared ensure_online Wi-Fi/clock/TLS preamble). pull_once does ls-refs first: the ref advertisement alone answers "anything new?", so up to date and LocalAhead (stranded commit — :sync's job) return without entering pack negotiation (run 1 paid a 9.7 s fetch to learn "up to date"); a needed download rides the same open connection. Divergence is refused — no merge on the device. Tracking-ref updates keep publish's radio-free up-to-date check honest.
  • The fast-forward is an O(changed) tree-diff apply, not checkout_tree (apply_tree_diff, the splice philosophy on the read side): diff_tree_to_tree skips identical subtree OIDs in memory, then exactly the changed paths are written (unlink+tmp+rename — FAT f_rename won't overwrite) or unlinked, and the ref moves last. libgit2's SAFE checkout instead iterates the whole working directory (readdir over SPI on all ~1100 files) — run 2 died there: internal DRAM was exhausted mid-walk and esp-idf's spi_master NULL-derefs on its own failed-DMA-alloc path (LoadProhibited in setup/uninstall_priv_desc). Belt kept O(changed): any to-be-clobbered file whose content no longer hashes to the old tree's blob aborts the pull before the first write (edits made behind git's back — e.g. desktop edits directly on the card). A half-applied working copy with the ref unmoved self-heals: the next :gl re-applies idempotently.
  • UI gate: :gl is refused while the dirty journal is non-empty ("unsynced changes - :sync first", also logged) — an unpublished save would fight the apply, and sync-then-pull is the single-writer appliance's natural order. A RAM-dirty (never-saved) buffer does not gate: its edits simply win over the pulled state (last-writer-wins, like the reconcile).
  • After a Pulled outcome the UI drops clean parked buffers (next switch re-reads the disk), re-reads the clean active buffer in place (Editor::refresh_active — boot posture, undo cleared), keeps a RAM-dirty active buffer untouched, and re-walks the palette file list in the background. Snackbar: pulled <oid> / up to date / ahead - :sync to publish / diverged - resolve on a computer / pull: <reason>.
  • Superseded (v0.9): divergence is no longer refused — :gl now rebases the device's local commit(s) onto origin (rebase_local_onto, last-writer-wins per note, no content merge) and ends LocalAhead for :gp to publish. The diverged - resolve on a computer snackbar is replaced by rebased <oid> - :gp to publish. See v0.9-robustness.md.

Memory + connection work landed here (2026-07-14, forced by the run-2 crash; all three compose):

• File-list interning — 1099 palette paths as individual Strings measured 182 KB of internal DRAM (each alloc under the 16 KB SPIRAM-malloc threshold), a main contributor to the DMA-pool exhaustion. The walk now emits one newline-joined blob (seeded at 64 KB → PSRAM); the editor stores blob + sorted spans (set_file_list_joined) and the palette reads slices. Confirmed run 3: the post-pull re-walk logged 0 KB consumed, blob 51 KB. (The boot walk's "125 KB consumed" is a measurement artifact — that window runs concurrently with USB attach and other boot allocations; the quiescent re-walk is the clean reading.)
• DMA reserve 32 → 64 KB (CONFIG_SPIRAM_MALLOC_RESERVE_INTERNAL) — the esp-idf SPI driver crashes rather than errors on a failed DMA alloc, so the floor must be generous enough that it never sees one.
• TLS session resumption + one-connection pull — every git op paid a full multi-second handshake (a rejected-push :sync runs three). esp_mbedtls_stream.c (second vendor delta) caches the mbedTLS session at stream close and offers it on the next connect to the same host. First same-session evidence (run 3): the reconcile fetch — always the second connection of a rejected-push :sync — fell from ~30 s (run 1) to ~5 s; a controlled first-op vs second-op ls-refs comparison is still worth logging when convenient.

Sync performance — inherited from the real-repo-sync kaizen (closed 2026-07-13 at 24.1 s end-to-end; target ≤ ~10 s):

• [~] Close the publish gap: 19.8 s at the kaizen close — the splice's 5.0 s and the ~7 s of repo-open/negotiation overhead are the next curves (tradeoff analysis). TLS session resumption (above) attacks the negotiation half; re-measure a warm :sync before hunting further. Run-3 data point: a rejected-push :sync (worst case, three connections) totalled 24.0 s vs 59 s in run 1. Warm clean publish measured 2026-07-14 (run 4, as :gp after the rename): 19.1 s — but the mix shifted entirely. The push leg collapsed to 5.9 s (6 objects; TLS-resumed connect 2.4 s vs 4.0 s for the session's first; post-repack marking cost invisible), and the splice grew to 10.3 s because the committed file sat four directories deep: O(depth) means ~5 loose tree writes instead of 1 (× the ~0.4 s dir-scan cost above) plus the session's first pack touch (5 mmaps / 1808 KB — the 1740 KB .idx map dominates). Same run: up-to-date :gl is now 4.7 s git-side via the ls-refs fast path ("no fetch" — was 9.7 s in run 1). So the residual curve is now splice depth × loose-write cost, which is the same FAT-dir-scan story as above and shares its levers (pack-not-loose foremost) — a future perf pass; a root-level file's warm :gp should sit ≈ 12–13 s.
• Instrument the residual ~360 ms/loose-write — measured 2026-07-14, CONVICTED: FAT linear directory scans (full analysis in the tradeoff doc, run 6). sd_bench's dir-entry-scaling section shows stat-miss cost linear in sibling count (~0.1 ms/entry/scan) and the loose-object composite reproducing the field number exactly: p50 402.9 ms at 256 siblings — the .git/objects/ fan-out is ~256 dirs, and every path resolution under it scans them. Composes with the run-1 splice counters (0 mmaps / 0 KB read — no pack I/O in the residual). Bounded (≤ ~2 s per typical commit) and accepted; the levers — fewer path resolutions in libgit2, or committing straight into a small pack instead of loose objects — are a future perf pass, not v0.x work. (The bench run also caught a cleanup bug: std's remove_dir_all mis-types directories via the generic-unix DT constants and dies on FatFS FR_DENIED; sd_bench now removes its scratch tree itself, same decoding as the palette walk. The aborted run left /sd/sdbench on the card — the next bench run's setup sweep removes it.)
• The images-off-card lever — decided + landed 2026-07-14 (skip-media-in-apply): the repo stays untouched (its images are remanso's blob-served CDN), and the device-side half shipped as media paths invisible to apply_tree_diff (never written/deleted/belt-hashed — a pulled 16 MB image was :gl's last OOM path). Accepted trade: the card's media go stale vs HEAD, so a computer sees phantom modified/deleted images in git status — device-owned card, refresh with just load, never hand-commit from it. Full write-up in notes/git-sync-images-and-repo-size.md.