The per-entry metadata() call makes FatFS re-walk the directory by
path every time — ~32ms/file on the card, 35s for a 1098-file tree.
esp-idf's FAT VFS always fills d_type (DT_DIR/DT_REG from the FILINFO
readdir already holds, never unknown) and Rust std maps file_type()
onto it stat-free, so the walk is now one readdir pass per directory.
With the file list now a recursive walk of the whole card, an unranked
short query pages through too much. Below PALETTE_MIN_QUERY (2) the
result list is the MRU only, keeping Cmd-P + Enter quick-switch one
keystroke away; two chars reveal the full fuzzy-ranked list. Commands
(>) and snippets ($) are short curated lists and keep matching from
the first char.
A nested repo (jcalixte/notes) showed only its top-level files in the
palette. Dot entries are skipped at every level so .git is never
descended; each directory is read fully before recursing so a single
FatFS dir handle is open at a time; depth capped at 8. The boot walk
logs its file count and duration to keep the FAT dir-IO cost visible.
The merged plan section now records the shipped design: single path
set (working tree as truth) instead of {changed, deleted}, the dirty
journal, stranded-commit recovery, radio-free up-to-date, and the
soft-reset reconcile with its carried-files side win.
The device's libgit2 has no SSH transport (HTTPS+PAT over mbedTLS
only), but _load-repo copied the desktop clone's origin verbatim — an
SSH-shaped URL fails every on-device push/fetch with "unsupported URL
protocol" (first real-repo :sync, 2026-07-13). Derive the HTTPS
equivalent for the card copy (git@host:path and ssh://git@host/path);
the source clone keeps its own URL. Warns on non-github hosts, since
the embedded trust store carries GitHub roots only.
The index pipeline (add_all → index.write → write_tree) is O(N_tree)
and cannot commit the real 1179-file / 570 MB-pack clone (index.write
measured up to 611 s on FAT's racy-clean re-hash). stage_and_commit is
now an O(depth) TreeBuilder splice of exactly the paths the editor
saved or deleted — ~2-2.8 s on the real clone — with the working tree
as source of truth (existing file → insert, missing → remove).
Storage records those repo-relative paths on every save/delete and
journals them to /sd/.typoena-dirty (atomic, only on growth), so a
power pull can't strand a saved-but-unpublished note now that nothing
walks the tree. take_dirty → publish_succeeded/publish_failed settles
each publish's snapshot from the UI outcome handler.
Also required by / discovered with the splice:
- mwindow opts at git-service start (32-bit defaults would OOM PSRAM
on the first real-pack access; bench-proven 256 KB / 4 MB).
- 16-FD mount for git builds (libgit2 holds pack+idx descriptors open;
the editor's 4-FD budget overruns).
- reconcile is a soft reset (no index to reset anymore); side win: a
remote-only added file is carried by the replay instead of dropped.
- stranded-commit recovery: tree-unchanged now pushes anyway when
origin/<branch> lacks HEAD (a commit whose push failed used to be
silently never retried).
- radio-free up-to-date: empty dirty set + origin at HEAD answers
without bringing Wi-Fi up.
- try_push splits ref rejection (reconcilable) from transport failure
(surfaced directly) — the first on-device run burned a doomed
reconcile on "unsupported URL protocol" and hid the cause.
Deliberate behavior change: files changed on the card outside the
editor are never committed anymore (also retires the macOS-cruft
filter). Trail: docs/tradeoff-curves/sync-commit-staging.md.
The bench phase the handoff was written for is closed (splice benched,
fast-seek landed, cache removed, decision made), so the note's live half —
the firmware plumbing plan — moves into sync-commit-staging.md as "The fix —
wiring the O(depth) splice into the firmware", reconciled to the run-5 state.
The duplicated TL;DR/measurement summaries are dropped in favor of the
curve's own trail; inbound references (notes index, git_bench comments)
now point at the curve.
Run 4: memory discipline verified (1833 KB flat, no OOM) but 0 hits
with the small maps demonstrably retained — window-repetition theory
refuted, sub-second bar failed (splice 2.83 s cold / ~2 s steady).
Decision: wire the splice in anyway (~9-10 s cold real-repo :sync vs
611 s/OOM for every alternative). Run 5: cache removal confirmed
free; the ~1.85 MB "resident" was mwindow's live window set, which
makes the mwindow opts in shipping git_sync.rs load-bearing.
Four instrumented real-repo bench runs scored 0 cache hits: libgit2's
mwindow layer reuses its open windows, so only genuinely new
(offset, len) ranges ever reach p_mmap — there is no repetition to
cache. The 7.4 MB OOM the v2 discipline fixed was caused by the cache
itself holding buffers past p_munmap; plain free-at-munmap is honest
with MWINDOW_MAPPED_LIMIT by construction. Run 5 confirmed removal is
I/O-neutral on device (byte-identical read pattern, warm splice 1953
vs 1949 ms) and even 15 reads better — v2's low-water eviction was
fighting mwindow. Stats counters kept to spot any future workload
that does repeat ranges.
Admission keyed on file size (>= 1 MB) instead of map length: the hot
set is the small repeated maps (pack trailer, idx fanout, delta bases)
that the 64 KB floor excluded — 0 hits across three real-repo runs.
Small mutable working-tree files stay uncacheable. p_munmap now evicts
unreferenced entries to a 2 MB low-water mark so released windows are
actually returned to git__malloc (fixes the 7.4 MB resident / 508 KB
heap zlib OOM and keeps MWINDOW_MAPPED_LIMIT honest).
Without CONFIG_FATFS_USE_FASTSEEK every long/backward lseek in the
263 MB pack walks the FAT cluster chain over SPI (~190 ms), and libgit2
pays ~8 such seeks per loose-object write. The CLMT makes lseek O(1)
for read-mode files; verified on device: far seek 198.7 -> 20.4 ms,
splice commit 6.5 -> 2.8 s.
Reads 4 KB at the start vs the end of the repo's largest pack (skipping
macOS ._ sidecars). Proved the ~1.5 s/loose-object cost was FatFS
walking the FAT cluster chain on every long/backward lseek: 5.8 ms at
offset 0 vs 198.7 ms at the end of the 263 MB pack.
Adds the splice prototype (patch one path onto HEAD's tree, O(depth))
as git_bench's headline op, run first so its cold number survives the
index ops' OOM, plus odb.read_header/odb.exists probes and strict-off
re-benches that localized the residual cost and refuted the
strict-object-creation theory.
The real jcalixte/notes clone (570 MB pack, 1179 files) proved both index
strategies are O(N_tree) and unshippable: index.write re-hashes the tree
(611 s), and the index-free read_tree is 77 s cold and OOMs the mmap cache
(zlib crash). Record the run in the tradeoff curve, revise the verdict to an
O(depth) TreeBuilder walk, and add a handoff note with the design, firmware
call sites, and bench steps for the next session.
Replace the index.write()/write_tree ops (which re-hash the whole working
tree on a fresh FAT clone -- up to 611 s on the real repo) with the
index-free candidate: Index::new + read_tree(HEAD) + blob + write_tree_to.
Add per-op mmap-cache + free-heap logging (via esp_map_stats), announce each
op before it runs so a hang reveals the culprit, mount_for_git for the FD
budget, tune the mwindow limits, and drop N 10 -> 3 for the slow real clone.
libgit2 keeps the pack, .idx and commit-graph descriptors open for the
repo's lifetime and opens loose objects on top, so a read_tree walk overruns
the editor's tight 4-FD budget with "no free file descriptors". Add
mount_for_git (16 FDs, matching the flash-FAT git binaries) alongside the
editor's mount; both route through a shared mount_with_max_files.
p_mmap emulated a mapping by malloc+read()ing the range from SD on every
call. libgit2 re-hits the pack idx/windows on every object write (via
git_odb__freshen -> git_odb_refresh), so a commit re-read pack bytes over
SPI repeatedly -- ~500ms-1.1s/op on a small repo, far worse on the real
570 MB pack.
Cache read-only mappings >= 64 KB (pack idx/windows, commit-graph, midx,
packed-refs -- all immutable on this device) in a refcounted, PSRAM-backed
slot table keyed by (dev, ino, size, mtime, offset, len), LRU-evicted under
a soft cap. Small mutable working-tree maps (diff_file.c) fall below the
floor and are never served stale.
Break the stage+commit window into sub-phases (FAT working-tree walk vs
object writes) via `commit split —` log lines, and add two micro-benchmarks
(sd_bench for SD/FAT primitive ops, git_bench for libgit2 object overhead)
with justfile recipes. Documents the walk-vs-writes cost model in
tradeoff-curves/sync-commit-staging.md to decide whether explicit-path
staging over the editor's dirty set is worth replacing add_all(["*"]).
esp-idf-hal always compiles its i2c.rs, which binds the old driver/i2c.h
API, so ESP-IDF links the legacy driver's constructor into every image and
prints its deprecation warning at boot. Typoena uses no I2C (EPD/SD are SPI,
keyboard is USB), so the warning is pure noise; this flag compiles the
constructor (and its conflict-abort check) out.
`just load` / `just init` now fast-forward the source repo from its
origin before rsyncing to /sd/repo, so the card picks up notes the
device already pushed. Fast-forward-only and best-effort: dirty tree,
diverged history, detached HEAD, or offline warns and copies the
current on-disk state rather than aborting. TW_NO_PULL=1 skips it.
Default espflash baud (115200) made a full flash take ~2-3 min of
serial transfer. Bump the cargo runner and the two direct-espflash
recipes (flash-git-push, flash-git-sync) to 921600. Monitor recipes
keep 115200 to match the device UART log rate.
Mark v0.6 delivered (core, host-tested; on-device gate pending): the
snippet engine, the Cmd-P/>/$ palette split retiring :e, and the just
init catalog. Note the →/≠ ISO-8859-15 caveat.
Record the firmware boot-read (serde_json builds for xtensa) and the
just-init catalog as delivered, and replace the proposed French catalog
with the finalised 17-snippet English set (three groups). Note the
→/≠ base-font caveat and the no-clobber seeding guard.
Add a curated snippet catalog (firmware/snippets-catalog/, three opt-in
groups: Symbols, Structure, Prose — 17 snippets, English prefixes,
translated from the source Zed set) and a _seed-configs step in `just
init` that jq-merges the chosen groups into repo/.typoena.snippets.json
and writes a starter repo/.typoena.toml. Seeds only files that are
absent, so re-running init never clobbers a synced library.
Mirror the prefs boot-read: load the git-tracked snippet library from the
SD repo, parse it with Snippets::parse, and hand it to the editor before
the first render. Missing/unreadable/malformed is non-fatal (no snippets,
editor runs). Verified serde_json builds for xtensa (cargo check, 0.6.0).
Record the > command registry, one-shot/two-step dispatch, and the :e
retirement as done in core; note the ISO-8859-15 label constraints
(ASCII `...`, `»` hint marker).
Turn the > palette from a settings-only list into an action registry
dispatched by PaletteCmd::kind: toggles stay open, one-shots (format,
publish) run and close, and the parameterised `new file...` morphs the
palette into a filename input step. Share run_publish between :sync and
the publish command, and retire :e (bare Cmd-P opens files).
Snapshot the snippet name inline Tab would expand into a panel field in
refresh_stats, so it rides the typing-pause throttle rather than
repainting per keystroke. Draw it as `» name` on the row above the mode
line (Latin-9 has no tab glyph), sharing the slot with the NO KBD flag,
which can't co-occur since the hint means you're typing.
Parse the Zed-compatible .typoena.snippets.json (serde_json), strip
${n:label} stops to bare $n, and drive a forward-only tab-stop session.
Expand a prefix inline on Tab in Insert; browse and insert from the
Cmd-P palette via a $ sigil (parallel to the > command mode), landing
on $1 as one undo group.
Reshape v0.6 snippets from a hard-coded table into a git-synced,
Zed-compatible JSON library (.typoena.snippets.json) with an inline Tab
path and a $ palette launcher. Add the file-format reference and
document the Cmd-P/>/$ verb split and the just init catalog.
Reference doc gains a theme key, an Options column, and the e-paper dark-mode
ghosting caveat. The v0.5 spec gets an amendment note superseding the two
reversed "decide-before-build" calls (auto_sync now a set-ahead palette
command; theme shipped early) without rewriting the verified-on-device
history; macroplan updated to match.
Generalise the palette so Enter advances any pref to its next value and
wraps; a boolean toggle is the two-option case. Both string prefs share a
next_option(current, &OPTIONS) helper (off-list values snap to the head).
- theme (light/dark): rendered as a single whole-frame Frame::invert at the
end of draw(), so text, selection, caret, panel and palette flip together.
- auto_sync: cycles 2m/5m/10m/15m/30m. Set-ahead only — still read by
nothing until the v0.7 periodic push.
toggle_pref -> cycle_pref; palette hint "Enter toggle" -> "Enter change".
Record the d14d9e7 behaviour the doc didn't cover: format-on-save keeps at
most one trailing blank line, and load-verbatim + guarded-save render a
file's POSIX terminator as a visible trailing empty line.
Move each version's scope checklist and status out of the monolithic
macroplan into a dedicated docs/vX.Y-<slug>.md page; macroplan keeps the
source block, the rollup status, and a one-line summary + link per
version. v0.1 reuses its existing product/technical pages.
Also drop the "Optional column ruler at 80" requirement from v0.6.
Read notes verbatim and insert the final newline only when it is missing,
instead of stripping the terminator on load and appending it unconditionally
on save. The editor's `rows = #\n + 1` model then renders a file's POSIX
terminator as a visible trailing blank line — what a writer expects: open a
note and see (and land the caret on) the empty line the newline stands for.
Supersedes the strip-on-load / unconditional-append handling that shipped
with the prefs work (c535864), which kept the buffer newline-free and hid the
terminator. Load + save are now an identity round-trip for any device-written
file (all end in '\n'); files stay git-clean (exactly one terminator); and a
trailing blank line the writer leaves is mirrored, never doubled.
- load_path: read verbatim (drop the strip)
- save_path: guarded final-newline (drop the unconditional append)
- Prefs::to_toml: ends in a newline again — the guarded save leaves exactly
one, so the prefs file is byte-identical to before and its device-verified
round-trip still holds
- sd_fat spike: payload ends in '\n' so its exact-equality round-trip holds
All four v0.5 slices are done and device-verified: the prefs round-trip is
confirmed both directions (boot-read + on-device palette edit -> push flipped
line_numbers on origin), and the slice-3 delete fix is recorded as confirmed on
device. Descoped items (buffer close, grey-Publish-in-Local cue, multi-file
publish count) noted, not blocking delivery.
Git-tracked editor preferences read at boot and toggled live on-device:
- Prefs type (line-based TOML parse/serialize, no crate on xtensa) held on
Editor; firmware reads /sd/repo/.typoena.toml before the first render and
falls back to per-key defaults. Keys: save_on_idle, format_on_save,
line_numbers (bool) + auto_sync (string, schema/default only until v0.7).
- line_numbers applied live (gutter_cols -> 0 when off).
- Palette > command mode toggles the three bools; the list stays open so
several flip in one visit, and :settings opens it directly. Each toggle
applies live and queues Effect::SavePrefs (host atomic-writes the file,
which rides the next :sync).
- save_on_idle honoured host-side as a silent, unformatted idle auto-save.
- to_toml is newline-free; save_path now appends exactly one terminator
unconditionally so buffers round-trip byte-for-byte (trailing blanks kept).
- Firmware 0.4.0 -> 0.5.0; new docs/typoena-toml.md reference. Also refreshes
the slice-3 macroplan status (delete fix was confirmed on device).
Add the POSIX line terminator on save and strip it on load, so files
written by the editor no longer trip git's "No newline at end of file".
Done at the persistence choke point, not in :fmt: the editor buffer is
newline-free by design (rows = #\n + 1), so a trailing '\n' in the buffer
would render a phantom blank last line. save_path adds exactly one '\n'
(guarded against doubling); load_path strips one back off so the buffer
stays newline-free and round-trips byte-stable.
Update the sd_fat spike payload to be newline-free so its byte-identity
round-trip assertion holds under the new normalizing contract.
:enew <name> creates a new file (empty, dirty, added to the palette list);
:delete unlinks the current file via a new Effect::Delete and switches to a
parked buffer or scratch. Scope is read from the path (local/x, repo/x) rather
than a modal prompt, and the /sd prefix is now optional in resolve_path.
On-device testing showed deletions never reached the remote: add_all(["*"])
alone does not stage a removal on this libgit2, so the tree came back unchanged
and the push was a silent no-op. stage_and_commit now runs add_all then
update_all(["*"]) (git add -u) — together git add -A. The :delete snackbar now
confirms the scoped file and flags that a Tracked file is local until :sync.
Cmd-P opens a modal transient palette over the writing column: a bare
fuzzy-search input, the ranked file list, and the selected row in reverse
video. A pure host-testable fuzzy_score (subsequence + word-boundary and
consecutive-run bonuses) ranks results; an in-core MRU floats recently
opened files first and is shared with :e (both route through open_path).
The host feeds the file list once at boot (enumerate_files over /sd/repo
and /sd/local). Ctrl-n/Ctrl-p navigate the list; Enter opens via the same
park/evict path as :e; Esc closes.
Ctrl-n/Ctrl-p also become down/up line motions in Normal and View (vim
CTRL-N/CTRL-P, count-aware), which is why the palette opener is Cmd-P
alone. No `>` prefix on the file input — `>` is reserved for the command
palette (slice 4).
112 editor + 28 keymap tests; the no-git firmware binary builds clean.
Rework the single Effect return into a drained effect queue
(Save{path,scope,contents} / Load / Publish / Pull) so one action can
ask the host for several ordered steps: opening a non-resident file
queues a Save of the outgoing dirty buffer then a Load of the target.
Keep the active buffer's fields inline on Editor and park inactive
buffers in a small LRU (<=3 resident = active + 2); switching back to a
resident buffer restores its caret/scroll/undo without touching the SD.
A dirty parked buffer is saved before eviction, so nothing leaves RAM
unsaved. `:e <path>` opens by prefix (/sd/repo -> Tracked, /sd/local ->
Local); `:sync` is refused in-core for a Local buffer.
Firmware drains the queue to empty each batch (a Load can cascade an
eviction Save) and persistence generalises the atomic save off the
hard-coded notes.md into load_path/save_path.
Also bump the side panel to FONT_9X15 and the `:` command line to
FONT_10X20 for legibility, word-wrapping the snackbar so a long notice
keeps its actionable tail.
Charwise `v` and linewise `V` selection with yank/delete/change on the
span; motions and counts extend it. Read-only View moves off v/V to
`gr` (go-read). Selection renders reverse-video on the 1-bit panel.
Normal motions factored into a shared move_by. Firmware -> 0.4.0.
A multi-line paste near the bottom of the screen left its later lines below
the viewport — adjust_scroll only kept the caret's (first) pasted line
visible. Add reveal(), which scrolls the end of the pasted block into view
while the caret stays on its first line (vim semantics unchanged).
Root cause, shipped safety net, and the specced eradication (a
persistent internal DMA scratch buffer in Epd so paints never allocate
mid-sync). Tracks hardware re-test and the eradication as follow-ups.
A screen refresh that ran while :sync had Wi-Fi + TLS up could fail to
allocate an internal DMA bounce buffer (ESP_ERR_NO_MEM); the error
propagated through ? out of main(), main_task returned from app_main(),
and the editor loop died while the USB and git threads kept running —
keys logged, panel frozen forever.
A paint is idempotent and retryable (the buffer is the source of truth),
so every editor-loop paint now logs and drops the frame instead of
propagating, leaves shown untouched so the next paint repaints the diff,
and forces a full refresh next to re-sync both RAM banks. Same contract
as save_note. The boot-time first render stays fatal on purpose.
