The per-entry metadata() call makes FatFS re-walk the directory by
path every time — ~32ms/file on the card, 35s for a 1098-file tree.
esp-idf's FAT VFS always fills d_type (DT_DIR/DT_REG from the FILINFO
readdir already holds, never unknown) and Rust std maps file_type()
onto it stat-free, so the walk is now one readdir pass per directory.
A nested repo (jcalixte/notes) showed only its top-level files in the
palette. Dot entries are skipped at every level so .git is never
descended; each directory is read fully before recursing so a single
FatFS dir handle is open at a time; depth capped at 8. The boot walk
logs its file count and duration to keep the FAT dir-IO cost visible.
The index pipeline (add_all → index.write → write_tree) is O(N_tree)
and cannot commit the real 1179-file / 570 MB-pack clone (index.write
measured up to 611 s on FAT's racy-clean re-hash). stage_and_commit is
now an O(depth) TreeBuilder splice of exactly the paths the editor
saved or deleted — ~2-2.8 s on the real clone — with the working tree
as source of truth (existing file → insert, missing → remove).
Storage records those repo-relative paths on every save/delete and
journals them to /sd/.typoena-dirty (atomic, only on growth), so a
power pull can't strand a saved-but-unpublished note now that nothing
walks the tree. take_dirty → publish_succeeded/publish_failed settles
each publish's snapshot from the UI outcome handler.
Also required by / discovered with the splice:
- mwindow opts at git-service start (32-bit defaults would OOM PSRAM
on the first real-pack access; bench-proven 256 KB / 4 MB).
- 16-FD mount for git builds (libgit2 holds pack+idx descriptors open;
the editor's 4-FD budget overruns).
- reconcile is a soft reset (no index to reset anymore); side win: a
remote-only added file is carried by the replay instead of dropped.
- stranded-commit recovery: tree-unchanged now pushes anyway when
origin/<branch> lacks HEAD (a commit whose push failed used to be
silently never retried).
- radio-free up-to-date: empty dirty set + origin at HEAD answers
without bringing Wi-Fi up.
- try_push splits ref rejection (reconcilable) from transport failure
(surfaced directly) — the first on-device run burned a doomed
reconcile on "unsupported URL protocol" and hid the cause.
Deliberate behavior change: files changed on the card outside the
editor are never committed anymore (also retires the macOS-cruft
filter). Trail: docs/tradeoff-curves/sync-commit-staging.md.
The bench phase the handoff was written for is closed (splice benched,
fast-seek landed, cache removed, decision made), so the note's live half —
the firmware plumbing plan — moves into sync-commit-staging.md as "The fix —
wiring the O(depth) splice into the firmware", reconciled to the run-5 state.
The duplicated TL;DR/measurement summaries are dropped in favor of the
curve's own trail; inbound references (notes index, git_bench comments)
now point at the curve.
Four instrumented real-repo bench runs scored 0 cache hits: libgit2's
mwindow layer reuses its open windows, so only genuinely new
(offset, len) ranges ever reach p_mmap — there is no repetition to
cache. The 7.4 MB OOM the v2 discipline fixed was caused by the cache
itself holding buffers past p_munmap; plain free-at-munmap is honest
with MWINDOW_MAPPED_LIMIT by construction. Run 5 confirmed removal is
I/O-neutral on device (byte-identical read pattern, warm splice 1953
vs 1949 ms) and even 15 reads better — v2's low-water eviction was
fighting mwindow. Stats counters kept to spot any future workload
that does repeat ranges.
Reads 4 KB at the start vs the end of the repo's largest pack (skipping
macOS ._ sidecars). Proved the ~1.5 s/loose-object cost was FatFS
walking the FAT cluster chain on every long/backward lseek: 5.8 ms at
offset 0 vs 198.7 ms at the end of the 263 MB pack.
Adds the splice prototype (patch one path onto HEAD's tree, O(depth))
as git_bench's headline op, run first so its cold number survives the
index ops' OOM, plus odb.read_header/odb.exists probes and strict-off
re-benches that localized the residual cost and refuted the
strict-object-creation theory.
Replace the index.write()/write_tree ops (which re-hash the whole working
tree on a fresh FAT clone -- up to 611 s on the real repo) with the
index-free candidate: Index::new + read_tree(HEAD) + blob + write_tree_to.
Add per-op mmap-cache + free-heap logging (via esp_map_stats), announce each
op before it runs so a hang reveals the culprit, mount_for_git for the FD
budget, tune the mwindow limits, and drop N 10 -> 3 for the slow real clone.
libgit2 keeps the pack, .idx and commit-graph descriptors open for the
repo's lifetime and opens loose objects on top, so a read_tree walk overruns
the editor's tight 4-FD budget with "no free file descriptors". Add
mount_for_git (16 FDs, matching the flash-FAT git binaries) alongside the
editor's mount; both route through a shared mount_with_max_files.
Break the stage+commit window into sub-phases (FAT working-tree walk vs
object writes) via `commit split —` log lines, and add two micro-benchmarks
(sd_bench for SD/FAT primitive ops, git_bench for libgit2 object overhead)
with justfile recipes. Documents the walk-vs-writes cost model in
tradeoff-curves/sync-commit-staging.md to decide whether explicit-path
staging over the editor's dirty set is worth replacing add_all(["*"]).
Mirror the prefs boot-read: load the git-tracked snippet library from the
SD repo, parse it with Snippets::parse, and hand it to the editor before
the first render. Missing/unreadable/malformed is non-fatal (no snippets,
editor runs). Verified serde_json builds for xtensa (cargo check, 0.6.0).
Read notes verbatim and insert the final newline only when it is missing,
instead of stripping the terminator on load and appending it unconditionally
on save. The editor's `rows = #\n + 1` model then renders a file's POSIX
terminator as a visible trailing blank line — what a writer expects: open a
note and see (and land the caret on) the empty line the newline stands for.
Supersedes the strip-on-load / unconditional-append handling that shipped
with the prefs work (c535864), which kept the buffer newline-free and hid the
terminator. Load + save are now an identity round-trip for any device-written
file (all end in '\n'); files stay git-clean (exactly one terminator); and a
trailing blank line the writer leaves is mirrored, never doubled.
- load_path: read verbatim (drop the strip)
- save_path: guarded final-newline (drop the unconditional append)
- Prefs::to_toml: ends in a newline again — the guarded save leaves exactly
one, so the prefs file is byte-identical to before and its device-verified
round-trip still holds
- sd_fat spike: payload ends in '\n' so its exact-equality round-trip holds
Git-tracked editor preferences read at boot and toggled live on-device:
- Prefs type (line-based TOML parse/serialize, no crate on xtensa) held on
Editor; firmware reads /sd/repo/.typoena.toml before the first render and
falls back to per-key defaults. Keys: save_on_idle, format_on_save,
line_numbers (bool) + auto_sync (string, schema/default only until v0.7).
- line_numbers applied live (gutter_cols -> 0 when off).
- Palette > command mode toggles the three bools; the list stays open so
several flip in one visit, and :settings opens it directly. Each toggle
applies live and queues Effect::SavePrefs (host atomic-writes the file,
which rides the next :sync).
- save_on_idle honoured host-side as a silent, unformatted idle auto-save.
- to_toml is newline-free; save_path now appends exactly one terminator
unconditionally so buffers round-trip byte-for-byte (trailing blanks kept).
- Firmware 0.4.0 -> 0.5.0; new docs/typoena-toml.md reference. Also refreshes
the slice-3 macroplan status (delete fix was confirmed on device).
Add the POSIX line terminator on save and strip it on load, so files
written by the editor no longer trip git's "No newline at end of file".
Done at the persistence choke point, not in :fmt: the editor buffer is
newline-free by design (rows = #\n + 1), so a trailing '\n' in the buffer
would render a phantom blank last line. save_path adds exactly one '\n'
(guarded against doubling); load_path strips one back off so the buffer
stays newline-free and round-trips byte-stable.
Update the sd_fat spike payload to be newline-free so its byte-identity
round-trip assertion holds under the new normalizing contract.
:enew <name> creates a new file (empty, dirty, added to the palette list);
:delete unlinks the current file via a new Effect::Delete and switches to a
parked buffer or scratch. Scope is read from the path (local/x, repo/x) rather
than a modal prompt, and the /sd prefix is now optional in resolve_path.
On-device testing showed deletions never reached the remote: add_all(["*"])
alone does not stage a removal on this libgit2, so the tree came back unchanged
and the push was a silent no-op. stage_and_commit now runs add_all then
update_all(["*"]) (git add -u) — together git add -A. The :delete snackbar now
confirms the scoped file and flags that a Tracked file is local until :sync.
Cmd-P opens a modal transient palette over the writing column: a bare
fuzzy-search input, the ranked file list, and the selected row in reverse
video. A pure host-testable fuzzy_score (subsequence + word-boundary and
consecutive-run bonuses) ranks results; an in-core MRU floats recently
opened files first and is shared with :e (both route through open_path).
The host feeds the file list once at boot (enumerate_files over /sd/repo
and /sd/local). Ctrl-n/Ctrl-p navigate the list; Enter opens via the same
park/evict path as :e; Esc closes.
Ctrl-n/Ctrl-p also become down/up line motions in Normal and View (vim
CTRL-N/CTRL-P, count-aware), which is why the palette opener is Cmd-P
alone. No `>` prefix on the file input — `>` is reserved for the command
palette (slice 4).
112 editor + 28 keymap tests; the no-git firmware binary builds clean.
Rework the single Effect return into a drained effect queue
(Save{path,scope,contents} / Load / Publish / Pull) so one action can
ask the host for several ordered steps: opening a non-resident file
queues a Save of the outgoing dirty buffer then a Load of the target.
Keep the active buffer's fields inline on Editor and park inactive
buffers in a small LRU (<=3 resident = active + 2); switching back to a
resident buffer restores its caret/scroll/undo without touching the SD.
A dirty parked buffer is saved before eviction, so nothing leaves RAM
unsaved. `:e <path>` opens by prefix (/sd/repo -> Tracked, /sd/local ->
Local); `:sync` is refused in-core for a Local buffer.
Firmware drains the queue to empty each batch (a Load can cascade an
eviction Save) and persistence generalises the atomic save off the
hard-coded notes.md into load_path/save_path.
Also bump the side panel to FONT_9X15 and the `:` command line to
FONT_10X20 for legibility, word-wrapping the snackbar so a long notice
keeps its actionable tail.
A screen refresh that ran while :sync had Wi-Fi + TLS up could fail to
allocate an internal DMA bounce buffer (ESP_ERR_NO_MEM); the error
propagated through ? out of main(), main_task returned from app_main(),
and the editor loop died while the USB and git threads kept running —
keys logged, panel frozen forever.
A paint is idempotent and retryable (the buffer is the source of truth),
so every editor-loop paint now logs and drops the frame instead of
propagating, leaves shown untouched so the next paint repaints the diff,
and forces a full refresh next to re-sync both RAM banks. Same contract
as save_note. The boot-time first render stays fatal on purpose.
Add Effect::Pull and the `:gl` command (fetch + fast-forward only, refuse on
divergence). The editor side is host-tested; the firmware arm is a stub posting
"pull: not wired yet (v0.7)" — the on-device fetch/fast-forward in git_sync is
v0.7 work (only push is wired today).
The pre-commit fetch cost ~6s on the measured cold sync (it did real work
absorbing a foreign push; ~3s + a full TLS handshake even when the remote is
unchanged). Drop it: push onto the current tip first, and only when the remote
rejects the push non-fast-forward do we fetch, mixed-reset onto origin, replay
the note, and retry. The happy path is now a single handshake.
- stage_and_commit extracted (used on the first attempt and the replay).
- reconcile_onto_origin replaces fast_forward_before_commit + fetch_and_integrate.
Mixed reset keeps the just-saved note; the replay lands it on origin's tip.
- Single-writer semantics: a foreign push resolves last-writer-wins instead of
bailing on divergence, so the device never gets stuck. A remote-only added file
it doesn't have would be dropped by the replay — needs the real merge path
(increment B), doesn't arise from this device's own use.
publish_cycle now times wifi assoc, SNTP, TLS setup and publish separately
and logs a one-line total. Cold sync shows all phases; a warm sync reads 0ms
for the first three. Makes the fetch added to publish measurable.
Two robustness fixes surfaced by commit 07d87772 (the first editor-driven
:sync), which shipped three macOS AppleDouble files to typoena-test:
- Staging filter: add_all now runs a per-path callback that skips ._* and
.DS_Store, so Finder/Spotlight cruft on the FAT card never lands in a
commit. Device-level, so it protects every repo without a per-repo
.gitignore.
- Pre-commit fast-forward: publish_once fetches origin and, if the local
branch is behind, fast-forwards to it via a MIXED reset before committing.
This lets the device absorb a foreign push (e.g. the sidecar cleanup) and
fast-forward cleanly instead of stacking a commit on a stale base and
diverging at push time. The mixed reset moves the ref+index but leaves the
working tree, so the just-saved unsynced note isn't lost.
Land the v0.1 editor integration: the git_sync module (libgit2 on the SD
/sd/repo, dedicated 96KB git thread, lazy Wi-Fi, :sync push with
synced/up-to-date/failed snackbars), the boot splash (Spike 9) plus its bin
and justfile recipes, and a power-on→cursor boot-timing log. Also re-syncs
the roadmap/spikes/v0.1-product status and adds the SD hardware reference
photo.
Boot posts "loaded <name>" (the note's filename without suffix); :w and
:sync post "saved" / "save FAILED - retry :w" via Editor::set_notice, so
save state is visible on-device, where there is no serial console.
Route :sync through publish(), whose git-push path is #[cfg(feature =
"git")]. Git code in the firmware binary is now only ever compiled with
--features git, so the default build stays a light editor build with no
git2 crate and (since the justfile only sets LIBGIT2_SRC for git recipes)
no libgit2 component. Feature is off by default; :sync saves locally.
Seed the editor from Storage::load at boot and persist on :w/:sync
through Storage::save. A missing card/repo/unreadable note halts boot
with the reason on the panel rather than starting empty and clobbering
the note on the next save. Save errors are logged, buffer kept in RAM.
The git-push half of :sync stays a TODO — it needs git_sync graduated
into a module on the dedicated git thread; :sync saves for now.
sd_fat is now a thin on-device harness over firmware::persistence instead of
duplicating the mount FFI. It mounts, reports FAT usage + negotiated clock,
loads notes.md (non-destructive), and only runs the write round-trip when
notes.md is empty so a provisioned card is never clobbered. Verified on
hardware 2026-07-11: mount at 10 MHz, 91-byte round-trip byte-identical.
Graduate the proven Spike 3 storage stack into firmware::persistence so the
editor and the spike share one implementation. Storage::mount brings up the
dedicated SPI3 bus (ADR-012) and mounts FAT with format_if_mount_failed=false
(never reformat the user's card). save() does the FAT-safe atomic write
(tmp -> fsync -> unlink -> rename); recover() reconciles a leftover *.tmp at
boot, keeping the committed file when the crash point is ambiguous and only
promoting the tmp when the target was already unlinked. load() caps reads at
256 KiB.
The boot banner, success line, and round-trip payload still said
"shared SPI2" / "shared bus" after the move to SPI3. Cosmetic only —
serial output now matches the dedicated-bus wiring.
The EPD's SpiBusDriver holds an exclusive SPI2 lock for its lifetime, so
an SD on SPI2 is locked out while the panel driver is alive. Give the SD
its own SPI3 (SCK 14, MOSI 15; MISO 13 / CS 10 unchanged) and drop the
now-pointless EPD-CS deselect. See ADR-012.
Flip format_if_mount_failed to true so a fresh exFAT/unformatted bench card is
reformatted to FAT on the device instead of failing — no Mac-side prep for the
spike. Fires only on a filesystem mount failure, not the CMD59 protocol
rejection. The real persistence module must keep this false.
FatFS's f_rename returns FR_EXIST on an existing destination — unlike POSIX
rename(2) it won't replace it, so re-running the round-trip failed at rename.
Unlink the target first (tolerating a missing one for the first save). This
opens a crash window the real persistence module must close with *.tmp boot
recovery (ADR-007).
The per-command sdmmc/sdspi DEBUG logs only existed to diagnose the CMD59
init failure, now resolved (verified 2026-07-11). Remove the build-time log
ceiling and the runtime esp_log_level_set block.
The pure, no-IO editor core can't persist or push, so `handle()` now
returns an `Effect` (None/Save/Publish) that the firmware actions. `:w`
signals Save, `:sync` signals Publish (save then git push); `:wq`/`:x`
alias Save with vim's "quit" half dropped, and the `:q` family stays
absent — an always-on appliance has nothing to quit to. `:fmt` remains
in-core and yields no effect.
The main loop keeps the batch's last effect and matches on it. Actual
SD-write and git-push are the v0.1 gate and still stubbed as log lines.
Route decoded keys through keymap::Composer, so on device `'`+e yields é
(grave/acute/circumflex/diaeresis/tilde, plus ç), and reset it on
keyboard detach. Enabling it is now safe because the editor buffer is
UTF-8-correct. Not verified on the xtensa build in this environment.
Move the editor core out of the firmware crate (pinned to the xtensa
target, so it can't run `cargo test`) into a standalone `editor` crate,
with the panel framebuffer + geometry split into a `display` crate. Both
depend only on embedded-graphics + keymap, so the editor is now
host-buildable and unit-tested (8 characterization tests). Firmware links
them and re-exports the geometry from epd.rs; behaviour is unchanged.
The xtensa firmware build was not verified in this environment.
The single-shot connect_wifi aborted boot on any transient association or
DHCP miss — common right after a reset, when the AP still holds the stale
pre-reset association. Add a bounded retry (5 attempts, 500ms→4s backoff,
disconnect between tries) so those transient failures recover instead of
failing the boot.
Extract the logic into firmware::net (new lib target) and rewire the
wifi_tls / git_push / git_sync spikes to it, removing three duplicated
copies. main.rs reuses this when Wi-Fi lands there instead of adding a
fourth copy.
Adopt the keymap crate (drops the PREV_KEYS/CAPS_USED statics for one
owned Decoder) and harden device teardown:
- #1 UAF: track REPORT_INFLIGHT (set on submit, cleared first thing in
report_cb). Teardown moves to close_device, which pumps client events
until the transfer quiesces before freeing it, and leaks rather than
frees if it never does — a leak is recoverable, a use-after-free is
not. usb_host_transfer_free's return is now checked.
- #3 leaks: a new attach while a device is still open tears the old one
down first; control_request and start_report_polling free the transfer
they allocated on a submit error.
Decode correctness is covered by the keymap tests; the teardown paths
are FFI and still need an on-device hot-plug run to confirm.
Move the mode indicator (and the NO KBD flag above it) down so the
mode sits flush at the bottom-left with a 2 px margin, reclaiming the
dead space that was left below it.
Move the mode + pending-command echo from the panel's top to its
bottom-left (vim `-- INSERT --` style); word count takes the top slot
and the NO KBD flag sits just above the mode. Also drop the
words-this-session field (not wanted) and its session_base_words state.
Carve the panel into a ~60-col writing column and a right-hand side
panel (per CONTEXT.md screen regions); the divider sits right of the
x=396 driver seam. The mode indicator and pending-command echo move
into the panel, while the ':' command line stays in the bottom strip.
The panel also shows word count and words-this-session (a snapshot
refreshed on a typing pause / non-Insert action, never per keystroke,
so ordinary typing keeps the fast windowed refresh) and a "NO KBD"
flag while the keyboard is dropped.
Add a persistent KBD_PRESENT atomic (set on setup, cleared on unplug)
and a keyboard_present() accessor. DEV_GONE is a one-shot detach event
the client loop consumes, not a state the panel can read.
v0.6 affordances plus the v0.4 command-line mechanism, all in editor.rs:
- soft-wrap display lines at word boundaries (layout() no longer cuts
mid-word; a single word wider than the panel still hard-breaks)
- Enter continues Markdown lists: -/*/+ repeat, N. increments, an empty
item exits the list; only at end-of-line, indentation preserved
- heading lines (# .. ######) render faux-bold via a 1px double-strike
(no bold Latin-9 font); checks the logical line so wrapped headings stay bold
- new `:` command mode (Command variant + status-strip echo, Esc/Backspace
cancel) with one command, `:fmt`: align pipe tables (honoring :--: colons),
collapse runs of blank lines to one, strip trailing whitespace; caret lands
on roughly the same line afterwards
std::fs::remove_dir_all deletes race-free via the openat/unlinkat/fdopendir
family; esp-idf's path-based FATFS VFS doesn't implement those, so it fails
with EACCES on the first entry. That looked like a read-only-attribute
problem but wasn't. A plain read_dir -> remove_file/remove_dir walk
(remove_tree) uses only what FATFS supports.
Hardware-verified: a diagnostic recursive delete removed the whole
/spiflash/repo (every entry reported writable, ro=false) then re-cloned +
pushed, twice. Wire remove_tree into both the RECLONE_EACH_BOOT reset and
the stale-clone recovery.
The p_open/p_creat shim (39e1155) stays: it keeps libgit2's objects writable
(ro=false), so neither this delete nor libgit2's own gc/repack/fetch-prune
is ever blocked by a read-only file on FAT.
Debug/recovery knob (ships false): when true, wipe /spiflash/repo and re-clone
every boot instead of opening the persistent clone. Doubles as the validation
harness for the FAT read-only-delete fix — the wipe deletes libgit2-created
objects, which only succeeds once they are writable.
859c478 aborted if /spiflash/repo already existed — libgit2 refuses to
clone into a non-empty dir (code=Exists). Remove a writable stale/partial
clone and re-clone.
Milestone #2A now hardware-verified end to end: clone (boot 1) + persistent
open + fast-forward push (boot 2) against origin/main, on the 96 KB git
thread, min heap 6.8 MB — clone/checkout fit the stack, no bump needed.
Known limit: the remove only works on a *writable* leftover. libgit2 marks
objects/packs read-only and FATFS won't f_unlink a read-only file (EACCES);
POSIX chmod does not clear AM_RDO on esp-idf's FATFS VFS (verified). So
re-cloning over a run that wrote objects needs an AM_RDO-clearing unlink
shim in esp_stubs.c (increment B, also needed for fetch/repack); until then
the fallback is a one-time `espflash erase-region 0x310000 0x400000`.
Milestone #2 increment A: the product's real publish flow, distinct from
git_push's fresh-init-per-boot throwaway branch. open-or-clone a persistent
/spiflash/repo, append to a tracked notes.md, commit on top of the branch,
and fast-forward push over mbedTLS HTTPS+PAT. Reconcile handles up-to-date
and fast-forward; a true divergence (merge commit on FATFS) is deferred to
increment B. Runs on the dedicated 96 KB git thread; gated behind `git`.
Compiles clean; on-device clone/open+fast-forward not yet hardware-verified
(clone/checkout is deeper than the proven init path — the flash may need a
GIT_STACK bump).
libgit2's init→push chain is deeply stack-hungry (~67 KB measured for a
trivial config write; the push is deeper). Spike 7 sized the shared main
task stack at 96 KB for it, which forced the editor build to over-reserve
~80 KB.
Move git_publish onto its own std::thread (GIT_STACK = 96 KB via
Builder::stack_size) and join it from main. This lets
CONFIG_ESP_MAIN_TASK_STACK_SIZE drop back to 12288 — the Spike-6 value
proven to run the editor plus a TLS-on-main handshake — so the editor no
longer pays for git's stack.
Hardware-verified: the push ran off-main (mbedTLS + FATFS) with no panic,
no stack overflow, and no ENOMEM on the 96 KB spawn. This retires the
earlier "time() only works on the main task" misdiagnosis for good — it
was always the default 4 KB pthread stack overflowing, never thread-vs-main.
Retire the cert-check bypass. Embed GitHub's root CAs (github_roots.pem:
USERTrust ECC/RSA + DigiCert G2/Global Root CA), write them to /spiflash/ca.pem
at boot, and point libgit2's mbedTLS stream at them via
GIT_OPT_SET_SSL_CERT_LOCATIONS (CONFIG_MBEDTLS_FS_IO is on). The push callback
now returns CertificatePassthrough instead of CertificateOk, so the http
transport maps it to `is_valid ? 0 : -1` (httpclient.c:805) -- an untrusted or
MITM cert fails the push (fail-closed), no blanket-accept.
Hardware-verified 2026-07-06: `TLS trust store installed`, chain validated
against the embedded USERTrust ECC root, push accepted (branch
device/1783372683 on jcalixte/typoena-test). Roots must be refreshed if GitHub
rotates CAs; a product would prefer esp-idf's bundle via a custom subtransport.
