Compare commits

..

35 Commits

Author SHA1 Message Date
Julien Calixte
e801d2d480 docs(palette): note the two on-device boot-log measurements pending
Walk time after the d_type fix, and the file-list internal-DRAM cost
with the decision rule for interning paths into one PSRAM buffer.
2026-07-13 10:26:25 +02:00
Julien Calixte
a771c3f6de chore(diag): log internal DRAM next to PSRAM-dominated heap totals
"free heap 5.8MB" in the crash log was PSRAM and masked the actual
internal-RAM exhaustion. git_sync's publish/commit/push lines now
carry the MALLOC_CAP_INTERNAL reading, and boot brackets the palette
file-list build with it — small Strings are forced internal by the
16 KB SPIRAM malloc threshold, so the 1098-path list is a suspected
~60-70 KB DRAM consumer competing with Wi-Fi/TLS; the log decides
whether interning into one PSRAM buffer is worth it.
2026-07-13 10:25:25 +02:00
Julien Calixte
4c92b0dddc fix(sync): move mbedTLS allocations to PSRAM
esp-idf's default is internal-RAM-only, and a TLS connection needs
~33 KB of it (two ~17 KB I/O buffers + contexts) at the exact moment
:sync pushes — with Wi-Fi, USB host, the editor and libgit2 resident,
mbedtls_ssl_setup failed there on the first real-repo push (and its
failure path then hit the stream double-free fixed in the previous
commit). TLS buffers are CPU-only data, so PSRAM is safe; the
handshake is network-bound.
2026-07-13 10:25:25 +02:00
Julien Calixte
af5b41a104 fix(sync): patch the mbedtls stream double-free that reset the chip
Upstream libgit2 v1.9.4 bug: mbedtls_stream_wrap's ssl_setup error
path closes and frees the caller's socket stream, then
git_mbedtls_stream_new frees it again — and wrap's other error paths
don't free it at all, so callers can't compensate either way. When
ssl_setup failed on the device (internal-RAM exhaustion during the
first real-repo push), the double git__free tripped tlsf ("block
already marked as free") and reset the chip mid-:sync.

Fixed the way esp_map.c replaces map.c: the vendored streams/mbedtls.c
is excluded from the component and esp_mbedtls_stream.c (verbatim copy
+ one-hunk fix: out_err leaves st->io to the caller, and frees the
leaked ssl struct) takes its place. Keep the copy in lockstep on
submodule bumps; worth reporting upstream.
2026-07-13 10:25:12 +02:00
Julien Calixte
2660a3e9dd perf(palette): trust dirent d_type instead of a per-entry stat
The per-entry metadata() call makes FatFS re-walk the directory by
path every time — ~32ms/file on the card, 35s for a 1098-file tree.
esp-idf's FAT VFS always fills d_type (DT_DIR/DT_REG from the FILINFO
readdir already holds, never unknown) and Rust std maps file_type()
onto it stat-free, so the walk is now one readdir pass per directory.
2026-07-13 01:05:10 +02:00
Julien Calixte
79fad4689c docs(palette): amend v0.5 with the recursive walk and search threshold 2026-07-13 00:55:34 +02:00
Julien Calixte
98fc817b3f feat(palette): show recents only until the query reaches two chars
With the file list now a recursive walk of the whole card, an unranked
short query pages through too much. Below PALETTE_MIN_QUERY (2) the
result list is the MRU only, keeping Cmd-P + Enter quick-switch one
keystroke away; two chars reveal the full fuzzy-ranked list. Commands
(>) and snippets ($) are short curated lists and keep matching from
the first char.
2026-07-13 00:55:25 +02:00
Julien Calixte
d306caacf7 feat(palette): walk the card recursively for the file list
A nested repo (jcalixte/notes) showed only its top-level files in the
palette. Dot entries are skipped at every level so .git is never
descended; each directory is read fully before recursing so a single
FatFS dir handle is open at a time; depth capped at 8. The boot walk
logs its file count and duration to keep the FAT dir-IO cost visible.
2026-07-13 00:55:16 +02:00
Julien Calixte
2166b932b6 docs(sync): update the wiring section to how-it-landed
The merged plan section now records the shipped design: single path
set (working tree as truth) instead of {changed, deleted}, the dirty
journal, stranded-commit recovery, radio-free up-to-date, and the
soft-reset reconcile with its carried-files side win.
2026-07-13 00:53:50 +02:00
Julien Calixte
02b7ed88b6 fix(provisioning): rewrite the card origin to HTTPS at load
The device's libgit2 has no SSH transport (HTTPS+PAT over mbedTLS
only), but _load-repo copied the desktop clone's origin verbatim — an
SSH-shaped URL fails every on-device push/fetch with "unsupported URL
protocol" (first real-repo :sync, 2026-07-13). Derive the HTTPS
equivalent for the card copy (git@host:path and ssh://git@host/path);
the source clone keeps its own URL. Warns on non-github hosts, since
the embedded trust store carries GitHub roots only.
2026-07-13 00:53:50 +02:00
Julien Calixte
a5edaed810 feat(sync): commit by splicing journaled dirty paths onto HEAD
The index pipeline (add_all → index.write → write_tree) is O(N_tree)
and cannot commit the real 1179-file / 570 MB-pack clone (index.write
measured up to 611 s on FAT's racy-clean re-hash). stage_and_commit is
now an O(depth) TreeBuilder splice of exactly the paths the editor
saved or deleted — ~2-2.8 s on the real clone — with the working tree
as source of truth (existing file → insert, missing → remove).

Storage records those repo-relative paths on every save/delete and
journals them to /sd/.typoena-dirty (atomic, only on growth), so a
power pull can't strand a saved-but-unpublished note now that nothing
walks the tree. take_dirty → publish_succeeded/publish_failed settles
each publish's snapshot from the UI outcome handler.

Also required by / discovered with the splice:
- mwindow opts at git-service start (32-bit defaults would OOM PSRAM
  on the first real-pack access; bench-proven 256 KB / 4 MB).
- 16-FD mount for git builds (libgit2 holds pack+idx descriptors open;
  the editor's 4-FD budget overruns).
- reconcile is a soft reset (no index to reset anymore); side win: a
  remote-only added file is carried by the replay instead of dropped.
- stranded-commit recovery: tree-unchanged now pushes anyway when
  origin/<branch> lacks HEAD (a commit whose push failed used to be
  silently never retried).
- radio-free up-to-date: empty dirty set + origin at HEAD answers
  without bringing Wi-Fi up.
- try_push splits ref rejection (reconcilable) from transport failure
  (surfaced directly) — the first on-device run burned a doomed
  reconcile on "unsupported URL protocol" and hid the cause.

Deliberate behavior change: files changed on the card outside the
editor are never committed anymore (also retires the macOS-cruft
filter). Trail: docs/tradeoff-curves/sync-commit-staging.md.
2026-07-13 00:53:39 +02:00
Julien Calixte
e86a3b8254 docs(sync): merge the commit handoff into the staging tradeoff curve
The bench phase the handoff was written for is closed (splice benched,
fast-seek landed, cache removed, decision made), so the note's live half —
the firmware plumbing plan — moves into sync-commit-staging.md as "The fix —
wiring the O(depth) splice into the firmware", reconciled to the run-5 state.
The duplicated TL;DR/measurement summaries are dropped in favor of the
curve's own trail; inbound references (notes index, git_bench comments)
now point at the curve.
2026-07-13 00:29:07 +02:00
Julien Calixte
9309f3f239 docs(sync): record the esp_map v2 verdict and cache removal
Run 4: memory discipline verified (1833 KB flat, no OOM) but 0 hits
with the small maps demonstrably retained — window-repetition theory
refuted, sub-second bar failed (splice 2.83 s cold / ~2 s steady).
Decision: wire the splice in anyway (~9-10 s cold real-repo :sync vs
611 s/OOM for every alternative). Run 5: cache removal confirmed
free; the ~1.85 MB "resident" was mwindow's live window set, which
makes the mwindow opts in shipping git_sync.rs load-bearing.
2026-07-13 00:20:12 +02:00
Julien Calixte
1a5e1736f5 refactor(sync): remove the never-hit mmap window cache
Four instrumented real-repo bench runs scored 0 cache hits: libgit2's
mwindow layer reuses its open windows, so only genuinely new
(offset, len) ranges ever reach p_mmap — there is no repetition to
cache. The 7.4 MB OOM the v2 discipline fixed was caused by the cache
itself holding buffers past p_munmap; plain free-at-munmap is honest
with MWINDOW_MAPPED_LIMIT by construction. Run 5 confirmed removal is
I/O-neutral on device (byte-identical read pattern, warm splice 1953
vs 1949 ms) and even 15 reads better — v2's low-water eviction was
fighting mwindow. Stats counters kept to spot any future workload
that does repeat ranges.
2026-07-13 00:20:02 +02:00
Julien Calixte
ce3204a350 docs(sync): record fast-seek root cause and splice bench trail
Splice bench (6.5 s, O(depth) shape confirmed), FatFS cluster-chain
root cause with elm-chan/ESP-IDF references, on-device fast-seek
verification, second localization round (strict-creation refuted,
read_header 470 ms), esp_map v2 design, and the two firmware plumbing
gaps (mwindow opts + 16-FD mount).
2026-07-13 00:02:55 +02:00
Julien Calixte
dd3d70cbec perf(sync): cache small pack windows and evict mmap cache on munmap
Admission keyed on file size (>= 1 MB) instead of map length: the hot
set is the small repeated maps (pack trailer, idx fanout, delta bases)
that the 64 KB floor excluded — 0 hits across three real-repo runs.
Small mutable working-tree files stay uncacheable. p_munmap now evicts
unreferenced entries to a 2 MB low-water mark so released windows are
actually returned to git__malloc (fixes the 7.4 MB resident / 508 KB
heap zlib OOM and keeps MWINDOW_MAPPED_LIMIT honest).
2026-07-13 00:02:54 +02:00
Julien Calixte
2c1c590c9e perf(sync): enable FatFS fast seek for pack reads
Without CONFIG_FATFS_USE_FASTSEEK every long/backward lseek in the
263 MB pack walks the FAT cluster chain over SPI (~190 ms), and libgit2
pays ~8 such seeks per loose-object write. The CLMT makes lseek O(1)
for read-mode files; verified on device: far seek 198.7 -> 20.4 ms,
splice commit 6.5 -> 2.8 s.
2026-07-13 00:02:42 +02:00
Julien Calixte
2c24ece3a5 feat(bench): add packfile long-seek op to sd_bench
Reads 4 KB at the start vs the end of the repo's largest pack (skipping
macOS ._ sidecars). Proved the ~1.5 s/loose-object cost was FatFS
walking the FAT cluster chain on every long/backward lseek: 5.8 ms at
offset 0 vs 198.7 ms at the end of the 263 MB pack.
2026-07-13 00:02:42 +02:00
Julien Calixte
da8ca7d8d2 feat(bench): bench the O(depth) TreeBuilder splice and odb probes
Adds the splice prototype (patch one path onto HEAD's tree, O(depth))
as git_bench's headline op, run first so its cold number survives the
index ops' OOM, plus odb.read_header/odb.exists probes and strict-off
re-benches that localized the residual cost and refuted the
strict-object-creation theory.
2026-07-13 00:02:29 +02:00
Julien Calixte
dade5e6bb3 docs(sync): record real-repo commit bench and TreeBuilder handoff
The real jcalixte/notes clone (570 MB pack, 1179 files) proved both index
strategies are O(N_tree) and unshippable: index.write re-hashes the tree
(611 s), and the index-free read_tree is 77 s cold and OOMs the mmap cache
(zlib crash). Record the run in the tradeoff curve, revise the verdict to an
O(depth) TreeBuilder walk, and add a handoff note with the design, firmware
call sites, and bench steps for the next session.
2026-07-12 15:04:59 +02:00
Julien Calixte
a6f52df8b6 perf(sync): bench the index-free commit path and mmap cache
Replace the index.write()/write_tree ops (which re-hash the whole working
tree on a fresh FAT clone -- up to 611 s on the real repo) with the
index-free candidate: Index::new + read_tree(HEAD) + blob + write_tree_to.
Add per-op mmap-cache + free-heap logging (via esp_map_stats), announce each
op before it runs so a hang reveals the culprit, mount_for_git for the FD
budget, tune the mwindow limits, and drop N 10 -> 3 for the slow real clone.
2026-07-12 15:04:52 +02:00
Julien Calixte
6c5e666f4b feat(persistence): add mount_for_git with a larger open-file budget
libgit2 keeps the pack, .idx and commit-graph descriptors open for the
repo's lifetime and opens loose objects on top, so a read_tree walk overruns
the editor's tight 4-FD budget with "no free file descriptors". Add
mount_for_git (16 FDs, matching the flash-FAT git binaries) alongside the
editor's mount; both route through a shared mount_with_max_files.
2026-07-12 15:04:44 +02:00
Julien Calixte
c5e119f6be perf(git): cache emulated pack mmap to avoid re-reading the pack
p_mmap emulated a mapping by malloc+read()ing the range from SD on every
call. libgit2 re-hits the pack idx/windows on every object write (via
git_odb__freshen -> git_odb_refresh), so a commit re-read pack bytes over
SPI repeatedly -- ~500ms-1.1s/op on a small repo, far worse on the real
570 MB pack.

Cache read-only mappings >= 64 KB (pack idx/windows, commit-graph, midx,
packed-refs -- all immutable on this device) in a refcounted, PSRAM-backed
slot table keyed by (dev, ino, size, mtime, offset, len), LRU-evicted under
a soft cap. Small mutable working-tree maps (diff_file.c) fall below the
floor and are never served stale.
2026-07-12 15:04:37 +02:00
Julien Calixte
456c4c43e7 perf(sync): instrument and benchmark commit-staging latency
Break the stage+commit window into sub-phases (FAT working-tree walk vs
object writes) via `commit split —` log lines, and add two micro-benchmarks
(sd_bench for SD/FAT primitive ops, git_bench for libgit2 object overhead)
with justfile recipes. Documents the walk-vs-writes cost model in
tradeoff-curves/sync-commit-staging.md to decide whether explicit-path
staging over the editor's dirty set is worth replacing add_all(["*"]).
2026-07-12 12:24:50 +02:00
Julien Calixte
beb11eda5e chore(firmware): skip legacy-I2C conflict check
esp-idf-hal always compiles its i2c.rs, which binds the old driver/i2c.h
API, so ESP-IDF links the legacy driver's constructor into every image and
prints its deprecation warning at boot. Typoena uses no I2C (EPD/SD are SPI,
keyboard is USB), so the warning is pure noise; this flag compiles the
constructor (and its conflict-abort check) out.
2026-07-12 12:23:10 +02:00
Julien Calixte
88bb2b99cf feat(provisioning): pull source clone before copying to the card
`just load` / `just init` now fast-forward the source repo from its
origin before rsyncing to /sd/repo, so the card picks up notes the
device already pushed. Fast-forward-only and best-effort: dirty tree,
diverged history, detached HEAD, or offline warns and copies the
current on-disk state rather than aborting. TW_NO_PULL=1 skips it.
2026-07-12 10:59:05 +02:00
Julien Calixte
69843086f9 chore(firmware): flash at 921600 baud
Default espflash baud (115200) made a full flash take ~2-3 min of
serial transfer. Bump the cargo runner and the two direct-espflash
recipes (flash-git-push, flash-git-sync) to 921600. Monitor recipes
keep 115200 to match the device UART log rate.
2026-07-12 10:49:16 +02:00
Julien Calixte
45664fc956 feat: add glyphs 2026-07-12 10:43:41 +02:00
Julien Calixte
56f5d18bd8 docs(macroplan): record v0.6 markdown complete in core (0.6.0)
Mark v0.6 delivered (core, host-tested; on-device gate pending): the
snippet engine, the Cmd-P/>/$ palette split retiring :e, and the just
init catalog. Note the →/≠ ISO-8859-15 caveat.
2026-07-12 10:42:56 +02:00
Julien Calixte
767742ba12 docs(v0.6): mark boot-read + catalog done, finalise the catalog list
Record the firmware boot-read (serde_json builds for xtensa) and the
just-init catalog as delivered, and replace the proposed French catalog
with the finalised 17-snippet English set (three groups). Note the
→/≠ base-font caveat and the no-clobber seeding guard.
2026-07-12 10:36:58 +02:00
Julien Calixte
85cbeceea9 feat(provisioning): seed a snippet catalog + starter prefs on just init
Add a curated snippet catalog (firmware/snippets-catalog/, three opt-in
groups: Symbols, Structure, Prose — 17 snippets, English prefixes,
translated from the source Zed set) and a _seed-configs step in `just
init` that jq-merges the chosen groups into repo/.typoena.snippets.json
and writes a starter repo/.typoena.toml. Seeds only files that are
absent, so re-running init never clobbers a synced library.
2026-07-12 10:36:46 +02:00
Julien Calixte
8527f75bf8 feat(firmware): read .typoena.snippets.json at boot into set_snippets
Mirror the prefs boot-read: load the git-tracked snippet library from the
SD repo, parse it with Snippets::parse, and hand it to the editor before
the first render. Missing/unreadable/malformed is non-fatal (no snippets,
editor runs). Verified serde_json builds for xtensa (cargo check, 0.6.0).
2026-07-12 10:30:02 +02:00
Julien Calixte
5a076e3226 docs(v0.6): mark the palette generalisation delivered in core
Record the > command registry, one-shot/two-step dispatch, and the :e
retirement as done in core; note the ISO-8859-15 label constraints
(ASCII `...`, `»` hint marker).
2026-07-12 10:22:24 +02:00
Julien Calixte
c969d3e051 feat(editor): generalise the > palette into a command registry
Turn the > palette from a settings-only list into an action registry
dispatched by PaletteCmd::kind: toggles stay open, one-shots (format,
publish) run and close, and the parameterised `new file...` morphs the
palette into a filename input step. Share run_publish between :sync and
the publish command, and retire :e (bare Cmd-P opens files).
2026-07-12 10:22:17 +02:00
Julien Calixte
baf9715570 feat(editor): hint the matching snippet in the panel on typing pause
Snapshot the snippet name inline Tab would expand into a panel field in
refresh_stats, so it rides the typing-pause throttle rather than
repainting per keystroke. Draw it as `» name` on the row above the mode
line (Latin-9 has no tab glyph), sharing the slot with the NO KBD flag,
which can't co-occur since the hint means you're typing.
2026-07-12 10:09:43 +02:00
25 changed files with 3807 additions and 313 deletions

341
display/src/glyphs.rs Normal file
View File

@@ -0,0 +1,341 @@
//! Extra glyphs the ISO-8859-15 render font lacks.
//!
//! The buffer is UTF-8, so it can hold codepoints outside Latin-9. The
//! `embedded-graphics` `iso_8859_15` font draws those as a fallback box. This
//! module carries hand-authored 10×20 bitmaps for a curated set of common
//! single-width characters — typographic punctuation that arrives in imported
//! prose (curly quotes, en/em dashes, ellipsis, bullet) and the math symbols the
//! user's Markdown snippets insert (→ ≠ Σ). [`Editor::draw`] overlays these over
//! the fallback box after the font has laid out the line.
//!
//! Every glyph is exactly one 10×20 cell (matching the body `FONT_10X20`, i.e.
//! `editor::CW`×`editor::CH`), so the editor's 1-char = 1-cell layout invariant
//! is untouched. A row is a 10-bit pattern: the leftmost column (x = 0) is the
//! high bit, so a binary literal reads left-to-right as the pixels of that row.
//! Rows are top (y = 0) to bottom (y = 19); 1 = ink.
use crate::Frame;
use embedded_graphics::pixelcolor::BinaryColor;
use embedded_graphics::prelude::*;
const GLYPH_W: usize = 10;
const GLYPH_H: usize = 20;
/// A 10×20 1-bit glyph: one `u16` per row, low `GLYPH_W` bits used, high bit =
/// leftmost pixel.
pub type Glyph = [u16; GLYPH_H];
// → U+2192 RIGHTWARDS ARROW: a horizontal shaft with a `>` head at the right.
#[rustfmt::skip]
const ARROW_RIGHT: Glyph = [
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000010000,
0b0000001000,
0b0000000100,
0b0000000010,
0b0111111111,
0b0111111111,
0b0000000010,
0b0000000100,
0b0000001000,
0b0000010000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
];
// ≠ U+2260 NOT EQUAL TO: two equals bars with a slash through them.
#[rustfmt::skip]
const NOT_EQUAL: Glyph = [
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000010,
0b0000000100,
0b0000000100,
0b0000001000,
0b0111111110,
0b0000010000,
0b0000010000,
0b0000100000,
0b0111111110,
0b0001000000,
0b0001000000,
0b0010000000,
0b0010000000,
0b0000000000,
0b0000000000,
0b0000000000,
];
// Σ U+03A3 GREEK CAPITAL LETTER SIGMA: top & bottom bars with the diagonals
// meeting at a vertex on the centre-right.
#[rustfmt::skip]
const SIGMA: Glyph = [
0b0000000000,
0b0000000000,
0b0000000000,
0b0111111110,
0b0100000000,
0b0010000000,
0b0001000000,
0b0000100000,
0b0000010000,
0b0000100000,
0b0001000000,
0b0010000000,
0b0100000000,
0b0100000000,
0b0111111110,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
];
// • U+2022 BULLET: a filled dot at mid-height.
#[rustfmt::skip]
const BULLET: Glyph = [
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0001111000,
0b0011111100,
0b0011111100,
0b0011111100,
0b0001111000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
];
// … U+2026 HORIZONTAL ELLIPSIS: three dots on the baseline.
#[rustfmt::skip]
const ELLIPSIS: Glyph = [
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0110110110,
0b0110110110,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
];
// U+2013 EN DASH: a short mid-height bar.
#[rustfmt::skip]
const EN_DASH: Glyph = [
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0011111100,
0b0011111100,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
];
// — U+2014 EM DASH: a full-width mid-height bar.
#[rustfmt::skip]
const EM_DASH: Glyph = [
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b1111111111,
0b1111111111,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
];
// ' U+2018 LEFT SINGLE QUOTATION MARK: raised, tail up (opening).
#[rustfmt::skip]
const LEFT_SINGLE_QUOTE: Glyph = [
0b0000000000,
0b0000000000,
0b0000100000,
0b0001100000,
0b0001100000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
];
// ' U+2019 RIGHT SINGLE QUOTATION MARK / apostrophe: raised, tail down (closing).
#[rustfmt::skip]
const RIGHT_SINGLE_QUOTE: Glyph = [
0b0000000000,
0b0000000000,
0b0001100000,
0b0001100000,
0b0000100000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
];
// " U+201C LEFT DOUBLE QUOTATION MARK: two opening quotes.
#[rustfmt::skip]
const LEFT_DOUBLE_QUOTE: Glyph = [
0b0000000000,
0b0000000000,
0b0010010000,
0b0011011000,
0b0011011000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
];
// " U+201D RIGHT DOUBLE QUOTATION MARK: two closing quotes.
#[rustfmt::skip]
const RIGHT_DOUBLE_QUOTE: Glyph = [
0b0000000000,
0b0000000000,
0b0011011000,
0b0011011000,
0b0010010000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
0b0000000000,
];
/// The 10×20 bitmap for a character the ISO-8859-15 font can't draw, or `None`
/// if the base font already covers it (all ASCII and Latin-9, including `œ €`).
pub fn extra_glyph(c: char) -> Option<&'static Glyph> {
Some(match c {
'\u{2192}' => &ARROW_RIGHT, // →
'\u{2260}' => &NOT_EQUAL, // ≠
'\u{03A3}' => &SIGMA, // Σ
'\u{2022}' => &BULLET, // •
'\u{2026}' => &ELLIPSIS, // …
'\u{2013}' => &EN_DASH, //
'\u{2014}' => &EM_DASH, // —
'\u{2018}' => &LEFT_SINGLE_QUOTE, // '
'\u{2019}' => &RIGHT_SINGLE_QUOTE, // '
'\u{201C}' => &LEFT_DOUBLE_QUOTE, // "
'\u{201D}' => &RIGHT_DOUBLE_QUOTE, // "
_ => return None,
})
}
/// Paint a glyph over one 10×20 cell at `(x, y)`. Fills the whole cell — set
/// bits in `ink`, unset bits in `ink.invert()` — so it fully overwrites whatever
/// the base font drew there (e.g. the fallback box). Pass `ink = On` for a
/// normal black-on-white cell, `ink = Off` for a reverse-video cell (a white
/// glyph over the black selection/caret fill).
pub fn blit_glyph(f: &mut Frame, x: i32, y: i32, g: &Glyph, ink: BinaryColor) {
let bg = ink.invert();
let pixels = g.iter().enumerate().flat_map(move |(row, &bits)| {
(0..GLYPH_W).map(move |c| {
let on = (bits >> (GLYPH_W - 1 - c)) & 1 == 1;
Pixel(
Point::new(x + c as i32, y + row as i32),
if on { ink } else { bg },
)
})
});
// Frame's DrawTarget error is Infallible.
let _ = f.draw_iter(pixels);
}

View File

@@ -13,6 +13,9 @@ use embedded_graphics::prelude::*;
use embedded_graphics::primitives::{Circle, PrimitiveStyleBuilder};
use embedded_graphics::text::{Alignment, Baseline, Text, TextStyleBuilder};
mod glyphs;
pub use glyphs::{blit_glyph, extra_glyph, Glyph};
pub const WIDTH: u16 = 792;
pub const HEIGHT: u16 = 272;

View File

@@ -61,8 +61,8 @@ learning = "Delivered 2026-07-12, well ahead of the 2026-09-28 baseline, and ful
name = "v0.6 markdown"
start = 2026-09-28
original = 2026-10-12
status = "on-track"
note = "Render affordances done early; snippet engine (added 2026-07-08) remains."
delivered = 2026-07-12
learning = "Core complete 2026-07-12, ~92 days ahead of the 2026-10-12 baseline, host-tested (187 editor tests). The snippet feature was reshaped 2026-07-08→07-12 from a hard-coded table into a git-synced, Zed-compatible .typoena.snippets.json library: a forward-only tab-stop session ($1..$n/$0, ${n:label} stripped to $n) driven by two surfaces — inline Tab-expansion in Insert and a $ palette launcher — plus a quiet pause hint in the side panel. The Cmd-P palette generalised into a verb split: bare = files, > = a real command registry (toggles stay open, one-shots format/publish close, the parameterised `new file` two-step), $ = snippets — retiring :e. Firmware bumped 0.5.0→0.6.0; the boot-read of the library was confirmed to build for xtensa (serde_json, the one new dep — cargo check passes). `just init` now seeds a curated 17-snippet catalog (three opt-in groups). On-device smoke-test still pending (pure editor-core + a mirror of the proven prefs boot-read, low risk). Known caveat: two symbols the catalog inserts (arrow →, neq ≠) are outside ISO-8859-15, so they store/sync correctly but need a display-layer glyph overlay (in flight) to draw on the panel; the other 15 render on the stock font."
[[feature]]
name = "v0.7 search + git"
@@ -125,6 +125,18 @@ on-device confirmed: the Cmd-P fuzzy palette, `:e`/`:enew`/delete across the
(boot-read plus a stay-open palette `>` command mode + `:settings` that edits them
live and syncs the change). Descoped to later: explicit buffer close, the
grey-Publish-in-Local panel cue, and the multi-file publish count.
**v0.6 Markdown is COMPLETE in core 2026-07-12** (firmware **0.6.0**), host-tested
(187 editor tests), on-device smoke-test pending. The render affordances (heading
bold, list continuation, soft-wrap) were done early; the headline is the
**snippet engine** — a forward-only tab-stop session reached both inline (type a
prefix + Tab in Insert) and from a **`$` palette** launcher, fed by a git-synced,
Zed-compatible `.typoena.snippets.json` read at boot (serde_json, confirmed to
build for xtensa). The `Cmd-P` palette **generalised** into a verb split — bare =
files, `>` = a command registry (toggles stay open, `format`/`publish` one-shots
close, a two-step `new file`), `$` = snippets — which **retired `:e`**. `just init`
seeds a curated 17-snippet catalog (Symbols · Structure · Prose, opt-in). One
caveat: `→`/`≠` sit outside ISO-8859-15 and need a display-layer glyph overlay
(in flight) to render on the panel; the other 15 draw on the stock font.
Marks: `[x]` done in core · `[~]` partially done · `[ ]` not started. An
inline `(✓)` marks the done half of a split item.
@@ -176,11 +188,15 @@ with a palette `>` command mode + `:settings`.
**DELIVERED 2026-07-12** (firmware 0.5.0), fully on-device confirmed.
Detail: [v0.5-palette-and-multi-file.md](v0.5-palette-and-multi-file.md).
## v0.6 — Markdown affordances — [~]
## v0.6 — Markdown affordances — [x]
Heading bolding, list continuation, and soft-wrap are done; the trigger-driven
snippet engine (net-new scope, added 2026-07-08) remains.
Detail: [v0.6-markdown.md](v0.6-markdown.md).
Heading bolding, list continuation, and soft-wrap, plus the trigger-driven
snippet engine (net-new scope, added 2026-07-08): a tab-stop session reached
inline (prefix + Tab) and from the `$` palette, fed by a git-synced,
Zed-compatible `.typoena.snippets.json`; the `Cmd-P` palette generalised into a
`files`/`>` commands/`$` snippets split (retiring `:e`); and a `just init`
catalog. **COMPLETE in core 2026-07-12** (firmware 0.6.0), on-device smoke-test
pending. Detail: [v0.6-markdown.md](v0.6-markdown.md).
## v0.7 — Search + better git — [~]

View File

@@ -75,10 +75,12 @@ The big rocks are physics or protocol, not slack:
- **TLS handshake ~2.4 s** and **push negotiate/upload ~4.4 s** are inherent to
libgit2-over-mbedTLS on this part; the payload is tiny, so there's little to
shave.
- **stage + commit ~3.1 s** is the one soft spot: staging `notes.md` directly
instead of `add_all(["*"])` would skip the SD/FAT tree walk (likely →
sub-second), at the cost of the file-agnostic design that a future multi-file
publish wants. Deferred, on purpose.
- **stage + commit ~3.1 s** is the one soft spot: staging over the editor's dirty
set (`add_path`) instead of `add_all(["*"])` would skip the SD/FAT tree walk
(likely → sub-second) *without* losing multi-file — the dirty set is the file
list. Whether the walk actually dominates the ~4 s commit is now being measured
by the `commit split —` log line; the cost model and the rule it decides live in
[`../tradeoff-curves/sync-commit-staging.md`](../tradeoff-curves/sync-commit-staging.md).
**Conclusion:** ~16 s cold / ~10 s warm is close to the floor for "commit to FAT +
one TLS push over Wi-Fi with a fresh clock." It reads as slow only if you wait on

View File

@@ -11,3 +11,4 @@
| --- | --- |
| [`wifi-auto-sync.md`](wifi-auto-sync.md) | `auto_sync` interval vs Wi-Fi energy (a `1/T` hyperbola) — why the default is 10 min and opportunistic, not a wall-clock timer. |
| [`epd-refresh-latency.md`](epd-refresh-latency.md) | E-ink refresh latency vs rows driven — the full / full-area-partial / windowed-Y cost model behind typing responsiveness and the boot splash→editor swap. |
| [`sync-commit-staging.md`](sync-commit-staging.md) | Commit-staging strategy vs working-tree size — RESOLVED: every index-based path is O(N_tree) and fails on the real repo (611 s / OOM); the O(depth) TreeBuilder splice is benched at ~22.8 s and ships. Holds the full measurement trail plus the firmware plumbing plan (merged from the retired handoff note). |

View File

@@ -0,0 +1,638 @@
# Commit-staging cost vs working-tree size
> **Decision (RESOLVED 2026-07-12, real-repo bench):** neither `add_all(["*"])`
> nor an index-free `read_tree`+`write_tree` is viable on the real
> `jcalixte/notes` clone — **both are O(N_tree)** and blow up on the 570 MB pack /
> 1179-file tree (611 s hash one end, 77 s tree-read + OOM the other; see
> [Real-repo run](#real-repo-run-2026-07-12-jcalixtenotes-570-mb-pack--the-index-is-the-wrong-primitive)
> below). The commit must be rebuilt with an **O(depth) TreeBuilder walk** —
> patch only the edited file's ancestor subtree chain onto HEAD's tree, never
> materialise all 1179 entries.
>
> **Final state (2026-07-13, after four localization rounds):** the splice walk
> is benched and the block is lifted. It first measured 6.5 s (each loose-object
> write cost ~1.5 s); the root cause was FatFS's lseek cluster-chain walk, and
> the fast-seek fix cut it to **2.8 s cold / ~2 s steady-state**. The `esp_map.c`
> window cache was **removed entirely** (0 hits across four instrumented runs —
> `mwindow` absorbs any true repetition above `p_mmap`). The sub-second bar
> failed, but ~22.8 s against 611 s/OOM for every alternative ships: a full
> cold real-repo `:sync` lands at ~910 s. **Decision: wire the splice in —
> done 2026-07-13** ([how it landed](#the-fix--wiring-the-odepth-splice-into-the-firmware),
> merged from the retired `notes/sync-commit-handoff.md`; on-device `:sync`
> against the real repo still pending). Shrinking the repo is **not** an option
> ([`../notes/git-sync-images-and-repo-size.md`](../notes/git-sync-images-and-repo-size.md):
> the images are load-bearing for another app), which is exactly why the O(depth)
> mechanism is the only lever left. This note records the cost model and the full
> measurement trail that got here.
>
> Tradeoff-curves index: [`README.md`](README.md). Docs index:
> [`../README.md`](../README.md). Where the whole sync goes:
> [`../notes/sync-latency.md`](../notes/sync-latency.md). Sibling curve on the
> radio cost of *how often* we sync: [`wifi-auto-sync.md`](wifi-auto-sync.md).
## The model
`:sync` commits the working tree on the SD/FAT card before it pushes. The commit
is two kinds of work against the card over SPI (10 MHz today, ADR-012):
```
stage write
─────────────────────────────── ─────────────────────────────
add_all(["*"]) + update_all(["*"]) index.write + write_tree + commit-obj
→ stat() every file in the tree, → serialise the index and three loose
hash the ones whose stat moved objects, each a FAT create+write+fsync
cost ∝ tree size (O(N_tree)) cost ∝ churn (O(N_changed)) + fixed
```
The two have different curves against **N = files in `/sd/repo`**:
- **Walk** rises with N. `add_all(["*"])` visits the whole working tree every
sync regardless of how little changed, and each visit is a FAT `stat` (and a
re-hash when the entry looks dirty) over SPI. This is the term explicit-path
staging removes: the editor already knows which buffers are dirty and which
were `:delete`d, so `index.add_path(p)` / `index.remove_path(p)` over that set
touches `N_changed` files (≈1 for a writing appliance), not `N`.
- **Write** is flat in N. A text commit is the index + a blob + a tree + a commit
object — a handful of small FAT writes whose cost is set by SPI clock and
`fsync`, not by tree size. Explicit-path staging cannot shave this; only a
faster card bus (SD 10 → 20 MHz on a clean PCB, `persistence.rs`) does.
```
Commit latency vs working-tree size two staging strategies
ms
| walk-all: add_all(["*"])
4000 | . * stat()s every file in the
| . * tree each sync → O(N)
| . *
3000 | . *
| . *
2000 | . *
| *
1000 |····································· explicit-path: add_path(dirty)
| FAT object-write floor → O(churn); flat in N. The gap
0 +----+----+----+----+----+----+----+---→ up to walk-all is the avoidable
10 50 100 200 400 800 1179 per-sync tree walk.
└── jcalixte/notes today (N files)
```
The gap between the lines at a given N is exactly what switching buys, and it
**grows without bound** as the notes tree fills.
### The real operating point (measured 2026-07-12, `jcalixte/notes`)
The device syncs into a clone of the actual notes repo, not a `notes.md` toy. Its
working tree is **not small**:
| | count | working-tree bytes |
| --- | ---: | ---: |
| Markdown (`.md`) | 875 | ~1.5 MB |
| Images (png/jpg/webp/bmp/gif) | ~260 | **~150 MB** |
| Other (json/ts/pdf/…) | ~44 | ~20 MB |
| **Total (N)** | **1179 files, 158 dirs** | **~170 MB** |
| `.git` history | | ~570 MB |
So `add_all(["*"])` walks **1179 files across 158 directories every sync** — and
~260 of them are images that a text edit never changes. That does two things the
toy-repo baseline hides:
1. **The walk term is large and paid on every sync** — 1179 `stat`s + 158 dir
reads over SPI, for a one-line note change. This is the O(N) cost the curve
above predicts, at N ≈ 1179 rather than N ≈ 2.
2. **Re-hash risk.** libgit2 decides a file is unchanged from `stat` metadata
(mtime/size). FAT's coarse mtime and lack of a stable inode can make entries
look racy, forcing a content re-hash. If even a slice of the ~150 MB of images
gets re-hashed over a 10 MHz SPI bus, the commit balloons far past 4 s. The
`walk` timer will show it; explicit-path staging sidesteps it entirely by never
visiting the images.
## Measurement (2026-07-12, toy `notes.md` tree, N ≈ 2)
Split from two back-to-back `:sync`es on the small test repo (commits `95ac56ef`
cold, `ab260bde` warm), via the `commit split —` log lines:
| Sub-phase | Kind | Cold (ms) | Warm (ms) |
| --- | --- | ---: | ---: |
| `walk(add_all+update_all)` | scan (O(N)) + likely 1 blob write | 1402 | 1456 |
| `index.write` | FAT write | 204 | 204 |
| `write_tree` | **1 tree object → FAT** | 710 | 715 |
| `parent-load` | FAT read | 102 | 105 |
| `commit-obj` | **1 commit object + ref → FAT** | 914 | 924 |
| **commit total** | | **3332** | **3404** |
### It is not the card — it's libgit2 (`sd_bench`, 2026-07-12)
My first read of the table was "a loose-object write to this SD card costs
~700900 ms." **That was wrong.** `sd_bench` (`firmware/src/bin/sd_bench.rs`) times
the raw FAT primitives on the same card at the same 10 MHz:
| Raw FAT op (200-byte payload) | p50 |
| --- | ---: |
| create + write + close | 21.7 ms |
| rename | 12.8 ms |
| stat (hit / miss) | ~5 ms |
| remove | 14.9 ms |
| **loose-object composite** (stat + create + write + rename) | **86 ms** |
The card does a *complete* loose-object write in **~86 ms**. Yet `write_tree`
(one tree object) took **710 ms** and `commit-obj` **914 ms** — an **~8× gap that
is pure libgit2 overhead, not FAT I/O.** So the earlier "object-write floor / SD
write amplification / better card / SPI-clock" framing is refuted: **the SD card is
not the bottleneck.** fsync is still confirmed off; the extra ~600 ms/op is CPU or
repeated `.git` I/O *inside* libgit2 (candidates: ODB refresh scanning
`objects/`, the treebuilder's per-entry `git_odb_exists`, ref-lock + reflog writes,
config/attributes re-reads). `git_bench` (`firmware/src/bin/git_bench.rs`) localizes
it — see below.
### It's the pack, read through an un-caching emulated mmap (`git_bench`, 2026-07-12)
`git_bench` times the git2 primitives in isolation on `/sd/repo` (git ops on the
same 96 KB thread the real service uses — the main-task stack overflows on
`index.write`, which is itself the reason the service has a dedicated thread):
| git2 op | p50 | note |
| --- | ---: | --- |
| `Repository::open` | 100 ms | one-time |
| `odb.write(blob)` (unique) | **45 ms** | writes a fresh object; touches no existing object |
| `repo.index()` open | ~0 ms | cached |
| `index.write()` | 376 ms | index + `index.lock` rename + tree-cache |
| `write_tree` [unchanged] | ~0 ms | tree exists → freshen-skips the write |
| **`write_tree` [changed]** | **1136 ms** | writes ONE 45 ms object |
| **`commit(None)` orphan obj** | **563 ms** | writes ONE 45 ms object, no ref/reflog |
Writing a fresh object is 45 ms; the ops that wrap one are 825×. The cause, from
the vendored source: `git_odb_write` calls `git_odb__freshen` (odb.c:1011), which
on a not-found object runs **`git_odb_refresh`** (re-reads the pack dir + reloads
pack indexes), and existence checks (`freshen(tree)` in `commit.c:169`, base-object
lookups in `write_tree`) hit the **pack**. Pack access goes through our
`p_mmap` (`esp_map.c`), which **`malloc`s and `read()`s the mapped range from the
card on every call — no cache** — with a 32 MB window on this 32-bit target. So
each write re-reads pack bytes from SD; `odb.write` of a fresh blob is 45 ms only
because it touches no packed object.
**This scales with pack size.** The toy repo's pack is tiny; the real
`jcalixte/notes` clone has a **570 MB pack**, and provisioning rsyncs a full clone
onto the card — so a real-repo commit has **never been benchmarked** and, on this
mechanism, will be far worse than the ~3.3 s toy number. That is the single biggest
open risk in sync.
### Real-repo run (2026-07-12, `jcalixte/notes`, 570 MB pack) — the index is the wrong primitive
`git_bench` was finally run against a full clone of the real repo (1179 files, 158
dirs, 570 MB pack). It settles the design: **any index-based commit is O(N_tree)
and does not fit this device.** Two independent walls:
| op | result | reading |
| --- | ---: | --- |
| `Repository::open` | 88 ms | fine |
| odb open (implicit) | ~6 s cold | maps the 1.7 MB pack `.idx` once (16 miss / 1790 KB) |
| `odb.write(blob)` | **142 ms** p50 | the mmap cache win **holds** (was 862 ms uncached) ✅ |
| `repo.index()` load (1179 entries) | 514 ms max | the on-disk index we were trying to avoid |
| `index.write()` | **min 360 ms / p50 12.8 s / max 611 s** | ⚠️ hangs — see root cause |
| **seed `read_tree(HEAD)` (cold, 1×)** | **~77 s** | ⚠️ reads all ~158 tree objects, 22.7 MB of pack windows |
| `Index::new + read_tree` (warm) | 447 ms p50 | windows still mapped → pure CPU |
| **index-free `stage→tree`** | **💥 crash** | `zlib (5)`: `deflateInit` failed, **508 KB heap left** |
**Wall 1 — `index.write()` hashes the whole working tree (up to 611 s).**
`git_index_write` unconditionally calls `truncate_racily_clean` (index.c:822),
which runs `git_diff_index_to_workdir` over **every** entry flagged "racy" and
re-hashes its file. On a fresh FAT clone the mtime granularity is 2 s and
`index.stamp.mtime <= entry.mtime` for ~all 1179 entries (index.h:117), so the
whole tree looks racy → it re-hashes ~170 MB (mostly the 150 MB of images) over
10 MHz SPI. The 611 s → 12.8 s → 360 ms decay across three iterations is the
signature: each write bumps the index mtime, shrinking the racy set. **Implication:
the shipping `stage_and_commit` calls `index.write()`, so `:sync` on the real repo
effectively bricks on the first commit** — the user sees a 10-minute freeze,
resets, the index mtime never advances, and it re-hashes forever. The real repo has
almost certainly never completed a sync on device (only the toy `typoena-test` has).
**Wall 2 — the index-free path is still O(N_tree), and the mmap cache OOMs.**
Skipping `index.write()` entirely (fresh `Index::new()`, stamp = 0, so
`truncate_racily_clean` can never fire) removes Wall 1. But to seed the in-memory
index, `read_tree(HEAD)` materialises all 1179 entries and reads every tree object
from the 570 MB pack — **77 s cold** (447 ms only once the windows are resident).
`write_tree_to` is O(changed), but you pay O(N_tree) to build the cache it needs, so
the index-free path only trades a 611 s hash for a 77 s tree-read. Worse, that
`read_tree` drove the `esp_map.c` cache to **7.4 MB resident** — past its own 4 MB
soft cap — which left 508 KB of heap and made `repo.blob()`'s zlib `deflateInit`
fail. **The one write we cannot skip crashed.** Root cause of the OOM: our cache
holds pack windows *after* libgit2 `p_munmap`s them (refcount 0, freed only lazily
on the next `p_mmap`), which **defeats `GIT_OPT_SET_MWINDOW_MAPPED_LIMIT`**
libgit2 thinks it released the memory; we didn't.
**Conclusion — use an O(depth) TreeBuilder walk.** "Replace K files in a 1179-file
tree" should touch `O(depth × K)` objects, not `O(N_tree)`. Walk HEAD's tree down
the edited path (`tree.get_name`/`get_path` → read ~depth subtree objects), then
rebuild bottom-up with `repo.treebuilder(Some(&subtree))``insert`/`remove`
`write()`, and `commit` the new root. That never materialises the 1179 entries,
never re-hashes anything, never visits the 150 MB of images, reads only a handful
of tree windows (so the cache stays small and zlib keeps its heap), and — crucially
— **carries the image entries forward untouched from HEAD's tree, so the device
does not even need the images in its working tree.** The `esp_map.c` cache still
needs an evict-on-`munmap` fix (drop the cap, free past a low-water mark) so it can
never again starve a downstream `git__malloc`, but with the TreeBuilder walk the
pressure it was under largely disappears.
### Splice bench (2026-07-12, second real-repo run) — the walk is right, the loose-object write is the new wall
The O(depth) splice op was added to `git_bench` (1 blob + 3 tree writes onto the
depth-3 path `.claude/commands/bsky.md`, run FIRST so its first iteration is
cold; the index ops moved last so their OOM can't cost the new data — it did
crash again, after everything was logged):
| op | result | reading |
| --- | ---: | --- |
| `splice stage→tree` (1 blob + 3 trees) | **6.5 s p50, warm ≈ cold** | O(depth) confirmed — cost is 4 loose writes × ~1.6 s |
| `commit(None)` orphan obj | 1.7 s p50 | one more loose write |
| `odb.write(blob)` | **1.5 s p50** | ⚠️ was 142 ms in the previous run |
| `repo.index()` load | 524 ms max | matches previous run |
| seed `read_tree(HEAD)` cold (now timed) | 81.6 s | reproduces the 77 s |
| `index-free stage→tree` | 💥 crash, 508 KB heap | reproduces the zlib OOM exactly |
Three readings:
1. **The splice mechanism is validated as a mechanism.** Pack reads stayed flat
(~40 KB per write; 6.4 MB heap free through splice + commit + odb.write), so
it really is O(depth) and it cannot OOM. The 6.5 s is not tree-walk cost.
2. **The wall moved to the loose-object write: ~1.5 s each, ×4 per splice.**
The isolated `odb.write(blob)` — one tiny orphan blob — took 1.5 s where the
raw FAT composite is 86 ms. Projected full commit (splice 6.5 s + commit-obj
1.7 s + ref/reflog update) ≈ **89 s**: enormously better than 611 s, still
far off the bar.
3. **The mmap cache scored 0 hits over the entire run** — the documented
862→142 ms `odb.write` win did **not reproduce** (same `esp_map.c`, same
card). Either the earlier run's conditions differed (orphan-object
population? FAT allocation state?) or the win was misattributed. Whatever the
1.5 s is, it is *not* SD data volume: each write moves ~40 KB read + ~1 KB
written.
### ROOT CAUSE FOUND (2026-07-12, `sd_bench` seek op): FatFS lseek walks the cluster chain
Two `sd_bench` re-runs on the ~740 MB-full card settled it:
1. **Free-cluster-scan hypothesis: refuted.** Raw FAT write ops are unchanged on
the full card — loose-object composite **77 ms p50** (was 86 ms), create
20 ms, rename 10 ms. The card is exonerated a second time.
2. **Long seeks are the cost.** A new op opens the repo's largest packfile
(263 MB — the "570 MB pack" was actually the whole `.git`) read-only and does
seek+read(4 KB): **@offset 0 = 5.8 ms; @end = 198.7 ms** — dead constant
across 20 iters. Without `CONFIG_FATFS_USE_FASTSEEK`, FatFS resolves lseek by
walking the file's FAT cluster chain over SPI: forward from the current
position, **from the chain head on any backward seek**. 263 MB ≈ 16.8k
clusters ≈ ~67 KB of FAT reads ≈ ~190 ms per long walk.
**Why FAT behaves this way:** FAT has no extent map or inode — a file is a
singly-linked list of clusters, and the only way to find "byte 260,000,000" is
to follow that list entry by entry through the allocation table. FatFS walks
forward from the current position when it can, but a backward seek restarts
from the chain head ([FatFS `f_lseek` docs, elm-chan.org](http://elm-chan.org/fsw/ff/doc/lseek.html)).
The fast-seek feature fixes exactly this: a pre-computed **cluster link map
table (CLMT)** per file object, "(fragments + 1) × 2" words, after which "no
FAT access is occured in subsequent f_read/f_write/f_lseek" (same page). On
esp-idf it's `CONFIG_FATFS_USE_FASTSEEK` — the official docs recommend it "for
read-heavy workloads with long backward seeks" and note it does not apply to
files opened in write mode
([ESP-IDF FatFS docs](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/fatfs.html)).
The budget closes: each loose write does ~810 small (~4 KB) `p_mmap`s (freshen
→ trailer/idx probes) interleaved with low-offset reads, so ~8 of them pay a
fresh ~190 ms walk → **~1.5 s per object**. It also explains everything the
cache couldn't: warm ≈ cold (the walk is paid inside `lseek` before any data
moves, and the maps are below the 64 KB cache floor), the 142 ms vs 1.5 s
run-1/run-2 discrepancy (run 1's `odb.write` bench ran first and hammered only
the trailer — the file position stayed there, so its seeks were forward/no-ops),
and a large slice of the 81.6 s `read_tree` (133 windows × backward seeks ≈ 25 s
of walking on top of the 25 MB of data).
**Fix (config, not code): `CONFIG_FATFS_USE_FASTSEEK=y` +
`CONFIG_FATFS_FAST_SEEK_BUFFER_SIZE=256`** (landed in `sdkconfig.defaults`
2026-07-12). Fast seek builds an in-memory cluster-link map per read-mode file —
exactly how the pack is opened — making lseek O(1); write-mode files fall back
to the walk transparently. 256 words = 1 KB per open read-only file, covering
~127 fragments (default 64 covers ~31; a fragmented pack would silently fall
back to slow seeks, so the headroom matters).
**A/B measured (same evening): a 2.3× partial win, not a full one.**
| op | fast-seek off | fast-seek on |
| --- | ---: | ---: |
| `splice stage→tree` | 6.5 s | **2.81 s** |
| `odb.write(blob)` | 1.5 s | **416 ms** |
| `commit(None)` | 1.7 s | **1.72 s — unchanged** |
`odb.write` dropped by almost exactly the ~6 chain walks the model predicted —
the seek theory holds — but two residuals remain: **~400 ms per loose write**
(vs the 77 ms raw-FAT floor) and **`commit(None)`'s ~1.3 s premium over a plain
write, which was never seek-bound at all**. Prime suspect for the commit
premium: strict object creation makes `git_commit_create` validate its parent +
tree OIDs with pack header resolves, and `git_treebuilder_insert` does the same
per inserted entry — `git_bench` grew `odb.read_header(packed)` /
`odb.exists(missing)` probes and strict-off re-benches to test it.
### Second localization round (2026-07-12, run 3b + sd_bench re-run)
**Fast-seek verified on the metal:** re-running the sd_bench seek op with
`CONFIG_FATFS_USE_FASTSEEK=y` dropped `pack seek+read 4KB @end` from
**198.7 ms → 20.4 ms** (the CLMT fits the pack in the 256-word buffer — the
pack is not too fragmented). A far seek is now ~15 ms, i.e. effectively fixed.
**The strict-creation theory is refuted; the probes found the real unit cost:**
| op | p50 | reading |
| --- | ---: | --- |
| `odb.read_header(packed)` | **470 ms** | ONE pack header resolve costs ~½ s |
| `odb.exists(missing)` | **968 ms** (±0.1 ms) | miss path (scan → refresh → rescan) ≈ 2× |
| `commit(None)` strict OFF | 1.80 s | vs 1.93 s strict on — validation is NOT the premium |
| `splice` strict OFF | 5.7 s | noise-worse; also not validation |
The ±0.1 ms constancy of `exists(missing)` = a fixed, deterministic SD-op
sequence. The map counters identify it: **~78 small (~4 KB) `p_mmap` reads per
op** — pack trailer probes, idx fanout reads and delta-base windows, repeated
at the *same offsets* on every freshen/refresh. Post-fast-seek those cost
~20 ms each (~150 ms/op); the rest of `read_header`'s 470 ms is CPU-side
delta-chain inflation on the 160 MHz core plus repeated re-reads. Two other
observations from run 3b: the loose-object orphan population from bench runs
is creeping costs upward (splice 2.81 s → 3.21 s between consecutive
fast-seek runs — a re-provision resets it), and the mmap cache STILL scored 0
hits — because its 64 KB map-length floor excluded exactly these hot small
maps.
**Fix built (esp_map.c v2):** cache admission re-keyed from
map length to **file size ≥ 1 MB** — the pack/idx's small repeated windows now
cache (RAM hits after first touch) while small mutable working-tree files stay
excluded — plus **evict-on-`p_munmap` down to a 2 MB low-water mark**, fixing
the 7.4 MB OOM from the first real-repo run (released windows are actually
returned to `git__malloc`, so `MWINDOW_MAPPED_LIMIT` stays honest). Expected:
`read_header` collapses toward CPU-only, `odb.write` toward ~150250 ms,
splice at or under the sub-second bar, and no end-of-run zlib OOM.
### Final bench (run 4, esp_map v2) — memory fix works, cache theory dead, bar failed
Run 4 (2026-07-12 evening, same card state as run 3b plus its orphans):
| op | run 3b (fast-seek) | run 4 (+ esp_map v2) |
| --- | ---: | ---: |
| `splice stage→tree` (cold, first op) | 2.81 s | **2.83 s — unchanged** |
| `splice` again (warm, strict-off phase) | 3.21 s | 1.95 s |
| `commit(None)` | 1.72 s | 713 ms |
| `odb.write(blob)` | 416 ms | 366 ms |
| `odb.read_header(packed)` | 470 ms | 412 ms |
| `odb.exists(missing)` | 968 ms | 852 ms |
| mmap-cache hits | 0 | **0** (313 misses) |
| cache resident / heap free | grew to 7.4 MB → zlib OOM | **1833 KB flat / 6.4 MB free all run** |
Three findings:
1. **The memory discipline is verified.** Resident sits at 1833 KB through
every phase (under the 2 MB low-water, so nothing is being churned) and
heap never drops below 6.2 MB. The one uncaptured datum is the index-free
`read_tree` tail (the section that OOM'd runs 13) — the monitor was cut
before it ran. Not blocking: the shipping splice path never calls
`read_tree`; the tail would only re-confirm eviction under burst.
2. **The repeated-small-window theory is REFUTED — theory #3 down** (after
strict-creation and free-cluster-scan). v2 demonstrably admits and retains
the small maps now — the 1833 KB resident *is* them, held below low-water so
nothing is evicted before reuse — and still scored 0 hits in 313 misses.
So the ~8 small reads per loose write hit **unique (offset, len) every
time**: `mwindow` was already absorbing any true repetition above `p_mmap`,
and what reaches the emulation layer is distinct data (different objects,
different delta bases). A window cache cannot help. The residual
~360 ms/loose-write ≈ 8 distinct small SD round-trips × ~45 ms each
(post-fast-seek) — I/O count, not I/O size or seek cost.
3. **Within-run drift cuts both ways, so cross-run tables are mushy.** In this
single run `commit(None)` degraded 713 ms → 1.79 s between the early and
late (strict-off) phases, while splice *improved* 2.83 → 1.95 s. Two
competing effects: first-touch warm-up fading (CLMT build, first pack
reads — helps later ops) and orphan loose objects accumulating in
`.git/objects/xx/` slowing every freshen existence check (FAT directory
lookups are linear scans — hurts later ops). Steady-state on a clean
objects dir ≈ **~2 s per splice+commit**.
**Run 5 (confirmation, cache removed entirely):** esp_map.c stripped back to
plain malloc-read/free-at-munmap (stats counters kept). Read pattern
**byte-identical** to run 4 at every checkpoint (118 maps / 2314 KB after
splice, 148/2434, 163/2494, 208/2674) — except the strict-off phase did **15
fewer reads (~60 KB)** than v2: the low-water eviction had been kicking out
buffers mwindow still wanted, forcing re-reads. The cache was marginally
worse than nothing. Warm splice identical (1953 vs 1949 ms); the cold-op
+1015 % drift (splice 2.83 → 3.26 s) is the known orphan-creep signature,
not the removal. Run 5 also reframes run 4's "resident": **1854 KB `live` is
mwindow's open-window working set** (pinned mappings under the bench's 4 MB
mapped limit), not retained cache buffers — the cache had been retaining
essentially nothing. Removal CONFIRMED free; simpler emulation ships.
**Verdict: the sub-second bar FAILED — wire the splice in anyway.** The bar
was aspirational; measured reality is ~22.8 s to commit on the real
263 MB-pack repo versus 611 s (or a hard OOM) for every alternative benched.
That puts a full real-repo `:sync` at roughly **910 s cold**, which ships.
The remaining ~2 s has survived four localization rounds; the next suspect —
FAT *directory-op* cost in the freshen/refresh path (open/stat/rename by path
walk FAT directories linearly; consistent with the orphan-creep signal) — is
one instrumentation pass for later (log each `p_mmap` miss + bench dir ops in
`sd_bench`), not a prerequisite for the plumbing.
### The walk is ~1.4 s even at N ≈ 2
Mostly fixed cost — the worktree-diff setup and the second (`update_all`) pass —
not per-file `stat` (one raw `stat` is ~5 ms, so N ≈ 2 can't be the 1.4 s). The
O(N) slope only bites on the real `jcalixte/notes` clone (N ≈ 1179), which this run
did **not** exercise. That slope is still unmeasured.
For orientation: `publish(commit+push)` was 9846 ms cold, so the **network half is
~6.5 s** — still the biggest single block of a warm sync (10.1 s total), a separate
floor ([`../notes/sync-latency.md`](../notes/sync-latency.md)).
## The verdict
The real-repo run (above) overturned the earlier ranking. Both index strategies
are O(N_tree) and fail on the 570 MB-pack clone, and the repo cannot be shrunk. The
work, ranked:
1. **Rewrite the commit as an O(depth) TreeBuilder walk (the fix — build this).**
Rebuild only the edited path's ancestor subtree chain onto HEAD's tree; never
materialise the 1179-entry index, never `index.write()`, never `read_tree` the
whole tree. This is the ONLY mechanism that fits: O(depth × dirty) reads/writes,
flat in repo size, small heap, images carried forward untouched. Replaces
`stage_and_commit`'s `add_all`/`update_all`/`index.write`/`write_tree`. Needs the
editor's dirty set (+ deleted set) plumbed to the git service — the editor
already knows both. **Benched to completion 2026-07-12: 6.5 s → 2.8 s cold /
~2 s steady-state after the fast-seek fix (run 4). The sub-second bar failed
but the block is lifted — wire it in** (see the final-bench section above:
the residual is unique small SD round-trips, not something a cache or seek
fix can remove).
2. **Fix the `esp_map.c` cache so it can't OOM — RESOLVED BY REMOVAL (run 5).**
The cache never scored a hit in four instrumented real-repo runs
(`mwindow` absorbs true repetition above `p_mmap`; only new ranges reach
the emulation), and the 7.4 MB OOM it was patched to avoid was caused by
the cache itself holding buffers past `p_munmap`. esp_map.c is now the
plain malloc-read/free-at-munmap emulation: honest with
`MWINDOW_MAPPED_LIMIT` by construction, ~120 lines lighter, and run 5
confirmed removal is I/O-neutral (even 15 reads *better* than v2, whose
low-water eviction fought mwindow). Stats counters kept to spot any future
workload that does repeat ranges.
3. **Retired: `add_all`/explicit-path *index* staging.** Explicit-path `add_path`
still goes through the index and `index.write``truncate_racily_clean`, so it
hits Wall 1 just the same. The TreeBuilder walk supersedes it entirely; the
"explicit-path staging" idea survives only as "the editor's dirty set feeds the
walk."
4. **Retired: SD clock / better card.** The card does a full object write in
~86 ms; raw I/O is not the bottleneck. Do not spend the PCB's 20 MHz budget
expecting a commit-latency win.
5. **Kept: the mmap cache + mwindow tuning** (`GIT_OPT_SET_MWINDOW_*`, 256 KB
window / 4 MB mapped limit). It fixed `odb.write` and the push read path; #2 just
makes it well-behaved under memory pressure.
**Recommendation:** build #1 (the O(depth) TreeBuilder walk) — #2 resolved
itself by removal. The concrete plumbing plan (exact call sites, dirty-set
threading, FD budget) is the
[next section](#the-fix--wiring-the-odepth-splice-into-the-firmware).
## The fix — wiring the O(depth) splice into the firmware
> Merged from `docs/notes/sync-commit-handoff.md` (written 2026-07-12, retired
> 2026-07-13 once the bench phase closed). The handoff's bench half is done —
> the splice op lives in `git_bench` and the numbers above are its output.
> **The firmware plumbing below SHIPPED 2026-07-13** (compile-verified both
> feature flavors; on-device `:sync` against the real repo still pending) —
> each item now records how it landed rather than what to do.
### The splice walk
Rebuild only the edited file's ancestor subtree chain onto HEAD's tree. Never
materialise the 1179-entry index; never `index.write()`; never `read_tree` the
whole tree. Cost is O(depth × dirty_files), flat in repo size, tiny heap, and it
carries every unchanged entry (all 260 images, the other 1176 files) forward
untouched — which means **the device doesn't even need the images in its
working tree.**
Shipped as `git_sync::splice` (git2 0.20: `Repository::treebuilder(Option<&Tree>)`
+ `TreeBuilder::{insert,remove,write}`). Signature:
```rust
fn splice(repo: &Repository, base: Option<&Tree>, path: &[&str], blob: Option<Oid>) -> Result<Oid>
```
`Some(blob)` inserts/replaces, `None` removes; `base: None` synthesizes a
missing intermediate directory on the way down, and a directory emptied by a
remove is pruned on the way up (the empty tree is never re-inserted).
`stage_and_commit` folds every dirty path through it (threading the running
root tree), then `commit(Some("HEAD"), …)` exactly as before — the
`tree unchanged → nothing to publish` check and the `commit split —` timing
log survive. The benched reference is `git_bench`'s `splice stage→tree` op.
### `firmware/src/git_sync.rs` — how it landed
- **mwindow options set at service start** (top of `run_git_service`, before
any `Repository::open`): `set_mwindow_size(256 KB)` +
`set_mwindow_mapped_limit(4 MB)`. Without them the 32-bit defaults (32 MB
window / 256 MB mapped limit, mwindow.c:16) would `git__malloc` a 32 MB
window on the first pack access and die on the 8 MB PSRAM heap. Run 5
sharpened the stakes: the ~1.85 MB bench "resident" **is** mwindow's live
open-window set, so these opts are the only thing bounding it.
- `stage_and_commit(repo, paths)` is the splice walk; `add_all`/`update_all`/
`index.write`/`write_tree` are gone. **The request carries one path set, not
`{changed, deleted}`:** the working tree is the source of truth, so at commit
time a recorded path that exists on the card is spliced in from disk and a
missing one is spliced out. An unchanged path is a no-op — over-reporting is
free, which is what makes the retry/journal semantics below simple.
- **Stranded-commit recovery (new):** when the splice yields the parent's tree
(nothing to commit), the service now compares HEAD against
`refs/remotes/origin/<branch>` and still pushes if origin lacks HEAD — a
previous cycle that committed and then failed mid-push used to strand that
commit forever (the old path reported "up to date" and never retried).
- **Radio-free up-to-date (new):** an empty dirty set + origin already at HEAD
short-circuits before Wi-Fi even comes up — `:sync` with nothing to do
answers in ~150 ms instead of a Wi-Fi/TLS round.
- `reconcile_onto_origin` now `ResetType::Soft` (ref move only) — there is no
index to reset, and a Mixed reset's index write is exactly the racy-clean
wall the splice avoids. Side win: a remote-only added file is now *carried*
by the replay (origin's tree is the splice base) where the old `add --all`
replay dropped it.
- The macOS-cruft filter (`skip_macos_cruft`) is gone with the walk — the
splice only touches paths the editor recorded, so `._*`/`.DS_Store` can't
sneak in the way they once did (07d87772, the Spike-14-era lesson).
- **Deliberate behavior change (now in effect):** only paths the editor
saved/deleted are ever committed. Files changed on the card outside the
editor (card mounted on a Mac) were swept in by `add_all` before; they will
now never be committed, and the working tree will show a permanent diff
against HEAD if inspected on a desktop. Correct for the appliance; recorded
here so it reads as intent, not accident.
### Dirty-set source — `firmware/src/persistence.rs` + `main.rs` (how it landed)
- `Storage` owns the record: `save_path`/`delete_path` note their repo-relative
path in a `RefCell<Dirty>` (paths outside `/sd/repo` are skipped), and the
set is **journaled to `/sd/.typoena-dirty`** — atomic write, rewritten only
when the set actually grows. Without the journal a power pull would strand
every file saved-but-not-published that session: nothing walks the tree
anymore, so an unrecorded change would never reach the remote. The journal
is loaded at mount and its paths ride the next `:sync`.
- Lifecycle: `take_dirty()` snapshots pending → in-flight for one publish
(journal keeps carrying the union); the outcome settles it —
`publish_succeeded()` forgets the snapshot and shrinks the journal,
`publish_failed()` returns it to pending for the next `:sync`. A save landing
*while* a publish runs re-enters pending and rides the next one. Recording
happens *before* the file write, so a crash between the two only
over-approximates (a no-op splice), never under-records.
- `Effect::Publish` in `main.rs` sends `PublishRequest { paths: take_dirty() }`;
the outcome handler in the idle branch calls the matching settle method.
- **FD budget:** a git build now mounts `Storage::mount_for_git()` (16 FDs) in
`boot_storage` — libgit2 keeps the pack + `.idx` descriptors open and opens
loose objects on top, which overruns the editor's 4-FD budget. The light
build keeps the editor's own budget.
### `esp_map.c` — nothing left to do
The handoff's third work item (an evict-on-`p_munmap` cache fix) is superseded:
run 5 removed the cache outright and `esp_map.c` is already the plain
malloc-read/free-at-munmap emulation (see verdict item 2). One surviving
operational note: editing `components/libgit2/*.c` won't rebuild via plain
`cargo build` — first
`rm -rf firmware/target/xtensa-esp32s3-espidf/release/.fingerprint/esp-idf-sys-*`
(the `esp-idf-component-rebuild` lesson).
### How to bench / flash
`git_bench.rs` runs git ops on the 96 KB `GIT_STACK` thread (the main task stack
overflows on these ops — that's why the real service has a dedicated thread). It's
Rust-only, so a plain rebuild picks it up (no fingerprint dance unless you also
touched `esp_map.c`).
```
just flash-gitbench
# = . ~/export-esp.sh && LIBGIT2_SRC=<repo>/firmware/components/libgit2/vendor \
# LIBGIT2_NO_VENDOR=1 PKG_CONFIG_ALLOW_CROSS=1 \
# PKG_CONFIG_LIBDIR=<repo>/firmware/pkgconfig \
# cargo run --release --bin git_bench --features git
```
Bench on the **real repo** clone (`/sd/repo` = full `jcalixte/notes`), not the
toy — the toy pack understates everything by ~2 orders of magnitude.
### Still open (none block the plumbing)
- The residual ~360 ms/loose-write ≈ 8 unique small SD round-trips; next
suspect is FAT **directory-op** cost in the freshen/refresh path. One
instrumentation pass for later (log each `p_mmap` miss + bench dir ops in
`sd_bench`).
- Ref/reflog-update cost on the real repo — the bench's `commit(None)` writes
no ref, so the shipping commit's last leg is unmeasured.
- The push's ~6.5 s network floor
([`../notes/sync-latency.md`](../notes/sync-latency.md)) — a separate curve.
## Adjacent lever: should the images be on the card at all?
Explicit-path staging makes the walk skip the images, but they still cost 150 MB
of SD space, inflate the 570 MB clone, and slow provisioning + the pull-before-push
paths. Whether the device should carry image blobs at all — vs. markdown-only, or
Git-LFS-style pointers — is a separate decision tracked in
[`../notes/git-sync-images-and-repo-size.md`](../notes/git-sync-images-and-repo-size.md).
That lever shrinks N and the clone; this one stops the walk from paying for N. They
compose.
## What this does *not* touch
The network half of `:sync` (TLS handshake + push round-trips, ~6.5 s of the warm
path) is a separate floor covered in [`sync-latency.md`](../notes/sync-latency.md);
this curve is only about the local commit. Radio *frequency* (how often we pay any
sync at all) is [`wifi-auto-sync.md`](wifi-auto-sync.md).

View File

@@ -83,9 +83,10 @@ handler, alongside the existing list-continuation transform.)
On a **typing pause** — the same throttle as the word-count / cursor refresh, so
never a per-keystroke e-ink flash — if the word before the caret is a prefix, the
right side panel shows a quiet hint (`↹ fiche de lecture`). The panel is ~17
columns, so the hint is the **snippet name / first line**, not the whole body;
the full preview is what the `$` palette is for.
right side panel shows a quiet hint (`» Book notes`), on the row above the mode
line. The panel is ~15 columns, so the hint is the **snippet name**, not the whole
body; the full preview is what the `$` palette is for. (The marker is `»`, not a
tab glyph: the panel font is ISO-8859-15, which has no `↹`.)
### `$` palette (browse + insert)
@@ -126,9 +127,11 @@ rather than refusing to start over a stray comma.
the device on the next clone/sync. This is deliberately where the heavy editing
happens; the appliance is for writing, not for maintaining a JSON library.
- **First-time setup.** [`just init`](../firmware/README.md#provisioning-an-sd-card)
seeds this file from a curated catalog — you pick which snippet groups you
want and it writes the selected subset into `repo/.typoena.snippets.json`
(committed on the device's first `:sync`). See
seeds this file from a curated catalog (`firmware/snippets-catalog/`) — you pick
which snippet groups you want and it `jq`-merges the selected subset into
`repo/.typoena.snippets.json` (committed on the device's first `:sync`). It
writes **only if the file is absent**, so re-running `init` on a card whose
clone already carries your library never overwrites it. See
[`v0.6-markdown.md`](v0.6-markdown.md) for the catalog.
- **On-device hand-edit — deferred.** The palette hides dotfiles, and `:e` was
dropped in v0.6, so there is no in-editor path to this file yet. When one is

View File

@@ -149,6 +149,43 @@ shown). On-device check: reflash → open a note (trailing empty line visible)
newline for the same reason; the guarded save leaves the prefs file with exactly
one.
**Amendment 2026-07-13 — recursive enumeration + a 2-char search threshold.**
Loading a real repo (`jcalixte/notes`) exposed that `enumerate_files` listed
only the **top-level** files of `/sd/repo` and `/sd/local` — a nested notes tree
showed a single file in the palette (subpaths always *opened* fine via
`:e repo/sub/x.md`; only the listing was flat). The enumeration is now a
recursive walk: dot entries are skipped at every level (so `.git` is never
descended into), each directory is read fully before recursing (one FatFS dir
handle open at a time — the `remove_dir_recursive` pattern, kind to the
FD-bounded mount), depth is capped at 8, and the boot-time walk logs its file
count and duration (`file walk: N files in Xms`) so the FAT dir-IO cost on a
big repo is measurable, not assumed. With the list now card-sized, the palette
gained a **search threshold**: below 2 typed chars the result list is the
**recents (MRU) only** — quick-switch (`Cmd-P`, `Enter`) stays one keystroke
away — and the full fuzzy-ranked list appears from 2 chars on
(`PALETTE_MIN_QUERY`). A fresh boot with no opens yet shows `(type to search)`.
`>` commands and `$` snippets are short curated lists; the threshold does not
apply to them.
**TODO (on-device, next time the device is on the bench)** — two measurements
from the same boot log, both already instrumented:
- [ ] **Re-measure the walk time** after the d_type fix (`2660a3e` — dirent
`file_type()` instead of a per-entry `metadata()` stat, which cost
~32 ms/file and made run 1 take 35 s for 1098 files). Read the
`file walk: N files in Xms` line. Only if it's still slow does the
async-walk idea come back on the table.
- [ ] **Read the file-list DRAM cost** from the new
`file list: internal heap <before> -> <after> (<N> KB consumed)` line
(the build is bracketed with `MALLOC_CAP_INTERNAL` readings in
`main.rs`). The 1098 path Strings are each below the 16 KB SPIRAM
malloc threshold, so they all land in internal DRAM — estimated
6070 KB, competing with Wi-Fi/TLS. Decision rule: **~6070 KB
confirms** interning the paths into one shared buffer (a single
>16 KB alloc goes to PSRAM; only a ~9 KB offset index stays in DRAM);
**well under that (≤~30 KB)** kills the idea — the next DRAM suspect
is then parked-buffer text.
- [x] `Cmd-P` opens fuzzy file palette over **both** `/sd/repo/` and
`/sd/local/`**landed and CONFIRMED ON DEVICE 2026-07-12** (Spike 11: no
ghosting on the transient panel); scope shows as the inline

View File

@@ -3,10 +3,14 @@
> Part of the [Typoena macro plan](macroplan.md). Requirements and targets:
> [qfd.md](qfd.md). Load-bearing decisions: [adr.md](adr.md).
**Status:** render affordances done early; the snippet engine is the remaining
work (snippets are net-new scope, added 2026-07-08; reshaped 2026-07-12 from a
hard-coded table into a git-synced, Zed-compatible library with a `$` palette
launcher — see [`typoena-snippets.md`](typoena-snippets.md) for the file format).
**Status:** render affordances done early. The snippet engine (tab-stop core,
inline Tab-expansion, hint-on-pause, `$` palette) and the `>` command-palette
generalisation are **done in core, host-tested** (187 editor tests); the
remaining work is the **firmware boot-read + `just init` catalog** (slice 5, the
on-device gate). Snippets are net-new scope, added 2026-07-08; reshaped
2026-07-12 from a hard-coded table into a git-synced, Zed-compatible library with
a `$` palette launcher — see [`typoena-snippets.md`](typoena-snippets.md) for the
file format.
- [x] Heading lines bolded in render (faux-bold double-strike)
- [x] List continuation on Enter inside `- ` / `1. ` (with empty-item exit)
@@ -34,62 +38,75 @@ behaviour.
handler alongside the existing `list_marker` transform
(`expand_snippet(word) -> Option<Snippet>`). Tab already arrives as
`Key::Char('\t')`, so no new key event.
- [ ] **Hint-on-pause.** On the typing pause (same throttle as the word-count /
- [x] **Hint-on-pause.** On the typing pause (same throttle as the word-count /
cursor refresh — never a per-keystroke repaint), if the word before the
caret is a prefix, the right side panel shows a quiet hint. The panel is
~17 cols, so the hint is the snippet **name / first line**, not the whole
body — the full preview is the `$` palette's job.
caret is a prefix, the right side panel shows a quiet hint (`» name`, on the
row above the mode line, sharing the slot with the `NO KBD` flag — the two
never co-occur since the hint means you're typing). The panel is ~15 cols,
so the hint is the snippet **name**, not the whole body — the full preview is
the `$` palette's job. Snapshotted in `refresh_stats` so it rides the pause,
and the firmware's Insert-pause repaint already carries it (no firmware
change). Latin-9 has no `↹` glyph, hence the `»` marker.
- [x] **`$` palette (browse + insert).** `Cmd-P` then `$` switches the palette to
the snippet list (the same sigil mechanism as `>`). Fuzzy-matches name /
prefix / description; Enter inserts the body at the caret and starts the
tab-stop session. Lists **all** snippets — the fuzzy filter handles clutter,
so there's no hidden "inline-only vs palette-only" split. Rows read
`Name [prefix]`, so browsing also teaches the inline trigger.
- [ ] **Boot wiring.** The host reads `/sd/repo/.typoena.snippets.json` at boot
- [x] **Boot wiring.** The host reads `/sd/repo/.typoena.snippets.json` at boot
and calls `Editor::set_snippets` (mirroring `set_prefs`); a missing or
malformed file is non-fatal (no snippets, editor runs). Parse lives in the
host-testable `editor` crate via `serde_json` — the one new dependency.
host-testable `editor` crate via `serde_json` — the one new dependency,
confirmed to build for xtensa (`cargo check`, firmware 0.6.0). On-device
smoke-test still pending.
## The palette, generalised (`Cmd-P` · `>` · `$`)
## The palette, generalised (`Cmd-P` · `>` · `$`) — done in core
v0.5 shipped `Cmd-P` = files and `>` = a five-entry settings list (`save_on_idle`,
`format_on_save`, `line_numbers` toggles + `theme`/`auto_sync` rotations). v0.6
makes the sigils a clean split by verb, and the empty-palette placeholder legends:
`Go to file · > settings · $ snippets`.
makes the sigils a clean split by verb, and the empty-palette placeholder legends
them: `Go to file · > settings · $ snippets`.
- **bare `Cmd-P`** → *navigate*: go to file (unchanged).
- **`>`** → *act on the editor* — the command palette. The pref toggles are just
its stateful entries, not a special section. Dispatch differs per entry: a
**toggle** flips and the list **stays open** (as today); a **one-shot** (e.g.
`format`, `publish`) runs and **closes**; a **parameterised** command morphs
the palette into a second **input step** (select `New file` → the box becomes a
filename prompt → Enter creates it, scope read from a `repo/`/`local/` prefix as
`:enew` does today). This retires `:e` (bare `Cmd-P` covers file-opening;
dotfiles get a dedicated `> edit …` command if/when wanted).
- **`>`** → *act on the editor* — the command palette, a real action registry (the
`PALETTE_CMDS` list, actions first then settings). The pref toggles are just its
stateful entries, not a special section. Dispatch is by `PaletteCmd::kind`: a
**toggle** flips and the list **stays open**; a **one-shot** (`format`,
`publish` — the latter shares `run_publish` with `:sync`) runs and **closes**; a
**parameterised** command (`new file...`) morphs the palette into a second
**input step** (the box becomes a filename prompt → Enter creates it, scope read
from a `repo/`/`local/` prefix as `:enew` does; backspacing past the start steps
back to the `>` list). This **retired `:e`** — bare `Cmd-P` covers file-opening;
dotfiles can get a dedicated `> edit ...` command if/when wanted.
- **`$`** → *insert content* — the snippet launcher above.
`format` / `publish` as `>` entries are a small opportunistic add (they prove `>`
is a real action registry, not a settings box); the headline is snippets.
Labels avoid `…`/`` — the palette/panel fonts are ISO-8859-15, which has neither
glyph — so `new file...` uses ASCII dots and the pause hint leads with `»`.
## First-time setup — snippet catalog (`just init`)
## First-time setup — snippet catalog (`just init`) — done
[`just init`](../firmware/README.md#provisioning-an-sd-card) gains a step that
seeds the two git-tracked config files into `repo/` (so they commit + sync on the
device's first `:sync`): a starter [`.typoena.toml`](typoena-toml.md) (the four
keys at their defaults, or confirmed at a prompt) and a
[`.typoena.snippets.json`](typoena-snippets.md) chosen from a **curated catalog**
checked into the repo. The catalog is grouped and opt-in — you pick the groups
you want rather than getting all of one writer's personal templates. Not every
Zed snippet is worth proposing: the Slidev/blog-pipeline ones (`@[youtube]`,
`<v-clicks>`, frontmatter, mermaid) and render-dependent or hyper-specific ones
are left out of the menu, since a distraction-free prose appliance can't render
them and you'd never miss them. **Proposed groups (pending sign-off):**
[`just init`](../firmware/README.md#provisioning-an-sd-card) seeds the two
git-tracked config files into `repo/` (so they commit + sync on the device's
first `:sync`), **writing only a file that is absent** so a re-`init` never
clobbers a synced library: a starter [`.typoena.toml`](typoena-toml.md) (the five
keys at their defaults) and a [`.typoena.snippets.json`](typoena-snippets.md)
assembled from a **curated catalog** (`firmware/snippets-catalog/`, `jq`-merged).
The catalog is grouped and opt-in — you pick the groups (each `[Y/n]`, all-yes on
a non-interactive run) rather than getting one writer's whole personal set. Not
every Zed snippet is worth proposing: the Slidev/blog-pipeline ones (`@[youtube]`,
`<v-clicks>`, frontmatter, mermaid) and hyper-specific personal ones are left out,
since a distraction-free prose appliance can't render them and you'd never miss
them. Prefixes are **English** (except `edanso`, the mnemonic for `œ`), bodies
translated from the source Zed snippets. **Three groups, 17 snippets:**
- [ ] **Symbols** (inline, keyboard can't type them): `fleche``→`,
`different``≠`, `fois``×`, `median``·`, `degre``°`, `euro``€`,
`edanso``œ` (dead keys in v0.2.5 cover accents, *not* these).
- [ ] **Structure**: `todo``- [ ] `, `link``[$1]($2)$0`, `img``![$1]($2)$0`
(net-new — obvious for Markdown), `table`, `code` (fenced block).
- [ ] **Prose / PKM templates** (`${n:label}` stripped to `$n`): `fiche`
(book notes), `reference`/`refangl` (reference block), `biais`,
`capture`, `standard`, `5w1h`.
- [x] **Symbols** (inline, the keyboard can't type them): `arrow``→`,
`neq``≠`, `times``×`, `middot``·`, `deg``°`, `euro``€`, `edanso``œ`
(dead keys in v0.2.5 cover accents, *not* these). **Caveat:** `→` (U+2192)
and `≠` (U+2260) are outside ISO-8859-15, so they store correctly and sync
but render as a missing-glyph box on the device panel until the font is
extended; the other five are Latin-9 and draw fine. Kept deliberately.
- [x] **Structure**: `todo``- [ ] `, `link``[$1]($2)$0`, `img``![$1]($2)$0`,
`table` (2-col), `code` (fenced block). `img`/`code` are net-new.
- [x] **Prose / PKM templates** (`${n:label}` stripped to `$n`): `booknotes`,
`reference` (the English reference block — `refangl` folded in), `bias`,
`capture`, `standard`. (`5w1h` dropped.)

File diff suppressed because it is too large Load Diff

View File

@@ -3,7 +3,7 @@ target = "xtensa-esp32s3-espidf"
[target.'cfg(target_os = "espidf")']
linker = "ldproxy"
runner = "espflash flash --monitor"
runner = "espflash flash --monitor --baud 921600"
rustflags = [ "--cfg", "espidf_time64"]
[unstable]

View File

@@ -1,6 +1,6 @@
[package]
name = "firmware"
version = "0.5.0"
version = "0.6.0"
authors = ["Julien Calixte <juliencalixte@gmail.com>"]
edition = "2024"
resolver = "2"
@@ -36,6 +36,22 @@ name = "sd_fat"
path = "src/bin/sd_fat.rs"
harness = false
# SD/FAT primitive-op micro-benchmark — attributes the ~700 ms/loose-object write
# floor (see docs/tradeoff-curves/sync-commit-staging.md). Pure SD, no git feature.
# Flash with `just flash-bench`.
[[bin]]
name = "sd_bench"
path = "src/bin/sd_bench.rs"
harness = false
# git-level micro-benchmark — localizes the ~700 ms/object libgit2 overhead (see
# docs/tradeoff-curves/sync-commit-staging.md). Needs the `git` feature.
[[bin]]
name = "git_bench"
path = "src/bin/git_bench.rs"
harness = false
required-features = ["git"]
# Spike 9 — boot splash. Standalone bench program that paints the Typoena
# wordmark-in-a-circle on the EPD. No git, no SD, no Wi-Fi. Flash with
# `just flash-splash`.

View File

@@ -41,6 +41,12 @@ file(GLOB LG2_SRCS
"${LG2}/deps/pcre/*.c"
"${LG2}/deps/zlib/*.c"
)
# streams/mbedtls.c is replaced by our patched copy: its wrap() error path
# double-freed the socket stream (tlsf abort on device when ssl_setup failed
# under memory pressure). See esp_mbedtls_stream.c's header for the one-hunk
# delta; keep the copy in lockstep on submodule bumps.
list(REMOVE_ITEM LG2_SRCS "${LG2}/src/libgit2/streams/mbedtls.c")
list(APPEND LG2_SRCS
"${LG2}/src/util/allocators/failalloc.c"
"${LG2}/src/util/allocators/stdalloc.c"
@@ -48,6 +54,7 @@ list(APPEND LG2_SRCS
"${LG2}/src/util/hash/mbedtls.c" # SHA1 + SHA256 via mbedtls
"${CMAKE_CURRENT_LIST_DIR}/esp_map.c" # p_mmap via malloc+read (no <sys/mman.h>)
"${CMAKE_CURRENT_LIST_DIR}/esp_stubs.c" # getuid/readlink/utimes/... stubs
"${CMAKE_CURRENT_LIST_DIR}/esp_mbedtls_stream.c" # streams/mbedtls.c + double-free fix
# NOTE: unix/map.c replaced by esp_map.c — picolibc has no <sys/mman.h>.
# NOTE: unix/process.c deliberately excluded — needs fork()/sys/wait.h,
# only used by the SSH-exec transport we don't enable.

View File

@@ -7,9 +7,19 @@
* Allocations go through git__malloc, so with PSRAM in the heap they land in
* the 8 MB external RAM rather than the ~340 KB internal DRAM.
*
* Limitation: writable/shared mappings are not written back (libgit2 does not
* mmap for writing in the paths we exercise). If that ever changes it will
* surface at runtime, not here.
* There is deliberately NO cache here. A window cache was built and removed
* on 2026-07-12: across four instrumented real-repo bench runs it scored
* exactly 0 hits, because libgit2's mwindow layer reuses its open windows and
* only genuinely new (offset, len) ranges ever reach p_mmap — and the one
* memory bug it "fixed" (7.4 MB resident starving zlib) was caused by the
* cache itself holding buffers past p_munmap. Free-at-munmap keeps
* GIT_OPT_SET_MWINDOW_MAPPED_LIMIT honest by construction: when libgit2
* releases a window, the memory really is back in git__malloc's pool. Full
* trail: docs/tradeoff-curves/sync-commit-staging.md (final-bench section).
* The stats counters below are kept so a future workload that *does* repeat
* ranges (push? reconcile?) can be spotted before anyone rebuilds a cache.
*
* Limitation: writable/shared mappings are not written back.
*/
#include "git2_util.h"
@@ -18,6 +28,7 @@
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <stdint.h>
int git__page_size(size_t *page_size)
{
@@ -31,10 +42,49 @@ int git__mmap_alignment(size_t *alignment)
return 0;
}
/* Diagnostics, read from the bench via esp_map_stats(). The signature predates
* the cache removal: `hits` is always 0, `misses` counts every mapping, and
* `cached_kb` now reports the LIVE mapped bytes (the mwindow working set). */
static uint32_t g_maps;
static uint64_t g_read_bytes;
static size_t g_live_bytes;
void esp_map_stats(uint32_t *hits, uint32_t *misses, uint32_t *read_kb, uint32_t *cached_kb)
{
if (hits) *hits = 0;
if (misses) *misses = g_maps;
if (read_kb) *read_kb = (uint32_t)(g_read_bytes / 1024);
if (cached_kb) *cached_kb = (uint32_t)(g_live_bytes / 1024);
}
static int read_range(int fd, off64_t offset, size_t len, unsigned char *data)
{
size_t got = 0;
if (lseek(fd, offset, SEEK_SET) < 0) {
git_error_set(GIT_ERROR_OS, "failed to seek for mmap emulation");
return -1;
}
while (got < len) {
ssize_t n = read(fd, data + got, len - got);
if (n < 0) {
if (errno == EINTR)
continue;
git_error_set(GIT_ERROR_OS, "failed to read for mmap emulation");
return -1;
}
if (n == 0)
break; /* short file: zero-fill the tail, like a real mapping */
got += (size_t)n;
}
if (got < len)
memset(data + got, 0, len - got);
return 0;
}
int p_mmap(git_map *out, size_t len, int prot, int flags, int fd, off64_t offset)
{
unsigned char *data;
size_t got = 0;
GIT_UNUSED(prot);
GIT_UNUSED(flags);
@@ -45,29 +95,13 @@ int p_mmap(git_map *out, size_t len, int prot, int flags, int fd, off64_t offset
data = git__malloc(len);
GIT_ERROR_CHECK_ALLOC(data);
if (lseek(fd, offset, SEEK_SET) < 0) {
git_error_set(GIT_ERROR_OS, "failed to seek for mmap emulation");
if (read_range(fd, offset, len, data) < 0) {
git__free(data);
return -1;
}
while (got < len) {
ssize_t n = read(fd, data + got, len - got);
if (n < 0) {
if (errno == EINTR)
continue;
git_error_set(GIT_ERROR_OS, "failed to read for mmap emulation");
git__free(data);
return -1;
}
if (n == 0)
break; /* short file: zero-fill the tail, like a real mapping */
got += (size_t)n;
}
if (got < len)
memset(data + got, 0, len - got);
g_maps++;
g_read_bytes += len;
g_live_bytes += len;
out->data = data;
out->len = len;
@@ -77,7 +111,12 @@ int p_mmap(git_map *out, size_t len, int prot, int flags, int fd, off64_t offset
int p_munmap(git_map *map)
{
GIT_ASSERT_ARG(map);
git__free(map->data);
if (g_live_bytes >= map->len)
g_live_bytes -= map->len;
else
g_live_bytes = 0;
map->data = NULL;
map->len = 0;
return 0;

View File

@@ -0,0 +1,498 @@
/*
* Copyright (C) the libgit2 contributors. All rights reserved.
*
* This file is part of libgit2, distributed under the GNU GPL v2 with
* a Linking Exception. For full terms see the included COPYING file.
*/
/*
* esp_mbedtls_stream.c — verbatim copy of the vendored
* src/libgit2/streams/mbedtls.c (v1.9.4) with ONE fix; it replaces the
* vendored file in CMakeLists.txt (same pattern as esp_map.c).
*
* THE FIX (2026-07-13): mbedtls_stream_wrap()'s `out_err` path closed and
* freed `st->io` — the caller's socket stream — but every caller frees that
* stream on error too (git_mbedtls_stream_new does close+free right after),
* and wrap's OTHER error paths (calloc/strdup failures) do NOT free it, so
* the caller cannot compensate either way. When mbedtls_ssl_setup failed on
* the device (internal-RAM exhaustion during the first real-repo push), the
* double git__free tripped tlsf ("block already marked as free") and reset
* the chip instead of surfacing a clean error. Delta from vendor: the
* `out_err` label no longer touches st->io — on error, ownership of `in`
* stays with the caller, consistently — and it frees the git__malloc'd
* st->ssl struct the vendored path leaked.
*
* Keep this file in lockstep with the vendored one on submodule bumps (diff
* against it; the delta must stay this one hunk).
*/
#include "streams/mbedtls.h"
#ifdef GIT_MBEDTLS
#include <ctype.h>
#include "runtime.h"
#include "stream.h"
#include "streams/socket.h"
#include "git2/transport.h"
#include "util.h"
#ifndef GIT_DEFAULT_CERT_LOCATION
#define GIT_DEFAULT_CERT_LOCATION NULL
#endif
/* Work around C90-conformance issues */
#if !defined(__STDC_VERSION__) || (__STDC_VERSION__ < 199901L)
# if defined(_MSC_VER)
# define inline __inline
# elif defined(__GNUC__)
# define inline __inline__
# else
# define inline
# endif
#endif
#include <mbedtls/ssl.h>
#include <mbedtls/error.h>
#include <mbedtls/entropy.h>
#include <mbedtls/ctr_drbg.h>
#undef inline
#define GIT_SSL_DEFAULT_CIPHERS "TLS1-3-AES-128-GCM-SHA256:TLS1-3-AES-256-GCM-SHA384:TLS1-3-CHACHA20-POLY1305-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-ECDSA-WITH-AES-256-GCM-SHA384:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-ECDHE-ECDSA-WITH-CHACHA20-POLY1305-SHA256:TLS-ECDHE-RSA-WITH-CHACHA20-POLY1305-SHA256:TLS-DHE-RSA-WITH-AES-128-GCM-SHA256:TLS-DHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-CHACHA20-POLY1305-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA256:TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA:TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA:TLS-ECDHE-ECDSA-WITH-AES-256-CBC-SHA384:TLS-ECDHE-RSA-WITH-AES-256-CBC-SHA384:TLS-ECDHE-ECDSA-WITH-AES-256-CBC-SHA:TLS-ECDHE-RSA-WITH-AES-256-CBC-SHA:TLS-DHE-RSA-WITH-AES-128-CBC-SHA256:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256:TLS-RSA-WITH-AES-128-GCM-SHA256:TLS-RSA-WITH-AES-256-GCM-SHA384:TLS-RSA-WITH-AES-128-CBC-SHA256:TLS-RSA-WITH-AES-256-CBC-SHA256:TLS-RSA-WITH-AES-128-CBC-SHA:TLS-RSA-WITH-AES-256-CBC-SHA"
#define GIT_SSL_DEFAULT_CIPHERS_COUNT 28
static int ciphers_list[GIT_SSL_DEFAULT_CIPHERS_COUNT];
static bool initialized = false;
static mbedtls_ssl_config mbedtls_config;
static mbedtls_ctr_drbg_context mbedtls_rng;
static mbedtls_entropy_context mbedtls_entropy;
static bool has_ca_chain = false;
static mbedtls_x509_crt mbedtls_ca_chain;
/**
* This function aims to clean-up the SSL context which
* we allocated.
*/
static void shutdown_ssl(void)
{
if (has_ca_chain) {
mbedtls_x509_crt_free(&mbedtls_ca_chain);
has_ca_chain = false;
}
if (initialized) {
mbedtls_ctr_drbg_free(&mbedtls_rng);
mbedtls_ssl_config_free(&mbedtls_config);
mbedtls_entropy_free(&mbedtls_entropy);
initialized = false;
}
}
int git_mbedtls_stream_global_init(void)
{
int loaded = 0;
char *crtpath = GIT_DEFAULT_CERT_LOCATION;
struct stat statbuf;
size_t ciphers_known = 0;
char *cipher_name = NULL;
char *cipher_string = NULL;
char *cipher_string_tmp = NULL;
mbedtls_ssl_config_init(&mbedtls_config);
mbedtls_entropy_init(&mbedtls_entropy);
mbedtls_ctr_drbg_init(&mbedtls_rng);
if (mbedtls_ssl_config_defaults(&mbedtls_config,
MBEDTLS_SSL_IS_CLIENT,
MBEDTLS_SSL_TRANSPORT_STREAM,
MBEDTLS_SSL_PRESET_DEFAULT) != 0) {
git_error_set(GIT_ERROR_SSL, "failed to initialize mbedTLS");
goto cleanup;
}
/* configure TLSv1.1 or better */
#ifdef MBEDTLS_SSL_MINOR_VERSION_2
mbedtls_ssl_conf_min_version(&mbedtls_config, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_2);
#endif
/* verify_server_cert is responsible for making the check.
* OPTIONAL because REQUIRED drops the certificate as soon as the check
* is made, so we can never see the certificate and override it. */
mbedtls_ssl_conf_authmode(&mbedtls_config, MBEDTLS_SSL_VERIFY_OPTIONAL);
/* set the list of allowed ciphersuites */
ciphers_known = 0;
cipher_string = cipher_string_tmp = git__strdup(GIT_SSL_DEFAULT_CIPHERS);
GIT_ERROR_CHECK_ALLOC(cipher_string);
while ((cipher_name = git__strtok(&cipher_string_tmp, ":")) != NULL) {
int cipherid = mbedtls_ssl_get_ciphersuite_id(cipher_name);
if (cipherid == 0) continue;
if (ciphers_known >= ARRAY_SIZE(ciphers_list)) {
git_error_set(GIT_ERROR_SSL, "out of cipher list space");
goto cleanup;
}
ciphers_list[ciphers_known++] = cipherid;
}
git__free(cipher_string);
if (!ciphers_known) {
git_error_set(GIT_ERROR_SSL, "no cipher could be enabled");
goto cleanup;
}
mbedtls_ssl_conf_ciphersuites(&mbedtls_config, ciphers_list);
/* Seeding the random number generator */
if (mbedtls_ctr_drbg_seed(&mbedtls_rng, mbedtls_entropy_func,
&mbedtls_entropy, NULL, 0) != 0) {
git_error_set(GIT_ERROR_SSL, "failed to initialize mbedTLS entropy pool");
goto cleanup;
}
mbedtls_ssl_conf_rng(&mbedtls_config, mbedtls_ctr_drbg_random, &mbedtls_rng);
/* load default certificates */
if (crtpath != NULL && stat(crtpath, &statbuf) == 0 && S_ISREG(statbuf.st_mode))
loaded = (git_mbedtls__set_cert_location(crtpath, NULL) == 0);
if (!loaded && crtpath != NULL && stat(crtpath, &statbuf) == 0 && S_ISDIR(statbuf.st_mode))
loaded = (git_mbedtls__set_cert_location(NULL, crtpath) == 0);
initialized = true;
return git_runtime_shutdown_register(shutdown_ssl);
cleanup:
mbedtls_ctr_drbg_free(&mbedtls_rng);
mbedtls_ssl_config_free(&mbedtls_config);
mbedtls_entropy_free(&mbedtls_entropy);
return -1;
}
static int bio_read(void *b, unsigned char *buf, size_t len)
{
git_stream *io = (git_stream *) b;
return (int) git_stream_read(io, buf, min(len, INT_MAX));
}
static int bio_write(void *b, const unsigned char *buf, size_t len)
{
git_stream *io = (git_stream *) b;
return (int) git_stream_write(io, (const char *)buf, min(len, INT_MAX), 0);
}
static int ssl_set_error(mbedtls_ssl_context *ssl, int error)
{
char errbuf[512];
int ret = -1;
GIT_ASSERT(error != MBEDTLS_ERR_SSL_WANT_READ);
GIT_ASSERT(error != MBEDTLS_ERR_SSL_WANT_WRITE);
if (error != 0)
mbedtls_strerror( error, errbuf, 512 );
switch(error) {
case 0:
git_error_set(GIT_ERROR_SSL, "SSL error: unknown error");
break;
case MBEDTLS_ERR_X509_CERT_VERIFY_FAILED:
git_error_set(GIT_ERROR_SSL, "SSL error: %#04x [%x] - %s", error, mbedtls_ssl_get_verify_result(ssl), errbuf);
ret = GIT_ECERTIFICATE;
break;
default:
git_error_set(GIT_ERROR_SSL, "SSL error: %#04x - %s", error, errbuf);
}
return ret;
}
static int ssl_teardown(mbedtls_ssl_context *ssl)
{
int ret = 0;
ret = mbedtls_ssl_close_notify(ssl);
if (ret < 0)
ret = ssl_set_error(ssl, ret);
mbedtls_ssl_free(ssl);
return ret;
}
static int verify_server_cert(mbedtls_ssl_context *ssl)
{
int ret = -1;
if ((ret = mbedtls_ssl_get_verify_result(ssl)) != 0) {
char vrfy_buf[512];
int len = mbedtls_x509_crt_verify_info(vrfy_buf, sizeof(vrfy_buf), "", ret);
if (len >= 1) vrfy_buf[len - 1] = '\0'; /* Remove trailing \n */
git_error_set(GIT_ERROR_SSL, "the SSL certificate is invalid: %#04x - %s", ret, vrfy_buf);
return GIT_ECERTIFICATE;
}
return 0;
}
typedef struct {
git_stream parent;
git_stream *io;
int owned;
bool connected;
char *host;
mbedtls_ssl_context *ssl;
git_cert_x509 cert_info;
} mbedtls_stream;
static int mbedtls_connect(git_stream *stream)
{
int ret;
mbedtls_stream *st = (mbedtls_stream *) stream;
if (st->owned && (ret = git_stream_connect(st->io)) < 0)
return ret;
st->connected = true;
mbedtls_ssl_set_hostname(st->ssl, st->host);
mbedtls_ssl_set_bio(st->ssl, st->io, bio_write, bio_read, NULL);
if ((ret = mbedtls_ssl_handshake(st->ssl)) != 0)
return ssl_set_error(st->ssl, ret);
return verify_server_cert(st->ssl);
}
static int mbedtls_certificate(git_cert **out, git_stream *stream)
{
unsigned char *encoded_cert;
mbedtls_stream *st = (mbedtls_stream *) stream;
const mbedtls_x509_crt *cert = mbedtls_ssl_get_peer_cert(st->ssl);
if (!cert) {
git_error_set(GIT_ERROR_SSL, "the server did not provide a certificate");
return -1;
}
/* Retrieve the length of the certificate first */
if (cert->raw.len == 0) {
git_error_set(GIT_ERROR_NET, "failed to retrieve certificate information");
return -1;
}
encoded_cert = git__malloc(cert->raw.len);
GIT_ERROR_CHECK_ALLOC(encoded_cert);
memcpy(encoded_cert, cert->raw.p, cert->raw.len);
st->cert_info.parent.cert_type = GIT_CERT_X509;
st->cert_info.data = encoded_cert;
st->cert_info.len = cert->raw.len;
*out = &st->cert_info.parent;
return 0;
}
static int mbedtls_set_proxy(git_stream *stream, const git_proxy_options *proxy_options)
{
mbedtls_stream *st = (mbedtls_stream *) stream;
return git_stream_set_proxy(st->io, proxy_options);
}
static ssize_t mbedtls_stream_write(git_stream *stream, const char *data, size_t len, int flags)
{
mbedtls_stream *st = (mbedtls_stream *) stream;
int written;
GIT_UNUSED(flags);
/*
* `mbedtls_ssl_write` can only represent INT_MAX bytes
* written via its return value. We thus need to clamp
* the maximum number of bytes written.
*/
len = min(len, INT_MAX);
if ((written = mbedtls_ssl_write(st->ssl, (const unsigned char *)data, len)) <= 0)
return ssl_set_error(st->ssl, written);
return written;
}
static ssize_t mbedtls_stream_read(git_stream *stream, void *data, size_t len)
{
mbedtls_stream *st = (mbedtls_stream *) stream;
int ret;
if ((ret = mbedtls_ssl_read(st->ssl, (unsigned char *)data, len)) <= 0)
ssl_set_error(st->ssl, ret);
return ret;
}
static int mbedtls_stream_close(git_stream *stream)
{
mbedtls_stream *st = (mbedtls_stream *) stream;
int ret = 0;
if (st->connected && (ret = ssl_teardown(st->ssl)) != 0)
return -1;
st->connected = false;
return st->owned ? git_stream_close(st->io) : 0;
}
static void mbedtls_stream_free(git_stream *stream)
{
mbedtls_stream *st = (mbedtls_stream *) stream;
if (st->owned)
git_stream_free(st->io);
git__free(st->host);
git__free(st->cert_info.data);
mbedtls_ssl_free(st->ssl);
git__free(st->ssl);
git__free(st);
}
static int mbedtls_stream_wrap(
git_stream **out,
git_stream *in,
const char *host,
int owned)
{
mbedtls_stream *st;
int error;
st = git__calloc(1, sizeof(mbedtls_stream));
GIT_ERROR_CHECK_ALLOC(st);
st->io = in;
st->owned = owned;
st->ssl = git__malloc(sizeof(mbedtls_ssl_context));
GIT_ERROR_CHECK_ALLOC(st->ssl);
mbedtls_ssl_init(st->ssl);
if (mbedtls_ssl_setup(st->ssl, &mbedtls_config)) {
git_error_set(GIT_ERROR_SSL, "failed to create ssl object");
error = -1;
goto out_err;
}
st->host = git__strdup(host);
GIT_ERROR_CHECK_ALLOC(st->host);
st->parent.version = GIT_STREAM_VERSION;
st->parent.encrypted = 1;
st->parent.proxy_support = git_stream_supports_proxy(st->io);
st->parent.connect = mbedtls_connect;
st->parent.certificate = mbedtls_certificate;
st->parent.set_proxy = mbedtls_set_proxy;
st->parent.read = mbedtls_stream_read;
st->parent.write = mbedtls_stream_write;
st->parent.close = mbedtls_stream_close;
st->parent.free = mbedtls_stream_free;
*out = (git_stream *) st;
return 0;
out_err:
/* ESP FIX: do NOT close/free st->io here — on error the caller keeps
* ownership of `in` (git_mbedtls_stream_new closes+frees it right after;
* the vendored code freed it here too → double free → tlsf abort). */
mbedtls_ssl_free(st->ssl);
git__free(st->ssl);
git__free(st);
return error;
}
int git_mbedtls_stream_wrap(
git_stream **out,
git_stream *in,
const char *host)
{
return mbedtls_stream_wrap(out, in, host, 0);
}
int git_mbedtls_stream_new(
git_stream **out,
const char *host,
const char *port)
{
git_stream *stream;
int error;
GIT_ASSERT_ARG(out);
GIT_ASSERT_ARG(host);
GIT_ASSERT_ARG(port);
if ((error = git_socket_stream_new(&stream, host, port)) < 0)
return error;
if ((error = mbedtls_stream_wrap(out, stream, host, 1)) < 0) {
git_stream_close(stream);
git_stream_free(stream);
}
return error;
}
int git_mbedtls__set_cert_location(const char *file, const char *path)
{
int ret = 0;
char errbuf[512];
GIT_ASSERT_ARG(file || path);
if (has_ca_chain)
mbedtls_x509_crt_free(&mbedtls_ca_chain);
mbedtls_x509_crt_init(&mbedtls_ca_chain);
if (file)
ret = mbedtls_x509_crt_parse_file(&mbedtls_ca_chain, file);
if (ret >= 0 && path)
ret = mbedtls_x509_crt_parse_path(&mbedtls_ca_chain, path);
/* mbedtls_x509_crt_parse_path returns the number of invalid certs on success */
if (ret < 0) {
mbedtls_x509_crt_free(&mbedtls_ca_chain);
mbedtls_strerror( ret, errbuf, 512 );
git_error_set(GIT_ERROR_SSL, "failed to load CA certificates: %#04x - %s", ret, errbuf);
return -1;
}
mbedtls_ssl_conf_ca_chain(&mbedtls_config, &mbedtls_ca_chain, NULL);
has_ca_chain = true;
return 0;
}
#else
#include "stream.h"
int git_mbedtls_stream_global_init(void)
{
return 0;
}
#endif

View File

@@ -80,6 +80,32 @@ flash-sd:
monitor-sd:
espflash monitor --elf {{elf_sd}}
# SD primitive-op micro-benchmark — attributes the ~700 ms/loose-object write
# floor (docs/tradeoff-curves/sync-commit-staging.md). Writes only to /sd/sdbench.
build-bench:
{{esp_env}} cargo build --release --bin sd_bench
# build + flash + monitor the SD bench
flash-bench:
{{esp_env}} cargo run --release --bin sd_bench
# serial monitor for the SD bench, with decoded backtraces
monitor-bench:
espflash monitor --elf target/xtensa-esp32s3-espidf/release/sd_bench
# git-level micro-benchmark — localizes the ~700 ms/object libgit2 overhead
# (docs/tradeoff-curves/sync-commit-staging.md). Read-mostly on /sd/repo.
build-gitbench:
{{esp_env}} {{git_env}} cargo build --release --bin git_bench --features git
# build + flash + monitor the git-level bench
flash-gitbench:
{{esp_env}} {{git_env}} cargo run --release --bin git_bench --features git
# serial monitor for the git-level bench, with decoded backtraces
monitor-gitbench:
espflash monitor --elf target/xtensa-esp32s3-espidf/release/git_bench
# Spike 9 — build the boot splash spike (no .env needed)
build-splash:
{{esp_env}} cargo build --release --bin splash
@@ -112,7 +138,7 @@ build-git-push:
# (so the `storage` FAT partition exists) and sets 16 MB flash. Uses espflash
# directly, not `cargo run`, so the table is applied only to this binary.
flash-git-push: build-git-push
espflash flash --monitor --partition-table {{partition_table}} --flash-size 16mb {{elf_git_push}}
espflash flash --monitor --baud 921600 --partition-table {{partition_table}} --flash-size 16mb {{elf_git_push}}
# serial monitor for the git push, with decoded backtraces
monitor-git-push:
@@ -125,7 +151,7 @@ build-git-sync:
# Milestone #2A — flash + monitor. Applies the same custom partition table as
# git-push (the `storage` FAT partition holds the persistent clone).
flash-git-sync: build-git-sync
espflash flash --monitor --partition-table {{partition_table}} --flash-size 16mb {{elf_git_sync}}
espflash flash --monitor --baud 921600 --partition-table {{partition_table}} --flash-size 16mb {{elf_git_sync}}
# serial monitor for the git sync, with decoded backtraces
monitor-git-sync:
@@ -161,18 +187,21 @@ ports:
# The `_`-prefixed recipes are shared internals (hidden from `just --list`).
sd_repo_dir := "repo"
# Full prep of a fresh card: copy the notes repo + write config, then eject.
# Run this once per card. <repo-src> is a clone made on this computer.
# Full prep of a fresh card: copy the notes repo, seed the git-tracked config
# files (first-time only), write the runtime config, then eject. Run this once
# per card. <repo-src> is a clone made on this computer.
init repo_src sd_volume="":
#!/usr/bin/env bash
set -euo pipefail
vol="$(just _card "{{sd_volume}}" | tail -n1)"
just _load-repo "{{repo_src}}" "$vol"
just _seed-configs "$vol"
just _write-conf "$vol" "{{repo_src}}"
just _eject "$vol"
# (Re)copy just the notes repo to /sd/repo (full clone, gitignored paths
# excluded), then eject — e.g. to refresh the card after big upstream changes.
# (Re)copy just the notes repo to /sd/repo: fast-forward the source clone from
# its remote first, then rsync it across (full clone, gitignored paths excluded)
# and eject — e.g. to refresh the card after the device pushed new notes.
load repo_src sd_volume="":
#!/usr/bin/env bash
set -euo pipefail
@@ -241,7 +270,10 @@ _card sd_volume="":
# .gitignore ignores is excluded (node_modules is 3.9 GB, gitignored, never in
# .git) — the device needs .git + the checkout (~720 MB), not the JS deps or
# secrets like firmware/.env. Copying (not a fresh `git clone` of the local
# path) preserves origin → GitHub so on-device fetch/push still work.
# path) preserves origin → GitHub so on-device fetch/push still work — except
# the URL scheme: the card copy's origin is rewritten to HTTPS at the end,
# because the device's libgit2 has no SSH transport (HTTPS+PAT over mbedTLS
# only). The source clone is never touched.
_load-repo repo_src vol:
#!/usr/bin/env bash
set -euo pipefail
@@ -259,6 +291,29 @@ _load-repo repo_src vol:
echo "warning: origin ($origin) != TW_REMOTE_URL (${TW_REMOTE_URL})"
fi
# Refresh the source clone from its remote before mirroring, so the card
# carries the latest — e.g. notes the device pushed since the last load.
# Fast-forward-only and best-effort: a dirty tree, diverged history, detached
# HEAD, or being offline just warns and copies the current on-disk state
# rather than aborting the card prep. Set TW_NO_PULL=1 to skip the pull.
case "$origin" in
https://*|git@*|ssh://*|git://*)
if [ -n "${TW_NO_PULL:-}" ]; then
echo "TW_NO_PULL set — skipping pull, copying current on-disk state"
else
branch="$(git -C "$src" symbolic-ref --quiet --short HEAD || true)"
if [ -z "$branch" ]; then
echo "warning: '$src' is in detached HEAD — skipping pull, copying current state" >&2
elif git -C "$src" pull --ff-only origin "$branch"; then
:
else
echo "warning: 'git pull --ff-only origin $branch' failed (dirty, diverged, or offline) —" >&2
echo " copying the current on-disk state instead" >&2
fi
fi
;;
esac
dest="{{vol}}/{{sd_repo_dir}}"
echo "copying repo: $src -> $dest"
mkdir -p "$dest"
@@ -280,6 +335,101 @@ _load-repo repo_src vol:
# -P for progress on the ~700 MB copy. Scoped to repo/, never the card root.
rsync -rtP --delete --modify-window=1 --exclude-from="$ignore_list" "$src/" "$dest/"
# The device's libgit2 speaks HTTPS+PAT only (mbedTLS — no SSH transport),
# so an SSH origin copied verbatim fails every on-device push/fetch with
# "unsupported URL protocol" (found the hard way, 2026-07-13). Rewrite the
# CARD copy's origin to the HTTPS equivalent; the source clone keeps its
# own URL. Its embedded trust store carries GitHub's roots, hence the
# non-github warning.
case "$origin" in
git@*:*)
host="${origin#git@}"; host="${host%%:*}"
card_origin="https://$host/${origin#git@*:}"
;;
ssh://git@*)
rest="${origin#ssh://git@}"
card_origin="https://${rest%%[:/]*}/${rest#*/}"
;;
https://*) card_origin="$origin" ;;
*) card_origin="" ;;
esac
if [ -n "$card_origin" ]; then
if [ "$card_origin" != "$origin" ]; then
echo "rewriting card origin for the device: $origin -> $card_origin"
fi
git -C "$dest" remote set-url origin "$card_origin"
case "$card_origin" in
https://github.com/*) ;;
*) echo "warning: origin host isn't github.com — the device's embedded trust store" \
"only carries GitHub roots, so on-device TLS will fail without its CA" ;;
esac
elif [ -n "$origin" ]; then
echo "warning: origin '$origin' has no HTTPS equivalent I can derive —"
echo " on-device push/fetch will fail; set it by hand:"
echo " git -C '$dest' remote set-url origin https://github.com/<owner>/<repo>.git"
fi
# First-time setup: seed the two git-tracked config files into the card's repo/
# so they commit + sync on the device's first `:sync`. Writes ONLY a file that is
# absent — a card whose clone already carries `.typoena.toml` /
# `.typoena.snippets.json` keeps its own, so a re-`init` never clobbers a synced
# library. The starter `.typoena.toml` mirrors the editor defaults; the snippet
# library is assembled from the curated catalog (snippets-catalog/, opt-in per
# group) with `jq`. Non-interactive runs take every default (all groups).
_seed-configs vol:
#!/usr/bin/env bash
set -euo pipefail
repo="{{vol}}/{{sd_repo_dir}}"
catalog="{{justfile_directory()}}/snippets-catalog"
interactive=1; [ -t 0 ] || interactive=0
# ask_yn "label" → 0 (yes) / 1 (no); default yes, non-interactive → yes.
ask_yn() {
local ans
if [ "$interactive" = 0 ]; then return 0; fi
read -r -p " $1 [Y/n]: " ans || ans=""
[[ "${ans:-Y}" =~ ^[Yy] ]]
}
# ── starter prefs (.typoena.toml) ────────────────────────────────────────
prefs="$repo/.typoena.toml"
if [ -e "$prefs" ]; then
echo "keeping existing .typoena.toml"
elif ask_yn "seed a starter .typoena.toml (editor defaults)?"; then
# printf per line (matching _write-conf), so no heredoc indentation trap.
{
printf '# Typoena editor preferences — hand-editable, git-tracked.\n'
printf '# Edit here, or change live from the Cmd-P palette (type `>`).\n'
printf 'save_on_idle = true\n'
printf 'format_on_save = true\n'
printf 'line_numbers = true\n'
printf 'theme = "light"\n'
printf 'auto_sync = "10m"\n'
} > "$prefs"
echo "wrote .typoena.toml"
fi
# ── snippet library (.typoena.snippets.json) from the catalog ────────────
snips="$repo/.typoena.snippets.json"
if [ -e "$snips" ]; then
echo "keeping existing .typoena.snippets.json"
elif ! command -v jq >/dev/null 2>&1; then
echo "warning: jq not found — skipping the snippet catalog (install jq, or" >&2
echo " hand-write $snips later)" >&2
else
echo "snippet catalog — choose groups to include:" >&2
picked=()
ask_yn "Symbols (→ ≠ × · ° € œ)" && picked+=("$catalog/symbols.json")
ask_yn "Structure (todo, link, img, table, code)" && picked+=("$catalog/structure.json")
ask_yn "Prose (booknotes, reference, bias, capture, standard)" && picked+=("$catalog/prose.json")
if [ "${#picked[@]}" -gt 0 ]; then
jq -s 'add' "${picked[@]}" > "$snips"
echo "wrote .typoena.snippets.json ($(jq 'length' "$snips") snippets, ${#picked[@]} group(s))"
else
echo "no groups chosen — skipping .typoena.snippets.json"
fi
fi
# Resolve + write <vol>/typoena.conf (Wi-Fi + PAT + git identity). Each value
# runs a ladder: firmware/.env (loaded via `set dotenv-load`) → derived from
# tools already on the machine (git config, gh, the active Wi-Fi network + its

View File

@@ -44,9 +44,42 @@ CONFIG_SPIRAM=y
CONFIG_SPIRAM_MODE_OCT=y
CONFIG_SPIRAM_USE_MALLOC=y
# FatFS fast seek (sync-latency root cause, 2026-07-12). Without it, lseek
# resolves by walking the file's FAT cluster chain over SPI — forward from the
# current position, from the CHAIN HEAD on any backward seek. sd_bench measured
# ~190 ms per long seek into the 263 MB packfile (5.8 ms at offset 0), and
# libgit2 pays ~8 such seeks per loose-object write via p_mmap's lseek+read →
# the ~1.5 s/object commit cost (docs/tradeoff-curves/sync-commit-staging.md).
# Fast seek builds an in-memory cluster-link map (CLMT) per READ-mode file,
# making lseek O(1); write-mode files transparently fall back to the walk. The
# buffer is per-open-file, 4 B × size: 256 words = 1 KB, covering ~127 fragments
# (the default 64 covers ~31 — a freshly-provisioned pack is near-contiguous,
# but don't let card aging silently disable the fix; vfs_fat falls back to slow
# seeks when the map doesn't fit).
CONFIG_FATFS_USE_FASTSEEK=y
CONFIG_FATFS_FAST_SEEK_BUFFER_SIZE=256
# Silence the legacy-I2C boot warning. esp-idf-hal's i2c.rs is always compiled
# and binds the old `driver/i2c.h` API, so the legacy driver's TU (and its
# `__attribute__((constructor))` deprecation warning) gets linked into every
# image — even though Typoena uses no I2C at all (EPD/SD are SPI, keyboard is
# USB). This flag wraps that constructor out. Safe here: we link neither the
# new i2c_master driver nor call the old one, so the conflict-abort check it
# also removes can never fire.
CONFIG_I2C_SKIP_LEGACY_CONFLICT_CHECK=y
# TLS trust store (Spike 6 — Wi-Fi + TLS, the gate for Spike 7 git push).
# The certificate bundle backs esp_crt_bundle_attach so an HTTPS GET to
# api.github.com validates against real roots. FULL rather than the common
# subset so a less common CA in the chain can't surprise us on the bench.
CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y
CONFIG_MBEDTLS_CERTIFICATE_BUNDLE_DEFAULT_FULL=y
# mbedTLS allocations go to PSRAM. The default (internal-only) needs ~33 KB of
# contiguous internal RAM per TLS connection (two ~17 KB I/O buffers + contexts)
# at the moment `:sync` pushes — with Wi-Fi, USB host, the editor and libgit2
# all resident, mbedtls_ssl_setup failed exactly there on the first real-repo
# push (2026-07-13; the failure then tripped the vendored stream double-free —
# see components/libgit2/esp_mbedtls_stream.c). TLS buffers are CPU-only data,
# so PSRAM is safe; the handshake is network-bound, not memory-bandwidth-bound.
CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=y

View File

@@ -0,0 +1,58 @@
{
"Book notes": {
"prefix": "booknotes",
"body": ["# $1", "", "## $2 - $3", "", "## What the book is about", ""],
"description": "Reading notes (title / author - year)"
},
"Reference block": {
"prefix": "reference",
"body": ["___", "", "## References", ""],
"description": "Reference block"
},
"Cognitive bias": {
"prefix": "bias",
"body": [
"# $1",
"",
"## Misconception",
"",
"$3",
"",
"## The reality",
"",
"$4",
"",
"___",
"",
"## References",
"",
"- You are not so smart - p. $2",
""
],
"description": "Cognitive-bias note"
},
"Capture": {
"prefix": "capture",
"body": ["# $1", "", "## Captured inspiration", "", "$2", ""],
"description": "Inspiration capture"
},
"Standard": {
"prefix": "standard",
"body": [
"# $1",
"",
"## Key points",
"",
"## Mistakes to avoid",
"",
"___",
"",
"## Examples",
"",
"___",
"",
"## References"
],
"description": "Standard / playbook"
}
}

View File

@@ -0,0 +1,27 @@
{
"Todo item": {
"prefix": "todo",
"body": "- [ ] ",
"description": "Task-list item"
},
"Link": {
"prefix": "link",
"body": "[$1]($2)$0",
"description": "Inline link"
},
"Image": {
"prefix": "img",
"body": "![$1]($2)$0",
"description": "Image"
},
"Table": {
"prefix": "table",
"body": ["| $1 | $2 |", "|-|-|", "| $3 | $4 |"],
"description": "Two-column table"
},
"Code block": {
"prefix": "code",
"body": ["```$1", "$2", "```"],
"description": "Fenced code block"
}
}

View File

@@ -0,0 +1,37 @@
{
"Arrow": {
"prefix": "arrow",
"body": "→",
"description": "Rightwards arrow (→)"
},
"Not equal": {
"prefix": "neq",
"body": "≠",
"description": "Not-equal sign (≠)"
},
"Times": {
"prefix": "times",
"body": "×",
"description": "Multiplication sign (×)"
},
"Middle dot": {
"prefix": "middot",
"body": "·",
"description": "Middle dot (·), inclusive writing"
},
"Degree": {
"prefix": "deg",
"body": "°",
"description": "Degree sign (°)"
},
"Euro": {
"prefix": "euro",
"body": "€",
"description": "Euro sign (€)"
},
"OE ligature": {
"prefix": "edanso",
"body": "œ",
"description": "Ligature œ (e dans l'o)"
}
}

View File

@@ -0,0 +1,371 @@
//! git-level micro-benchmark — localizes the ~700 ms/object libgit2 overhead the
//! `:sync` commit split showed (2026-07-12), now that `sd_bench` proved the raw
//! card does a *full* loose-object write (stat+create+write+rename) in ~86 ms.
//! The ~8× gap between that and `write_tree`'s 710 ms lives inside libgit2, not
//! FAT — this bench times the git2 ODB/index primitives in isolation to find it.
//!
//! HEADLINE OP (since the 2026-07-12 real-repo run): `splice stage→tree` — the
//! O(depth) TreeBuilder walk that replaces the index-based commit entirely
//! (docs/tradeoff-curves/sync-commit-staging.md). It runs FIRST so its first iteration is
//! the cold number; acceptance bar: **sub-second cold on the real 570 MB-pack
//! clone, heap staying healthy**. The index paths it supersedes run LAST, for
//! regression tracking — they previously OOM'd, and a late crash can't cost the
//! splice data.
//!
//! Read-mostly on `/sd/repo`: the only writes are unreferenced ("orphan") loose
//! blobs/trees/commits — never reachable from a ref, so never pushed, and
//! gc-able. Safe on the test card.
//!
//! Flash with `just flash-gitbench` (needs the `git` feature; env in the recipe).
use std::time::Instant;
use anyhow::{Context, Result};
use esp_idf_svc::hal::delay::FreeRtos;
use git2::{IndexEntry, IndexTime, ObjectType, Oid, Repository, Signature, Tree};
use firmware::git_sync::GIT_STACK;
use firmware::persistence::{Storage, REPO_DIR};
const BUILD_TAG: &str = concat!("build ", env!("BUILD_TIME"), " @", env!("BUILD_GIT"));
/// Iterations per op. Small — some ops write to the card, and the first vs rest
/// spread (min vs max) is itself the signal (e.g. cold vs warm, write vs
/// freshen-skip). Kept low (3) on the real 570 MB-pack clone so a slow op still
/// finishes in seconds.
const N: usize = 3;
fn main() -> Result<()> {
esp_idf_svc::sys::link_patches();
esp_idf_svc::log::EspLogger::initialize_default();
log::info!("Typoena — git-level bench, {BUILD_TAG}");
// libgit2 nests ~67 KB of GIT_PATH_MAX stack buffers (postmortem #3), so the
// git work must run on the same 96 KB stack the real git service uses. On the
// small main-task stack `index.write()` overflows → nested panic → boot loop.
let handle = std::thread::Builder::new()
.name("git_bench".into())
.stack_size(GIT_STACK)
.spawn(run)
.expect("spawn git_bench thread");
match handle.join() {
Ok(Ok(())) => log::info!("git_bench: done"),
Ok(Err(e)) => log::error!("git_bench failed: {e:?}"),
Err(_) => log::error!("git_bench thread panicked"),
}
loop {
FreeRtos::delay_ms(1000);
}
}
fn run() -> Result<()> {
// libgit2 holds the pack + idx (+ commit-graph) fds open and reads loose
// objects on top; the editor's default 4-FD budget can't cover read_tree.
let _sd = Storage::mount_for_git().context("mounting SD")?;
// A 32 MB default mwindow window (mwindow.c) would git__malloc > PSRAM on the
// real 570 MB pack; small windows keep each p_mmap read cheap, and the
// esp_map cache keeps them from being re-read on every freshen→refresh.
// SAFETY: process-global libgit2 options, set once before any repo work.
unsafe {
git2::opts::set_mwindow_size(256 * 1024).ok();
git2::opts::set_mwindow_mapped_limit(4 * 1024 * 1024).ok();
}
// Repository open — one-time, but shows the cost of scanning .git (config,
// refs, ODB backends/packs) which every later op may implicitly refresh.
let t = Instant::now();
let repo = Repository::open(REPO_DIR)
.with_context(|| format!("opening git repo at {REPO_DIR}"))?;
log::info!("Repository::open {:.1} ms", t.elapsed().as_micros() as f64 / 1000.0);
log_map_stats("open");
// 1) THE FIX — `splice stage→tree`, the O(depth) TreeBuilder walk
// (docs/tradeoff-curves/sync-commit-staging.md): patch the edited file's ancestor
// subtree chain onto HEAD's tree; never materialise the 1179-entry index,
// never index.write(), never read_tree the whole tree. Runs FIRST so
// iteration #1 is genuinely cold (only `open` has touched the pack).
let head_tree = repo
.head()?
.peel_to_commit()
.context("HEAD → commit")?
.tree()
.context("HEAD tree")?;
// A nested path that already exists in HEAD's tree, found by an O(depth)
// descent — NOT read_tree, which is itself the 77 s op — so the splice
// REPLACES a real file and rebuilds a real ancestor chain, not just the root.
let edit_path = find_edit_path(&repo, &head_tree)?;
log::info!(
"splice: editing {} (depth {})",
edit_path.join("/"),
edit_path.len()
);
// The blob write is inside the timing: the real commit pays blob + trees, and
// it keeps the number comparable to `index-free stage→tree` below. Not
// measured here: the ref/reflog update (commit(Some("HEAD"))) — flat FAT
// writes, ~350 ms on the toy repo.
bench("splice stage→tree", |i| {
let data = format!("typoena splice bench edit #{i}\n");
let oid = repo.blob(data.as_bytes()).context("write blob")?;
let parts: Vec<&str> = edit_path.iter().map(String::as_str).collect();
splice(&repo, Some(&head_tree), &parts, Some(oid)).map(|_| ())
})?;
log_map_stats("splice");
// 2) commit(None, …) — create a commit OBJECT without moving HEAD or writing a
// reflog (update_ref = None → an orphan commit, gc-able). Isolates commit-
// object creation from the ref-update + reflog cost; splice + this projects
// the full real-repo commit. Reuses the parent's tree (no new tree needed);
// unique message each iter forces a real write.
let parent = repo.head()?.peel_to_commit().context("HEAD → commit")?;
let sig = Signature::now("typoena-bench", "bench@typoena.local").context("sig")?;
bench("commit(None) orphan obj", |i| {
let msg = format!("typoena git_bench orphan commit #{i}");
repo.commit(None, &sig, &sig, &msg, &head_tree, &[&parent])
.map(|_| ())
.context("commit(None)")
})?;
log_map_stats("commit");
// 3) odb.write(blob) in isolation — unique content each iter forces a real
// write (no freshen-skip). If ~100 ms the ODB write path is fine and any
// slow op above is in the tree/ref layer; if ~1 s the cost is inside the
// ODB write itself (deflate/sha/freshen) and the mmap cache regressed.
let odb = repo.odb().context("opening odb")?;
bench("odb.write(blob)", |i| {
let data = format!("typoena git_bench orphan blob #{i} — unique so the write is real\n");
odb.write(ObjectType::Blob, data.as_bytes())
.map(|_| ())
.context("odb.write")
})?;
log_map_stats("odb.write");
// 3b) LOCALIZE the commit-vs-blob gap. The fast-seek A/B (2026-07-12) left
// `commit(None)` at 1.7 s while `odb.write` dropped to ~0.4 s — commit
// additionally VALIDATES its parent + tree OIDs against the odb (strict
// object creation → pack header resolves) and freshens the packed tree.
// Price the two suspects, then re-bench commit + splice with strict
// creation OFF. If commit collapses toward odb.write, validation was the
// gap — and git_sync can ship with strict off (every OID it inserts
// comes from HEAD's tree or a blob it just wrote).
let parent_id = parent.id();
let tree_id = head_tree.id();
bench("odb.read_header(packed)", |i| {
let id = if i % 2 == 0 { tree_id } else { parent_id };
odb.read_header(id).map(|_| ()).context("read_header")
})?;
bench("odb.exists(missing)", |i| {
let id = Oid::from_str(&format!("{:040x}", 0xdead_beef_u64 + i as u64))?;
let _ = odb.exists(id); // miss → freshen fails → git_odb_refresh path
Ok(())
})?;
log_map_stats("probes");
// Process-global libgit2 flag; this bench owns the process.
git2::opts::strict_object_creation(false);
bench("commit(None) [strict off]", |i| {
let msg = format!("typoena git_bench strict-off commit #{i}");
repo.commit(None, &sig, &sig, &msg, &head_tree, &[&parent])
.map(|_| ())
.context("commit strict-off")
})?;
bench("splice [strict off]", |i| {
let data = format!("typoena splice strict-off edit #{i}\n");
let oid = repo.blob(data.as_bytes()).context("write blob")?;
let parts: Vec<&str> = edit_path.iter().map(String::as_str).collect();
splice(&repo, Some(&head_tree), &parts, Some(oid)).map(|_| ())
})?;
git2::opts::strict_object_creation(true);
log_map_stats("strict-off");
// 4) on-disk index LOAD (no write). Times loading all ~1179 entries from the
// card and prints the count. We deliberately do NOT bench index.write():
// it calls truncate_racily_clean, which diffs the whole working tree
// against the index and — because a fresh FAT clone makes every entry look
// "racy" (2 s mtime granularity) — re-hashes ~170 MB over SPI, up to ~10 min
// on this repo (proven 2026-07-12, index.write max 611 s). The splice
// never touches the on-disk index, so that path never runs.
bench("repo.index() load", |_| {
repo.index().map(|_| ()).context("index open")
})?;
let n_entries = repo.index().map(|i| i.len()).unwrap_or(0);
log::info!("on-disk index has {n_entries} entries");
log_map_stats("index load");
// 5) REFUTED ALTERNATIVE — the index-free in-memory-index commit
// (read_tree(HEAD) + add + write_tree_to). It dodges truncate_racily_clean
// but is still O(N_tree): the 2026-07-12 real-repo run measured ~77 s for
// the cold read_tree and drove the mmap cache to 7.4 MB (zlib OOM). Kept
// for regression tracking, run LAST so a crash here can't cost the splice
// data above. The cold read_tree is now timed explicitly (the 77 s was
// previously visible only via log timestamps); the ops above warmed only
// ~depth of the ~158 tree windows, so this is still ~cold.
let t = Instant::now();
{
let mut idx = git2::Index::new().context("Index::new")?;
idx.read_tree(&head_tree).context("seed read_tree")?;
}
log::info!(
"seed read_tree(HEAD) cold {:.1} ms",
t.elapsed().as_micros() as f64 / 1000.0
);
log_map_stats("read_tree");
// Warm repeats: windows resident → pure CPU + cache lookups.
bench("Index::new + read_tree", |_| {
let mut idx = git2::Index::new().context("Index::new")?;
idx.read_tree(&head_tree).context("read_tree")?;
Ok(())
})?;
let edit_path_bytes = edit_path.join("/").into_bytes();
bench("index-free stage→tree", |i| {
let mut idx = git2::Index::new().context("Index::new")?;
idx.read_tree(&head_tree).context("read_tree")?;
let data = format!("typoena index-free bench edit #{i}\n");
let oid = repo.blob(data.as_bytes()).context("write blob")?;
idx.add(&blob_entry(&edit_path_bytes, oid)).context("index.add")?;
idx.write_tree_to(&repo).map(|_| ()).context("write_tree_to")
})?;
log_map_stats("index-free");
Ok(())
}
/// PROTOTYPE of the real fix (destined for `git_sync::stage_and_commit`): return
/// a new tree OID equal to `base` with `path` set to `new` — `Some(blob)` to
/// add/replace, `None` to delete. Reads ~depth subtree objects, writes ~depth
/// trees; every other entry (all 1179 files, the 150 MB of images) is carried
/// forward by OID without ever being read. `base = None` builds a fresh subtree
/// chain (new file in a new directory). The git_sync version must additionally
/// drop a directory entry when a delete empties its subtree; the bench only
/// exercises replace.
fn splice(repo: &Repository, base: Option<&Tree>, path: &[&str], new: Option<Oid>) -> Result<Oid> {
let (head, rest) = path.split_first().context("splice: empty path")?;
let mut tb = repo.treebuilder(base).context("treebuilder")?;
if rest.is_empty() {
match new {
Some(oid) => {
tb.insert(*head, oid, 0o100644).context("insert blob")?;
}
None => {
let _ = tb.remove(*head); // already absent ⇒ nothing to delete
}
}
} else {
let sub = match base.and_then(|b| b.get_name(head)) {
Some(e) if e.kind() == Some(ObjectType::Tree) => {
Some(repo.find_tree(e.id()).context("loading subtree")?)
}
_ => None, // no such dir yet (or a blob in the way): build from empty
};
let new_sub = splice(repo, sub.as_ref(), rest, new)?;
tb.insert(*head, new_sub, 0o040000).context("insert subtree")?;
}
tb.write().context("treebuilder write")
}
/// Find a real file to "edit": descend the first subtree at each level (capped),
/// then take the first blob of the deepest tree reached. Reads O(depth) tree
/// objects — never `read_tree`/materialise the whole tree (that's the 77 s op
/// this bench exists to retire).
fn find_edit_path(repo: &Repository, root: &Tree) -> Result<Vec<String>> {
let mut path = Vec::new();
let mut cur_id = root.id();
for _ in 0..6 {
let cur = repo.find_tree(cur_id).context("descending tree")?;
match cur.iter().find(|e| e.kind() == Some(ObjectType::Tree)) {
Some(sub) => {
path.push(sub.name().context("non-utf8 tree name")?.to_string());
cur_id = sub.id();
}
None => break,
}
}
let cur = repo.find_tree(cur_id).context("leaf tree")?;
let blob = cur
.iter()
.find(|e| e.kind() == Some(ObjectType::Blob))
.context("no blob along the first-subtree chain — pick an edit path manually")?;
path.push(blob.name().context("non-utf8 blob name")?.to_string());
Ok(path)
}
unsafe extern "C" {
/// Counters from the p_mmap emulation in `components/libgit2/esp_map.c`.
/// Post cache-removal: `hits` is always 0, `misses` counts every mapping,
/// `cached_kb` reports the LIVE mapped bytes (the mwindow working set).
fn esp_map_stats(hits: *mut u32, misses: *mut u32, read_kb: *mut u32, cached_kb: *mut u32);
}
/// Log the p_mmap counters — mappings performed, total KB read from the card,
/// and KB currently live-mapped (should track mwindow's open windows and stay
/// well under MWINDOW_MAPPED_LIMIT now that munmap frees immediately).
fn log_map_stats(label: &str) {
let (mut hits, mut misses, mut read_kb, mut cached_kb) = (0u32, 0u32, 0u32, 0u32);
unsafe { esp_map_stats(&mut hits, &mut misses, &mut read_kb, &mut cached_kb) };
let _ = hits; // always 0 since the cache removal; slot kept for ABI stability
// Free heap spans PSRAM here; a drop toward 0 during write_tree/commit on the
// real repo would point at mwindow/idx allocation pressure (or thrash) as the
// cause of an apparent hang, not CPU.
let free_kb = unsafe { esp_idf_svc::sys::esp_get_free_heap_size() } / 1024;
log::info!(
"mmap @ {label:<11} {misses} maps, {read_kb} KB read, {cached_kb} KB live, {free_kb} KB heap free"
);
}
/// Announce, time, and summarize an op. The `→ label …` line prints BEFORE the op
/// runs, so if an op hangs on the real 570 MB-pack repo we can see which one it
/// entered — a bare `summarize` prints only after all N iters, hiding the culprit.
fn bench<F: FnMut(usize) -> Result<()>>(label: &str, op: F) -> Result<()> {
log::info!("→ {label} …");
summarize(label, time_each(op)?);
Ok(())
}
/// A minimal index entry pointing at an already-written blob — for `index.add`,
/// which (unlike `add_frombuffer`) needs no repo owner, so it works on a bare
/// in-memory index. Only `id`, `path` and `mode` feed the tree write.
fn blob_entry(path: &[u8], oid: Oid) -> IndexEntry {
IndexEntry {
ctime: IndexTime::new(0, 0),
mtime: IndexTime::new(0, 0),
dev: 0,
ino: 0,
mode: 0o100644,
uid: 0,
gid: 0,
file_size: 0,
id: oid,
flags: 0,
flags_extended: 0,
path: path.to_vec(),
}
}
/// Run `op(i)` for `i in 0..N`, returning each call's wall time in microseconds.
fn time_each<F: FnMut(usize) -> Result<()>>(mut op: F) -> Result<Vec<u64>> {
let mut times = Vec::with_capacity(N);
for i in 0..N {
let t = Instant::now();
op(i)?;
times.push(t.elapsed().as_micros() as u64);
}
Ok(times)
}
/// Log min / p50 / mean / max in ms for a set of per-call microsecond timings.
fn summarize(label: &str, mut times: Vec<u64>) {
times.sort_unstable();
let n = times.len();
let mean = times.iter().sum::<u64>() / n as u64;
let ms = |us: u64| us as f64 / 1000.0;
log::info!(
"{label:<26} min {:>6.1} p50 {:>6.1} mean {:>6.1} max {:>6.1} ms",
ms(times[0]),
ms(times[n / 2]),
ms(mean),
ms(times[n - 1]),
);
}

View File

@@ -0,0 +1,212 @@
//! SD/FAT primitive-op micro-benchmark — investigating the ~700 ms-per-loose-
//! object write floor found in the `:sync` commit split (2026-07-12, see
//! `docs/tradeoff-curves/sync-commit-staging.md`).
//!
//! The split showed a single small git loose object (`write_tree` = one tree
//! object) takes ~710 ms to land on the card, and it is **not** fsync
//! (`GIT_OPT_ENABLE_FSYNC_GITDIR` is off). libgit2's loose-object write
//! (`odb_loose.c` `loose_backend__write` → `git_filebuf_commit_at`) is, per object:
//!
//! stat(final) — freshen probe, misses (our `utimes` stub → `stat`)
//! open+write+close — a temp file (`GIT_FILEBUF_TEMPORARY`)
//! [mkdir objects/xx once per fan-out]
//! p_rename — our stub: remove(final) [ENOENT] + rename(temp → final)
//!
//! i.e. **two directory-mutating writes** (temp create + rename) per object. This
//! bench times each FAT primitive in isolation, then a composite that mirrors the
//! sequence above, so we can attribute the ~700 ms to specific ops and get a
//! baseline to compare an A1/A2 card or a 20 MHz bus against. All writes go to
//! `/sd/sdbench` (cleaned up at the end); the pack-seek op additionally opens
//! `/sd/repo`'s packfile READ-ONLY — it never writes there.
//!
//! Flash with `just flash-bench`. Needs no `.env`, no `git` feature (pure SD).
use std::fs::{self, File};
use std::io::{Read, Seek, SeekFrom, Write};
use std::time::Instant;
use anyhow::{Context, Result};
use esp_idf_svc::hal::delay::FreeRtos;
use firmware::persistence::Storage;
/// Injected by build.rs so serial output identifies the exact build.
const BUILD_TAG: &str = concat!("build ", env!("BUILD_TIME"), " @", env!("BUILD_GIT"));
/// Scratch dir on the card ROOT — outside `/sd/repo`, so a later `:sync` never
/// stages it and the user's notes are never touched.
const BENCH_DIR: &str = "/sd/sdbench";
/// Iterations per op: enough to read min/p50/mean past controller jitter, few
/// enough that total write volume stays tiny.
const N: usize = 20;
/// ~ the size of a small deflated git loose object (blob/tree/commit).
const PAYLOAD: [u8; 200] = [b'x'; 200];
fn main() -> Result<()> {
// Required once before any esp-idf-svc call (see esp-idf-template#71).
esp_idf_svc::sys::link_patches();
esp_idf_svc::log::EspLogger::initialize_default();
log::info!("Typoena — SD primitive bench, {BUILD_TAG}");
match run() {
Ok(()) => log::info!("sd_bench: done"),
Err(e) => log::error!("sd_bench failed: {e:?}"),
}
loop {
FreeRtos::delay_ms(1000);
}
}
fn run() -> Result<()> {
let sd = Storage::mount().context("mounting SD")?;
let (max_khz, real_khz) = sd.negotiated_khz();
log::info!("bus: max {max_khz} kHz, negotiated {real_khz} kHz — {N} iters, {}-byte payload", PAYLOAD.len());
// Fresh scratch dir.
let _ = fs::remove_dir_all(BENCH_DIR);
fs::create_dir_all(BENCH_DIR).with_context(|| format!("creating {BENCH_DIR}"))?;
// Warm-up: the first write after mount pays one-time settling — don't measure it.
{
let mut f = File::create(format!("{BENCH_DIR}/warmup"))?;
f.write_all(&PAYLOAD)?;
}
// 1) create + write(200B) + close, a fresh unique file each time. The drop at
// the block's end is the close (FatFS f_close flushes dir entry + data).
summarize("create+write(200B)+close", time_each(|i| {
let mut f = File::create(format!("{BENCH_DIR}/c{i}"))?;
f.write_all(&PAYLOAD)?;
Ok(())
})?);
// 2) rename c{i} -> o{i}. Sources exist from step 1 (untimed setup).
summarize("rename", time_each(|i| {
fs::rename(format!("{BENCH_DIR}/c{i}"), format!("{BENCH_DIR}/o{i}"))
.map_err(Into::into)
})?);
// 3) stat, hit.
summarize("stat (hit)", time_each(|i| {
fs::metadata(format!("{BENCH_DIR}/o{i}")).map(|_| ()).map_err(Into::into)
})?);
// 4) stat, miss (ENOENT) — the freshen-probe analogue. A read, expected cheap.
summarize("stat (miss/ENOENT)", time_each(|i| {
let _ = fs::metadata(format!("{BENCH_DIR}/nope{i}"));
Ok(())
})?);
// 5) remove o{i}.
summarize("remove", time_each(|i| {
fs::remove_file(format!("{BENCH_DIR}/o{i}")).map_err(Into::into)
})?);
// 6) Composite: the exact loose-object write sequence libgit2 performs, with a
// git-length (38-hex) final name so LFN directory-entry cost is included.
// If the model is right this lands near the ~700 ms/object from the split.
summarize("loose-object composite", time_each(|i| {
let tmp = format!("{BENCH_DIR}/tmp_obj{i}");
let fin = format!("{BENCH_DIR}/{i:038x}");
let _ = fs::metadata(&fin); // freshen probe, misses
{
let mut f = File::create(&tmp)?; // temp create + write + close
f.write_all(&PAYLOAD)?;
}
let _ = fs::remove_file(&fin); // p_rename's remove(to) — ENOENT
fs::rename(&tmp, &fin)?; // temp -> final
Ok(())
})?);
// Clean up so the card is left as we found it.
fs::remove_dir_all(BENCH_DIR).with_context(|| format!("removing {BENCH_DIR}"))?;
// 7) THE ~1.5 s LOOSE-WRITE SUSPECT (git_bench, 2026-07-12 second real-repo
// run): lseek inside a huge file. Without CONFIG_FATFS_USE_FASTSEEK,
// FatFS resolves lseek by walking the file's FAT cluster chain — forward
// from the current position, from the CHAIN HEAD on any backward seek.
// The 570 MB pack is ~36k clusters ≈ ~146 KB of FAT reads over SPI per
// long walk. `p_mmap` (esp_map.c) does lseek+read per window, and
// libgit2's freshen path probes the pack TRAILER (near the end) while
// tree windows sit at low offsets — so each loose write pays ~one full
// walk. Prediction: "@start" stays ~ms; "@end" costs ~1.5 s per iter.
// If so, the fix is CONFIG_FATFS_USE_FASTSEEK=y (fast-seek applies to
// read-mode files only — exactly how the pack is opened).
match find_pack()? {
Some(pack) => {
let len = fs::metadata(&pack)?.len();
log::info!("pack seek bench: {pack} ({} MB)", len / (1024 * 1024));
if len < 1024 * 1024 {
log::info!("pack too small to show chain-walk cost — skipping (toy card?)");
} else {
let mut f = File::open(&pack).with_context(|| format!("opening {pack}"))?;
let mut buf = vec![0u8; 4096];
// Baseline: rewind + read at the chain head — no walk to resolve.
summarize("pack seek+read 4KB @start", time_each(|_| {
f.seek(SeekFrom::Start(0))?;
f.read_exact(&mut buf)?;
Ok(())
})?);
// Rewind (cheap, measured above), then seek near the end — pays
// one full cluster-chain walk per iteration if fast-seek is off.
let high = len - 4096;
summarize("pack seek+read 4KB @end", time_each(|_| {
f.seek(SeekFrom::Start(0))?;
f.read_exact(&mut buf)?;
f.seek(SeekFrom::Start(high))?;
f.read_exact(&mut buf)?;
Ok(())
})?);
}
}
None => log::info!("no packfile under /sd/repo/.git/objects/pack — skipping seek bench"),
}
Ok(())
}
/// Largest `*.pack` under the repo's pack dir, if the card carries a clone.
/// Skips macOS AppleDouble sidecars (`._pack-*.pack`, 4 KB of Finder metadata) —
/// the Spike-14 cruft in its latest disguise.
fn find_pack() -> Result<Option<String>> {
let Ok(entries) = fs::read_dir("/sd/repo/.git/objects/pack") else {
return Ok(None);
};
let mut best: Option<(u64, String)> = None;
for e in entries.flatten() {
let p = e.path();
let name = p.file_name().and_then(|n| n.to_str()).unwrap_or("");
if name.starts_with("._") || !name.ends_with(".pack") {
continue;
}
let len = fs::metadata(&p).map(|m| m.len()).unwrap_or(0);
if best.as_ref().is_none_or(|(l, _)| len > *l) {
best = Some((len, p.to_string_lossy().into_owned()));
}
}
Ok(best.map(|(_, p)| p))
}
/// Run `op(i)` for `i in 0..N`, returning each call's wall time in microseconds.
fn time_each<F: FnMut(usize) -> Result<()>>(mut op: F) -> Result<Vec<u64>> {
let mut times = Vec::with_capacity(N);
for i in 0..N {
let t = Instant::now();
op(i)?;
times.push(t.elapsed().as_micros() as u64);
}
Ok(times)
}
/// Log min / p50 / mean / max in ms for a set of per-call microsecond timings.
fn summarize(label: &str, mut times: Vec<u64>) {
times.sort_unstable();
let n = times.len();
let mean = times.iter().sum::<u64>() / n as u64;
let ms = |us: u64| us as f64 / 1000.0;
log::info!(
"{label:<26} min {:>6.1} p50 {:>6.1} mean {:>6.1} max {:>6.1} ms",
ms(times[0]),
ms(times[n / 2]),
ms(mean),
ms(times[n - 1]),
);
}

View File

@@ -17,15 +17,23 @@
//! notes. A `/sd/repo` that isn't a valid repo is a provisioning error
//! (`just init`), surfaced as such, not papered over.
//! 3. **No synthetic content.** The spike appended a marker line; here the
//! editor has already saved the user's `notes.md` before `:sync` signals us,
//! so we just stage + commit + push what's on disk.
//! editor has already saved the user's buffers before `:sync` signals us,
//! so we just commit + push what's on disk.
//! 4. **The commit is an O(depth) TreeBuilder splice, not an index pass.**
//! The request carries the repo-relative paths saved/deleted since the last
//! confirmed publish (`Storage`'s journaled dirty set); `stage_and_commit`
//! patches exactly those onto HEAD's tree. The index pipeline it replaced
//! (`add_all` → `index.write` → `write_tree`) is O(N_tree) and measured up
//! to 611 s on the real 1179-file / 570 MB-pack clone — see
//! docs/tradeoff-curves/sync-commit-staging.md for the whole trail.
//!
//! Runs on a dedicated 96 KB thread (libgit2's init→push chain nests ~67 KB of
//! `GIT_PATH_MAX` stack buffers — see git_push.rs / postmortem #3). Config is
//! baked at build time (`TW_*`, ADR-007: v0.1 device config is compiled in).
use std::cell::RefCell;
use std::path::Path;
use std::collections::BTreeSet;
use std::fs;
use std::rc::Rc;
use std::sync::mpsc::{Receiver, Sender};
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
@@ -39,8 +47,8 @@ use esp_idf_svc::sntp::{EspSntp, SyncStatus};
use esp_idf_svc::sys;
use esp_idf_svc::wifi::{BlockingWifi, EspWifi};
use git2::{
CertificateCheckStatus, Commit, Cred, CredentialType, FetchOptions, IndexAddOption,
PushOptions, RemoteCallbacks, Repository, Signature,
CertificateCheckStatus, Commit, Cred, CredentialType, FetchOptions, ObjectType, Oid,
PushOptions, RemoteCallbacks, Repository, Signature, Tree,
};
use crate::net::connect_wifi;
@@ -73,10 +81,15 @@ const SNTP_TIMEOUT: Duration = Duration::from_secs(20);
/// now also runs here, but it's shallow next to libgit2's path-buffer nesting.
pub const GIT_STACK: usize = 96 * 1024;
/// A request to publish. The note is already saved to `/sd/repo/notes.md` by the
/// UI task before this is sent, so the request carries no payload (a future
/// multi-file publish can grow one).
pub struct PublishRequest;
/// A request to publish. The UI task has already saved every dirty buffer to
/// the card before sending this; `paths` is `Storage::take_dirty`'s snapshot —
/// the repo-relative paths saved or `:delete`d since the last confirmed
/// publish. The working tree stays the source of truth: at commit time a path
/// that exists on the card is spliced into the tree from disk, a missing one
/// is spliced out. An unchanged path is a no-op, so over-reporting is safe.
pub struct PublishRequest {
pub paths: BTreeSet<String>,
}
/// Result of a publish attempt, sent back to the UI task for the snackbar. The
/// detailed error always goes to the serial log; the panel gets a short line.
@@ -103,6 +116,22 @@ pub fn run_git_service(
rx: Receiver<PublishRequest>,
tx: Sender<PublishOutcome>,
) {
// Process-global libgit2 tuning, once, before any repo work. The 32-bit
// defaults (32 MB window / 256 MB mapped budget, mwindow.c) would
// git__malloc past PSRAM on the first pack access of the real 570 MB-pack
// clone; these are the bench-proven values (git_bench), and ~1.9 MB of
// windows stays live during git ops — the p_mmap emulation (esp_map.c)
// relies on this mapped limit being real.
// SAFETY: set on the git thread before any Repository is opened.
unsafe {
if let Err(e) = git2::opts::set_mwindow_size(256 * 1024) {
log::error!("set_mwindow_size failed ({e}); first pack access may OOM");
}
if let Err(e) = git2::opts::set_mwindow_mapped_limit(4 * 1024 * 1024) {
log::error!("set_mwindow_mapped_limit failed ({e}); first pack access may OOM");
}
}
// Lazily initialised on the first request, then reused across publishes.
let mut wifi: Option<BlockingWifi<EspWifi<'static>>> = None;
let mut modem = Some(modem);
@@ -110,7 +139,7 @@ pub fn run_git_service(
let mut clock_synced = false;
let mut tls_ready = false;
while rx.recv().is_ok() {
while let Ok(req) = rx.recv() {
let outcome = publish_cycle(
&sys_loop,
&mut wifi,
@@ -118,6 +147,7 @@ pub fn run_git_service(
&mut nvs,
&mut clock_synced,
&mut tls_ready,
&req.paths,
);
let msg = match outcome {
Ok(o) => o,
@@ -143,11 +173,22 @@ fn publish_cycle(
nvs: &mut Option<EspDefaultNvsPartition>,
clock_synced: &mut bool,
tls_ready: &mut bool,
paths: &BTreeSet<String>,
) -> Result<PublishOutcome> {
if REMOTE_URL.is_empty() || GH_USER.is_empty() || PAT.is_empty() || WIFI_SSID.is_empty() {
bail!("git config missing — set TW_WIFI_SSID / TW_REMOTE_URL / TW_GH_USER / TW_PAT in firmware/.env and rebuild");
}
// Nothing recorded dirty and origin's tracking ref already has HEAD: this
// `:sync` has nothing to do — say so without touching the radio (~150 ms
// instead of a Wi-Fi + TLS round). A stranded local commit (committed but
// never pushed, e.g. a push that failed mid-air) makes the check false and
// takes the full path below, where publish_once pushes it.
if paths.is_empty() && remote_current().unwrap_or(false) {
log::info!(":sync — no dirty paths and origin has HEAD; up to date, radio untouched");
return Ok(PublishOutcome::UpToDate);
}
// Phases are timed so a cold :sync reports where the seconds go. Wi-Fi, clock
// and TLS run only on the first sync of a session; a warm sync skips them, so
// they read 0 ms and the total collapses to just publish(fetch+commit+push).
@@ -186,7 +227,7 @@ fn publish_cycle(
}
let t_publish = Instant::now();
let outcome = publish_once()?;
let outcome = publish_once(paths)?;
log::info!(
":sync timing — wifi {wifi_ms}ms, clock {clock_ms}ms, tls {tls_ms}ms, publish(commit+push) {}ms, total {}ms",
t_publish.elapsed().as_millis(),
@@ -205,15 +246,17 @@ fn publish_cycle(
///
/// Never clones or wipes: a `/sd/repo` that isn't a valid repo is a provisioning
/// error, surfaced as such.
fn publish_once() -> Result<PublishOutcome> {
log::info!("publish started — free heap {}", free_heap());
fn publish_once(paths: &BTreeSet<String>) -> Result<PublishOutcome> {
log::info!(
"publish started — {} dirty path(s), free heap {} ({} internal)",
paths.len(),
free_heap(),
internal_free_heap()
);
let repo = Repository::open(REPO_DIR).with_context(|| {
format!("opening git repo at {REPO_DIR} — provision the card with a clone (just init) whose origin is your remote")
})?;
let Some(mut oid) = stage_and_commit(&repo)? else {
return Ok(PublishOutcome::UpToDate);
};
let branch = repo
.head()?
.shorthand()
@@ -221,17 +264,46 @@ fn publish_once() -> Result<PublishOutcome> {
.to_string();
let refspec = format!("refs/heads/{branch}:refs/heads/{branch}");
// Optimistic push. A non-fast-forward rejection means the remote moved under
// us: reconcile onto origin and replay the note on the new tip, then retry
// once. reconcile_onto_origin mixed-resets, so the just-saved note survives in
// the working tree and stage_and_commit lands it on top of origin.
if let Err(first) = try_push(&repo, &refspec) {
log::warn!("push rejected ({first}); reconciling onto origin and replaying the note");
let mut oid = match stage_and_commit(&repo, paths)? {
Some(oid) => oid,
None => {
// Nothing new to commit. Usually genuinely up to date — but a
// previous cycle may have committed and then failed to push,
// stranding a local-only commit (the old add_all path silently
// never retried those). Push whenever origin's tracking ref
// doesn't already have HEAD.
let head = repo.head()?.peel_to_commit()?.id();
if tracking_tip(&repo, &branch) == Some(head) {
return Ok(PublishOutcome::UpToDate);
}
log::info!(
"tree unchanged but origin/{branch} lacks HEAD {} — pushing the stranded commit",
short(head)
);
head
}
};
// Optimistic push. A non-fast-forward *rejection* means the remote moved
// under us: reconcile onto origin and replay the dirty paths on the new
// tip, then retry once (reconcile_onto_origin soft-resets — ref move only —
// so the notes stay on the card and stage_and_commit splices them on top of
// origin). A transport-level failure is surfaced as-is: its fetch would die
// the same way, and the commit is safe locally — the stranded-commit check
// above pushes it once the transport works again.
if let Err(failure) = try_push(&repo, &refspec) {
let rejection = match failure {
PushFailure::Rejected(msg) => msg,
PushFailure::Other(e) => return Err(e),
};
log::warn!("push rejected ({rejection}); reconciling onto origin and replaying the note");
reconcile_onto_origin(&repo, &branch).context("reconciling after a rejected push")?;
match stage_and_commit(&repo)? {
match stage_and_commit(&repo, paths)? {
Some(replayed) => {
oid = replayed;
try_push(&repo, &refspec).context("push after reconcile")?;
try_push(&repo, &refspec)
.map_err(PushFailure::into_error)
.context("push after reconcile")?;
}
// The note was already on origin (nothing to replay) — treat as done.
None => {
@@ -242,53 +314,67 @@ fn publish_once() -> Result<PublishOutcome> {
}
log::info!(
"push done — free heap {}, min-ever {}",
"push done — free heap {} ({} internal), min-ever {}",
free_heap(),
internal_free_heap(),
min_free_heap()
);
Ok(PublishOutcome::Pushed(short(oid)))
}
/// Stage the working tree and commit it on top of the current branch tip.
/// Returns the new commit id, or `None` when the tree already matches the parent
/// (nothing to publish). Called on the first attempt and again to replay the note
/// after a reconcile.
/// Build the commit for `paths` as an O(depth) TreeBuilder splice onto HEAD's
/// tree and return the new commit id or `None` when the result matches the
/// parent (nothing to publish). Called on the first attempt and again to
/// replay the dirty paths after a reconcile.
///
/// Staging is `add --all` **plus** `add -u`, which together equal `git add -A`.
/// `add_all` stages new + modified files; `update_all` re-syncs already-tracked
/// entries to the working tree, which is what actually removes an index entry
/// whose file was deleted. Spike 14 found `add_all` alone did **not** stage a
/// `:delete`d file's removal on this libgit2 build (the tree came back unchanged,
/// so the publish was a silent no-op), so the `update_all` pass is load-bearing,
/// not belt-and-braces — do not drop it.
/// This replaces the index pipeline (`add_all` → `index.write` → `write_tree`),
/// which is O(N_tree) and cannot run on the real 1179-file / 570 MB-pack clone:
/// `index.write`'s racy-clean pass re-hashes ~every entry on FAT's 2 s mtimes
/// (measured up to **611 s**), and even the index-free `read_tree` walk was
/// 77 s. The splice reads and writes only the dirty paths' ancestor chains —
/// O(depth × dirty), flat in repo size, **~22.8 s measured on the real
/// clone** — and carries every untouched entry (including the ~150 MB of
/// images) forward by OID without ever opening it. Trail + bench numbers:
/// docs/tradeoff-curves/sync-commit-staging.md.
///
/// Both run a per-path filter that drops macOS AppleDouble sidecars (`._name`)
/// and `.DS_Store` that Finder/Spotlight sprinkle onto the FAT card whenever it's
/// mounted on a Mac — without it, a blind add --all sweeps them into the commit
/// (it did once: 07d87772 shipped `._.git`, `._README.md`, `._notes.md`).
/// Filtering here fixes it for *every* repo at the device level, so no per-repo
/// `.gitignore` is needed.
fn stage_and_commit(repo: &Repository) -> Result<Option<git2::Oid>> {
let mut index = repo.index().context("opening index")?;
let mut skip_macos_cruft = |path: &Path, _matched: &[u8]| -> i32 {
match path.file_name().and_then(|n| n.to_str()) {
Some(name) if name.starts_with("._") || name == ".DS_Store" => 1, // skip
_ => 0, // add
}
};
index
.add_all(["*"], IndexAddOption::DEFAULT, Some(&mut skip_macos_cruft))
.context("staging new/modified (add --all)")?;
// Stage deletions: update_all removes index entries whose working-tree file is
// gone. add_all does not do this reliably here (Spike 14), so this is required.
index
.update_all(["*"], Some(&mut skip_macos_cruft))
.context("staging deletions (add -u)")?;
index.write().context("writing index")?;
let tree = repo.find_tree(index.write_tree().context("writing tree")?)?;
// Commit on top of the current branch tip (None on an empty/unborn remote).
/// The working tree is the source of truth: a recorded path that exists on the
/// card is spliced in from disk, a missing one is spliced out (a `:delete`).
/// Unrecorded paths are never visited — so Finder cruft (`._*`, `.DS_Store`)
/// on the FAT card can no longer ride into a commit the way it once did with
/// `add_all` (07d87772), and the old cruft filter is gone with the walk.
fn stage_and_commit(repo: &Repository, paths: &BTreeSet<String>) -> Result<Option<Oid>> {
// Commit on top of the current branch tip (None on an empty/unborn remote,
// where the splice starts from an empty base and makes a parentless commit).
let parent = repo.head().ok().and_then(|h| h.peel_to_commit().ok());
let base = match &parent {
Some(c) => Some(c.tree().context("loading HEAD tree")?),
None => None,
};
let t_splice = Instant::now();
let mut tree = base;
for path in paths {
let parts: Vec<&str> = path.split('/').filter(|p| !p.is_empty()).collect();
if parts.is_empty() {
continue;
}
let blob = match fs::read(format!("{REPO_DIR}/{path}")) {
Ok(bytes) => Some(
repo.blob(&bytes)
.with_context(|| format!("writing blob for {path}"))?,
),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => None, // deleted → splice out
Err(e) => return Err(e).with_context(|| format!("reading {path}")),
};
let spliced = splice(repo, tree.as_ref(), &parts, blob)
.with_context(|| format!("splicing {path}"))?;
tree = Some(repo.find_tree(spliced).context("loading spliced tree")?);
}
let Some(tree) = tree else {
return Ok(None); // unborn branch and nothing dirty — nothing to commit
};
let splice_ms = t_splice.elapsed().as_millis();
if let Some(p) = &parent {
if p.tree_id() == tree.id() {
log::info!("nothing to publish — tree unchanged @ {}", short(p.id()));
@@ -299,18 +385,116 @@ fn stage_and_commit(repo: &Repository) -> Result<Option<git2::Oid>> {
let sig = Signature::now(AUTHOR_NAME, AUTHOR_EMAIL).context("building signature")?;
let message = format!("Typoena publish — unix {}", now_unix());
let parents: Vec<&Commit> = parent.iter().collect();
let t_commit = Instant::now();
let oid = repo
.commit(Some("HEAD"), &sig, &sig, &message, &tree, &parents)
.context("creating commit")?;
log::info!("committed {} — free heap {}", short(oid), free_heap());
log::info!(
"commit split — splice {splice_ms}ms ({} path(s)), commit-obj {}ms; committed {} — free heap {} ({} internal)",
paths.len(),
t_commit.elapsed().as_millis(),
short(oid),
free_heap(),
internal_free_heap()
);
Ok(Some(oid))
}
/// Return a new tree = `base` with `path` set to `blob` (`Some` inserts or
/// replaces, `None` removes). Recurses down the path's subtree chain: reads
/// ~depth tree objects and writes ~depth new ones, leaving every sibling entry
/// untouched (carried by OID — never opened). A missing intermediate directory
/// is synthesized on the way down; a directory emptied by a remove is pruned
/// on the way up rather than left behind as an empty tree entry.
fn splice(repo: &Repository, base: Option<&Tree>, path: &[&str], blob: Option<Oid>) -> Result<Oid> {
let (head, rest) = path.split_first().context("splice: empty path")?;
let mut tb = repo.treebuilder(base).context("treebuilder")?;
if rest.is_empty() {
match blob {
Some(oid) => {
tb.insert(*head, oid, 0o100644)
.context("inserting blob entry")?;
}
// Removing a never-committed path is a no-op, not an error (a note
// created and deleted between two syncs).
None => {
let _ = tb.remove(*head);
}
}
} else {
let sub = match base.and_then(|b| b.get_name(head)) {
Some(e) if e.kind() == Some(ObjectType::Tree) => {
Some(repo.find_tree(e.id()).context("loading subtree")?)
}
// Absent (a new directory) or a non-tree shadowing the name —
// build the subtree from scratch either way.
_ => None,
};
let new_sub = splice(repo, sub.as_ref(), rest, blob)?;
if repo.find_tree(new_sub)?.len() == 0 {
let _ = tb.remove(*head); // the remove emptied this directory — prune it
} else {
tb.insert(*head, new_sub, 0o040000)
.context("inserting subtree entry")?;
}
}
tb.write().context("writing spliced tree")
}
/// Origin's remote-tracking tip for `branch`, if the ref exists. libgit2
/// updates it after a successful push/fetch, so it is "the newest commit we
/// know origin has" — without touching the network.
fn tracking_tip(repo: &Repository, branch: &str) -> Option<Oid> {
repo.find_reference(&format!("refs/remotes/origin/{branch}"))
.ok()?
.peel_to_commit()
.ok()
.map(|c| c.id())
}
/// Whether origin is known to already have HEAD (local refs only, no network).
/// Errors read as "not current", so the caller falls through to the full
/// publish path where the real failure surfaces with context.
fn remote_current() -> Result<bool> {
let repo = Repository::open(REPO_DIR)?;
let head = repo.head()?.peel_to_commit()?.id();
let branch = repo
.head()?
.shorthand()
.context("HEAD has no branch shorthand")?
.to_string();
Ok(tracking_tip(&repo, &branch) == Some(head))
}
/// How a push attempt failed — this decides whether reconciling can help.
enum PushFailure {
/// The server processed the push but refused the ref update (arrives via
/// the `push_update_reference` callback — e.g. non-fast-forward): the
/// remote moved under us, and reconcile + replay is the right response.
Rejected(String),
/// Transport / TLS / auth / URL — the push never reached a ref decision,
/// so a reconcile (whose fetch needs the same transport) cannot help.
/// Surfaced directly; the 2026-07-13 on-device run burned a doomed
/// reconcile on an "unsupported URL protocol" because this wasn't split.
Other(anyhow::Error),
}
impl PushFailure {
fn into_error(self) -> anyhow::Error {
match self {
Self::Rejected(msg) => anyhow::anyhow!("remote rejected ref: {msg}"),
Self::Other(e) => e,
}
}
}
/// One push attempt over HTTPS. Binds the PAT credential + the cert-verify
/// callback, and surfaces a server-side ref rejection (e.g. non-fast-forward) as
/// an error (it arrives via `push_update_reference`, not as a `push()` error).
fn try_push(repo: &Repository, refspec: &str) -> Result<()> {
let mut remote = repo.find_remote("origin")?;
/// callback, and separates a server-side ref rejection (reconcilable) from a
/// transport-level failure (not).
fn try_push(repo: &Repository, refspec: &str) -> Result<(), PushFailure> {
let mut remote = repo
.find_remote("origin")
.map_err(|e| PushFailure::Other(anyhow::Error::new(e).context("finding remote origin")))?;
let rejection: Rc<RefCell<Option<String>>> = Rc::new(RefCell::new(None));
let mut cbs = auth_callbacks();
@@ -328,27 +512,30 @@ fn try_push(repo: &Repository, refspec: &str) -> Result<()> {
opts.remote_callbacks(cbs);
remote
.push(&[refspec], Some(&mut opts))
.context("push transport")?;
.map_err(|e| PushFailure::Other(anyhow::Error::new(e).context("push transport")))?;
if let Some(msg) = rejection.borrow().clone() {
bail!("remote rejected ref: {msg}");
return Err(PushFailure::Rejected(msg));
}
log::info!("push accepted by remote");
Ok(())
}
/// Fetch origin and mixed-reset the local branch onto it, so our just-made commit
/// can be replayed on the current tip. Only runs after a non-fast-forward push
/// Fetch origin and *soft*-reset the local branch onto it, so our changes can
/// be replayed on the current tip. Only runs after a non-fast-forward push
/// rejection — i.e. the remote moved under us.
///
/// **MIXED**, deliberately not a force checkout: the note we're publishing lives
/// in the working tree, and a force checkout would clobber it. Mixed moves the
/// branch ref + index onto origin but leaves the working tree, so the note
/// survives and `stage_and_commit` replays it on top. For a single-writer
/// appliance this resolves last-writer-wins — a concurrent remote *edit* to the
/// same note loses to ours, and a remote-only *added* file the card doesn't have
/// is dropped by the replay's add --all. Both need a real merge (increment B) and
/// don't arise from this device's own use.
/// **SOFT**, deliberately: it moves only the branch ref. The previous Mixed
/// reset also rewrote the index — pure waste now that the splice commit never
/// reads the index, and on the real repo an index write is exactly the
/// racy-clean wall the splice exists to avoid. Neither flavor touches the
/// working tree, so the notes being published survive on the card and the
/// replay splices them onto the new tip. For a single-writer appliance this
/// resolves last-writer-wins: a concurrent remote *edit* to a note we're
/// publishing loses to ours, while a remote-only added/changed file is simply
/// carried forward — origin's tree is now the splice base, so the replay
/// keeps it (an improvement over the old `add --all` replay, which dropped
/// files the card didn't have). A real merge stays increment-B work.
fn reconcile_onto_origin(repo: &Repository, branch: &str) -> Result<()> {
let mut remote = repo.find_remote("origin")?;
let mut fo = FetchOptions::new();
@@ -362,12 +549,12 @@ fn reconcile_onto_origin(repo: &Repository, branch: &str) -> Result<()> {
.context("no FETCH_HEAD after fetch")?;
let theirs = repo.reference_to_annotated_commit(&fetch_head)?;
log::info!(
"reconcile: resetting local {branch} onto origin @ {} (mixed, keeps the note)",
"reconcile: resetting local {branch} onto origin @ {} (soft — ref move only, notes stay on the card)",
short(theirs.id())
);
let their_obj = repo.find_object(theirs.id(), None)?;
repo.reset(&their_obj, git2::ResetType::Mixed, None)
.context("mixed reset onto origin")?;
repo.reset(&their_obj, git2::ResetType::Soft, None)
.context("soft reset onto origin")?;
Ok(())
}
@@ -452,6 +639,14 @@ fn free_heap() -> u32 {
unsafe { sys::esp_get_free_heap_size() }
}
/// Free INTERNAL RAM (DRAM), excluding PSRAM. `free_heap` is dominated by the
/// 8 MB PSRAM pool and masks internal exhaustion — which is what actually
/// killed the first real-repo push (mbedTLS's ssl_setup could not get its
/// ~33 KB while Wi-Fi + USB + editor + libgit2 were resident).
fn internal_free_heap() -> u32 {
unsafe { sys::heap_caps_get_free_size(sys::MALLOC_CAP_INTERNAL) as u32 }
}
fn min_free_heap() -> u32 {
unsafe { sys::esp_get_minimum_free_heap_size() }
}

View File

@@ -10,7 +10,10 @@ use esp_idf_svc::hal::spi::{Dma, SpiBusDriver, SpiDriver};
use esp_idf_svc::hal::units::FromValueType;
use display::Frame;
use editor::{Editor, Effect, Mode, Prefs, Scope, CH, LOCAL_DIR, PREFS_PATH, REPO_DIR};
use editor::{
Editor, Effect, Mode, Prefs, Scope, Snippets, CH, LOCAL_DIR, PREFS_PATH, REPO_DIR,
SNIPPETS_PATH,
};
use firmware::epd::{self, Epd};
use firmware::persistence::{Storage, NOTES};
@@ -120,7 +123,18 @@ fn main() -> anyhow::Result<()> {
ed.set_notice(format!("loaded {name}"));
// Feed the file palette (Ctrl-P). Enumerated once at boot — the v0.5 slices
// that create/delete files (`:enew`, delete) will re-feed it then.
// Bracketed with internal-DRAM readings: each path is a small String, kept
// internal by the SPIRAM malloc threshold (16 KB), so the list competes
// with Wi-Fi/TLS for DRAM. Estimate to confirm: ~60-70 KB at 1098 files —
// this number decides whether interning the paths into one shared buffer
// (a single >16 KB alloc, which lands in PSRAM) is worth the refactor.
let dram_before = internal_free_heap();
ed.set_file_list(enumerate_files());
let dram_after = internal_free_heap();
log::info!(
"file list: internal heap {dram_before} -> {dram_after} ({} KB consumed)",
dram_before.saturating_sub(dram_after) / 1024
);
// Editor preferences (.typoena.toml, git-tracked). Read before the first
// render so `line_numbers` shapes the opening frame. A missing / unreadable /
// partial file falls back to defaults, so a fresh card just works.
@@ -130,6 +144,21 @@ fn main() -> anyhow::Result<()> {
};
log::info!("prefs: {prefs:?}");
ed.set_prefs(prefs);
// Snippet library (.typoena.snippets.json, git-tracked). Parsed with
// serde_json in the editor crate; a missing / unreadable / malformed file is
// non-fatal — the editor simply has no snippets and runs unchanged.
let snippets = match storage.load_path(SNIPPETS_PATH) {
Ok(src) => match Snippets::parse(&src) {
Ok(s) => s,
Err(e) => {
log::warn!("snippets parse FAILED ({e}); none loaded");
Snippets::default()
}
},
Err(_) => Snippets::default(),
};
log::info!("snippets: {} loaded", snippets.0.len());
ed.set_snippets(snippets);
let mut updates: u32 = 0;
let mut cursor_shown = true; // the initial render includes the caret
let mut last_activity = Instant::now();
@@ -206,11 +235,21 @@ fn main() -> anyhow::Result<()> {
// The outcome returns on `git_rx` and updates the snackbar
// (see the idle branch below). The Save that preceded this
// in the batch already persisted the buffer, so this is a
// pure git push.
// pure git publish of the recorded dirty paths — the
// outcome decides whether the snapshot is forgotten
// (publish_succeeded) or retried (publish_failed).
#[cfg(feature = "git")]
match git_tx.send(firmware::git_sync::PublishRequest) {
Ok(()) => ed.set_notice("syncing..."),
Err(_) => ed.set_notice("sync: git thread down"),
{
let paths = storage.take_dirty();
match git_tx.send(firmware::git_sync::PublishRequest { paths }) {
Ok(()) => ed.set_notice("syncing..."),
Err(_) => {
// Thread gone — nothing will report back, so
// return the snapshot to pending ourselves.
storage.publish_failed();
ed.set_notice("sync: git thread down");
}
}
}
#[cfg(not(feature = "git"))]
log::info!(":sync — saved; light build (no `git` feature) — push skipped");
@@ -241,6 +280,13 @@ fn main() -> anyhow::Result<()> {
#[cfg(feature = "git")]
if let Ok(outcome) = git_rx.try_recv() {
use firmware::git_sync::PublishOutcome::*;
// Settle the dirty snapshot this publish took: confirmed
// published (or up to date) → forget it; failed → back to
// pending so the next :sync retries the same paths.
match &outcome {
Pushed(_) | UpToDate => storage.publish_succeeded(),
Failed(_) => storage.publish_failed(),
}
ed.set_notice(match outcome {
Pushed(oid) => format!("synced {oid}"),
UpToDate => "up to date".to_string(),
@@ -392,7 +438,15 @@ fn main() -> anyhow::Result<()> {
/// `main`): the note is the whole point of the appliance, so we refuse to run
/// in a state where the next save could destroy it.
fn boot_storage(epd: &mut Epd) -> (Storage, String) {
let storage = match Storage::mount() {
// A git build shares this mount with the git thread, and libgit2 keeps the
// pack + idx descriptors open across a publish — that overruns the
// editor's tight 4-FD budget, so mount with the 16-FD one (persistence.rs,
// MAX_FILES_GIT). The light build keeps the editor's own budget.
#[cfg(feature = "git")]
let mounted = Storage::mount_for_git();
#[cfg(not(feature = "git"))]
let mounted = Storage::mount();
let storage = match mounted {
Ok(s) => s,
Err(e) => boot_halt(epd, "SD card not ready", &format!("{e:#}")),
};
@@ -508,36 +562,71 @@ fn delete_buffer(storage: &Storage, ed: &mut Editor, path: String, scope: Scope)
}
}
/// Enumerate the palette's openable files: the top-level regular files in
/// `/sd/repo` and `/sd/local`, as absolute paths. Skips dotfiles (so `.git`,
/// `.typoena.toml`, and the like never show) and anything that isn't a plain
/// file. Best-effort: an unreadable directory (e.g. no `/sd/local` yet)
/// contributes nothing rather than failing. The editor sorts and dedupes.
/// Enumerate the palette's openable files: the regular files under `/sd/repo`
/// and `/sd/local`, recursively, as absolute paths. Skips dot entries at every
/// level (so `.git` and its thousands of object files, `.typoena.toml`, and the
/// like never show or get walked). Best-effort: an unreadable directory (e.g.
/// no `/sd/local` yet) contributes nothing rather than failing. The editor
/// sorts and dedupes. Runs once at boot, so the walk time is logged — on a big
/// repo the FAT directory IO is the cost to watch.
fn enumerate_files() -> Vec<String> {
let start = std::time::Instant::now();
let mut out = Vec::new();
for dir in [REPO_DIR, LOCAL_DIR] {
let Ok(entries) = std::fs::read_dir(dir) else {
walk_files(std::path::Path::new(dir), 0, &mut out);
}
log::info!("file walk: {} files in {}ms", out.len(), start.elapsed().as_millis());
out
}
/// Depth bound for [`walk_files`] — belt-and-braces against pathological
/// nesting on a hand-edited card; notes trees are a couple of levels deep.
const WALK_MAX_DEPTH: usize = 8;
/// Recursive helper for [`enumerate_files`]: push `dir`'s files onto `out`,
/// then descend into its subdirectories. Reads each directory fully before
/// recursing (the `remove_dir_recursive` pattern in `git_sync`), so only one
/// FatFS directory handle is open at a time regardless of depth — relevant on
/// the FD-bounded SD mount.
fn walk_files(dir: &std::path::Path, depth: usize, out: &mut Vec<String>) {
if depth > WALK_MAX_DEPTH {
log::warn!("file walk: {} exceeds depth {WALK_MAX_DEPTH}, skipped", dir.display());
return;
}
let Ok(entries) = std::fs::read_dir(dir) else {
return;
};
// Keep the dirent's own file type: esp-idf's FAT VFS always fills d_type
// (DT_DIR/DT_REG, straight from the FILINFO readdir already holds), so
// `file_type()` is free. A per-entry `metadata()` stat instead re-walks
// the directory by path every time — measured at ~32ms/file on the SD
// card, it turned a 1098-file walk into 35s.
let children: Vec<_> = entries
.flatten()
.filter_map(|e| e.file_type().ok().map(|t| (e.path(), t)))
.collect();
for (path, ftype) in children {
let Some(name) = path.file_name().and_then(|n| n.to_str()) else {
continue;
};
for entry in entries.flatten() {
let path = entry.path();
let Some(name) = path.file_name().and_then(|n| n.to_str()) else {
continue;
};
if name.starts_with('.') {
continue;
}
// Stat rather than trust d_type — FatFS's dirent type can read back
// as unknown; a plain metadata call is reliable here.
if !std::fs::metadata(&path).map(|m| m.is_file()).unwrap_or(false) {
continue;
}
if name.starts_with('.') {
continue;
}
if ftype.is_file() {
if let Some(p) = path.to_str() {
out.push(p.to_string());
}
} else if ftype.is_dir() {
walk_files(&path, depth + 1, out);
}
}
out
}
/// Free internal DRAM (excludes the 8 MB PSRAM pool, which dominates the total
/// free-heap number and masks DRAM exhaustion). Same reading `git_sync` logs.
fn internal_free_heap() -> u32 {
use esp_idf_svc::sys;
unsafe { sys::heap_caps_get_free_size(sys::MALLOC_CAP_INTERNAL) as u32 }
}
/// A file's display name — its basename without extension (`/sd/repo/notes.md`

View File

@@ -34,6 +34,8 @@
//! sits in the tmp. [`Storage::recover`] closes the loop at boot — see its docs
//! for the exact case analysis, which is subtler than "promote the tmp."
use std::cell::RefCell;
use std::collections::BTreeSet;
use std::fs;
use std::io::Write as _;
use std::mem::MaybeUninit;
@@ -79,6 +81,23 @@ pub const MAX_FILE_BYTES: u64 = 256 * 1024;
/// The C mount point (`/sd\0`) for the esp-idf FFI calls.
const MOUNT_C: &std::ffi::CStr = c"/sd";
/// Dirty-path journal — one repo-relative path per line, mirroring the in-RAM
/// dirty set (see [`Storage::take_dirty`]). At the card root, *outside*
/// `/sd/repo`, so it can never itself be committed. Without it a power pull
/// would strand every file saved-but-not-yet-published in that session: the
/// splice commit only visits recorded paths (nothing walks the tree anymore),
/// so an unrecorded change would never reach the remote.
const DIRTY_JOURNAL: &str = "/sd/.typoena-dirty";
/// VFS open-file budget for the editor path: it opens only a note and its
/// `*.tmp`, so a tight budget keeps FatFS's per-file buffers off the heap.
const MAX_FILES_EDITOR: i32 = 4;
/// VFS open-file budget for the git tooling. libgit2 keeps the pack + `.idx`
/// (and commit-graph) descriptors open for the repo's lifetime and opens loose
/// objects on top, so a `read_tree` walk overruns [`MAX_FILES_EDITOR`] with a
/// "no free file descriptors" error. Matches the flash-FAT git binaries' 16.
const MAX_FILES_GIT: i32 = 16;
/// A mounted SD card. Holds the live card handle for its lifetime; v0.1 never
/// unmounts (the card stays up for the whole power session). Not `Send` — the
/// handle lives on the task that mounted it (the ui/main task). The git thread
@@ -86,6 +105,23 @@ const MOUNT_C: &std::ffi::CStr = c"/sd";
/// lock serialises the two, so no extra mutex is needed here.
pub struct Storage {
card: *mut sys::sdmmc_card_t,
/// Repo-relative paths saved or `:delete`d since the last confirmed
/// publish — the editor-side half of the O(depth) splice commit
/// (`git_sync::stage_and_commit` visits exactly these paths and nothing
/// else). Mirrored to [`DIRTY_JOURNAL`] whenever it changes, so the record
/// survives a power pull. `RefCell` because recording happens inside
/// `&self` save/delete calls; `Storage` already lives on one task only.
dirty: RefCell<Dirty>,
}
/// The two halves of the dirty record: `pending` accumulates between syncs;
/// `take_dirty` moves it to `in_flight` for the duration of a publish so a
/// failure can put it back (and a save landing *during* the publish re-enters
/// `pending`, riding the next one). The journal always carries the union.
#[derive(Default)]
struct Dirty {
pending: BTreeSet<String>,
in_flight: BTreeSet<String>,
}
/// What [`Storage::recover`] did with a leftover `*.tmp` at boot.
@@ -113,6 +149,17 @@ impl Storage {
/// (The Spike 3 bench binary sets it true for convenience on blank cards;
/// this path must not.)
pub fn mount() -> Result<Self> {
Self::mount_with_max_files(MAX_FILES_EDITOR)
}
/// Like [`Storage::mount`], but with the larger [`MAX_FILES_GIT`] open-file
/// budget the git tooling (bench / sync) needs — libgit2 holds several
/// descriptors open at once, which the editor's default budget can't cover.
pub fn mount_for_git() -> Result<Self> {
Self::mount_with_max_files(MAX_FILES_GIT)
}
fn mount_with_max_files(max_files: i32) -> Result<Self> {
// 1) SPI3 with the SD's four lines. Dedicated bus (ADR-012) — no EPD
// deselect needed: the panel is on SPI2 and can't contend here.
// SAFETY: a zeroed spi_bus_config_t is valid (all pins default 0); we
@@ -177,7 +224,7 @@ impl Storage {
// 4) Mount config. format_if_mount_failed = FALSE — see method docs.
let mount = sys::esp_vfs_fat_mount_config_t {
format_if_mount_failed: false,
max_files: 4,
max_files,
allocation_unit_size: 16 * 1024,
disk_status_check_enable: false,
use_one_fat: false,
@@ -203,7 +250,10 @@ impl Storage {
}
esp!(rc).context("esp_vfs_fat_sdspi_mount (card present? inserted? FAT-formatted?)")?;
let storage = Storage { card };
let storage = Storage {
card,
dirty: RefCell::new(Dirty::default()),
};
let (max_khz, real_khz) = storage.negotiated_khz();
log::info!("SD mounted at {MOUNT} — max {max_khz} kHz, negotiated {real_khz} kHz");
@@ -218,6 +268,13 @@ impl Storage {
newest complete copy)"
),
}
let carried = storage.load_dirty_journal();
if carried > 0 {
log::info!(
"dirty journal: {carried} unpublished path(s) carried over from a previous \
session — the next :sync will commit them"
);
}
Ok(storage)
}
@@ -294,6 +351,16 @@ impl Storage {
/// buffers is deferred to the v0.9 crash-safety work — the atomic swap here
/// already protects each individual save.
pub fn save_path(&self, path: &str, contents: &str) -> Result<()> {
// Record BEFORE writing: a crash in between leaves an over-approximate
// journal (the splice of an unchanged path is a no-op), whereas the
// reverse order could leave a changed file no record ever points at.
self.record_dirty(path);
Self::atomic_write(path, contents)
}
/// The atomic write primitive behind [`Storage::save_path`] and the dirty
/// journal: write `{path}.tmp`, fsync, unlink the target, rename over it.
fn atomic_write(path: &str, contents: &str) -> Result<()> {
let tmp = format!("{path}.tmp");
{
let mut f = fs::File::create(&tmp)
@@ -330,6 +397,9 @@ impl Storage {
/// file half-present after a delete. For a Tracked file this leaves the
/// working copy short one file; the next publish's `add --all` stages it.
pub fn delete_path(&self, path: &str) -> Result<()> {
// Same record-first rule as `save_path`: the splice treats a recorded
// path with no file behind it as "remove from the tree".
self.record_dirty(path);
let _ = fs::remove_file(format!("{path}.tmp"));
match fs::remove_file(path) {
Ok(()) => Ok(()),
@@ -338,6 +408,88 @@ impl Storage {
}
}
/// Note a working-copy file as (possibly) differing from HEAD. Paths
/// outside `/sd/repo` (`/sd/local`, `/sd/ca.pem`, the journal itself) are
/// not git's business and are skipped. The journal is rewritten only when
/// the set actually grows, so re-saving the same note between syncs costs
/// no extra card I/O.
fn record_dirty(&self, abs_path: &str) {
let Some(rel) = abs_path
.strip_prefix(REPO_DIR)
.and_then(|r| r.strip_prefix('/'))
else {
return;
};
if rel.is_empty() {
return;
}
let grew = self.dirty.borrow_mut().pending.insert(rel.to_string());
if grew {
self.persist_dirty();
}
}
/// Snapshot the dirty paths for a publish (repo-relative). The snapshot
/// moves to `in_flight` — the journal keeps carrying it — until the UI
/// task reports the outcome: [`Storage::publish_succeeded`] forgets it,
/// [`Storage::publish_failed`] returns it to pending for the next `:sync`.
pub fn take_dirty(&self) -> BTreeSet<String> {
let mut d = self.dirty.borrow_mut();
let taken = std::mem::take(&mut d.pending);
d.in_flight.extend(taken.iter().cloned());
taken
}
/// The publish that took the last snapshot committed (or confirmed
/// up-to-date): drop its paths and shrink the journal. Anything saved
/// while it ran is still in `pending` and rides the next sync.
pub fn publish_succeeded(&self) {
self.dirty.borrow_mut().in_flight.clear();
self.persist_dirty();
}
/// The publish failed: return its snapshot to pending so the next `:sync`
/// retries it (the splice is idempotent, so a retry of an already-clean
/// path is free). The journal already carries these paths — no rewrite.
pub fn publish_failed(&self) {
let mut d = self.dirty.borrow_mut();
let inflight = std::mem::take(&mut d.in_flight);
d.pending.extend(inflight);
}
/// Mirror `pending in_flight` to [`DIRTY_JOURNAL`], atomically.
/// Best-effort: a failed journal write must not fail the save that
/// triggered it — the set stays correct in RAM and the journal heals on
/// the next change.
fn persist_dirty(&self) {
let contents = {
let d = self.dirty.borrow();
let mut out = String::new();
for p in d.pending.union(&d.in_flight) {
out.push_str(p);
out.push('\n');
}
out
};
if let Err(e) = Self::atomic_write(DIRTY_JOURNAL, &contents) {
log::warn!("dirty journal write FAILED ({e:#}); set kept in RAM only");
}
}
/// Seed the dirty set from the journal at mount — the paths a previous
/// session saved but never got confirmed as published (power pull, failed
/// sync, or simply no `:sync` before shutdown). Returns how many.
fn load_dirty_journal(&self) -> usize {
let Ok(text) = fs::read_to_string(DIRTY_JOURNAL) else {
return 0; // no journal yet — nothing carried over
};
let mut d = self.dirty.borrow_mut();
for line in text.lines().map(str::trim).filter(|l| !l.is_empty()) {
d.pending.insert(line.to_string());
}
d.pending.len()
}
/// Reconcile a leftover `notes.md.tmp` at boot. The save sequence is
/// write-tmp → fsync → unlink-target → rename, so a lingering tmp means the
/// last save was interrupted. Which way to recover depends on whether the