Compare commits

..

4 Commits

Author SHA1 Message Date
Julien Calixte
470a9d25d0 docs(macroplan): note v0.3 on-device smoke-test and paste-scroll fix 2026-07-11 20:26:15 +02:00
Julien Calixte
937868dd85 fix(editor): reveal the pasted block when it runs past the fold
A multi-line paste near the bottom of the screen left its later lines below
the viewport — adjust_scroll only kept the caret's (first) pasted line
visible. Add reveal(), which scrolls the end of the pasted block into view
while the caret stays on its first line (vim semantics unchanged).
2026-07-11 20:26:15 +02:00
Julien Calixte
884c8f2e48 docs(postmortems): record the SPI-DMA OOM editor freeze
Root cause, shipped safety net, and the specced eradication (a
persistent internal DMA scratch buffer in Epd so paints never allocate
mid-sync). Tracks hardware re-test and the eradication as follow-ups.
2026-07-11 20:21:49 +02:00
Julien Calixte
32b9bc6a15 fix(firmware): keep the editor alive when a panel paint fails
A screen refresh that ran while :sync had Wi-Fi + TLS up could fail to
allocate an internal DMA bounce buffer (ESP_ERR_NO_MEM); the error
propagated through ? out of main(), main_task returned from app_main(),
and the editor loop died while the USB and git threads kept running —
keys logged, panel frozen forever.

A paint is idempotent and retryable (the buffer is the source of truth),
so every editor-loop paint now logs and drops the frame instead of
propagating, leaves shown untouched so the next paint repaints the diff,
and forces a full refresh next to re-sync both RAM banks. Same contract
as save_note. The boot-time first render stays fatal on purpose.
2026-07-11 20:21:49 +02:00
5 changed files with 284 additions and 27 deletions

View File

@@ -40,7 +40,7 @@ name = "v0.3 editing"
start = 2026-08-03
original = 2026-08-24
delivered = 2026-07-11
learning = "Core complete 44 days early, host-tested. Register + yank/paste (yy/p/P), snapshot undo/redo (u/Ctrl-r, bounded 100 groups in PSRAM), and keystroke-recorded `.` repeat all landed 2026-07-11; the d/c operator grammar + text objects were already done ahead of schedule. Firmware version bumped to 0.3.0. On-device smoke-test still pending — pure editor-core, no new render surface."
learning = "Core complete 44 days early, host-tested and partially smoke-tested on the panel. Register + yank/paste (yy/p/P), snapshot undo/redo (u/Ctrl-r, bounded 100 groups in PSRAM), and keystroke-recorded `.` repeat all landed 2026-07-11; the d/c operator grammar + text objects were already done ahead of schedule. Firmware bumped to 0.3.0. On device dd/yy/Ctrl-r confirmed; the one bug found was a multi-line paste leaving its later lines below the fold (adjust_scroll only tracked the caret) — fixed with a reveal() that scrolls the block end into view."
[[feature]]
name = "v0.4 visual + ex"
@@ -108,8 +108,9 @@ recovery, 1000-word no-drop, and `Ctrl-G`'s not-yet-built pull-then-retry
refresh check passed (single-line edit repaints only rows at/below it, no extra
full refresh), closing the last gate. **v0.2.5 international input** is
hardware-verified (2026-07-11), and **v0.3 editing is complete in core** the same
day (register + yank/paste, snapshot undo/redo, `.` repeat — host-tested,
on-device smoke-test pending); the firmware crate is bumped to **0.3.0**. Most of
day (register + yank/paste, snapshot undo/redo, `.` repeat — host-tested, and
partially smoke-tested on the panel: `dd`/`yy`/`Ctrl-r` good, a multi-line-paste
scroll bug found + fixed); the firmware crate is bumped to **0.3.0**. Most of
v0.6 Markdown also already runs. Version numbers track shippable device releases,
not raw core progress — the 0.3.0 bump reflects the v0.3 feature set being met.
@@ -253,15 +254,19 @@ v0.2 UTF-8-correct buffer and the ISO-8859-15 render font. Host-tested.
## v0.3 — Vim editing — [x]
**Status:** COMPLETE in core 2026-07-11, host-tested (64 editor + 28 keymap
tests). The three remaining pieces landed together: a single unnamed **register**
with `y`/`yy`/`p`/`P` (and `x`/`d`/`c` filling it, so `dd`…`p` moves a line),
**Status:** COMPLETE in core 2026-07-11, host-tested (65 editor + 28 keymap
tests) and **partially smoke-tested on the panel 2026-07-11**. The three
remaining pieces landed together: a single unnamed **register** with
`y`/`yy`/`p`/`P` (and `x`/`d`/`c` filling it, so `dd`…`p` moves a line),
**undo/redo** (`u`/`Ctrl-r`, snapshot-based, bounded to 100 groups in PSRAM — a
whole Insert session undoes as one group), and **`.` repeat** (keystroke-recorded,
so it replays an insert session like `ciwfoo<Esc>`). The `d`/`c` operator grammar
and text objects had already landed ahead of schedule. Pure editor-core work,
riding the already-verified draw/refresh path; **on-device smoke-test pending**
(nothing new to render, so low risk).
and text objects had already landed ahead of schedule. On device, `dd`, `yy`, and
`Ctrl-r` confirmed good; the one issue found was that a **multi-line paste near
the bottom left its later lines below the fold** — `adjust_scroll` only kept the
caret's (first) pasted line visible. Fixed by a `reveal()` that scrolls the end of
the pasted block into view while the caret stays on its first line (reflash to
re-confirm on panel).
- [x] `x dd`, `dw dd d$` (✓); `yy p P` (✓) and `.` repeat (✓) — register + a
keystroke-recorded last-change both landed 2026-07-11

View File

@@ -0,0 +1,164 @@
# Editor freeze — SPI-DMA out-of-memory during a background `:sync`
> Date: 2026-07-11 · Build at time of failure: `07-11 18:02Z @229c259-dirty`
> Status: **Safety net shipped** (paints are non-fatal, the appliance no longer
> bricks) — pending hardware re-test. **Root cause not yet eradicated**: a paint
> that lands during a sync still fails and drops the frame; the permanent fix (a
> persistent internal DMA scratch buffer) is specced below and tracked in the
> follow-ups.
>
> Context: editor loop [`../../firmware/src/main.rs`](../../firmware/src/main.rs),
> EPD driver [`../../firmware/src/epd.rs`](../../firmware/src/epd.rs), git
> transport [`../../firmware/src/git_sync.rs`](../../firmware/src/git_sync.rs).
> Display medium [ADR-003](../adr.md#adr-003-display-medium--e-ink-gdey0579t93-panel);
> concurrency model (dedicated git thread) [ADR-006](../adr.md).
## Summary
First-ever field freeze. After an hour of clean editing, the user ran `:sync`;
the commit and push **succeeded** (`push accepted by remote`, commit
`48a2c0a8`), but partway through the push the panel stopped updating. Keystrokes
kept being logged, so the device wasn't hung — but nothing repainted again.
The editor task had died. A screen refresh that happened to run **while Wi-Fi +
TLS were up for the push** failed to allocate an internal DMA buffer
(`ESP_ERR_NO_MEM`), and that error propagated through a `?` straight out of
`main()`. ESP-IDF's `main_task` returned from `app_main()`; the editor loop lived
on that task, so it stopped. The USB-keyboard and git threads are **separate
FreeRTOS tasks** (ADR-006) and kept running — hence keys still logged while the
panel was frozen. A zombie, not a crash.
## Symptom
```
I (305749) firmware::git_sync: verifying github.com TLS chain against embedded GitHub CA bundle
I (305979) firmware::usb_kbd: key: HalfPageUp ← scroll arrives → editor loop paints
E (306259) spi_master: setup_dma_priv_buffer(1206): Failed to allocate priv TX buffer
Error: ESP_ERR_NO_MEM
I (306259) main_task: Returned from app_main() ← editor task exits
...
I (310119) firmware::git_sync: push accepted by remote ← git thread unaffected; commit landed
I (311069) firmware::usb_kbd: key: Escape ← keys still logged, no refresh ever again
I (312729) firmware::usb_kbd: key: HalfPageDown
```
## Root cause
The EPD is on SPI2 configured `Dma::Auto(4096)` (`main.rs`), and the driver hands
frame data to `spi.write()` as ordinary Rust `Vec<u8>` buffers (`epd.rs`
`write_frame_bank` / `data`). With PSRAM added to the heap allocator, those
`Vec`s can be allocated **from PSRAM**, which is **not DMA-capable**. When a
transfer buffer isn't DMA-capable, esp-idf's `spi_master` bounces it through a
temporary internal buffer it mallocs on the fly (`setup_dma_priv_buffer`, using
`MALLOC_CAP_DMA`**internal RAM only**).
For the whole session that bounce allocation succeeded, because internal RAM was
plentiful. The instant `:sync` brought up **Wi-Fi + TLS**, they consumed the
small internal pool. The next paint's bounce allocation returned
`ESP_ERR_NO_MEM`. The real defect is not the low-memory moment itself — it's that
the paint's `?` made a **transient, retryable** I/O failure **fatal to the whole
appliance**.
Two things made this easy to misread:
- **"8.4 MB free heap" is a red herring.** That figure is dominated by PSRAM.
The starved pools are the tiny internal ones (~265 KB / 21 KB / 32 KB), and DMA
bounce buffers can only come from those.
- **It looked like a git/sync bug, but the push was fine.** The git thread
finished and the remote accepted the commit. Only the UI task died.
## Timeline
1. Hour of normal editing — hundreds of ~630 ms partial refreshes, no issue
(internal RAM never under pressure; Wi-Fi off — the radio is lazy, ADR-006).
2. `:sync` → save → git thread brings up Wi-Fi, SNTP, TLS, commits `48a2c0a8`,
starts the push. Internal RAM now under heavy Wi-Fi/TLS load.
3. User keeps scrolling (`HalfPageUp`) during the push. Each scroll triggers a
full-area partial refresh → SPI-DMA transfer → bounce-buffer malloc.
4. One such malloc fails (`ESP_ERR_NO_MEM`); the `?` in the refresh call
propagates out of `main()`; `main_task` returns from `app_main()`.
5. Push completes normally; Wi-Fi torn down; internal RAM freed — but the editor
task is already gone, so nothing repaints. Keys log into the void.
## What it was *not*
- **Not out of heap** — plenty of PSRAM free; it was internal *DMA-capable* RAM
specifically.
- **Not a git or TLS bug** — the sync succeeded end to end.
- **Not an editor-core bug** — `editor`/`keymap` never ran; the failure is in the
firmware paint path's error handling.
- **Not a panel/wiring fault** — the same paint path worked for an hour and works
again after Wi-Fi is down.
## Remediation shipped — paints are non-fatal
A screen refresh is idempotent and retryable: the editor buffer is the source of
truth, so a dropped frame costs nothing and the next paint recovers. This is the
exact contract already written into `save_note` ("errors are logged, never
propagated"). Every paint site in the editor loop (`main.rs`) now:
- logs the failure and **drops the frame** instead of `?`-propagating it;
- leaves `shown` untouched so the next paint repaints the same diff;
- sets a `force_full` flag so the next paint is a **full refresh**, which
rewrites both RAM banks and recovers from a partial that may have died
mid-transfer and left the `0x24`/`0x26` banks inconsistent.
Effect: a paint that lands during a sync is dropped (panel goes stale for the
~15 s of the push), then self-heals the moment the push finishes and internal RAM
frees. A permanent brick becomes a brief stale window. The boot-time first render
stays `?`-fatal on purpose — it runs before Wi-Fi, can't hit this, and a dead
panel at boot is a legitimate hard fault.
## Root-cause eradication (specced, not yet built)
The safety net stops the brick but not the underlying **contention**: paints
during a sync still fail and drop. To keep the editor fully live while a push
runs — the whole point of the async git thread (ADR-006) — remove the on-the-fly
DMA allocation entirely.
**Fix: a persistent, internal, DMA-capable scratch buffer owned by `Epd`.**
- Allocate it **once at `Epd` construction** (boot, before Wi-Fi is ever up, when
internal RAM is plentiful) via
`heap_caps_malloc(SPI_CHUNK, MALLOC_CAP_DMA | MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT)`.
- `data()` already chunks writes to `SPI_CHUNK` (4096 B), so a **4 KB** scratch
buffer is sufficient: copy each chunk into it and hand that DMA-capable slice to
`spi.write()`.
- Because the source is now DMA-capable, `spi_master` DMAs directly from it —
`setup_dma_priv_buffer` is never invoked, there is **no per-refresh
allocation**, and paints no longer compete with Wi-Fi/TLS for internal RAM.
Refreshes then succeed during a sync, and `force_full` recovery is only ever
exercised by genuine faults, not by normal syncing.
**Cost:** ~4 KB of internal RAM reserved for the life of the device; one small
`unsafe` block for the alloc + slice; a `memcpy` per 4 KB chunk (negligible
against the ~630 ms panel waveform).
**Alternatives considered and rejected:**
- *Trim Wi-Fi's internal buffer counts* to leave headroom — fragile tuning that
risks Wi-Fi stability/throughput and only widens the margin instead of removing
the failure mode.
- *Force the whole ~13.6 KB frame buffer internal* (custom allocator / full
`heap_caps` buffer) — larger reservation and more churn than the 4 KB
chunk-scratch, for no extra benefit since `data()` already chunks.
- *Serialize paints against sync* (skip painting while a push is in flight) —
defeats the async-git design and freezes the panel for the whole push; the
safety net already makes this unnecessary.
**Verification when built:** reproduce the exact failing scenario — edit,
`:sync`, and keep scrolling/paging through the *entire* push. Expect: every
refresh succeeds (no `refresh … FAILED` warnings, no dropped frames), min-ever
internal heap stays comfortably above zero throughout, and `force_full` is not
triggered by the sync.
## Follow-ups
- [x] Make all editor-loop paints non-fatal + `force_full` recovery (`main.rs`);
release build green.
- [ ] Reflash and hardware-verify the safety net against the repro (edit →
`:sync` → scroll through the push): panel must recover, not freeze.
- [ ] Implement the persistent internal DMA scratch buffer in `Epd` (eradication
above) if the stale-during-sync window proves annoying in real use.
- [ ] After eradication, confirm refreshes succeed *during* a push and drop the
stale window entirely.

View File

@@ -12,3 +12,4 @@
| ---------- | ------------------------------------------------------------------------ | ------ |
| 2026-07-05 | [Spike 3 (SD) — card rejects CMD59 (SPI-mode CRC)](2026-07-05-spike3-sd-cmd59.md) | Paused — awaiting a compliant microSD; wiring + firmware proven |
| 2026-07-05 | [Spike 7 (git push) — ADR-004 kill-switch fired: gix can't push over HTTPS](2026-07-05-spike7-gix-https-push.md) | Turned — pivoted to libgit2; git mechanics proven on desktop, device build next |
| 2026-07-11 | [Editor freeze — SPI-DMA OOM during a background `:sync`](2026-07-11-editor-freeze-spi-dma-oom-during-sync.md) | Safety net shipped (paints non-fatal); root-cause eradication specced, not yet built |

View File

@@ -1036,24 +1036,32 @@ impl Editor {
}
self.checkpoint();
let content = self.register.repeat(n);
if self.register_linewise {
// `end`: byte offset of the last pasted char, so the viewport can reveal
// the whole block even when the caret stays on its first line.
let end = if self.register_linewise {
let le = self.line_end(self.caret);
if le < self.text.len() {
let at = le + 1; // start of the following line
self.text.insert_str(at, &content);
self.caret = at;
at + content.len() - 1
} else {
// Last line has no trailing newline: prefix one, drop the
// block's trailing newline so we don't leave a blank line.
let block = content.strip_suffix('\n').unwrap_or(&content);
self.text.insert_str(le, &format!("\n{block}"));
let inserted = format!("\n{block}");
let end = le + inserted.len() - 1;
self.text.insert_str(le, &inserted);
self.caret = le + 1;
end
}
} else {
let at = if self.text.is_empty() { 0 } else { self.next_char(self.caret) };
self.text.insert_str(at, &content);
self.caret = self.prev_char(at + content.len()); // onto the last char
}
self.caret
};
self.reveal(end);
}
/// `P` — paste the register `n` times before the caret. Linewise content
@@ -1065,15 +1073,18 @@ impl Editor {
}
self.checkpoint();
let content = self.register.repeat(n);
if self.register_linewise {
let end = if self.register_linewise {
let ls = self.line_start(self.caret);
self.text.insert_str(ls, &content);
self.caret = ls;
ls + content.len() - 1
} else {
let at = self.caret;
self.text.insert_str(at, &content);
self.caret = self.prev_char(at + content.len()); // onto the last char
}
self.caret
};
self.reveal(end);
}
/// Apply a pending operator over the buffer range `[start, end)` (order
@@ -1349,6 +1360,32 @@ impl Editor {
(row, col)
}
/// Scroll so the display row holding byte offset `pos` is visible,
/// bottom-aligning it when it sits below the viewport (never scrolls up).
/// Called after an insert that can run past the fold — chiefly a multi-line
/// paste — so the *whole* pasted block is revealed, not just the caret's
/// first line. The caret is left where the edit put it; `draw`'s
/// `adjust_scroll` won't override this as long as the caret stays within the
/// resulting window (true for any block up to a screen tall).
fn reveal(&mut self, pos: usize) {
let lay = self.layout();
if lay.is_empty() {
return;
}
let pos = pos.min(self.text.len());
let mut row = 0;
for (i, l) in lay.iter().enumerate() {
if l.start <= pos {
row = i;
} else {
break;
}
}
if row >= self.scroll_top + ROWS {
self.scroll_top = row + 1 - ROWS;
}
}
/// Move the viewport so the caret stays visible (Normal/Insert), or just
/// clamp it to the content (View).
fn adjust_scroll(&mut self, caret_row: usize, total: usize) {
@@ -2239,6 +2276,28 @@ mod tests {
assert_eq!(e.text(), "abc");
}
#[test]
fn multiline_paste_at_the_bottom_reveals_the_whole_block() {
// A screenful+ of lines, caret on the last line; paste two lines after
// it. Both pasted lines must be visible without a manual scroll — the
// caret stays on the first pasted line, but the viewport reveals the end.
let mut e = Editor::with_text(vec!["x"; 20].join("\n")); // 20 display rows
e.handle(Key::Char('g'));
e.handle(Key::Char('g'));
e.handle(Key::Char('2'));
e.handle(Key::Char('y'));
e.handle(Key::Char('y')); // yank two lines
e.handle(Key::Char('G')); // to the last line
e.handle(Key::Char('p')); // paste two lines below it (22 rows total)
e.draw(true); // adjust_scroll runs; reveal already applied by paste
let last_row = e.layout().len() - 1; // the second pasted line
assert!(
last_row >= e.scroll_top() && last_row < e.scroll_top() + ROWS,
"pasted block end (row {last_row}) off-screen at scroll_top {}",
e.scroll_top()
);
}
// ---- Undo / redo (v0.3) ----
#[test]

View File

@@ -113,6 +113,10 @@ fn main() -> anyhow::Result<()> {
let mut updates: u32 = 0;
let mut cursor_shown = true; // the initial render includes the caret
let mut last_activity = Instant::now();
// Set when a paint fails (see the refresh block below): the next paint then
// does a full refresh to re-establish both RAM banks, since a partial that
// died mid-transfer may have left them inconsistent.
let mut force_full = false;
// Keyboard attach/detach state drives the panel's disconnect flag; seed it
// (and the word-count snapshot) before the first render.
@@ -202,7 +206,11 @@ fn main() -> anyhow::Result<()> {
Failed(reason) => reason,
});
let f = ed.draw(true);
epd.display_frame_partial_window(f.bytes(), 0, epd::HEIGHT)?;
if let Err(e) = epd.display_frame_partial_window(f.bytes(), 0, epd::HEIGHT) {
log::warn!("sync-notice repaint FAILED ({e}); full refresh next");
force_full = true;
continue;
}
shown = f;
cursor_shown = true;
continue;
@@ -211,7 +219,11 @@ fn main() -> anyhow::Result<()> {
// no keystroke will arrive to trigger it otherwise.
if kbd_changed {
let f = ed.draw(true);
epd.display_frame_partial_window(f.bytes(), 0, epd::HEIGHT)?;
if let Err(e) = epd.display_frame_partial_window(f.bytes(), 0, epd::HEIGHT) {
log::warn!("kbd-flag repaint FAILED ({e}); full refresh next");
force_full = true;
continue;
}
shown = f;
cursor_shown = true;
log::info!("keyboard {}", if kbd { "connected" } else { "disconnected" });
@@ -226,10 +238,14 @@ fn main() -> anyhow::Result<()> {
{
ed.refresh_stats();
let f = ed.draw(true);
epd.display_frame_partial_window(f.bytes(), 0, epd::HEIGHT)?;
shown = f;
cursor_shown = true;
log::info!("caret shown");
if let Err(e) = epd.display_frame_partial_window(f.bytes(), 0, epd::HEIGHT) {
log::warn!("caret repaint FAILED ({e}); full refresh next");
force_full = true;
} else {
shown = f;
cursor_shown = true;
log::info!("caret shown");
}
} else {
FreeRtos::delay_ms(8);
}
@@ -273,17 +289,29 @@ fn main() -> anyhow::Result<()> {
&& only_adds_ink(shown.bytes(), frame.bytes(), y0, y1);
let t0 = Instant::now();
let refresh = if periodic {
epd.display_frame(frame.bytes())?;
"FULL"
// `force_full` promotes to a full refresh after a failed paint: it
// rewrites both RAM banks, recovering from a partial that may have died
// mid-transfer and desynced them.
let (result, refresh) = if periodic || force_full {
(epd.display_frame(frame.bytes()), "FULL")
} else if additive {
epd.display_frame_partial_window(frame.bytes(), y0, y1 - y0 + 1)?;
"windowed"
(epd.display_frame_partial_window(frame.bytes(), y0, y1 - y0 + 1), "windowed")
} else {
epd.display_frame_partial_window(frame.bytes(), 0, epd::HEIGHT)?;
"full-area"
(epd.display_frame_partial_window(frame.bytes(), 0, epd::HEIGHT), "full-area")
};
let ms = t0.elapsed().as_millis();
if let Err(e) = result {
// Never fatal — the buffer is the source of truth and safe in RAM,
// exactly like a failed `save_note`. Drop this frame, leave `shown`
// untouched so the next paint repaints the same diff, and force a
// clean full refresh then. Typical cause: internal DMA-capable RAM
// briefly starved by Wi-Fi/TLS during a background `:sync`; it frees
// the moment the push finishes.
log::warn!("{refresh} refresh #{updates} FAILED ({e}); frame dropped, full refresh next");
force_full = true;
continue;
}
force_full = false;
log::info!(
"{refresh} refresh #{updates} [{:?}]: {ms} ms (rows {y0}..={y1}, {keys} key(s))",
ed.mode()