From 0b52f3494025485fef35a4560d0ddee48de027b0 Mon Sep 17 00:00:00 2001 From: Julien Calixte Date: Sun, 5 Jul 2026 18:16:49 +0200 Subject: [PATCH] docs(postmortems): record Spike 3 SD CMD59 card incompatibility New docs/postmortems/ folder. The bench card (133 GB SDXC) rejects CMD59 (SPI-mode CRC) as illegal; CMD0/CMD8 succeed, so wiring and firmware are proven and the fix is a compliant card. Captures the root cause, the keep-CRC-required decision, and the reusable findings (EPD bus lock, LFN requirement, R1-byte diagnostic) before the work pauses for hardware. --- .../postmortems/2026-07-05-spike3-sd-cmd59.md | 164 ++++++++++++++++++ docs/postmortems/README.md | 13 ++ 2 files changed, 177 insertions(+) create mode 100644 docs/postmortems/2026-07-05-spike3-sd-cmd59.md create mode 100644 docs/postmortems/README.md diff --git a/docs/postmortems/2026-07-05-spike3-sd-cmd59.md b/docs/postmortems/2026-07-05-spike3-sd-cmd59.md new file mode 100644 index 0000000..7e005ac --- /dev/null +++ b/docs/postmortems/2026-07-05-spike3-sd-cmd59.md @@ -0,0 +1,164 @@ +# Spike 3 (SD) — blocked on a card that rejects CMD59 (SPI-mode CRC) + +> Date: 2026-07-05 · Build at time of failure: `07-05 16:07Z @f77f669-dirty` +> Status: **paused, not failed** — bench card is incompatible; awaiting a +> compliant microSD. Wiring and firmware are proven good. +> +> Context: Spike 3 in +> [`../v0.1-mvp-technical.md`](../v0.1-mvp-technical.md#hardware-bring-up-order), +> storage split [ADR-007](../adr.md#adr-007-storage-split--fat-on-sd-for-working-copy-littlefs-on-flash-for-config), +> firmware notes [`../../firmware/README.md`](../../firmware/README.md). +> Spike program: [`../../firmware/src/bin/sd_fat.rs`](../../firmware/src/bin/sd_fat.rs). + +## Summary + +Built the Spike 3 bench program (`sd_fat`): bring up the SD card over the EPD's +shared SPI2 bus, mount FAT, and prove the persistence module's atomic-save +pattern (write `*.tmp` → fsync → rename → read-back). The card **never mounts**: +SD init gets cleanly through CMD0 and CMD8 (the card even identifies as an +SDHC/SDXC v2 card), then fails at **CMD59 (CRC_ON_OFF)**, which the card rejects +as an illegal command (`ESP_ERR_NOT_SUPPORTED`, `0x106`). `sdmmc_card_init` +treats that as fatal. + +Root cause is the **card**, not our wiring or code: the bench card is a 133 GB +SDXC that doesn't implement CMD59 in SPI mode — common on large / cheap / +counterfeit SDXC cards. It works fine on the Mac because macOS uses native SD +mode, never the SPI fallback the ESP32 uses. + +**Decision:** keep CRC required (don't disable it to limp the bad card along — +see below) and reject incompatible cards with a clear message. Resume when a +genuine ≤32 GB card is on the bench. + +## Bench wiring (shared SPI2, proven good) + +| Signal | GPIO | Shared with EPD? | +| ------ | ---- | ---------------- | +| SCK | 12 | yes (epd.rs) | +| MOSI | 11 | yes (epd.rs) | +| MISO | 13 | **no** — new line; the EPD is write-only and never used MISO | +| SD CS | 10 | no — EPD CS is 7 | + +The SD sits on the same SPI2 bus as the panel, on its own chip-select. The spike +runs **SD-only** (see "EPD bus lock" below). + +## Timeline + +1. First flash → `sdmmc_send_cmd_crc_on_off returned 0x106`, `sdmmc_card_init + failed (0x106)`. Cryptic; could be wiring, pull-ups, speed, or card. +2. Added internal pull-ups on SCK/MOSI/MISO/CS and drove the EPD CS (GPIO 7) + high (deselect the panel on the shared bus). **No change** — identical + failure at the same command. A deterministic failure at one command argues + against marginal signal integrity. +3. Verified the card on the Mac: `Windows_FAT_32 TYPOENA`, 33.6 GB partition on + a 133 GB card, **healthy and readable**. Rules out a dead card / wrong FS. +4. Read the esp-idf SD init source: the CMD8 handler *silently tolerates* the + same `ESP_ERR_NOT_SUPPORTED` (treats it as "not a v2 card"), while CMD59's + does not. So the failure might be a persistent bad response, not a CMD59 + one-off — needed the raw R1 bytes to tell. +5. Raised the compile-time log ceiling (`CONFIG_LOG_MAXIMUM_LEVEL_DEBUG`) and + bumped the `sdmmc_*` / `sdspi_*` tags to DEBUG at runtime. The dump was + decisive: + - `cmd=52` / `cmd=5` fail (`0x107`/`0x106`) — **normal**: those are SDIO + probes (CMD52/CMD5) a memory card doesn't answer. + - `sdmmc_sd: SDHC/SDXC card` — printed **only when CMD8 succeeds**. So CMD0 + and CMD8 returned clean, correct responses (right 0xAA echo, no error bit). + - `cmd=59, R1 response: command not supported` — only CMD59 is rejected. +6. Confirmed the breakout is a bare microSD→pin adapter (no level shifter), so + the 3.3 V SPI path is clean. + +## Root cause + +The card responds correctly to CMD0 and CMD8 (proving wiring, signal integrity, +pull-ups, bus sharing, and the hand-built FFI mount path are all correct) but +**rejects CMD59 (CRC_ON_OFF) as an illegal command**. CMD59 is mandatory per the +SD spec; some large/counterfeit SDXC cards don't implement it in SPI mode. Since +esp-idf's `sdmmc_init_spi_crc` hard-fails when CMD59 fails (and there is **no** +Kconfig or host flag to skip it), the mount aborts. + +## What it was *not* + +- **Not wiring / a swap** — CMD0/CMD8 responses are clean and correct. +- **Not signal integrity** — internal pull-ups changed nothing; failure is + deterministic at one command; init runs at 400 kHz where the jumpers are fine. +- **Not the filesystem** — failure is at SD *protocol* init, before any FAT + access. The card is FAT32 and mounts on the Mac. +- **Not a level-shifter breakout** — it's a bare 3.3 V adapter. +- **Not our FFI** — the hand-rolled `SDSPI_HOST_DEFAULT()` descriptors work; the + driver reaches and drives the card correctly. + +## Decision: keep CRC required, reject bad cards clearly + +CMD59's job is to enable **CRC on data transfers**. The only way to mount this +card is to tolerate CMD59 failing, i.e. run with data CRC **off**. For a device +whose entire value is not losing the user's writing, giving up integrity +checking — precisely on the cards that are already the sketchy ones — is the +wrong trade. It would also require patching a vendored esp-idf function that +re-applies on every update. + +So `sd_fat` now maps `ESP_ERR_NOT_SUPPORTED` from the mount to an actionable +message ("card rejected CMD59… use a genuine card, ideally ≤32 GB…") instead of +a raw code. This is the behavior the real `persistence` module should carry too: +a bad card at boot should say "swap the SD," not hang on a hex code. + +A CRC-off patch remains *possible* if bench work ever needs this exact card, but +it is explicitly **not recommended** and not applied. + +## Other findings worth keeping + +- **EPD bus lock forces SD-only (for now).** The EPD driver uses esp-idf-hal's + `SpiBusDriver`, whose constructor calls `spi_device_acquire_bus(BLOCK)` and + holds that **exclusive** lock for its whole lifetime (it needs CS held across + a cmd→data sequence while DC toggles). While held, any other device on SPI2 — + the SD — blocks. So the EPD and an arbitrated SD device can't both be live on + one host as things stand. This spike proves the SD stack; the shared-bus + **arbitration decision** (release/re-acquire around EPD ops, vs. the + risk-table fallback of giving the SD its own SPI3) is still open and is what a + follow-up must settle before integration. +- **FatFS long filenames are required.** The atomic-save temp name + (`notes.md.tmp`, two dots) is not a valid 8.3 name, and FatFS defaults to + 8.3-only. `CONFIG_FATFS_LFN_HEAP=y` is needed by this spike **and** the real + persistence path. +- **`SDSPI_*_DEFAULT()` macros are bindgen-invisible.** Built the `sdmmc_host_t` + / `sdspi_device_config_t` descriptors by hand; the `SDMMC_HOST_FLAG_*` values + are `BIT(n)` macros bindgen can't fold, so they're inlined (`BIT(3)`/`BIT(5)`). +- **Conservative SD clock.** `SD_FREQ_KHZ = 10_000` (vs. the 20 MHz default), + since the EPD needed 4 MHz on these bench jumpers. Init runs at 400 kHz + regardless, so this only affects post-init throughput — revisit on a real PCB. +- **Diagnostic technique.** Raising the log ceiling + `esp_log_level_set` to read + the driver's per-command R1 bytes turned a cryptic error into a one-glance + root cause. Reusable for any esp-idf init failure. + +## Where the end-to-end (clone → SD → edit → push) stands + +- **Clone is out-of-band in v0.1** — the dev clones the remote onto the mounted + SD from a laptop; there is no first-clone on device + ([ADR-007](../adr.md#adr-007-storage-split--fat-on-sd-for-working-copy-littlefs-on-flash-for-config), + technical doc "Provisioning"). The bench card mounts on the Mac, so that step + is not blocked. +- **Edit → save-to-SD** becomes testable the moment a compliant card mounts + (Spike 3 + the existing editor's atomic save). Closest milestone. +- **SD → push** needs **Spike 7 (gitoxide over HTTPS+PAT)**, which is unbuilt and + needs **PSRAM enabled first** (not yet done). Spike 6 (Wi-Fi + TLS) proved the + network/TLS gate. So the full loop is not testable yet even with a good card. + +## Follow-ups + +- [ ] Re-run with a genuine ≤32 GB card → expect clean mount + round-trip. +- [ ] Strip the debug logging once green: remove `CONFIG_LOG_MAXIMUM_LEVEL_DEBUG` + from `sdkconfig.defaults` and the `esp_log_level_set` block in `sd_fat.rs`. +- [ ] Write up Spike 3 as verified in + [`../../firmware/README.md`](../../firmware/README.md) (wiring, LFN + requirement, card-compatibility note), matching the other spikes. +- [ ] Record a **recommended-SD-card note** for v0.1 (genuine, ≤32 GB; + large/counterfeit SDXC may fail CMD59) — product doc / ADR-007. +- [ ] Settle the **shared-bus arbitration** decision (EPD lock vs. SPI3 for SD). +- [ ] Enable PSRAM, then build Spike 7 (gitoxide push) for the push leg. + +## Artifacts (this session) + +- `firmware/src/bin/sd_fat.rs` — the spike (mount + atomic round-trip + clear + rejection + debug logging). +- `firmware/Cargo.toml` — `[[bin]] sd_fat`. +- `firmware/justfile` — `build-sd` / `flash-sd` / `monitor-sd`. +- `firmware/sdkconfig.defaults` — `CONFIG_FATFS_LFN_HEAP=y` (keep) and + `CONFIG_LOG_MAXIMUM_LEVEL_DEBUG=y` (temporary, strip when green). diff --git a/docs/postmortems/README.md b/docs/postmortems/README.md new file mode 100644 index 0000000..ece9fb1 --- /dev/null +++ b/docs/postmortems/README.md @@ -0,0 +1,13 @@ +# Postmortems + +> Bench + bring-up debugging write-ups: what broke, how we found the root cause, +> and the decisions that came out of it. One file per incident, named +> `YYYY-MM-DD-.md`. These capture *why* a spike stalled or a design turned +> — the kind of context that's expensive to reconstruct later. +> +> Project overview: [`../../README.md`](../../README.md). Bring-up spikes: +> [`../v0.1-mvp-technical.md`](../v0.1-mvp-technical.md#hardware-bring-up-order). + +| Date | Incident | Status | +| ---------- | ------------------------------------------------------------------------ | ------ | +| 2026-07-05 | [Spike 3 (SD) — card rejects CMD59 (SPI-mode CRC)](2026-07-05-spike3-sd-cmd59.md) | Paused — awaiting a compliant microSD; wiring + firmware proven |