docs(postmortems): record Spike 3 SD CMD59 card incompatibility

New docs/postmortems/ folder. The bench card (133 GB SDXC) rejects CMD59 (SPI-mode CRC) as illegal; CMD0/CMD8 succeed, so wiring and firmware are proven and the fix is a compliant card. Captures the root cause, the keep-CRC-required decision, and the reusable findings (EPD bus lock, LFN requirement, R1-byte diagnostic) before the work pauses for hardware.
2026-07-05 18:16:49 +02:00
parent 9fc21568e7
commit 0b52f34940
2 changed files with 177 additions and 0 deletions
--- a/docs/postmortems/2026-07-05-spike3-sd-cmd59.md
+++ b/docs/postmortems/2026-07-05-spike3-sd-cmd59.md
@@ -0,0 +1,164 @@
+# Spike 3 (SD) — blocked on a card that rejects CMD59 (SPI-mode CRC)
+
+> Date: 2026-07-05 · Build at time of failure: `07-05 16:07Z @f77f669-dirty`
+> Status: **paused, not failed** — bench card is incompatible; awaiting a
+> compliant microSD. Wiring and firmware are proven good.
+>
+> Context: Spike 3 in
+> [`../v0.1-mvp-technical.md`](../v0.1-mvp-technical.md#hardware-bring-up-order),
+> storage split [ADR-007](../adr.md#adr-007-storage-split--fat-on-sd-for-working-copy-littlefs-on-flash-for-config),
+> firmware notes [`../../firmware/README.md`](../../firmware/README.md).
+> Spike program: [`../../firmware/src/bin/sd_fat.rs`](../../firmware/src/bin/sd_fat.rs).
+
+## Summary
+
+Built the Spike 3 bench program (`sd_fat`): bring up the SD card over the EPD's
+shared SPI2 bus, mount FAT, and prove the persistence module's atomic-save
+pattern (write `*.tmp` → fsync → rename → read-back). The card **never mounts**:
+SD init gets cleanly through CMD0 and CMD8 (the card even identifies as an
+SDHC/SDXC v2 card), then fails at **CMD59 (CRC_ON_OFF)**, which the card rejects
+as an illegal command (`ESP_ERR_NOT_SUPPORTED`, `0x106`). `sdmmc_card_init`
+treats that as fatal.
+
+Root cause is the **card**, not our wiring or code: the bench card is a 133 GB
+SDXC that doesn't implement CMD59 in SPI mode — common on large / cheap /
+counterfeit SDXC cards. It works fine on the Mac because macOS uses native SD
+mode, never the SPI fallback the ESP32 uses.
+
+**Decision:** keep CRC required (don't disable it to limp the bad card along —
+see below) and reject incompatible cards with a clear message. Resume when a
+genuine ≤32 GB card is on the bench.
+
+## Bench wiring (shared SPI2, proven good)
+
+| Signal | GPIO | Shared with EPD? |
+| ------ | ---- | ---------------- |
+| SCK    | 12   | yes (epd.rs)     |
+| MOSI   | 11   | yes (epd.rs)     |
+| MISO   | 13   | **no** — new line; the EPD is write-only and never used MISO |
+| SD CS  | 10   | no — EPD CS is 7 |
+
+The SD sits on the same SPI2 bus as the panel, on its own chip-select. The spike
+runs **SD-only** (see "EPD bus lock" below).
+
+## Timeline
+
+1. First flash → `sdmmc_send_cmd_crc_on_off returned 0x106`, `sdmmc_card_init
+   failed (0x106)`. Cryptic; could be wiring, pull-ups, speed, or card.
+2. Added internal pull-ups on SCK/MOSI/MISO/CS and drove the EPD CS (GPIO 7)
+   high (deselect the panel on the shared bus). **No change** — identical
+   failure at the same command. A deterministic failure at one command argues
+   against marginal signal integrity.
+3. Verified the card on the Mac: `Windows_FAT_32 TYPOENA`, 33.6 GB partition on
+   a 133 GB card, **healthy and readable**. Rules out a dead card / wrong FS.
+4. Read the esp-idf SD init source: the CMD8 handler *silently tolerates* the
+   same `ESP_ERR_NOT_SUPPORTED` (treats it as "not a v2 card"), while CMD59's
+   does not. So the failure might be a persistent bad response, not a CMD59
+   one-off — needed the raw R1 bytes to tell.
+5. Raised the compile-time log ceiling (`CONFIG_LOG_MAXIMUM_LEVEL_DEBUG`) and
+   bumped the `sdmmc_*` / `sdspi_*` tags to DEBUG at runtime. The dump was
+   decisive:
+   - `cmd=52` / `cmd=5` fail (`0x107`/`0x106`) — **normal**: those are SDIO
+     probes (CMD52/CMD5) a memory card doesn't answer.
+   - `sdmmc_sd: SDHC/SDXC card` — printed **only when CMD8 succeeds**. So CMD0
+     and CMD8 returned clean, correct responses (right 0xAA echo, no error bit).
+   - `cmd=59, R1 response: command not supported` — only CMD59 is rejected.
+6. Confirmed the breakout is a bare microSD→pin adapter (no level shifter), so
+   the 3.3 V SPI path is clean.
+
+## Root cause
+
+The card responds correctly to CMD0 and CMD8 (proving wiring, signal integrity,
+pull-ups, bus sharing, and the hand-built FFI mount path are all correct) but
+**rejects CMD59 (CRC_ON_OFF) as an illegal command**. CMD59 is mandatory per the
+SD spec; some large/counterfeit SDXC cards don't implement it in SPI mode. Since
+esp-idf's `sdmmc_init_spi_crc` hard-fails when CMD59 fails (and there is **no**
+Kconfig or host flag to skip it), the mount aborts.
+
+## What it was *not*
+
+- **Not wiring / a swap** — CMD0/CMD8 responses are clean and correct.
+- **Not signal integrity** — internal pull-ups changed nothing; failure is
+  deterministic at one command; init runs at 400 kHz where the jumpers are fine.
+- **Not the filesystem** — failure is at SD *protocol* init, before any FAT
+  access. The card is FAT32 and mounts on the Mac.
+- **Not a level-shifter breakout** — it's a bare 3.3 V adapter.
+- **Not our FFI** — the hand-rolled `SDSPI_HOST_DEFAULT()` descriptors work; the
+  driver reaches and drives the card correctly.
+
+## Decision: keep CRC required, reject bad cards clearly
+
+CMD59's job is to enable **CRC on data transfers**. The only way to mount this
+card is to tolerate CMD59 failing, i.e. run with data CRC **off**. For a device
+whose entire value is not losing the user's writing, giving up integrity
+checking — precisely on the cards that are already the sketchy ones — is the
+wrong trade. It would also require patching a vendored esp-idf function that
+re-applies on every update.
+
+So `sd_fat` now maps `ESP_ERR_NOT_SUPPORTED` from the mount to an actionable
+message ("card rejected CMD59… use a genuine card, ideally ≤32 GB…") instead of
+a raw code. This is the behavior the real `persistence` module should carry too:
+a bad card at boot should say "swap the SD," not hang on a hex code.
+
+A CRC-off patch remains *possible* if bench work ever needs this exact card, but
+it is explicitly **not recommended** and not applied.
+
+## Other findings worth keeping
+
+- **EPD bus lock forces SD-only (for now).** The EPD driver uses esp-idf-hal's
+  `SpiBusDriver`, whose constructor calls `spi_device_acquire_bus(BLOCK)` and
+  holds that **exclusive** lock for its whole lifetime (it needs CS held across
+  a cmd→data sequence while DC toggles). While held, any other device on SPI2 —
+  the SD — blocks. So the EPD and an arbitrated SD device can't both be live on
+  one host as things stand. This spike proves the SD stack; the shared-bus
+  **arbitration decision** (release/re-acquire around EPD ops, vs. the
+  risk-table fallback of giving the SD its own SPI3) is still open and is what a
+  follow-up must settle before integration.
+- **FatFS long filenames are required.** The atomic-save temp name
+  (`notes.md.tmp`, two dots) is not a valid 8.3 name, and FatFS defaults to
+  8.3-only. `CONFIG_FATFS_LFN_HEAP=y` is needed by this spike **and** the real
+  persistence path.
+- **`SDSPI_*_DEFAULT()` macros are bindgen-invisible.** Built the `sdmmc_host_t`
+  / `sdspi_device_config_t` descriptors by hand; the `SDMMC_HOST_FLAG_*` values
+  are `BIT(n)` macros bindgen can't fold, so they're inlined (`BIT(3)`/`BIT(5)`).
+- **Conservative SD clock.** `SD_FREQ_KHZ = 10_000` (vs. the 20 MHz default),
+  since the EPD needed 4 MHz on these bench jumpers. Init runs at 400 kHz
+  regardless, so this only affects post-init throughput — revisit on a real PCB.
+- **Diagnostic technique.** Raising the log ceiling + `esp_log_level_set` to read
+  the driver's per-command R1 bytes turned a cryptic error into a one-glance
+  root cause. Reusable for any esp-idf init failure.
+
+## Where the end-to-end (clone → SD → edit → push) stands
+
+- **Clone is out-of-band in v0.1** — the dev clones the remote onto the mounted
+  SD from a laptop; there is no first-clone on device
+  ([ADR-007](../adr.md#adr-007-storage-split--fat-on-sd-for-working-copy-littlefs-on-flash-for-config),
+  technical doc "Provisioning"). The bench card mounts on the Mac, so that step
+  is not blocked.
+- **Edit → save-to-SD** becomes testable the moment a compliant card mounts
+  (Spike 3 + the existing editor's atomic save). Closest milestone.
+- **SD → push** needs **Spike 7 (gitoxide over HTTPS+PAT)**, which is unbuilt and
+  needs **PSRAM enabled first** (not yet done). Spike 6 (Wi-Fi + TLS) proved the
+  network/TLS gate. So the full loop is not testable yet even with a good card.
+
+## Follow-ups
+
+- [ ] Re-run with a genuine ≤32 GB card → expect clean mount + round-trip.
+- [ ] Strip the debug logging once green: remove `CONFIG_LOG_MAXIMUM_LEVEL_DEBUG`
+      from `sdkconfig.defaults` and the `esp_log_level_set` block in `sd_fat.rs`.
+- [ ] Write up Spike 3 as verified in
+      [`../../firmware/README.md`](../../firmware/README.md) (wiring, LFN
+      requirement, card-compatibility note), matching the other spikes.
+- [ ] Record a **recommended-SD-card note** for v0.1 (genuine, ≤32 GB;
+      large/counterfeit SDXC may fail CMD59) — product doc / ADR-007.
+- [ ] Settle the **shared-bus arbitration** decision (EPD lock vs. SPI3 for SD).
+- [ ] Enable PSRAM, then build Spike 7 (gitoxide push) for the push leg.
+
+## Artifacts (this session)
+
+- `firmware/src/bin/sd_fat.rs` — the spike (mount + atomic round-trip + clear
+  rejection + debug logging).
+- `firmware/Cargo.toml` — `[[bin]] sd_fat`.
+- `firmware/justfile` — `build-sd` / `flash-sd` / `monitor-sd`.
+- `firmware/sdkconfig.defaults` — `CONFIG_FATFS_LFN_HEAP=y` (keep) and
+  `CONFIG_LOG_MAXIMUM_LEVEL_DEBUG=y` (temporary, strip when green).
--- a/docs/postmortems/README.md
+++ b/docs/postmortems/README.md
@@ -0,0 +1,13 @@
+# Postmortems
+
+> Bench + bring-up debugging write-ups: what broke, how we found the root cause,
+> and the decisions that came out of it. One file per incident, named
+> `YYYY-MM-DD-<slug>.md`. These capture *why* a spike stalled or a design turned
+> — the kind of context that's expensive to reconstruct later.
+>
+> Project overview: [`../../README.md`](../../README.md). Bring-up spikes:
+> [`../v0.1-mvp-technical.md`](../v0.1-mvp-technical.md#hardware-bring-up-order).
+
+| Date       | Incident                                                                 | Status |
+| ---------- | ------------------------------------------------------------------------ | ------ |
+| 2026-07-05 | [Spike 3 (SD) — card rejects CMD59 (SPI-mode CRC)](2026-07-05-spike3-sd-cmd59.md) | Paused — awaiting a compliant microSD; wiring + firmware proven |