Compare commits
10 Commits
79fad4689c
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5f50cf1b45 | ||
|
|
5c97e019ec | ||
|
|
7712de6153 | ||
|
|
1df25f75d6 | ||
|
|
acbafa3d6b | ||
|
|
e801d2d480 | ||
|
|
a771c3f6de | ||
|
|
4c92b0dddc | ||
|
|
af5b41a104 | ||
|
|
2660a3e9dd |
@@ -33,3 +33,4 @@
|
||||
| [`postmortems/`](postmortems/README.md) | Bring-up debugging write-ups: what broke, the root cause, and the decisions that came out of it. |
|
||||
| [`notes/`](notes/README.md) | Longer-form essays on the thinking behind specific choices — e.g. where the ~16 s cold [`:sync`](notes/sync-latency.md) goes. |
|
||||
| [`tradeoff-curves/`](tradeoff-curves/README.md) | Cost-vs-knob curves behind chosen defaults — energy, latency, memory. |
|
||||
| [`kaizen/real-repo-sync.md`](kaizen/real-repo-sync.md) | Six-step kaizen write-ups — the problem→analysis→fix story behind an improvement, e.g. the real-repo `:sync` brick. |
|
||||
|
||||
280
docs/kaizen/real-repo-sync.md
Normal file
280
docs/kaizen/real-repo-sync.md
Normal file
@@ -0,0 +1,280 @@
|
||||
# Kaizen — `:sync` never completes on the real notes repo
|
||||
|
||||
The writer presses `:sync` on the Typoena appliance and the device freezes for
|
||||
ten minutes; a power-cycle re-enters the same freeze forever. The real
|
||||
`jcalixte/notes` clone (1179 files, 263 MB pack) has never been synced from the
|
||||
device — only the toy test repo has. Full measurement trail:
|
||||
[`../tradeoff-curves/sync-commit-staging.md`](../tradeoff-curves/sync-commit-staging.md).
|
||||
|
||||
## 1) Improvement potential
|
||||
|
||||
**Value model for the writer (Julien on the appliance)**
|
||||
|
||||
| Want more of | Do less of |
|
||||
| ----------------------------------------------- | ------------------------------------------------------- |
|
||||
| + words safely on the remote with one keystroke | – waiting on a frozen device |
|
||||
| + trust that `:sync` completes, every time | – power-cycling mid-sync and losing the session's trust |
|
||||
| + keep writing while it publishes | – babysitting git from a desktop to rescue the card |
|
||||
|
||||
**Measurement**: seconds from `:sync` to the snackbar, on the real
|
||||
`jcalixte/notes` clone on the device.
|
||||
|
||||
```mermaid
|
||||
---
|
||||
config:
|
||||
xyChart:
|
||||
width: 900
|
||||
height: 500
|
||||
---
|
||||
xychart-beta
|
||||
title "Improvement potential — :sync → snackbar"
|
||||
x-axis ["Before kaizen (~611 s, never completes)", "After kaizen (target ≤ ~10 s)"]
|
||||
y-axis "seconds" 0 --> 650
|
||||
bar [611]
|
||||
line [10, 10]
|
||||
```
|
||||
|
||||
The before bar understates the reality: a reset re-enters the freeze, so the
|
||||
true value is ∞ — 0 successful syncs ever. The line is the ≤ ~10 s target;
|
||||
|
||||
## 2) Current method analysis
|
||||
|
||||
```
|
||||
:sync (current method, real repo)
|
||||
│
|
||||
▼
|
||||
┌─ local commit ─────────────────────────────────────┐ ┌─ network push ──────┐
|
||||
│ stage: add_all(["*"]) + update_all(["*"]) │ │ TLS + push │
|
||||
│ → stat()s 1179 files / 158 dirs over 10 MHz SPI │──▶│ ~6.5 s floor │
|
||||
│ index.write ⚡ ~611 s │ │ (separate curve) │
|
||||
│ → FAT's 2 s mtime marks ~every entry "racy" │ └─────────────────────┘
|
||||
│ → re-hashes ~170 MB (150 MB of images) │
|
||||
│ write_tree + commit-obj ⚡ seconds each │
|
||||
│ → loose writes trigger pack reads through │
|
||||
│ emulated mmap; each far lseek walks the │
|
||||
│ FAT cluster chain (~190 ms) │
|
||||
└────────────────────────────────────────────────────┘
|
||||
⚡ = weak points, measured on device
|
||||
```
|
||||
|
||||
Every factor was benched on the device (`sd_bench` for raw FAT, `git_bench` for
|
||||
git2 primitives on the real clone) — details and full tables in
|
||||
[`../tradeoff-curves/sync-commit-staging.md`](../tradeoff-curves/sync-commit-staging.md):
|
||||
|
||||
| Factor # | Factor | Hypothesis | Test method | Test result | True or false? |
|
||||
| -------- | --------------------------- | ------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | -------------------- |
|
||||
| 1 | SD card raw I/O | The card itself is slow: a loose-object write costs ~700–900 ms | `sd_bench`: raw FAT create/write/rename composite, same card, same 10 MHz | complete loose-object composite **86 ms** — 8× under libgit2's number | **FALSE** |
|
||||
| 2 | `index.write()` racy-clean | FAT's 2 s mtime granularity makes ~all 1179 entries look racy → full ~170 MB re-hash over SPI | `git_bench` on the real clone, 3 consecutive `index.write()` | **611 s → 12.8 s → 360 ms** — the decay is the racy set shrinking as the index mtime advances | **TRUE** |
|
||||
| 3 | Index-free alternative | Skipping `index.write` (fresh in-memory index) escapes the wall | `git_bench`: `Index::new` + `read_tree(HEAD)` + `write_tree_to` | seed `read_tree` **77 s cold** + mmap grows to 7.4 MB → **zlib OOM crash** — still O(N_tree) | **FALSE** (as a fix) |
|
||||
| 4 | FatFS `lseek` | Long/backward seeks walk the FAT cluster chain over SPI (~67 KB of FAT reads for a 263 MB file) | `sd_bench`: seek+read 4 KB @offset 0 vs @end of the pack; A/B with `CONFIG_FATFS_USE_FASTSEEK` | @0 = 5.8 ms, @end = **198.7 ms**; fast-seek → **20.4 ms** | **TRUE** |
|
||||
| 5 | Free-cluster scan | A ~740 MB-full card slows FAT allocation | `sd_bench` re-run on the full card | composite 77 ms — unchanged | **FALSE** |
|
||||
| 6 | Strict object creation | OID validation on commit/treebuilder causes the ~1.3 s commit premium | strict-off A/B in `git_bench` | 1.80 s vs 1.93 s — no premium removed | **FALSE** |
|
||||
| 7 | Repeated small mmap windows | A window cache in `esp_map.c` would absorb the ~8 small pack reads per loose write | instrumented cache (v2), 4 real-repo runs | **0 hits / 313 misses** — every read hits a unique (offset, len); `mwindow` already absorbs true repetition | **FALSE** |
|
||||
| 8 | Cache memory discipline | The OOM in factor 3 is our own cache holding buffers past `p_munmap`, defeating `MWINDOW_MAPPED_LIMIT` | run-4 memory monitor after evict-on-munmap fix, then full removal (run 5) | resident flat at 1833 KB, heap ≥ 6.2 MB, no OOM; removal I/O-neutral | **TRUE** |
|
||||
|
||||
### Details on hypothesis #2 (the freeze itself)
|
||||
|
||||
`git_index_write` unconditionally runs `truncate_racily_clean`, which re-hashes
|
||||
every entry whose mtime is not strictly older than the index stamp. On a fresh
|
||||
FAT clone the 2 s mtime granularity makes the whole tree racy, so a one-line
|
||||
note edit re-hashes ~170 MB — mostly the ~260 images a text edit never touches —
|
||||
over the 10 MHz SPI bus. This is why the device freezes ~10 minutes and why a
|
||||
reset never helps: the index mtime doesn't advance, so it re-hashes forever.
|
||||
|
||||
**Selected weak point**: the local commit stage — its staging strategy is
|
||||
O(N_tree) in files and bytes, on a device where N_tree ≈ 1179 and the bus is
|
||||
10 MHz SPI. (The ~6.5 s network half is real but is a separate curve,
|
||||
[`../notes/sync-latency.md`](../notes/sync-latency.md), and it does not brick
|
||||
the device.)
|
||||
|
||||
## 3) Ideas
|
||||
|
||||
### Bibliography
|
||||
|
||||
- [FatFS `f_lseek` docs (elm-chan.org)](http://elm-chan.org/fsw/ff/doc/lseek.html) — the cluster-chain walk and the CLMT fast-seek mechanism.
|
||||
- [ESP-IDF FatFS docs](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/fatfs.html) — `CONFIG_FATFS_USE_FASTSEEK`, recommended for read-heavy workloads with long backward seeks.
|
||||
- Vendored libgit2 v1.9.4 source — `index.c` `truncate_racily_clean` (the re-hash wall), `odb.c` `git_odb__freshen`/`git_odb_refresh` (the per-write pack probes), `mwindow.c` (32-bit defaults: 32 MB window / 256 MB mapped limit).
|
||||
|
||||
### New ideas
|
||||
|
||||
| Name | Estimated gain | Estimated lead time | Cause addressed | Comments |
|
||||
| ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | --------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **O(depth) TreeBuilder splice** — patch only the edited path's ancestor chain onto HEAD's tree | commit 611 s → ~2–3 s; **flat in repo size forever** | ~2 days (bench op, then plumbing) | #2 + #3 — never opens the index, never materialises the tree | **Chosen.** Bonus: carries the 150 MB of images forward by OID, so the device doesn't even need them in its working tree |
|
||||
| `CONFIG_FATFS_USE_FASTSEEK` + 256-word CLMT buffer | 2.3× on the splice (6.5 → 2.8 s); far seek 198.7 → 20.4 ms | hours — config, not code | #4 | Shipped as the companion fix; write-mode files fall back transparently |
|
||||
| Explicit-path index staging (`add_path` over the editor's dirty set) | removes the O(N) walk term only | ~1 day | #2 partially | Retired — still calls `index.write()`, so it hits the same racy-clean wall |
|
||||
| Index-free `write_tree_to` on a fresh in-memory index | — | ~1 day | #2 | Refuted by factor 3: seeding `read_tree(HEAD)` is 77 s + OOM — trades one O(N) for another |
|
||||
| `esp_map.c` window cache (v2: size-keyed admission, evict-on-munmap) | hoped ~150–250 ms per loose write | ~1 day | #7 | Built and instrumented → **0 hits in 4 runs** → removed entirely. The one build made on an unverified cause; see learnings |
|
||||
| Faster card / SD at 20 MHz | none for the commit | PCB respin budget | #1 | Rejected — factor 1 refuted; the card was exonerated twice (86 ms then 77 ms composites) |
|
||||
| Shrink the repo (images off-card / LFS-style pointers) | shrinks N and the 570 MB clone | unknown | N itself | Rejected for now — the images are load-bearing for another app ([`../notes/git-sync-images-and-repo-size.md`](../notes/git-sync-images-and-repo-size.md)). Composes with the splice later |
|
||||
|
||||
**Chosen idea**: the O(depth) TreeBuilder splice, with fast-seek as its config
|
||||
companion — it is the only candidate whose cost is independent of repo size, so
|
||||
the problem cannot come back as the notes tree grows. Everything index-shaped
|
||||
stays O(N_tree) and merely moves the wall.
|
||||
|
||||
## 4) Test plan
|
||||
|
||||
**What could go wrong?**
|
||||
|
||||
| Lens | Anticipated consequence | Mitigation |
|
||||
| ------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Stable | libgit2's 32-bit mwindow defaults `malloc` a 32 MB window on first pack access → instant OOM on the 8 MB PSRAM heap | `GIT_OPT_SET_MWINDOW_SIZE` 256 KB / `MAPPED_LIMIT` 4 MB set at service start, before any `Repository::open` |
|
||||
| Stable | Power pull between a save and the next `:sync` loses the dirty set — nothing walks the tree anymore, so an unrecorded file would **never** sync | Dirty set journaled to `/sd/.typoena-dirty` (atomic write), recorded _before_ the file write; over-reporting is a free no-op splice, so the semantics stay simple |
|
||||
| Stable | Commit lands, push fails → the commit strands forever behind "up to date" | Stranded-commit recovery: compare HEAD against `refs/remotes/origin/<branch>` and push even when there is nothing new to commit |
|
||||
| Method | Reconcile's `Mixed` reset writes the index → re-enters the racy-clean wall through the back door | `ResetType::Soft` — ref move only; there is no index to reset anymore |
|
||||
| Method | **Deliberate behavior change:** files edited on the card from a desktop are never committed anymore (`add_all` used to sweep them in) | Accepted and documented as intent — the appliance's editor is the only writer that counts |
|
||||
| Machine | libgit2 holds the pack + `.idx` descriptors open and opens loose objects on top → blows the editor's 4-FD mount | git builds mount with 16 FDs (`Storage::mount_for_git()`) |
|
||||
|
||||
The mechanism was stress-tested as a prototype first, never in production: the
|
||||
splice ran as a `git_bench` op against a full real-repo clone through four
|
||||
localization rounds before any firmware plumbing.
|
||||
|
||||
**Who must we convince?**
|
||||
|
||||
- Upstream libgit2 maintainers — the tlsf double-free in the mbedTLS stream
|
||||
error path (found during rollout; `wrap`'s error path frees the caller's
|
||||
socket stream, `new()` frees it again). Needs an upstream report; we ship a
|
||||
whole-file override meanwhile.
|
||||
- Desktop-me (and any future contributor) — the card's working tree now shows a
|
||||
permanent diff against HEAD when inspected on a Mac. That is intent, not a
|
||||
bug.
|
||||
|
||||
**Rollback**: the `git` feature flag gates the whole sync stack (the light
|
||||
build never links it), and the plumbing is a small commit range to revert. The
|
||||
journal file is additive — an old build simply ignores it.
|
||||
|
||||
**Measurement protocol for step 6**: full `:sync` on the device against the
|
||||
real clone — `commit split —` log lines plus snackbar-to-snackbar wall time,
|
||||
cold and steady-state.
|
||||
|
||||
## 5) Implementation
|
||||
|
||||
**Before** — `stage_and_commit` walked and hashed the world through the index
|
||||
(condensed; `firmware/src/git_sync.rs` at `a5edaed~1`):
|
||||
|
||||
```rust
|
||||
fn stage_and_commit(repo: &Repository) -> Result<Option<Oid>> {
|
||||
let mut index = repo.index()?;
|
||||
index.add_all(["*"], IndexAddOption::DEFAULT, Some(&mut skip_macos_cruft))?;
|
||||
index.update_all(["*"], Some(&mut skip_macos_cruft))?; // stat 1179 files
|
||||
index.write()?; // ⚡ racy-clean re-hash: ~611 s on FAT
|
||||
let tree = repo.find_tree(index.write_tree()?)?;
|
||||
// … commit(Some("HEAD"), …, &tree, &parents)
|
||||
}
|
||||
```
|
||||
|
||||
**After** — the editor's journaled dirty paths are spliced onto HEAD's tree;
|
||||
the index never exists (condensed; `firmware/src/git_sync.rs:345`):
|
||||
|
||||
```rust
|
||||
fn stage_and_commit(repo: &Repository, paths: &BTreeSet<String>) -> Result<Option<Oid>> {
|
||||
let parent = repo.head().ok().and_then(|h| h.peel_to_commit().ok());
|
||||
let mut tree = parent.as_ref().map(|c| c.tree()).transpose()?;
|
||||
for path in paths {
|
||||
let blob = match fs::read(format!("{REPO_DIR}/{path}")) {
|
||||
Ok(bytes) => Some(repo.blob(&bytes)?), // present → splice in
|
||||
Err(e) if e.kind() == NotFound => None, // deleted → splice out
|
||||
Err(e) => return Err(e.into()),
|
||||
};
|
||||
let spliced = splice(repo, tree.as_ref(), &parts(path), blob)?;
|
||||
tree = Some(repo.find_tree(spliced)?);
|
||||
}
|
||||
// … same "tree unchanged → nothing to publish" check, same commit call
|
||||
}
|
||||
|
||||
/// Reads ~depth tree objects, writes ~depth new ones; every sibling entry is
|
||||
/// carried by OID — never opened.
|
||||
fn splice(repo: &Repository, base: Option<&Tree>, path: &[&str], blob: Option<Oid>) -> Result<Oid>
|
||||
```
|
||||
|
||||
The same pipeline, redrawn with the change applied:
|
||||
|
||||
```
|
||||
:sync (new method, real repo)
|
||||
│
|
||||
▼
|
||||
┌─ local commit ───────────────────────────────┐ ┌─ network push ──────┐
|
||||
│ splice: for each journaled dirty path (≈1) │ │ TLS + push │
|
||||
│ read file → blob → rebuild its ancestor │──▶│ ~6.5 s floor │
|
||||
│ subtree chain (~depth tree objects) │ │ (untouched — next │
|
||||
│ commit-obj │ │ curve to attack) │
|
||||
│ no index · no walk · no re-hash · images │ └─────────────────────┘
|
||||
│ carried by OID · lseek O(1) via fast-seek │
|
||||
└──────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Around the mechanism, the plumbing that makes it safe day-to-day: the
|
||||
`/sd/.typoena-dirty` journal with its pending → in-flight → settled lifecycle,
|
||||
stranded-commit recovery, a radio-free "up to date" answer (~150 ms instead of
|
||||
a Wi-Fi/TLS round), soft-reset reconcile, the 16-FD git mount, and the mwindow
|
||||
options at service start. Details:
|
||||
[the fix — wiring](../tradeoff-curves/sync-commit-staging.md#the-fix--wiring-the-odepth-splice-into-the-firmware).
|
||||
|
||||
## 6) Evaluation
|
||||
|
||||
**Measurement redone** (seconds, `:sync` → snackbar on the real repo):
|
||||
∞ (never completed) → **PENDING end-to-end** (target was ≤ ~10 s).
|
||||
|
||||
What is measured so far, on device against the real clone (2026-07-13):
|
||||
|
||||
- The commit half works for the first time ever: splice **4.2 s** + commit-obj
|
||||
**3.2 s** with the UI running concurrently (commits `a73bca0e`, then
|
||||
`8939168f` carrying a 2-path journal across a power cycle).
|
||||
- Projected full cold `:sync` ≈ **9–10 s** (commit + the ~6.5 s network half).
|
||||
- The push half is not yet verified: the first attempt hit the card's
|
||||
SSH-shaped origin (now rewritten to HTTPS at load), the second a tlsf
|
||||
double-free in libgit2's mbedTLS stream error path (fixed via the
|
||||
`esp_mbedtls_stream.c` override + `CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=y`).
|
||||
The step-1 number lands when the next on-device `:sync` completes
|
||||
snackbar-to-snackbar — protocol in step 4.
|
||||
|
||||
**Learnings**:
|
||||
|
||||
- **Bench the real thing.** The toy repo understated everything by ~2 orders
|
||||
of magnitude; "works on the toy" hid a total brick. The real clone is now
|
||||
the only valid bench target.
|
||||
- **Refute before building.** Four theories died to measurement (card speed,
|
||||
free-cluster scan, strict creation, repeated windows). The one build made on
|
||||
an unverified cause — the `esp_map.c` window cache — was the one wasted
|
||||
build: 0 hits in 4 instrumented runs, removed entirely.
|
||||
- **FAT breaks POSIX assumptions.** 2 s mtime granularity, no inode,
|
||||
cluster-chain seeks: any POSIX-shaped library (libgit2's racy-clean check,
|
||||
the mmap emulation) needs its cost model audited on FAT before trusting it.
|
||||
- **Prefer mechanisms flat in N.** The splice cannot regress as the notes tree
|
||||
grows — the fix that prevents the problem from ever coming back, which is
|
||||
the kind kaizen prefers.
|
||||
|
||||
**Standard to update**:
|
||||
|
||||
- Bench and verify against the **real repo clone**, never the toy (the toy is
|
||||
~2 orders of magnitude too kind) — recorded in the tradeoff doc's
|
||||
[how to bench](../tradeoff-curves/sync-commit-staging.md#how-to-bench--flash).
|
||||
- Any new libgit2 entry point **must set the mwindow options before its first
|
||||
`Repository::open`** — the 32-bit defaults OOM the 8 MB heap (done in
|
||||
`git_sync` and `git_bench`; the rule is the standard).
|
||||
- `CONFIG_FATFS_USE_FASTSEEK=y` + 256-word CLMT buffer stay pinned in
|
||||
`sdkconfig.defaults`.
|
||||
|
||||
**Share with**:
|
||||
|
||||
- Upstream libgit2 — file the tlsf double-free report (mbedTLS stream error
|
||||
path: `wrap`'s error path frees the caller's socket stream, `new()` frees it
|
||||
again; v1.9.4).
|
||||
- A write-up for the notes/blog — the four-round localization story (611 s →
|
||||
~2.8 s commit on a microcontroller, four theories refuted on the way) stands
|
||||
on its own.
|
||||
|
||||
**Next steps**
|
||||
|
||||
- Run the end-to-end on-device `:sync` verify and close the step-1
|
||||
measurement (the stranded `8939168f` should push first).
|
||||
- File the libgit2 upstream bug report.
|
||||
- Instrument the residual ~360 ms/loose-write (suspect: FAT directory-op cost
|
||||
in the freshen/refresh path) — one `sd_bench` + `p_mmap`-miss logging pass.
|
||||
- Measure the ref/reflog leg of the shipping commit (the bench's
|
||||
`commit(None)` writes no ref).
|
||||
- Attack the next curve: the ~6.5 s network floor
|
||||
([`../notes/sync-latency.md`](../notes/sync-latency.md)); the
|
||||
images-off-card lever
|
||||
([`../notes/git-sync-images-and-repo-size.md`](../notes/git-sync-images-and-repo-size.md))
|
||||
composes with the splice.
|
||||
@@ -26,7 +26,9 @@
|
||||
> measurement trail that got here.
|
||||
>
|
||||
> Tradeoff-curves index: [`README.md`](README.md). Docs index:
|
||||
> [`../README.md`](../README.md). Where the whole sync goes:
|
||||
> [`../README.md`](../README.md). The six-step kaizen retelling of this whole
|
||||
> investigation (problem → factor analysis → chosen fix → evaluation):
|
||||
> [`../kaizen/real-repo-sync.md`](../kaizen/real-repo-sync.md). Where the whole sync goes:
|
||||
> [`../notes/sync-latency.md`](../notes/sync-latency.md). Sibling curve on the
|
||||
> radio cost of *how often* we sync: [`wifi-auto-sync.md`](wifi-auto-sync.md).
|
||||
|
||||
|
||||
@@ -167,6 +167,25 @@ away — and the full fuzzy-ranked list appears from 2 chars on
|
||||
`>` commands and `$` snippets are short curated lists; the threshold does not
|
||||
apply to them.
|
||||
|
||||
**TODO (on-device, next time the device is on the bench)** — two measurements
|
||||
from the same boot log, both already instrumented:
|
||||
|
||||
- [ ] **Re-measure the walk time** after the d_type fix (`2660a3e` — dirent
|
||||
`file_type()` instead of a per-entry `metadata()` stat, which cost
|
||||
~32 ms/file and made run 1 take 35 s for 1098 files). Read the
|
||||
`file walk: N files in Xms` line. Only if it's still slow does the
|
||||
async-walk idea come back on the table.
|
||||
- [ ] **Read the file-list DRAM cost** from the new
|
||||
`file list: internal heap <before> -> <after> (<N> KB consumed)` line
|
||||
(the build is bracketed with `MALLOC_CAP_INTERNAL` readings in
|
||||
`main.rs`). The 1098 path Strings are each below the 16 KB SPIRAM
|
||||
malloc threshold, so they all land in internal DRAM — estimated
|
||||
60–70 KB, competing with Wi-Fi/TLS. Decision rule: **~60–70 KB
|
||||
confirms** interning the paths into one shared buffer (a single
|
||||
>16 KB alloc goes to PSRAM; only a ~9 KB offset index stays in DRAM);
|
||||
**well under that (≤~30 KB)** kills the idea — the next DRAM suspect
|
||||
is then parked-buffer text.
|
||||
|
||||
- [x] `Cmd-P` opens fuzzy file palette over **both** `/sd/repo/` and
|
||||
`/sd/local/` — **landed and CONFIRMED ON DEVICE 2026-07-12** (Spike 11: no
|
||||
ghosting on the transient panel); scope shows as the inline
|
||||
|
||||
@@ -41,6 +41,12 @@ file(GLOB LG2_SRCS
|
||||
"${LG2}/deps/pcre/*.c"
|
||||
"${LG2}/deps/zlib/*.c"
|
||||
)
|
||||
# streams/mbedtls.c is replaced by our patched copy: its wrap() error path
|
||||
# double-freed the socket stream (tlsf abort on device when ssl_setup failed
|
||||
# under memory pressure). See esp_mbedtls_stream.c's header for the one-hunk
|
||||
# delta; keep the copy in lockstep on submodule bumps.
|
||||
list(REMOVE_ITEM LG2_SRCS "${LG2}/src/libgit2/streams/mbedtls.c")
|
||||
|
||||
list(APPEND LG2_SRCS
|
||||
"${LG2}/src/util/allocators/failalloc.c"
|
||||
"${LG2}/src/util/allocators/stdalloc.c"
|
||||
@@ -48,6 +54,7 @@ list(APPEND LG2_SRCS
|
||||
"${LG2}/src/util/hash/mbedtls.c" # SHA1 + SHA256 via mbedtls
|
||||
"${CMAKE_CURRENT_LIST_DIR}/esp_map.c" # p_mmap via malloc+read (no <sys/mman.h>)
|
||||
"${CMAKE_CURRENT_LIST_DIR}/esp_stubs.c" # getuid/readlink/utimes/... stubs
|
||||
"${CMAKE_CURRENT_LIST_DIR}/esp_mbedtls_stream.c" # streams/mbedtls.c + double-free fix
|
||||
# NOTE: unix/map.c replaced by esp_map.c — picolibc has no <sys/mman.h>.
|
||||
# NOTE: unix/process.c deliberately excluded — needs fork()/sys/wait.h,
|
||||
# only used by the SSH-exec transport we don't enable.
|
||||
|
||||
498
firmware/components/libgit2/esp_mbedtls_stream.c
Normal file
498
firmware/components/libgit2/esp_mbedtls_stream.c
Normal file
@@ -0,0 +1,498 @@
|
||||
/*
|
||||
* Copyright (C) the libgit2 contributors. All rights reserved.
|
||||
*
|
||||
* This file is part of libgit2, distributed under the GNU GPL v2 with
|
||||
* a Linking Exception. For full terms see the included COPYING file.
|
||||
*/
|
||||
|
||||
/*
|
||||
* esp_mbedtls_stream.c — verbatim copy of the vendored
|
||||
* src/libgit2/streams/mbedtls.c (v1.9.4) with ONE fix; it replaces the
|
||||
* vendored file in CMakeLists.txt (same pattern as esp_map.c).
|
||||
*
|
||||
* THE FIX (2026-07-13): mbedtls_stream_wrap()'s `out_err` path closed and
|
||||
* freed `st->io` — the caller's socket stream — but every caller frees that
|
||||
* stream on error too (git_mbedtls_stream_new does close+free right after),
|
||||
* and wrap's OTHER error paths (calloc/strdup failures) do NOT free it, so
|
||||
* the caller cannot compensate either way. When mbedtls_ssl_setup failed on
|
||||
* the device (internal-RAM exhaustion during the first real-repo push), the
|
||||
* double git__free tripped tlsf ("block already marked as free") and reset
|
||||
* the chip instead of surfacing a clean error. Delta from vendor: the
|
||||
* `out_err` label no longer touches st->io — on error, ownership of `in`
|
||||
* stays with the caller, consistently — and it frees the git__malloc'd
|
||||
* st->ssl struct the vendored path leaked.
|
||||
*
|
||||
* Keep this file in lockstep with the vendored one on submodule bumps (diff
|
||||
* against it; the delta must stay this one hunk).
|
||||
*/
|
||||
|
||||
#include "streams/mbedtls.h"
|
||||
|
||||
#ifdef GIT_MBEDTLS
|
||||
|
||||
#include <ctype.h>
|
||||
|
||||
#include "runtime.h"
|
||||
#include "stream.h"
|
||||
#include "streams/socket.h"
|
||||
#include "git2/transport.h"
|
||||
#include "util.h"
|
||||
|
||||
#ifndef GIT_DEFAULT_CERT_LOCATION
|
||||
#define GIT_DEFAULT_CERT_LOCATION NULL
|
||||
#endif
|
||||
|
||||
/* Work around C90-conformance issues */
|
||||
#if !defined(__STDC_VERSION__) || (__STDC_VERSION__ < 199901L)
|
||||
# if defined(_MSC_VER)
|
||||
# define inline __inline
|
||||
# elif defined(__GNUC__)
|
||||
# define inline __inline__
|
||||
# else
|
||||
# define inline
|
||||
# endif
|
||||
#endif
|
||||
|
||||
#include <mbedtls/ssl.h>
|
||||
#include <mbedtls/error.h>
|
||||
#include <mbedtls/entropy.h>
|
||||
#include <mbedtls/ctr_drbg.h>
|
||||
|
||||
#undef inline
|
||||
|
||||
#define GIT_SSL_DEFAULT_CIPHERS "TLS1-3-AES-128-GCM-SHA256:TLS1-3-AES-256-GCM-SHA384:TLS1-3-CHACHA20-POLY1305-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-ECDSA-WITH-AES-256-GCM-SHA384:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-ECDHE-ECDSA-WITH-CHACHA20-POLY1305-SHA256:TLS-ECDHE-RSA-WITH-CHACHA20-POLY1305-SHA256:TLS-DHE-RSA-WITH-AES-128-GCM-SHA256:TLS-DHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-CHACHA20-POLY1305-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA256:TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA:TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA:TLS-ECDHE-ECDSA-WITH-AES-256-CBC-SHA384:TLS-ECDHE-RSA-WITH-AES-256-CBC-SHA384:TLS-ECDHE-ECDSA-WITH-AES-256-CBC-SHA:TLS-ECDHE-RSA-WITH-AES-256-CBC-SHA:TLS-DHE-RSA-WITH-AES-128-CBC-SHA256:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256:TLS-RSA-WITH-AES-128-GCM-SHA256:TLS-RSA-WITH-AES-256-GCM-SHA384:TLS-RSA-WITH-AES-128-CBC-SHA256:TLS-RSA-WITH-AES-256-CBC-SHA256:TLS-RSA-WITH-AES-128-CBC-SHA:TLS-RSA-WITH-AES-256-CBC-SHA"
|
||||
#define GIT_SSL_DEFAULT_CIPHERS_COUNT 28
|
||||
|
||||
static int ciphers_list[GIT_SSL_DEFAULT_CIPHERS_COUNT];
|
||||
|
||||
static bool initialized = false;
|
||||
static mbedtls_ssl_config mbedtls_config;
|
||||
static mbedtls_ctr_drbg_context mbedtls_rng;
|
||||
static mbedtls_entropy_context mbedtls_entropy;
|
||||
|
||||
static bool has_ca_chain = false;
|
||||
static mbedtls_x509_crt mbedtls_ca_chain;
|
||||
|
||||
/**
|
||||
* This function aims to clean-up the SSL context which
|
||||
* we allocated.
|
||||
*/
|
||||
static void shutdown_ssl(void)
|
||||
{
|
||||
if (has_ca_chain) {
|
||||
mbedtls_x509_crt_free(&mbedtls_ca_chain);
|
||||
has_ca_chain = false;
|
||||
}
|
||||
|
||||
if (initialized) {
|
||||
mbedtls_ctr_drbg_free(&mbedtls_rng);
|
||||
mbedtls_ssl_config_free(&mbedtls_config);
|
||||
mbedtls_entropy_free(&mbedtls_entropy);
|
||||
initialized = false;
|
||||
}
|
||||
}
|
||||
|
||||
int git_mbedtls_stream_global_init(void)
|
||||
{
|
||||
int loaded = 0;
|
||||
char *crtpath = GIT_DEFAULT_CERT_LOCATION;
|
||||
struct stat statbuf;
|
||||
|
||||
size_t ciphers_known = 0;
|
||||
char *cipher_name = NULL;
|
||||
char *cipher_string = NULL;
|
||||
char *cipher_string_tmp = NULL;
|
||||
|
||||
mbedtls_ssl_config_init(&mbedtls_config);
|
||||
mbedtls_entropy_init(&mbedtls_entropy);
|
||||
mbedtls_ctr_drbg_init(&mbedtls_rng);
|
||||
|
||||
if (mbedtls_ssl_config_defaults(&mbedtls_config,
|
||||
MBEDTLS_SSL_IS_CLIENT,
|
||||
MBEDTLS_SSL_TRANSPORT_STREAM,
|
||||
MBEDTLS_SSL_PRESET_DEFAULT) != 0) {
|
||||
git_error_set(GIT_ERROR_SSL, "failed to initialize mbedTLS");
|
||||
goto cleanup;
|
||||
}
|
||||
|
||||
/* configure TLSv1.1 or better */
|
||||
#ifdef MBEDTLS_SSL_MINOR_VERSION_2
|
||||
mbedtls_ssl_conf_min_version(&mbedtls_config, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_2);
|
||||
#endif
|
||||
|
||||
/* verify_server_cert is responsible for making the check.
|
||||
* OPTIONAL because REQUIRED drops the certificate as soon as the check
|
||||
* is made, so we can never see the certificate and override it. */
|
||||
mbedtls_ssl_conf_authmode(&mbedtls_config, MBEDTLS_SSL_VERIFY_OPTIONAL);
|
||||
|
||||
/* set the list of allowed ciphersuites */
|
||||
ciphers_known = 0;
|
||||
cipher_string = cipher_string_tmp = git__strdup(GIT_SSL_DEFAULT_CIPHERS);
|
||||
GIT_ERROR_CHECK_ALLOC(cipher_string);
|
||||
|
||||
while ((cipher_name = git__strtok(&cipher_string_tmp, ":")) != NULL) {
|
||||
int cipherid = mbedtls_ssl_get_ciphersuite_id(cipher_name);
|
||||
if (cipherid == 0) continue;
|
||||
|
||||
if (ciphers_known >= ARRAY_SIZE(ciphers_list)) {
|
||||
git_error_set(GIT_ERROR_SSL, "out of cipher list space");
|
||||
goto cleanup;
|
||||
}
|
||||
|
||||
ciphers_list[ciphers_known++] = cipherid;
|
||||
}
|
||||
git__free(cipher_string);
|
||||
|
||||
if (!ciphers_known) {
|
||||
git_error_set(GIT_ERROR_SSL, "no cipher could be enabled");
|
||||
goto cleanup;
|
||||
}
|
||||
mbedtls_ssl_conf_ciphersuites(&mbedtls_config, ciphers_list);
|
||||
|
||||
/* Seeding the random number generator */
|
||||
|
||||
if (mbedtls_ctr_drbg_seed(&mbedtls_rng, mbedtls_entropy_func,
|
||||
&mbedtls_entropy, NULL, 0) != 0) {
|
||||
git_error_set(GIT_ERROR_SSL, "failed to initialize mbedTLS entropy pool");
|
||||
goto cleanup;
|
||||
}
|
||||
|
||||
mbedtls_ssl_conf_rng(&mbedtls_config, mbedtls_ctr_drbg_random, &mbedtls_rng);
|
||||
|
||||
/* load default certificates */
|
||||
if (crtpath != NULL && stat(crtpath, &statbuf) == 0 && S_ISREG(statbuf.st_mode))
|
||||
loaded = (git_mbedtls__set_cert_location(crtpath, NULL) == 0);
|
||||
|
||||
if (!loaded && crtpath != NULL && stat(crtpath, &statbuf) == 0 && S_ISDIR(statbuf.st_mode))
|
||||
loaded = (git_mbedtls__set_cert_location(NULL, crtpath) == 0);
|
||||
|
||||
initialized = true;
|
||||
|
||||
return git_runtime_shutdown_register(shutdown_ssl);
|
||||
|
||||
cleanup:
|
||||
mbedtls_ctr_drbg_free(&mbedtls_rng);
|
||||
mbedtls_ssl_config_free(&mbedtls_config);
|
||||
mbedtls_entropy_free(&mbedtls_entropy);
|
||||
|
||||
return -1;
|
||||
}
|
||||
|
||||
static int bio_read(void *b, unsigned char *buf, size_t len)
|
||||
{
|
||||
git_stream *io = (git_stream *) b;
|
||||
return (int) git_stream_read(io, buf, min(len, INT_MAX));
|
||||
}
|
||||
|
||||
static int bio_write(void *b, const unsigned char *buf, size_t len)
|
||||
{
|
||||
git_stream *io = (git_stream *) b;
|
||||
return (int) git_stream_write(io, (const char *)buf, min(len, INT_MAX), 0);
|
||||
}
|
||||
|
||||
static int ssl_set_error(mbedtls_ssl_context *ssl, int error)
|
||||
{
|
||||
char errbuf[512];
|
||||
int ret = -1;
|
||||
|
||||
GIT_ASSERT(error != MBEDTLS_ERR_SSL_WANT_READ);
|
||||
GIT_ASSERT(error != MBEDTLS_ERR_SSL_WANT_WRITE);
|
||||
|
||||
if (error != 0)
|
||||
mbedtls_strerror( error, errbuf, 512 );
|
||||
|
||||
switch(error) {
|
||||
case 0:
|
||||
git_error_set(GIT_ERROR_SSL, "SSL error: unknown error");
|
||||
break;
|
||||
|
||||
case MBEDTLS_ERR_X509_CERT_VERIFY_FAILED:
|
||||
git_error_set(GIT_ERROR_SSL, "SSL error: %#04x [%x] - %s", error, mbedtls_ssl_get_verify_result(ssl), errbuf);
|
||||
ret = GIT_ECERTIFICATE;
|
||||
break;
|
||||
|
||||
default:
|
||||
git_error_set(GIT_ERROR_SSL, "SSL error: %#04x - %s", error, errbuf);
|
||||
}
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
static int ssl_teardown(mbedtls_ssl_context *ssl)
|
||||
{
|
||||
int ret = 0;
|
||||
|
||||
ret = mbedtls_ssl_close_notify(ssl);
|
||||
if (ret < 0)
|
||||
ret = ssl_set_error(ssl, ret);
|
||||
|
||||
mbedtls_ssl_free(ssl);
|
||||
return ret;
|
||||
}
|
||||
|
||||
static int verify_server_cert(mbedtls_ssl_context *ssl)
|
||||
{
|
||||
int ret = -1;
|
||||
|
||||
if ((ret = mbedtls_ssl_get_verify_result(ssl)) != 0) {
|
||||
char vrfy_buf[512];
|
||||
int len = mbedtls_x509_crt_verify_info(vrfy_buf, sizeof(vrfy_buf), "", ret);
|
||||
if (len >= 1) vrfy_buf[len - 1] = '\0'; /* Remove trailing \n */
|
||||
git_error_set(GIT_ERROR_SSL, "the SSL certificate is invalid: %#04x - %s", ret, vrfy_buf);
|
||||
return GIT_ECERTIFICATE;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
typedef struct {
|
||||
git_stream parent;
|
||||
git_stream *io;
|
||||
int owned;
|
||||
bool connected;
|
||||
char *host;
|
||||
mbedtls_ssl_context *ssl;
|
||||
git_cert_x509 cert_info;
|
||||
} mbedtls_stream;
|
||||
|
||||
|
||||
static int mbedtls_connect(git_stream *stream)
|
||||
{
|
||||
int ret;
|
||||
mbedtls_stream *st = (mbedtls_stream *) stream;
|
||||
|
||||
if (st->owned && (ret = git_stream_connect(st->io)) < 0)
|
||||
return ret;
|
||||
|
||||
st->connected = true;
|
||||
|
||||
mbedtls_ssl_set_hostname(st->ssl, st->host);
|
||||
|
||||
mbedtls_ssl_set_bio(st->ssl, st->io, bio_write, bio_read, NULL);
|
||||
|
||||
if ((ret = mbedtls_ssl_handshake(st->ssl)) != 0)
|
||||
return ssl_set_error(st->ssl, ret);
|
||||
|
||||
return verify_server_cert(st->ssl);
|
||||
}
|
||||
|
||||
static int mbedtls_certificate(git_cert **out, git_stream *stream)
|
||||
{
|
||||
unsigned char *encoded_cert;
|
||||
mbedtls_stream *st = (mbedtls_stream *) stream;
|
||||
|
||||
const mbedtls_x509_crt *cert = mbedtls_ssl_get_peer_cert(st->ssl);
|
||||
if (!cert) {
|
||||
git_error_set(GIT_ERROR_SSL, "the server did not provide a certificate");
|
||||
return -1;
|
||||
}
|
||||
|
||||
/* Retrieve the length of the certificate first */
|
||||
if (cert->raw.len == 0) {
|
||||
git_error_set(GIT_ERROR_NET, "failed to retrieve certificate information");
|
||||
return -1;
|
||||
}
|
||||
|
||||
encoded_cert = git__malloc(cert->raw.len);
|
||||
GIT_ERROR_CHECK_ALLOC(encoded_cert);
|
||||
memcpy(encoded_cert, cert->raw.p, cert->raw.len);
|
||||
|
||||
st->cert_info.parent.cert_type = GIT_CERT_X509;
|
||||
st->cert_info.data = encoded_cert;
|
||||
st->cert_info.len = cert->raw.len;
|
||||
|
||||
*out = &st->cert_info.parent;
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int mbedtls_set_proxy(git_stream *stream, const git_proxy_options *proxy_options)
|
||||
{
|
||||
mbedtls_stream *st = (mbedtls_stream *) stream;
|
||||
|
||||
return git_stream_set_proxy(st->io, proxy_options);
|
||||
}
|
||||
|
||||
static ssize_t mbedtls_stream_write(git_stream *stream, const char *data, size_t len, int flags)
|
||||
{
|
||||
mbedtls_stream *st = (mbedtls_stream *) stream;
|
||||
int written;
|
||||
|
||||
GIT_UNUSED(flags);
|
||||
|
||||
/*
|
||||
* `mbedtls_ssl_write` can only represent INT_MAX bytes
|
||||
* written via its return value. We thus need to clamp
|
||||
* the maximum number of bytes written.
|
||||
*/
|
||||
len = min(len, INT_MAX);
|
||||
|
||||
if ((written = mbedtls_ssl_write(st->ssl, (const unsigned char *)data, len)) <= 0)
|
||||
return ssl_set_error(st->ssl, written);
|
||||
|
||||
return written;
|
||||
}
|
||||
|
||||
static ssize_t mbedtls_stream_read(git_stream *stream, void *data, size_t len)
|
||||
{
|
||||
mbedtls_stream *st = (mbedtls_stream *) stream;
|
||||
int ret;
|
||||
|
||||
if ((ret = mbedtls_ssl_read(st->ssl, (unsigned char *)data, len)) <= 0)
|
||||
ssl_set_error(st->ssl, ret);
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
static int mbedtls_stream_close(git_stream *stream)
|
||||
{
|
||||
mbedtls_stream *st = (mbedtls_stream *) stream;
|
||||
int ret = 0;
|
||||
|
||||
if (st->connected && (ret = ssl_teardown(st->ssl)) != 0)
|
||||
return -1;
|
||||
|
||||
st->connected = false;
|
||||
|
||||
return st->owned ? git_stream_close(st->io) : 0;
|
||||
}
|
||||
|
||||
static void mbedtls_stream_free(git_stream *stream)
|
||||
{
|
||||
mbedtls_stream *st = (mbedtls_stream *) stream;
|
||||
|
||||
if (st->owned)
|
||||
git_stream_free(st->io);
|
||||
|
||||
git__free(st->host);
|
||||
git__free(st->cert_info.data);
|
||||
mbedtls_ssl_free(st->ssl);
|
||||
git__free(st->ssl);
|
||||
git__free(st);
|
||||
}
|
||||
|
||||
static int mbedtls_stream_wrap(
|
||||
git_stream **out,
|
||||
git_stream *in,
|
||||
const char *host,
|
||||
int owned)
|
||||
{
|
||||
mbedtls_stream *st;
|
||||
int error;
|
||||
|
||||
st = git__calloc(1, sizeof(mbedtls_stream));
|
||||
GIT_ERROR_CHECK_ALLOC(st);
|
||||
|
||||
st->io = in;
|
||||
st->owned = owned;
|
||||
|
||||
st->ssl = git__malloc(sizeof(mbedtls_ssl_context));
|
||||
GIT_ERROR_CHECK_ALLOC(st->ssl);
|
||||
mbedtls_ssl_init(st->ssl);
|
||||
if (mbedtls_ssl_setup(st->ssl, &mbedtls_config)) {
|
||||
git_error_set(GIT_ERROR_SSL, "failed to create ssl object");
|
||||
error = -1;
|
||||
goto out_err;
|
||||
}
|
||||
|
||||
st->host = git__strdup(host);
|
||||
GIT_ERROR_CHECK_ALLOC(st->host);
|
||||
|
||||
st->parent.version = GIT_STREAM_VERSION;
|
||||
st->parent.encrypted = 1;
|
||||
st->parent.proxy_support = git_stream_supports_proxy(st->io);
|
||||
st->parent.connect = mbedtls_connect;
|
||||
st->parent.certificate = mbedtls_certificate;
|
||||
st->parent.set_proxy = mbedtls_set_proxy;
|
||||
st->parent.read = mbedtls_stream_read;
|
||||
st->parent.write = mbedtls_stream_write;
|
||||
st->parent.close = mbedtls_stream_close;
|
||||
st->parent.free = mbedtls_stream_free;
|
||||
|
||||
*out = (git_stream *) st;
|
||||
return 0;
|
||||
|
||||
out_err:
|
||||
/* ESP FIX: do NOT close/free st->io here — on error the caller keeps
|
||||
* ownership of `in` (git_mbedtls_stream_new closes+frees it right after;
|
||||
* the vendored code freed it here too → double free → tlsf abort). */
|
||||
mbedtls_ssl_free(st->ssl);
|
||||
git__free(st->ssl);
|
||||
git__free(st);
|
||||
|
||||
return error;
|
||||
}
|
||||
|
||||
int git_mbedtls_stream_wrap(
|
||||
git_stream **out,
|
||||
git_stream *in,
|
||||
const char *host)
|
||||
{
|
||||
return mbedtls_stream_wrap(out, in, host, 0);
|
||||
}
|
||||
|
||||
int git_mbedtls_stream_new(
|
||||
git_stream **out,
|
||||
const char *host,
|
||||
const char *port)
|
||||
{
|
||||
git_stream *stream;
|
||||
int error;
|
||||
|
||||
GIT_ASSERT_ARG(out);
|
||||
GIT_ASSERT_ARG(host);
|
||||
GIT_ASSERT_ARG(port);
|
||||
|
||||
if ((error = git_socket_stream_new(&stream, host, port)) < 0)
|
||||
return error;
|
||||
|
||||
if ((error = mbedtls_stream_wrap(out, stream, host, 1)) < 0) {
|
||||
git_stream_close(stream);
|
||||
git_stream_free(stream);
|
||||
}
|
||||
|
||||
return error;
|
||||
}
|
||||
|
||||
int git_mbedtls__set_cert_location(const char *file, const char *path)
|
||||
{
|
||||
int ret = 0;
|
||||
char errbuf[512];
|
||||
|
||||
GIT_ASSERT_ARG(file || path);
|
||||
|
||||
if (has_ca_chain)
|
||||
mbedtls_x509_crt_free(&mbedtls_ca_chain);
|
||||
|
||||
mbedtls_x509_crt_init(&mbedtls_ca_chain);
|
||||
|
||||
if (file)
|
||||
ret = mbedtls_x509_crt_parse_file(&mbedtls_ca_chain, file);
|
||||
|
||||
if (ret >= 0 && path)
|
||||
ret = mbedtls_x509_crt_parse_path(&mbedtls_ca_chain, path);
|
||||
|
||||
/* mbedtls_x509_crt_parse_path returns the number of invalid certs on success */
|
||||
if (ret < 0) {
|
||||
mbedtls_x509_crt_free(&mbedtls_ca_chain);
|
||||
mbedtls_strerror( ret, errbuf, 512 );
|
||||
git_error_set(GIT_ERROR_SSL, "failed to load CA certificates: %#04x - %s", ret, errbuf);
|
||||
return -1;
|
||||
}
|
||||
|
||||
mbedtls_ssl_conf_ca_chain(&mbedtls_config, &mbedtls_ca_chain, NULL);
|
||||
has_ca_chain = true;
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
#else
|
||||
|
||||
#include "stream.h"
|
||||
|
||||
int git_mbedtls_stream_global_init(void)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
#endif
|
||||
@@ -74,3 +74,12 @@ CONFIG_I2C_SKIP_LEGACY_CONFLICT_CHECK=y
|
||||
# subset so a less common CA in the chain can't surprise us on the bench.
|
||||
CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y
|
||||
CONFIG_MBEDTLS_CERTIFICATE_BUNDLE_DEFAULT_FULL=y
|
||||
|
||||
# mbedTLS allocations go to PSRAM. The default (internal-only) needs ~33 KB of
|
||||
# contiguous internal RAM per TLS connection (two ~17 KB I/O buffers + contexts)
|
||||
# at the moment `:sync` pushes — with Wi-Fi, USB host, the editor and libgit2
|
||||
# all resident, mbedtls_ssl_setup failed exactly there on the first real-repo
|
||||
# push (2026-07-13; the failure then tripped the vendored stream double-free —
|
||||
# see components/libgit2/esp_mbedtls_stream.c). TLS buffers are CPU-only data,
|
||||
# so PSRAM is safe; the handshake is network-bound, not memory-bandwidth-bound.
|
||||
CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=y
|
||||
|
||||
@@ -248,9 +248,10 @@ fn publish_cycle(
|
||||
/// error, surfaced as such.
|
||||
fn publish_once(paths: &BTreeSet<String>) -> Result<PublishOutcome> {
|
||||
log::info!(
|
||||
"publish started — {} dirty path(s), free heap {}",
|
||||
"publish started — {} dirty path(s), free heap {} ({} internal)",
|
||||
paths.len(),
|
||||
free_heap()
|
||||
free_heap(),
|
||||
internal_free_heap()
|
||||
);
|
||||
let repo = Repository::open(REPO_DIR).with_context(|| {
|
||||
format!("opening git repo at {REPO_DIR} — provision the card with a clone (just init) whose origin is your remote")
|
||||
@@ -313,8 +314,9 @@ fn publish_once(paths: &BTreeSet<String>) -> Result<PublishOutcome> {
|
||||
}
|
||||
|
||||
log::info!(
|
||||
"push done — free heap {}, min-ever {}",
|
||||
"push done — free heap {} ({} internal), min-ever {}",
|
||||
free_heap(),
|
||||
internal_free_heap(),
|
||||
min_free_heap()
|
||||
);
|
||||
Ok(PublishOutcome::Pushed(short(oid)))
|
||||
@@ -388,11 +390,12 @@ fn stage_and_commit(repo: &Repository, paths: &BTreeSet<String>) -> Result<Optio
|
||||
.commit(Some("HEAD"), &sig, &sig, &message, &tree, &parents)
|
||||
.context("creating commit")?;
|
||||
log::info!(
|
||||
"commit split — splice {splice_ms}ms ({} path(s)), commit-obj {}ms; committed {} — free heap {}",
|
||||
"commit split — splice {splice_ms}ms ({} path(s)), commit-obj {}ms; committed {} — free heap {} ({} internal)",
|
||||
paths.len(),
|
||||
t_commit.elapsed().as_millis(),
|
||||
short(oid),
|
||||
free_heap()
|
||||
free_heap(),
|
||||
internal_free_heap()
|
||||
);
|
||||
Ok(Some(oid))
|
||||
}
|
||||
@@ -636,6 +639,14 @@ fn free_heap() -> u32 {
|
||||
unsafe { sys::esp_get_free_heap_size() }
|
||||
}
|
||||
|
||||
/// Free INTERNAL RAM (DRAM), excluding PSRAM. `free_heap` is dominated by the
|
||||
/// 8 MB PSRAM pool and masks internal exhaustion — which is what actually
|
||||
/// killed the first real-repo push (mbedTLS's ssl_setup could not get its
|
||||
/// ~33 KB while Wi-Fi + USB + editor + libgit2 were resident).
|
||||
fn internal_free_heap() -> u32 {
|
||||
unsafe { sys::heap_caps_get_free_size(sys::MALLOC_CAP_INTERNAL) as u32 }
|
||||
}
|
||||
|
||||
fn min_free_heap() -> u32 {
|
||||
unsafe { sys::esp_get_minimum_free_heap_size() }
|
||||
}
|
||||
|
||||
@@ -123,7 +123,18 @@ fn main() -> anyhow::Result<()> {
|
||||
ed.set_notice(format!("loaded {name}"));
|
||||
// Feed the file palette (Ctrl-P). Enumerated once at boot — the v0.5 slices
|
||||
// that create/delete files (`:enew`, delete) will re-feed it then.
|
||||
// Bracketed with internal-DRAM readings: each path is a small String, kept
|
||||
// internal by the SPIRAM malloc threshold (16 KB), so the list competes
|
||||
// with Wi-Fi/TLS for DRAM. Estimate to confirm: ~60-70 KB at 1098 files —
|
||||
// this number decides whether interning the paths into one shared buffer
|
||||
// (a single >16 KB alloc, which lands in PSRAM) is worth the refactor.
|
||||
let dram_before = internal_free_heap();
|
||||
ed.set_file_list(enumerate_files());
|
||||
let dram_after = internal_free_heap();
|
||||
log::info!(
|
||||
"file list: internal heap {dram_before} -> {dram_after} ({} KB consumed)",
|
||||
dram_before.saturating_sub(dram_after) / 1024
|
||||
);
|
||||
// Editor preferences (.typoena.toml, git-tracked). Read before the first
|
||||
// render so `line_numbers` shapes the opening frame. A missing / unreadable /
|
||||
// partial file falls back to defaults, so a fresh card just works.
|
||||
@@ -585,29 +596,39 @@ fn walk_files(dir: &std::path::Path, depth: usize, out: &mut Vec<String>) {
|
||||
let Ok(entries) = std::fs::read_dir(dir) else {
|
||||
return;
|
||||
};
|
||||
let paths: Vec<_> = entries.flatten().map(|e| e.path()).collect();
|
||||
for path in paths {
|
||||
// Keep the dirent's own file type: esp-idf's FAT VFS always fills d_type
|
||||
// (DT_DIR/DT_REG, straight from the FILINFO readdir already holds), so
|
||||
// `file_type()` is free. A per-entry `metadata()` stat instead re-walks
|
||||
// the directory by path every time — measured at ~32ms/file on the SD
|
||||
// card, it turned a 1098-file walk into 35s.
|
||||
let children: Vec<_> = entries
|
||||
.flatten()
|
||||
.filter_map(|e| e.file_type().ok().map(|t| (e.path(), t)))
|
||||
.collect();
|
||||
for (path, ftype) in children {
|
||||
let Some(name) = path.file_name().and_then(|n| n.to_str()) else {
|
||||
continue;
|
||||
};
|
||||
if name.starts_with('.') {
|
||||
continue;
|
||||
}
|
||||
// Stat rather than trust d_type — FatFS's dirent type can read back
|
||||
// as unknown; a plain metadata call is reliable here.
|
||||
let Ok(meta) = std::fs::metadata(&path) else {
|
||||
continue;
|
||||
};
|
||||
if meta.is_file() {
|
||||
if ftype.is_file() {
|
||||
if let Some(p) = path.to_str() {
|
||||
out.push(p.to_string());
|
||||
}
|
||||
} else if meta.is_dir() {
|
||||
} else if ftype.is_dir() {
|
||||
walk_files(&path, depth + 1, out);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Free internal DRAM (excludes the 8 MB PSRAM pool, which dominates the total
|
||||
/// free-heap number and masks DRAM exhaustion). Same reading `git_sync` logs.
|
||||
fn internal_free_heap() -> u32 {
|
||||
use esp_idf_svc::sys;
|
||||
unsafe { sys::heap_caps_get_free_size(sys::MALLOC_CAP_INTERNAL) as u32 }
|
||||
}
|
||||
|
||||
/// A file's display name — its basename without extension (`/sd/repo/notes.md`
|
||||
/// → `notes`), for the snackbar. Falls back to the raw path if it has no stem.
|
||||
fn file_stem(path: &str) -> &str {
|
||||
|
||||
Reference in New Issue
Block a user