Compare commits

..

8 Commits

Author SHA1 Message Date
Julien Calixte
e801d2d480 docs(palette): note the two on-device boot-log measurements pending
Walk time after the d_type fix, and the file-list internal-DRAM cost
with the decision rule for interning paths into one PSRAM buffer.
2026-07-13 10:26:25 +02:00
Julien Calixte
a771c3f6de chore(diag): log internal DRAM next to PSRAM-dominated heap totals
"free heap 5.8MB" in the crash log was PSRAM and masked the actual
internal-RAM exhaustion. git_sync's publish/commit/push lines now
carry the MALLOC_CAP_INTERNAL reading, and boot brackets the palette
file-list build with it — small Strings are forced internal by the
16 KB SPIRAM malloc threshold, so the 1098-path list is a suspected
~60-70 KB DRAM consumer competing with Wi-Fi/TLS; the log decides
whether interning into one PSRAM buffer is worth it.
2026-07-13 10:25:25 +02:00
Julien Calixte
4c92b0dddc fix(sync): move mbedTLS allocations to PSRAM
esp-idf's default is internal-RAM-only, and a TLS connection needs
~33 KB of it (two ~17 KB I/O buffers + contexts) at the exact moment
:sync pushes — with Wi-Fi, USB host, the editor and libgit2 resident,
mbedtls_ssl_setup failed there on the first real-repo push (and its
failure path then hit the stream double-free fixed in the previous
commit). TLS buffers are CPU-only data, so PSRAM is safe; the
handshake is network-bound.
2026-07-13 10:25:25 +02:00
Julien Calixte
af5b41a104 fix(sync): patch the mbedtls stream double-free that reset the chip
Upstream libgit2 v1.9.4 bug: mbedtls_stream_wrap's ssl_setup error
path closes and frees the caller's socket stream, then
git_mbedtls_stream_new frees it again — and wrap's other error paths
don't free it at all, so callers can't compensate either way. When
ssl_setup failed on the device (internal-RAM exhaustion during the
first real-repo push), the double git__free tripped tlsf ("block
already marked as free") and reset the chip mid-:sync.

Fixed the way esp_map.c replaces map.c: the vendored streams/mbedtls.c
is excluded from the component and esp_mbedtls_stream.c (verbatim copy
+ one-hunk fix: out_err leaves st->io to the caller, and frees the
leaked ssl struct) takes its place. Keep the copy in lockstep on
submodule bumps; worth reporting upstream.
2026-07-13 10:25:12 +02:00
Julien Calixte
2660a3e9dd perf(palette): trust dirent d_type instead of a per-entry stat
The per-entry metadata() call makes FatFS re-walk the directory by
path every time — ~32ms/file on the card, 35s for a 1098-file tree.
esp-idf's FAT VFS always fills d_type (DT_DIR/DT_REG from the FILINFO
readdir already holds, never unknown) and Rust std maps file_type()
onto it stat-free, so the walk is now one readdir pass per directory.
2026-07-13 01:05:10 +02:00
Julien Calixte
79fad4689c docs(palette): amend v0.5 with the recursive walk and search threshold 2026-07-13 00:55:34 +02:00
Julien Calixte
98fc817b3f feat(palette): show recents only until the query reaches two chars
With the file list now a recursive walk of the whole card, an unranked
short query pages through too much. Below PALETTE_MIN_QUERY (2) the
result list is the MRU only, keeping Cmd-P + Enter quick-switch one
keystroke away; two chars reveal the full fuzzy-ranked list. Commands
(>) and snippets ($) are short curated lists and keep matching from
the first char.
2026-07-13 00:55:25 +02:00
Julien Calixte
d306caacf7 feat(palette): walk the card recursively for the file list
A nested repo (jcalixte/notes) showed only its top-level files in the
palette. Dot entries are skipped at every level so .git is never
descended; each directory is read fully before recursing so a single
FatFS dir handle is open at a time; depth capped at 8. The boot walk
logs its file count and duration to keep the FAT dir-IO cost visible.
2026-07-13 00:55:16 +02:00
7 changed files with 709 additions and 36 deletions

View File

@@ -149,6 +149,43 @@ shown). On-device check: reflash → open a note (trailing empty line visible)
newline for the same reason; the guarded save leaves the prefs file with exactly
one.
**Amendment 2026-07-13 — recursive enumeration + a 2-char search threshold.**
Loading a real repo (`jcalixte/notes`) exposed that `enumerate_files` listed
only the **top-level** files of `/sd/repo` and `/sd/local` — a nested notes tree
showed a single file in the palette (subpaths always *opened* fine via
`:e repo/sub/x.md`; only the listing was flat). The enumeration is now a
recursive walk: dot entries are skipped at every level (so `.git` is never
descended into), each directory is read fully before recursing (one FatFS dir
handle open at a time — the `remove_dir_recursive` pattern, kind to the
FD-bounded mount), depth is capped at 8, and the boot-time walk logs its file
count and duration (`file walk: N files in Xms`) so the FAT dir-IO cost on a
big repo is measurable, not assumed. With the list now card-sized, the palette
gained a **search threshold**: below 2 typed chars the result list is the
**recents (MRU) only** — quick-switch (`Cmd-P`, `Enter`) stays one keystroke
away — and the full fuzzy-ranked list appears from 2 chars on
(`PALETTE_MIN_QUERY`). A fresh boot with no opens yet shows `(type to search)`.
`>` commands and `$` snippets are short curated lists; the threshold does not
apply to them.
**TODO (on-device, next time the device is on the bench)** — two measurements
from the same boot log, both already instrumented:
- [ ] **Re-measure the walk time** after the d_type fix (`2660a3e` — dirent
`file_type()` instead of a per-entry `metadata()` stat, which cost
~32 ms/file and made run 1 take 35 s for 1098 files). Read the
`file walk: N files in Xms` line. Only if it's still slow does the
async-walk idea come back on the table.
- [ ] **Read the file-list DRAM cost** from the new
`file list: internal heap <before> -> <after> (<N> KB consumed)` line
(the build is bracketed with `MALLOC_CAP_INTERNAL` readings in
`main.rs`). The 1098 path Strings are each below the 16 KB SPIRAM
malloc threshold, so they all land in internal DRAM — estimated
6070 KB, competing with Wi-Fi/TLS. Decision rule: **~6070 KB
confirms** interning the paths into one shared buffer (a single
>16 KB alloc goes to PSRAM; only a ~9 KB offset index stays in DRAM);
**well under that (≤~30 KB)** kills the idea — the next DRAM suspect
is then parked-buffer text.
- [x] `Cmd-P` opens fuzzy file palette over **both** `/sd/repo/` and
`/sd/local/`**landed and CONFIRMED ON DEVICE 2026-07-12** (Spike 11: no
ghosting on the transient panel); scope shows as the inline

View File

@@ -743,8 +743,9 @@ pub struct Editor {
/// key batch. See [`Effect`].
requests: Vec<Effect>,
/// Every openable file, as absolute paths, fed by the host at boot via
/// [`set_file_list`](Self::set_file_list) (an enumeration of `/sd/repo` and
/// `/sd/local`). The palette fuzzy-filters this; empty until the host feeds it.
/// [`set_file_list`](Self::set_file_list) (a recursive walk of `/sd/repo`
/// and `/sd/local`). The palette fuzzy-filters this once the query reaches
/// [`PALETTE_MIN_QUERY`] chars; empty until the host feeds it.
files: Vec<String>,
/// Recently-opened files, most-recent-first (an MRU), deduped and bounded to
/// [`MRU_MAX`]. Every `:e`/palette open pushes to the front
@@ -798,12 +799,20 @@ struct Buffer {
/// evicted; it is saved first if dirty, so an evicted buffer is never lost.
const MAX_RESIDENT: usize = 3;
/// Recent-files (MRU) list length — how many opens the palette remembers to
/// float to the top on an empty query. Far more than [`MAX_RESIDENT`] (recency
/// Recent-files (MRU) list length — how many opens the palette remembers; they
/// are the whole result list below [`PALETTE_MIN_QUERY`] chars and float to the
/// top above it. Far more than [`MAX_RESIDENT`] (recency
/// outlives residency: a file evicted from memory is still recently *used*), but
/// bounded so the list can't grow without limit over a long session.
const MRU_MAX: usize = 16;
/// Query length (chars) at which the file palette searches the full file list.
/// Shorter queries show only the recents ([`MRU_MAX`]) — the list is a
/// recursive walk of the card, and one char can't rank hundreds of paths
/// usefully. `>` commands and `$` snippets are short curated lists, so the
/// threshold does not apply to them.
const PALETTE_MIN_QUERY: usize = 2;
/// Maximum undo depth (change-groups). A full-buffer snapshot per group means
/// worst-case memory is `UNDO_DEPTH × buffer size`; for note-sized files on the
/// 8 MB PSRAM this is negligible, and prose editing rarely nears 100 groups
@@ -1924,6 +1933,11 @@ impl Editor {
/// Base order is MRU-first (recents in use order, then the rest as sorted). A
/// non-empty query keeps only fuzzy matches and stable-sorts them by score, so
/// equal scores keep their MRU/base position. See [`fuzzy_score`].
///
/// Below [`PALETTE_MIN_QUERY`] chars the candidate set is the recents only:
/// the file list is a recursive walk of the whole card, too long to page
/// through unranked, but the MRU keeps quick-switch (`Cmd-P`, `Enter`) one
/// keystroke away. Two typed chars reveal the full list.
fn palette_matches(&self) -> Vec<usize> {
let mut order: Vec<usize> = Vec::with_capacity(self.files.len());
for r in &self.recent {
@@ -1931,9 +1945,11 @@ impl Editor {
order.push(i);
}
}
for i in 0..self.files.len() {
if !order.contains(&i) {
order.push(i);
if self.palette_query.chars().count() >= PALETTE_MIN_QUERY {
for i in 0..self.files.len() {
if !order.contains(&i) {
order.push(i);
}
}
}
if self.palette_query.is_empty() {
@@ -3562,6 +3578,10 @@ impl Editor {
"(no command)"
} else if self.files.is_empty() {
"(no files on card)"
} else if self.palette_query.chars().count() < PALETTE_MIN_QUERY {
// No recents yet and the query is below the search threshold —
// the full list needs 2+ chars.
"(type to search)"
} else {
"(no match)"
};
@@ -5124,6 +5144,9 @@ mod tests {
e.take_effects();
assert!(!e.files.contains(&"/sd/repo/notes.md".to_string()));
e.handle(Key::Palette);
for c in "md".chars() {
e.handle(Key::Char(c)); // reach the search threshold
}
assert_eq!(palette_labels(&e), vec!["repo/todo.md"]); // only the survivor
}
@@ -5272,6 +5295,9 @@ mod tests {
fn half_page_keys_move_the_selection_clamped() {
let mut e = palette_editor(&["/sd/repo/a.md", "/sd/repo/b.md", "/sd/repo/c.md"]);
e.handle(Key::Palette);
for ch in "md".chars() {
e.handle(Key::Char(ch)); // reach the search threshold: all three match
}
assert_eq!(e.palette_sel, 0);
e.handle(Key::HalfPageDown);
assert_eq!(e.palette_sel, 1);
@@ -5286,6 +5312,9 @@ mod tests {
fn ctrl_n_p_navigate_the_palette() {
let mut e = palette_editor(&["/sd/repo/a.md", "/sd/repo/b.md", "/sd/repo/c.md"]);
e.handle(Key::Palette);
for ch in "md".chars() {
e.handle(Key::Char(ch)); // reach the search threshold: all three match
}
e.handle(Key::Down); // Ctrl-n
assert_eq!(e.palette_sel, 1);
e.handle(Key::Down);
@@ -5337,6 +5366,9 @@ mod tests {
fn editing_the_query_resets_the_selection_to_the_top() {
let mut e = palette_editor(&["/sd/repo/a.md", "/sd/repo/b.md"]);
e.handle(Key::Palette);
for ch in "md".chars() {
e.handle(Key::Char(ch)); // reach the search threshold: both match
}
e.handle(Key::HalfPageDown);
assert_eq!(e.palette_sel, 1);
e.handle(Key::Char('a')); // a query edit resets the selection
@@ -5352,17 +5384,46 @@ mod tests {
}
#[test]
fn empty_query_orders_recents_first_then_sorted() {
fn short_query_lists_recents_only() {
let mut e = palette_editor(&["/sd/repo/b.md", "/sd/repo/a.md", "/sd/repo/c.md"]);
// No opens yet: pure sorted order.
assert_eq!(palette_labels(&e), vec!["repo/a.md", "repo/b.md", "repo/c.md"]);
// Open c.md through the palette; it should float to the front next time.
// No opens yet: below the search threshold there is nothing to show.
assert!(palette_labels(&e).is_empty());
// Open c.md through the palette; it becomes the recents-only result.
e.handle(Key::Palette);
for ch in "c.md".chars() {
e.handle(Key::Char(ch));
}
e.handle(Key::Enter);
e.take_effects(); // drop the queued Load; we only care about the MRU
assert_eq!(palette_labels(&e), vec!["repo/c.md"]);
}
#[test]
fn two_char_query_reveals_the_full_file_list() {
let mut e = palette_editor(&["/sd/repo/b.md", "/sd/repo/a.md", "/sd/repo/c.md"]);
e.handle(Key::Palette);
e.handle(Key::Char('m')); // one char: still recents-only (none yet)
assert!(e.palette_matches().is_empty());
e.handle(Key::Char('d')); // "md": the full list, fuzzy-ranked
assert_eq!(palette_labels(&e), vec!["repo/a.md", "repo/b.md", "repo/c.md"]);
}
#[test]
fn recents_float_above_the_full_list_on_a_matching_query() {
let mut e = palette_editor(&["/sd/repo/b.md", "/sd/repo/a.md", "/sd/repo/c.md"]);
// Open c.md so it is the MRU head.
e.handle(Key::Palette);
for ch in "c.md".chars() {
e.handle(Key::Char(ch));
}
e.handle(Key::Enter);
e.take_effects();
// "md" scores the three labels equally; the stable sort keeps the
// recently-opened c.md in front of the sorted rest.
e.handle(Key::Palette);
for ch in "md".chars() {
e.handle(Key::Char(ch));
}
assert_eq!(palette_labels(&e), vec!["repo/c.md", "repo/a.md", "repo/b.md"]);
}
@@ -5370,6 +5431,10 @@ mod tests {
fn draw_in_palette_mode_does_not_panic() {
let mut e = palette_editor(&["/sd/repo/a.md", "/sd/local/j.md"]);
e.handle(Key::Palette);
let _ = e.draw(true); // empty query, no recents: "(type to search)"
e.handle(Key::Char('j')); // one char, still below the threshold
let _ = e.draw(true);
e.handle(Key::Char('m')); // at the threshold: the ranked list
let _ = e.draw(true);
// Empty file list: the "(no files on card)" path must also be safe.
let mut empty = Editor::new();

View File

@@ -41,6 +41,12 @@ file(GLOB LG2_SRCS
"${LG2}/deps/pcre/*.c"
"${LG2}/deps/zlib/*.c"
)
# streams/mbedtls.c is replaced by our patched copy: its wrap() error path
# double-freed the socket stream (tlsf abort on device when ssl_setup failed
# under memory pressure). See esp_mbedtls_stream.c's header for the one-hunk
# delta; keep the copy in lockstep on submodule bumps.
list(REMOVE_ITEM LG2_SRCS "${LG2}/src/libgit2/streams/mbedtls.c")
list(APPEND LG2_SRCS
"${LG2}/src/util/allocators/failalloc.c"
"${LG2}/src/util/allocators/stdalloc.c"
@@ -48,6 +54,7 @@ list(APPEND LG2_SRCS
"${LG2}/src/util/hash/mbedtls.c" # SHA1 + SHA256 via mbedtls
"${CMAKE_CURRENT_LIST_DIR}/esp_map.c" # p_mmap via malloc+read (no <sys/mman.h>)
"${CMAKE_CURRENT_LIST_DIR}/esp_stubs.c" # getuid/readlink/utimes/... stubs
"${CMAKE_CURRENT_LIST_DIR}/esp_mbedtls_stream.c" # streams/mbedtls.c + double-free fix
# NOTE: unix/map.c replaced by esp_map.c — picolibc has no <sys/mman.h>.
# NOTE: unix/process.c deliberately excluded — needs fork()/sys/wait.h,
# only used by the SSH-exec transport we don't enable.

View File

@@ -0,0 +1,498 @@
/*
* Copyright (C) the libgit2 contributors. All rights reserved.
*
* This file is part of libgit2, distributed under the GNU GPL v2 with
* a Linking Exception. For full terms see the included COPYING file.
*/
/*
* esp_mbedtls_stream.c — verbatim copy of the vendored
* src/libgit2/streams/mbedtls.c (v1.9.4) with ONE fix; it replaces the
* vendored file in CMakeLists.txt (same pattern as esp_map.c).
*
* THE FIX (2026-07-13): mbedtls_stream_wrap()'s `out_err` path closed and
* freed `st->io` — the caller's socket stream — but every caller frees that
* stream on error too (git_mbedtls_stream_new does close+free right after),
* and wrap's OTHER error paths (calloc/strdup failures) do NOT free it, so
* the caller cannot compensate either way. When mbedtls_ssl_setup failed on
* the device (internal-RAM exhaustion during the first real-repo push), the
* double git__free tripped tlsf ("block already marked as free") and reset
* the chip instead of surfacing a clean error. Delta from vendor: the
* `out_err` label no longer touches st->io — on error, ownership of `in`
* stays with the caller, consistently — and it frees the git__malloc'd
* st->ssl struct the vendored path leaked.
*
* Keep this file in lockstep with the vendored one on submodule bumps (diff
* against it; the delta must stay this one hunk).
*/
#include "streams/mbedtls.h"
#ifdef GIT_MBEDTLS
#include <ctype.h>
#include "runtime.h"
#include "stream.h"
#include "streams/socket.h"
#include "git2/transport.h"
#include "util.h"
#ifndef GIT_DEFAULT_CERT_LOCATION
#define GIT_DEFAULT_CERT_LOCATION NULL
#endif
/* Work around C90-conformance issues */
#if !defined(__STDC_VERSION__) || (__STDC_VERSION__ < 199901L)
# if defined(_MSC_VER)
# define inline __inline
# elif defined(__GNUC__)
# define inline __inline__
# else
# define inline
# endif
#endif
#include <mbedtls/ssl.h>
#include <mbedtls/error.h>
#include <mbedtls/entropy.h>
#include <mbedtls/ctr_drbg.h>
#undef inline
#define GIT_SSL_DEFAULT_CIPHERS "TLS1-3-AES-128-GCM-SHA256:TLS1-3-AES-256-GCM-SHA384:TLS1-3-CHACHA20-POLY1305-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-ECDSA-WITH-AES-256-GCM-SHA384:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-ECDHE-ECDSA-WITH-CHACHA20-POLY1305-SHA256:TLS-ECDHE-RSA-WITH-CHACHA20-POLY1305-SHA256:TLS-DHE-RSA-WITH-AES-128-GCM-SHA256:TLS-DHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-CHACHA20-POLY1305-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA256:TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA:TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA:TLS-ECDHE-ECDSA-WITH-AES-256-CBC-SHA384:TLS-ECDHE-RSA-WITH-AES-256-CBC-SHA384:TLS-ECDHE-ECDSA-WITH-AES-256-CBC-SHA:TLS-ECDHE-RSA-WITH-AES-256-CBC-SHA:TLS-DHE-RSA-WITH-AES-128-CBC-SHA256:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256:TLS-RSA-WITH-AES-128-GCM-SHA256:TLS-RSA-WITH-AES-256-GCM-SHA384:TLS-RSA-WITH-AES-128-CBC-SHA256:TLS-RSA-WITH-AES-256-CBC-SHA256:TLS-RSA-WITH-AES-128-CBC-SHA:TLS-RSA-WITH-AES-256-CBC-SHA"
#define GIT_SSL_DEFAULT_CIPHERS_COUNT 28
static int ciphers_list[GIT_SSL_DEFAULT_CIPHERS_COUNT];
static bool initialized = false;
static mbedtls_ssl_config mbedtls_config;
static mbedtls_ctr_drbg_context mbedtls_rng;
static mbedtls_entropy_context mbedtls_entropy;
static bool has_ca_chain = false;
static mbedtls_x509_crt mbedtls_ca_chain;
/**
* This function aims to clean-up the SSL context which
* we allocated.
*/
static void shutdown_ssl(void)
{
if (has_ca_chain) {
mbedtls_x509_crt_free(&mbedtls_ca_chain);
has_ca_chain = false;
}
if (initialized) {
mbedtls_ctr_drbg_free(&mbedtls_rng);
mbedtls_ssl_config_free(&mbedtls_config);
mbedtls_entropy_free(&mbedtls_entropy);
initialized = false;
}
}
int git_mbedtls_stream_global_init(void)
{
int loaded = 0;
char *crtpath = GIT_DEFAULT_CERT_LOCATION;
struct stat statbuf;
size_t ciphers_known = 0;
char *cipher_name = NULL;
char *cipher_string = NULL;
char *cipher_string_tmp = NULL;
mbedtls_ssl_config_init(&mbedtls_config);
mbedtls_entropy_init(&mbedtls_entropy);
mbedtls_ctr_drbg_init(&mbedtls_rng);
if (mbedtls_ssl_config_defaults(&mbedtls_config,
MBEDTLS_SSL_IS_CLIENT,
MBEDTLS_SSL_TRANSPORT_STREAM,
MBEDTLS_SSL_PRESET_DEFAULT) != 0) {
git_error_set(GIT_ERROR_SSL, "failed to initialize mbedTLS");
goto cleanup;
}
/* configure TLSv1.1 or better */
#ifdef MBEDTLS_SSL_MINOR_VERSION_2
mbedtls_ssl_conf_min_version(&mbedtls_config, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_2);
#endif
/* verify_server_cert is responsible for making the check.
* OPTIONAL because REQUIRED drops the certificate as soon as the check
* is made, so we can never see the certificate and override it. */
mbedtls_ssl_conf_authmode(&mbedtls_config, MBEDTLS_SSL_VERIFY_OPTIONAL);
/* set the list of allowed ciphersuites */
ciphers_known = 0;
cipher_string = cipher_string_tmp = git__strdup(GIT_SSL_DEFAULT_CIPHERS);
GIT_ERROR_CHECK_ALLOC(cipher_string);
while ((cipher_name = git__strtok(&cipher_string_tmp, ":")) != NULL) {
int cipherid = mbedtls_ssl_get_ciphersuite_id(cipher_name);
if (cipherid == 0) continue;
if (ciphers_known >= ARRAY_SIZE(ciphers_list)) {
git_error_set(GIT_ERROR_SSL, "out of cipher list space");
goto cleanup;
}
ciphers_list[ciphers_known++] = cipherid;
}
git__free(cipher_string);
if (!ciphers_known) {
git_error_set(GIT_ERROR_SSL, "no cipher could be enabled");
goto cleanup;
}
mbedtls_ssl_conf_ciphersuites(&mbedtls_config, ciphers_list);
/* Seeding the random number generator */
if (mbedtls_ctr_drbg_seed(&mbedtls_rng, mbedtls_entropy_func,
&mbedtls_entropy, NULL, 0) != 0) {
git_error_set(GIT_ERROR_SSL, "failed to initialize mbedTLS entropy pool");
goto cleanup;
}
mbedtls_ssl_conf_rng(&mbedtls_config, mbedtls_ctr_drbg_random, &mbedtls_rng);
/* load default certificates */
if (crtpath != NULL && stat(crtpath, &statbuf) == 0 && S_ISREG(statbuf.st_mode))
loaded = (git_mbedtls__set_cert_location(crtpath, NULL) == 0);
if (!loaded && crtpath != NULL && stat(crtpath, &statbuf) == 0 && S_ISDIR(statbuf.st_mode))
loaded = (git_mbedtls__set_cert_location(NULL, crtpath) == 0);
initialized = true;
return git_runtime_shutdown_register(shutdown_ssl);
cleanup:
mbedtls_ctr_drbg_free(&mbedtls_rng);
mbedtls_ssl_config_free(&mbedtls_config);
mbedtls_entropy_free(&mbedtls_entropy);
return -1;
}
static int bio_read(void *b, unsigned char *buf, size_t len)
{
git_stream *io = (git_stream *) b;
return (int) git_stream_read(io, buf, min(len, INT_MAX));
}
static int bio_write(void *b, const unsigned char *buf, size_t len)
{
git_stream *io = (git_stream *) b;
return (int) git_stream_write(io, (const char *)buf, min(len, INT_MAX), 0);
}
static int ssl_set_error(mbedtls_ssl_context *ssl, int error)
{
char errbuf[512];
int ret = -1;
GIT_ASSERT(error != MBEDTLS_ERR_SSL_WANT_READ);
GIT_ASSERT(error != MBEDTLS_ERR_SSL_WANT_WRITE);
if (error != 0)
mbedtls_strerror( error, errbuf, 512 );
switch(error) {
case 0:
git_error_set(GIT_ERROR_SSL, "SSL error: unknown error");
break;
case MBEDTLS_ERR_X509_CERT_VERIFY_FAILED:
git_error_set(GIT_ERROR_SSL, "SSL error: %#04x [%x] - %s", error, mbedtls_ssl_get_verify_result(ssl), errbuf);
ret = GIT_ECERTIFICATE;
break;
default:
git_error_set(GIT_ERROR_SSL, "SSL error: %#04x - %s", error, errbuf);
}
return ret;
}
static int ssl_teardown(mbedtls_ssl_context *ssl)
{
int ret = 0;
ret = mbedtls_ssl_close_notify(ssl);
if (ret < 0)
ret = ssl_set_error(ssl, ret);
mbedtls_ssl_free(ssl);
return ret;
}
static int verify_server_cert(mbedtls_ssl_context *ssl)
{
int ret = -1;
if ((ret = mbedtls_ssl_get_verify_result(ssl)) != 0) {
char vrfy_buf[512];
int len = mbedtls_x509_crt_verify_info(vrfy_buf, sizeof(vrfy_buf), "", ret);
if (len >= 1) vrfy_buf[len - 1] = '\0'; /* Remove trailing \n */
git_error_set(GIT_ERROR_SSL, "the SSL certificate is invalid: %#04x - %s", ret, vrfy_buf);
return GIT_ECERTIFICATE;
}
return 0;
}
typedef struct {
git_stream parent;
git_stream *io;
int owned;
bool connected;
char *host;
mbedtls_ssl_context *ssl;
git_cert_x509 cert_info;
} mbedtls_stream;
static int mbedtls_connect(git_stream *stream)
{
int ret;
mbedtls_stream *st = (mbedtls_stream *) stream;
if (st->owned && (ret = git_stream_connect(st->io)) < 0)
return ret;
st->connected = true;
mbedtls_ssl_set_hostname(st->ssl, st->host);
mbedtls_ssl_set_bio(st->ssl, st->io, bio_write, bio_read, NULL);
if ((ret = mbedtls_ssl_handshake(st->ssl)) != 0)
return ssl_set_error(st->ssl, ret);
return verify_server_cert(st->ssl);
}
static int mbedtls_certificate(git_cert **out, git_stream *stream)
{
unsigned char *encoded_cert;
mbedtls_stream *st = (mbedtls_stream *) stream;
const mbedtls_x509_crt *cert = mbedtls_ssl_get_peer_cert(st->ssl);
if (!cert) {
git_error_set(GIT_ERROR_SSL, "the server did not provide a certificate");
return -1;
}
/* Retrieve the length of the certificate first */
if (cert->raw.len == 0) {
git_error_set(GIT_ERROR_NET, "failed to retrieve certificate information");
return -1;
}
encoded_cert = git__malloc(cert->raw.len);
GIT_ERROR_CHECK_ALLOC(encoded_cert);
memcpy(encoded_cert, cert->raw.p, cert->raw.len);
st->cert_info.parent.cert_type = GIT_CERT_X509;
st->cert_info.data = encoded_cert;
st->cert_info.len = cert->raw.len;
*out = &st->cert_info.parent;
return 0;
}
static int mbedtls_set_proxy(git_stream *stream, const git_proxy_options *proxy_options)
{
mbedtls_stream *st = (mbedtls_stream *) stream;
return git_stream_set_proxy(st->io, proxy_options);
}
static ssize_t mbedtls_stream_write(git_stream *stream, const char *data, size_t len, int flags)
{
mbedtls_stream *st = (mbedtls_stream *) stream;
int written;
GIT_UNUSED(flags);
/*
* `mbedtls_ssl_write` can only represent INT_MAX bytes
* written via its return value. We thus need to clamp
* the maximum number of bytes written.
*/
len = min(len, INT_MAX);
if ((written = mbedtls_ssl_write(st->ssl, (const unsigned char *)data, len)) <= 0)
return ssl_set_error(st->ssl, written);
return written;
}
static ssize_t mbedtls_stream_read(git_stream *stream, void *data, size_t len)
{
mbedtls_stream *st = (mbedtls_stream *) stream;
int ret;
if ((ret = mbedtls_ssl_read(st->ssl, (unsigned char *)data, len)) <= 0)
ssl_set_error(st->ssl, ret);
return ret;
}
static int mbedtls_stream_close(git_stream *stream)
{
mbedtls_stream *st = (mbedtls_stream *) stream;
int ret = 0;
if (st->connected && (ret = ssl_teardown(st->ssl)) != 0)
return -1;
st->connected = false;
return st->owned ? git_stream_close(st->io) : 0;
}
static void mbedtls_stream_free(git_stream *stream)
{
mbedtls_stream *st = (mbedtls_stream *) stream;
if (st->owned)
git_stream_free(st->io);
git__free(st->host);
git__free(st->cert_info.data);
mbedtls_ssl_free(st->ssl);
git__free(st->ssl);
git__free(st);
}
static int mbedtls_stream_wrap(
git_stream **out,
git_stream *in,
const char *host,
int owned)
{
mbedtls_stream *st;
int error;
st = git__calloc(1, sizeof(mbedtls_stream));
GIT_ERROR_CHECK_ALLOC(st);
st->io = in;
st->owned = owned;
st->ssl = git__malloc(sizeof(mbedtls_ssl_context));
GIT_ERROR_CHECK_ALLOC(st->ssl);
mbedtls_ssl_init(st->ssl);
if (mbedtls_ssl_setup(st->ssl, &mbedtls_config)) {
git_error_set(GIT_ERROR_SSL, "failed to create ssl object");
error = -1;
goto out_err;
}
st->host = git__strdup(host);
GIT_ERROR_CHECK_ALLOC(st->host);
st->parent.version = GIT_STREAM_VERSION;
st->parent.encrypted = 1;
st->parent.proxy_support = git_stream_supports_proxy(st->io);
st->parent.connect = mbedtls_connect;
st->parent.certificate = mbedtls_certificate;
st->parent.set_proxy = mbedtls_set_proxy;
st->parent.read = mbedtls_stream_read;
st->parent.write = mbedtls_stream_write;
st->parent.close = mbedtls_stream_close;
st->parent.free = mbedtls_stream_free;
*out = (git_stream *) st;
return 0;
out_err:
/* ESP FIX: do NOT close/free st->io here — on error the caller keeps
* ownership of `in` (git_mbedtls_stream_new closes+frees it right after;
* the vendored code freed it here too → double free → tlsf abort). */
mbedtls_ssl_free(st->ssl);
git__free(st->ssl);
git__free(st);
return error;
}
int git_mbedtls_stream_wrap(
git_stream **out,
git_stream *in,
const char *host)
{
return mbedtls_stream_wrap(out, in, host, 0);
}
int git_mbedtls_stream_new(
git_stream **out,
const char *host,
const char *port)
{
git_stream *stream;
int error;
GIT_ASSERT_ARG(out);
GIT_ASSERT_ARG(host);
GIT_ASSERT_ARG(port);
if ((error = git_socket_stream_new(&stream, host, port)) < 0)
return error;
if ((error = mbedtls_stream_wrap(out, stream, host, 1)) < 0) {
git_stream_close(stream);
git_stream_free(stream);
}
return error;
}
int git_mbedtls__set_cert_location(const char *file, const char *path)
{
int ret = 0;
char errbuf[512];
GIT_ASSERT_ARG(file || path);
if (has_ca_chain)
mbedtls_x509_crt_free(&mbedtls_ca_chain);
mbedtls_x509_crt_init(&mbedtls_ca_chain);
if (file)
ret = mbedtls_x509_crt_parse_file(&mbedtls_ca_chain, file);
if (ret >= 0 && path)
ret = mbedtls_x509_crt_parse_path(&mbedtls_ca_chain, path);
/* mbedtls_x509_crt_parse_path returns the number of invalid certs on success */
if (ret < 0) {
mbedtls_x509_crt_free(&mbedtls_ca_chain);
mbedtls_strerror( ret, errbuf, 512 );
git_error_set(GIT_ERROR_SSL, "failed to load CA certificates: %#04x - %s", ret, errbuf);
return -1;
}
mbedtls_ssl_conf_ca_chain(&mbedtls_config, &mbedtls_ca_chain, NULL);
has_ca_chain = true;
return 0;
}
#else
#include "stream.h"
int git_mbedtls_stream_global_init(void)
{
return 0;
}
#endif

View File

@@ -74,3 +74,12 @@ CONFIG_I2C_SKIP_LEGACY_CONFLICT_CHECK=y
# subset so a less common CA in the chain can't surprise us on the bench.
CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y
CONFIG_MBEDTLS_CERTIFICATE_BUNDLE_DEFAULT_FULL=y
# mbedTLS allocations go to PSRAM. The default (internal-only) needs ~33 KB of
# contiguous internal RAM per TLS connection (two ~17 KB I/O buffers + contexts)
# at the moment `:sync` pushes — with Wi-Fi, USB host, the editor and libgit2
# all resident, mbedtls_ssl_setup failed exactly there on the first real-repo
# push (2026-07-13; the failure then tripped the vendored stream double-free —
# see components/libgit2/esp_mbedtls_stream.c). TLS buffers are CPU-only data,
# so PSRAM is safe; the handshake is network-bound, not memory-bandwidth-bound.
CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=y

View File

@@ -248,9 +248,10 @@ fn publish_cycle(
/// error, surfaced as such.
fn publish_once(paths: &BTreeSet<String>) -> Result<PublishOutcome> {
log::info!(
"publish started — {} dirty path(s), free heap {}",
"publish started — {} dirty path(s), free heap {} ({} internal)",
paths.len(),
free_heap()
free_heap(),
internal_free_heap()
);
let repo = Repository::open(REPO_DIR).with_context(|| {
format!("opening git repo at {REPO_DIR} — provision the card with a clone (just init) whose origin is your remote")
@@ -313,8 +314,9 @@ fn publish_once(paths: &BTreeSet<String>) -> Result<PublishOutcome> {
}
log::info!(
"push done — free heap {}, min-ever {}",
"push done — free heap {} ({} internal), min-ever {}",
free_heap(),
internal_free_heap(),
min_free_heap()
);
Ok(PublishOutcome::Pushed(short(oid)))
@@ -388,11 +390,12 @@ fn stage_and_commit(repo: &Repository, paths: &BTreeSet<String>) -> Result<Optio
.commit(Some("HEAD"), &sig, &sig, &message, &tree, &parents)
.context("creating commit")?;
log::info!(
"commit split — splice {splice_ms}ms ({} path(s)), commit-obj {}ms; committed {} — free heap {}",
"commit split — splice {splice_ms}ms ({} path(s)), commit-obj {}ms; committed {} — free heap {} ({} internal)",
paths.len(),
t_commit.elapsed().as_millis(),
short(oid),
free_heap()
free_heap(),
internal_free_heap()
);
Ok(Some(oid))
}
@@ -636,6 +639,14 @@ fn free_heap() -> u32 {
unsafe { sys::esp_get_free_heap_size() }
}
/// Free INTERNAL RAM (DRAM), excluding PSRAM. `free_heap` is dominated by the
/// 8 MB PSRAM pool and masks internal exhaustion — which is what actually
/// killed the first real-repo push (mbedTLS's ssl_setup could not get its
/// ~33 KB while Wi-Fi + USB + editor + libgit2 were resident).
fn internal_free_heap() -> u32 {
unsafe { sys::heap_caps_get_free_size(sys::MALLOC_CAP_INTERNAL) as u32 }
}
fn min_free_heap() -> u32 {
unsafe { sys::esp_get_minimum_free_heap_size() }
}

View File

@@ -123,7 +123,18 @@ fn main() -> anyhow::Result<()> {
ed.set_notice(format!("loaded {name}"));
// Feed the file palette (Ctrl-P). Enumerated once at boot — the v0.5 slices
// that create/delete files (`:enew`, delete) will re-feed it then.
// Bracketed with internal-DRAM readings: each path is a small String, kept
// internal by the SPIRAM malloc threshold (16 KB), so the list competes
// with Wi-Fi/TLS for DRAM. Estimate to confirm: ~60-70 KB at 1098 files —
// this number decides whether interning the paths into one shared buffer
// (a single >16 KB alloc, which lands in PSRAM) is worth the refactor.
let dram_before = internal_free_heap();
ed.set_file_list(enumerate_files());
let dram_after = internal_free_heap();
log::info!(
"file list: internal heap {dram_before} -> {dram_after} ({} KB consumed)",
dram_before.saturating_sub(dram_after) / 1024
);
// Editor preferences (.typoena.toml, git-tracked). Read before the first
// render so `line_numbers` shapes the opening frame. A missing / unreadable /
// partial file falls back to defaults, so a fresh card just works.
@@ -551,36 +562,71 @@ fn delete_buffer(storage: &Storage, ed: &mut Editor, path: String, scope: Scope)
}
}
/// Enumerate the palette's openable files: the top-level regular files in
/// `/sd/repo` and `/sd/local`, as absolute paths. Skips dotfiles (so `.git`,
/// `.typoena.toml`, and the like never show) and anything that isn't a plain
/// file. Best-effort: an unreadable directory (e.g. no `/sd/local` yet)
/// contributes nothing rather than failing. The editor sorts and dedupes.
/// Enumerate the palette's openable files: the regular files under `/sd/repo`
/// and `/sd/local`, recursively, as absolute paths. Skips dot entries at every
/// level (so `.git` and its thousands of object files, `.typoena.toml`, and the
/// like never show or get walked). Best-effort: an unreadable directory (e.g.
/// no `/sd/local` yet) contributes nothing rather than failing. The editor
/// sorts and dedupes. Runs once at boot, so the walk time is logged — on a big
/// repo the FAT directory IO is the cost to watch.
fn enumerate_files() -> Vec<String> {
let start = std::time::Instant::now();
let mut out = Vec::new();
for dir in [REPO_DIR, LOCAL_DIR] {
let Ok(entries) = std::fs::read_dir(dir) else {
walk_files(std::path::Path::new(dir), 0, &mut out);
}
log::info!("file walk: {} files in {}ms", out.len(), start.elapsed().as_millis());
out
}
/// Depth bound for [`walk_files`] — belt-and-braces against pathological
/// nesting on a hand-edited card; notes trees are a couple of levels deep.
const WALK_MAX_DEPTH: usize = 8;
/// Recursive helper for [`enumerate_files`]: push `dir`'s files onto `out`,
/// then descend into its subdirectories. Reads each directory fully before
/// recursing (the `remove_dir_recursive` pattern in `git_sync`), so only one
/// FatFS directory handle is open at a time regardless of depth — relevant on
/// the FD-bounded SD mount.
fn walk_files(dir: &std::path::Path, depth: usize, out: &mut Vec<String>) {
if depth > WALK_MAX_DEPTH {
log::warn!("file walk: {} exceeds depth {WALK_MAX_DEPTH}, skipped", dir.display());
return;
}
let Ok(entries) = std::fs::read_dir(dir) else {
return;
};
// Keep the dirent's own file type: esp-idf's FAT VFS always fills d_type
// (DT_DIR/DT_REG, straight from the FILINFO readdir already holds), so
// `file_type()` is free. A per-entry `metadata()` stat instead re-walks
// the directory by path every time — measured at ~32ms/file on the SD
// card, it turned a 1098-file walk into 35s.
let children: Vec<_> = entries
.flatten()
.filter_map(|e| e.file_type().ok().map(|t| (e.path(), t)))
.collect();
for (path, ftype) in children {
let Some(name) = path.file_name().and_then(|n| n.to_str()) else {
continue;
};
for entry in entries.flatten() {
let path = entry.path();
let Some(name) = path.file_name().and_then(|n| n.to_str()) else {
continue;
};
if name.starts_with('.') {
continue;
}
// Stat rather than trust d_type — FatFS's dirent type can read back
// as unknown; a plain metadata call is reliable here.
if !std::fs::metadata(&path).map(|m| m.is_file()).unwrap_or(false) {
continue;
}
if name.starts_with('.') {
continue;
}
if ftype.is_file() {
if let Some(p) = path.to_str() {
out.push(p.to_string());
}
} else if ftype.is_dir() {
walk_files(&path, depth + 1, out);
}
}
out
}
/// Free internal DRAM (excludes the 8 MB PSRAM pool, which dominates the total
/// free-heap number and masks DRAM exhaustion). Same reading `git_sync` logs.
fn internal_free_heap() -> u32 {
use esp_idf_svc::sys;
unsafe { sys::heap_caps_get_free_size(sys::MALLOC_CAP_INTERNAL) as u32 }
}
/// A file's display name — its basename without extension (`/sd/repo/notes.md`