chore: remove bar
This commit is contained in:
Julien Calixte
@@ -10,11 +10,11 @@ device — only the toy test repo has. Full measurement trail:
|
||||
|
||||
**Value model for the writer (Julien on the appliance)**
|
||||
|
||||
| Want more of | Do less of |
|
||||
|---|---|
|
||||
| + words safely on the remote with one keystroke | – waiting on a frozen device |
|
||||
| + trust that `:sync` completes, every time | – power-cycling mid-sync and losing the session's trust |
|
||||
| + keep writing while it publishes | – babysitting git from a desktop to rescue the card |
|
||||
| Want more of | Do less of |
|
||||
| ----------------------------------------------- | ------------------------------------------------------- |
|
||||
| + words safely on the remote with one keystroke | – waiting on a frozen device |
|
||||
| + trust that `:sync` completes, every time | – power-cycling mid-sync and losing the session's trust |
|
||||
| + keep writing while it publishes | – babysitting git from a desktop to rescue the card |
|
||||
|
||||
**Measurement**: seconds from `:sync` to the snackbar, on the real
|
||||
`jcalixte/notes` clone on the device.
|
||||
@@ -30,7 +30,7 @@ xychart-beta
|
||||
title "Improvement potential — :sync → snackbar"
|
||||
x-axis ["Before kaizen (~611 s, never completes)", "After kaizen (target ≤ ~10 s)"]
|
||||
y-axis "seconds" 0 --> 650
|
||||
bar [611, 0]
|
||||
bar [611]
|
||||
line [10, 10]
|
||||
```
|
||||
|
||||
@@ -44,13 +44,13 @@ after column has no bar until the step-6 measurement lands.
|
||||
:sync (current method, real repo)
|
||||
│
|
||||
▼
|
||||
┌─ local commit ────────────────────────────────────┐ ┌─ network push ──────┐
|
||||
│ stage: add_all(["*"]) + update_all(["*"]) │ │ TLS + push │
|
||||
│ → stat()s 1179 files / 158 dirs over 10 MHz SPI │──▶│ ~6.5 s floor │
|
||||
│ index.write ⚡ ~611 s │ │ (separate curve) │
|
||||
│ → FAT's 2 s mtime marks ~every entry "racy" │ └─────────────────────┘
|
||||
│ → re-hashes ~170 MB (150 MB of images) │
|
||||
│ write_tree + commit-obj ⚡ seconds each │
|
||||
┌─ local commit ─────────────────────────────────────┐ ┌─ network push ──────┐
|
||||
│ stage: add_all(["*"]) + update_all(["*"]) │ │ TLS + push │
|
||||
│ → stat()s 1179 files / 158 dirs over 10 MHz SPI │──▶│ ~6.5 s floor │
|
||||
│ index.write ⚡ ~611 s │ │ (separate curve) │
|
||||
│ → FAT's 2 s mtime marks ~every entry "racy" │ └─────────────────────┘
|
||||
│ → re-hashes ~170 MB (150 MB of images) │
|
||||
│ write_tree + commit-obj ⚡ seconds each │
|
||||
│ → loose writes trigger pack reads through │
|
||||
│ emulated mmap; each far lseek walks the │
|
||||
│ FAT cluster chain (~190 ms) │
|
||||
@@ -62,16 +62,16 @@ Every factor was benched on the device (`sd_bench` for raw FAT, `git_bench` for
|
||||
git2 primitives on the real clone) — details and full tables in
|
||||
[`../tradeoff-curves/sync-commit-staging.md`](../tradeoff-curves/sync-commit-staging.md):
|
||||
|
||||
| Factor # | Factor | Hypothesis | Test method | Test result | True or false? |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | SD card raw I/O | The card itself is slow: a loose-object write costs ~700–900 ms | `sd_bench`: raw FAT create/write/rename composite, same card, same 10 MHz | complete loose-object composite **86 ms** — 8× under libgit2's number | **FALSE** |
|
||||
| 2 | `index.write()` racy-clean | FAT's 2 s mtime granularity makes ~all 1179 entries look racy → full ~170 MB re-hash over SPI | `git_bench` on the real clone, 3 consecutive `index.write()` | **611 s → 12.8 s → 360 ms** — the decay is the racy set shrinking as the index mtime advances | **TRUE** |
|
||||
| 3 | Index-free alternative | Skipping `index.write` (fresh in-memory index) escapes the wall | `git_bench`: `Index::new` + `read_tree(HEAD)` + `write_tree_to` | seed `read_tree` **77 s cold** + mmap grows to 7.4 MB → **zlib OOM crash** — still O(N_tree) | **FALSE** (as a fix) |
|
||||
| 4 | FatFS `lseek` | Long/backward seeks walk the FAT cluster chain over SPI (~67 KB of FAT reads for a 263 MB file) | `sd_bench`: seek+read 4 KB @offset 0 vs @end of the pack; A/B with `CONFIG_FATFS_USE_FASTSEEK` | @0 = 5.8 ms, @end = **198.7 ms**; fast-seek → **20.4 ms** | **TRUE** |
|
||||
| 5 | Free-cluster scan | A ~740 MB-full card slows FAT allocation | `sd_bench` re-run on the full card | composite 77 ms — unchanged | **FALSE** |
|
||||
| 6 | Strict object creation | OID validation on commit/treebuilder causes the ~1.3 s commit premium | strict-off A/B in `git_bench` | 1.80 s vs 1.93 s — no premium removed | **FALSE** |
|
||||
| 7 | Repeated small mmap windows | A window cache in `esp_map.c` would absorb the ~8 small pack reads per loose write | instrumented cache (v2), 4 real-repo runs | **0 hits / 313 misses** — every read hits a unique (offset, len); `mwindow` already absorbs true repetition | **FALSE** |
|
||||
| 8 | Cache memory discipline | The OOM in factor 3 is our own cache holding buffers past `p_munmap`, defeating `MWINDOW_MAPPED_LIMIT` | run-4 memory monitor after evict-on-munmap fix, then full removal (run 5) | resident flat at 1833 KB, heap ≥ 6.2 MB, no OOM; removal I/O-neutral | **TRUE** |
|
||||
| Factor # | Factor | Hypothesis | Test method | Test result | True or false? |
|
||||
| -------- | --------------------------- | ------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | -------------------- |
|
||||
| 1 | SD card raw I/O | The card itself is slow: a loose-object write costs ~700–900 ms | `sd_bench`: raw FAT create/write/rename composite, same card, same 10 MHz | complete loose-object composite **86 ms** — 8× under libgit2's number | **FALSE** |
|
||||
| 2 | `index.write()` racy-clean | FAT's 2 s mtime granularity makes ~all 1179 entries look racy → full ~170 MB re-hash over SPI | `git_bench` on the real clone, 3 consecutive `index.write()` | **611 s → 12.8 s → 360 ms** — the decay is the racy set shrinking as the index mtime advances | **TRUE** |
|
||||
| 3 | Index-free alternative | Skipping `index.write` (fresh in-memory index) escapes the wall | `git_bench`: `Index::new` + `read_tree(HEAD)` + `write_tree_to` | seed `read_tree` **77 s cold** + mmap grows to 7.4 MB → **zlib OOM crash** — still O(N_tree) | **FALSE** (as a fix) |
|
||||
| 4 | FatFS `lseek` | Long/backward seeks walk the FAT cluster chain over SPI (~67 KB of FAT reads for a 263 MB file) | `sd_bench`: seek+read 4 KB @offset 0 vs @end of the pack; A/B with `CONFIG_FATFS_USE_FASTSEEK` | @0 = 5.8 ms, @end = **198.7 ms**; fast-seek → **20.4 ms** | **TRUE** |
|
||||
| 5 | Free-cluster scan | A ~740 MB-full card slows FAT allocation | `sd_bench` re-run on the full card | composite 77 ms — unchanged | **FALSE** |
|
||||
| 6 | Strict object creation | OID validation on commit/treebuilder causes the ~1.3 s commit premium | strict-off A/B in `git_bench` | 1.80 s vs 1.93 s — no premium removed | **FALSE** |
|
||||
| 7 | Repeated small mmap windows | A window cache in `esp_map.c` would absorb the ~8 small pack reads per loose write | instrumented cache (v2), 4 real-repo runs | **0 hits / 313 misses** — every read hits a unique (offset, len); `mwindow` already absorbs true repetition | **FALSE** |
|
||||
| 8 | Cache memory discipline | The OOM in factor 3 is our own cache holding buffers past `p_munmap`, defeating `MWINDOW_MAPPED_LIMIT` | run-4 memory monitor after evict-on-munmap fix, then full removal (run 5) | resident flat at 1833 KB, heap ≥ 6.2 MB, no OOM; removal I/O-neutral | **TRUE** |
|
||||
|
||||
### Details on hypothesis #2 (the freeze itself)
|
||||
|
||||
@@ -98,15 +98,15 @@ the device.)
|
||||
|
||||
### New ideas
|
||||
|
||||
| Name | Estimated gain | Estimated lead time | Cause addressed | Comments |
|
||||
|---|---|---|---|---|
|
||||
| **O(depth) TreeBuilder splice** — patch only the edited path's ancestor chain onto HEAD's tree | commit 611 s → ~2–3 s; **flat in repo size forever** | ~2 days (bench op, then plumbing) | #2 + #3 — never opens the index, never materialises the tree | **Chosen.** Bonus: carries the 150 MB of images forward by OID, so the device doesn't even need them in its working tree |
|
||||
| `CONFIG_FATFS_USE_FASTSEEK` + 256-word CLMT buffer | 2.3× on the splice (6.5 → 2.8 s); far seek 198.7 → 20.4 ms | hours — config, not code | #4 | Shipped as the companion fix; write-mode files fall back transparently |
|
||||
| Explicit-path index staging (`add_path` over the editor's dirty set) | removes the O(N) walk term only | ~1 day | #2 partially | Retired — still calls `index.write()`, so it hits the same racy-clean wall |
|
||||
| Index-free `write_tree_to` on a fresh in-memory index | — | ~1 day | #2 | Refuted by factor 3: seeding `read_tree(HEAD)` is 77 s + OOM — trades one O(N) for another |
|
||||
| `esp_map.c` window cache (v2: size-keyed admission, evict-on-munmap) | hoped ~150–250 ms per loose write | ~1 day | #7 | Built and instrumented → **0 hits in 4 runs** → removed entirely. The one build made on an unverified cause; see learnings |
|
||||
| Faster card / SD at 20 MHz | none for the commit | PCB respin budget | #1 | Rejected — factor 1 refuted; the card was exonerated twice (86 ms then 77 ms composites) |
|
||||
| Shrink the repo (images off-card / LFS-style pointers) | shrinks N and the 570 MB clone | unknown | N itself | Rejected for now — the images are load-bearing for another app ([`../notes/git-sync-images-and-repo-size.md`](../notes/git-sync-images-and-repo-size.md)). Composes with the splice later |
|
||||
| Name | Estimated gain | Estimated lead time | Cause addressed | Comments |
|
||||
| ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | --------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **O(depth) TreeBuilder splice** — patch only the edited path's ancestor chain onto HEAD's tree | commit 611 s → ~2–3 s; **flat in repo size forever** | ~2 days (bench op, then plumbing) | #2 + #3 — never opens the index, never materialises the tree | **Chosen.** Bonus: carries the 150 MB of images forward by OID, so the device doesn't even need them in its working tree |
|
||||
| `CONFIG_FATFS_USE_FASTSEEK` + 256-word CLMT buffer | 2.3× on the splice (6.5 → 2.8 s); far seek 198.7 → 20.4 ms | hours — config, not code | #4 | Shipped as the companion fix; write-mode files fall back transparently |
|
||||
| Explicit-path index staging (`add_path` over the editor's dirty set) | removes the O(N) walk term only | ~1 day | #2 partially | Retired — still calls `index.write()`, so it hits the same racy-clean wall |
|
||||
| Index-free `write_tree_to` on a fresh in-memory index | — | ~1 day | #2 | Refuted by factor 3: seeding `read_tree(HEAD)` is 77 s + OOM — trades one O(N) for another |
|
||||
| `esp_map.c` window cache (v2: size-keyed admission, evict-on-munmap) | hoped ~150–250 ms per loose write | ~1 day | #7 | Built and instrumented → **0 hits in 4 runs** → removed entirely. The one build made on an unverified cause; see learnings |
|
||||
| Faster card / SD at 20 MHz | none for the commit | PCB respin budget | #1 | Rejected — factor 1 refuted; the card was exonerated twice (86 ms then 77 ms composites) |
|
||||
| Shrink the repo (images off-card / LFS-style pointers) | shrinks N and the 570 MB clone | unknown | N itself | Rejected for now — the images are load-bearing for another app ([`../notes/git-sync-images-and-repo-size.md`](../notes/git-sync-images-and-repo-size.md)). Composes with the splice later |
|
||||
|
||||
**Chosen idea**: the O(depth) TreeBuilder splice, with fast-seek as its config
|
||||
companion — it is the only candidate whose cost is independent of repo size, so
|
||||
@@ -117,14 +117,14 @@ stays O(N_tree) and merely moves the wall.
|
||||
|
||||
**What could go wrong?**
|
||||
|
||||
| Lens | Anticipated consequence | Mitigation |
|
||||
|---|---|---|
|
||||
| Stable | libgit2's 32-bit mwindow defaults `malloc` a 32 MB window on first pack access → instant OOM on the 8 MB PSRAM heap | `GIT_OPT_SET_MWINDOW_SIZE` 256 KB / `MAPPED_LIMIT` 4 MB set at service start, before any `Repository::open` |
|
||||
| Stable | Power pull between a save and the next `:sync` loses the dirty set — nothing walks the tree anymore, so an unrecorded file would **never** sync | Dirty set journaled to `/sd/.typoena-dirty` (atomic write), recorded *before* the file write; over-reporting is a free no-op splice, so the semantics stay simple |
|
||||
| Stable | Commit lands, push fails → the commit strands forever behind "up to date" | Stranded-commit recovery: compare HEAD against `refs/remotes/origin/<branch>` and push even when there is nothing new to commit |
|
||||
| Method | Reconcile's `Mixed` reset writes the index → re-enters the racy-clean wall through the back door | `ResetType::Soft` — ref move only; there is no index to reset anymore |
|
||||
| Method | **Deliberate behavior change:** files edited on the card from a desktop are never committed anymore (`add_all` used to sweep them in) | Accepted and documented as intent — the appliance's editor is the only writer that counts |
|
||||
| Machine | libgit2 holds the pack + `.idx` descriptors open and opens loose objects on top → blows the editor's 4-FD mount | git builds mount with 16 FDs (`Storage::mount_for_git()`) |
|
||||
| Lens | Anticipated consequence | Mitigation |
|
||||
| ------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Stable | libgit2's 32-bit mwindow defaults `malloc` a 32 MB window on first pack access → instant OOM on the 8 MB PSRAM heap | `GIT_OPT_SET_MWINDOW_SIZE` 256 KB / `MAPPED_LIMIT` 4 MB set at service start, before any `Repository::open` |
|
||||
| Stable | Power pull between a save and the next `:sync` loses the dirty set — nothing walks the tree anymore, so an unrecorded file would **never** sync | Dirty set journaled to `/sd/.typoena-dirty` (atomic write), recorded _before_ the file write; over-reporting is a free no-op splice, so the semantics stay simple |
|
||||
| Stable | Commit lands, push fails → the commit strands forever behind "up to date" | Stranded-commit recovery: compare HEAD against `refs/remotes/origin/<branch>` and push even when there is nothing new to commit |
|
||||
| Method | Reconcile's `Mixed` reset writes the index → re-enters the racy-clean wall through the back door | `ResetType::Soft` — ref move only; there is no index to reset anymore |
|
||||
| Method | **Deliberate behavior change:** files edited on the card from a desktop are never committed anymore (`add_all` used to sweep them in) | Accepted and documented as intent — the appliance's editor is the only writer that counts |
|
||||
| Machine | libgit2 holds the pack + `.idx` descriptors open and opens loose objects on top → blows the editor's 4-FD mount | git builds mount with 16 FDs (`Storage::mount_for_git()`) |
|
||||
|
||||
The mechanism was stress-tested as a prototype first, never in production: the
|
||||
splice ran as a `git_bench` op against a full real-repo clone through four
|
||||
|
||||
Reference in New Issue
Block a user