docs(notes): record why the notes repo stays full-size (pre-seed the SD)

The 150 MB of images are remanso's image CDN (it reads image bytes straight
out of git as blobs), not offloadable bloat — LFS/filter-repo/git-rm all
break remanso. libgit2 also lacks blobless partial clone. Decision: leave
the repo untouched and pre-seed the SD with a full clone from a computer, so
the device only ever takes the open + incremental fetch/push path. Also
notes the increment-C constraint: stage specific paths (not add_all) so a
media-excluding sparse checkout is safe.
This commit is contained in:
Julien Calixte
2026-07-07 22:55:59 +02:00
parent ef755bac81
commit 2aa4032e1f

View File

@@ -0,0 +1,131 @@
# Why we don't shrink the notes repo
> The 150 MB of images in my notes repo isn't bloat to offload — it's the
> image CDN for the web app that reads the same repo. Shrinking it for
> Typoena's sake breaks remanso. So we don't.
The question that started this: **Typoena** keeps a persistent clone of my
notes repo (`github.com/jcalixte/notes`) on its SD card and fast-forwards it
on every `Ctrl-G`. The most likely first failure is a cold clone that's too
big — so the instinct was "clone the least data possible; ignore
`node_modules`; maybe strip the media." That instinct is mostly wrong, for
reasons worth writing down before anyone force-pushes a rewritten history.
## The measured reality
| Metric (target repo, measured 2026-07-07) | Value |
| --- | --- |
| Working tree | 3.9 GB (dominated by `node_modules`) |
| `.git` (what a clone actually transfers) | **566 MB** |
| Commits / objects | 13,852 / 63,252 |
| Depth-1 snapshot (HEAD tree + blobs) | **154.7 MB** |
| — of which markdown (the notes) | **1.4 MB** |
| — of which media (png/jpg/pdf/gif/bmp) | **153 MB** |
| Media across *all* history (dedup) | 566 MB (715 PNG objects = 463 MB) |
Two early assumptions corrected:
- **`node_modules` is a non-issue.** It's gitignored and was never committed,
so a clone never touches it. There's nothing to filter. The 3.9 GB working
tree is a red herring; a clone transfers the 566 MB `.git`.
- **The risk isn't stack overflow, it's transfer size.** 566 MB over Wi-Fi +
mbedTLS to an SPI SD card, with no resume, is 30+ minutes and one dropout
away from failure. Stack/memory pressure is a symptom of asking the device
to cold-clone half a gigabyte.
## Why shrinking is off the table
The clean fix for "device only needs 1.4 MB of text" would be a blobless
partial clone (`--filter=blob:none`) — **libgit2 does not support it**. The
fallbacks are LFS migration, a filter-repo history purge, or `git rm` of media
at HEAD. All three remove the image *blobs* from what a git client sees.
That breaks **remanso**, the web app that reads the same repo. remanso is a
frontend with no server-side git; it displays a note image by reading the
image straight out of git as a blob and inlining it as a data URI
(`useImages.hook.ts` + `repo.ts`):
1. Markdown `<img src="relative/path.png">`
2. resolve path → find the file's `sha` in the GitHub tree
3. `GET /repos/{owner}/{repo}/git/blobs/{sha}`**base64 of the git blob**
4. `img.src = "data:image/jpeg;base64," + thatContent`
The image *is* the git blob. GitHub's Git Blobs / Trees / Contents APIs do
**not** resolve LFS (only `media.githubusercontent.com` does, which remanso
doesn't use). So after an LFS migration those endpoints return the ~130-byte
**pointer text**, remanso wraps it in `data:image/jpeg;base64,…`, and every
image renders broken. Uploads break too: remanso writes images via
`PUT /contents`, which ignores `.gitattributes`/LFS and commits a plain blob.
And it's not specific to LFS — `git rm` at HEAD or a filter-repo purge remove
the blobs from the tree, so remanso can't find them either. **Any approach
that takes the images out of the git repo breaks remanso**, because those
150 MB are load-bearing infrastructure for the web app, not offloadable bloat.
| Consumer | How it reads images | If we shrink the repo |
| --- | --- | --- |
| Typoena (libgit2) | doesn't render images; needs valid git objects | fine — would get tiny pointers |
| remanso (Blobs API → data URI) | reads image bytes straight out of git | **broken** — pointer/missing bytes render as a dead image; uploads bypass LFS |
## Decision
**Leave the notes repo untouched. Pre-seed the device SD card with a full
`git clone` from a computer.**
Repo size is only a *device* constraint when the *device* does the cold clone.
A laptop clones 566 MB in ~2 minutes onto the SD via a card reader; the SD has
GB to spare. The device then only ever takes the `open` + incremental
fetch/commit/push path (`open_or_clone` already splits on this). A *full*
pre-seed (not depth-1) also sidesteps the shallow-push sharp edge. remanso
keeps working, the device gets everything, and repo size stops being anyone's
problem.
## What happens on an ongoing pull
In the single-writer model the device usually doesn't pull at all:
`fetch_and_integrate` runs only from the rejected-push arm of
`push_with_retry`, and a sole writer always fast-forwards. "Images to pull"
only arises when a **second writer** (remanso or the desktop) pushed them.
When it does, today the device fetches (downloading the image blobs) and then
hits the divergence bail (`increment B, deferred`) — no data loss, but no
integration until the merge path exists.
Once integration lands, the costs of carrying media the device never renders:
1. **Bandwidth for unusable bytes.** No partial fetch in libgit2, so a fetch
pulls the full new image blobs. 20 MB of pasted screenshots = a 20 MB fetch
before a one-line note can publish.
2. **~2× SD storage.** Each image lives in `.git` *and* in the working-tree
checkout that `checkout_head(force)` writes.
3. **Memory — the real edge.** libgit2 tends to materialize a whole blob in
RAM for checkout. History already has a 38 MB mp3 and 16 MB PNGs. Against
**8 MB PSRAM**, a single large image arriving in a pull is a genuine OOM
risk on checkout. Verify on hardware (push a 20 MB image from remanso, watch
`min_free_heap`) rather than trusting a read of libgit2 internals.
**The trap:** `publish()` stages with `add_all(["*"])`. A sparse checkout that
omitted images would make `add_all` see them as deleted → the device would
commit "delete all images" and push it → remanso loses every image. So with
the current staging, a full checkout is mandatory — which is what feeds costs
2 and 3.
## Fix for increment C (git module in the editor)
Change two things together:
- **Stage specific paths, not `add_all(["*"])`.** The editor knows which note
file it wrote — commit that path explicitly.
- Then a **sparse checkout that excludes media** is safe: the device never
writes images to its working tree, killing the checkout OOM and the 2×
storage. The bytes still transit `.git` on fetch (no partial clone), but
writing objects to disk is far lower memory risk than a checkout that
materializes them in RAM.
## Related
- `firmware/src/bin/git_sync.rs` — the persistent-clone publish cycle analysed
here (milestone #2A).
- ADR-010 — "writing tool, not sync engine": the principle this decision
serves.
- remanso: `src/hooks/useImages.hook.ts`, `src/modules/repo/services/repo.ts`
(`queryFileContent`) — the image-as-blob pipeline.