Compare commits

...

22 Commits

Author SHA1 Message Date
Julien Calixte
7623fff227 fix(test): match getSession URL by suffix, not /getSession path
All checks were successful
CI / check (push) Successful in 13s
The PDS endpoint URL is /xrpc/com.atproto.server.getSession —
those are dots before getSession, not slashes, so the previous
includes("/getSession") filter never matched and the Authorization
header assertion silently short-circuited via optional chaining.
2026-06-07 22:31:26 +02:00
Julien Calixte
2105d75cb5 ci(gitea): scope deno test and check to the new tests directory
Some checks failed
CI / check (push) Failing after 11s
2026-06-07 22:27:23 +02:00
Julien Calixte
e937863bc6 test(server): cover HTTP routes, auth, admin gate, and CORS preflight
Some checks failed
CI / check (push) Failing after 12s
2026-06-07 22:26:54 +02:00
Julien Calixte
5c4b189136 test(jetstream): cover bulk-create debounce and flush semantics
Some checks failed
CI / check (push) Failing after 11s
Uses @std/testing/time FakeTime to deterministically advance the
500ms debounce. Covers accumulation, timer restart on each push,
the bulk-create verb + payload shape, and per-did buffer
isolation with cleared state after flush.
2026-06-07 22:25:48 +02:00
Julien Calixte
764ba09f77 test(jetstream): cover dispatchAll resilience and header handling
Some checks failed
CI / check (push) Failing after 13s
2026-06-07 22:24:51 +02:00
Julien Calixte
365deb305a test(search): cover cursor, snippet, and searchNotes query body
Some checks failed
CI / check (push) Failing after 10s
2026-06-07 22:24:21 +02:00
Julien Calixte
02f3139f46 refactor(search): read OPENSEARCH_* env per call
Some checks failed
CI / check (push) Failing after 12s
Tests can't override values captured at module load because ES
imports hoist before any Deno.env.set call. Reading env on each
helper invocation makes the module trivially testable and has
no production cost (env doesn't change at runtime).
2026-06-07 22:23:36 +02:00
Julien Calixte
6159ec00e0 ci(gitea): cache DENO_DIR keyed on deno.lock
Some checks failed
CI / check (push) Failing after 16s
setup-deno's built-in cache stores fetched JSR/npm modules and
compiled dependencies across runs. Keying on the lockfile hash
invalidates the cache only when dependencies change.
2026-06-07 22:22:10 +02:00
Julien Calixte
3611add21a test(auth): cover JWT decoding and authenticateRequest flow 2026-06-07 22:21:48 +02:00
Julien Calixte
34b8221376 fix(test): pass explicit did to seedWebhook in CRUD test
All checks were successful
CI / check (push) Successful in 15s
The webhook CRUD test combined an explicit did:plc:user webhook
with seedWebhook calls that defaulted to did:plc:test, so
listWebhooksByDid("did:plc:user") only returned the one
explicit row instead of three.
2026-06-07 22:20:52 +02:00
Julien Calixte
292fe8f85a test(migrations): cover initial schema, idempotency, and legacy upgrade
Some checks failed
CI / check (push) Failing after 17s
2026-06-07 22:17:54 +02:00
Julien Calixte
2d7863d9cc test(db): cover note CRUD, pagination, and webhook helpers 2026-06-07 22:17:23 +02:00
Julien Calixte
c62936b819 test(helpers): add db, fetch, app, and auth test helpers
withTestDb sets up an in-memory SQLite handle with migrations
applied, swaps the production singleton via _setDbForTest, and
restores in a finally block. installFetchStub captures every
outbound request and throws if restored twice. requestApp wraps
Oak's app.handle so route tests can run without listen().
stubAuthenticate flips the server's authenticator indirection.
2026-06-07 22:16:47 +02:00
Julien Calixte
62a55fd3e4 fix(jetstream): use ReturnType<typeof setTimeout> for bulk timer
All checks were successful
CI / check (push) Successful in 14s
setTimeout returns an opaque Timeout type in newer Deno
releases, so the previous timer: number annotation fails type
checking in CI. Using ReturnType<typeof setTimeout> is portable
across Deno versions and runtimes.
2026-06-07 22:14:03 +02:00
Julien Calixte
49608cff20 refactor(search,auth): export pure helpers for direct testing
encodeCursor/decodeCursor/firstSnippet (opensearch) and
decodeJwtPayload (verify) become public exports so tests can
hit them without round-tripping through fetch.
2026-06-07 22:06:49 +02:00
Julien Calixte
01048c753e refactor(jetstream): extract webhook dispatch and bulk-create modules
dispatchAll, fireWebhooks, and the bulk-create debounce buffer
move out of jetstream.ts into src/jetstream/{webhooks,bulk}.ts.
fireWebhooks and the bulk helpers now take their DB lookup and
dispatch as parameters so tests can inject stubs. Top-level
Jetstream construction, event handlers, ensureIndex call, and
the cursor-saving interval are gated behind import.meta.main so
importing jetstream.ts no longer opens a WebSocket.
2026-06-07 22:06:19 +02:00
Julien Calixte
8964f66184 refactor(server): extract createApp and add auth/admin test seams
createApp() builds the Application without listening, so tests
can call app.handle() directly. The listen call is gated behind
import.meta.main so importing server.ts no longer boots a port.
ADMIN_DIDS is read inside requireAdmin on each call (was frozen
at module load), and a mutable authenticator indirection lets
tests stub authenticateRequest via _setAuthenticatorForTest.
2026-06-07 22:05:12 +02:00
Julien Calixte
6760c434b4 refactor(db): lazy-init connection and add test seam
The Database handle is now constructed on first use and held in
a module-level slot. New _setDbForTest and _resetDbForTest
exports let tests swap in a :memory: handle without rewriting
callers. closeDb() replaces the previous db.close() pattern used
by short-lived scripts. Adds listDistinctNoteDids() to keep the
backfill script from poking the raw connection.
2026-06-07 22:04:12 +02:00
Julien Calixte
dc6daf3f54 refactor(migrations): extract applyMigrations as callable function 2026-06-07 22:01:57 +02:00
Julien Calixte
7875d24d48 refactor(search): use typed OpenSearch SDK request body
Some checks failed
CI / check (push) Failing after 1m0s
Pull in @opensearch-project/opensearch only for its
Search_RequestBody type so the inline search body is
type-checked. Also drops the unsupported
highlight.content.boundary_scanner_locale option.
2026-06-07 12:04:43 +02:00
Julien Calixte
4ff3ea6644 docs(api): add Bruno collection for Jetstream API
Excludes docs/ from deno fmt and deno lint so the Bruno
files don't break formatting and lint checks in CI.
2026-06-07 12:04:37 +02:00
Julien Calixte
a6f1a96c4a ci(gitea): migrate pipeline from GitLab to Gitea Actions 2026-06-07 12:04:25 +02:00
48 changed files with 3340 additions and 367 deletions

30
.gitea/workflows/ci.yml Normal file
View File

@@ -0,0 +1,30 @@
name: CI
on:
push:
branches: [main]
pull_request:
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: denoland/setup-deno@v2
with:
deno-version: v2.x
cache: true
cache-hash: ${{ hashFiles('deno.lock') }}
- name: Format check
run: deno fmt --check
- name: Lint
run: deno lint
- name: Type check
run: deno check jetstream.ts server.ts scripts/*.ts src/migrations/init.ts tests/
- name: Test
run: deno test --allow-all tests/

View File

@@ -1,23 +0,0 @@
stages:
- build
- deploy
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: "/certs"
IMAGE_TAG: $CI_REGISTRY/litenote:$CI_COMMIT_REF_SLUG
IMAGE_LATEST: $CI_REGISTRY/litenote:latest
build:
stage: build
image: docker:24-dind
services:
- docker:24-dind
before_script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
script:
- docker build -t $IMAGE_TAG -t $IMAGE_LATEST .
- docker push $IMAGE_TAG
- docker push $IMAGE_LATEST
only:
- main

View File

@@ -1,4 +1,10 @@
{
"fmt": {
"exclude": ["docs/"]
},
"lint": {
"exclude": ["docs/"]
},
"tasks": {
"jetstream:prod": "deno run --allow-net --allow-read --allow-write --allow-env --allow-ffi --unstable-ffi jetstream.ts",
"jetstream": "deno run --watch --allow-net --allow-read --allow-write --allow-env --allow-ffi --unstable-ffi jetstream.ts",
@@ -12,7 +18,9 @@
"imports": {
"@db/sqlite": "jsr:@db/sqlite@^0.13.0",
"@oak/oak": "jsr:@oak/oak@^17.2.0",
"@opensearch-project/opensearch": "npm:@opensearch-project/opensearch@^3.6.0",
"@skyware/jetstream": "npm:@skyware/jetstream@^0.2.5",
"@std/assert": "jsr:@std/assert@1"
"@std/assert": "jsr:@std/assert@1",
"@std/testing": "jsr:@std/testing@1"
}
}

53
deno.lock generated
View File

@@ -7,8 +7,10 @@
"jsr:@oak/commons@1": "1.0.0",
"jsr:@oak/oak@^17.2.0": "17.2.0",
"jsr:@std/assert@1": "1.0.18",
"jsr:@std/async@^1.4.0": "1.4.0",
"jsr:@std/bytes@1": "1.0.4",
"jsr:@std/crypto@1": "1.0.3",
"jsr:@std/data-structures@^1.1.0": "1.1.0",
"jsr:@std/encoding@1": "1.0.6",
"jsr:@std/encoding@^1.0.5": "1.0.6",
"jsr:@std/fmt@1": "1.0.9",
@@ -20,6 +22,9 @@
"jsr:@std/path@1": "1.1.4",
"jsr:@std/path@1.0": "1.0.9",
"jsr:@std/path@^1.1.1": "1.1.4",
"jsr:@std/testing@1": "1.0.19",
"npm:@opensearch-project/opensearch@3": "3.6.0",
"npm:@opensearch-project/opensearch@^3.6.0": "3.6.0",
"npm:@skyware/jetstream@~0.2.5": "0.2.5",
"npm:path-to-regexp@^6.3.0": "6.3.0"
},
@@ -69,12 +74,18 @@
"jsr:@std/internal@^1.0.12"
]
},
"@std/async@1.4.0": {
"integrity": "4d70b008634f571cff9b554090d628c76141c32613aae0ff283fd5fa23d0c379"
},
"@std/bytes@1.0.4": {
"integrity": "11a0debe522707c95c7b7ef89b478c13fb1583a7cfb9a85674cd2cc2e3a28abc"
},
"@std/crypto@1.0.3": {
"integrity": "a2a32f51ddef632d299e3879cd027c630dcd4d1d9a5285d6e6788072f4e51e7f"
},
"@std/data-structures@1.1.0": {
"integrity": "c35ae4ad5d8e41a38573c2fe3e19b18ea2505f63cfea201edcb8720aca1f7f58"
},
"@std/encoding@1.0.6": {
"integrity": "ca87122c196e8831737d9547acf001766618e78cd8c33920776c7f5885546069"
},
@@ -108,6 +119,13 @@
"dependencies": [
"jsr:@std/internal@^1.0.12"
]
},
"@std/testing@1.0.19": {
"integrity": "f4236172365b216728dc3cc8b5e80a9f4c33083d1e4ede7613d5b25b4014898e",
"dependencies": [
"jsr:@std/async",
"jsr:@std/data-structures"
]
}
},
"npm": {
@@ -142,6 +160,17 @@
"unicode-segmenter"
]
},
"@opensearch-project/opensearch@3.6.0": {
"integrity": "sha512-ow1A/Z7MBlNL/JZzTY4+M+8psiVh66g0Z7EQSl/qbT0U6zLBsVDYh6nvm8D4e4tIzQtn6ODbA2OsqYGvqZj76g==",
"dependencies": [
"aws4",
"debug",
"hpagent",
"json11",
"ms",
"secure-json-parse"
]
},
"@skyware/jetstream@0.2.5": {
"integrity": "sha512-fM/zs03DLwqRyzZZJFWN20e76KrdqIp97Tlm8Cek+vxn96+tu5d/fx79V6H85L0QN6HvGiX2l9A8hWFqHvYlOA==",
"dependencies": [
@@ -155,12 +184,31 @@
"@standard-schema/spec@1.1.0": {
"integrity": "sha512-l2aFy5jALhniG5HgqrD6jXLi/rUWrKvqN/qJx6yoJsgKhblVd+iqqU4RCXavm/jPityDo5TCvKMnpjKnOriy0w=="
},
"aws4@1.13.2": {
"integrity": "sha512-lHe62zvbTB5eEABUVi/AwVh0ZKY9rMMDhmm+eeyuuUQbQ3+J+fONVQOZyj+DdrvD4BY33uYniyRJ4UJIaSKAfw=="
},
"debug@4.4.3": {
"integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==",
"dependencies": [
"ms"
]
},
"esm-env@1.2.2": {
"integrity": "sha512-Epxrv+Nr/CaL4ZcFGPJIYLWFom+YeV1DqMLHJoEd9SYRxNbaFruBwfEX/kkHUJf55j2+TUbmDcmuilbP1TmXHA=="
},
"event-target-polyfill@0.0.4": {
"integrity": "sha512-Gs6RLjzlLRdT8X9ZipJdIZI/Y6/HhRLyq9RdDlCsnpxr/+Nn6bU2EFGuC94GjxqhM+Nmij2Vcq98yoHrU8uNFQ=="
},
"hpagent@1.2.0": {
"integrity": "sha512-A91dYTeIB6NoXG+PxTQpCCDDnfHsW9kc06Lvpu1TEe9gnd6ZFeiBoRO9JvzEv6xK7EX97/dUE8g/vBMTqTS3CA=="
},
"json11@2.0.2": {
"integrity": "sha512-HIrd50UPYmP6sqLuLbFVm75g16o0oZrVfxrsY0EEys22klz8mRoWlX9KAEDOSOR9Q34rcxsyC8oDveGrCz5uLQ==",
"bin": true
},
"ms@2.1.3": {
"integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="
},
"partysocket@1.1.12": {
"integrity": "sha512-079YDW1QZsFwJ8syLmr9xFMbE+VwbK4SpcKSOPVApX8rpIMennkxx0MeWl4oWutP/Zjgy8TMnQZ3FOXTQvu3DA==",
"dependencies": [
@@ -170,6 +218,9 @@
"path-to-regexp@6.3.0": {
"integrity": "sha512-Yhpw4T9C6hPpgPeA28us07OJeqZ5EzQTkbfwuhsUg0c237RomFoETJgmp2sa3F/41gfLE6G5cqcYwznmeEeOlQ=="
},
"secure-json-parse@2.7.0": {
"integrity": "sha512-6aU+Rwsezw7VR8/nyvKTx8QpWH9FrcYiXXlqC4z5d5XQBDRqtbfsRjnwGyqbi3gddNtWHuEk9OANUotL26qKUw=="
},
"tiny-emitter@2.1.0": {
"integrity": "sha512-NB6Dk1A9xgQPMoGqC5CVXn123gWyte215ONT5Pp5a0yt4nlEoO1ZWeCwpncaekPHXO60i47ihFnZPiRPjRMq4Q=="
},
@@ -196,6 +247,8 @@
"jsr:@db/sqlite@0.13",
"jsr:@oak/oak@^17.2.0",
"jsr:@std/assert@1",
"jsr:@std/testing@1",
"npm:@opensearch-project/opensearch@^3.6.0",
"npm:@skyware/jetstream@~0.2.5"
]
}

View File

@@ -0,0 +1,114 @@
meta {
name: List every webhook subscription
type: http
seq: 1
}
get {
url: {{baseUrl}}/admin/webhooks
body: none
auth: bearer
}
auth:bearer {
token: {{token}}
}
example {
name: 200 Response
description: All subscriptions across all DIDs.
request: {
url: {{baseUrl}}/admin/webhooks
method: GET
mode: none
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 200
text: OK
}
body: {
type: json
content: '''
[
{
"id": 0,
"did": "",
"method": "",
"url": "",
"verb": ""
}
]
'''
}
}
}
example {
name: 401 Response
description: Missing or invalid bearer token.
request: {
url: {{baseUrl}}/admin/webhooks
method: GET
mode: none
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 401
text: Unauthorized
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}
example {
name: 403 Response
description: Token is valid but the caller is not authorized for this resource.
request: {
url: {{baseUrl}}/admin/webhooks
method: GET
mode: none
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 403
text: Forbidden
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}

View File

@@ -0,0 +1,7 @@
meta {
name: admin
}
auth {
mode: inherit
}

View File

@@ -0,0 +1,84 @@
meta {
name: GitHub OAuth proxy
type: http
seq: 1
}
get {
url: {{baseUrl}}/auth/github
body: none
auth: inherit
}
params:query {
code:
~type:
}
example {
name: 200 Response
description: GitHub access-token response (shape determined by GitHub).
request: {
url: {{baseUrl}}/auth/github
method: GET
mode: none
params:query: {
code:
~type:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 200
text: OK
}
body: {
type: json
content: '''
{}
'''
}
}
}
example {
name: 400 Response
description: Malformed input.
request: {
url: {{baseUrl}}/auth/github
method: GET
mode: none
params:query: {
code:
~type:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 400
text: Bad Request
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}

View File

@@ -0,0 +1,7 @@
meta {
name: auth
}
auth {
mode: inherit
}

View File

@@ -0,0 +1,9 @@
{
"version": "1",
"name": "Remanso Jetstream API",
"type": "collection",
"ignore": [
"node_modules",
".git"
]
}

View File

@@ -0,0 +1,7 @@
meta {
name: Remanso Jetstream API
}
auth {
mode: none
}

View File

@@ -0,0 +1,3 @@
vars {
baseUrl: http://localhost:8080
}

View File

@@ -0,0 +1,3 @@
vars {
baseUrl: https://api.remanso.space
}

View File

@@ -0,0 +1,42 @@
meta {
name: Health probe
type: http
seq: 2
}
get {
url: {{baseUrl}}/health
body: none
auth: inherit
}
example {
name: 200 Response
description: Service is up
request: {
url: {{baseUrl}}/health
method: GET
mode: none
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 200
text: OK
}
body: {
type: json
content: '''
{
"status": ""
}
'''
}
}
}

View File

@@ -0,0 +1,40 @@
meta {
name: Hello world
type: http
seq: 1
}
get {
url: {{baseUrl}}/
body: none
auth: inherit
}
example {
name: 200 Response
description: Plain text greeting
request: {
url: {{baseUrl}}/
method: GET
mode: none
}
response: {
headers: {
Content-Type: text/plain
}
status: {
code: 200
text: OK
}
body: {
type: text
content: '''
Hello world
'''
}
}
}

View File

@@ -0,0 +1,7 @@
meta {
name: meta
}
auth {
mode: inherit
}

View File

@@ -0,0 +1,61 @@
meta {
name: List discoverable notes across all users
type: http
seq: 1
}
get {
url: {{baseUrl}}/notes
body: none
auth: inherit
}
params:query {
~cursor:
~limit:
}
example {
name: 200 Response
description: A page of notes ordered by rkey descending.
request: {
url: {{baseUrl}}/notes
method: GET
mode: none
params:query: {
~cursor:
~limit:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 200
text: OK
}
body: {
type: json
content: '''
{
"notes": [
{
"did": "",
"rkey": "",
"title": "",
"publishedAt": "",
"createdAt": "",
"language": ""
}
],
"cursor": ""
}
'''
}
}
}

View File

@@ -0,0 +1,69 @@
meta {
name: List discoverable notes for a single DID
type: http
seq: 2
}
get {
url: {{baseUrl}}/:did/notes
body: none
auth: inherit
}
params:query {
~cursor:
~limit:
}
params:path {
did:
}
example {
name: 200 Response
description: A page of notes for the given DID.
request: {
url: {{baseUrl}}/:did/notes
method: GET
mode: none
params:query: {
~cursor:
~limit:
}
params:path: {
did:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 200
text: OK
}
body: {
type: json
content: '''
{
"notes": [
{
"did": "",
"rkey": "",
"title": "",
"publishedAt": "",
"createdAt": "",
"language": ""
}
],
"cursor": ""
}
'''
}
}
}

View File

@@ -0,0 +1,105 @@
meta {
name: Multi-DID feed
type: http
seq: 3
}
post {
url: {{baseUrl}}/notes/feed
body: json
auth: inherit
}
body:json {
{
"dids": [],
"cursor": "",
"limit": 0
}
}
example {
name: 200 Response
description: A page of notes from the requested DIDs.
request: {
url: {{baseUrl}}/notes/feed
method: POST
mode: json
body:json: {
{
"dids": [],
"cursor": "",
"limit": 0
}
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 200
text: OK
}
body: {
type: json
content: '''
{
"notes": [
{
"did": "",
"rkey": "",
"title": "",
"publishedAt": "",
"createdAt": "",
"language": ""
}
],
"cursor": ""
}
'''
}
}
}
example {
name: 400 Response
description: Malformed input.
request: {
url: {{baseUrl}}/notes/feed
method: POST
mode: json
body:json: {
{
"dids": [],
"cursor": "",
"limit": 0
}
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 400
text: Bad Request
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}

View File

@@ -0,0 +1,7 @@
meta {
name: notes
}
auth {
mode: inherit
}

View File

@@ -0,0 +1,195 @@
meta {
name: Owner-scoped full-text fuzzy search
type: http
seq: 2
}
get {
url: {{baseUrl}}/:did/search
body: none
auth: bearer
}
params:query {
q:
~cursor:
~limit:
}
params:path {
did: did:plc:4m3kouplb7s7xozjd3whinvl
}
auth:bearer {
token: {{token}}
}
example {
name: 200 Response
description: A page of search hits for the caller's DID.
request: {
url: {{baseUrl}}/:did/search
method: GET
mode: none
params:query: {
q:
~cursor:
~limit:
}
params:path: {
did:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 200
text: OK
}
body: {
type: json
content: '''
{
"results": [
{
"did": "",
"rkey": "",
"title": "",
"snippet": "",
"score": 0,
"publishedAt": ""
}
],
"cursor": ""
}
'''
}
}
}
example {
name: 400 Response
description: Malformed input.
request: {
url: {{baseUrl}}/:did/search
method: GET
mode: none
params:query: {
q:
~cursor:
~limit:
}
params:path: {
did:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 400
text: Bad Request
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}
example {
name: 401 Response
description: Missing or invalid bearer token.
request: {
url: {{baseUrl}}/:did/search
method: GET
mode: none
params:query: {
q:
~cursor:
~limit:
}
params:path: {
did:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 401
text: Unauthorized
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}
example {
name: 403 Response
description: Token is valid but the caller is not authorized for this resource.
request: {
url: {{baseUrl}}/:did/search
method: GET
mode: none
params:query: {
q:
~cursor:
~limit:
}
params:path: {
did:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 403
text: Forbidden
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}

View File

@@ -0,0 +1,99 @@
meta {
name: Public full-text fuzzy search
type: http
seq: 1
}
get {
url: {{baseUrl}}/search?q=
body: none
auth: inherit
}
params:query {
q:
~cursor:
~limit:
}
example {
name: 200 Response
description: A page of search hits.
request: {
url: {{baseUrl}}/search
method: GET
mode: none
params:query: {
q:
~cursor:
~limit:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 200
text: OK
}
body: {
type: json
content: '''
{
"results": [
{
"did": "",
"rkey": "",
"title": "",
"snippet": "",
"score": 0,
"publishedAt": ""
}
],
"cursor": ""
}
'''
}
}
}
example {
name: 400 Response
description: Malformed input.
request: {
url: {{baseUrl}}/search
method: GET
mode: none
params:query: {
q:
~cursor:
~limit:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 400
text: Bad Request
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}

View File

@@ -0,0 +1,7 @@
meta {
name: search
}
auth {
mode: inherit
}

View File

@@ -0,0 +1,206 @@
meta {
name: Add one or more webhook subscriptions
type: http
seq: 2
}
post {
url: {{baseUrl}}/:did/webhooks
body: json
auth: bearer
}
params:path {
did:
}
auth:bearer {
token: {{token}}
}
body:json {
{
"method": "",
"url": "",
"token": "",
"verb": ""
}
}
example {
name: 201 Response
description: One subscription per requested verb.
request: {
url: {{baseUrl}}/:did/webhooks
method: POST
mode: json
params:path: {
did:
}
body:json: {
{
"method": "",
"url": "",
"token": "",
"verb": ""
}
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 201
text: Created
}
body: {
type: json
content: '''
[
{
"id": 0,
"did": "",
"method": "",
"url": "",
"verb": ""
}
]
'''
}
}
}
example {
name: 400 Response
description: Malformed input.
request: {
url: {{baseUrl}}/:did/webhooks
method: POST
mode: json
params:path: {
did:
}
body:json: {
{
"method": "",
"url": "",
"token": "",
"verb": ""
}
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 400
text: Bad Request
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}
example {
name: 401 Response
description: Missing or invalid bearer token.
request: {
url: {{baseUrl}}/:did/webhooks
method: POST
mode: json
params:path: {
did:
}
body:json: {
{
"method": "",
"url": "",
"token": "",
"verb": ""
}
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 401
text: Unauthorized
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}
example {
name: 403 Response
description: Token is valid but the caller is not authorized for this resource.
request: {
url: {{baseUrl}}/:did/webhooks
method: POST
mode: json
params:path: {
did:
}
body:json: {
{
"method": "",
"url": "",
"token": "",
"verb": ""
}
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 403
text: Forbidden
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}

View File

@@ -0,0 +1,189 @@
meta {
name: Delete a single webhook subscription by id
type: http
seq: 4
}
delete {
url: {{baseUrl}}/:did/webhooks/:id
body: none
auth: bearer
}
params:path {
did:
id:
}
auth:bearer {
token: {{token}}
}
example {
name: 204 Response
description: Subscription deleted.
request: {
url: {{baseUrl}}/:did/webhooks/:id
method: DELETE
mode: none
params:path: {
did:
id:
}
}
response: {
status: {
code: 204
text: No Content
}
body: {
type: text
content: '''
'''
}
}
}
example {
name: 400 Response
description: Malformed input.
request: {
url: {{baseUrl}}/:did/webhooks/:id
method: DELETE
mode: none
params:path: {
did:
id:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 400
text: Bad Request
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}
example {
name: 401 Response
description: Missing or invalid bearer token.
request: {
url: {{baseUrl}}/:did/webhooks/:id
method: DELETE
mode: none
params:path: {
did:
id:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 401
text: Unauthorized
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}
example {
name: 403 Response
description: Token is valid but the caller is not authorized for this resource.
request: {
url: {{baseUrl}}/:did/webhooks/:id
method: DELETE
mode: none
params:path: {
did:
id:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 403
text: Forbidden
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}
example {
name: 404 Response
description: No subscription with that id owned by this DID.
request: {
url: {{baseUrl}}/:did/webhooks/:id
method: DELETE
mode: none
params:path: {
did:
id:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 404
text: Not Found
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}

View File

@@ -0,0 +1,115 @@
meta {
name: Delete every webhook subscription for the caller
type: http
seq: 3
}
delete {
url: {{baseUrl}}/:did/webhooks
body: none
auth: bearer
}
params:path {
did:
}
auth:bearer {
token: {{token}}
}
example {
name: 204 Response
description: All subscriptions deleted.
request: {
url: {{baseUrl}}/:did/webhooks
method: DELETE
mode: none
params:path: {
did:
}
}
response: {
status: {
code: 204
text: No Content
}
body: {
type: text
content: '''
'''
}
}
}
example {
name: 401 Response
description: Missing or invalid bearer token.
request: {
url: {{baseUrl}}/:did/webhooks
method: DELETE
mode: none
params:path: {
did:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 401
text: Unauthorized
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}
example {
name: 403 Response
description: Token is valid but the caller is not authorized for this resource.
request: {
url: {{baseUrl}}/:did/webhooks
method: DELETE
mode: none
params:path: {
did:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 403
text: Forbidden
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}

View File

@@ -0,0 +1,127 @@
meta {
name: List a DID's webhook subscriptions
type: http
seq: 1
}
get {
url: {{baseUrl}}/:did/webhooks
body: none
auth: bearer
}
params:path {
did: did:plc:4m3kouplb7s7xozjd3whinvl
}
auth:bearer {
token: {{token}}
}
example {
name: 200 Response
description: All subscriptions owned by the caller.
request: {
url: {{baseUrl}}/:did/webhooks
method: GET
mode: none
params:path: {
did:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 200
text: OK
}
body: {
type: json
content: '''
[
{
"id": 0,
"did": "",
"method": "",
"url": "",
"verb": ""
}
]
'''
}
}
}
example {
name: 401 Response
description: Missing or invalid bearer token.
request: {
url: {{baseUrl}}/:did/webhooks
method: GET
mode: none
params:path: {
did:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 401
text: Unauthorized
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}
example {
name: 403 Response
description: Token is valid but the caller is not authorized for this resource.
request: {
url: {{baseUrl}}/:did/webhooks
method: GET
mode: none
params:path: {
did:
}
}
response: {
headers: {
Content-Type: application/json
}
status: {
code: 403
text: Forbidden
}
body: {
type: json
content: '''
{
"error": ""
}
'''
}
}
}

View File

@@ -0,0 +1,7 @@
meta {
name: webhooks
}
auth {
mode: inherit
}

View File

@@ -5,11 +5,12 @@ import {
getWebhooksByDidAndVerb,
saveCursor,
upsertNote,
type WebhookVerb,
} from "./src/data/db.ts";
import { Note } from "./src/data/note.ts";
import { log } from "./src/log.ts";
import { ensureIndex, indexNote, removeNote } from "./src/search/opensearch.ts";
import { dispatchAll, fireWebhooks } from "./src/jetstream/webhooks.ts";
import { type BulkDeps, queueBulkCreate } from "./src/jetstream/bulk.ts";
const safeIndex = async (note: Note): Promise<void> => {
try {
@@ -27,6 +28,12 @@ const safeRemove = async (did: string, rkey: string): Promise<void> => {
}
};
const bulkDeps: BulkDeps = {
getSubs: getWebhooksByDidAndVerb,
dispatch: dispatchAll,
};
if (import.meta.main) {
globalThis.addEventListener("unhandledrejection", (e) => {
log("[jetstream] unhandled rejection:", e.reason);
});
@@ -35,91 +42,6 @@ globalThis.addEventListener("error", (e) => {
log("[jetstream] uncaught error:", e.error);
});
type WebhookTarget = {
method: string;
url: string;
token?: string;
};
const dispatchAll = async (
webhooks: WebhookTarget[],
payload: Record<string, unknown>,
label: string,
): Promise<void> => {
if (webhooks.length === 0) return;
const results = await Promise.allSettled(
webhooks.map(({ method, url, token }) => {
const hasBody = method !== "GET" && method !== "HEAD";
return fetch(url, {
method,
headers: {
...(hasBody ? { "Content-Type": "application/json" } : {}),
...(token ? { Authorization: `Bearer ${token}` } : {}),
},
body: hasBody ? JSON.stringify(payload) : undefined,
});
}),
);
for (const result of results) {
if (result.status === "rejected") {
log(`[jetstream] ${label} webhook error:`, result.reason);
}
}
};
const fireWebhooks = async (
did: string,
verb: WebhookVerb,
payload: Record<string, unknown>,
): Promise<void> => {
const webhooks = getWebhooksByDidAndVerb(did, verb);
await dispatchAll(webhooks, payload, `${verb} ${did}`);
};
const BULK_CREATE_DEBOUNCE_MS = 500;
type BulkBuffer = {
records: Record<string, unknown>[];
timer: number;
};
const bulkBuffers = new Map<string, BulkBuffer>();
const flushBulkCreate = async (did: string): Promise<void> => {
const buffer = bulkBuffers.get(did);
if (!buffer) return;
bulkBuffers.delete(did);
const webhooks = getWebhooksByDidAndVerb(did, "bulk-create");
log(`[jetstream] bulk-create ${did}: ${buffer.records.length} record(s)`);
await dispatchAll(
webhooks,
{ event: "bulk-create", did, records: buffer.records },
`bulk-create ${did}`,
);
};
// Buffered records are not persisted: if jetstream restarts mid-window, those
// `bulk-create` notifications are lost. Subscribers reconcile on cold start
// because the underlying notes are already saved to the `note` table.
const queueBulkCreate = (
did: string,
record: Record<string, unknown>,
): void => {
const existing = bulkBuffers.get(did);
if (existing) {
clearTimeout(existing.timer);
existing.records.push(record);
existing.timer = setTimeout(
() => flushBulkCreate(did),
BULK_CREATE_DEBOUNCE_MS,
);
return;
}
bulkBuffers.set(did, {
records: [record],
timer: setTimeout(() => flushBulkCreate(did), BULK_CREATE_DEBOUNCE_MS),
});
};
const cursor = getCursor();
log(`[jetstream] starting with cursor: ${cursor ?? "none"}`);
@@ -140,10 +62,15 @@ jetstream.onCreate("space.remanso.note", async (event) => {
upsertNote({ did, rkey, ...note });
log(`[jetstream] create ${did}/${rkey}: ${note.title}`);
await Promise.allSettled([
fireWebhooks(did, "create", { event: "create", did, rkey, ...note }),
fireWebhooks(
did,
"create",
{ event: "create", did, rkey, ...note },
getWebhooksByDidAndVerb,
),
safeIndex({ did, rkey, ...note }),
]);
queueBulkCreate(did, { rkey, ...note });
queueBulkCreate(did, { rkey, ...note }, bulkDeps);
} catch (error) {
log(`[jetstream] error on create:`, error);
}
@@ -161,10 +88,15 @@ jetstream.onUpdate("space.remanso.note", async (event) => {
log(`[jetstream] update ${did}/${rkey}: ${note.title}`);
// Updates fold into the `create` verb — subscribers reconcile by (did, rkey).
await Promise.allSettled([
fireWebhooks(did, "create", { event: "create", did, rkey, ...note }),
fireWebhooks(
did,
"create",
{ event: "create", did, rkey, ...note },
getWebhooksByDidAndVerb,
),
safeIndex({ did, rkey, ...note }),
]);
queueBulkCreate(did, { rkey, ...note });
queueBulkCreate(did, { rkey, ...note }, bulkDeps);
} catch (error) {
log(`[jetstream] error on update:`, error);
}
@@ -180,7 +112,12 @@ jetstream.onDelete("space.remanso.note", async (event) => {
deleteNote({ did, rkey });
log(`[jetstream] delete ${did}/${rkey}`);
await Promise.allSettled([
fireWebhooks(did, "delete", { event: "delete", did, rkey }),
fireWebhooks(
did,
"delete",
{ event: "delete", did, rkey },
getWebhooksByDidAndVerb,
),
safeRemove(did, rkey),
]);
} catch (error) {
@@ -214,3 +151,4 @@ setInterval(() => {
saveCursor(jetstream.cursor);
}
}, 5000);
}

View File

@@ -9,7 +9,7 @@
// Requires OPENSEARCH_URL to be set; falls back to a no-op otherwise.
// SQLite path follows the same SQLITE_PATH convention as the server/jetstream.
import { db } from "../src/data/db.ts";
import { closeDb, listDistinctNoteDids } from "../src/data/db.ts";
import { resolvePds } from "../src/auth/verify.ts";
import { ensureIndex, indexNote } from "../src/search/opensearch.ts";
import { log } from "../src/log.ts";
@@ -100,14 +100,12 @@ const main = async () => {
Deno.exit(1);
}
const rows = db.prepare("SELECT DISTINCT did FROM note ORDER BY did").all<
{ did: string }
>();
log(`[backfill] ${rows.length} DID(s) to process`);
const dids = listDistinctNoteDids();
log(`[backfill] ${dids.length} DID(s) to process`);
let totalIndexed = 0;
let totalFailed = 0;
for (const { did } of rows) {
for (const did of dids) {
try {
const { indexed, failed } = await backfillDid(did);
log(`[backfill] ${did}: indexed=${indexed} failed=${failed}`);
@@ -119,7 +117,7 @@ const main = async () => {
}
}
log(`[backfill] done. indexed=${totalIndexed} failed=${totalFailed}`);
db.close();
closeDb();
};
await main();

View File

@@ -19,13 +19,25 @@ type AuthCtx = {
response: { status: number; body: unknown };
};
let authenticator: typeof authenticateRequest = authenticateRequest;
export const _setAuthenticatorForTest = (
fn: typeof authenticateRequest,
): void => {
authenticator = fn;
};
export const _resetAuthenticatorForTest = (): void => {
authenticator = authenticateRequest;
};
const requireDidOwnership = async (
ctx: AuthCtx,
did: string,
): Promise<boolean> => {
let verifiedDid: string;
try {
verifiedDid = await authenticateRequest(
verifiedDid = await authenticator(
ctx.request.headers.get("Authorization"),
);
} catch {
@@ -41,7 +53,8 @@ const requireDidOwnership = async (
return true;
};
const ADMIN_DIDS = new Set(
const adminDids = (): Set<string> =>
new Set(
(Deno.env.get("ADMIN_DIDS") ?? "")
.split(",")
.map((d) => d.trim())
@@ -51,7 +64,7 @@ const ADMIN_DIDS = new Set(
const requireAdmin = async (ctx: AuthCtx): Promise<boolean> => {
let verifiedDid: string;
try {
verifiedDid = await authenticateRequest(
verifiedDid = await authenticator(
ctx.request.headers.get("Authorization"),
);
} catch {
@@ -59,7 +72,7 @@ const requireAdmin = async (ctx: AuthCtx): Promise<boolean> => {
ctx.response.body = { error: "Unauthorized" };
return false;
}
if (!ADMIN_DIDS.has(verifiedDid)) {
if (!adminDids().has(verifiedDid)) {
ctx.response.status = 403;
ctx.response.body = { error: "Admin only" };
return false;
@@ -260,6 +273,7 @@ router.delete("/:did/webhooks/:id", async (ctx) => {
// ctx.response.status = 204;
// })
export const createApp = (): Application => {
const app = new Application();
app.use(async (ctx, next) => {
@@ -282,5 +296,11 @@ app.use(async (ctx, next) => {
app.use(router.routes());
app.use(router.allowedMethods());
return app;
};
if (import.meta.main) {
const app = createApp();
log("[server] listening on port 8080");
app.listen({ port: 8080 });
}

View File

@@ -7,7 +7,7 @@ interface DidDocument {
service?: { id: string; serviceEndpoint: string }[];
}
function decodeJwtPayload(token: string): JwtPayload {
export function decodeJwtPayload(token: string): JwtPayload {
const parts = token.split(".");
if (parts.length !== 3) throw new Error("Invalid JWT format");
const payload = parts[1].replace(/-/g, "+").replace(/_/g, "/");

View File

@@ -1,18 +1,39 @@
import { Database } from "@db/sqlite";
import type { Note } from "./note.ts";
export const db = new Database(Deno.env.get("SQLITE_PATH") ?? "notes.db");
let _db: Database | null = null;
const openDefaultDb = (): Database => {
const handle = new Database(Deno.env.get("SQLITE_PATH") ?? "notes.db");
try {
db.exec("PRAGMA busy_timeout=10000");
db.exec("PRAGMA journal_mode=WAL");
const [row] = db.prepare("PRAGMA journal_mode").all<
handle.exec("PRAGMA busy_timeout=10000");
handle.exec("PRAGMA journal_mode=WAL");
const [row] = handle.prepare("PRAGMA journal_mode").all<
{ journal_mode: string }
>();
console.log(`[db] journal_mode=${row.journal_mode}, busy_timeout=10000`);
} catch (e) {
console.error("[db] failed to set PRAGMAs:", e);
}
return handle;
};
const getDb = (): Database => _db ??= openDefaultDb();
export const _setDbForTest = (handle: Database): void => {
_db = handle;
};
export const _resetDbForTest = (): void => {
_db = null;
};
export const closeDb = (): void => {
if (_db) {
_db.close();
_db = null;
}
};
type NoteRow = {
did: string;
@@ -23,6 +44,7 @@ type NoteRow = {
};
export const getNotes = (cursor?: string, limit = 20) => {
const db = getDb();
const notes = cursor
? db.prepare(
"SELECT did, rkey, title, publishedAt, createdAt, language FROM note WHERE discoverable = 1 AND rkey < ? ORDER BY rkey DESC LIMIT ?",
@@ -38,6 +60,7 @@ export const getNotes = (cursor?: string, limit = 20) => {
};
export const getNotesByDid = (did: string, cursor?: string, limit = 20) => {
const db = getDb();
const notes = cursor
? db.prepare(
"SELECT did, rkey, title, publishedAt, createdAt, language FROM note WHERE discoverable = 1 AND did = ? AND rkey < ? ORDER BY rkey DESC LIMIT ?",
@@ -54,6 +77,7 @@ export const getNotesByDid = (did: string, cursor?: string, limit = 20) => {
export const getNotesByDids = (dids: string[], cursor?: string, limit = 20) => {
if (dids.length === 0) return { notes: [] };
const db = getDb();
const placeholders = dids.map(() => "?").join(", ");
const notes = cursor
? db.prepare(
@@ -70,18 +94,18 @@ export const getNotesByDids = (dids: string[], cursor?: string, limit = 20) => {
};
export const deleteNote = ({ did, rkey }: { did: string; rkey: string }) => {
db.exec("DELETE FROM note WHERE did = ? AND rkey = ?", did, rkey);
getDb().exec("DELETE FROM note WHERE did = ? AND rkey = ?", did, rkey);
};
export const getCursor = (): string | undefined => {
const row = db.prepare(
const row = getDb().prepare(
"SELECT value FROM state WHERE key = 'cursor'",
).get<{ value: string }>();
return row?.value;
};
export const saveCursor = (cursor: number) => {
db.exec(
getDb().exec(
"INSERT OR REPLACE INTO state (key, value) VALUES ('cursor', ?)",
String(cursor),
);
@@ -101,6 +125,7 @@ type WebhookSubscriptionRow = {
export const addWebhookSubscription = (
{ did, method, url, token, verb }: Omit<WebhookSubscriptionRow, "id">,
): WebhookSubscriptionRow => {
const db = getDb();
db.exec(
"INSERT INTO webhook_subscription (did, method, url, token, verb) VALUES (?, ?, ?, ?, ?)",
did,
@@ -116,13 +141,13 @@ export const addWebhookSubscription = (
};
export const deleteWebhooksByDid = (did: string): void => {
db.exec("DELETE FROM webhook_subscription WHERE did = ?", did);
getDb().exec("DELETE FROM webhook_subscription WHERE did = ?", did);
};
export const deleteWebhookById = (
{ did, id }: { did: string; id: number },
): boolean => {
const result = db.prepare(
const result = getDb().prepare(
"DELETE FROM webhook_subscription WHERE did = ? AND id = ?",
).run(did, id);
return result > 0;
@@ -131,13 +156,13 @@ export const deleteWebhookById = (
export const listWebhooksByDid = (
did: string,
): Omit<WebhookSubscriptionRow, "token">[] => {
return db.prepare(
return getDb().prepare(
"SELECT id, did, method, url, verb FROM webhook_subscription WHERE did = ? ORDER BY id DESC",
).all<Omit<WebhookSubscriptionRow, "token">>(did);
};
export const listAllWebhooks = (): Omit<WebhookSubscriptionRow, "token">[] => {
return db.prepare(
return getDb().prepare(
"SELECT id, did, method, url, verb FROM webhook_subscription ORDER BY did, id",
).all<Omit<WebhookSubscriptionRow, "token">>();
};
@@ -146,14 +171,20 @@ export const getWebhooksByDidAndVerb = (
did: string,
verb: WebhookVerb,
): WebhookSubscriptionRow[] => {
return db.prepare(
return getDb().prepare(
"SELECT id, did, method, url, token, verb FROM webhook_subscription WHERE did = ? AND verb = ? ORDER BY id DESC LIMIT 10",
).all<WebhookSubscriptionRow>(did, verb);
};
export const listDistinctNoteDids = (): string[] => {
return getDb().prepare(
"SELECT DISTINCT did FROM note ORDER BY did",
).all<{ did: string }>().map((row) => row.did);
};
export const upsertNote = (note: Note) => {
const now = new Date().toISOString();
db.exec(
getDb().exec(
`
INSERT INTO note (
title,

66
src/jetstream/bulk.ts Normal file
View File

@@ -0,0 +1,66 @@
import { log } from "../log.ts";
import type { GetSubsByVerb, WebhookTarget } from "./webhooks.ts";
export const BULK_CREATE_DEBOUNCE_MS = 500;
export type BulkDeps = {
getSubs: GetSubsByVerb;
dispatch: (
webhooks: WebhookTarget[],
payload: Record<string, unknown>,
label: string,
) => Promise<void>;
debounceMs?: number;
};
type BulkBuffer = {
records: Record<string, unknown>[];
timer: ReturnType<typeof setTimeout>;
};
const bulkBuffers = new Map<string, BulkBuffer>();
export const flushBulkCreate = async (
did: string,
deps: BulkDeps,
): Promise<void> => {
const buffer = bulkBuffers.get(did);
if (!buffer) return;
bulkBuffers.delete(did);
const webhooks = deps.getSubs(did, "bulk-create");
log(`[jetstream] bulk-create ${did}: ${buffer.records.length} record(s)`);
await deps.dispatch(
webhooks,
{ event: "bulk-create", did, records: buffer.records },
`bulk-create ${did}`,
);
};
// Buffered records are not persisted: if jetstream restarts mid-window, those
// `bulk-create` notifications are lost. Subscribers reconcile on cold start
// because the underlying notes are already saved to the `note` table.
export const queueBulkCreate = (
did: string,
record: Record<string, unknown>,
deps: BulkDeps,
): void => {
const ms = deps.debounceMs ?? BULK_CREATE_DEBOUNCE_MS;
const existing = bulkBuffers.get(did);
if (existing) {
clearTimeout(existing.timer);
existing.records.push(record);
existing.timer = setTimeout(() => flushBulkCreate(did, deps), ms);
return;
}
bulkBuffers.set(did, {
records: [record],
timer: setTimeout(() => flushBulkCreate(did, deps), ms),
});
};
export const _resetBulkBuffersForTest = (): void => {
for (const buffer of bulkBuffers.values()) {
clearTimeout(buffer.timer);
}
bulkBuffers.clear();
};

49
src/jetstream/webhooks.ts Normal file
View File

@@ -0,0 +1,49 @@
import type { WebhookVerb } from "../data/db.ts";
import { log } from "../log.ts";
export type WebhookTarget = {
method: string;
url: string;
token?: string;
};
export const dispatchAll = async (
webhooks: WebhookTarget[],
payload: Record<string, unknown>,
label: string,
): Promise<void> => {
if (webhooks.length === 0) return;
const results = await Promise.allSettled(
webhooks.map(({ method, url, token }) => {
const hasBody = method !== "GET" && method !== "HEAD";
return fetch(url, {
method,
headers: {
...(hasBody ? { "Content-Type": "application/json" } : {}),
...(token ? { Authorization: `Bearer ${token}` } : {}),
},
body: hasBody ? JSON.stringify(payload) : undefined,
});
}),
);
for (const result of results) {
if (result.status === "rejected") {
log(`[jetstream] ${label} webhook error:`, result.reason);
}
}
};
export type GetSubsByVerb = (
did: string,
verb: WebhookVerb,
) => WebhookTarget[];
export const fireWebhooks = async (
did: string,
verb: WebhookVerb,
payload: Record<string, unknown>,
getSubs: GetSubsByVerb,
): Promise<void> => {
const webhooks = getSubs(did, verb);
await dispatchAll(webhooks, payload, `${verb} ${did}`);
};

83
src/migrations/apply.ts Normal file
View File

@@ -0,0 +1,83 @@
import type { Database } from "@db/sqlite";
export const applyMigrations = (db: Database): void => {
db.exec(`
CREATE TABLE IF NOT EXISTS note (
title TEXT NOT NULL,
publishedAt DATETIME,
createdAt DATETIME NOT NULL,
did TEXT NOT NULL,
rkey TEXT NOT NULL,
discoverable INTEGER NOT NULL DEFAULT 1,
PRIMARY KEY (did, rkey)
);
`);
try {
db.exec(
`ALTER TABLE note ADD COLUMN listed INTEGER NOT NULL DEFAULT 1;`,
);
} catch {
// Column already exists — no-op
}
try {
db.exec(
`ALTER TABLE note ADD COLUMN language STRING;`,
);
} catch {
// Column already exists — no-op
}
try {
db.exec(`ALTER TABLE note RENAME COLUMN listed TO discoverable;`);
} catch {
// Column already renamed or doesn't exist — no-op
}
db.exec(`
CREATE TABLE IF NOT EXISTS state (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
`);
db.exec(`
CREATE TABLE IF NOT EXISTS webhook_subscription (
id INTEGER PRIMARY KEY,
did TEXT NOT NULL,
method TEXT NOT NULL,
url TEXT NOT NULL
);
`);
db.exec(`
CREATE INDEX IF NOT EXISTS idx_webhook_subscription_did
ON webhook_subscription(did);
`);
try {
db.exec(`ALTER TABLE webhook_subscription ADD COLUMN token TEXT;`);
} catch {
// Column already exists — no-op
}
try {
db.exec(
`ALTER TABLE webhook_subscription ADD COLUMN verb TEXT NOT NULL DEFAULT 'create';`,
);
db.exec(`
INSERT INTO webhook_subscription (did, method, url, token, verb)
SELECT did, method, url, token, 'delete'
FROM webhook_subscription
WHERE verb = 'create';
`);
} catch {
// Column already exists — backfill already happened on a previous run.
}
db.exec(`
CREATE INDEX IF NOT EXISTS idx_webhook_subscription_did_verb
ON webhook_subscription(did, verb);
`);
};

View File

@@ -1,83 +1,6 @@
import { db } from "../data/db.ts";
db.exec(`
CREATE TABLE IF NOT EXISTS note (
title TEXT NOT NULL,
publishedAt DATETIME,
createdAt DATETIME NOT NULL,
did TEXT NOT NULL,
rkey TEXT NOT NULL,
discoverable INTEGER NOT NULL DEFAULT 1,
PRIMARY KEY (did, rkey)
);
`);
try {
db.exec(
`ALTER TABLE note ADD COLUMN listed INTEGER NOT NULL DEFAULT 1;`,
);
} catch {
// Column already exists — no-op
}
try {
db.exec(
`ALTER TABLE note ADD COLUMN language STRING;`,
);
} catch {
// Column already exists — no-op
}
try {
db.exec(`ALTER TABLE note RENAME COLUMN listed TO discoverable;`);
} catch {
// Column already renamed or doesn't exist — no-op
}
db.exec(`
CREATE TABLE IF NOT EXISTS state (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
`);
db.exec(`
CREATE TABLE IF NOT EXISTS webhook_subscription (
id INTEGER PRIMARY KEY,
did TEXT NOT NULL,
method TEXT NOT NULL,
url TEXT NOT NULL
);
`);
db.exec(`
CREATE INDEX IF NOT EXISTS idx_webhook_subscription_did
ON webhook_subscription(did);
`);
try {
db.exec(`ALTER TABLE webhook_subscription ADD COLUMN token TEXT;`);
} catch {
// Column already exists — no-op
}
try {
db.exec(
`ALTER TABLE webhook_subscription ADD COLUMN verb TEXT NOT NULL DEFAULT 'create';`,
);
db.exec(`
INSERT INTO webhook_subscription (did, method, url, token, verb)
SELECT did, method, url, token, 'delete'
FROM webhook_subscription
WHERE verb = 'create';
`);
} catch {
// Column already exists — backfill already happened on a previous run.
}
db.exec(`
CREATE INDEX IF NOT EXISTS idx_webhook_subscription_did_verb
ON webhook_subscription(did, verb);
`);
import { Database } from "@db/sqlite";
import { applyMigrations } from "./apply.ts";
const db = new Database(Deno.env.get("SQLITE_PATH") ?? "notes.db");
applyMigrations(db);
db.close();

View File

@@ -1,20 +1,19 @@
import type { API } from "@opensearch-project/opensearch";
import { log } from "../log.ts";
const OPENSEARCH_URL = Deno.env.get("OPENSEARCH_URL");
const OPENSEARCH_USERNAME = Deno.env.get("OPENSEARCH_USERNAME");
const OPENSEARCH_PASSWORD = Deno.env.get("OPENSEARCH_PASSWORD");
const OPENSEARCH_INDEX = Deno.env.get("OPENSEARCH_INDEX") ?? "notes";
const MAX_LIMIT = 100;
const enabled = (): boolean => {
if (!OPENSEARCH_URL) return false;
return true;
};
const openSearchUrl = (): string | undefined => Deno.env.get("OPENSEARCH_URL");
const openSearchIndex = (): string =>
Deno.env.get("OPENSEARCH_INDEX") ?? "notes";
const enabled = (): boolean => Boolean(openSearchUrl());
const authHeaders = (): Record<string, string> => {
if (OPENSEARCH_USERNAME && OPENSEARCH_PASSWORD) {
const creds = btoa(`${OPENSEARCH_USERNAME}:${OPENSEARCH_PASSWORD}`);
const username = Deno.env.get("OPENSEARCH_USERNAME");
const password = Deno.env.get("OPENSEARCH_PASSWORD");
if (username && password) {
const creds = btoa(`${username}:${password}`);
return { Authorization: `Basic ${creds}` };
}
return {};
@@ -28,7 +27,7 @@ const request = async (
path: string,
body?: unknown,
): Promise<Response> => {
const url = `${OPENSEARCH_URL}${path}`;
const url = `${openSearchUrl()}${path}`;
const headers: Record<string, string> = { ...authHeaders() };
if (body !== undefined) headers["Content-Type"] = "application/json";
return await fetch(url, {
@@ -58,11 +57,12 @@ const indexMappings = {
export const ensureIndex = async (): Promise<void> => {
if (!enabled()) return;
const head = await request("HEAD", `/${OPENSEARCH_INDEX}`);
const index = openSearchIndex();
const head = await request("HEAD", `/${index}`);
if (head.status === 200) return;
const res = await request("PUT", `/${OPENSEARCH_INDEX}`, indexMappings);
const res = await request("PUT", `/${index}`, indexMappings);
if (res.ok) {
log(`[opensearch] created index ${OPENSEARCH_INDEX}`);
log(`[opensearch] created index ${index}`);
return;
}
const text = await res.text();
@@ -95,7 +95,7 @@ export const indexNote = async (note: IndexedNote): Promise<void> => {
};
const res = await request(
"PUT",
`/${OPENSEARCH_INDEX}/_doc/${docId(note.did, note.rkey)}`,
`/${openSearchIndex()}/_doc/${docId(note.did, note.rkey)}`,
doc,
);
if (!res.ok) {
@@ -113,7 +113,7 @@ export const removeNote = async (
if (!enabled()) return;
const res = await request(
"DELETE",
`/${OPENSEARCH_INDEX}/_doc/${docId(did, rkey)}`,
`/${openSearchIndex()}/_doc/${docId(did, rkey)}`,
);
if (!res.ok && res.status !== 404) {
const text = await res.text();
@@ -138,7 +138,7 @@ export type SearchOptions = {
limit: number;
};
const decodeCursor = (cursor: string): [number, string] | undefined => {
export const decodeCursor = (cursor: string): [number, string] | undefined => {
try {
const parsed = JSON.parse(atob(cursor));
if (
@@ -156,10 +156,10 @@ const decodeCursor = (cursor: string): [number, string] | undefined => {
return undefined;
};
const encodeCursor = (score: number, rkey: string): string =>
export const encodeCursor = (score: number, rkey: string): string =>
btoa(JSON.stringify([score, rkey]));
const firstSnippet = (
export const firstSnippet = (
highlight: Record<string, string[]> | undefined,
source: { content?: string; title?: string },
): string => {
@@ -185,7 +185,9 @@ export const searchNotes = async (
filters.push({ term: { did: opts.did } });
}
const body: Record<string, unknown> = {
const searchAfter = opts.cursor ? decodeCursor(opts.cursor) : undefined;
const body: API.Search_RequestBody = {
size: limit,
query: {
bool: {
@@ -207,21 +209,16 @@ export const searchNotes = async (
fragment_size: 200,
number_of_fragments: 1,
boundary_scanner: "sentence",
boundary_scanner_locale: "en-US",
},
title: { number_of_fragments: 0 },
},
},
sort: [{ _score: "desc" }, { rkey: "desc" }],
_source: ["did", "rkey", "title", "content", "publishedAt"],
...(searchAfter ? { search_after: searchAfter } : {}),
};
if (opts.cursor) {
const decoded = decodeCursor(opts.cursor);
if (decoded) body.search_after = decoded;
}
const res = await request("POST", `/${OPENSEARCH_INDEX}/_search`, body);
const res = await request("POST", `/${openSearchIndex()}/_search`, body);
if (!res.ok) {
const text = await res.text();
throw new Error(`Search failed: ${res.status} ${text}`);

49
tests/_helpers/app.ts Normal file
View File

@@ -0,0 +1,49 @@
import type { Application } from "@oak/oak";
export type RequestOptions = {
method?: string;
path: string;
headers?: HeadersInit;
body?: unknown;
};
export type TestResponse = {
status: number;
headers: Headers;
text: string;
json: () => unknown;
};
export const requestApp = async (
app: Application,
opts: RequestOptions,
): Promise<TestResponse> => {
const method = opts.method ?? "GET";
const url = `http://test.local${opts.path}`;
const init: RequestInit = {
method,
headers: opts.headers,
};
if (opts.body !== undefined && method !== "GET" && method !== "HEAD") {
init.body = typeof opts.body === "string"
? opts.body
: JSON.stringify(opts.body);
const headers = new Headers(opts.headers);
if (!headers.has("Content-Type")) {
headers.set("Content-Type", "application/json");
}
init.headers = headers;
}
const req = new Request(url, init);
const res = await app.handle(req);
if (!res) {
throw new Error("app.handle returned undefined");
}
const text = await res.text();
return {
status: res.status,
headers: res.headers,
text,
json: () => JSON.parse(text),
};
};

18
tests/_helpers/auth.ts Normal file
View File

@@ -0,0 +1,18 @@
import {
_resetAuthenticatorForTest,
_setAuthenticatorForTest,
} from "../../server.ts";
export type AuthStub = { restore: () => void };
export const stubAuthenticate = (didOrError: string | Error): AuthStub => {
_setAuthenticatorForTest(() => {
if (didOrError instanceof Error) {
return Promise.reject(didOrError);
}
return Promise.resolve(didOrError);
});
return {
restore: () => _resetAuthenticatorForTest(),
};
};

60
tests/_helpers/db.ts Normal file
View File

@@ -0,0 +1,60 @@
import { Database } from "@db/sqlite";
import {
_resetDbForTest,
_setDbForTest,
addWebhookSubscription,
upsertNote,
type WebhookVerb,
} from "../../src/data/db.ts";
import { applyMigrations } from "../../src/migrations/apply.ts";
import type { Note } from "../../src/data/note.ts";
export const withTestDb = (
fn: (db: Database) => Promise<void> | void,
): () => Promise<void> => {
return async () => {
const db = new Database(":memory:");
applyMigrations(db);
_setDbForTest(db);
try {
await fn(db);
} finally {
_resetDbForTest();
db.close();
}
};
};
let seedCounter = 0;
export const seedNote = (overrides: Partial<Note> = {}): Note => {
seedCounter++;
const note: Note = {
did: overrides.did ?? "did:plc:test",
rkey: overrides.rkey ?? `seed${seedCounter.toString().padStart(6, "0")}`,
title: overrides.title ?? "Test title",
publishedAt: overrides.publishedAt ?? "2026-01-01T00:00:00.000Z",
createdAt: overrides.createdAt ?? "2026-01-01T00:00:00.000Z",
discoverable: overrides.discoverable,
language: overrides.language,
content: overrides.content,
};
upsertNote(note);
return note;
};
export const seedWebhook = (overrides: {
did?: string;
method?: string;
url?: string;
token?: string;
verb?: WebhookVerb;
} = {}) => {
return addWebhookSubscription({
did: overrides.did ?? "did:plc:test",
method: overrides.method ?? "POST",
url: overrides.url ?? "https://example.com/hook",
token: overrides.token,
verb: overrides.verb ?? "create",
});
};

View File

@@ -0,0 +1,57 @@
export type FetchHandler = (
req: Request,
) => Response | Promise<Response>;
export type CapturedRequest = {
method: string;
url: string;
headers: Headers;
body: string;
};
export type FetchStub = {
calls: CapturedRequest[];
restore: () => void;
};
export const installFetchStub = (handler: FetchHandler): FetchStub => {
const original = globalThis.fetch;
const calls: CapturedRequest[] = [];
let restored = false;
globalThis.fetch = async (input, init) => {
const req = new Request(input as RequestInfo, init);
const body = req.method === "GET" || req.method === "HEAD"
? ""
: await req.clone().text();
calls.push({
method: req.method,
url: req.url,
headers: new Headers(req.headers),
body,
});
return await handler(req);
};
return {
calls,
restore: () => {
if (restored) throw new Error("FetchStub.restore() called twice");
restored = true;
globalThis.fetch = original;
},
};
};
export const jsonResponse = (
body: unknown,
init: ResponseInit = {},
): Response => {
return new Response(JSON.stringify(body), {
status: init.status ?? 200,
headers: { "Content-Type": "application/json", ...(init.headers ?? {}) },
});
};
export const errorResponse = (status: number, text = "error"): Response =>
new Response(text, { status });

118
tests/auth_verify_test.ts Normal file
View File

@@ -0,0 +1,118 @@
import { assertEquals, assertRejects, assertThrows } from "@std/assert";
import { authenticateRequest, decodeJwtPayload } from "../src/auth/verify.ts";
import { installFetchStub, jsonResponse } from "./_helpers/fetch_stub.ts";
const makeJwt = (payload: Record<string, unknown>): string => {
const header = btoa(JSON.stringify({ alg: "HS256", typ: "JWT" }));
const body = btoa(JSON.stringify(payload));
return `${header}.${body}.sig`;
};
Deno.test("decodeJwtPayload parses sub from a 3-segment JWT", () => {
const token = makeJwt({ sub: "did:plc:abc", iat: 1 });
const payload = decodeJwtPayload(token);
assertEquals(payload.sub, "did:plc:abc");
});
Deno.test("decodeJwtPayload throws on malformed token", () => {
assertThrows(() => decodeJwtPayload("not.a.valid.jwt"));
assertThrows(() => decodeJwtPayload("onlytwo.parts"));
});
Deno.test("authenticateRequest rejects missing or non-Bearer Authorization", async () => {
await assertRejects(() => authenticateRequest(null));
await assertRejects(() => authenticateRequest(""));
await assertRejects(() => authenticateRequest("Basic abc"));
});
Deno.test(
"authenticateRequest resolves PDS then verifies session and returns did",
async () => {
const did = "did:plc:user";
const stub = installFetchStub((req) => {
if (req.url === `https://plc.directory/${did}`) {
return jsonResponse({
service: [{
id: "#atproto_pds",
serviceEndpoint: "https://pds.example.com",
}],
});
}
if (
req.url ===
"https://pds.example.com/xrpc/com.atproto.server.getSession"
) {
return jsonResponse({ did });
}
return jsonResponse({ error: "unexpected" }, { status: 500 });
});
try {
const result = await authenticateRequest(
`Bearer ${makeJwt({ sub: did })}`,
);
assertEquals(result, did);
const sessionCall = stub.calls.find((c) => c.url.endsWith("getSession"));
assertEquals(
sessionCall?.headers.get("Authorization")?.startsWith("Bearer "),
true,
);
} finally {
stub.restore();
}
},
);
Deno.test(
"authenticateRequest throws when plc.directory or getSession is non-2xx",
async () => {
const did = "did:plc:user";
const plcFailStub = installFetchStub(() =>
new Response("not found", { status: 404 })
);
try {
await assertRejects(() =>
authenticateRequest(`Bearer ${makeJwt({ sub: did })}`)
);
} finally {
plcFailStub.restore();
}
const sessionFailStub = installFetchStub((req) => {
if (req.url.startsWith("https://plc.directory/")) {
return jsonResponse({
service: [{
id: "#atproto_pds",
serviceEndpoint: "https://pds.example.com",
}],
});
}
return new Response("nope", { status: 401 });
});
try {
await assertRejects(() =>
authenticateRequest(`Bearer ${makeJwt({ sub: did })}`)
);
} finally {
sessionFailStub.restore();
}
},
);
Deno.test(
"authenticateRequest throws when DID document has no atproto_pds service",
async () => {
const did = "did:plc:user";
const stub = installFetchStub(() =>
jsonResponse({ service: [{ id: "#other", serviceEndpoint: "x" }] })
);
try {
await assertRejects(() =>
authenticateRequest(`Bearer ${makeJwt({ sub: did })}`)
);
} finally {
stub.restore();
}
},
);

141
tests/db_test.ts Normal file
View File

@@ -0,0 +1,141 @@
import { assertEquals, assertFalse } from "@std/assert";
import {
addWebhookSubscription,
deleteWebhookById,
getNotes,
getNotesByDids,
getWebhooksByDidAndVerb,
listWebhooksByDid,
upsertNote,
} from "../src/data/db.ts";
import { seedNote, seedWebhook, withTestDb } from "./_helpers/db.ts";
Deno.test(
"upsertNote inserts then updates on (did, rkey) conflict",
withTestDb(() => {
upsertNote({
did: "did:plc:user",
rkey: "rkey1",
title: "First",
publishedAt: "2026-01-01T00:00:00.000Z",
createdAt: "2026-01-01T00:00:00.000Z",
});
upsertNote({
did: "did:plc:user",
rkey: "rkey1",
title: "Second",
publishedAt: "2026-02-01T00:00:00.000Z",
createdAt: "2026-01-01T00:00:00.000Z",
});
const { notes } = getNotes(undefined, 10);
assertEquals(notes.length, 1);
assertEquals(notes[0].title, "Second");
assertEquals(notes[0].publishedAt, "2026-02-01T00:00:00.000Z");
}),
);
Deno.test(
"upsertNote treats undefined discoverable as 1 and false as 0",
withTestDb(() => {
upsertNote({
did: "did:plc:user",
rkey: "default",
title: "Default",
publishedAt: "2026-01-01T00:00:00.000Z",
createdAt: "2026-01-01T00:00:00.000Z",
});
upsertNote({
did: "did:plc:user",
rkey: "hidden",
title: "Hidden",
publishedAt: "2026-01-01T00:00:00.000Z",
createdAt: "2026-01-01T00:00:00.000Z",
discoverable: false,
});
const { notes } = getNotes(undefined, 10);
const rkeys = notes.map((n) => n.rkey);
assertEquals(rkeys, ["default"]);
}),
);
Deno.test(
"getNotes paginates rkey desc and returns cursor only when page is full",
withTestDb(() => {
for (const rkey of ["a", "b", "c", "d", "e"]) {
seedNote({ rkey });
}
const firstPage = getNotes(undefined, 3);
assertEquals(firstPage.notes.map((n) => n.rkey), ["e", "d", "c"]);
assertEquals(firstPage.cursor, "c");
const secondPage = getNotes(firstPage.cursor, 3);
assertEquals(secondPage.notes.map((n) => n.rkey), ["b", "a"]);
assertEquals(secondPage.cursor, undefined);
}),
);
Deno.test(
"getNotesByDids returns empty without running SQL on empty did list",
withTestDb(() => {
seedNote({ did: "did:plc:alice", rkey: "a" });
const result = getNotesByDids([], undefined, 10);
assertEquals(result, { notes: [] });
}),
);
Deno.test(
"webhook CRUD: add omits token, list orders desc, delete by id reports hit",
withTestDb(() => {
const created = addWebhookSubscription({
did: "did:plc:user",
method: "POST",
url: "https://example.com/a",
token: "secret",
verb: "create",
});
assertEquals(created.token, undefined);
assertEquals(created.url, "https://example.com/a");
seedWebhook({
did: "did:plc:user",
url: "https://example.com/b",
verb: "create",
});
seedWebhook({
did: "did:plc:user",
url: "https://example.com/c",
verb: "delete",
});
const list = listWebhooksByDid("did:plc:user");
assertEquals(list.length, 3);
const ids = list.map((w) => w.id);
assertEquals(ids, [...ids].sort((a, b) => b - a));
assertFalse(deleteWebhookById({ did: "did:plc:user", id: 9999 }));
const deleted = deleteWebhookById({ did: "did:plc:user", id: created.id });
assertEquals(deleted, true);
assertEquals(listWebhooksByDid("did:plc:user").length, 2);
}),
);
Deno.test(
"getWebhooksByDidAndVerb filters by verb and caps at 10",
withTestDb(() => {
for (let i = 0; i < 12; i++) {
seedWebhook({ url: `https://example.com/c${i}`, verb: "create" });
}
seedWebhook({ url: "https://example.com/d", verb: "delete" });
const creates = getWebhooksByDidAndVerb("did:plc:test", "create");
assertEquals(creates.length, 10);
for (const row of creates) assertEquals(row.verb, "create");
const deletes = getWebhooksByDidAndVerb("did:plc:test", "delete");
assertEquals(deletes.length, 1);
assertEquals(deletes[0].verb, "delete");
}),
);

View File

@@ -0,0 +1,147 @@
import { assertEquals } from "@std/assert";
import { FakeTime } from "@std/testing/time";
import {
_resetBulkBuffersForTest,
type BulkDeps,
queueBulkCreate,
} from "../src/jetstream/bulk.ts";
import type { WebhookTarget } from "../src/jetstream/webhooks.ts";
import type { WebhookVerb } from "../src/data/db.ts";
type DispatchCall = {
webhooks: WebhookTarget[];
payload: Record<string, unknown>;
label: string;
};
const flushMicrotasks = async (): Promise<void> => {
for (let i = 0; i < 5; i++) {
await Promise.resolve();
}
};
const makeDeps = (
webhooks: WebhookTarget[] = [],
): {
deps: BulkDeps;
getSubsCalls: { did: string; verb: WebhookVerb }[];
dispatchCalls: DispatchCall[];
} => {
const getSubsCalls: { did: string; verb: WebhookVerb }[] = [];
const dispatchCalls: DispatchCall[] = [];
const deps: BulkDeps = {
getSubs: (did, verb) => {
getSubsCalls.push({ did, verb });
return webhooks;
},
dispatch: (w, payload, label) => {
dispatchCalls.push({ webhooks: w, payload, label });
return Promise.resolve();
},
debounceMs: 500,
};
return { deps, getSubsCalls, dispatchCalls };
};
Deno.test("queueBulkCreate accumulates records and flushes once after debounce", async () => {
const time = new FakeTime();
try {
_resetBulkBuffersForTest();
const { deps, dispatchCalls } = makeDeps();
queueBulkCreate("did:plc:a", { rkey: "r1" }, deps);
queueBulkCreate("did:plc:a", { rkey: "r2" }, deps);
queueBulkCreate("did:plc:a", { rkey: "r3" }, deps);
assertEquals(dispatchCalls.length, 0);
await time.tickAsync(500);
await flushMicrotasks();
assertEquals(dispatchCalls.length, 1);
const payload = dispatchCalls[0].payload as {
records: { rkey: string }[];
};
assertEquals(payload.records.map((r) => r.rkey), ["r1", "r2", "r3"]);
} finally {
_resetBulkBuffersForTest();
time.restore();
}
});
Deno.test("queueBulkCreate restarts the timer on each push (debounce)", async () => {
const time = new FakeTime();
try {
_resetBulkBuffersForTest();
const { deps, dispatchCalls } = makeDeps();
queueBulkCreate("did:plc:a", { rkey: "r1" }, deps);
await time.tickAsync(400);
queueBulkCreate("did:plc:a", { rkey: "r2" }, deps);
await time.tickAsync(400);
assertEquals(dispatchCalls.length, 0);
await time.tickAsync(100);
await flushMicrotasks();
assertEquals(dispatchCalls.length, 1);
} finally {
_resetBulkBuffersForTest();
time.restore();
}
});
Deno.test("flushBulkCreate uses bulk-create verb and the standard payload shape", async () => {
const time = new FakeTime();
try {
_resetBulkBuffersForTest();
const targets: WebhookTarget[] = [
{ method: "POST", url: "https://hook.example/bulk" },
];
const { deps, getSubsCalls, dispatchCalls } = makeDeps(targets);
queueBulkCreate("did:plc:a", { rkey: "r1", title: "T" }, deps);
await time.tickAsync(500);
await flushMicrotasks();
assertEquals(getSubsCalls, [{ did: "did:plc:a", verb: "bulk-create" }]);
assertEquals(dispatchCalls.length, 1);
assertEquals(dispatchCalls[0].webhooks, targets);
assertEquals(dispatchCalls[0].label, "bulk-create did:plc:a");
assertEquals(dispatchCalls[0].payload, {
event: "bulk-create",
did: "did:plc:a",
records: [{ rkey: "r1", title: "T" }],
});
} finally {
_resetBulkBuffersForTest();
time.restore();
}
});
Deno.test("bulkBuffers are cleared after flush and isolated per did", async () => {
const time = new FakeTime();
try {
_resetBulkBuffersForTest();
const { deps, dispatchCalls } = makeDeps();
queueBulkCreate("did:plc:a", { rkey: "a1" }, deps);
queueBulkCreate("did:plc:b", { rkey: "b1" }, deps);
await time.tickAsync(500);
await flushMicrotasks();
assertEquals(dispatchCalls.length, 2);
const dids = dispatchCalls.map((c) => (c.payload as { did: string }).did)
.sort();
assertEquals(dids, ["did:plc:a", "did:plc:b"]);
// Another flush window should not re-dispatch the same records.
await time.tickAsync(500);
await flushMicrotasks();
assertEquals(dispatchCalls.length, 2);
} finally {
_resetBulkBuffersForTest();
time.restore();
}
});

View File

@@ -0,0 +1,94 @@
import { assertEquals } from "@std/assert";
import { dispatchAll } from "../src/jetstream/webhooks.ts";
import { installFetchStub, jsonResponse } from "./_helpers/fetch_stub.ts";
Deno.test("dispatchAll is a noop on empty webhook list", async () => {
const stub = installFetchStub(() =>
jsonResponse({ should: "not be called" })
);
try {
await dispatchAll([], { event: "create" }, "label");
assertEquals(stub.calls.length, 0);
} finally {
stub.restore();
}
});
Deno.test(
"dispatchAll sends Authorization Bearer only when token is present",
async () => {
const stub = installFetchStub(() => new Response(null, { status: 204 }));
try {
await dispatchAll(
[
{ method: "POST", url: "https://a.example/hook" },
{
method: "POST",
url: "https://b.example/hook",
token: "shh",
},
],
{ event: "create" },
"label",
);
assertEquals(stub.calls.length, 2);
const [first, second] = stub.calls;
assertEquals(first.headers.get("Authorization"), null);
assertEquals(second.headers.get("Authorization"), "Bearer shh");
assertEquals(first.headers.get("Content-Type"), "application/json");
} finally {
stub.restore();
}
},
);
Deno.test(
"dispatchAll omits body and Content-Type for GET and HEAD",
async () => {
const stub = installFetchStub(() => new Response(null, { status: 204 }));
try {
await dispatchAll(
[
{ method: "GET", url: "https://a.example/ping" },
{ method: "HEAD", url: "https://b.example/ping" },
],
{ event: "create", data: "ignored" },
"label",
);
assertEquals(stub.calls.length, 2);
for (const call of stub.calls) {
assertEquals(call.body, "");
assertEquals(call.headers.get("Content-Type"), null);
}
} finally {
stub.restore();
}
},
);
Deno.test(
"dispatchAll uses Promise.allSettled — one rejection does not abort others",
async () => {
const stub = installFetchStub((req) => {
if (req.url === "https://reject.example/hook") {
return Promise.reject(new Error("connection refused"));
}
return Promise.resolve(new Response(null, { status: 204 }));
});
try {
await dispatchAll(
[
{ method: "POST", url: "https://reject.example/hook" },
{ method: "POST", url: "https://ok.example/hook" },
],
{ event: "create" },
"label",
);
assertEquals(stub.calls.length, 2);
const urls = stub.calls.map((c) => c.url);
assertEquals(urls.includes("https://ok.example/hook"), true);
} finally {
stub.restore();
}
},
);

115
tests/migrations_test.ts Normal file
View File

@@ -0,0 +1,115 @@
import { assertEquals } from "@std/assert";
import { Database } from "@db/sqlite";
import { applyMigrations } from "../src/migrations/apply.ts";
type TableInfoRow = { name: string };
type ColumnInfoRow = { name: string };
const tableNames = (db: Database): string[] =>
db.prepare(
"SELECT name FROM sqlite_master WHERE type = 'table' ORDER BY name",
).all<TableInfoRow>().map((r) => r.name);
const columnNames = (db: Database, table: string): string[] =>
db.prepare(`PRAGMA table_info(${table})`)
.all<ColumnInfoRow>()
.map((r) => r.name)
.sort();
const indexNames = (db: Database): string[] =>
db.prepare(
"SELECT name FROM sqlite_master WHERE type = 'index' AND name NOT LIKE 'sqlite_%' ORDER BY name",
).all<TableInfoRow>().map((r) => r.name);
Deno.test("applyMigrations creates expected tables, columns, and indices", () => {
const db = new Database(":memory:");
try {
applyMigrations(db);
const tables = tableNames(db);
assertEquals(
tables.includes("note") &&
tables.includes("state") &&
tables.includes("webhook_subscription"),
true,
);
const noteCols = columnNames(db, "note");
for (
const col of [
"title",
"publishedAt",
"createdAt",
"did",
"rkey",
"discoverable",
"language",
]
) {
assertEquals(noteCols.includes(col), true, `note.${col} missing`);
}
const webhookCols = columnNames(db, "webhook_subscription");
for (const col of ["id", "did", "method", "url", "token", "verb"]) {
assertEquals(
webhookCols.includes(col),
true,
`webhook_subscription.${col} missing`,
);
}
const indices = indexNames(db);
assertEquals(indices.includes("idx_webhook_subscription_did"), true);
assertEquals(indices.includes("idx_webhook_subscription_did_verb"), true);
} finally {
db.close();
}
});
Deno.test("applyMigrations is idempotent", () => {
const db = new Database(":memory:");
try {
applyMigrations(db);
db.exec(
"INSERT INTO webhook_subscription (did, method, url, verb) VALUES ('did:plc:x', 'POST', 'https://e.com', 'create')",
);
applyMigrations(db);
applyMigrations(db);
const [{ count }] = db.prepare(
"SELECT COUNT(*) AS count FROM webhook_subscription WHERE did = 'did:plc:x' AND verb = 'create'",
).all<{ count: number }>();
assertEquals(count, 1);
} finally {
db.close();
}
});
Deno.test(
"applyMigrations upgrades a legacy schema missing language and discoverable",
() => {
const db = new Database(":memory:");
try {
db.exec(`
CREATE TABLE note (
title TEXT NOT NULL,
publishedAt DATETIME,
createdAt DATETIME NOT NULL,
did TEXT NOT NULL,
rkey TEXT NOT NULL,
listed INTEGER NOT NULL DEFAULT 1,
PRIMARY KEY (did, rkey)
);
`);
applyMigrations(db);
const cols = columnNames(db, "note");
assertEquals(cols.includes("discoverable"), true);
assertEquals(cols.includes("language"), true);
assertEquals(cols.includes("listed"), false);
} finally {
db.close();
}
},
);

204
tests/opensearch_test.ts Normal file
View File

@@ -0,0 +1,204 @@
import { assertEquals, assertRejects } from "@std/assert";
import {
decodeCursor,
encodeCursor,
firstSnippet,
searchNotes,
} from "../src/search/opensearch.ts";
import { installFetchStub, jsonResponse } from "./_helpers/fetch_stub.ts";
const withOpenSearchEnv = (fn: () => Promise<void> | void) => {
return async () => {
const previousUrl = Deno.env.get("OPENSEARCH_URL");
const previousIndex = Deno.env.get("OPENSEARCH_INDEX");
Deno.env.set("OPENSEARCH_URL", "http://opensearch.test");
Deno.env.set("OPENSEARCH_INDEX", "notes");
try {
await fn();
} finally {
if (previousUrl === undefined) Deno.env.delete("OPENSEARCH_URL");
else Deno.env.set("OPENSEARCH_URL", previousUrl);
if (previousIndex === undefined) Deno.env.delete("OPENSEARCH_INDEX");
else Deno.env.set("OPENSEARCH_INDEX", previousIndex);
}
};
};
Deno.test("encodeCursor and decodeCursor round-trip", () => {
const cursor = encodeCursor(3.14, "abc123");
assertEquals(decodeCursor(cursor), [3.14, "abc123"]);
});
Deno.test("decodeCursor returns undefined on garbage input", () => {
assertEquals(decodeCursor("not-base64-at-all!!!"), undefined);
assertEquals(decodeCursor(btoa("not-json")), undefined);
assertEquals(decodeCursor(btoa(JSON.stringify([1]))), undefined);
assertEquals(
decodeCursor(btoa(JSON.stringify(["wrong", "type"]))),
undefined,
);
});
Deno.test("firstSnippet prefers highlight.content > title > source content > title fallback", () => {
assertEquals(
firstSnippet(
{ content: ["hl-content"], title: ["hl-title"] },
{ title: "src-title", content: "src-content" },
),
"hl-content",
);
assertEquals(
firstSnippet({ title: ["hl-title"] }, {
title: "src-title",
content: "src-content",
}),
"hl-title",
);
const longContent = "a".repeat(500);
assertEquals(
firstSnippet(undefined, { content: longContent, title: "ignored" }).length,
200,
);
assertEquals(firstSnippet(undefined, { title: "only-title" }), "only-title");
});
Deno.test(
"searchNotes returns empty when OPENSEARCH_URL is unset",
async () => {
const previousUrl = Deno.env.get("OPENSEARCH_URL");
Deno.env.delete("OPENSEARCH_URL");
try {
const stub = installFetchStub(() =>
jsonResponse({ should: "not be called" })
);
try {
const result = await searchNotes({
q: "hello",
discoverableOnly: true,
limit: 10,
});
assertEquals(result, { results: [] });
assertEquals(stub.calls.length, 0);
} finally {
stub.restore();
}
} finally {
if (previousUrl !== undefined) {
Deno.env.set("OPENSEARCH_URL", previousUrl);
}
}
},
);
Deno.test(
"searchNotes builds multi_match body with filters and cursor",
withOpenSearchEnv(async () => {
const stub = installFetchStub(() =>
jsonResponse({
hits: {
hits: [
{
_source: {
did: "did:plc:a",
rkey: "rk-a",
title: "T",
content: "c",
},
_score: 1.5,
highlight: { content: ["snippet"] },
},
{
_source: {
did: "did:plc:b",
rkey: "rk-b",
title: "U",
content: "d",
},
_score: 1.0,
},
],
},
})
);
try {
const result = await searchNotes({
q: "hello world",
discoverableOnly: true,
did: "did:plc:a",
limit: 2,
cursor: encodeCursor(2.0, "rk-prev"),
});
assertEquals(result.results.length, 2);
assertEquals(result.results[0].snippet, "snippet");
assertEquals(result.results[1].snippet, "d");
assertEquals(result.cursor, encodeCursor(1.0, "rk-b"));
assertEquals(stub.calls.length, 1);
const sent = JSON.parse(stub.calls[0].body) as Record<string, unknown>;
assertEquals(sent.size, 2);
assertEquals(sent.search_after, [2.0, "rk-prev"]);
const query = sent.query as {
bool: { must: unknown[]; filter: unknown[] };
};
const multi =
(query.bool.must[0] as { multi_match: Record<string, unknown> })
.multi_match;
assertEquals(multi.query, "hello world");
assertEquals(multi.fields, ["title^2", "content"]);
assertEquals(multi.fuzziness, "AUTO");
assertEquals(query.bool.filter, [
{ term: { discoverable: true } },
{ term: { did: "did:plc:a" } },
]);
} finally {
stub.restore();
}
}),
);
Deno.test(
"searchNotes returns cursor only when the page is full",
withOpenSearchEnv(async () => {
const stub = installFetchStub(() =>
jsonResponse({
hits: {
hits: [
{
_source: { did: "did:plc:a", rkey: "rk", title: "T" },
_score: 1,
},
],
},
})
);
try {
const result = await searchNotes({
q: "hi",
discoverableOnly: false,
limit: 5,
});
assertEquals(result.results.length, 1);
assertEquals(result.cursor, undefined);
} finally {
stub.restore();
}
}),
);
Deno.test(
"searchNotes throws on non-ok response",
withOpenSearchEnv(async () => {
const stub = installFetchStub(() => new Response("boom", { status: 500 }));
try {
await assertRejects(() =>
searchNotes({
q: "hi",
discoverableOnly: false,
limit: 5,
})
);
} finally {
stub.restore();
}
}),
);

287
tests/server_test.ts Normal file
View File

@@ -0,0 +1,287 @@
import { assertEquals } from "@std/assert";
import { createApp } from "../server.ts";
import { addWebhookSubscription, listWebhooksByDid } from "../src/data/db.ts";
import { seedNote, withTestDb } from "./_helpers/db.ts";
import { requestApp } from "./_helpers/app.ts";
import { stubAuthenticate } from "./_helpers/auth.ts";
import { installFetchStub, jsonResponse } from "./_helpers/fetch_stub.ts";
const withAdminEnv = (
dids: string,
fn: () => Promise<void> | void,
): () => Promise<void> => {
return async () => {
const previous = Deno.env.get("ADMIN_DIDS");
Deno.env.set("ADMIN_DIDS", dids);
try {
await fn();
} finally {
if (previous === undefined) Deno.env.delete("ADMIN_DIDS");
else Deno.env.set("ADMIN_DIDS", previous);
}
};
};
const withOpenSearchEnv = (fn: () => Promise<void> | void) => {
return async () => {
const previous = Deno.env.get("OPENSEARCH_URL");
Deno.env.set("OPENSEARCH_URL", "http://opensearch.test");
try {
await fn();
} finally {
if (previous === undefined) Deno.env.delete("OPENSEARCH_URL");
else Deno.env.set("OPENSEARCH_URL", previous);
}
};
};
Deno.test(
"GET /health returns 200 with status ok",
withTestDb(async () => {
const res = await requestApp(createApp(), { path: "/health" });
assertEquals(res.status, 200);
assertEquals(res.json(), { status: "ok" });
}),
);
Deno.test(
"GET /notes paginates rkey desc and honors cursor + limit",
withTestDb(async () => {
for (const rkey of ["a", "b", "c", "d", "e"]) {
seedNote({ rkey });
}
const app = createApp();
const first = await requestApp(app, { path: "/notes?limit=3" });
assertEquals(first.status, 200);
const firstBody = first.json() as {
notes: { rkey: string }[];
cursor?: string;
};
assertEquals(firstBody.notes.map((n) => n.rkey), ["e", "d", "c"]);
assertEquals(firstBody.cursor, "c");
const second = await requestApp(app, {
path: `/notes?limit=3&cursor=${firstBody.cursor}`,
});
const secondBody = second.json() as { notes: { rkey: string }[] };
assertEquals(secondBody.notes.map((n) => n.rkey), ["b", "a"]);
}),
);
Deno.test(
"POST /notes/feed rejects empty dids and accepts a valid list",
withTestDb(async () => {
seedNote({ did: "did:plc:alice", rkey: "rk1" });
const app = createApp();
const empty = await requestApp(app, {
method: "POST",
path: "/notes/feed",
body: { dids: [] },
});
assertEquals(empty.status, 400);
const notArray = await requestApp(app, {
method: "POST",
path: "/notes/feed",
body: { dids: "did:plc:alice" },
});
assertEquals(notArray.status, 400);
const ok = await requestApp(app, {
method: "POST",
path: "/notes/feed",
body: { dids: ["did:plc:alice"] },
});
assertEquals(ok.status, 200);
const body = ok.json() as { notes: { rkey: string }[] };
assertEquals(body.notes.length, 1);
}),
);
Deno.test(
"POST /:did/webhooks: 401 without auth, 403 on DID mismatch, 201 inserts create+delete by default",
withTestDb(async () => {
const app = createApp();
const noAuth = await requestApp(app, {
method: "POST",
path: "/did:plc:alice/webhooks",
body: { method: "POST", url: "https://hook.example/x" },
});
assertEquals(noAuth.status, 401);
const mismatch = stubAuthenticate("did:plc:bob");
try {
const res = await requestApp(app, {
method: "POST",
path: "/did:plc:alice/webhooks",
headers: { Authorization: "Bearer fake" },
body: { method: "POST", url: "https://hook.example/x" },
});
assertEquals(res.status, 403);
} finally {
mismatch.restore();
}
const owned = stubAuthenticate("did:plc:alice");
try {
const res = await requestApp(app, {
method: "POST",
path: "/did:plc:alice/webhooks",
headers: { Authorization: "Bearer fake" },
body: { method: "POST", url: "https://hook.example/x" },
});
assertEquals(res.status, 201);
const created = res.json() as { verb: string }[];
const verbs = created.map((w) => w.verb).sort();
assertEquals(verbs, ["create", "delete"]);
} finally {
owned.restore();
}
}),
);
Deno.test(
"POST /:did/webhooks rejects unknown verb with 400",
withTestDb(async () => {
const auth = stubAuthenticate("did:plc:alice");
try {
const res = await requestApp(createApp(), {
method: "POST",
path: "/did:plc:alice/webhooks",
headers: { Authorization: "Bearer fake" },
body: {
method: "POST",
url: "https://hook.example/x",
verb: "destroy",
},
});
assertEquals(res.status, 400);
} finally {
auth.restore();
}
}),
);
Deno.test(
"DELETE /:did/webhooks/:id: 400 on non-positive, 404 on unknown, 204 on success",
withTestDb(async () => {
const created = addWebhookSubscription({
did: "did:plc:alice",
method: "POST",
url: "https://hook.example/x",
verb: "create",
});
const auth = stubAuthenticate("did:plc:alice");
const app = createApp();
try {
const bad = await requestApp(app, {
method: "DELETE",
path: "/did:plc:alice/webhooks/0",
headers: { Authorization: "Bearer fake" },
});
assertEquals(bad.status, 400);
const missing = await requestApp(app, {
method: "DELETE",
path: "/did:plc:alice/webhooks/9999",
headers: { Authorization: "Bearer fake" },
});
assertEquals(missing.status, 404);
const ok = await requestApp(app, {
method: "DELETE",
path: `/did:plc:alice/webhooks/${created.id}`,
headers: { Authorization: "Bearer fake" },
});
assertEquals(ok.status, 204);
assertEquals(listWebhooksByDid("did:plc:alice").length, 0);
} finally {
auth.restore();
}
}),
);
Deno.test(
"GET /admin/webhooks: 403 for non-admin, 200 when ADMIN_DIDS contains caller",
withAdminEnv(
"did:plc:admin",
withTestDb(async () => {
const app = createApp();
const nonAdmin = stubAuthenticate("did:plc:other");
try {
const res = await requestApp(app, {
path: "/admin/webhooks",
headers: { Authorization: "Bearer fake" },
});
assertEquals(res.status, 403);
} finally {
nonAdmin.restore();
}
const admin = stubAuthenticate("did:plc:admin");
try {
const res = await requestApp(app, {
path: "/admin/webhooks",
headers: { Authorization: "Bearer fake" },
});
assertEquals(res.status, 200);
assertEquals(Array.isArray(res.json()), true);
} finally {
admin.restore();
}
}),
),
);
Deno.test(
"GET /search: 400 on blank q, forwards discoverableOnly:true to opensearch",
withOpenSearchEnv(
withTestDb(async () => {
const app = createApp();
const blank = await requestApp(app, { path: "/search?q=" });
assertEquals(blank.status, 400);
const stub = installFetchStub(() => jsonResponse({ hits: { hits: [] } }));
try {
const res = await requestApp(app, { path: "/search?q=hello" });
assertEquals(res.status, 200);
assertEquals(stub.calls.length, 1);
const sent = JSON.parse(stub.calls[0].body) as {
query: { bool: { filter: Record<string, unknown>[] } };
};
assertEquals(sent.query.bool.filter, [
{ term: { discoverable: true } },
]);
} finally {
stub.restore();
}
}),
),
);
Deno.test(
"OPTIONS preflight returns 204 with CORS headers",
withTestDb(async () => {
const res = await requestApp(createApp(), {
method: "OPTIONS",
path: "/notes",
});
assertEquals(res.status, 204);
assertEquals(res.headers.get("Access-Control-Allow-Origin"), "*");
assertEquals(
res.headers.get("Access-Control-Allow-Methods")?.includes("POST"),
true,
);
assertEquals(
res.headers.get("Access-Control-Allow-Headers")?.includes(
"Authorization",
),
true,
);
}),
);