Architecture and trade-offs

knotel is a tracing backend, dashboard and MCP server in a single Cloudflare Worker. Spans are kept in a columnar Iceberg table on R2 and queried with R2 SQL, with a small D1 database in front as a fast cache of recent spans. This page explains how that works, why it's built this way, and where the design stops being the right fit.

The short version

• One deployable. Ingest, queries, the web app, auth, MCP and the nightly cleanup are one Worker. There are no servers, containers or queues to run.
• Your account, your data. Spans are written to R2 and D1 in your own Cloudflare account and nowhere else.
• Columnar storage with no fixed ceiling. Cloudflare Pipelines streams spans into Parquet files in an Iceberg table on R2, so history grows as cheaply as object storage.
• A hot cache keeps it fast. R2 SQL takes seconds per query, so the last 7 days also live in D1 and answer in milliseconds.
• About $5 a month for small teams: at hundreds of thousands of spans a month everything fits inside the Workers Paid plan's allowances.
• OpenTelemetry-native. Spans arrive as OTLP over HTTP from the bundled browser script, the Workers SDK, or any OTel SDK.

The pieces

Browser t.js, Workers SDK, OpenTelemetry SDKs and Collector
  │
  │ POST /v1/traces
  ▼
Worker: auth (memory → KV → D1) → parse → filter
  ├──► Pipelines stream ──► Iceberg table on R2   (everything, forever)
  └──► D1                                          (last 7 days, fast)

query module ─┬─ window ≤ 7 days  ──► D1
              └─ older            ──► R2 SQL      (KV caches the result)
   ▲
   ├─ dashboard (server-rendered)
   └─ /mcp (Claude, IDEs)

Cron, daily 03:17 UTC ─► trim D1 back to the hot window

Life of a span

1. Captured. The browser script patches fetch, listens for errors and reads the navigation timing API; the Workers SDK wraps your handler and uses AsyncLocalStorage to know which request an outgoing fetch belongs to. Both add a W3C traceparent header so the next service continues the same trace.
2. Batched. The browser sends up to 50 spans per request (sooner when the tab hides, with keepalive). A Worker sends a request's spans after the response, with ctx.waitUntil, so users never wait on telemetry.
3. Authenticated. The ingest key is hashed and looked up in an in-memory map, then KV, then D1. Most requests never touch D1 to authenticate.
4. Parsed. OTLP JSON and protobuf are normalised the same way: hex or base64 ids, numeric or named enums, nanosecond strings. Both read the same OTLP fields, so neither encoding carries less. Well-known attributes (HTTP method, route, status, URL, database system, operation, collection, query) become columns, as do the instrumentation scope, trace state and flags, and the counts of what the sender's SDK dropped; everything else stays as JSON.
5. Filtered. Project filters drop scanner probes and other noise in memory, before any write.
6. Stored twice. The batch goes to the Pipelines stream, which writes Parquet into the Iceberg table about once a minute, and to D1 as one INSERT … SELECT FROM json_each(?) statement, so 50 spans cost one query rather than 50.
7. Metered. Spans, bytes, D1 rows and R2 SQL queries are added to daily usage counters after the response.

Reading: one question, two stores

Every chart, table and MCP answer comes from one query module. It looks at the time range and picks the store: anything inside the hot window goes to D1, anything older to R2 SQL. Callers never know which answered.

Entry spans

Request counts, error rates and latency use only server spans and trace roots, so a request with twenty database calls counts once, with its end-to-end latency.

Percentiles

In D1, p50, p95 and p99 are exact, computed with ROW_NUMBER() OVER (PARTITION BY … ORDER BY duration). In R2 SQL they come from approx_percentile_cont, which is approximate but scans far less data.

The service map is a self-join

Joining each span to its parent finds every place where the parent belongs to a different service or project: that's an edge. Database and external-host edges come from client spans that no instrumented service answered. No extra instrumentation or topology data is needed, and the same join works in both stores.

Home signals are computed on read

The Needs attention list compares each service's current window with the one before (error rate, p95, traffic) when you open Home. There's no background job or alert state to maintain.

Caching

• Credentials: memory (60 s) → KV (5 min) → D1. Revoking deletes the KV entry immediately.
• Dashboards: every query is cached in memory, then KV, for 30 s (15m) up to 30 min (30d), and everyone looking at the same range shares one result. Past that, the old result is still served while a new one loads in the background, so a viewer never waits on a query just because the cache turned over. This matters most for the columnar store, where a query takes seconds. Settled traces (nothing new for 5 minutes) are cached for a day.
• Filters: compiled rules in memory, then KV, then D1.

Auth and MCP

Better Auth handles email and password sign-in, with sessions in D1. There's no public sign-up: the first account claims the instance with a setup code. The auth instance binds to whichever origin a request comes in on, so workers.dev, a custom domain and localhost all work without configuration.

The MCP server is stateless Streamable HTTP: each request builds the tool set, answers, and exits, which suits Workers. Tools call the same query module as the dashboard, so the AI and the UI never disagree.

Why Cloudflare

• No operations. Nothing to patch, scale or keep running. The Worker scales from zero to whatever arrives, and Iceberg tables are managed by R2 Data Catalog rather than by you.
• Ingest at the edge. Spans are received in the Cloudflare location nearest each user, so the browser's telemetry requests are fast wherever your users are.
• Cheap storage. Receiving spans is free, writing them into Iceberg is $0.06/GB after 50 GB a month, and R2 storage is $0.015/GB-month with no egress fees.
• Next to what you trace. If your apps already run on Workers, instrumenting them and deploying knotel use the same tools and account.

Trade-offs and limits

Compared with the alternatives

Where it goes next

• Rollups: precomputed daily aggregates so long-range dashboards don't wait on R2 SQL at all.
• A bigger hot store: one SQLite-backed Durable Object per project, lifting the single 10 GB cache limit for busy instances.
• History management: compaction settings and a way to drop old partitions from the Iceberg table.
• Alerts, a visual query builder with heatmaps, logs, and OAuth for MCP.

Data model

The Iceberg table knotel.spans holds one row per span, with the same 37 columns the stream's schema defines: ids, service, name, kind, timings, status, the promoted HTTP and database attributes, the instrumentation scope, trace state and flags, the sender's dropped counts, when ingest received the span, and attributes, resource, events and links as JSON text. D1 holds the same shape for the hot window, plus everything else:

Code layout