How far an observability backend goes on SQLite
A single Go binary writing OTLP metrics into SQLite on a 2 vCPU, 8 GB box sustained 58k metric points a second. Here is the method, and where it stops.
SQLite is not the database anyone reaches for when they hear "observability backend", and that is mostly habit rather than measurement. The TracePath backend keeps telemetry behind a storage interface with three implementations: ClickHouse, which is what TracePath Cloud runs, and two embedded engines, SQLite and DuckDB. This post is the SQLite measurement, on the cheapest dedicated-vCPU machine worth benchmarking: what a single binary writing OpenTelemetry metrics into a local SQLite file actually sustains, and where it stops being a good idea.
Key takeaways
- On a 2 dedicated vCPU, 8 GB, 80 GB NVMe box, the SQLite storage mode sustained 58,000 metric points per second at its best payload shape, and 36,000 points/sec at an SDK-shaped 100 points per request.
- A 200 response means the rows are in the database. There is no in-memory buffer in front of the write, so the throughput figures are durable writes, not queue depth.
- The single biggest win was not a pragma. Wrapping each ingest batch in one transaction, plus a larger page cache, took the ceiling from roughly 15k to the numbers above.
- Writes were never the limit. Read latency is, and that is the subject of the follow-up work on DuckDB.
- None of this is a sizing exercise you have to do on TracePath Cloud, which runs the ClickHouse backend. It is a useful one if you are deciding what your own service should store data in.
Why benchmark an embedded database at all
A complete observability stack has to carry six things: metrics, logs, traces, real-user monitoring, alerting, and symbolicated exceptions. There are two conventional ways to get them.
The first is to buy them. That works, the bill is predictable until it isn't, and free tiers are easy to blow through on a project with barely any users.
The second is to assemble them: a collector, a metrics store, a log store, a trace store, a profiler, an alert manager, a dashboard layer, a RUM pipeline, an error tracker. Each is good. Together they are a dozen processes to run, upgrade, and keep alive, and each one wants enough headroom to do its job. The payoff is that the assembly scales to enormous volume, if you ever need that.
Strip the architecture diagrams away, though, and ingest is a small loop: accept a protobuf, decode it, write rows, and later answer aggregations over those rows. It is fair to ask how much database that loop actually requires at a scale most services never leave.
The system under test
Everything below ran on a Hetzner CCX13: 2 dedicated vCPU, 8 GB RAM, 80 GB NVMe SSD, about $16.49/month. Load was generated from a separate machine on the same private network, so the generator never competed with the backend for CPU.
On the box: one binary, the whole stack, telemetry persisted to a SQLite file on the local disk. The SQLite configuration is not stock — the telemetry database is opened with a tuned DSN:
_pragma=journal_mode(WAL) # readers never block the writer
_pragma=busy_timeout(5000) # absorb short lock contention
_pragma=synchronous(NORMAL) # fsync at checkpoints, not per commit
_pragma=cache_size(-524288) # 512 MB page cache
_pragma=temp_store(MEMORY) # sorts and temp b-trees stay off disk
_pragma=mmap_size(1073741824) # 1 GB memory-mapped read window
_pragma=wal_autocheckpoint(50000) # ~50k pages of WAL between checkpoints
_pragma=auto_vacuum(incremental) # reclaim freed pages without a full VACUUMTwo details behind that block matter more than any individual pragma.
The first is that the pragmas ride on the connection string rather than being executed after opening. A PRAGMA statement run through db.Exec configures whichever pooled connection happens to serve it; every connection opened later starts unconfigured. Putting them in the DSN means every connection in the pool boots identically.
The second is that there are two databases, not one. Relational state (users, projects, dashboards, alert rules) lives in a transactional SQLite file limited to a single connection. Telemetry lives in its own file with a pool of four. WAL mode lets those four readers work while a write is in flight, but SQLite still serializes writes at the file level, which is exactly why batching matters so much below.
What the benchmark measures
Three phases, metrics only, all writes:
- Large payload. Up to 16k metric points per request, capped at 5 requests/sec.
- Largest sustainable payload, maximum request rate. Take the biggest payload that held, then ramp request rate until something breaks.
- SDK-shaped. 100 metric points per request, ramping request count. This is the shape a fleet of instrumented services actually produces.
What "accepted" means here
The ingest endpoint returns 200 only after the payload has been written to SQLite. There is no in-memory buffer in front of the database, so a successful response is a durable write. P99 latency stayed under 400 ms even on the largest payloads, which is the evidence that these are sustained ingest rates rather than a queue filling faster than it drains.
That property is a deliberate trade. Buffering in memory before the write would raise every number here, and it would mean a 200 no longer promises the data survives a restart.
Pass and fail
A step fails when either of these is true:
- combined error rate above 5%
- achieved rate below 70% of target, the soft cliff
A step passes once it has survived two minutes under load. The harness then ramps and repeats. The headline number is the highest sustained rate across all passing steps.
Results
| Phase | Payload shape | Sustained |
|---|---|---|
| Large payload | 8,000 points/request | 31,000 points/sec |
| Max rate at best payload | largest sustainable batch | 58,000 points/sec |
| SDK-shaped | 100 points/request | 36,000 points/sec |
What actually moved the number
The first runs peaked around 15k points/sec. Almost all of the distance to 58k came from one change: wrapping each batch of inserts in a single transaction instead of committing row by row. In WAL mode every commit is a write barrier, so a thousand single-row commits pay a thousand barriers to store the same thousand rows. One transaction pays one. Combined with the larger page cache, that nearly quadrupled throughput.
Two further ideas remain on the table and are worth naming because of what each costs:
- A centralized writer that batches in memory before inserting. Faster, and it gives up the "200 means persisted" property described above. That is a product decision, not a performance one.
- Multi-row insert statements instead of single-row ones. Strictly cheaper, no durability change, just work.
Where SQLite stops
The honest framing of a benchmark like this is not "look how fast", it is "here is the edge". Three edges showed up.
Reads fail long before writes do. Follow-up runs on the same hardware probed real dashboard pages against filled tables. The metrics views stayed usable to roughly 1M rows; the spans and logs views to roughly 100k. Above that there was no slow-but-working middle ground: queries either answered under the threshold or ran past the timeout entirely. If you plan capacity for an embedded telemetry store, plan it in rows you will query, not rows you can write.
Logs are the weakest signal. Log records were the lowest write ceiling of the three signals and the first to hit a hard wall, where a modest increase in request rate took the error rate from near zero to near total. That result is what motivated benchmarking a columnar engine next.
One box, one process. The embedded builds are single-instance by construction: the databases are local files, and the in-process caches only learn about writes made by the same process. Horizontal scale is not a configuration change, it is a different storage backend.
What that adds up to
Match the storage engine to the volume. If you are running exception tracking and RUM for a mobile or web app, a monolith's internal tooling, a side project, or an early-stage product, an embedded store on a small machine has a genuinely small failure surface: one process to watch, one file to back up, no coordination.
A concrete envelope, at roughly the shape a small fleet emits:
- Metrics: 5 services × 20 points/sec = 100 inserts/sec
- Traces: 5 services × 10 requests/sec = 50 inserts/sec
- Logs and spans: 5 services × 10 requests/sec × (10 spans + 50 log records) = 3,000 inserts/sec
About 3,150 inserts per second in total, comfortably inside what the benchmark sustained — and, per the read cliffs above, a retention window measured in days rather than months is what keeps the dashboards fast. Exception tracking alone stretches much further, because exceptions are rare; hitting even a few thousand a second takes real effort.
Where TracePath Cloud sits
TracePath Cloud does not run this configuration. The hosted platform runs the ClickHouse storage backend with PostgreSQL for relational state, which is the combination built for many tenants and long retention rather than one machine. You point an OTLP exporter at the ingest host and the storage sizing stops being yours:
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.tracepath.dev/api/otel
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <project_token>Retention and ingest volume become a plan setting instead of a pragma — see pricing for the limits on each plan, and the OpenTelemetry setup guide for collector and per-language configuration. The signals themselves are the same ones benchmarked here: metrics, logs, and traces.
FAQ
Is SQLite a reasonable database for telemetry?
For writes, yes, further than its reputation suggests: tens of thousands of rows a second on a small machine, provided inserts are batched into transactions. For reads, it depends entirely on how many rows a dashboard query has to touch. Aggregations over millions of rows are where a row-store runs out, and that arrives long before the write ceiling does.
Why does the response time matter if the numbers are about throughput?
Because it separates ingest from buffering. If a service answers 200 immediately and writes later, its throughput number describes how fast it accepts bytes, not how fast it stores them. Keeping P99 under 400 ms while the write is synchronous is what makes the throughput figure meaningful.
What is the single highest-leverage change for SQLite write throughput?
Batching writes into one transaction. Every commit in WAL mode is a synchronization point; the fewer commits per row, the better. It was worth more here than cache sizing, synchronous level, and mmap window combined.
Does TracePath Cloud run on SQLite?
No. TracePath Cloud runs the ClickHouse backend. The embedded engines exist because the same backend supports several storage layers, which is what makes a comparison like this possible in the first place.
What replaces SQLite when the read cliffs get close?
A columnar engine. The follow-up benchmark ran the identical harness against the DuckDB storage backend on the identical hardware and moved every read cliff by two orders of magnitude: SQLite vs DuckDB.
Questions, or a result that disagrees with ours: [email protected].