I Ran My Entire Observability Stack on SQLite. You Probably Could Too.

TL;DR

TracePath runs on ClickHouse or SQLite depending on your scale. This post benchmarks the SQLite mode. On a $16.49/month Hetzner box (2 dedicated vCPU, 8 GB RAM, 80 GB NVMe SSD), a single binary running the full observability stack (metrics, logs, traces, RUM, alerting, exceptions) on SQLite hit:

• 31k metric points/sec on large payloads (8k points/request)
• 58k metric points/sec at the largest sustainable payload
• 36k metric points/sec at 100 points per request

Methodology and full results below. Short version: SQLite is doing a lot more than people give it credit for, and you almost certainly do not need a dozen-plus-component LGTM deployment to run a side project.

Why are you doing this?

Observability is a set of trade-offs.

A good observability stack will provide: metrics, logs, traces, RUM, alerting and exceptions (symbolicated). To get that stack, the two most popular options are:

1. Pay Datadog, New Relic, Sentry.
2. Self-host the LGTM stack: Alloy, Prometheus or Mimir or VictoriaMetrics, Loki or VictoriaLogs, Tempo, Pyroscope, Alertmanager, Grafana, Grafana OnCall, Faro, Sentry…

Option one is the most expensive. You pay for convenience, the free tiers are tight, and I've managed to blow through them on side projects that barely had users.

Option two covers everything but introduces two problems. It's not easy to host. That's a lot of moving parts and a lot of potential points of failure. And it's resource-intensive; giving each component enough headroom to actually do its job adds up fast. The upside is that it scales incredibly well, if you need it to.

But we're talking about ingesting a protobuf and writing it to a database when the backend gets a request. Let's find out how hard that is to do.

The real question I wanted to answer: what is the smallest server I could run a full observability stack on, and what does throughput look like when I stop caring about scaling to a million pods?

Methodology

I've been building a ClickHouse-first observability platform. The goal of the platform was accessibility and providing the experience of Datadog/New Relic/Sentry on top of OpenTelemetry data. Obviously hosting ClickHouse is not going to happen on the $5 tier server from Railway, so I've implemented the database layer with SQLite as well. This is the setup we will be measuring: a single binary packed with all parts of the observability stack running with SQLite persisted on disk.

All of the tests have been performed on a dedicated-vCPU instance, specifically the cheapest dedicated-vCPU Hetzner instance, the CCX13 (2 dedicated vCPU, 8 GB RAM, 80 GB NVMe SSD), starting at $16.49 per month. Load generation is happening on a separate server.

We will be measuring throughput of metrics in three phases:

1. Large payload - 16k max metric points per payload, 5 req/sec max
2. Largest sustainable payload, max request rate - taking the largest sustainable payload we max out req/sec to find the breaking point
3. 100 metric points per req - how many requests per second can we ingest

What "accepted" means here

TracePath's ingestion endpoint only returns 200 once the payload has been written to SQLite. There's no in-memory buffer in front of the database, so a successful response is a durable write. P99 latency stayed under 400ms even on the largest payloads, which means the throughput numbers below reflect sustained ingest, not a queue filling up faster than it drains.

Pass/fail thresholds

A test is considered failed if:

• Combined error rate > 5%
• Soft cliff: achieved rate < 70% of target

A given request rate is considered successful after it survives for 2min under that load. The benchmark then ramps to the next level and repeats. The headline number is the maximum sustained items/sec across all passing steps.

These three phases measure write throughput only; read performance is out of scope for this post, but will be covered in the future.

Analysis and SQLite optimizations

When I first ran the tests, the max throughput I was able to achieve peaked at 15k metrics/sec on the CCX13 instance. The biggest optimization win came from wrapping the batches of inserts into a transaction. Combined with a slightly bigger cache it almost quadrupled the throughput. No other real optimizations were done at this stage. Possible ideas I'll be exploring over the next few weeks to push this further:

• Centralized writer that batches events in memory before inserting. Note that this is a deliberate durability/throughput trade-off: buffering in memory before the write means a successful response would no longer guarantee a durable write, so it gives up the "200 = persisted" property described above in exchange for higher throughput.
• Building multi-row insert statements instead of single-row ones

Systems usually fail after 2 weeks, not instantly

The obvious pushback: systems like this usually fail two weeks in, not at hour one. Agreed, you should NOT use SQLite as your production system pushing 50k metrics/sec anyway.

I am proposing that you optimize your costs to your scale. If your scale is small you should use tools that give you the best cost/performance at that scale. If your scale allows, there is nothing that can beat running a single binary with SQLite, your surface area for failure is minimal, a single tool that is easy to set up and monitor in production. This all makes sense only if you're self-hosting due to regulatory reasons or personal preference.

Here is when this configuration makes sense:

• Exception tracking + RUM for Mobile (Flutter/Android/iOS)
• Exception tracking + RUM for Frontend
• Monolith backends for internal tooling
• Side projects
• Early startups

Here is the breakdown of potential loads that this setup can handle easily:

• Metrics: 5 servers × 20 metrics/sec = 100 inserts/sec
• Traces: 5 servers × 10 req/sec = 50 inserts/sec
• Logs and Spans: 5 servers × 10 req/sec × (10 spans + 50 logs) = 3000 inserts/sec

This adds up to 3.15k inserts per second, which SQLite will handle easily. As we're assuming costs are the main concern, you could limit your retention to seven days. With that you are almost guaranteed to stay within your server's limits, even when both memory and CPU are constrained.

A much larger deployment can be supported if you're only doing exception tracking with RUM, as exceptions are rare in general and it's pretty hard to hit even 10k exceptions/sec.

Conclusion

I've had a bunch of fun setting up the benchmarks and seeing the results. Honestly, SQLite has performed better than expected. It also helps with specifying the minimum requirements for running TracePath and the performance that can be expected. Now that the benchmarks are set up, I am looking forward to all the future data I'll be able to pull from the system.

You can find the results of all runs in the TracePath repo: https://github.com/tracepathhq/tracepath. If you have any questions, feel free to reach out at [email protected].

Next post: same setup, a full week of real data flowing through it. Logs, traces, what actually breaks, and whether the SQLite story holds up past hour one, querying common endpoints while writing. After that we move to ClickHouse and find out what the bigger machine actually buys you.