System design of a fast symbolicator

A symbolicator turns an unreadable production stack trace — at n (app.min.js:1:5114) — back into the code somebody wrote: validateCheckout (checkout.js:48:10). The algorithm for one frame is a lookup. The system design is everything that decides whether the ten-thousandth lookup during a crash spike costs the same as the first. This post is the second part: how TracePath's symbolicator is shaped so that the expensive work happens once per build artifact instead of once per frame, and how the operating system ends up doing the caching.

Premature optimization is the root of all evil.

Knuth wrote that in 1974 and it has aged well. Quality software is built by understanding the problem first and improving the solution pragmatically, one measured step at a time. That is the structure of what follows: the slow version, then the two changes that made it fast, with the numbers each one moved.

Key takeaways

• Building a lookup structure from a source map costs roughly 975 µs. The lookup it enables costs 44 ns. Everything downstream follows from that ratio.
• Caching the parsed object is the obvious answer and the wrong one: it hands the garbage collector thousands of pointers per artifact.
• Serializing to a flat, pointer-free byte layout makes the in-memory cache and the on-disk file the same bytes, searched by the same function.
• mmap then hands eviction to the kernel's page cache, which survives restarts and lets replicas share prebuilt artifacts.
• In a head-to-head benchmark, the disk-backed design held ~30k stack traces/sec under cache churn at 361 MB resident, against 938/sec at 4,491 MB for a parse-on-miss implementation.

The vocabulary, quickly

When you ship JavaScript, a build step shrinks it into one file, the bundle. validateCheckout becomes n, whitespace vanishes, and the whole program collapses onto one line. That is minification — great for download size, miserable the moment something breaks.

When that code crashes, what you get is a stack trace: the chain of calls that led to the crash. Each call is a frame, printed as a line like at n (app.min.js:1:5114) — a spot in a file nobody can read, under a name that used to mean something. Stack a dozen of those and the report is unactionable.

Turning that back into real names and line numbers is symbolication, and the service that does it is a symbolicator. It needs the minified frame plus a build artifact: for JavaScript, the source map the build tool emits, which records where each scrap of minified code came from. Every serious error tracker has one.

The algorithm inside has been taken apart here before, by hand, twice: once for JavaScript source maps and once for an obfuscated Flutter crash. Neither is required reading for this post, which is about the part both skipped — the part that decides whether the thing works in production.

The job sounds simple: run that algorithm for every crash, from every release, of every customer, fast enough that symbolication never makes an event show up late. The simple version is painfully slow. The road from slow to fast runs through caching, binary file layout, and the operating system's memory manager. None of it is specific to crashes. A symbolicator is a sharp example of a common shape: data that is expensive to build and cheap to look up. Parse once, cache the right shape, let the kernel do the rest.

The problem

Strip everything away and a symbolicator is one function:

Symbolicate(file, line, col) -> (origFile, origLine, origCol, fnName)

Calling it once is cheap. But exceptions are rarely one-offs; they arrive in bursts. Ship a bug and the same crash spams in, and the larger the app the more of them come. Client-side code is the worst case: a million browsers can hit the same crash in the same minute. Each error carries a dozen frames, spread across whatever bundles and releases those users happened to be running.

So the real question is not "how do I symbolicate a frame". It is what sits behind Symbolicate so the ten-thousandth call in a spike is as cheap as the first.

Stage 1: the dumb version that parses on every request

A request arrives carrying a stack frame: a file, a line, a column. To turn it back into source you need the build's debug artifact, and which one depends on the platform. JavaScript ships a source map plus the bundle. iOS ships a .dSYM. Dart ships a .symbols file. Different formats, one job: each maps a compiled position back to an original file, line and name.

The simplest possible service does the whole thing inline, fresh on every request:

Step 2 is where the cost is. Parsing the artifact walks the whole file and builds a sizable in-memory index: milliseconds of work, thousands of allocations. The lookup in step 3 is a plain search by comparison, effectively free.

Here is the waste. A release's debug artifact is frozen the moment you ship it; it never changes. Yet this version rebuilds the same index from the same bytes on every request and then throws it away. During a burst — a thousand frames a second off one broken release — it runs that identical parse a thousand times a second and spends almost nothing on the lookups it came for.

The fix names itself: parse once per artifact, look up once per frame. That is the easy part. Making it hold up in production is the rest of the work.

Stage 2: build the structure once, keep it in memory

Just how lopsided is that split? Here it is in TracePath, where the parse-and-resolve step is BuildTW and the per-frame lookup is LookupTW. A local go test -bench on a real pair, preact.module.js (10 KB) and its 69 KB source map:

BenchmarkBuildTW    975232 ns/op    771942 B/op    10260 allocs/op
BenchmarkLookupTW       44.24 ns/op      24 B/op        2 allocs/op

The lookup is 44 nanoseconds. The parse in front of it is 975 microseconds, three quarters of a megabyte of garbage, ten thousand allocations. The build is ~22,000x more expensive than the lookup it enables. So you cache the parse and never repeat it. The interesting question is what to cache, and the whole design hinges on the answer.

The lazy answer is to keep the live parsed object: a Go struct full of slices and a map[string]int32 name index. It works, but it is a sprawling object graph — thousands of little allocations, every one a pointer the garbage collector chases on every cycle, for a value that is immutable and that you would like to keep around for hours. Cache ten thousand of those and you have turned a symbolicator into a GC benchmark.

So BuildTW does not return an object graph. It serializes to bytes, in the .tw resolver format: the compact, little-endian layout the engine compiles every build artifact into and runs on in production. A .tw is a cache artifact, never the source of truth, so it can afford to be dumb. A header, a sorted array of fixed-size token records (one per mapped position), then two interned string tables for file and function names:

Every token is a fixed 24 bytes, six little-endian uint32s: the generated line and column (where it sits in the minified bundle), the original line and column (where it came from in your source), and two indices into the string tables (the source file and the function name). No pointers anywhere. A string is an index, and the tables turn an index into bytes. Here is the heart of the serializer:

out = append(out, twMagic[:]...)                              // "TWSM"
out = binary.LittleEndian.AppendUint32(out, twVersion)
out = binary.LittleEndian.AppendUint32(out, uint32(len(b.tokens)))
// ... file + fn counts + a reserved word ...
 
for i := range b.tokens {
	t := &b.tokens[i]
	out = binary.LittleEndian.AppendUint32(out, t.genLine)
	out = binary.LittleEndian.AppendUint32(out, t.genCol)
	out = binary.LittleEndian.AppendUint32(out, t.srcLine)
	out = binary.LittleEndian.AppendUint32(out, t.srcCol)
	out = binary.LittleEndian.AppendUint32(out, uint32(t.fileIdx))
	out = binary.LittleEndian.AppendUint32(out, uint32(t.fnIdx))
}

The tokens are sorted by generated position. That is the whole point. A lookup becomes a binary search over a slice of fixed-size records, with no decoding involved. LookupTW is sort.Search plus a couple of field reads:

idx := sort.Search(l.tokenCount, func(i int) bool {
	gl, gc := tokAt(i)                            // read token i's generated pos
	return gl > genLine || (gl == genLine && gc > genCol)
})
// idx-1 is the floor token: the greatest position <= the frame.
// read its srcLine/srcCol/fileIdx/fnIdx straight out of the bytes,
// resolve the two indices through the string tables, done.

That is the 44-nanosecond number. The two allocations it reports are the two strings it copies out at the end, the file name and the function name. Everything else is read in place. No parse, no garbage.

On the preact pair, BuildTW turns a 69 KB map and a 10 KB bundle into a 48 KB .tw blob with about 2,000 tokens. That blob is one allocation the GC never looks inside, and it is the unit that gets cached. The cache is a plain LRU keyed by file name, bounded by entry count and total bytes. (LRU means least-recently-used: when it fills, whatever nobody has touched in the longest gets dropped.)

func (s *twcachemem) get(name string) ([]byte, func(), bool) {
	s.mu.Lock()
	defer s.mu.Unlock()
	el, ok := s.items[name]
	if !ok {
		return nil, noop, false
	}
	s.order.MoveToFront(el)               // mark most-recently-used
	return el.Value.(*memEntry).data, noop, true
}

The cache wrapper builds each file exactly once, even under a stampede: the first request for a file builds it while any concurrent requests for the same file wait on that in-flight load, instead of all calling BuildTW at once. The per-request path becomes:

data, done, err := cache.Get(ctx, fileKey, func(ctx context.Context) ([]byte, error) {
	return sourcemap.BuildTW(mapBytes, bundleBytes)   // runs once per file
})
defer done()
frame, ok := sourcemap.LookupTW(data, line, col)      // 44 ns, every frame after the first

Now the first frame from a bundle pays the 975 µs and every frame after it pays 44 ns, for the rest of that bundle's life in cache. At a thousand errors a second over a working set that fits in memory, nearly all the CPU just disappeared.

Why serialize to bytes instead of caching the parsed struct? Three reasons, and the third is the one that matters:

1. It is compact. 48 KB beats a sprawl of thousands of small objects.
2. It is invisible to the GC. One []byte with no inner pointers. The collector skips straight over it, however many thousands are held.
3. It is already the exact shape you would write to a file. A flat, pointer-free, offset-addressed byte array is a file format. That is the entire next stage.

Hold that third point. But first, the catch that ends Stage 2: this cache lives entirely in RAM.

To keep every lookup fast, every .tw you might need has to be resident at once — the whole working set has to fit in memory. For one small app it does, a handful of bundles. A platform symbolicating at scale is the opposite. A mobile app has a long tail of versions still installed on real devices. A website ships a new bundle on every deploy, and crashes keep arriving from every release a user is still running. That is thousands of distinct artifacts, each with its own .tw, and the old ones never fully go quiet. Hold them all in memory, on every replica, and the bill is unpayable. The LRU's only answer is to evict, which turns the overflow back into 975 µs rebuilds.

Two smaller cracks sit next to it. A restart is a cold start: the cache is gone, and the first wave of every bundle re-parses from scratch. And replicas do not share — ten servers behind a load balancer keep ten separate caches, each re-parsing the same bundle the first time a frame lands there.

Three problems, one root. The structure lives on one heap, in one process, for only as long as that process runs. The next stage moves it off the heap entirely and fixes all three at once.

Stage 3: store it and mmap it

Reread reason 3 above. The thing being cached is a flat []byte with no pointers; every "reference" is an integer offset from the start of the buffer. A structure like that does not care whether its bytes live in a heap allocation or in a file on disk. The bytes are the structure.

So persisting it is os.WriteFile. And loading it is mmap, the system call that maps a file straight into a process's address space. You get back a byte array that is the file on disk, and the OS pulls in each chunk the first time you touch it — a page fault — instead of copying the whole thing up front:

func mmapFile(path string) ([]byte, func(), error) {
	f, err := os.Open(path)
	if err != nil {
		return nil, nil, err
	}
	defer f.Close()
 
	st, _ := f.Stat()
	size := st.Size()
 
	data, err := syscall.Mmap(int(f.Fd()), 0, int(size),
		syscall.PROT_READ, syscall.MAP_SHARED)
	if err != nil {
		return nil, nil, err
	}
	return data, func() { _ = syscall.Munmap(data) }, nil
}

Now look at the signature that has been called all along:

func LookupTW(data []byte, genLine, genCol uint32) (StackTraceFrame, bool)

It takes a []byte. It never cared where the bytes came from. A heap buffer from make, a memory-mapped file from syscall.Mmap: to LookupTW they are the same thing. The structure was designed so they could be. Stage 2 and Stage 3 run identical lookup code over identical bytes; the only thing that changed is who owns the pages.

And that is what the binary search buys: the file is searched in place. No read, no decode, no copy into a struct. A search over ~2,000 tokens touches about eleven 24-byte records, which on a 48 KB file land in a few pages. So the first lookup costs a few page faults and every one after it is a plain memory read at RAM speed. The disk store wires mmapFile straight into the cache's get:

func (s *twcachedisk) get(name string) ([]byte, func(), bool) {
	path, err := s.path(name)
	if err != nil {
		return nil, noop, false
	}
	data, unmap, err := mmapFile(path)   // []byte backed by the file
	if err != nil {
		return nil, noop, false
	}
	s.noteUse(name, int64(len(data)))
	return data, unmap, true             // caller calls unmap when done
}

That done/unmap is the one bit of discipline mmap adds. The []byte is only valid until it is unmapped, so a request borrows it, does its lookups, and releases it at the end. That is safe because LookupTW copies out the two strings it returns: the file name and function name become real Go strings on the heap, so the resolved frame outlives the mapping even though everything else was read straight from the file's pages.

Here is what quietly falls out of this — the payoff for all three of Stage 2's leftover problems:

• The OS page cache becomes the cache tier, for free. The page cache is the RAM the operating system already uses to hold recently-read file data. Hot .tw files stay resident there because the kernel keeps touched pages around; cold ones are dropped under memory pressure by the kernel's own reclaim. The eviction problem is handed to an algorithm that has been tuned for fifty years.
• It survives restarts. The .tw files are on disk. A restart re-mmaps them in microseconds. No re-parse, no cold-start stampede.
• Replicas share. Put the prebuilt .tw blobs in shared object storage and every replica pulls the same 48 KB artifact instead of re-deriving it from 79 KB of map-plus-bundle. A fresh server is warm almost immediately.

In production these stack into one cascade, cheapest first:

1. Local mmap cache. The .tw file is on this box. Map it, search it, done in microseconds.
2. Object storage. Not local yet. Download the prebuilt 48 KB .tw, write it down, map it. Still no parse.
3. Rebuild. No .tw exists anywhere. This is the first crash of a brand-new release. Run the full 975 µs BuildTW once, then write the result to storage so steps 1 and 2 cover every replica after that.

The expensive path — Stage 1's full parse — is now the rare cold-cold case. It runs once per release, ever, instead of once per frame.

The shape of the win

The same LookupTW, fed from four different places:

What carried the whole thing was a data-layout choice in Stage 2, not a clever caching layer. Making the searchable structure a flat, pointer-free []byte instead of a Go object graph collapsed two problems into one: caching it in memory and storing it on disk became the same bytes handed to the same function. And the kernel's page cache, which nobody here wrote or tunes, does the eviction.

Does it survive contact with production?

A 44-nanosecond lookup is a nice number on a bench. The real question is what happens under a firehose: thousands of stack traces a second, hundreds of releases, a working set bigger than RAM. So the repo's standard processor benchmark was run against the best available baseline.

That baseline is Honeycomb's open-source OpenTelemetry symbolicator processors, the sourcemapprocessor for JavaScript and the dsymprocessor for iOS. Under the hood they wrap Sentry's symbolic, the Rust library behind Sentry's own symbolication. It is the engine plenty of production error tracking already runs on, and it is mature, well-optimized Rust. The only way a Go service beats that is by doing less work, not faster work — which is the entire design: on the hot path, skip the parse.

To compare the two symbolicators and nothing else, both run as OpenTelemetry Collector processors, the plugin slot that rewrites telemetry as it passes through. Both sit in the same collector, behind the same Go load generator firing crash reports over OTLP, in front of the same small Rust receiver that checks every frame came back resolved, on the same 8 vCPU / 32 GB box. Swap the processor, change nothing else. Load ramps from 8 to 256 connections, and the peak is kept.

Memory needs one caveat, because it is where the gap is widest. RAM is sampled from outside the process, since Go's own heap stats cannot see Honeycomb's parsed maps (they live on the C heap inside symbolic) and would over-count TracePath's mmap'd pages (the kernel reclaims those on demand). The corpus is a real minified bundle padded to a production-realistic size, so a cache miss costs what it would in production. Then three scenarios, each pointed at a different failure mode:

• hot: one bundle, always cache-warm. The ceiling: pure lookup throughput with zero misses.
• churn: 512 bundles cycling through a 128-entry cache, so almost every request misses. This is the real world, where traffic spans every release ever shipped, and it is where the cost of a miss gets exposed.
• oom: thousands of fat bundles on a small 2 vCPU, 8 GB box, a working set far bigger than RAM. The survival test: who stays up, and who the kernel kills. (The padding grows TracePath's token table too, so nothing is tilted in its favour.)

Four columns carry the story. max stacks/s is throughput, stack traces resolved per second. p99 ms is tail latency, the slowest 1% of requests — the part users feel. peak RSS MB is the most RAM the process ever held. symb% is the correctness check, the fraction of frames that came back resolved. outcome is whether the process lived.

JavaScript, TracePath against Honeycomb:

Correctness is a tie. Everything resolves ~100% of frames — that is the price of admission, not a differentiator. The differentiator is everything else:

• Churn throughput: 938 → 29,923 stacks/s, about 32x. This is the whole post in one number. On every cache miss the baseline re-parses the raw source map through symbolic, the Stage 1 tax paid forever, and it pins about one core: its average CPU sat near 104%, while TracePath spread across seven cores at ~690%. TracePath on a miss re-opens a precompiled .tw file and maps it. The 32x is the distance between "re-derive the structure" and "the structure is already the file".
• Latency: hot p99 373 ms → 4.1 ms (~90x), churn p99 2,962 ms → 17.7 ms (~168x). When the cache is warm both are quick-ish per core. At saturation the baseline's tail blows out to multiple seconds, and a symbolicator with a 3-second p99 is why errors show up late.
• Memory under churn: 4,491 MB → 361 MB, about 12x less. The baseline keeps the raw map JSON and the minified bundle resident per cache entry. TracePath throws both away the instant it has compiled the compact .tw. (Both hit ~6.8 GB peak in the oom run, but read that with the asterisk above: TracePath's are reclaimable page-cache pages, and it sustains 30k/s there where the baseline manages 938.)

Which knobs actually matter: parser and cache tier

Those TracePath rows were one configuration. The benchmark swept four — two parsers crossed with two cache tiers — and the full matrix says which knobs to turn:

Two independent decisions fall out, and the data picks both.

Disk over memory, because memory dies. This is the Stage 2 → Stage 3 jump, now with a body count. Both -mem variants get OOM-killed on the 8 GB box (oxc-mem at 40 seconds, goja-mem at 236). The in-memory cache holds every .tw on the Go heap, and the heap cannot be reclaimed under pressure, so the kernel kills the process. Both -disk variants survive the same workload, because an mmap'd .tw is just file pages the kernel can drop and re-fault on demand. That is eviction nobody has to write and nobody can get wrong. Disk also keeps churn memory flat — 361 MB against goja-mem's 1,309 MB — because the working set lives in the page cache, not the heap.

The bundle parser is the miss cost. It only runs on a cache miss (the BuildTW parse), so it is invisible in hot and decisive in churn. goja is a pure-Go JavaScript parser: zero dependencies, always available, and what the default build ships. oxc is a Rust parser linked in over cgo. On the churn run oxc-disk holds a 17.7 ms p99 against goja-disk's 22.7 ms, and the gap widens the harder you lean on misses. Pay the build complexity and the cold path gets cheaper.

Put them together and oxc + disk has the best latency, and it is the only corner that stays flat on memory and survives the OOM box. The default build is goja + memory with cgo off, because it needs no Rust toolchain and is plenty when the corpus fits in RAM. Switching is a build flag plus two environment variables:

# one-time: compile the Rust parser shim, then build with cgo + the oxc tag
#   scripts/build-oxc-shim.sh && go build -tags oxc ./...
 
SYMBOLICATOR_PARSER=oxc
SOURCEMAP_CACHE_TYPE=disk
SOURCEMAP_DISK_CACHE_PATH=/var/lib/tracepath/twcache
SOURCEMAP_DISK_CACHE_MAX_MB=2048

The full matrix, including Android and the sizing guidance behind those variables, is in the symbolicator performance docs.

The same trick, every language

Nothing in Stage 2 or Stage 3 was about JavaScript. The .tw format is a sorted token table plus two string tables. It does not care whether those tokens came from a JavaScript source map, a Dart .symbols file, or an iOS .dSYM — the last two being the debug files mobile platforms ship instead of source maps, both built on DWARF, the same debug format a native debugger reads. Every one of them is flattened into the same .tw on a cache miss, and from there it is the same binary search over the same mmap'd bytes. The disk tier (survival, flat memory, a cache that outlives a restart) comes for free the moment the symbolicator learns a new format.

The benchmark bears that out. Dart first (no Honeycomb column, since their processor does not support Dart or Flutter):

The same pattern holds. disk wins churn throughput 2.7x and holds memory at 138 MB against mem's 529 MB (and 2,165 MB on the oom soak). mem is much faster on the always-warm hot path — a 5.5 ms p99, where disk's tail runs far longer — but hot is the one scenario that never happens at scale. The flatten-once, mmap-forever pattern does the identical job for DWARF that it did for source maps.

And iOS, head-to-head with the dsymprocessor:

tracepath-ios-disk pushes ~4x the throughput at 94 MB against 9,727 MB, about 100x less memory. The baseline keeps each parsed symbolic archive resident; TracePath keeps a compact .tw and lets the kernel page it. Different binary format, same win, and not a line of new cache code to get it.

Why this matters for a stack trace you actually read

Symbolication is the least glamorous corner of an observability platform, and "close enough" is what fails you at 3 a.m., when a frame says n and you need validateCheckout. It happens once, at ingest: a report arrives, each trace is resolved frame by frame, and the resolved trace is what gets hashed for grouping and stored. Nothing is re-resolved when you open an issue later, which is what keeps issue pages fast — and it is also why uploading better debug info improves future events rather than past ones.

On TracePath that is the whole workflow: upload the source map, .dSYM, .symbols file or R8 mapping at build time, and exceptions arrive already readable, grouped by a hash computed over the resolved trace. The same engine runs as an OpenTelemetry Collector processor and as a Go package inside the backend, which is what the benchmark above harnesses. Setup is in the symbolicator docs, and what each plan includes is on the pricing page.

The part most underrated going in was mmap. Hand the kernel a file, get back a byte array that is the file, and the page cache does the eviction. The fallback, os.ReadFile, copies every byte onto the heap for the garbage collector to chase. Handing the cache to the operating system, instead of writing a worse one, is the lesson worth carrying to the next problem of this shape.

One idea is still open for a future format version. Every token is a full 24 bytes of absolute values, so a lookup scatters across the file even though mmap already hands back a whole 4 to 16 KB page each time it faults. Frame-of-reference encoding could tighten that: group tokens into fixed-length frames, keep one base per frame, and store each token as a small delta from it. The file would shrink, and a lookup would stay inside a single frame — one page fault instead of a dozen. How much that actually buys is untested, which is exactly what makes it interesting.

FAQ

What is symbolication?

Mapping a position in compiled or minified output back to the source location that produced it: a file, a line, a column and a function name. For JavaScript the mapping comes from a source map; for iOS from a .dSYM; for Dart from a .symbols file; for Android from an R8 mapping.txt.

Why cache a serialized byte buffer instead of the parsed object?

Three reasons. It is far more compact, it has no inner pointers so the garbage collector skips it entirely, and — the decisive one — a flat pointer-free buffer is already a file format, which makes the memory cache and the on-disk cache literally the same bytes read by the same function.

Why is mmap better than reading the file into memory?

os.ReadFile copies the whole file onto the heap, where the process owns it until it is dropped. mmap maps the file into the address space and lets the OS page cache hold the pages, so the kernel reclaims them under pressure instead of the kernel killing the process. In the benchmark, both in-memory variants were OOM-killed on an 8 GB box and both disk variants survived.

Does symbolication run on every view of an issue?

No. It runs once, at ingest, and the resolved trace is what is stored and hashed for grouping. That keeps issue pages fast, and it means a debug artifact uploaded after the fact improves future events, not existing ones.

Does this design work for languages other than JavaScript?

Yes, and that is the point of the format. Source maps, DWARF-based .dSYM and Dart symbols, and R8 mappings all flatten into the same sorted token table, so the cache, the binary search and the mmap tier are shared across every language the symbolicator supports.

Built something that turned out faster on disk than in memory because the on-disk shape was the right shape? We would like to hear about it: [email protected].