How to Use flame graphs to See Where Time Goes With Examples

A profiler's text report lists functions and percentages, and it hides the structure: which caller makes the expensive calls and where the time sits in the tree. A flame graph draws that tree in one picture, and a pattern becomes visible in seconds that a table takes minutes to reveal. It is the usual next step after perf stat and perf record. Examples use perf 6.12, the FlameGraph scripts and inferno 0.11.

Record and render

git clone --depth 1 https://github.com/brendangregg/FlameGraph.git
sudo perf record -F 99 -a -g -- sleep 30
sudo perf script | ./FlameGraph/stackcollapse-perf.pl | ./FlameGraph/flamegraph.pl > cpu.svg

-a records all CPUs, so it shows the whole system for 30 seconds while the problem happens; use -p PID for one process or run a command in place of sleep. perf script prints each sample with its stack, stackcollapse-perf.pl folds identical stacks into lines with a count, and flamegraph.pl draws the SVG. Open it in a browser, where it is interactive: hover shows names and percentages, a click zooms into a frame, and a search highlights matches.

The Rust tool inferno does the same job faster: perf script | inferno-collapse-perf | inferno-flamegraph > cpu.svg.

Read it right

• The x-axis is not time. Frames are sorted alphabetically within each level, and the width is the number of samples that included that frame.
• Each box is a function; the box below it is its caller. The top edge shows what was running when the sample was taken.
• Look for wide plateaus on top: a function that is wide and has nothing above it is spending its own time. A wide function with many narrow boxes above it is spending time in callees.
• Tall towers are deep call chains and only matter if they are also wide.

The first thing to find is the widest box near the top, then the stack below it to see who called it. That tells you whether to fix the function or its caller, which is often the larger win: a function that takes 40 percent because it is called in a loop one level up.

When stacks are broken

sudo perf record -F 99 -a --call-graph dwarf -- sleep 10
sudo perf record -F 99 -a --call-graph lbr -- sleep 10

A graph full of [unknown] frames or stacks cut off after one or two levels means the unwinder could not follow the stack. The default method walks frame pointers, which many binaries omit. --call-graph dwarf unwinds with debugging information and gives complete stacks, with much bigger recordings and more overhead. Installing debug symbols for the binaries and libraries you care about turns hex addresses into names. For interpreted languages, the native stack shows the interpreter, not your code: use the language's profiler.

Language-specific

py-spy record -o profile.svg --pid 4121 --duration 30
py-spy record -o profile.svg -- python3 script.py

py-spy samples a Python process from outside, without changing it, and writes a flame graph of Python functions. Go has go tool pprof -http=:8080, which includes a flame graph view; Java has async-profiler. These show your code's frames, which perf cannot.

Compare two runs

./FlameGraph/difffolded.pl before.folded after.folded | ./FlameGraph/flamegraph.pl > diff.svg

A differential graph colors frames red where the code got more expensive and blue where it got cheaper, which is a clear way to show the effect of a change. Save the folded text files from each run.

Where this does not apply

A CPU flame graph shows time on the CPU only. A program that is slow because it waits for the disk, the network or a lock has little CPU use, and its graph is flat or misleading; off-CPU analysis (perf sched, offcputime from bcc) shows the waits. Short runs give few samples and noisy graphs, so record long enough, at least a few hundred samples. Sampling at 99 Hz misses brief events. See profiling a Bash script for shell programs.

For a first look at where a slow job spends its time before drawing anything, profiling a Bash script and finding the slow part of a pipeline cover the cheaper tools. A flame graph is the right next step when those point at a program and not at the shell.

What the picture actually encodes

A flame graph is a visualization of sampled stacks. Each box is a function, the box beneath it is its caller, and the width of a box is the share of samples in which that function was on the stack. The vertical axis is stack depth and carries no meaning of time. The horizontal axis is not time either: boxes at the same level are sorted alphabetically so that identical stacks merge, which means left to right order tells you nothing about what ran first. The part to look for is a wide box with little above it, a plateau, since that is a function where the program spent time itself and not in functions it called. A wide box with many narrow children above it is a function that mostly delegates, and its cost is spread over the children. Reading a flame graph starts with the widest boxes near the top edge and asks why the program is there so much.

Wide does not mean wrong. A function that takes forty percent of the samples may be doing exactly the work the program exists to do, and the question is whether that share is reasonable for the task, or whether the work is repeated, unnecessary or done in a slow way.

Making broken stacks whole

If the graph shows towers cut off after one or two frames, or large boxes labelled unknown, the stack walking failed. The cause is almost always missing frame pointers or missing symbols. Compiled programs built with -fno-omit-frame-pointer give clean stacks with the default method, and Go has used frame pointers for years, so Go programs usually just work. Several runtimes need help to expose their frames. Python 3.12 added support for the Linux perf tool through an option that generates small trampolines so each Python function appears by name, enabled with the -X perf flag or an environment variable. Node.js can write a map file of its generated code with --perf-basic-prof. Java needs an agent such as async-profiler, which also handles the inlining that hides frames. Without these, the graph shows the interpreter's internals and not your code, and the picture teaches you about the interpreter instead.

The same applies to libraries: a profile of a program that spends its time inside a shared library shows only the library's address range unless its symbols are available, and installing the debug package for that library fixes it.

Comparing and choosing what to record

A single graph tells you where time goes. A comparison tells you what changed, which is more useful for regression hunting. Record the same workload before and after a change, collapse both, and use the differential tool in the FlameGraph repository, difffolded.pl, to render a graph colored by the change, red for more samples and blue for fewer. Normalize for different run lengths, which the tool does with an option, or compare runs of equal work. Remember that a flame graph made from on-CPU sampling shows only time spent running. A program that is slow because it waits for disk, a lock or the network looks nearly empty, and for that you need an off-CPU profile, which records where threads blocked and for how long, using tracing instead of sampling. Choose by the symptom: high CPU use means on-CPU, and slow with idle CPUs means off-CPU. Check top or pidstat first, so you do not spend an hour tuning code that was not the bottleneck.

See Brendan Gregg's flame graph page and inferno.