Find a Go Goroutine Leak or Scheduler Stall With Profiles and Execution Traces

A Go service keeps serving traffic while goroutine count and memory climb, then latency spikes during a deployment or dependency slowdown. A high count alone is not proof of a leak: workers, keep-alive loops, and in-flight requests may be legitimate. The diagnostic question is whether one stack cohort grows and fails to leave after its work, deadline, or owner ends.

Use the sequence below as a diagnostic method, not as a promise that one setting fits every host; verify its example paths, privileges, and backup assumptions while working to distinguish legitimate concurrency from goroutines stuck on channels, locks, I/O, or missing cancellation by comparing profiles and a bounded runtime trace.

Likely failure paths

• A producer exits while a consumer remains blocked on a channel, or a send has no receiver and no cancellation branch.
• An outbound request, stream, timer, ticker, response body, or worker lifecycle lacks a deadline and deterministic cleanup.
• Lock contention, runnable-queue pressure, blocking syscalls, or CPU saturation delays goroutines that are not actually leaked.

The branches are ordered to protect the strongest evidence around this possibility: a producer exits while a consumer remains blocked on a channel, or a send has no receiver and no cancellation branch. The observed scope and logs—not a familiar-looking error screen—decide which one applies.

Go runtime incidents are best separated into application blocking, scheduler delay, allocation pressure, garbage collection, and external resource waits; counts and averages can hide a growing cohort, so compare profiles and traces from equivalent traffic windows before changing runtime settings; in this guide, the practical goal is to distinguish legitimate concurrency from goroutines stuck on channels, locks, I/O, or missing cancellation by comparing profiles and a bounded runtime trace.

A controlled investigation

1. Graph runtime.NumGoroutine with request rate, active work, heap, file descriptors, and dependency latency; look for a count that does not return after the load drains.
2. Capture goroutine profiles at a healthy baseline and after growth, then compare stack signatures and counts rather than reading one enormous dump.
3. Collect CPU, block, and mutex profiles when their sampling is intentionally enabled, and take a short execution trace during a reproducible window to see runnable, blocked, syscall, network, and GC delays.
4. Audit every goroutine start for an owner, stop condition, context propagation, channel closure rule, ticker/timer stop, response-body close, and WaitGroup completion.

The sequence moves from observation toward intervention. Preserve the result of the final check—audit every goroutine start for an owner, stop condition, context propagation, channel closure rule, ticker/timer stop, response-body close, and WaitGroup completion—because it provides a useful comparison after the repair.

curl -fsS 'http://127.0.0.1:6060/debug/pprof/goroutine' -o goroutine-before.pb.gz
# apply the controlled workload, then allow its cancellation grace period
curl -fsS 'http://127.0.0.1:6060/debug/pprof/goroutine' -o goroutine-after.pb.gz

go tool pprof -top goroutine-after.pb.gz
curl -fsS 'http://127.0.0.1:6060/debug/pprof/trace?seconds=5' -o runtime.trace
go tool trace runtime.trace

Make the smallest durable change

1. Give long-lived goroutines explicit ownership and propagate cancellation; in channel operations, select on ctx.Done() so shutdown does not depend on another component making progress.
2. Close response bodies and streams, stop tickers and timers, bound queues, and ensure workers exit after their producer or service lifecycle ends.
3. If the trace shows scheduler or lock contention rather than a leak, shorten critical sections, remove accidental blocking, and enforce downstream concurrency limits before adding workers.

Before applying “Give long-lived goroutines explicit ownership and propagate cancellation; in channel operations, select on ctx.Done() so shutdown does not depend on another component making progress,” name its rollback point and the evidence that will count as success. Afterward, repeat the original request and specifically check whether you can run repeated load-and-drain cycles and confirm the same goroutine stack cohorts return to a stable baseline within a defined grace period; a changed symptom at that point is new evidence, not permission to make several more changes at once.

Close the incident with evidence

• Run repeated load-and-drain cycles and confirm the same goroutine stack cohorts return to a stable baseline within a defined grace period.
• Compare pprof and trace evidence after the patch while holding traffic, dependency behavior, Go version, and GOMAXPROCS constant.
• Exercise cancellation, timeout, peer disconnect, failed startup, and graceful shutdown paths under the race detector in tests where practical.

One successful refresh is not closure. Keep the incident open until you can also compare pprof and trace evidence after the patch while holding traffic, dependency behavior, Go version, and GOMAXPROCS constant, adjacent paths have not regressed, temporary diagnostics are gone, and another operator can explain what changed.

Prepare a useful escalation if the boundary is outside your control

A Go escalation should include the Go version and build settings, deployment and traffic window, GOMAXPROCS and memory limit, goroutine/heap/CPU profile summaries, a short execution trace when safe, and the smallest reproducible workload; profiles can contain URLs or labels, so restrict the endpoint and sanitize artifacts; include the result of this first observation: graph runtime.NumGoroutine with request rate, active work, heap, file descriptors, and dependency latency; look for a count that does not return after the load drains.

State what was tested, including the result of “Graph `runtime.NumGoroutine` with request rate, active work, heap, file descriptors, and dependency latency; look for a count that does not return after the load drains,” and what changed between attempts; evidence tied to that observation is safer and more actionable than granting broad access or sending an unnecessary full database export.

Do not trade visibility for a green screen

• Do not expose net/http/pprof on an internet-facing listener; profiles can reveal endpoints, labels, stack details, and operational state.
• Do not call runtime.GC or raise resource limits to hide a growing goroutine cohort; goroutines retain referenced memory and external resources.

Primary references

• Go diagnostics — official reference consulted for this guide.
• Go runtime scheduler notes — official reference consulted for this guide.
• Go data race detector — official reference consulted for this guide.

Editorial note: The scenario above illustrates how to approach “Give long-lived goroutines explicit ownership and propagate cancellation; in channel operations, select on ctx.Done() so shutdown does not depend on another component making progress”; it is a documented example, not a claim about a reader’s server, so verify the cited documentation, take the appropriate backup, and follow the real environment’s access and change-control rules.