How to profile Go applications inside a docker container
August 11, 2017 - Jan Pieter Bruins Slot
This article has been updated and you can view its updated version here
In this post I’ll give a quick overview of several methods you can use for profiling/debugging Go applications that are running in a docker container. To get a more in-depth overview of the several methods, I’ve added the source links you can reference.
general sources
- Defer Panic - understanding golang memory usage
- golang.org - profiling go programs
- Jimmy-Xu - pprof in docker daemon
pprof
When you want to use pprof
to profile your Go applications, that are running in
a container, we need to make sure we turn on the internal pprof
HTTP endpoints.
We’ll do this by updating the source code of your application. What follows is
an example of how you would enable those endpoints.
// main.go
import (
_ "net/http/pprof"
_ "expvar"
"github.com/gorilla/mux"
)
func main() {
// OPTION 1: Add this to your application
go func() {
http.ListenAndServe("0.0.0.0:6060", nil)
}()
// OPTION 2: When using gorilla mux you can do the following:
router := mux.NewRouter()
router.HandleFunc("/debug/pprof/", pprof.Index)
router.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
router.HandleFunc("/debug/pprof/profile", pprof.Profile)
router.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
router.HandleFunc("/debug/pprof/trace", pprof.Trace)
router.Handle("/debug/pprof/goroutine", pprof.Handler("goroutine"))
router.Handle("/debug/pprof/heap", pprof.Handler("heap"))
router.Handle("/debug/pprof/threadcreate", pprof.Handler("threadcreate"))
router.Handle("/debug/pprof/block", pprof.Handler("block"))
router.Handle("/debug/vars", http.DefaultServeMux)
http.ListenAndServe("0.0.0.0:6060", router)
}
Now, we need to make sure that we’re able to access those endpoints. When
you’re using docker-compose update your docker-compose.yaml
file as follows:
// docker-compose.yaml
version: "3"
services:
app:
build: ./build-dir
ports:
- "6060:6060"
Once you’ve updated the files you’ll be able to access the endpoints by using
your browser and to go to
http://localhost:6060/debug/pprof/,
or you can use go tool pprof
:
# Heap profile
#
# optional flags:
# –alloc_space tells you how many megabytes have been allocated.
# –inuse_space tells you know how many are still in use.
$ go tool pprof http://localhost:6060/debug/pprof/heap
# CPU profile
$ go tool pprof http://localhost:6060/debug/pprof/profile
# Using trace tool
$ curl http://localhost:6060/debug/pprof/trace?seconds=5 > trace.out
$ go tool trace trace.out
# Create a heap dump and analyze it locally
$ curl http://localhost:6060/debug/pprof/heap > heap.out
$ go tool pprof -runtime --inuse_space [your-binary-here] heap.out
One thing to remember is: if you’re making cpu profiles that takes a certain
amount of time you need to specify that your server is able to handle those
long request times. You typically want to add the following to your
http.Server
:
srv := &http.Server{
WriteTimeout: 30 * time.Second,
}
sources
- Go - Package profile
- Go Blog - Profiling Go Programs
- Svet Ralchev - Performance and memory analysis of Golang programs
- Stack Overflow - Profiling Go web application built with Gorilla’s mux with net/http/pprof
- Brad Fritzpatrick - Profiling & Optimizing in Go
- Pusher - go tool trace
- Pusher - Introduction to go tool trace
- Rhys Hiltner - Go’s execution tracer
- Defer Panic - Goroutine Tracing
gcvis
To use gcvis to profile our applications we need to first install it:
go get -u github.com/davecheney/gcvis
Now update your docker-compose.yaml
file, important is to set the GODEBUG
environment variable:
// docker-compose.yaml
version: "3"
services:
app:
build: ./build-dir
environment:
- GODEBUG=gctrace=1
Once you’ve updated the docker-compose.yaml
file, run your container with the
command below, and gcvis will start a browser for you automatically.
$ docker-compose up app | gcvis
sources
go-torch
When using go-torch, first update your code to enable the pprof endpoints.
Reference the pprof
section on how to do that. Next, you need to
get go-torch:
$ go get -u github.com/uber/go-torch
$ git clone git@github.com:brendangregg/FlameGraph.git
$ export PATH=$PATH:/path/to/FlameGraph
And you can run it as follows:
$ go-torch -u http://localhost:6060/