How to profile Go applications inside a docker container

August 11, 2017 - Jan Pieter Bruins Slot

This article has been updated and you can view its updated version here

In this post I’ll give a quick overview of several methods you can use for profiling/debugging Go applications that are running in a docker container. To get a more in-depth overview of the several methods, I’ve added the source links you can reference.

general sources

pprof

When you want to use pprof to profile your Go applications, that are running in a container, we need to make sure we turn on the internal pprof HTTP endpoints. We’ll do this by updating the source code of your application. What follows is an example of how you would enable those endpoints.

// main.go
import (
    _ "net/http/pprof"
    _ "expvar"
    "github.com/gorilla/mux"
)

func main() {

    // OPTION 1: Add this to your application
    go func() {
        http.ListenAndServe("0.0.0.0:6060", nil)
    }()

    // OPTION 2: When using gorilla mux you can do the following:
    router := mux.NewRouter()
    router.HandleFunc("/debug/pprof/", pprof.Index)
    router.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
    router.HandleFunc("/debug/pprof/profile", pprof.Profile)
    router.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
    router.HandleFunc("/debug/pprof/trace", pprof.Trace)
    router.Handle("/debug/pprof/goroutine", pprof.Handler("goroutine"))
    router.Handle("/debug/pprof/heap", pprof.Handler("heap"))
    router.Handle("/debug/pprof/threadcreate", pprof.Handler("threadcreate"))
    router.Handle("/debug/pprof/block", pprof.Handler("block"))
    router.Handle("/debug/vars", http.DefaultServeMux)
    http.ListenAndServe("0.0.0.0:6060", router)
}

Now, we need to make sure that we’re able to access those endpoints. When you’re using docker-compose update your docker-compose.yaml file as follows:

// docker-compose.yaml
version: "3"
services:
  app:
    build: ./build-dir
    ports:
      - "6060:6060"

Once you’ve updated the files you’ll be able to access the endpoints by using your browser and to go to http://localhost:6060/debug/pprof/, or you can use go tool pprof:

# Heap profile
#
# optional flags:
#   –alloc_space tells you how many megabytes have been allocated.
#   –inuse_space tells you know how many are still in use.
$ go tool pprof http://localhost:6060/debug/pprof/heap

# CPU profile
$ go tool pprof http://localhost:6060/debug/pprof/profile

# Using trace tool
$ curl http://localhost:6060/debug/pprof/trace?seconds=5 > trace.out
$ go tool trace trace.out

# Create a heap dump and analyze it locally
$ curl http://localhost:6060/debug/pprof/heap > heap.out
$ go tool pprof -runtime --inuse_space [your-binary-here] heap.out

One thing to remember is: if you’re making cpu profiles that takes a certain amount of time you need to specify that your server is able to handle those long request times. You typically want to add the following to your http.Server:

srv := &http.Server{
    WriteTimeout: 30 * time.Second,
}

sources

gcvis

To use gcvis to profile our applications we need to first install it:

go get -u github.com/davecheney/gcvis

Now update your docker-compose.yaml file, important is to set the GODEBUG environment variable:

// docker-compose.yaml
version: "3"
services:
  app:
    build: ./build-dir
  environment:
    - GODEBUG=gctrace=1

Once you’ve updated the docker-compose.yaml file, run your container with the command below, and gcvis will start a browser for you automatically.

$ docker-compose up app | gcvis

sources

go-torch

When using go-torch, first update your code to enable the pprof endpoints. Reference the pprof section on how to do that. Next, you need to get go-torch:

$ go get -u github.com/uber/go-torch
$ git clone git@github.com:brendangregg/FlameGraph.git
$ export PATH=$PATH:/path/to/FlameGraph

And you can run it as follows:

$ go-torch -u http://localhost:6060/

sources