Remote Profiling of go Programs

← Blog Home

RSS

By

Introduction

The go programming language’s profiling system supports an interesting feature: http access. If your application already has an HTTP interface, and it uses the net/http default multiplexer (i.e., it calls http.Handle() and http.ListenAndServe()), enabling http profiling is as simple as:

import _ "net/http/pprof"

and the profiling APIs will be visible under /debug/pprof/ alongside your application’s URL paths, where you can access them with:

go tool pprof http://host:port/debug/pprof/profile

While this is a slick way of profiling a running program, the fact that this capability is installed by an import makes turning it off and on cumbersome: the import either has to be added and removed, or placed in a separate file with build tags to turn compilation on and off. If left in shipped code, it has the side effect of publishing the profiling hooks (with no access control) on the application http listener. This is far from ideal, but there are a few good workarounds.

Access Control

A simple and effective option is to put the pprof http server on a separate port on localhost, separate from the application http server. If the application does not use the http default multiplexer, starting the profiling http server is as simple as:

	go func() {
		log.Fatal(http.ListenAndServe("localhost:8081"))
	}()

Otherwise, you can run the following additional setup prior to any http.Handle() calls:

	pprofMux := http.DefaultServeMux
	http.DefaultServeMux = http.NewServeMux()
	go func() {
		log.Fatal(&http.Server{
			Addr: "localhost:8081",
			Handler: pprofMux,
		}.ListenAndServe())
	}()

Either of the above snippets can be conditionally run, so the profiling server may be turned on or off by command line flags or application configuration. However, since they are now only locally accessible, there is no downside to leaving them on.

Remote Control

Now that you have the profiling hooks safely exposed via a localhost-only interface, you can invoke the following:

go tool pprof http://localhost:8081/debug/pprof/profile

and profile the program with the local go tools. However, since go compiles to a static binary which can be installed without any go-related dependencies, chances are you don’t have go tools where your program is running. But you probably have SSH, so:

ssh -L 8081:localhost:8081 user@remote-host

or if you are already logged in:

~C
-L 8081:localhost:8081

will make the local go tool pprof on your local machine profile the remote code over the ssh tunnel.

Conclusion

If you are running go services on servers with only trusted local users, exposing the profiling http port through port forwarding limits access to the trusted local users, making it practical to leave profiling on as just another administrative feature of the system. This allows the developer to observe and/or troubleshoot the program under a real-world, and real, load.

Chris Mikkelson is a Senior Distributed Systems Engineer at Farsight Security, Inc.

← Blog Home

Protect against cybercriminal activity in real-time.

Request demo

Email: sales@farsightsecurity.com Phone: +1-650-489-7919