<HTML>
<HEAD>
<title>pprof and Remote Servers</title>
</HEAD>
<BODY>
<h1><code>pprof</code> and Remote Servers</h2>
<p>In mid-2006, we added an experimental facility to <A
HREF="cpu_profiler.html">pprof</A>, the tool that analyzes CPU and
heap profiles. This facility allows you to collect profile
information from running applications. It makes it easy to collect
profile information without having to stop the program first, and
without having to log into the machine where the application is
running. This is meant to be used on webservers, but will work on any
application that can be modified to accept TCP connections on a port
of its choosing, and to respond to HTTP requests on that port.</p>
<p>We do not currently have infrastructure, such as apache modules,
that you can pop into a webserver or other application to get the
necessary functionality "for free." However, it's easy to generate
the necessary data, which should allow the interested developer to add
the necessary support into his or her applications.</p>
<p>To use <code>pprof</code> in this experimental "server" mode, you
give the script a host and port it should query, replacing the normal
commandline arguments of application + profile file:</p>
<pre>
% pprof internalweb.mycompany.com:80
</pre>
<p>The host must be listening on that port, and be able to accept HTTP/1.0
requests -- sent via <code>wget</code> and <code>curl</code> -- for
several urls. The following sections list the urls that
<code>pprof</code> can send, and the responses it expects in
return.</p>
<p>Here are examples that pprof will recognize, when you give them
on the commandline, are urls. In general, you
specify the host and a port (the port-number is required), and put
the service-name at the end of the url.:</p>
<blockquote><pre>
http://myhost:80/pprof/heap # retrieves a heap profile
http://myhost:8008/pprof/profile # retrieves a CPU profile
http://myhost:80 # retrieves a CPU profile (the default)
http://myhost:8080/ # retrieves a CPU profile (the default)
myhost:8088/pprof/growth # "http://" is optional, but port is not
http://myhost:80/myservice/pprof/heap # /pprof/heap just has to come at the end
</pre></blockquote>
<h2> <code><b>/pprof/heap</b></code> </h2>
<p><code>pprof</code> asks for the url <code>/pprof/heap</code> to
get heap information. The actual url is controlled via the variable
<code>HEAP_PAGE</code> in the <code>pprof</code> script, so you
can change it if you'd like.</p>
<p>The server should respond by calling</p>
<pre>
MallocExtension::instance()->GetHeapSample(&output);
</pre>
<p>and sending <code>output</code> back as an HTTP response to
<code>pprof</code>. <code>MallocExtension</code> is defined in the
header file <code>google/malloc_extension.h</code>.</p>
<p>Here's an example, from an actual Google webserver, of what the
output should look like:</p>
<pre>
heap profile: 9369: 126987529 [ 9369: 126987529] @ heap
2: 1024 [ 2: 1024] @ 0x87da913 0x8923ad4 0x891d4c2 0x892de12 0x8930519 0x83a16c2 0x836cb38 0x834cd1c 0x8349ba5 0x10a3177 0x8349961
1: 36 [ 1: 36] @ 0x87da913 0x83a0929 0x836cb38 0x834cd1c 0x8349ba5 0x10a3177 0x8349961
308: 10092544 [ 308: 10092544] @ 0x87da913 0x8970d66 0x8970e64 0x896e8e2 0x88e69d2 0x88e6add 0x88e6dec 0x88e7384 0x88e73fa 0x8838793 0x8838b36 0x88395f8 0x88f5a4b 0x890d03a 0x890d65a 0x8917666 0x890d1f3 0x890e6e4 0x8349c1b 0x10a3177 0x8349961
[...]
</pre>
<h2> <code><b>/pprof/growth</b></code> </h2>
<p><code>pprof</code> asks for the url <code>/pprof/growth</code> to
get heap-profiling delta (growth) information. The actual url is
controlled via the variable <code>GROWTH_PAGE</code> in the
<code>pprof</code> script, so you can change it if you'd like.</p>
<p>The server should respond by calling</p>
<pre>
MallocExtension::instance()->GetHeapGrowthStacks(&output);
</pre>
<p>and sending <code>output</code> back as an HTTP response to
<code>pprof</code>. <code>MallocExtension</code> is defined in the
header file <code>google/malloc_extension.h</code>.</p>
<p>Here's an example, from an actual Google webserver, of what the
output should look like:</p>
<pre>
heap profile: 741: 812122112 [ 741: 812122112] @ growth
1: 1572864 [ 1: 1572864] @ 0x87da564 0x87db8a3 0x84787a4 0x846e851 0x836d12f 0x834cd1c 0x8349ba5 0x10a3177 0x8349961
1: 1048576 [ 1: 1048576] @ 0x87d92e8 0x87d9213 0x87d9178 0x87d94d3 0x87da9da 0x8a364ff 0x8a437e7 0x8ab7d23 0x8ab7da9 0x8ac7454 0x8348465 0x10a3161 0x8349961
[...]
</pre>
<h2> <code><b>/pprof/profile</b></code> </h2>
<p><code>pprof</code> asks for the url
<code>/pprof/profile?seconds=XX</code> to get cpu-profiling
information. The actual url is controlled via the variable
<code>PROFILE_PAGE</code> in the <code>pprof</code> script, so you can
change it if you'd like.</p>
<p>The server should respond by calling
<code>ProfilerStart(filename)</code>, continuing to do its work, and
then, XX seconds later, calling <code>ProfilerStop()</code>. (These
functions are declared in <code>google/profiler.h</code>.) The
application is responsible for picking a unique filename for
<code>ProfilerStart()</code>. After calling
<code>ProfilerStop()</code>, the server should read the contents of
<code>filename</code> and send them back as an HTTP response to
<code>pprof</code>.</p>
<p>Obviously, to get useful profile information the application must
continue to run in the XX seconds that the profiler is running. Thus,
the profile start-stop calls should be done in a separate thread, or
be otherwise non-blocking.</p>
<p>The profiler output file is binary, but near the end of it, it
should have lines of text somewhat like this:</p>
<pre>
01016000-01017000 rw-p 00015000 03:01 59314 /lib/ld-2.2.2.so
</pre>
<h2> <code><b>/pprof/contention</b></code> </h2>
<p>This is intended to be able to profile (thread) lock contention in
addition to CPU and memory use. It's not yet usable.</p>
<h2> <code><b>/pprof/cmdline</b></code> </h2>
<p><code>pprof</code> asks for the url <code>/pprof/cmdline</code> to
figure out what application it's profiling. The actual url is
controlled via the variable <code>PROGRAM_NAME_PAGE</code> in the
<code>pprof</code> script, so you can change it if you'd like.</p>
<p>The server should respond by reading the contents of
<code>/proc/self/cmdline</code>, converting all internal NUL (\0)
characters to newlines, and sending the result back as an HTTP
response to <code>pprof</code>.</p>
<p>Here's an example return value:<p>
<pre>
/root/server/custom_webserver
80
--configfile=/root/server/ws.config
</pre>
<h2> <code><b>/pprof/symbol</b></code> </h2>
<p><code>pprof</code> asks for the url <code>/pprof/symbol</code> to
map from hex addresses to variable names. The actual url is
controlled via the variable <code>SYMBOL_PAGE</code> in the
<code>pprof</code> script, so you can change it if you'd like.</p>
<p>This is perhaps the hardest request to write code for, because
it must accept POST requests. This means that after the HTTP headers,
pprof will pass in a list of hex addresses connected by
<code>+</code>, like so:</p>
<pre>
curl -d '0x0824d061+0x0824d1cf' http://remote_host:80/pprof/symbol
</pre>
<p>The server should read the POST data, which will be in one line,
and for each hex value, should write one line of output to the output
stream, like so:</p>
<pre>
<hex address><tab><function name>
</pre>
<p>For instance:</p>
<pre>
0x08b2dabd _Update
</pre>
<p>The other reason this is the most difficult request to implement,
is that the application will have to figure out for itself how to map
from address to function name. One possibility is to run <code>nm -C
-n <program name></code> to get the mappings, either statically
(say at program-compile time), or dynamically, by having the
application call out to <code>nm</code> for every
<code>pprof/symbol</code> call (presumably with some caching!).</p>
<p><code>pprof</code> itself does just this for local profiles (not
ones that talk to remote servers); look at the subroutine
<code>GetProcedureBoundaries</code>.</p>
<hr>
Last modified: Mon Jun 12 21:30:14 PDT 2006
</body>
</html>
distrib
>
Mandriva
>
2010.0
>
i586
>
media
>
contrib-release
>
by-pkgid
>
d8a194b87cd3249c72f0ca8dc4d3e593
>
files
>
33
