With application development, a common step is to improve runtime performance. To not waste time on optimizing functions which are rarely used, one needs to know in which parts of the program most of the time is spent.

This is done with a technique called profiling. The program is run under control of a profiling tool, which gives the time distribution of executed functions in the run. After examination of the program's profile, it should be clear if and where optimization is useful. Afterwards, one should verify any runtime changes by another profile run.

Most widely known is the GCC profiling tool GProf: one needs to compile an application with the compiler option -pg. Running the program generates a file gmon.out, which can be transformed into human readable form with the command line tool gprof. A disadvantage here is the the need to recompile everything, and also the need to statically link the executable.

Another profiling tool is Cachegrind, part of Valgrind. It uses the processor emulation of Valgrind to run the executable, and catches all memory accesses, which are used to drive a cache simulator. The program does not need to be recompiled, it can use shared libraries and plugins, and the profile measurement doesn't influence the memory access behaviour. The trace includes the number of instruction/data memory accesses and 1st/2nd level cache misses, and relates it to source lines and functions of the run program. A disadvantage is the slowdown involved in the processor emulation, around 50 times slower.

Cachegrind can only deliver a flat profile. There is no call relationship among the functions of an application stored. Thus, inclusive costs, i.e. costs of a function including the cost of all functions called from there, cannot be calculated. Callgrind extends Cachegrind by including call relationship and exact event counts spent while doing a call.

Because Callgrind (and Cachegrind) is based on simulation, the slowdown due to processing the synthetic runtime events does not influence the results. See Usage for more details on the possibilities.

To start a profile run for a program, execute:

callgrind [callgrind options] your-program [program options]

While the simulation is running, you can observe execution with

callgrind_control -b

This will print out a current backtrace. To annotate the backtrace with event counts, run

callgrind_control -e -b

After program termination, a profile data file named callgrind.out.pid is generated with pid being the process ID of the execution of this profile run.

The data file contains information about the calls made in the program among the functions executed, together with events of type Instruction Read Accesses (Ir).

If you are additionally interested in measuring the cache behaviour of your program, use Callgrind with the option --simulate-cache=yes. This will further slow down the run approximately by a factor of 2.

If the program section you want to profile is somewhere in the middle of the run, it is beneficial to fast forward to this section without any profiling at all, and switch it on later. This is achieved by using --instr-atstart=no and interactively use callgrind_control -i on before the interesting code section is about to be executed.

If you want to be able to see assembler annotation, specify --dump-instr=yes. This will produce profile data at instruction granularity. Note that the resulting profile data can only be viewed with KCachegrind. For assembler annotation, it also is interesting to see more details of the control flow inside of functions, ie. (conditional) jumps. This will be collected by further specifying --collect-jumps==yes.

Often, you aren't interested in time characteristics of a full program run, but only of a small part of it (e.g. execution of one algorithm). If there are multiple algorithms or one algorithm running with different input data, it's even useful to get different profile information for multiple parts of one program run.

Profile data files have names of the form

callgrind.out.pid.part-threadID

where pid is the PID of the running program, part is a number incremented on each dump (".part" is skipped for the dump at program termination), and threadID is a thread identification ("-threadID" is only used if you request dumps of individual threads with --separate-threads=yes).

There are different ways to generate multiple profile dumps while a program is running under Callgrind's supervision. Nevertheless, all methods trigger the same action, which is "dump all profile information since the last dump or program start, and zero cost counters afterwards". To allow for zeroing cost counters without dumping, there is a second action "zero all cost counters now". The different methods are:

If you are running a multi-threaded application and specify the command line option --separate-threads=yes, every thread will be profiled on its own and will create its own profile dump. Thus, the last two methods will only generate one dump of the currently running thread. With the other methods, you will get multiple dumps (one for each thread) on a dump request.

For aggregating events (function enter/leave, instruction execution, memory access) into event numbers, first, the events must be recognizable by Callgrind, and second, the collection state must be switched on.

Event collection is only possible if instrumentation for program code is switched on. This is the default, but for faster execution (identical to valgrind --tool=none), it can be switched off until the program reaches a state in which you want to start collecting profiling data. Callgrind can start without instrumentation by specifying option --instr-atstart=no. Instrumentation can be switched on interactively with

callgrind_control -i on

and off by specifying "off" instead of "on". Furthermore, instrumentation state can be programatically changed with the macros CALLGRIND_START_INSTRUMENTATION; and CALLGRIND_STOP_INSTRUMENTATION;.

In addition to enabling instrumentation, you must also enable event collection for the parts of your program you are interested in. By default, event collection is enabled everywhere. You can limit collection to specific function(s) by using --toggle-collect=funcprefix. This will toggle the collection state on entering and leaving the specified functions. When this option is in effect, the default collection state at program start is "off". Only events happening while running inside of functions starting with funcprefix will be collected. Recursive calls of functions with funcprefix do not trigger any action.

It is important to note that with instrumentation switched off, the cache simulator cannot see any memory access events, and thus, any simulated cache state will be frozen and wrong without instrumentation. Therefore, to get useful cache events (hits/misses) after switching on instrumentation, the cache first must warm up, probably leading to many cold misses which would not have happened in reality. If you do not want to see these, start event collection a few million instructions after you have switched on instrumentation

.

Each group of functions with any two of them happening to have a call chain from one to the other, is called a cycle. For example, with A calling B, B calling C, and C calling A, the three functions A,B,C build up one cycle.

If a call chain goes multiple times around inside of a cycle, with profiling, you can not distinguish event counts coming from the first round or the second. Thus, it makes no sense to attach any inclusive cost to a call among functions inside of one cycle. If "A > B" appears multiple times in a call chain, you have no way to partition the one big sum of all appearances of "A > B". Thus, for profile data presentation, all functions of a cycle are seen as one big virtual function.

Unfortunately, if you have an application using some callback mechanism (like any GUI program), or even with normal polymorphism (as in OO languages like C++), it's quite possible to get large cycles. As it is often impossible to say anything about performance behaviour inside of cycles, it is useful to introduce some mechanisms to avoid cycles in call graphs. This is done by treating the same function in different ways, depending on the current execution context, either by giving them different names, or by ignoring calls to functions.

There is an option to ignore calls to a function with --fn-skip=funcprefix. E.g., you usually do not want to see the trampoline functions in the PLT sections for calls to functions in shared libraries. You can see the difference if you profile with --skip-plt=no. If a call is ignored, cost events happening will be attached to the enclosing function.

If you have a recursive function, you can distinguish the first 10 recursion levels by specifying --fn-recursion10=funcprefix. Or for all functions with --fn-recursion=10, but this will give you much bigger profile data files. In the profile data, you will see the recursion levels of "func" as the different functions with names "func", "func'2", "func'3" and so on.

If you have call chains "A > B > C" and "A > C > B" in your program, you usually get a "false" cycle "B <> C". Use --fn-caller2=B --fn-caller2=C, and functions "B" and "C" will be treated as different functions depending on the direct caller. Using the apostrophe for appending this "context" to the function name, you get "A > B'A > C'B" and "A > C'A > B'C", and there will be no cycle. Use --fn-caller=3 to get a 2-caller dependency for all functions. Note that doing this will increase the size of profile data files.

These options influence the name and format of the profile data files.

These options specify when actions relating to event counts are to be executed. For interactive control use callgrind_control.

These options specify when events are to be aggregated into event counts. Also see Limiting range of event collection.

These options specify how event counts should be attributed to execution contexts. More specifically, they specify e.g. if the recursion level or the call chain leading to a function should be accounted for, and whether the thread ID should be remembered. Also see Avoiding cycles.