PTfS22: Likwid + OpenMP | NHR Learning Platform

Dear PTfS-Team,

while working on assignment 9, exercise 1, I set out to measure the arithmetic intensity of the raytrace loop using likwid. At first I just ran likwid-perfctr on the whole program, but then realized that the values I get are not correct due to the zeroing loop over the image array, which creates extra data traffic. So I used the Marker API to isolate the benchmark loop. And then things started getting weird.

When profiling the entire program, I get a total data volume of 0.6 - 0.7 GB, which is expected when accounting for the zeroing loop, write-allocates, etc.. With the marker API, I call `LIKWID_MARKER_START` ahead of the loop for each thread in a short `#pragma omp parallel` region, containing only that statement (calling it from serial code initialized the markers only for the main thread). I call `LIKWID_MARKER_STOP` in the same way after the loop. But then, likwid gives me a measured data volume of 6 GB for the loop region, ten times as much as before! Also, the DP FLOP/s measurement is four to five times higher than measured for the entire program. Am I doing something wrong, or is there some reasonable explanation for this, and some way to sensibly interpret these numbers?

Best,

Frederik

Output for entire program (no marker API): https://moodle.nhr.fau.de/pluginfile.php/2955/mod_forum/post/130/raytrace_perf_nomarker.txt
Output for benchmark loop, with marker API: https://moodle.nhr.fau.de/pluginfile.php/2955/mod_forum/post/130/raytrace_perf.txt

Re: Likwid + OpenMP

by Georg Hager - Tuesday, 5 July 2022, 6:47 PM

Hi Frederik,

there are two issues here: The massive performance reduction for the end-to-end measurement is just a consequence of the short runtime and the significant impact from allocation and initialization. The increase in memory volume for activated markers, however, is not expected. You can solve the problem by registering each marker region at the beginning of your program (after LIKWID_MARKER_INIT) in a parallel region (see also https://github.com/RRZE-HPC/likwid/wiki/LikwidMarkerAPIPitfalls#recommended-usage-of-likwid_marker_register):

#pragma omp parallel
{
  LIKWID_MARKER_REGISTER("raytrace");
}

This way, the individual LIKWID threads that are responsible for the monitoring are started already at that point and not when you call LIKWID_MARKER_START for the first time. In addition, internal data structures are set up, and all this overhead is out of the way as soon as you start counting later on. It is unclear to me why not registering the region(s) incurs such a large memory traffic, but the LIKWID developers have been notified about the problem.

Be that as it may, you do not actually need likwid-perfctr to answer the question in the assignment, and we did not expect you to use it at all (or we would probably have spotted the problem ourselves - but still, thank you for pointing it out).

Best,
Georg.