PTfS22: Assignment 5: Performance counters, energy to solution (due June 10 11:59 p.m.)

Opened: Thursday, 2 June 2022, 12:00 AM

Due: Saturday, 11 June 2022, 12:00 AM

Hardware performance counters

In order to study the interaction software and hardware, hardware performance monitoring is instrumental. Here you will use the likwid-perfctr tool to investigate code execution and data transfer of the serial Schönauer triad loop:
```
for(int i=0; i<100000000; ++i)
  a[i] = b[i] + c[i] * d[i];
```
Use your benchmarking harness to run this loop for at least 1 second.

(a) Instrument the code with LIKWID markers as shown in the lecture (include the code and command line for building the binary in your assignment submission!). You can find a template code in ~ptfs100h/GettingStarted/scan_c_markers.c. Instrument the inner benchmark loop so that every full loop traversal is recognized. Make sure it compiles and links correctly. On Fritz, you have to load the "likwid" module to get the LIKWID commands and libraries. Use the $LIKWID_INC and $LIKWID_LIB environment variables to make your life easier. You could do, e.g.:

$ icx $LIKWID_INC -O3 -xHost -fno-alias $LIKWID_LIB -DLIKWID_PERFMON source.c timing.o -llikwid

This assumes that your source code is in "source.c". (10 crd)
(b) Run the benchmark on core 0 of a Fritz node with Turbo mode enabled (i.e., not fixing the clock speed) and using the MEM_DP performance group. You will need the "-C hwperf" command line option to sbatch (or salloc) so that access to the hardware performance counters is possible:

$ sbatch -N 1 -C hwperf job.sh

If you forget this option, likwid-perfctr will not get access to the counters and will fail with an error message. In order to run the code with likwid-perfctr, you can do:

$ likwid-perfctr -C 0 -m -g MEM_DP ./a.out

Include the output of likwid-perfctr in your submission and answer the following questions:
(b1) What was the actual clock frequency the core was running at? (5 crd)
(b2) How often was the inner (benchmark) loop executed? (5 crd)
(b3) Was the code vectorized? If yes, what register width was used? (5 crd)
(b4) What is the (measured) code balance of the loop? Explain why the measured value meets (or does not meet) your expectations. (5 crd)

General documentation about likwid-perfctr is available at: https://github.com/RRZE-HPC/likwid/wiki/likwid-perfctr
Optimal energy to solution
We want to model the power consumption of a 12-core processor under a certain workload, and make the following assumptions:
- The chip dissipates a "baseline power" $W_0$ when all cores are idle, independent of the clock frequency.
- If a core is executing instructions, it dissipates the additional ("dynamic") power $W_d$, which adds to the chip's baseline power.
- The performance of a certain code running on a single core of this processor is $P(1)$.
- The maximum performance of the parallel version of this code is $P_{limit}>P(1)$. Hence, when solving a given problem in parallel with $n$ cores, the overall performance is $P(n)=\min\left(nP(1),P_{limit}\right)$. This models a behavior where a code is limited by a bottleneck if it is running on multiple cores (i.e., the performance "saturates" at some point).
We want to solve a given problem with $n=1\ldots 12$ cores and calculate the energy $E(n)$ it takes to solve it. This is our "energy to solution" metric. You can assume that the amount of work to be done is normalized to 1, i.e., the time to solution on $n$ cores is $T(n)=1/P(n)$. For the following tasks you may find it helpful to use a spreadsheet program:
(a) (20 crd) Calculate $E(n)$ for $W_0=10 \,\mathrm W$, $W_d=8 \,\mathrm W$, $P(1)=1 \,\mathrm s^{-1}$, and $P_{limit}=6 \,\mathrm s^{-1}$ and draw a diagram of $E(n)$ vs. $n$. For which $n_\min$ is $E(n_\min)$ minimal? Can it make sense to use more than $n_\min$ cores?
(b) (15 crd) Now assume we apply a code optimization (such as SIMD vectorization) that improves the single-core performance to $P(1)=2\,\mathrm s^{-1}$, but $P_{limit}$ is unaffected. How does that change $n_\min$ and $E(n_\min)$? Draw the new function $E(n)$ in a diagram. What is the general conclusion from this result?
(c) (15 crd) Keeping $P(1)$ as it is (i.e., the optimized value from (b)), we now perform an optimization that improves the saturated performance $P_{limit}$ to $10\,\mathrm s^{-1}$. How does that change $n_\min$ and $E(n_\min)$? What happens if $P_{limit}=50\,\mathrm s^{-1}$? What is the general conclusion from this result?
(d) (20 crd) What impact do you expect from changing the clock speed away from the base clock speed $f_0$ (higher or lower)? Remember that the dynamic power dissipation of each core goes with the third power of the frequency and assume that that $P_{limit}$ is unaffected by the clock speed. This yields $P(n,f)=\min\left(P(n,f_0)\times f/f_0,P_{limit}\right)$. What happens to the optimal energy to solution if you (i) increase or (ii) decrease the clock speed by a factor of two?