Assignment 7: Machine topology, Basic OpenMP (due June 24 11:59 p.m.)

Note 1: It is essential to take control of core-thread affinity in OpenMP programs. This means that it should not be left to chance where in the machine the threads of an OpenMP program are running. The OpenMP standard defines a way to bind threads to sockets, cores, or hardware threads. For example,

$ OMP_NUM_THREADS=10 OMP_PLACES=cores OMP_PROC_BIND=close ./a.out

binds the ten threads of the binary to the 10 "first" physical cores of the machine. On Fritz, these would be the ten first cores of a socket. The OMP_PLACES variable denotes the entities ("places") used for pinning threads. You can set it to "threads" (this means hardware threads), "cores" (this means full cores), or "sockets" (this means full sockets). E.g., with OMP_PLACES=cores, each OpenMP thread will be bound to its own physical core. The OMP_PROC_BIND variable determines how the OpenMP threads are pinned to the places. Here "close" means to fill the places from "left to right," while "spread" keeps an even spacing between the OpenMP threads. 

Further example: 

$ OMP_NUM_THREADS=20 OMP_PLACES=cores OMP_PROC_BIND=spread ./a.out 

will run 10 threads on one socket and 10 on the other socket of a Fritz node. In fact, the runtime will "spread out" the threads evenly across the machine. There are more options in this scheme, but this information will be sufficient to get you going.

Note 2: The LIKWID tool suite is a collection of simple, easy-to-use tools that ease the handling of multicore nodes. You know likwid-perfctr already from previous assignments. The two most important other tools you may want to look at are likwid-topology and likwid-pin. You can watch a couple of short videos we made about these tools:

•  (finding out about node topology)
•  (enforcing thread affinity)

In order use LIKWID on Fritz, you have to load the module first:

$ module load likwid

Then, to bind the threads of an OpenMP application to cores, you use likwid-pin as a wrapper:

$ likwid-pin -c S0:0-9 ./a.out

This has the same effect as the first example above: It pins the threads of a.out to the first ten physical cores of socket 0. Note that we did not have to set OMP_NUM_THREADS; likwid-pin will set it for you and infer its value from the pin mask given with the -c option. If you set it explicitly, likwid-pin will leave it alone. If the pin mask comprises more cores than OMP_NUM_THREADS, that's OK - only the first OMP_NUM_THREADS entries in the pin mask will be used then.

It does not matter whether you use OMP_PLACES/OMP_PROC_BIND or likwid-pin to pin your threads. Choose the mechanism that suits you best.

1. Machine Topology. Use "likwid-topology -g" to find out about the organization of a "Fritz" node.
   (a) How many cores are there in one node? (10 crd)
   (b) How much memory (RAM) is installed? (10 crd)
   (c) What are the L1/l2/L3 cache sizes? How many cores share an L1/L2/L3 cache segment? (10 crd)
   (d) How many sockets and how many ccNUMA domains per socket does the node have? (10 crd)
   
   
2. Parallel update benchmark. Parallelize the update benchmark loop (a[i] = s*a[i]) with OpenMP. To do this, compile and link with the -qopenmp switch and use the "fused" parallel for directive to distribute iterations across threads (details will be presented in the lecture on Monday, June 20):
   
   

   #pragma omp parallel for
   for(int i=0; i<N; ++i)
     a[i] = s*a[i];
   

   For the following measurements, you can fix the clock speed (to 2 GHz) or you can also leave Turbo mode on. Whatever you do, document it in your solution.
   
   (a) (20 crd) Perform benchmark runs with this loop on one Fritz socket. Draw graphs of performance in Gflop/s versus loop length N for N = 10¹...10⁸ (use a log scale on the x axis and a scaling factor of 1.2 from one N to the next) for 1,2,4,8,16 threads (draw them all in the same diagram). Hint: Remember to use the benchmarking "harness" that we showed you; the whole loop should be repeated often enough to make the overall runtime long enough to be accurately measured.
   (b) (20 crd) Comment on the scalability of the benchmark for different working set sizes (L1 cache, L2 cache, L3 cache, memory); i.e., can n cores really compute n times faster? (Remember that the LLC of the Ice Lake processor is an exclusive victim cache!) To do this, compare with the purely serial code (compiled without -qopenmp). For which working set sizes do you get a "good" speedup with multiple threads? 
   (c) (20 crd) For N=10⁸ run the benchmark with 1...18 cores on one ccNUMA domain and draw a scaling graph, i.e., performance in Gflop/s vs. number of cores. From the data, calculate the maximum memory bandwidth that you can achieve.