MPI+OpenMP: he-hy - Hello Hybrid! - compiling, starting

Prepare for these Exercises:

cd ~/HY-VSC/he-hy          #   change into your he-hy directory

Contents:

job_*.sh                              #   job-scripts to run the provided tools, 2 x job_*_exercise.sh

*.[c|f90]                             #   various codes (hello world & tests) - NO need to look into these!

vsc4/vsc4_slurm.out_*     #   vsc4 output files --> sorted (note: physical cores via modulo 48)

IN THE ONLINE COURSE he-hy shall be done in two parts:

    first exercise       =   1. + 2. + 3. + 4.

    second exercise   =   5. + 6. + 7.   (after the talk on pinning)

1. FIRST THINGS FIRST - PART 1: :DEMO: find out about a (new) cluster - login node

    module (avail, load, list, unload); compiler (name & --version)

    Some suggestions what you might want to try:

    module list                                 # are there any default modules loaded ?   are these okay ?
                                                        # ==> no default modules loaded on vsc4

    module avail                               # which modules are available ?
                                                        # default versions ?   latest versions ?   spack ?
                                                        # ==> look for: compiler, mpi, likwid

# likwid is (temporarily) not available on vsc4...
    module load likwid                    # let's try to load a module...
    likwid-topology -c -g                # ... and use it on the login node
    module purge                            # clean up and purge all loaded modules

    numactl --hardware                  # provides the same information...

    module load intel intel-mpi   # decide on the modules to be used (Intel compiler and Intel MPI library)

    module list                                 # list modules loaded

    echo $CC                                   # check compiler names
    echo $CXX                                 #                     
    echo $F90                                  #

    icc --version                              # check versions for standard Intel compilers
    icpc --version                            #
    ifort  --version                           #

    mpi<tab><tab>                         # figure out which MPI wrappers are available

    mpicc --version                        # try standard names for the MPI wrappers
    mpicxx --version                      #
    mpif90 --version                      # ==> oh no, these are old gnu compilers...

    mpiicc --version                       # try special names, e.g., for Intel...
    mpiicpc --version                     #
    mpiifort --version                     # ==> finally, we get what we expect (note the double ii)

    Always check compiler names and versions, e.g. with: mpiicc --version !!!

2. FIRST THINGS FIRST - PART 2: :DEMO: find out about a (new) cluster - batch jobs

    job environment, job scripts (clean) & batch system (SLURM); test compiler and MPI version

    job_env.sh,  job_te-ve_[c|f].sh,  te-ve*    

   ! there might be several hardware partitions and qos on a cluster & a default !
   !  you have to check all hardware partitions you would like to use separately  !

   ! in the course we have a node reservation on vsc4 in the skylake_0096 partition !
   ! (it's switched on via your ~/.bashrc file (done with setup.sh))                                     !

    sbatch job_env.sh                                                        # check job environment

    ./job_te-ve_c.sh             |   ./job_te-ve_f.sh               # run on login nodes  --> test version (te-ve)

    sbatch job_te-ve_c.sh   |   sbatch job_te-ve_f.sh     # submit job               --> test version (te-ve)

3. MPI+OpenMP: :TODO: how to compile and start an application
                                              how to do conditional compilation

    job_co-co_[c|f].sh,  co-co.[c|f90]

    Recap with Intel compiler & Intel MPI (→ see also slide ##):

	compiler:		? USE_MPI	? _OPENMP		START APPLICATION:
C:						export OMP_NUM_THREADS=#
with MPI	mpiicc		-DUSE_MPI	-qopenmp	…	mpirun -n # ./<exe>
no MPI	icc			-qopenmp	…	./<exe>
Fortran:						export OMP_NUM_THREADS=#
with MPI	mpiifort	-fpp	-DUSE_MPI	-qopenmp	…	mpirun -n # ./<exe>
no MPI	ifort	-fpp		-qopenmp	…	./<exe>

      TODO:

    → Compile and Run (4 possibilities) co-co.[c|f90] = Demo for conditional compilation.

    → Do it by hand - compile and run it directly on the login node (e.g., with #=4).
         export I_MPI_FABRICS=shm                          # needed on (some) login nodes

    → Have a look into the code:  co-co.[c|f90]  to see how it works.

    → It's also available as a script:  job_co-co_[c|f].sh

    Always check compiler names and versions, e.g. with: mpiicc --version !!!

4. MPI+OpenMP: :TODO: get to know the hardware - needed for pinning

                                                                                                              (→ see also slide ##)

      TODO:

    → Find out about the hardware of compute nodes:

    → Write and Submit: job_check-hw_exercise.sh

        # likwid is (temporarily) not available on vsc4...

    → Describe the compute nodes... (core numbering?)

    → solution = job_check-hw_solution.sh

    → solution.out =  vsc4/vsc4_slurm.out_check-hw_solution