Hybrid Programming in HPC - MPI+X @ HLRS

Copy additional files for the pinning exercises:

cd ~; cp -af ~sct50054/MPIX-HLRS/jobs_he-hy/*  MPIX-HLRS/he-hy/.       #   copy

cd ~/MPIX-HLRS/he-hy          #   change into your he-hy directory

 

 

Contents:

job_*.sh                              #   job-scripts to run the provided tools, 2 x job_*_exercise.sh

*.[c|f90]                             #   various codes (hello world & tests) - NO need to look into these!

hawk/*.o*                            #   hawk job output files

 

IN THE ONLINE COURSE he-hy shall be done in two parts:

    first exercise       =   1. + 2. + 3. + 4.   (already done before)

    second exercise   =   5. + 6. + 7.   (after the talk on pinning)

 

 

 

4. Recap from previous exercise: get to know the hardware

    → Find out about the hardware of compute nodes:

    → solution output =  check-hw.o_solution

 

 

5. MPI-pure MPI:      compile and run the MPI "Hello world!" program (pinning)

    job_he-mpi_[default|ordered].sh,  he-mpi.[c|f90],  help_fortran_find_core_id.c    

    compile he-mpi - either C or Fortran: 
    C:             mpicc -o he-mpi he-mpi.c
    Fortran:   gcc -c help_fortran_find_core_id.c
    Fortran:   mpif90 -o he-mpi he-mpi.f90 help_fortran_find_core_id.o

    run he-mpi twice on login node with only 4 procs:
    mpirun -n 4 ./he-mpi                     # unsorted

    mpirun -n 4 ./he-mpi | sort -n      # sorted
    ? Why is the output (most of the time) unsorted ?
        ==> here (he-mpi) you can user: ... | sort -n

    submit he-mpi to a compute node (mpirun):
    qsub job_he-mpi_default.sh                          # HLRS training cluster --> pinning is perfect
    qsub job_he-mpi_ordered.sh                        # HLRS training cluster -->  pinning is perfect

    ? Can you rely on the defaults for pinning ? ==> Always take care of & check correct pinning yourself !

 

 

6. MPI+OpenMP: :TODO: compile and run the Hybrid "Hello world!" program

    job_he-hy_exercise.sh,  he-hy.[c|f90],  help_fortran_find_core_id.c

    compile he-hy - either C or Fortran:
    C:             mpicc -fopenmp -o he-hy he-hy.c
    Fortran:   gcc -c help_fortran_find_core_id.c
    Fortran:   mpif90 -fopenmp -o he-hy he-hy.f90 help_fortran_find_core_id.o

    run he-hy twice on login node  with only 4 procs & 4 threads:
    export OMP_NUM_THREADS=4
    mpirun -n 4 ./he-hy                    # unsorted
    mpirun -n 4 ./he-hy | sort -n      # sorted
    ? Why is the output (most of the time) unsorted ?
       ==> here (he-hy) you can use: ... | sort -n

 

      TODO:

    → Run he-hy on a compute node, i.e.:  qsub job_he-hy_exercise.sh

    → Oh no, this is not going to fly... --> we have to do better !

    → Look into: job_he-hy_exercise.sh 

    → Do NOT YET do the pinning exercise, see below 7.

    ? Can you rely on the defaults for pinning ?

    ==> Always take care of & check correct pinning yourself !

 

 

7. MPI+OpenMP: :TODO: how to do pinning

    job_he-hy_[exercise|solution].sh,  he-hy.[c|f90]

 

      TODO (see below for info):

    → Do the pinning exercise in:  job_he-hy_exercise.sh

    → one possible (short) solution = job_he-hy_solution.sh

 

 

PINNING: (→ see also slides)

Pinning depends on:

batch system	PBSPro	\
MPI library	OpenMPI	|   interaction between these !
startup	mpirun	/

Always check your pinning !

          → job_he-hy...sh (he-hy.[c|f90] prints core_id)

          → print core_id in your application (see he-hy.*)

          → turn on debugging info & verbose output in job

          → monitor your job → login to nodes: top [1 q]