MPI+OpenMP: he-hy - Hello Hybrid! - compiling, starting
Prepare for these exercises:
cd ~; cp -a ~xwwclabs/MPIX-HLRS . # copy the exercises
cd ~/MPIX-HLRS/he-hy # change into your he-hy directory
Contents:
job_*.sh # job-scripts to run the provided tools, 2 x job_*_exercise.sh
*.[c|f90] # various codes (hello world & tests) - NO need to look into these!
hawk/*.o* # hawk job output files
IN THE ONLINE COURSE he-hy shall be done in two parts:
first exercise = 1. + 2. + 3. + 4.
second exercise = 5. + 6. + 7. (after the talk on pinning)
1. FIRST THINGS FIRST - PART 1: :DEMO: find out about a (new) cluster - login node
module (avail, load, list, unload); compiler (name & --version)
Some suggestions what you might want to try:
module list # are there any default modules loaded ? are these okay ?
module avail # which modules are available ?
# default versions ? latest versions ? spack ?
# ==> look for: compiler, mpi, likwid
module load likwid # let's try to load a module...
likwid-topology -c -g # ... and use it on the login node
module unload likwid # clean up and get rid of loaded modules
numactl --hardware # provides the same information...
module list # list modules loaded
echo $CC # check compiler names
echo $CXX #
echo $F90 #
gcc --version # check versions for standard Intel compilers
g++ --version #
gfortran --version #
mpi<tab><tab> # figure out which MPI wrappers are available
mpicc --version # try standard names for the MPI wrappers
mpicxx --version #
mpif90 --version #
mpif08 --version
2. FIRST THINGS FIRST - PART 2: :DEMO: find out about a (new) cluster - batch jobs
job environment, job scripts (clean) & batch system (PBSPro); test compiler and MPI version
job_env.sh, job_te-ve_[c|f].sh, te-ve*
! there might be several hardware partitions and qos on a cluster & a default !
! you have to check all hardware partitions you would like to use separately !
! in the course we have a node reservation, activate with -q R_mpix !
PBSPro (Hawk):
PBSPro (Hawk):
qsub -q R_mpix job*.sh # submit a job
qstat # check status
qdel JOB_ID # delete a job
*.o* & *.e* # job output & error
qsub -q R_mpix job_env.sh # check job environment
mpicc -o te-ve te-ve.c | mpif08 -o te-ve te-ve-mpi_f08.f90 # compile on login node
| mpif90 -o te-ve te-ve-mpi_old.f90
| mpif90 -o te-ve te-ve-mpi_old-key.f90
mpirun -n 1 ./te-ve # run on login node
qsub -q R_mpix job_te-ve_c.sh | qsub -q R_mpix job_te-ve_f.sh # submit job (te-ve)
3. MPI+OpenMP: :TODO: how to compile and start an application
how to do conditional compilation
job_co-co_[c|f].sh, co-co.[c|f90]
Recap with gcc & mpt (@Hawk) (→ see also slide ##):
| compiler: | ? USE_MPI | ? _OPENMP | START APPLICATION: | |||
|---|---|---|---|---|---|---|
| C: | export OMP_NUM_THREADS=# | |||||
| with MPI | mpicc | -DUSE_MPI | -fopenmp | … | mpirun -n # ./<exe> | |
| no MPI | gcc | -fopenmp | … | ./<exe> | ||
| Fortran: | export OMP_NUM_THREADS=# | |||||
| with MPI | mpif08 | -cpp | -DUSE_MPI | -fopenmp | … | mpirun -n # ./<exe> |
| no MPI | gfortran | -cpp | -fopenmp | … |
./<exe> |
TODO:
→ Compile and Run (4 possibilities) co-co.[c|f90] = Demo for conditional compilation.
→ Do it by hand - compile and run it directly on the login node (e.g., with #=4).
→ Have a look into the code: co-co.[c|f90] to see how it works.
→ It's also available as a script: job_co-co_[c|f].sh
Always check compiler names and versions, e.g. with: mpicc --version !!!
4. MPI+OpenMP: :TODO: get to know the hardware - needed for pinning
(→ see also slide ##)
TODO:
→ Find out about the hardware of compute nodes:
→ Write and Submit: job_check-hw_exercise.sh
→ Describe the compute nodes... (core numbering?)
→ solution = job_check-hw_solution.sh
→ solution output = hawk/check-hw.o*