Korištenje
| Verzija | Modul |
|---|---|
| 4.1.5 | siesta/4.1.5 |
% module load siesta/4.1.5 % siesta Siesta Version : 4.1.5 Architecture : unknown Compiler version: ifort (IFORT) 19.0.0.117 20180804 Compiler flags : mpifort -O2 -fPIC -fp-model source -qopenmp -I/apps/netcdf-fortran/4.5.3/include/ PP flags : -DFC_HAVE_ABORT -DMPI -qopenmp -DNCDF -DNCDF_4 -DNCDF_PARALLEL -DSIESTA__METIIS -DSIESTA__ELPA -I/apps/elpa/2020.11.001/avx2/include/elpa-2020.11.001/modules -DSIESTA__DIAG_2STAGE Libraries : libsiestaLAPACK.a libsiestaBLAS.a libfdict.a libncdf.a -qopenmp -mkl=paralleel -i8 -lmkl_core -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lmkl_sequential -lmkl_blas95_lp64 -L/apps/netcdf-c/4.7.4/lib/ -L/apps/netcdf-fortran/4.5.3/lib/ -L/apps/hdf5/1.12.0/lib/ -lnetcdff -lnetcdf -lhdf5_fortran -lhdf5 -lz -L/apps/metis/5.1.0/lib/ -lmetis -L/apps/elpa/2020.11.001/avx2/lib/ -lelpa PARALLEL version NetCDF-4 support NetCDF-4 MPI-IO support METIS ordering support ...
Primjer skripte za opis posla:
#$ -N siesta-parallel #$ -q p28.q #$ -pe *mpi 16 #$ -cwd module load siesta/4.1.5 export OMP_NUM_THREADS=4 mpirun --bind-to socket -hostfile $TMPDIR/machines -np 4 siesta < fe.fdf > fe.out
Važno
Aplikacija se širi hibridno procesima (MPI paradigma) i threadovima (OpenMP paradigma). Varijabla okruženja OMP_NUM_THREADS definira koliko threadova će svaki MPI proces kreirati. Iz tog razloga umnožak procesa i threadova mora odgovarati zatraženom broju slotova definiranim u paralelnom okruženju. U primjer je to 16 = 4 (MPI procesa) x 4 (OpenMPI threadsa)
Instalacija
Programski paket SIESTA je pripremljen s kompajlerom Intel 2019, Intel MKL Scalapack i MPI implementacijom OpenMPI 3.0.0. Instalacija je obavljena pripremom pomoćnih biblioteka na koje se paket oslanja, dopremanjem izvornog koda, učitavanjem potrebnih modula za kompajler i MPI paralelizaciju te kompajliranjem samog paketa.
Priprema i instalacija se vršila prema uputstvima (uz male preinake specifične za Isabellu): https://gitlab.com/siesta-project/siesta/-/releases/v4.1.5/downloads/siesta.pdf
Izvorni kod se može pronaći na sjedištu: https://gitlab.com/siesta-project/siesta
Priprema
HD5
https://www.hdfgroup.org/downloads/hdf5/source-code/
module load mpi/openmpi3-intel-x86_64 CC=mpicc CXX=mpicxx FC=mpifort F90=mpif90 ./configure --prefix=/apps/hdf5/1.12.0 --enable-parallel --enable-fortran make -j8 make install
NetCDF
https://parallel-netcdf.github.io/
module load mpi/openmpi3-intel-x86_64 ./configure --prefix=/apps/pnetcdf/1.12.2 --enable-shared make -j8 make install
https://www.unidata.ucar.edu/downloads/netcdf/
module load mpi/openmpi3-intel-x86_64 LD_LIBRARY_PATH="/apps/hdf5/1.12.0/lib/:/apps/pnetcdf/1.12.2/lib:$LD_LIBRARY_PATH" LIBS="-lhdf5_hl -lhdf5" LDFLAGS="-L/apps/hdf5/1.12.0/lib -L/apps/pnetcdf/1.12.2/lib" CC=mpicc CPPFLAGS="-I/apps/hdf5/1.12.0/include -I/apps/pnetcdf/1.12.2/include" CXX=mpicxx ./configure --prefix=/apps/netcdf-c/4.7.4/ --enable-pnetcdf make -j8 make install
module load mpi/openmpi3-intel-x86_64 LD_LIBRARY_PATH="/apps/netcdf-c/4.7.4/lib/:$LD_LIBRARY_PATH" CC=mpicc CXX=mpicxx FC=mpifort F90=ifort CPPFLAGS="-I/apps/netcdf-c/4.7.4/include/" LDFLAGS="-L/apps/netcdf-c/4.7.4/lib/" ./configure --prefix=/apps/netcdf-fortran/4.5.3/ make -j8 make install
METIS
http://glaros.dtc.umn.edu/gkhome/metis/metis/download
make config prefix=/apps/metis/5.1.0/ cc=icc make install
ELPA
https://elpa.mpcdf.mpg.de/elpa-tar-archive
module load mpi/openmpi3-intel-x86_64 LIBS="-qopenmp -mkl=parallel -i8 -lmkl_core -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lmkl_sequential -lmkl_blas95_lp64" ./configure --prefix=/apps/elpa/2020.11.001/avx2 make -j8 make install
MUMPS
http://mumps.enseeiht.fr/index.php?page=dwnld#form
module load mpi/openmpi3-intel-x86_64 cd MUMPS_5.3.5 cp Make.inc/Makefile.INTEL.PAR Makefile.inc make all cp -R lib/ include/ /apps/mumps/5.3.5/
Kompajliranje i instalacija SIESTA
wget -c https://gitlab.com/siesta-project/siesta/-/package_files/6659062/download -O siesta-4.1.5.tar.gz tar -v -xzf siesta-4.1.5.tar.gz
cd siesta-4.1.5/Obj/ sh ../Src/obj_setup.sh cp intel.make arch.make module load mpi/openmpi3-intel-x86_64
Datoteku arch.make je potrebno editirati i dodati postavke za MPI paralelizaciju, Intel kompajler te povezivanje na pomoćne biblioteke:
# parallel compile @ Isabella CC = mpicc FPP = $(FC) -E -P FC = mpifort FC_SERIAL = ifort FFLAGS = -O2 -fPIC -fp-model source -qopenmp MPI_INTERFACE=libmpi_f90.a MPI_INCLUDE=. FPPFLAGS += -DMPI -qopenmp LIBS += -qopenmp -mkl=parallel -i8 -lmkl_core -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lmkl_sequential -lmkl_blas95_lp64 LIBS += -L/apps/netcdf-c/4.7.4/lib/ -L/apps/netcdf-fortran/4.5.3/lib/ -L/apps/hdf5/1.12.0/lib/ -lnetcdff -lnetcdf -lhdf5_fortran -lhdf5 -lz # fdict COMP_LIBS += libfdict.a # ncdf COMP_LIBS += libncdf.a FPPFLAGS += -DNCDF -DNCDF_4 -DNCDF_PARALLEL FFLAGS += -I/apps/netcdf-fortran/4.5.3/include/ # metis LIBS += -L/apps/metis/5.1.0/lib/ -lmetis FPPFLAGS += -DSIESTA__METIS # elpa LIBS += -L/apps/elpa/2020.11.001/avx2/lib/ -lelpa FPPFLAGS += -DSIESTA__ELPA -I/apps/elpa/2020.11.001/avx2/include/elpa-2020.11.001/modules # mumps LIBS += -L/apps/mumps/5.3.5/lib/ -lzmumps -lmumps_common -lpord FPPFLAGS += -DSIESTA__MUMPS FFLAGS += -I/apps/mumps/5.3.5/include/
make -j8 cp siesta /apps/siesta/4.1.5/avx2
cd siesta-4.1.5/Util
find . -name Makefile -exec sed -i 's/$(LDFLAGS)/$(LDFLAGS) -qopenmp/g' {} \;
./build_all
find . -type f -executable -exec cp {} /apps/siesta/4.1.5/avx2 \;
ls /apps/siesta/4.1.5/avx2/
h2o/ eig2bxsf* fractional* ionplot.sh* pblas_prb* simple* surf.py*
2dplot.py* eigfat2plot* get_chem_labels* macroave* pi3* simple_mpi_parallel* tbtrans*
Eig2DOS* f2fmaster* gnubands* md2axsf* plstm* simple_mpi_serial* ts2ts*
blacs_prb* f2fslave* grid1d* mixps* plsts* simple_pipes_parallel* tselecs.sh*
ccViz* fat* grid2d* mprop* protoNEB* simple_pipes_serial* vib2xsf*
clean_all.sh* fcbuild* hs2hsx* optical* rho2xsf* simple_sockets_parallel* vibra*
denchar* fdf2grimme* hsx2hs* optical_input* sies2arc* simple_sockets_serial* xv2xsf*
driver* fmpdos* int_explorer* para* siesta* spin_texture*