Page History

Introduction

To schedule and manage jobs on the Parachute computer cluster, PBS (Portable Batch System) is used, which performs job scheduling within the cluster. Its primary task is the distribution of computer tasks, i.e. batch jobs, among the available computer resources.

...

...

#!/bin/bash

#PBS -P test example
#PBS -e /home/my_directory
#PBS -q cpu
#PBS -l walltime=00:01:00
#PBS -l select=1:ncpus=10

module load mpi/openmpi-x86_64

mpicc --version

Osnovni PBS parametri

Options for sending notifications by mail option -m:

#!/bin/bash

#PBS -q cpu
#PBS -l walltime=00:01:00
#PBS -l select=1:ncpus=2
#PBS -M <name>@srce.hr,<name2>@srce.hr
#PBS -m be

echo $PBS_JOBNAME > out
echo $PBS_O_HOST

...

Options for requesting resources with the -l option

PBS environmental variables

...

PBS makes it possible to define the necessary resources in several ways. The main unit for resource allocation is the so-called "Chunk" or piece of node. A chunk is defined with the select option. The number of processor cores per chunk can be defined with ncpus, the number of mpi processes with mpiprocs and the amount of working memory with mem. It is also possible to define walltime (maximum job execution time) and place (chunk allocation method by nodes).

If some of the parameters are not defined, the default value will be used:

Memory control using cgroups

...

In this case, the user gets 4 processor cores and a total of 14GB of memory on one chunk. When jobs are described without the select option, it is not possible to "chain resources" (separate the required resources with a colon, it is necessary to put the -l option on a new line for each resource)

...

Primjeri korištenja

Usage examples

1. Example of simple use of $TMPDIR variablePrimjer jednostavnog korištenja $TMPDIR varijable:

   Code Block
   #!/bin/bash #PBS -q cpu #PBS -l walltime=00:00:05 cd $TMPDIR pwd > test cp test $PBS_O_WORKDIR

   
   
2. Primjer kopiranja ulaznih podataka u $TMPDIR, pokretanje aplikacije, i kopiranje u radni direktorijAn example of copying the input data to $TMPDIR, running the application, and copying it to the working directory:
   

   Code Block

   #!/bin/bash #PBS -q cpu #PBS -l walltime=00:00:05 # StvaranjaCreating directories direktorijafor zainput ulaznedata podatkein ua privremenomtemporary direktorijudirectory mkdir -p $TMPDIR/data # KopiratiCopy sveall potrebnerequired inputeinputs uto privremenia temporary direktorijdirectory cp -r $HOME/data/* $TMPDIR/data # Pokrenuti aplikaciju i preusmjeriti outpute u "aktualniRun the application and redirect the outputs to the "current" (privremenitemporary) direktorijdirectory cd $TMPDIR <izvršna<application naredbaexecutable aplikacije>command> 1>output.log 2>error.log # KopiratiCopy željenidesired output uto radniworking direktorijdirectory cp -r /$TMPDIR/output $PBS_O_WORKDIR

...

Parallel jobs

OpenMP

...

parallelization

If your application uses parallelization exclusively at the level of OpenMP threads and cannot expand beyond one worker node (that is, it works with shared memory), you can call the job as shown in the xTB application example below.

Ako Vaša aplikacija koristi paralelizaciju isključivo na razini OpenMP dretvi (engl. threads) i ne može se širiti van jednog radnog čvora (odnosno radi s dijeljenom memorijom), posao možete pozvati na način kako je prikazano u primjeru xTB aplikacije niže.

...

#!/bin/bash

#PBS -q cpu
#PBS -l walltime=10:00:00
#PBS -l ncpus=8:mem=28GB

cd ${PBS_O_WORKDIR}

xtb C2H4BrCl.xyz --chrg 0 --uhf 0 --opt vtight

MPI paralelizacija

Ako Vaša aplikacija koristi paralelizaciju isključivo na razini MPI procesa i može se širiti van jednog radnog čvora (odnosno radi s raspodijeljenom memorijom), posao možete pozvati na način kako je prikazano u primjeru Quantum ESPRESSO aplikacije niže. Za izvođenje aplikacija koje koriste paralelizaciju MPI (ili hibridno MPI+OMP) potrebno je učitati mpi modul  prije pozivanja naredbe mpiexec ili mpirun.

MPI parallelization

If your application uses parallelization exclusively at the MPI process level and can extend beyond a single worker node (that is, it works with distributed memory), you can call the job as shown in the Quantum ESPRESSO application example below. To run applications using MPI (or hybrid MPI+OMP) parallelization, the mpi module must be loaded before calling mpiexec or mpirun.

#!/bin/bash

#PBS -q cpu
#PBS -l walltime=10:00:00
#PBS -l select=16

module load mpi/openmpi-x86_64

cd ${PBS_O_WORKDIR}

mpiexec pw.x -i calcite.in

MPI + OpenMP (

...

hybrid)

...

parallelization

If your application can be parallelized hybridly, i.e. divide its MPI processes into OpenMP threads, you can call the job as shown in the GROMACS application example belowAko se Vaša aplikacije može paralelizirati hibridno, odnosno dijeliti svoje MPI procese u OpenMP threadove, možete posao pozvati na način kako je prikazano u primjeru GROMACS aplikacije niže:

#!/bin/bash

#PBS -q cpu
#PBS -l walltime=10:00:00
#PBS -l select=8:ncpus=4:mem=14GB

module load mpi/openmpi-x86_64

cd ${PBS_O_WORKDIR}

mpiexec -d ${OMP_NUM_THREADS} --cpu-bind depth gmx mdrun -v -deffnm md

Praćenje i upravljanje izvođenja posla

Praćenje posla

Monitoring and management of job performance

Job monitoring

The PBS command qstat is used to display the status of jobs. Command syntax isZa prikaz stanja poslova koristi se PBS-ova naredba qstat. Osnovna sintaksa naredbe je:

qstat <opcije><options> <ID<job_posla>ID>

Izvršavanjem naredbe qstat bez dodatnih opcija dobiva se ispis svih trenuthi poslova svih korisnikaExecuting the qstat command without additional options displays a printout of all current jobs of all users:

Job id            Name             User              Time Use S Queue
----------------  ---------------- ----------------  -------- - -----
111.admin         mpi+omp_s        kmrkalj           00:36:09 R cpu             

Neke od korištenijih opcija suSome of the more frequently used options are:

Primjeri korištenja:

...

Usage examples:

Detailed job description:

qstat -fxw 2648

Tracejob naredba vadi i prikazuje log poruke za PBS posao po kronološkom reduThe tracejob command extracts and displays log messages for a PBS job in chronological order.

tracejob <ID<job_posla>ID>

PrimjerExample:

$ tracejob 111

Job: 111.admin

03/30/2023 11:23:24  L    Considering job to run
03/30/2023 11:23:24  S    Job Queued at request of mhrzenja@node034, owner =
                          mhrzenja@node034, job name = mapping, queue = cpu
03/30/2023 11:23:24  S    Job Run at request of Scheduler@node034 on exec_vnode
                          (node034:ncpus=40:mem=104857600kb)
03/30/2023 11:23:24  L    Job run
03/30/2023 11:23:24  S    enqueuing into cpu, state Q hop 1
03/30/2023 11:23:56  S    Holds u set at request of mhrzenja@node034
03/30/2023 11:24:22  S    Holds u released at request of mhrzenja@node034

Upravljanje poslovima

Poslom se može upravljati i nakon pokretanja.

Job management

The job can be managed even after it has started.

While the job is in the queue, it is possible to temporarily stop its execution with the command:Dok je posao u redu čekanja, moguće je privremeno zaustaviti njegovo izvršavanje naredbom:

qhold <ID<job_posla>ID>

To return to the queueVraćanje natrag na red čekanja:

qrls <ID<job_posla>ID>

The job is completely stopped or unqueued with the commandPosao se u potpunosti zaustavlja ili miče iz reda čekanja naredbom:

qdel <ID<job_posla>ID>

Force stop should be used for stuck jobsZa zaglavljene poslove treba koristiti prisilno zaustavljanje:

qdel -W force -x <ID<job_posla>

Odgađanje izvođenja

PBS pruža mogućnost izvođenja poslova u ovisnosti o drugima, što je korisno u slučajevima poput:

• izvršavanje poslova ovisi o izlazu ili stanju prethodno izvršenog
• aplikacija zahtijeva sekvencijalno izvođenje raznih komponenata
• ispis podataka jednog posla može ugroziti izvođenje drugog

ID>

Delay of execution

PBS provides the ability to perform jobs in dependence on others, which is useful in cases such as:

• the execution of jobs depends on the output or state of the previously executed
• the application requires the sequential execution of various components
• printing data from one job may compromise the execution of another

The directive that enables this functionality when submitting a job immediately isDirektiva koja omogućuje ovu funkcionalnost pri trenutnom podnošenju posla je:

qsub -W depend=<tip><type>:<ID<job_posla>ID>[:<ID<job_posla>ID>] ...

Gdje < tip> može bitiWhere < type> can be:

• after* - pokretanje trenutnog s obzirom na ostale
  
• after - izvršavanje trenutnog nakon početka izvršavanja navedenih
• afterok - izvršavanje trenutnog nakon uspješnog završetka navedenih
  
afternotok -izvršavanje trenutnognakon greške u završetku navedenih

• afterany - izvršavanje trenutnog nakon završetka navedenih
before* - pokretanje ostalih s obzirom na trenutni before - pokretanjenavedenih nakon početka trenutnog
beforeok - pokretanjenavedenih nakon usprešnog završetka trenutnog
beforenotok - pokretanjenavedenih nakon greške u izvršavanju trenutnog beforeany - pokretanjenavedenih nakon završetka trenutnog
• on:<broj> - izvršavanje posla koji će ovisiti o naknadno navedenom broju before*   tipa poslova

Primjeri

• starting the current one with respect to the others
  
  • after - execution of the current one after the start of execution of the specified ones
  • afterok - execution of the current one after the successful completion of the specified ones
    
  • afternotok -execution of the current after an error in the completion of the specified
    
  • afterany - execution of the current one after the end of the specified ones
• before* - starting the others with respect to the current one
  • before - execution of the specified ones after the start of the current one
    
  • beforeok - execution of the specified ones after the successful completion of the current one
    
  • beforenotok - execution of the specified ones after the an error in the completion of the current one
  • beforeany - execution of the specified ones after the end of the current one
• on:<number> - execution of a job that will depend on the subsequently specified number of before* types of jobs

Usage examples:

If we want  posao1 to start after successful completion of Ako želimo da posao1 započne nakon uspješnog završetka posao0:

[korisnik@padobran] $ qsub posao0
1000.admin

[korisnik@padobran] $ qsub -W depend=afterok:1000 posao1
1001.admin

[korisnik@padobran] $ qstat 1000 1001
Job id                 Name             User              Time Use S Queue
---------------------  ---------------- ----------------  -------- - -----
1000.admin             posao0           korisnik           00:00:00 R cpu             
1001.admin             posao1           korisnik                  0 H cpu

Ako želimo da posao0 započne tek nakon uspješnog završetka If we want posao0 to start after successful completion of  posao1 :

[korisnik@padobran] $ qsub -W depend=on:1 posao0
1002.admin
[korisnik@padobran] $ qsub -W depend=beforeok:1002 posao1
1003.admin

[korisnik@padobran] $ qstat 1002 1003
Job id                 Name             User              Time Use S Queue
---------------------  ---------------- ----------------  -------- - -----
1002.admin             posao0           korisnik                  0 H cpu             
1003.admin             posao1           korisnik           00:00:00 R cpu

...