Table of Contents

Intro

For deploying and managing jobs on Isabella computer cluster, SGE or Sun Grid Engine is used and job managment system JMS. In this document use of SGE ver. 9 8 is described.

Running jobs

User applications (in continuation jobs) which are run with SGE system have to be described with the startup shell script. Withing starting script, alongside the usual commands, SGE parameters are stated. It is possible to state the same parameters outside of starting script, during job submission.

...

1. Example of a job that requires 20 CPU cores and 10 GB of RAM per process (the job will be allocated a total of 200 GB of RAM):

   Code Block
   #$ -pe *mpi 20 #$ -l memory=10

   
   

2. The job requires 100 GB of scratch space:

   Code Block
   #$ -pe *mpisingle 4 #$ -l scratch=25

   
   

Job queue

SGE supports multiple job queues, for different job types and maximum job duration, for different job types (interactive, vSMP jobs, ...) etc.

The desired job queue can be defined with the -q parameter:

#$ -q <queueName>

Popis dostupnih redova poslova na Isabelli može se pronaći na Redovi poslova i paralelne okoline.

Job Status Notifications

SGE supports sending e-mail notifications about job status changes.

The -M parameter determines the email address to which job status notifications will be sent:

#$ -M <email_address> 

The -m parameter determines in which cases the notification will be sent:

#$ -m [b][a][e][n]

• b - start of execution

• a - execution error

• e - completion of execution

• n - do not send notifications (default option)

Example of use:

#$ -m ae
#$ -M my@mail.com

Notification will be sent to the address moj@mail.com when the work is interrupted or completed.

Monitoring and management of job performance

Display of job status

SGE's qstat command is used to display job status. The command syntax is:

$ qstat <options>

By executing the qstat command without additional options, a printout of all current user jobs is obtained:

[tsmolcic@teran ~]$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
 131982 0.00001 md5-10k.0  tsmolcic     r     05/28/2019 06:24:06 gpu.0.q@c4140-03.isabella          1
 131988 0.00001 md5-10k.2  tsmolcic     r     05/28/2019 09:29:26 gpu.2.q@c4140-01.isabella          1
 131990 0.00001 md1-5k.3   tsmolcic     r     05/28/2019 09:29:41 gpu.3.q@c4140-01.isabella          1
 131991 0.00001 md10-15.4  tsmolcic     r     05/28/2019 10:18:08 gpu.1.q@c4140-01.isabella          1
 131994 0.04119 md1kp.5    tsmolcic     qw    05/27/2019 13:14:14                                   20
 131996 0.00005 md1-5k.6   tsmolcic     qw    05/27/2019 13:16:30                                    1

Some of more important options:

-s [r|p|s|h] : filter jobs by state: r - active, s - stopped, h - stopped in queue, p - queued
-j [job_ID] : detailed display of job status (if job_ID is not specified, information on all active jobs is displayed)
-f : detailed information about node load and job allocation to nodes is displayed
-F : detailed display of node data
-u <user_name> : only jobs from user <user_name> are displayed (* - for all users)
-q [queue_name] : command retrieves job queue information only
-g c : retrieve summary information about job queues: total queue load, total number of nodes in the queue, number of busy ones
and free nodes and the number of nodes in special states
-g t : slave and master tasks are marked in the jobs view
-l <resource>=<value> : filter jobs by resources
-ext : additional information about jobs
-pri : detailed data on job priorities

The options of the qstat command can be combined, which is especially important when it is necessary to filter through several hundred jobs.

Examples of use:

1. View all jobs in the queue

   Code Block
   $ qstat -u "*" -s p

   
   

2. 
   

   Display of all user jobs that are currently running and required a16-mpi parallel environment

   Code Block
   $ qstat -u "*" -s r -pe a16-mpi

   
   

3. Display of all jobs and nodes user pero

   Code Block
   $ qstat –s r –f –u pero

   
   

4. Display of all jobs currently running on  nodes with graphical processors:

   Code Block
   $ qstat –u "*" -q gpu.*.q

   
   

Display of working nodes

To print the number of processors, cores and the amount of working memory per node with use of the following command:

$ qhost

The values ​​of the available resources per work nodes can be printed with the following command:

$ qhost -F vendor,scratch,memory

Job management

The job can also be managed after launch.

While a job is in the queue, it is possible to temporarily stop its execution with the following command:

$ qhold <ID_posla>

A stopped job can be requeued with the following command:

$ qrls <ID_posla>

Greater control over the job is provided by the qmod command. The command makes it possible to temporarily stop active jobs by sending a SIGSTOP signal to the job. The job will enter the idle state (T) but will not make resources available (memory, file descriptors). The command also enables saving the state of the process to disk (checkpointing) for jobs that have this option. Furthermore, using the qmod command, the user can stop the active process and return it to the job queue.

The command parameters are:

-c : the command performs a process state saving
-f : parameter defines to execute the command whether it is possible or not
     (useful when requeueing jobs marked with the -r n parameter).
-r: the command stops the job and puts it back in the queue.
-s : the command stops the execution of the active process.
-us : command resumes execution of stopped active job.

The job is completely stopped or unqueued with the command:

$ qdel <job_ID>

It is possible to stop all user jobs:

$ qdel -u <username>

Force stop should be used for stuck jobs or jobs in loop:

$ qdel -f <job_ID>

Access to information about completed jobs

The qacct command is used to retrieve information about completed jobs. The syntax is:

$ qacct <ptions>

Most common example:

$ qacct -j <job_ID>

Prints all the information about the completed work.

Other useful qacct command options are:

-j <job_id> : detailed description of individual jobs
-h <node_address> : statistics for individual nodes
-q <queue_name> : statistics for individual queues
-o <username> : statistics for individual users
-pe <parallel_env_name> : statistics for individual parallel environments
-slots <number_of_processors> : statistics for jobs for a specified number of processors
-P <project> : display the consumption summary of the defined project

Examples of use:

1. Detailed information about all jobs performed on the cluster (caution: large amount of data):

   Code Block
   $ qacct -j

   
   

2. Display of information about all jobs of the defined user:

   Code Block
   $ qacct -j -o <user>

   Display a summary of the consumption of computer resources of a defined user (if <user> is not defined, data for all users is displayed):

   Code Block
   $ qacct -o <user>

   
   

3. Display information about all jobs for the defined project:

   Code Block
   $ qacct -j -P <projekt>

   Display of the consumption summary of the defined project (if <project> is not defined, data for all projects are displayed):

   Code Block
   $ qacct -P <project>

   
   

Cheatsheets

Linux cheatsheet

Navigating the file system

Directory management

Kopiranje datoteka i direktorija

Move and rename files and directories

Password change

Auto-fill and search of command history

SGE cheatsheet

Job submit

Checking job status

Stopping jobs

Information about completed jobs