Cluster Resources are managed by the system for batch tasks Sun Grid Engine, version 6.2, also called for a shorter SGE.

Jobs

Jobs are the basic units with which SGE operates and runs . They are plain shell scripts, supplemented with special SGE parameters in the form of special syntax comments. Here is an example task with the more commonly used parameters:

#!/bin/bash
# File mytask.job

# Comment
# These lines are not processed by the SGE or the shell

# The following lines are processed only by SGE
#$ -N myjob
#$ -cwd
#$ -m bea
#$ -M user@domain.com,user2@domain2.com
#$ -P project043
#$ -l h_rt=1:0:0

# Followed simple shell script
./process_result > $TMPDIR/alabala.out
mv $TMPDIR/alabala.out

The parameters for SGE are placed in comment rows that start with the special combination #$ and have the same look as the queue command options in the queue of SGE qsub. Since all rows that begin with # are perceived as comments in the script, the last ones only make sense to the system SGE and are not interpreted by the shell. * The default wrapper is /bin/bash, and other shells are not supported by the administration *. Different parameters have the following meaning:

  • -N myjob — sets the name of the job under which it will be visible in the job list;
  • -cwd — indicates that the output files containing the standard output and error of the task should be located in the current directory instead of the home;
  • -m bea — indicates that an email should be sent at start, finish, and / or unusual interruption;
  • -M user1<AT>domain1.com,user2<AT>domain2.com — Specifies the addresses to which e-mails should be sent when the events listed in the parameter -m (insted of <AT>, @ shoud be placed);
  • -P project043 — indicates that the task should be executed as part of a named project project043. The resources used will be deducted from the “account” of the project. The assignment of the master project you are assigned to this parameter is redundant as it is implied. * You must be assigned to a project so you can use its resources and quotas *;
  • -l h_rt=1:0:0 — dicates that the job requires a maximum of 1 hour of work time. Upon expiration of this time interval, the task is forced out of SGE . * Setting this parameter is required! Omitting it will cause the task to terminate 5 minutes after it starts. *

The job is passed to SGE using the command:

qsub mytask.job

Instead of specifying in the job file, the parameters for SGE can also be submitted using the command line via qsub. For example:

qsub -N myjob -cwd -l h_rt=0:30:0 mytask.job

Command line parameter values override those set in the script file. This allows the same script to run several tasks with different parameters.

All resources that are not explicitly requested receive default values. If the requirements of the task exceed the limits of available resources, it will either not be executed or will be put in wait for an indefinite period of time.

Launching background processes within the job (using the & operator) is not recommended as it leads to over usage of the requested resources!

Non-interactive tasks

Non-interactive tasks do not use a graphical or text interface to interact with the user during their work. These are all the processes that could be run in the background, as well as the processes that allow you to redirect their standard input and output to and from external files. Typical examples are Quantum ESPRESSO, GROMACS, AMD, etc., which read their file configurations or standard input in non-interactive mode and write their results either in files or in standard exit.

The conversion of a non-interactive application into a SGE task is in most cases trivial, as most finished products provide the so-called driver scripts performing a sequence of actions on the data for and from the simulation. It is only necessary to extend the corresponding driver script with the parameters for SGE.

Interactive tasks

Interactive are tasks that require a dialogue with the user and can not work in batch mode. As such, all graphical applications (Mathematica, Maple, MATLAB, etc.) are classified when running in the dialog, as well as some of the command line applications whose input and output can not be redirected.

Starting graphical interactive tasks

Graphic interactive tasks are started with the command

qsh -l h_rt=hh:mm:ss [other parameters...]

where hh:mm:ss is the time of maximum session length. Other resources, such as virtual memory size with -l h_vmem=8G, can be requested. After submitting this command SGE attempts to satisfy within a few seconds the desired resources, with the xterm graphical terminal. In case of failure, an error message is received and the interactive session is denied. This happens if there are no free slots available for interactive work or if the requested resources exceed the available ones. For proper startup of the graphical terminal, you should have a working X server on your computer and the $DISPLAY environment variable set correctly. The latter happens automatically if you use tunneling X11 through the SSH session to the cluster. For more details, see the “graphic apps” page:graphical-apps-en.

Non-graphical interactive tasks

For tasks that do not require the presence of an X11 server, non-graphic plug-in is recommended. For this purpose, the qsh command from the previous point is replaced by qlogin. The parameters remain the same.

Parallel jobs

Parallel tasks are a special class of tasks that require more than one processor (or processor core) to perform. For this purpose, such tasks are started in the so-called Parallel environments. The latter are a mechanism that points to SGE the ways in which a parallel task is started, executed and ended. At the moment, the following parallel environments are available:

  • threads — for multi-threaded applications;
  • ipar — for parallel / multi-threaded interactive applications;
  • ompisn — for parallel applications using the library OpenMPI and from 1 to 8 slots on one machine;
  • ompix8 — for parallel applications using the library OpenMPI and multiple of 8 slots.
  • ompix12 — for parallel applications using the library==OpenMPI== and multiple of 8 slots.

Within a parallel environment, tasks use a certain number of slots, one slot corresponding to one processor core. For threads in multi-threaded applications, the number of slots must match the maximum number of threads, and due to the architectural features of the cluster, no more than 8 (eight) or 12 (twelve) slots can be requested in this mode. If the specified number of slots is up to 8 (eight), your assignments can be allocated to either 8 core processor nodes or 12 (twelve) core processor nodes. When the number of slots requested is between 9 (nine) and 12 (twelve) your tasks will be allocated to 12 (twelve) core processor nodes. Starting more threads than the requested number of slots leads to over-consumption and affects the foreign tasks running on the same computing node. If your application uses more than 12 threads, please specify 12 slots!

For parallel MPI applications, the number of slots sets the number of processes that will participate in the parallel task. Because the different processes in MPI applications are not limited to the requirement to run on the same node, thus more than 8 slots may be requested. The current hardware configuration allows up to 160 slots per job (up to 184 slots in special cases).

Usage of parallel environments

To start a task as a parallel one, it is enough to set the following option to SGE:

-pe environment #slots

environment is the name of the parallel environment (e.g., ompix8) and #slots is the requested number of slots. This parameter can be embedded in the job script or specified as an argument via qsub.

For example, for a 16-processor MPI application in the job script the following can be placed:

#$ -pe ompix8 16

It should be borne in mind that all resources, except real time h_rt, are treated as slotted, i.e. the total claimed consumption is obtained as a product of the requested resources by the number of slots.

Administrative restrictions

Since parallel tasks block large amounts of resources simultaneously, each user can use up to 184 slots in all parallel environments combined. The restriction is subject to modification in special cases.

Parallel interactive tasks

It is possible to launch parallel interactive tasks in which parallelism is achieved using multiple threads. For this purpose of the qsh command (or qlogin in the non-graphical case), should be instructed to use the ipar parallel environment. For example:

qsh -l h_rt=1:0:0 -pe ipar 4

This command specifies 4 cores for a multi-threaded application (Java, OpenMP, or other mechanism). After starting the interactive session, you should indicate to the applications you are working with that they may use 4 threads. The mechanisms by which this is done are specific to the various applications. You should find the mechanism to determine the level of … of the applications you work with — SGE can not restrict the programs to the number of cores on which they are in practice. The actual use of more cores than the requested number of slots violates the work of the other tasks and should be considered as “unsportsmanlike” use of the machine.

Аdvance reservations

It is possible to reserve some of the cluster resources for exclusive use within a certain time. To do this, use the qrsub command:

qrsub -d 16:0:0 -pe ompix8 32 -m b -M user@domain

This command reserves 32 slots in the parallel environment ompix8 for 16 hours. Upon successful booking, the command returns the unique reservation number. SGE will send an e-mail to the specified address when all required resources become available. It is possible to explicitly indicate the start time of the reservation with the option -a YYMMDDhhmm and / or the end time with the option -e YYMMDDhhmm.

Jobs can be sent within the reservation by specifying the reservation number in the -ar parameter of the qsub, qlogin and qsh commands:

qsub -ar 30 -l h_rt=1:0:0 -pe ompix8 32 myjob

Remarks

Аdvance reservations are subject to special status within the system SGE. The resources used by tasks within the reservations are not subject to standard constraints and controls. For this reason, the creation of reservations is restricted to users who are members of a special group. Please contact system administrators if you need prior reservation for your tasks.

Limits

In order to share the cluster resources fairly, there are some limitations detailed here.

More information