About SLURM Scheduler

In the Central Cluster, we use SLURM as cluster workload manager which used to schedule and manage user jobs running. SLURM will handle the job queueing and compute nodes allocating also start and executing the jobs. By running any application or script on the compute nodes, you need to submit the job to SLURM. SLURM system consider a fair-share factor when scheduling your job priority. The job's priority influences the order in which a user's queued jobs are scheduled to run based on the portion of the computing resources they have been allocated and the resources their jobs have already consumed. You can always check your fair-share details and the usage of your account by sshare command. For the details of the fair-share calculation, you may refere to SLURM Offical fair-share documentation. The usage of different compute node consumed will calculate as following table:

Submitting Job

To submit job in SLURM, sbatch, srun and salloc are the commands use to allocate resource and run the job. All of these commands have the standard options for specifiy the resources required to run your job. Usually, we will create a script for setup the environment and run the command or program which require to run in HPC platform. In this situation, the command sbatch is the best option for running a script because we can add the job specification using #SBATCH at the top of the script. Here is an example of how to run the script with the specification in the command line and the equivalent way to add the specification inside the script:

sbatch -J JobName -N 2 -c 16 jobscript.sh

By entering the above command, SLURM will allocate 2 nodes with 16 core each with a job name "JobName" to run jobscript.sh. this specification is required for running this job script every time, you may mention them inside the shell script as below:

#!/bin/bash
#SBATCH -J JobName
#SBATCH -N 2 -c 16

Then you just need to run the script with sbatch and SLURM will load the specification from your script.

sbatch jobscript.sh

Here is a list of the option for the command sbatch, srun, and salloc:

Here is a video reference for how to use Batch Scripting with SLURM:

Submit Job to GPU Nodes

You need to use --gres option to request GPU card for your job. You may mention the option inside you batch like this:

#!/bin/bash
#SBATCH -J JobName
#SBATCH -N 1
#SBATCH --gres=gpu:2

GPU is a type of Generic Resource(GRES) inside SLURM. In the above example, the SLURM will assign 2 GPU card for you job.You may also mention the GPU type like this:

#!/bin/bash
#SBATCH -J JobName
#SBATCH -N 1
#SBATCH --gres=gpu:K80:2

If your batch require to assign different job to run in separate GPU, you may need to use srun with --exclusive option inside your batch file:

#!/bin/bash
#SBATCH -J JobName
#SBATCH -N 1
#SBATCH -n 4
#SBATCH --gres=gpu:4
srun --gres=gpu:2 -n2 --exclusive task1.sh &
srun --gres=gpu:1 -n1 --exclusive task2.sh &
srun --gres=gpu:1 -n1 --exclusive task3.sh &
wait

Checking the Job Status

After you submit the job, you can check the job status by using the command squeue. You may use the option -l with squeue to show the long report of the running or pending job. When the job are cancelled or completed, these jobs are not shown by the squeue command. By default the squeue output contains the job id, partition of job, name of the job, job owner name, job state, time used by the job, nodes allocated information and the and the job pending reason​. If you want more information of the job, you may use the command scontrol show job to have a full information about the job. You may specify the job id to view the target job details by the command scontrol show job <job_id>.

Controlling the Submitted Job

If the running or pending job does not run as your desire, you may use the command scancel to cancel the running or scheduled job. To cancel a job you must mention the job id with the command scancel <job_id>. After the cancel request submit, SLURM will stop the job as soon as possible. If you want to cancel all the jobs you have submitted, you may specify you username with the command scancel -u <username>. All your scheduled jobs will be terminated.

Other than cancelling the job, you may also modify the job by using scontrol command but some of the specifications cannot be modified after the job is running. For example, you may use the command scontrol update JobId=<job_id> NumNodes=<minnodes> to modify the node number requested by the job. If the job has not started, you can increase or decrease the node number requested. If the job has already started, you can only decrease the number of nodes runnning the job only. For more details about modifing the job, you may reference to SLRUM FAQ: 24.Can I change my job's size after it has started running.

PBS to SLURM

If you are using PBS batch script to submit a job in PBS, you only need to change the command for submitting job to SLURM from qsub to sbatch. SLURM will understand all PBS batch script option used in the script. Just run the script with sbatch and the SLURM will submit the job according to your PBS batch script option. The common problem you may face will be the "$PBS -q" option. SLURM uses the PBS queue name as SLURM partition name. If you face this problem, just remove the "$PBS -q" line and the job will be accepted. Even SLURM accept PBS batch script options, you should either use the PBS batch script option or SLURM bactch script option.

The following charts list the difference between PBS and SLURM commands, environment variables and job specifications: