--- title: "SLURM Guide" canonical: "https://kb.uconn.edu/space/SH/26032963685/SLURM%20Guide" format: markdown --- ## Overview To optimally and fairly use the cluster, all application programs must be run using the job scheduler, SLURM. When you use SLURM's `sbatch` command, your application program gets submitted as a "job". To better understand how applications get submitted as jobs, let's review the difference between login nodes and compute nodes. ***Login nodes***: When you connect to the cluster and see `[@login4 ~]`, you are connected to a single shared computer with all your fellow users, known as the "login node". The purpose of the "login" node is for you to submit jobs, copy data, edit programs, etc. The programs that are allowed to run on login nodes is listed in our [usage policy](https://uconn.atlassian.net/wiki/spaces/SH/pages/26032800737). ***Compute nodes: ***These computers do the heavy lifting of running your programs. However, you do not directly interact with compute nodes. You ask the scheduler for compute nodes to run your application program using SLURM, and then SLURM will find available compute nodes and run your application program on them. > ℹ️ Please **do not** run computationally-intensive programs on the login nodes. Doing so may slow down performance for other users, and your commands will be automatically throttled or terminated. ## Job Submission 1. First, log in to the cluster with your NetID: 2. Use `nano` or your favorite text editor to create your job submission script. Here is a very simple job example: 3. Save your submission script and then submit your job `sbatch`: 4. You can view the status of your job with the `squeue --me`command (described later in this guide: > ℹ️ The output of your job will be in the current working directory in a file named `slurm-JobID.out` by default, where `JobID` is the number returned by `sbatch` in the example above. ## Job Examples The resources of the HPC cluster are segmented into groups called Partitions. All jobs submitted to the cluster run within one of these Partitions. If you do not select a Partition explicitly the scheduler will put your job into the default Partition, which is called `general`. Each Partition has defined limits for job runtime and core usage, with specific details available on the [usage policy](https://wiki.hpc.uconn.edu/index.php/Usage_policy) page. You can view a list of all partitions and their status by running the `sinfo` command. There is also a knowledge base article referencing the available Partitions and the options that can be used to submit to the partitions located [here](https://kb.uconn.edu/space/SH/26032963610/SLURM+Partitions+and+Job+Scheduling). Below are multiple examples of how to submit a job in different scenarios. **Default (general) partition** The example job below requests 48 CPU cores for two hours, and emails the specified address upon completion: ``` #!/bin/bash #SBATCH --partition=general # Name of partition #SBATCH --ntasks=48 # Request 48 CPU cores #SBATCH --time=02:00:00 # Job should run for up to 2 hours (for example) #SBATCH --mail-type=END # Event(s) that triggers email notification (BEGIN,END,FAIL,ALL) #SBATCH --mail-user=first.last@uconn.edu # Destination email address myapp --some-options path/to/app/parameters # Replace with your application's commands ``` **Test (debug) partition** This Partition allows you to request a single node and run for up to 30 minutes. ``` #!/bin/bash #SBATCH --partition=debug # Name of Partition #SBATCH --ntasks=4 # Maximum CPU cores for job #SBATCH --nodes=1 # Ensure all cores are from the same node #SBATCH --time=5 # Job should run for up to 5 minutes (for example) #SBATCH --mail-type=END # Event(s) that triggers email notification (BEGIN,END,FAIL,ALL) #SBATCH --mail-user=first.last@uconn.edu # Destination email address myapp --some-options path/to/app/parameters # Replace with your application's commands ``` **MPI-optimized (hi-core) partition** ** **This Partition allows you to request up to 384 cores and run for up to 6 hours. ``` #!/bin/bash #SBATCH --partition=hi-core # Name of Partition #SBATCH --ntasks=240 # Request 256 CPU cores #SBATCH --time=01:30:00 # Job should run for up to 1.5 hours (for example) #SBATCH --mail-type=END # Event(s) that triggers email notification (BEGIN,END,FAIL,ALL) #SBATCH --mail-user=first.last@uconn.edu # Destination email address mpirun myapp --some-options path/to/app/parameters # Replace with your application's commands ``` **Single node (lo-core) partition** ** **The `lo-core` partition allow you to request up to 7 days. ``` #!/bin/bash #SBATCH --partition=lo-core # Name of Partition #SBATCH --ntasks=24 # Maximum CPU cores for job #SBATCH --nodes=1 # Ensure all cores are from the same node #SBATCH --time=02-00:00:00 # Job should run for up to 2 days (for example) #SBATCH --mail-type=END # Event(s) that triggers email notification (BEGIN,END,FAIL,ALL) #SBATCH --mail-user=first.last@uconn.edu # Destination email address myapp --some-options path/to/app/parameters # Replace with your application's commands ``` --- **Running a multi-threaded job on a complete node** In some cases, one is running a single multi-threaded program that needs to be spawned on a given node. We need the program to have access to all available CPU cores on that node. To ensure all the CPUs of a given node are allocated to the program on that node, we can use the `--ntasks=1` flag combined with the `--cpus-per-task=` flag. The value `--cpus-per-task=` flag is set to should be equal to the number of cores available on a given node, but that will vary from node to node. We have included a couple of examples below. **AMD-Epyc Nodes (128 or 64 cores)** ``` #!/bin/bash #SBATCH --partition=general # selecting a partition #SBATCH --constraint=epyc128 # requesting an AMD-Epyc 128-core node #SBATCH --cpus-per-task=126 # 126 cores will be used per task #SBATCH --ntasks=1 # number of tasks in this batch job #SBATCH --nodes=1 # all 126 cores to come from 1 node #SBATCH --mem=16G # memory allocation #SBATCH --time=02:00:00 # job runtime (hh:mm:ss) bash my_script.sh ``` Change the constraint to `epyc64` and CPUs per task to 62 to target the 64-core nodes. Oftentimes, instead of specifying both `--cpus-per-task=` and `--ntasks=`, only `--ntasks=` needs to be defined with `--cpus-per-task` adopting the default value of `1`. It is recommended to test these options with your desired programs to ensure complete utilization of your requested CPU resources. > ℹ️ **NOTE:** Two CPU cores of each compute node are reserved for administrative processes required for the HPC to function. This means that of the 128 cores in a Epyc128 node, only 126 can be requested through SLURM, and the same applies to other node architectures. > ℹ️ > ℹ️ ![image-20250904-163709.png](media://40b53b7f-a836-48e4-a66b-97ba822aa187) **General GPU Nodes** ``` #!/bin/bash #SBATCH --partition=general-gpu # selecting a partition #SBATCH --constraint=epyc64,a100 # requesting an NVIDIA-A100 Epyc64 node #SBATCH --ntasks=62 # request 62 CPU cores for this batch job #SBATCH --nodes=1 # all 126 cores to come from 1 node #SBATCH --gres=gpu:1 # requesting 1 GPU #SBATCH --mem=16G # memory allocation #SBATCH --time=02:00:00 # job runtime (hh:mm:ss) bash my_script.sh ``` Replace the `bash my_script.sh` line with your multi-threaded program command. For more information on available resources, please see our **[Storrs HPC Resources page](https://kb.uconn.edu/space/SH/26032963610/Partitions+%2F+Storrs+HPC+Resources)** for the number of cores available on nodes of different architectures. > ℹ️ For a more extensive list of flags that can be used with the #SBATCH header or the `srun `command, see this **[table](https://kb.uconn.edu/space/SH/26449379370/SLURM+Cheatsheet#Job-Submission)**** **from our SLURM Cheatsheet. --- ## How to test jobs using the Debug Partition The debug partition is a great way to troubleshoot and test code before running on a node quickly without running into long wait times. The above Debug Partition test example shows how to submit to the debug partition, request 4 cores and 1 node to the debug job. The following example will go into further detail for the Debug Partition and show a different way that can be used to debug code to determine if there is a potential issue or to confirm that the code can run on the HPC hardware at the time of submission. Different hardware is available to test within the Debug partition, which allows for users to troubleshoot their code on specific architectures. **AMD EPYC node debug test** **Test (debug) partition for AMD EPYC nodes: **This example will allow you to request a single node, and run for up to 30 minutes on an AMD EPYC node using the max of 128 CPU cores on an AMD EPYC node and run for 5 minutes. ``` #!/bin/bash #SBATCH --partition=debug # Name of Partition #SBATCH --ntasks=126 # Maximum CPU cores for job #SBATCH --nodes=1 # Ensure all cores are from the same node #SBATCH --constraint='epyc128' #SBATCH --time=5 # Job should run for up to 5 minutes (for example) #SBATCH --mail-type=END # Event(s) that triggers email notification (BEGIN,END,FAIL,ALL) #SBATCH --mail-user=first.last@uconn.edu # Destination email address myapp --some-options path/to/app/parameters # Replace with your application's commands ``` ## RAM Job Submission allocation There are multiple options that can be used to allocate memory to a SLURM job submission script. Out Of memory issues are fairly common now, if memory is not specified within a job submission script. To avoid potential Out of Memory errors, there are a couple of ways to designate a RAM assignment within the job script. There will be a couple of examples below showcasing different ways to allocate memory to a Slurm job that submits to the general partition. **Full node memory allocation AMD EPYC node with 128 CPU cores** ``` #!/bin/bash #SBATCH --partition=general # Name of Partition #SBATCH --ntasks=126 # Maximum CPU cores for job #SBATCH --nodes=1 # Ensure all cores are from the same node #SBATCH --constraint=epyc128 # Target epyc128 nodes #SBATCH --mem=492G # Request 500GB of available RAM on an AMD EPYC node with 128 cores #SBATCH --mail-type=END # Event(s) that triggers email notification (BEGIN,END,FAIL,ALL) #SBATCH --mail-user=first.last@uconn.edu # Destination email address myapp --some-options path/to/app/parameters # Replace with your application's commands ``` **Full node memory allocation and request all available GPU cards** ``` #!/bin/bash #SBATCH --partition=general-gpu # Name of Partition #SBATCH --ntasks=62 # Maximum CPU cores for job #SBATCH --nodes=1 # Ensure all cores are from the same node #SBATCH --constraint=a100,epyc64 # Target NVIDIA-A100 Epyc64 nodes #SBATCH --mem=492GB # Request 500 GB of available RAM #SBATCH --gres=gpu:3 # Request 3 GPU cards for the job #SBATCH --mail-type=END # Event(s) that triggers email notification (BEGIN,END,FAIL,ALL) #SBATCH --mail-user=first.last@uconn.edu # Destination email address myapp --some-options path/to/app/parameters # Replace with your application's commands ``` ## Testing/Debugging code if a job fails It is recommended (if there is a job failure with potential coding issues) to debug and test the code under the debug partition. It is also recommended, to test the code with a simple test program to see if a simple program works when submitting to the debug partition. If a simple test program works, then there might be issues with the current input file or code being executed within the job submission script. ## Monitoring Jobs **Monitoring active jobs** Submission information such as the start time, deadline, requested resources, and file paths to the submission script, working directory, and the slurm.out file are recorded and accessible with the following command: ``` scontrol show job {JobID} ``` Note that this information is only available from jobs that are actively running. `scontrol show job` will not work for completed/failed jobs. **Monitoring completed jobs - CPU and memory usage** SLURM records statistics for every job, including how much memory and CPU was used. After the job completes, you can run `seff ` to get some useful information about your job, including the memory used and what percent of your allocated memory that amounts to. ``` [jth10001@node ~]$ seff 2262973 Job ID: 2262973 Cluster: slurm User/Group: jth10001/domain users State: COMPLETED (exit code 0) Nodes: 1 Cores per node: 62 CPU Utilized: 00:19:42 CPU Efficiency: 0.20% of 6-16:16:12 core-walltime Job Wall-clock time: 02:35:06 Memory Utilized: 11.67 GB (estimated maximum) Memory Efficiency: 9.41% of 124.00 GB (2.00 GB/core) ``` It is recommended to test various job submissions to utilize memory based on the code being submitted at the time. > ℹ️ **NOTE:** By default, 2 GB of RAM is assigned per core allocated to each job. For example, a 24-core job will have 48 GB of RAM by default. ## Interactive jobs > ⚠️ As of August 2024, we are no long supporting `fisbatch` on the HPC. We recommend using `srun` for all of your interactive job needs. If you require an interactive job, use the `srun` command instead of `sbatch`. This command does not use a submission script. Instead, all of the options from the submission script are given on the command line, without the `#SBATCH` keyword. 1. Basic example: start an interactive job with 12 cores 2. To use a custom partition with interactive jobs, specify the `--partition` parameter: 3. If you suddenly lost the connection from the interactive screen, you can try to re-attach. First you need to get the JobID of the `srun` job: Then, you can re-attach the job by JobID using the`sattach` command: > ℹ️ **NOTE:** Please don't forget to exit when you finish your job. And, although many programs have a graphical interface, we recommend that all jobs use a command-line interface if supported by the application. > ℹ️ **NOTE:** It is not guaranteed to access the screen successfully every time. If it fails, it means this interactive job is not accessible anymore. Please `scancel` it. > ℹ️ **NOTE:** If you would like to spawn an interactive job to run GUI software, please see the following guide: > ℹ️ > ℹ️ [https://kb.uconn.edu/space/SH/26033914267](https://kb.uconn.edu/space/SH/26033914267) > ℹ️ **NOTE: **If a job submission fails with the error: **Unable to allocate resources: Invalid account or account/partition combination'**, then the following SLURM header would need to be specified in the job submission: > ℹ️ > ℹ️ #SBATCH --account=PINetidHere > ℹ️ > ℹ️ Or > ℹ️ > ℹ️ Through an interactive SLURM srun command below: > ℹ️ > ℹ️ srun --account=PINetidHere --ntasks=12 --nodes=1 --partition=general --pty bash ## Checking the Status of a Job To view your active jobs: 2. Alternatively, the `squeue` command may be more descriptive for jobs in a `PENDING` state: 3. To view your job history: 4. To view all the jobs in the cluster: 5. To view details about nodes and partitions: 6. To review all the job logs: ## How to Release a Job from “JobHeldUser” If you have a job that has been “held,” it will not be able to run it is “released.” You will know your job is being held if [squeue ](https://kb.uconn.edu/space/SH/26449379370/SLURM+Cheatsheet#squeue)says the job state is SE (Special Exit) and the reason says “JobHeldUser.” We have a deeper explanation for why this happens in our FAQ [here](#). But in brief, you can release a job using its jobID or jobName. 1. To release a single job 2. To release all jobs with a given job name > ℹ️ Please note that jobs which are left in the queue in the ‘SE’ state will be **cancelled** after 2 days. ## How to Terminate a Job 1. To terminate a single job: 2. To terminate all of your jobs: ## Priority Jobs If you have been granted access to priority resources through [our condo model](https://kb.uconn.edu/space/SH/26698678562/Priority+Access) then you need to submit your jobs using a custom Partition in order to avoid resource limits. > ℹ️ **NOTE:** Priority user jobs will time out after 24 hours if a time frame was not specified in the job submission script. 1. For example: The following example can be used to specify a 2 day run time for a job: ``` #SBATCH --time=2-00:00:00 ``` > ℹ️ For an explanation of available partitions, please refer to [Partitions and Job Scheduling](https://uconn.atlassian.net/wiki/spaces/SH/pages/26032963610) ## Checking the state of nodes and partitions You can view a listing of nodes and see what resources are currently being used. with the `nodeinfo`command. This may be helpful for determining whether there are available nodes with sufficient resources available to backfill your jobs. The following example will show all idle or mix state nodes available in the general-gpu partition: ``` [jth10001@login6 ~]$ nodeinfo -p general-gpu -t idle,mix PARTITION NODES STATE TIMELIMIT CPUS GRES MEMORY ACTIVE_FEATURES NODELIST general-gpu 5 mix 12:00:00 64 gpu:3 515404 epyc64,a100,gpu gpu[14,21-22,29,37] general-gpu 4 mix 12:00:00 64 gpu:1 515404 epyc64,a100,gpu gpu[17-18,33-34] general-gpu 1 idle 12:00:00 64 gpu:1 515404 epyc64,a100,gpu gpu26 general-gpu 1 idle 12:00:00 64 gpu:3 515404 epyc64,a100,gpu gpu38 ``` Jobs in the “idle” state have all cores available, whereas jobs in the “mix” state have at least one core available. If you want to target the remaining available cores on a node in the mix state, it can be helpful to figure out how many cores are available. To check that we can use the `sinfo -n ` command to look at a specific node like so, in this case `gpu14`: ``` [jth10001@login6 ~]$ sinfo -n gpu14 -p general-gpu -o%10n%20R%10t%20C HOSTNAMES PARTITION STATE CPUS(A/I/O/T) gpu14 general-gpu mix 36/28/0/64 ``` This output reports CPUs in one of four states: A, allocated; I, idle; O, other (ignore this one); and T, total. From this output we can see that 36 of gpu14’s 64 cores are allocated and 28 are available. Now, the one tricky part about the Storrs HPC is that even though it says 28 are available, there are really only 26 available because the first two cores on every node in the general access partitions are reserved for HPC administration tasks. So if we wanted to target gpu14, we would have to use a command like this which asks for 26 cores (or less). ``` srun -p general-gpu --nodelist=gpu14 -N 1 -n 26 --pty bash ``` ## Job State Codes | Status | Code | Explaination | | --- | --- | --- | | COMPLETED | `CD` | The job has completed successfully. | | COMPLETING | `CG` | The job is finishing but some processes are still active. | | FAILED | `F` | The job terminated with a non-zero exit code and failed to execute. | | PENDING | `PD` | The job is waiting for resource allocation. It will eventually run. | | PREEMPTED | `PR` | The job was terminated because of preemption by another job. | | RUNNING | `R` | The job currently is allocated to a node and is running. | | SUSPENDED | `S` | A running job has been stopped with its cores released to other jobs. | | STOPPED | `ST` | A running job has been stopped with its cores retained. | | SPECIAL EXIT | `SE` | The job failed. Check logs to determine reason and either release or cancel. | A full list of these Job State codes can be found in [Slurm’s documentation.](https://slurm.schedmd.com/squeue.html#lbAG) ## Job Reason Codes | Reason Code | Explaination | | --- | --- | | `Priority` | One or more higher priority jobs is in queue for running. Your job will eventually run. | | `Dependency` | This job is waiting for a dependent job to complete and will run afterwards. | | `Resources` | The job is waiting for resources to become available and will eventually run. | | `InvalidAccount` | The job’s account is invalid. Cancel the job and rerun with correct account. | | `InvaldQoS` | The job’s QoS is invalid. Cancel the job and rerun with correct account. | | `QOSGrpCpuLimit` | All CPUs assigned to your job’s specified QoS are in use; job will run eventually. | | `QOSGrpMaxJobsLimit` | Maximum number of jobs for your job’s QoS have been met; job will run eventually. | | `QOSGrpNodeLimit` | All nodes assigned to your job’s specified QoS are in use; job will run eventually. | | `PartitionCpuLimit` | All CPUs assigned to your job’s specified partition are in use; job will run eventually. | | `PartitionMaxJobsLimit` | Maximum number of jobs for your job’s partition have been met; job will run eventually. | | `PartitionNodeLimit` | Node number requested exceeds limit; user needs to decrease nodes requested to below limit. | | `AssociationCpuLimit` | All CPUs assigned to your job’s specified association are in use; job will run eventually. | | `AssociationMaxJobsLimit` | Maximum number of jobs for your job’s association have been met; job will run eventually. | | `AssociationNodeLimit` | All nodes assigned to your job’s specified association are in use; job will run eventually. | A full list of these Job Reason Codes can be found [in Slurm’s documentation.](https://slurm.schedmd.com/squeue.html#lbAF) ## \uD83D\uDCCB Related articles > Macro (contentbylabel)