> For the complete documentation index, see [llms.txt](https://docs-ksc.gitbook.io/neuron-user-guide-eng/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs-ksc.gitbook.io/neuron-user-guide-eng/system/running-jobs-through-scheduler-slurm.md). # Running Jobs Through Scheduler (SLURM) The Neuron system’s job scheduler uses SLURM. This chapter introduces the method that is used to submit jobs through SLURM and relevant commands. For information on how to write a job script when submitting a job to SLURM, refer to \[Appendix 1] and the example showing how to write a job script file. **※ User jobs can only be submitted from /scratch/$USER.** ## A. Queue Configuration wall clock time : 2 days (48 h) In all queues (partitions), job allocation follows the shared node policy (multiple jobs can run simultaneously on a single node).\ (22.03.17. Update) : To enhance resource efficiency, the policy has been changed from exclusive node usage to shared node usage. Job queue (partitions) * The partitions available to general users include jupyter, cas\_v100nv\_8, cas\_v100nv\_4, cas\_v100\_4, cas\_v100\_2, amd\_a100nv\_8, skl, and bigmem. (You can check the number of nodes, maximum job runtime, and node list using the sinfo command.) Job Submission Quantity Limitations * Maximum number of jobs per user :\ An error occurs if jobs exceed the submission limit. * Maximum number of concurrent jobs per user :\ Jobs wait if they exceed the running job limit. Resource Occupancy Limit * Maximum number of nodes per job :\ Jobs will not execute if they exceed the node limit. This limit is independent of the total number of nodes occupied by running jobs. * Maximum number of GPUs per user :\ This setting limits the total number of GPUs each user can occupy, and if the limit is exceeded, jobs wait until previous jobs are completed. It limits the number of GPUs that a user's running jobs can occupy at any given time. **※ Node configuration and partitioning may be adjusted during system operation based on system usage.** ## B. Job Submission and Monitoring ### 1. Summary of the basic commands | **Command** | Description | | --------------------------- | -------------------------------------------- | | $ sbatch \[option..] script | Submit a job | | $ scancel Job ID | Delete a job | | $ squeue | Check job status | | $ smap | Check job status and node status graphically | | $ sinfo \[option..] | Check node information | ※ You can use the “sinfo –help” command to check the “sinfo” command options. **※ To accurately analyze user resource usage on the Neuron system and to establish effective operational policies based on AI/HPC application fields, it is mandatory to specify the application field (field) and application name (application) with the #SBATCH option.** \ **Specifically, before submitting a job, you must include the --comment option in your job script according to the field and application you are using, referring to the below table.**\ **\[Format] #SBATCH --comment="field=\;appl=\"** **※ Only pre-defined values are allowed for the field (field) and application (application) parameters. You can check the supported values by referring to the tables below (SBATCH option values for field/application) or by running the following command on the login node:**\ **\[Command] $ showappl** **※ The classification of applications will be updated periodically based on collected user requests. If you wish to add an application, please submit a request to .** **\[Table: field values for SBATCH option]**
CategoryFieldDescription
AIagenticaiAgentic AI
AIroboticsPhysical AI & Robotics
AIaimAutonomous Driving & AI Mobility
AInlpNatural Language Processing, Large Language Models
AIcvComputer Vision & Spatial Intelligence
AIspeechSpeech, Audio AI
AIrecsysRecommendation System & Decision Making
AImultimodalMultimodal, Generative AI, World Model, Media Synthesis
AIsciaiAI for Science & Graph AI
AIefficientaiEfficient & Scalable AI Systems, Quantization, Edge AI
AIhealthaiHealthcare AI
AIfinaiFinance AI
AImiscMiscellaneous
HPCmathMathematics
HPCphysPhysics
HPCchemChemistry
HPCaacAstrophysics and Cosmology
HPCeesEarth and Environmental Science
HPCbioBio/life science
HPCmedMedical/Health science
HPCbnsBrain and Neurosciences
HPCagfAgriculture, Fishery and Food
HPCqstQuantum Science and Technology
HPCmseMaterial Science and engineering
HPCereEnergy and Resources engineering
HPCmaeMechanical and Aerospace engineering
HPCneNuclear engineering
HPCcheChemical Engineering
HPCeeeElectrical and electronic engineering
HPCictInformation/Communication Technology
HPCcseComputer science and engineering
HPCcteCivil and transportation engineering
HPCssSocial Science
HPCcaaCulture and Arts
HPChumHumanities
HPCmiscMiscellaneous
**\[Table: application values for SBATCH option]**
categoryappl namecategoryappl namecategoryappl name
AIpytorchAIjaxAItriton
AIpytorch-ddpAItensorflowAIjupyter
AIpytorch-fsdpAItensorrtAIr
AIdeepspeedAIollamaAImisc
AImegatron-lmAIvllm
HPCabaqusHPCgrimsHPCq-chem
HPCamberHPCgromacsHPCqmcpack
HPCansysHPCkflowHPCquantum_espresso
HPCberkeleygwHPClammpsHPCr
HPCbwaHPCls-dynaHPCramses
HPCcastepHPCmaterials_studioHPCraspa
HPCcesmHPCmatlabHPCroms
HPCcgyroHPCmfdnHPCrosetta
HPCcharmmHPCmomHPCsiesta
HPCcp2kHPCmpasHPCstar-ccm
HPCfhi_aimsHPCmsc_oneHPCsu2
HPCfvcomHPCnamdHPCvasp
HPCgadgetHPCnw_chemHPCwannier90
HPCgatkHPCopenfoamHPCwien2k
HPCgaussianHPCopenmmHPCwrf
HPCgeant4HPCopenmxHPCxtb
HPCgotpmHPCorcaHPCin_house
HPCgpawHPCpyfrHPCmisc
### **2. Batch job submission** Submit a job by using the sbatch command as in “sbatch {script file}”. ```shell-session $ sbath [UserJob.script] ``` Check job progress You can check the status of the job by accessing the allocated node. #### 1) Use the squeue command to check the node name (NODELIST) to which the running job is assigned. ``` $ squeue -u [userID] JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON) 99792 cas_v100_4 ior userID RUNNING 0:12 5:00:00 1 gpu25 ``` #### 2) Access the corresponding node using the ssh command ```shell-session $ ssh gpu25 ``` #### 3) Once on the compute node, you can check the job's progress using the top or nvidia-smi commands ※ Example of monitoring GPU usage at 2-second intervals ```shell-session $ nvidia-smi -l 2 ``` Example of writing a job script file * To perform batch jobs in SLURM, you need to create a job script file using SLURM keywords. ※ Refer to “\[Appendix 1] Main Keywords for Job Scripts”. ※ Refer to the KISTI supercomputing blog () for information on how to use the machine-learning framework conda. * SLURM keywords | Keyword | Description | | ------------------------------------------------------ | ----------------------------------------------- | | #SBATCH –J | Specify the job name | | #SBATCH --time | Specify the maximum time to run the job | | #SBATCH –o | Specify the file name of the job log | | #SBATCH –e | Specify the file name of the error log | | #SBATCH –p | Specify the partition to be used | | #SBATCH --comment="field=\;appl=\" | Specify the field & application to be used | | #SBATCH -–nodelist=(Node List) | Specify the node on which the job will be run | | #SBATCH -–nodes=(Number of Nodes) | Specify the number of nodes to use for the job | | #SBATCH —ntasks-per-node=(Number of Processes) | Specify the number of processes to run per node | | #SBATCH —cpus-per-task=(Number of CPU Cores) | Number of CPU cores allocated per process | | #SBATCH —cpus-per-gpu=(Number of CPU Cores) | Number of CPU cores allocated per GPU | | #SBATCH --exclusive | Option for exclusive use of nodes | **Set memory allocation under the Neuron shared node policy** To ensure efficient resource use and stable job execution on the Neuron system, memory allocation is automatically adjusted as follows: ``` memory-per-node = ntasks-per-node * cpus-per-task * (95% of available memory per node / total number of cores per node) ``` ※ When using the '--exclusive option, 95% of the available memory on a single node is allocated to the job, allowing exclusive use of the node. However, the wait time may increase until a node is available for exclusive use. **Set the number of CPU cores allocated per GPU under the Neuron shared node policy** To ensure stable execution of GPU applications, the number of CPU cores per node is proportionally allocated to the GPUs as follows: Memory allocation is also automatically set according to the Neuron shared node policy for memory allocation. ``` cpus-per-gpu = total number of cores in the node / total number of GPUs in the node * requested number of GPUs (--gres=gpu:x) ``` ※ If additional memory is required, you can request more resources than the default cpus-per-gpu allocation to secure the necessary memory allocation. * **CPU Serial Program** ```bash #!/bin/sh #SBATCH -J Serial_cpu_job #SBATCH -p skl #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH -o %x_%j.out #SBATCH -e %x_%j.err #SBATCH --time=01:00:00 #SBATCH --comment="field=xxx;appl=yyy" #Refer to the Table of SBATCH option values for field and application export OMP_NUM_THREADS=1 module purge module load intel/19.1.2 srun ./test.exe exit 0 ``` ※ Example of occupying 1 node and running a sequential job * **CPU OpenMP Program** ```bash #!/bin/sh #SBATCH -J OpenMP_cpu_job #SBATCH -p skl #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=10 #SBATCH -o %x_%j.out #SBATCH -e %x_%j.err #SBATCH --time=01:00:00 #SBATCH --comment="field=xxx;appl=yyy" #Refer to the Table of SBATCH option values for field and application export OMP_NUM_THREADS=10 module purge module load intel/19.1.2 mpirun ./test_omp.exe exit 0 ``` ※ Example of occupying 1 node and using 10 threads per node * **CPU MPI Program** ```bash #!/bin/sh #SBATCH -J MPI_cpu_job #SBATCH -p skl #SBATCH --nodes=2 #SBATCH --ntasks-per-node=4 #SBATCH -o %x_%j.out #SBATCH -e %x_%j.err #SBATCH --time=01:00:00 #SBATCH --comment="field=xxx;appl=yyy" #Refer to the Table of SBATCH option values for field and application module purge module load intel/19.1.2 mpi/impi-19.1.2 mpirun ./test_mpi.exe ``` ※ Example of occupying 2 nodes and using 4 processes per node (total of 8 MPI processes) * **CPU Hybrid (OpenMP+MPI) Program** ```bash #!/bin/sh #SBATCH -J hybrid_cpu_job #SBATCH -p skl #SBATCH --nodes=1 #SBATCH --ntasks-per-node=2 #SBATCH --cpus-per-task=10 #SBATCH -o %x_%j.out #SBATCH -e %x_%j.err #SBATCH --time=01:00:00 #SBATCH --comment="field=xxx;appl=yyy" #Refer to the Table of SBATCH option values for field and application module purge module load intel/19.1.2 mpi/impi-19.1.2 export OMP_NUM_THREADS=10 mpirun ./test_mpi.exe ``` ※ Example of occupying 1 node and using 2 processes per node, with 10 threads per process (total of 2 MPI processes and 20 OpenMP threads) * **GPU Serial Program** ```bash #!/bin/sh #SBATCH -J Serial_gpu_job #SBATCH -p cas_v100_4 #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH -o %x_%j.out #SBATCH -e %x_%j.err #SBATCH --time=01:00:00 #SBATCH --gres=gpu:1 # using 2 gpus per node #SBATCH --comment="field=xxx;appl=yyy" #Refer to the Table of SBATCH option values for field and application export OMP_NUM_THREADS=1 module purge module load intel/19.1.2 cuda/11.4 srun ./test.exe exit 0 ``` ※ Example of occupying 1 node and running a sequential job * **GPU OpenMP Program** ```bash #!/bin/sh #SBATCH -J openmp_gpu_job #SBATCH -p cas_v100_4 #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=10 #SBATCH -o %x_%j.out #SBATCH -e %x_%j.err #SBATCH --time=01:00:00 #SBATCH --gres=gpu:2 # using 2 gpus per node #SBATCH --comment="field=xxx;appl=yyy" #Refer to the Table of SBATCH option values for field and application export OMP_NUM_THREADS=10 module purge module load intel/19.1.2 cuda/11.4 srun ./test_omp.exe exit 0 ``` ※ Example of occupying 1 node, using 10 threads and 2 GPUs per node * **GPU MPI Program** ```bash #!/bin/sh #SBATCH -J mpi_gpu_job #SBATCH -p cas_v100_4 #SBATCH --nodes=2 #SBATCH --ntasks-per-node=4 #SBATCH -o %x_%j.out #SBATCH -e %x_%j.err #SBATCH --time=01:00:00 #SBATCH --gres=gpu:2 # using 2 gpus per node #SBATCH --comment="field=xxx;appl=yyy" #Refer to the Table of SBATCH option values for field and application module purge module load intel/19.1.2 cuda/11.4 cudampi/mvapich2-2.3.6 srun ./test_mpi.exe ``` ※ Example of occupying 2 nodes, using 4 processes and 2 GPUs per node (total of 8 MPI processes) * **CPU MPI Program - Execution example of utilizing all CPUs on 1 node** ``` #!/bin/sh #SBATCH -J mpi_gpu_job #SBATCH -p cas_v100_4 #SBATCH --nodes=1 #SBATCH --ntasks-per-node=40 #SBATCH -o %x_%j.out #SBATCH -e %x_%j.err #SBATCH --time=01:00:00 #SBATCH --gres=gpu:2 #SBATCH --comment="field=xxx;appl=yyy" #Refer to the Table of SBATCH option values for field and application module purge module load intel/19.1.2 cuda/11.4 cudampi/mvapich2-2.3.6 srun ./test_mpi.exe ``` ※ Example of occupying all cores and using 2 GPUs on 1 cas\_v100\_4 node * **GPU MPI Program - Execution example of utilizing only half of the CPUs on 1 node** ``` #!/bin/sh #SBATCH -J mpi_gpu_job #SBATCH -p cas_v100nv_8 #SBATCH --nodes=1 #SBATCH --ntasks-per-node=16 #SBATCH -o %x_%j.out #SBATCH -e %x_%j.err #SBATCH --time=01:00:00 #SBATCH --gres=gpu:4 #SBATCH --comment="field=xxx;appl=yyy" #Refer to the Table of SBATCH option values for field and application module purge module load intel/19.1.2 cuda/11.4 cudampi/mvapich2-2.3.6 srun ./test_mpi.exe ``` ※ Example of occupying half the cores and using 4 GPUs on 1 cas\_v100nv\_8 node ※ The total number of cores per partition is determined when executing jobs via the scheduler (SLURM) > A. Refer to the queue configuration > Total number of CPU cores * **Execution example of a program that requires a large memory allocation** ``` #!/bin/sh #SBATCH -J mem_alloc_job #SBATCH -p cas_v100_4 #SBATCH --nodes=1 #SBATCH --ntasks-per-node=40 #SBATCH -o %x_%j.out #SBATCH -e %x_%j.err #SBATCH --time=01:00:00 #SBATCH --comment="field=xxx;appl=yyy" #Refer to the Table of SBATCH option values for field and application module purge module load intel/19.1.2 mpi/impi-19.1.2 mpirun -n 2 ./test.exe exit 0 ``` ※ When a program uses few cores but requires a large amount of memory, an example of adjusting memory allocation by setting the number of processes per node to run the program efficiently ※ The '--mem' (memory allocation per node) option is not available. When you enter the number of processes per node (ntasks-per-node) and the number of CPU cores per process (cpus-per-task), the memory allocation is automatically calculated based on the following formula:\ (memory-per-node = ntasks-per-node \* cpus-per-task \*\ (95% of the available memory on a single node / total number of cores on a single node) ※ When using the '--exclusive option, 95% of the available memory on a single node is allocated to the job, allowing exclusive use of the node. However, the wait time may increase until a node is available for exclusive use. * **An example of using GPU singularity** ```bash 1) Single node #!/bin/sh #SBATCH -J singularity #SBATCH --time=1:00:00 #SBATCH -p ivy_v100_2 #SBATCH --comment="field=cv;appl=tensorflow" #Refer to the Table of SBATCH option name per application #SBATCH -N 1 # number of nodes #SBATCH -n 2 #SBATCH –o %x_%j.out #SBATCH -e %x_%j.err #SBATCH --gres=gpu:2 module load singularity/3.6.4 singularity run —nv /apps/applications/singularity_images/ngc/tensorflow:20.09-tf1-py3.sif python test.py 2-1) Multi-node (using horovod) #!/bin/sh #SBATCH -J singularity #SBATCH --time=1:00:00 #SBATCH -p ivy_v100_2 #SBATCH --comment="field=cv;appl=tensorflow" #Refer to the Table of SBATCH option name per application #SBATCH -N 2 # number of nodes #SBATCH –n 4 #SBATCH –o %x_%j.out #SBATCH -e %x_%j.err #SBATCH --gres=gpu:2 export Base=/scratch/ID/work_dir module load singularity/3.6.4 gcc/4.8.5 mpi/openmpi-3.1.5 srun singularity run —nv /apps/applications/singularity_images/ngc/tensorflow:20.09-tf1-py3.sif \ python $Base/horovod/examples/keras/keras_imagenet_resnet50.py 2-2) Multi-node (using horovod): When the module is loaded, the specified singularity container is automatically run when the user program is executed. #!/bin/sh #SBATCH -J singularity #SBATCH --time=1:00:00 #SBATCH -p ivy_v100_2 #SBATCH --comment="field=cv;appl=tensorflow" #Refer to the Table of SBATCH option name per application #SBATCH -N 2 # number of nodes #SBATCH –n 4 #SBATCH –o %x_%j.out #SBATCH -e %x_%j.err #SBATCH --gres=gpu:2 export Base=/scratch/ID/work_dir module load singularity/3.6.4 ngc/tensorflow:20.09-tf1-py3 # tensorflow:20.09-tf1-py3.sif After the container is run automatically, the resnet50 model is trained in parallel on multiple nodes. mpirun_wrapper python $Base/horovod/examples/keras/keras_imagenet_resnet50.py ``` ※ Container images that support deep learning frameworks, such as TensorFlow, Caffe, and PyTorch, can be accessed in the /apps/applications/singularity\_images and /apps/applications/singularity\_images/ngc directories. ※ Refer to "\[Appendix 3] Method for Using Singularity Container Images - Method for running the user program from the module-based NGC container" for the list of deep learning and HPC application modules that support the automatic execution of the singularity container. * \[Example] * You can use the container images that support deep learning frameworks, such as TensorFlow, Caffe, and PyTorch, by copying them from the /apps/applications/singularity\_images directory to the user work directory. * Use the following example if the image file is “/home01/userID/tensorflow-1.13.0-py3.simg”. ``` singularity exec /home01/userID/tensorflow-1.13.0-py3.simg python test.py ``` ### **3. Interactive job submission** * Resource allocation * Use the GPU 2-node configuration (each with 2 cores and 2 GPUs) in the cas\_v100\_4 partition for interactive purposes. ```shell-session $ salloc --partition=ivy_v100_2 -N 2 -n 4 --tasks-per-node=2 --gres=gpu:2 --comment={SBATCH option name} # Refer to the Table of SBATCH option name per application ``` ※ Refer to the SBATCH option labels for each application to find the appropriate {SBATCH option name} ※ The walltime for interactive jobs is fixed at 8 hours * Job execution ```shell-session $ srun ./(executable file) (run option) ``` * Exit from the connected node, or cancel the resource allocation. ```shell-session $ exit ``` * Delete a job using a command. ```shell-session $ scancel [Job_ID] ``` ※ Job ID can be checked using the squeue command. ### **4. Job monitoring** Check the partition status Use the sinfo command to check the status ```shell-session $ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST jupyter up 2-00:00:00 6 mix jupyter[01-05,07] cas_v100nv_8 up 2-00:00:00 4 idle gpu[01-04] cas_v100nv_4 up 2-00:00:00 2 alloc gpu[06-07] cas_v100nv_4 up 2-00:00:00 2 idle gpu[08-09] cas_v100_4 up 2-00:00:00 1 mix gpu10 cas_v100_4 up 2-00:00:00 10 idle gpu[11-20] cas_v100_2 up 2-00:00:00 1 mix gpu25 ``` ※ The node configuration may be adjusted during system operation according to the system load. * PARTITION : the name of the partition set in the current SLURM * AVAIL : partition status (up or down) * TIMELIMIT : wall clock time * NODES : the number of nodes * STATE : node status (alloc-resource is being used/Idle-resource is available) * NODELIST : node list * **Detailed information per node** Adding the "-Nel" option to the sinfo command provides detailed information. ```shell-session $ sinfo -Nel Fri Mar 18 10:52:13 2022 NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON gpu01 1 cas_v100nv_8 idle 32 2:16:1 384000 0 1 TeslaV10 none gpu02 1 cas_v100nv_8 idle 32 2:16:1 384000 0 1 TeslaV10 none gpu03 1 cas_v100nv_8 idle 32 2:16:1 384000 0 1 TeslaV10 none gpu04 1 cas_v100nv_8 idle 32 2:16:1 384000 0 1 TeslaV10 none - - Omitted Below - - ``` *** * **Check job status** Use the squeue command to view the job list and status ```shell-session $ squeue JOBID PARTITION NAME USER ST TIME NODES. NODELIST(REASON) 760 cas_v100_4 gpu_burn userid R 0:00 10 gpu10 761 cas_v100_4 gpu_burn userid R 0:00 10 gpu11 762 cas_v100_4 gpu_burn userid R 0:00 10 gpu12 ``` * **Check job status and node status through graphical displays** ```shell-session $ smap ``` * **View detailed information on submitted jobs** ```shell-session $ scontrol show job [Job ID] ``` ## C. Controlling Jobs Delete a job (cancel) Use the scancel command to delete a job by entering “scancel \[Job\_ID]”. The Job\_ID can be found using the squeue command. ```shell-session $ scancel 761 ``` ## D. Compile, Debugging, Job submission Loacation Debugging nodes, which can be directly accessed via SSH from the login node, are available. Compilation, debugging, and job submission to all partitions are possible from the login/debugging nodes. The CPU time limit for debugging nodes is 120 minutes. If needed, you can use the SLURM Interactive Job feature to compile and debug in any partition. {% hint style="info" %} Last updated on August 12, 2026. {% endhint %}