# Mastering Slurm: Your Gateway to High-Performance Computing Welcome to the world of High-Performance Computing (HPC)! Slurm is a powerful tool that helps manage and schedule workloads on computing clusters. This tutorial will guide you through the essentials of Slurm, enabling you to efficiently manage your computational tasks. Our goal is to help you learn workload management through hands-on practice using a simulated Slurm environment. ---interactive-terminal--- Welcome to the Slurm Tutorial Terminal! Type 'help' to see available commands. --- ---command-reference-trigger--- ## What is Slurm? Slurm (Simple Linux Utility for Resource Management) is an open-source workload manager and job scheduler designed for Linux clusters of all sizes. It is responsible for allocating resources to users, managing a queue of pending work, and scheduling jobs to run on compute nodes. Key functions of Slurm include: * **Resource Manager:** It keeps track of all available compute nodes and their resources (CPUs, memory, GPUs, etc.). * **Job Scheduler:** It decides when and where to run jobs based on priorities, policies, and available resources. * **Queue Management:** It manages a queue of jobs submitted by users, holding them until resources become available. Try it out: In the terminal above, type `sinfo` to see the simulated cluster information. ## Slurm Architecture Understanding Slurm's architecture helps in using it effectively: * **Compute Nodes (or Slurm Nodes):** These are the machines that perform the actual computations. * **Partitions:** Nodes are often grouped into partitions. Partitions can be thought of as job queues, each with specific limits (e.g., runtime, number of nodes) or features (e.g., GPU-equipped nodes). * **Jobs:** A job is a request for resource allocation. Slurm allocates resources to a job, and then the job can perform its tasks. * **Slurm Controller (slurmctld):** This is the central daemon that monitors resources and jobs. * **Slurm Daemons (slurmd):** These run on each compute node, manage tasks, monitor resources, and execute jobs. ## Key Concepts * **Nodes:** Individual computers in the cluster where your programs run. * **Partitions:** Logical groupings of nodes. You submit jobs to a specific partition. * **Jobs:** Allocations of resources (nodes, CPUs, memory) for a specified amount of time. * **Job Steps:** Individual commands or tasks within a job. Practice: Explore the cluster partitions by typing `sinfo` in the terminal. ## Essential Commands Let's dive into the most common Slurm commands. ### Information Commands * **`sinfo` - Cluster Information** `sinfo` provides information about the cluster's partitions and nodes. Instruction: In the terminal, type `sinfo`. * **`squeue` - Job Queue Status** `squeue` shows the status of jobs in the queue (pending, running, etc.). By default, in our simulation, it shows jobs for 'testuser'. Instruction: In the terminal, type `squeue`. ### Exercise 1: Exploring the Cluster 1. Use `sinfo` to see all partitions and their states. 2. Use `squeue` to see if there are any running or pending jobs for 'testuser'. ## Job Submission ### Interactive Jobs - `srun` `srun` is used to submit jobs for interactive use or to launch job steps. It's great for tasks that require real-time interaction or for quick tests. * Example: `srun hostname` (This will simulate running the `hostname` command on an allocated compute node). Instruction: In the terminal, type `srun hostname`. * Example: `srun echo "Hello Slurm!"` Instruction: In the terminal, type `srun echo "Hello Slurm!"`. ### Batch Jobs - `sbatch` `sbatch` is used to submit a job script for later execution. Job scripts are text files containing Slurm directives (lines starting with `#SBATCH`) and the commands to be executed. Here's an example of a simple batch script (e.g., `my_script.sh`): ```bash #!/bin/bash #SBATCH --job-name=myFirstJob #SBATCH --output=myFirstJob_output.txt # Simulated: output won't actually be written to a file here #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --time=00:05:00 echo "Hello from my batch job!" date echo "My Job ID is $SLURM_JOB_ID" # This variable is available in actual Slurm jobs ``` You would typically save this to a file (e.g., `my_script.sh`) and then submit it using `sbatch my_script.sh`. In our simulated environment, the script content isn't read, but the command simulates the submission process. Instruction: Try submitting a simulated batch job: `sbatch my_script.sh`. A job with this name will be created. ### Exercise 2: Your First Job 1. Submit an interactive job that prints a message: `srun echo "My interactive job is running!"` 2. Submit a batch job (the name can be anything, e.g., `example_job.sh`): `sbatch example_job.sh` 3. Check the job queue to see your jobs: `squeue`. 4. Note the `JOBID` of one of your jobs from the `squeue` output. 5. Cancel one of your jobs: `scancel ` (replace `` with the actual ID). ## Job Monitoring Slurm provides commands to monitor your jobs. * **`scontrol show job ` - Detailed Job Information** This command provides detailed information about a specific job. Instruction: 1. Submit a job: `sbatch another_job.sh` 2. Check the queue for its ID: `squeue` 3. View details: `scontrol show job ` (replace `` with the ID you found). * **`scontrol show node ` - Detailed Node Information** This command provides detailed information about a specific node. Instruction: In the terminal, type `scontrol show node node01`. You can also try `gpu01`. * **`sacct` - Job Accounting** `sacct` is used to view accounting information for jobs and job steps, including completed jobs. Instruction: After submitting and possibly canceling a few jobs, type `sacct` to see their simulated history. ### Exercise 3: Monitoring Jobs 1. Submit a batch job: `sbatch monitor_test.sh`. 2. Find its `JOBID` using `squeue`. 3. View detailed information about this job: `scontrol show job `. 4. Check the details of a compute node: `scontrol show node node02`. 5. View the accounting information for your jobs: `sacct`. ## Advanced Features (Conceptual) Slurm offers many advanced features. Here are a few concepts: * **Resource Constraints:** You can request specific resources for your jobs. * `--cpus-per-task=`: Request a specific number of CPUs for each task. * `--mem=`: Request a specific amount of memory (e.g., `--mem=4G` for 4 Gigabytes). * `--time=`: Set a time limit for your job. These options are typically used within an `sbatch` script or with `srun`. * **Job Arrays:** If you need to run the same script multiple times with slightly different parameters, job arrays are very efficient. For example, `sbatch --array=1-10 my_array_script.sh` would submit 10 instances of the script. An environment variable like `SLURM_ARRAY_TASK_ID` would be available inside the script to differentiate tasks. * **Environment Variables:** Slurm sets several environment variables within a job's environment, such as: * `SLURM_JOB_ID`: The ID of the current job. * `SLURM_SUBMIT_DIR`: The directory where `sbatch` was run. * `SLURM_NNODES`: Number of nodes allocated to the job. * `SLURM_NTASKS`: Total number of tasks for the job. These are not interactively simulated in this terminal but are fundamental concepts in real Slurm environments. ## Slurm Command Reference (Simulated) Here's a quick reference for the Slurm commands available in this simulated terminal: * **`sinfo`**: Show cluster nodes and partitions. * **`squeue [-u ]`**: Display job queue status (default user is 'testuser'). * **`sbatch `**: Submit a batch job script (simulated submission). * **`srun [args...]`**: Run an interactive job (e.g., `srun hostname`, `srun echo Hello`). * **`scancel `**: Cancel a job by its ID. * **`scontrol show job `**: Show detailed information for a specific job. * **`scontrol show node `**: Show detailed information for a specific node (e.g., `node01`, `gpu01`). * **`sacct`**: Show job accounting information for submitted jobs. * **`help`**: Shows all available commands in the terminal. * **`echo [text]`**: Prints text to the terminal. * **`clear`**: Clears the terminal history. * **`date`**: Shows the current date and time. ## Conclusion Congratulations! You've completed the basic Slurm tutorial. You've learned about Slurm's core concepts, how to submit and manage jobs, and how to monitor their progress in this simulated environment. High-Performance Computing can seem daunting, but tools like Slurm make it manageable. We encourage you to continue practicing with the commands in the terminal. The more you use them, the more comfortable you'll become with Slurm's workflow. Happy computing!