[SLURM] Checking consumed resources on SLURM

[Phil Kauffman via Slurm]

Take a look at 'man squeue' and look at the -O option. It has a ton of useful things you can check.

Here are a couple of examples that show you how to check the consumed generic resources (GRES) on any machine in the cluster:


Here we can see that 'pramodkm' is currently consuming 3x GPUs and 'ady' is consuming only 1x.

kauffman at hester2:~$ squeue -O username,nodelist,gres
USER                NODELIST            GRES                
ady                 gpu3                gpu:1               
pramodkm            gpu3                gpu:3               
alexliu0809         slurm4              (null)              
alexliu0809         slurm4              (null)              
alexliu0809         slurm3              (null)              
alexliu0809         slurm3              (null)              
alexliu0809         slurm3              (null)


kauffman at hester2:~$ squeue -O jobid,partition,name,username,nodelist,gres,state
JOBID               PARTITION           NAME                USER                NODELIST            GRES                STATE               
78237               titan               bash                ady                 gpu3                gpu:1               RUNNING             
78174               titan               bash                pramodkm            gpu3                gpu:3               RUNNING             
78322               general             xato-m-10           alexliu0809         slurm4              (null)              RUNNING             
78321               general             xato-m-9            alexliu0809         slurm4              (null)              RUNNING             
78320               general             xato-m-8            alexliu0809         slurm3              (null)              RUNNING             
78319               general             xato-m-7            alexliu0809         slurm3              (null)              RUNNING             
78318               general             xato-m-6            alexliu0809         slurm3              (null)              RUNNING


-- 
Phil Kauffman
Systems Administrator
Dept. of Computer Science
University of Chicago
kauffman at cs.uchicago.edu
773-702-3913