[SLURM] [AI cluster] Fairshare and Backfill scheduler changes

[Phil Kauffman]

In order to preserve fairness across the cluster I have implemented two 
changes:

1. The backfill scheduler will now only consider 5 jobs per user to 
backfill with each cycle (one cycle every 15 seconds).

This should prevent any one user from filling a queue with 100s of jobs 
and not allowing anyone one to run jobs until they are finished.


2. Fairshare values are computed but don't seem to useful as usage grows

My current hypothesis:

Adding in all CS ldap users (10k+) into Slurm skews the fairshare 
calculation such that when you get to any normal usage your fairshare 
value goes down to zero. Which effectively means fairshare for users 
that use the cluster is not being used.

- By default every user gets one RawShare


1 share per user: list of some active users


root at ai-mgmt2:/usr/local/slurm-tools# sshare 
--accounts=user1,user2,user3,user4,user5,user6,user7,user8,user9,user10,user11 
--noheader --format=account,RawShares,FairShare --parsable2
user1|1|0.561084
user2|1|0.885498
user3|1|0.000000
user4|1|0.000000
user5|1|0.713917
user6|1|0.999548
user7|1|0.000000
user8|1|0.000000
user9|1|0.000000
user10|1|0.000000
user11|1|0.000000


Notice that the FairShare value is useless among the active users. The 
more you use the cluster the more likely it is for your FairShare 
calculation to be at 0.000000.

Slurm documentation says that you should be able to add more zeros to 
the fairshare calculation by adding them to various PriorityWeight* 
variables, however I haven't had any luck with this method.


What happens if we increase RawShares for every user who has ever 
submitted a job? Let's assume that anyone who has submitted a job at any 
time is a cluster user and give them more RawShares. For this example 
we'll use 100.


root at ai-mgmt2:/usr/local/slurm-tools# 
./set-fairshare-for-users-who-have-submitted-one-job.py

View calculated FairShare values:

root at ai-mgmt2:/usr/local/slurm-tools# sshare 
--accounts=user1,user2,user3,user4,user5,user6,user7,user8,user9,user10,user11 
--noheader --format=account,RawShares,FairShare --parsable2
user1|100|0.993638
user2|100|0.998658
user3|100|0.000548
user4|100|0.000000
user5|100|0.996285
user6|100|0.999995
user7|100|0.236102
user8|100|0.000544
user9|100|0.002677
user10|100|0.073332
user11|100|0.093049
</code>

This looks much more useful.


Assuming my hypothesis is correct; I think we have two options:

1. Make all users request a Slurm account.

   + All accounts in the SlurmDB will be an actual user of the cluster.
   + It decreases the time to sync users into the SlumDB.
   - Techstaff needs to write some web form for people to "register" to 
use the cluster.
   - Increases the barrier to entry to using the cluster.

2. Assume any user who has ever submitted a job is a Slurm user and set 
a default RawFairShare value for them. Scan periodically for users who 
have submitted jobs and ensure their RawShare value is set to a default.

   + no need to rebuild database (no downtime)
   + techstaff doesn't have to build a registration site
   + user doesn't have to "register"
   - somewhat unknown consequences of setting RawShares.
   - First job for any user may take some time to be submitted. If 
that's the case we may need a "registration" page or email techstaff to 
set RawShares for that user.

The RawShare of 100 is mutable, and will be the same for all "active" 
users. I see that setting this value to 1000 provides more resolution 
and encompasses the most active user.

I think option #2 is best for now and have deployed this solution this 
morning.



3. The above has been deployed to the AI Cluster only, but expect that I 
will push it to the Peanut cluster this week.


Feedback is welcome as always.


Cheers,

Phil



* users anonymized in above output (userN).


-- 
Phil Kauffman
Systems Admin
Dept. of Computer Science
University of Chicago
kauffman at cs.uchicago.edu
W: 773-702-3913