Multi-Category Security (MCS) Guide
The MCS Plugin is meant to extend the current Slurm functionality related to job nodes exclusivity and job/node information display privacy.
Slurm OverSubscribe option controls the ability of a partition to execute more than one job at a time on each resource, no matter "what type" of job. Slurm job submission clients can also use the --exclusive and --oversubscribe parameters to request how the job can be shared. In Slurm 15.08, ExclusiveUser slurm.conf parameter and the --exclusive=user client parameter value were introduced, extending the original exclusivity functionality. With this parameter enabled, the "type of job" now matters when considering exclusivity, so that now jobs can share resources based on job users, meaning that only jobs whose user is the same can share resources. This indeed added a new dimension to how Slurm manages exclusivity. With the introduction of the MCS Plugin, Slurm can now be configured to associate a MCS_label to jobs and optionally ensure that nodes can only be shared among jobs having the same label. This adds even more degrees of freedom to how Slurm manages exclusivity, providing end users with way much more flexibility on this area.
Slurm also has the PrivateData slurm.conf parameter, which is used to control what type of information is hidden from regular users. Similarly to the exclusivity property, the MCS Plugin also extends the privacy one by filtering jobs and/or nodes information based on the users access to their MCS_label. This means that privacy is now less restrictive, and information is not just hidden or not to regular users, but now it is filtered depending on these configurable/requestable labels in coordination with the PrivateData option.
Two parameters are currently available to configure MCS: MCSPlugin and MCSParameters.
- MCSPlugin Specifies which plugin should be used. Plugins are mutually exclusive, and the type of label to be associated depends on the loaded plugin.
- mcs/none is the default and disables MCS labels and functionality.
- mcs/account MCS labels can only take a value equal to the job's --account. NOTE: this option requires enabling of accounting.
- mcs/group MCS labels can only take a value equal to the job's user group.
- mcs/user MCS labels can only take a value equal to the username of the job's --uid.
MCS_labels on jobs can be displayed through 'squeue' with the format option mcslabel or through 'scontrol show job'. Nodes also acquire a MCS_label, which is inherited from the allocated job's MCS_label. The nodes label can be displayed with 'scontrol show nodes'. The 'sview' command can also be used to see these MCS_labels.
Users may either request a particular category label for a job (through the --mcs-label option), or use the default value generated by the specific MCS plugin implementation. So labels can be configured to be enforced or set on demand, and the specific MCS Plugin is responsible for checking the validity of these labels. When enforced, the MCS Plugin implementation will always associate a MCS label to a submited job, either the default value or the one requested by the user (if it's considered correct).
The selection (exclusivity) of nodes can be filtered on MCS labels either on demand (ondemand) or always (select) or never (noselect). User can force the filter with --exclusive=mcs option (except if noselect mode is enabled).
Label enforcement, node selection filtering policy, private data based on labels and a list of user groups allowed to be mapped to MCS labels can be configured through MCSParameters option.
- MCSParameters Specifies the options to pass to the specific MCS
Plugin implementation. Options should satisfy the following expression:
"[ondemand|enforced][,noselect|select|ondemandselect][,privatedata]:[mcs_plugin_parameters]". The defaults are "ondemand,ondemandselect" and no privatedata.
- ondemand|enforced set MCS label on jobs either on demand (using --mcs-label option) or always
- noselect|select|ondemandselect select nodes with filter on MCS label: never, always or on demand (using --exclusive=mcs option)
- privatedata accordingly with PrivateData specific option:
if MCS' privatedata and PrivateData's privatedata=jobs, jobs information is
filtered based on their MCS labels
if MCS' privatedata and PrivateData's privatedata=nodes, nodes informations is
filtered based on their MCS labels
- if MCS' privatedata and PrivateData's privatedata=jobs, jobs information is filtered based on their MCS labels
- mcs_plugin_parameters Only mcs/group is currently supporting this option. It can be used to specify the list of user groups (separated by the '|' character) that are allowed to be mapped to MCS labels by the mcs/group plugin.
Label enforcement: ondemand
(MCS_label set only if requested.)
Label enforcement: enforced
(MCS_label is mandatory.)
|noselect||No filter on nodes even if --exclusive=mcs requested.||No filter on nodes even if --exclusive=mcs requested.|
|select||Filter on nodes only if job MCS_label is set.||Always filter on nodes.|
|ondemandselect||Filter on nodes only if --exclusive=mcs.||Filter on nodes only if --exclusive=mcs.|
View MCS parameters
Configured MCS parameters can be viewed using the
scontrol show config |grep MCS MCSPlugin = mcs/group MCSParameters = ondemand,noselect:groupA|groupB|groupC
Put a mcs_label in job using salloc, sbatch or srun command
srun -n10 -t 1000 --mcs-label=groupB ./job &
Put a mcs_label in job with exclusivity
srun -n10 -t 1000 --mcs-label=groupB --exclusive=mcs ./job &
With mcs/account plugin : put a different account/mcs-label in job
srun -n10 -t 1000 -A another_account_than_default ./job &
View MCS parameters in usage
squeue -O jobid,username,mcslabel JOBID USER MCSLABEL 2 foo groupA 3 bar groupB
scontrol show nodes NodeName=node0001 Arch=x86_64 CoresPerSocket=4 CPUAlloc=0 CPUErr=0 CPUTot=8 CPULoad=0.01 Features=(null) Gres=(null) NodeAddr=noder0001 NodeHostName=node0001 Version=15.08 OS=Linux RealMemory=23780 AllocMem=0 Sockets=2 Boards=1 State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=40 Owner=N/A MCS_label=groupA BootTime=2015-08-05T10:14:41 SlurmdStartTime=2015-09-07T13:42:28 CapWatts=n/a CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
MCS Plugin development
Please see the MCS Plugin Programmer Guide for details on developing additional MCS plugins.
Last modified 27 April 2017