It is important to know that the Remote Procedure Calls (RPCs) that are received by Slurm are coming from trusted sources. There are a few different authentication mechanisms available within Slurm to verify the legitimacy and integrity of the requests.
MUNGE can be used to create and validate credentials. It allows Slurm to authenticate the UID and GID of a request from another host that has matching users and groups. MUNGE libraries must exist when building Slurm in order for it to be able to use munge for authentication. All hosts in the cluster must have a shared cryptographic key.
- MUNGE requires that there be a shared key on the machines running
slurmctld, slurmdbd and the nodes. You can create a key file by entering your
own text or by generating random data:
dd if=/dev/random of=/etc/munge/munge.key bs=1024 count=1
- This key should be owned by the "munge" user and should not be readable
or writable by other users:
chown munge:munge /etc/munge/munge.key chmod 400 /etc/munge/munge.key
- Distribute the key file to the machines on the cluster. It needs to be on the machines running slurmctld, slurmdbd, slurmd and any submit hosts you have configured. Use the distribution method of your choice.
- The "munge" service should be running on any machines that need to use it
for authentication. It should be enabled and started on all the machines you
distributed a key to:
systemctl enable munge systemctl start munge
- Changing the authentication mechanism requires a restart of Slurm daemons. The daemons need to be stopped before updating the slurm.conf so that client commands do not use a mechanism other than what the running daemons are expecting.
- Update your slurm.conf to use MUNGE authentication:
AuthType = auth/munge CredType = cred/munge
- Start the Slurm daemons back up with the appropriate method for your cluster.
Beginning with version 23.11, Slurm has its own plugin that can create and validate credentials. It validates that the requests come from legitimate UIDs and GIDs on other hosts with matching users and groups. All hosts in the cluster must have a shared cryptographic key.
- For the authentication to happen correctly you must have a shared key on
the machine running slurmctld, slurmdbd and the nodes. You can create a key
file by entering your own text or by generating random data:
dd if=/dev/random of=/etc/slurm/slurm.key bs=1024 count=1
- This key should be owned by SlurmUser and should not be readable or
writable by other users. This example assumes the SlurmUser is 'slurm':
chown slurm:slurm /etc/slurm/slurm.key chmod 600 /etc/slurm/slurm.key
- Distribute the key file to the machines on the cluster. It needs to be on the machines running slurmctld, slurmdbd, slurmd and sackd.
- Update your slurm.conf to use the Slurm authentication type:
AuthType = auth/slurm CredType = cred/slurm
- Start the daemons back up with the appropriate method for your cluster.
Slurm's internal authentication makes use of a subsystem — the Slurm Auth and Cred Kiosk (SACK) — that is responsible for handling requests from the auth/slurm and cred/slurm plugins. This subsystem is automatically started and managed by the slurmctld, slurmdbd, and slurmd daemons on each system.
For login nodes not running one of these Slurm daemons, the sackd daemon should be run to allow the Slurm client commands to authenticate to the rest of the cluster. This daemon can also manage cached configuration files for configless environments.
Slurm can be configured to use JSON Web Tokens (JWT) for authentication purposes. This is configured with the AuthAltType parameter and is used only for client to server communication. You can read more about this authentication mechanism and how to install it here.
Last modified 11 January 2024