Namespace Plugins

Overview

A namespace plugin can be enabled to provide job-specific, private temporary file system space.

When enabled on the cluster, a filesystem namespace will be created for each job with a unique, private instance of /tmp and /dev/shm for the job to use. These directories can be changed with the Dirs= option in the plugin-specific configuration file. The contents of these directories will be removed at job termination. Additionally, the namespace/linux plugin can be configured to create new PID and user namespaces.

Installation

These plugins are built and installed as part of the default build, no extra installation steps are required.

Setup

Slurm must be configured to load the namespace plugin by adding PrologFlags=contain and setting NamespaceType to the desired plugin in slurm.conf. Additional configuration must be done in the plugin-specific configuration file file, which should be placed in the same directory as slurm.conf.

namespace/linux plugin

The namespace/linux plugin (added 25.11) uses the configuration file namespace.yaml. This plugin can be configured to create user and PID namespaces in addition to a temporary filesystem namespace. Namespaces can be configured for all nodes or for a subset of nodes. As an example, if all nodes will be configured the same way, you could put the following in your namespace.yaml:

defaults:
  auto_base_path: true
  base_path: "/var/nvme/storage"

Note the following important details with this plugin:

  • This plugin requires cgroup/v2 to operate correctly.
  • When using user namespaces, bpf token support (added in kernel 6.9) is required to use ConstrainDevices in cgroup.conf.

namespace/tmpfs plugin

The namespace/tmpfs plugin (formerly job_container/tmpfs) uses the configuration file job_container.conf. Namespaces can be configured for all nodes, or for a subset of nodes. As an example, if all nodes will be configured the same way, you could put the following in your job_container.conf:

AutoBasePath=true
BasePath=/var/nvme/storage

Initial Testing

An easy way to verify that the container is working is to run a job and ensure that the /tmp directory is empty (since it normally has some other files) and that "." is owned by the user that submitted the job.

tim@slurm-ctld:~$ srun ls -al /tmp
total 8
drwx------  2 tim    root 4096 Feb 10 17:14 .
drwxr-xr-x 21 root   root 4096 Nov 15 08:46 ..

While a job is running, root should be able to confirm that /$BasePath/$JobID/_tmp exists and is empty. This directory is bind mounted into the job. /$BasePath/$JobID should be owned by root, and is not intended to be accessible to the user.

Additionally, when the Linux plugin is in use, you can confirm that a PID namespace is in effect by running a job and running "ps". The only visible PIDs should be related to the job and PID 1 will be named slurmstepd: [${job_id}.namespace].

SPANK

This plugin interfaces with the SPANK api, and automatically joins the job's namespace in the following functions:

  • spank_task_init_privileged()
  • spank_task_init()

In addition to the job itself, the TaskProlog will also be executed inside the container.