Prolog and Epilog Guide

Slurm supports a multitude of prolog and epilog programs. Note that for security reasons, these programs do not have a search path set. Either specify fully qualified path names in the program or set the "PATH" environment variable. The first table below identifies what prologs and epilogs are available for job allocations, when and where they run.

Parameter

Location

Invoked by

User

When executed

Prolog (from slurm.conf)

Compute or front end node

slurmd daemon

SlurmdUser (normally user root)

First job or job step initiation on that node (by default); PrologFlags=Alloc will force the script to be executed at job allocation

PrologSlurmctld (from slurm.conf)

Head node (where slurmctld daemon runs)

slurmctld daemon

SlurmctldUser

At job allocation

Epilog (from slurm.conf)

Compute or front end node

slurmd daemon

SlurmdUser (normally user root)

At job termination

EpilogSlurmctld (from slurm.conf)

Head node (where slurmctld daemon runs)

slurmctld daemon

SlurmctldUser

At job termination


This second table below identifies what prologs and epilogs are available for job step allocations, when and where they run.

Parameter

Location

Invoked by

User

When executed

SrunProlog (from slurm.conf) or srun --prolog

srun invocation node

srun command

User invoking srun command

Prior to launching job step

TaskProlog (from slurm.conf)

Compute node

slurmstepd daemon

User invoking srun command

Prior to launching job step

srun --task-prolog

Compute node

slurmstepd daemon

User invoking srun command

Prior to launching job step

TaskEpilog (from slurm.conf)

Compute node

slurmstepd daemon

User invoking srun command

Completion job step

srun --task-epilog

Compute node

slurmstepd daemon

User invoking srun command

Completion job step

SrunEpilog (from slurm.conf) or srun --epilog

srun invocation node

srun command

User invoking srun command

Completion job step

By default the Prolog script is only run on any individual node when it first sees a job step from a new allocation; it does not run the Prolog immediately when an allocation is granted. If no job steps from an allocation are run on a node, it will never run the Prolog for that allocation. This Prolog behaviour can be changed by the PrologFlags parameter. The Epilog, on the other hand, always runs on every node of an allocation when the allocation is released.

The task prolog is executed with the same environment as the user tasks to be initiated. The standard output of that program is read and processed as follows:
export name=value sets an environment variable for the user task
unset name clears an environment variable from the user task
print ... writes to the task's standard output.

Unless otherwise specified, these environment variables are available to all of the programs.

  • BASIL_RESERVATION_ID Basil reservation ID. Available on Cray systems with ALPS only.
  • MPIRUN_PARTITION BlueGene partition name. Available on BlueGene systems only.
  • SLURM_ARRAY_JOB_ID If this job is part of a job array, this will be set to the job ID. Otherwise it will not be set. To reference this specific task of a job array, combine SLURM_ARRAY_JOB_ID with SLURM_ARRAY_TASK_ID (e.g. "scontrol update ${SLURM_ARRAY_JOB_ID}_{$SLURM_ARRAY_TASK_ID} ..."); Available in PrologSlurmctld and EpilogSlurmctld only.
  • SLURM_ARRAY_TASK_ID If this job is part of a job array, this will be set to the task ID. Otherwise it will not be set. To reference this specific task of a job array, combine SLURM_ARRAY_JOB_ID with SLURM_ARRAY_TASK_ID (e.g. "scontrol update ${SLURM_ARRAY_JOB_ID}_{$SLURM_ARRAY_TASK_ID} ..."); Available in PrologSlurmctld and EpilogSlurmctld only.
  • SLURM_ARRAY_TASK_MAX If this job is part of a job array, this will be set to the maximum task ID. Otherwise it will not be set. Available in PrologSlurmctld and EpilogSlurmctld only.
  • SLURM_ARRAY_TASK_MIN If this job is part of a job array, this will be set to the minimum task ID. Otherwise it will not be set. Available in PrologSlurmctld and EpilogSlurmctld only.
  • SLURM_ARRAY_TASK_STEP If this job is part of a job array, this will be set to the step size of task IDs. Otherwise it will not be set. Available in PrologSlurmctld and EpilogSlurmctld only.
  • SLURM_CLUSTER_NAME Name of the cluster executing the job.
  • SLURM_JOB_GPUS GPU IDs allocated to the job (if any). Available in the Prolog only.
  • SLURM_JOB_ACCOUNT Account name used for the job. Available in PrologSlurmctld and EpilogSlurmctld only.
  • SLURM_JOB_CONSTRAINTS Features required to run the job. Available in Prolog, PrologSlurmctld and EpilogSlurmctld only.
  • SLURM_JOB_DERIVED_EC The highest exit code of all of the job steps. Available in EpilogSlurmctld only.
  • SLURM_JOB_EXIT_CODE The exit code of the job script (or salloc). The value is the status as returned by the wait() system call (See wait(2)) Available in EpilogSlurmctld only.
  • SLURM_JOB_EXIT_CODE2 The exit code of the job script (or salloc). The value has the format :. The first number is the exit code, typically as set by the exit() function. The second number of the signal that caused the process to terminante if it was terminated by a signal. Available in EpilogSlurmctld only.
  • SLURM_JOB_GID Group ID of the job's owner. Available in PrologSlurmctld and EpilogSlurmctld only.
  • SLURM_JOB_GROUP Group name of the job's owner. Available in PrologSlurmctld and EpilogSlurmctld only.
  • SLURM_JOB_ID Job ID. CAUTION: If this job is the first task of a job array, then Slurm commands using this job ID will refer to the entire job array rather than this specific task of the job array.
  • SLURM_JOB_NAME Name of the job. Available in PrologSlurmctld and EpilogSlurmctld only.
  • SLURM_JOB_NODELIST Nodes assigned to job. A Slurm hostlist expression. "scontrol show hostnames" can be used to convert this to a list of individual host names. Available in PrologSlurmctld and EpilogSlurmctld only.
  • SLURM_JOB_PARTITION Partition that job runs in. Available in Prolog, PrologSlurmctld and EpilogSlurmctld only.
  • SLURM_JOB_UID User ID of the job's owner.
  • SLURM_JOB_USER User name of the job's owner.

Plugins functions are may also be useful to execute logic at various well defined points.

SPANK is another mechanism that may be useful to invoke logic in the user commands, slurmd daemon, and slurmstepd daemon.

Failure Handling

If the Epilog fails (returns a non-zero exit code), this will result in the node being set to a DOWN state. If the EpilogSlurmctld fails (returns a non-zero exit code), this will only be logged. If the Prolog fails (returns a non-zero exit code), this will result in the node being set to a DRAIN state and the job requeued in a held state (unless nohold_on_prolog_fail is configured in SchedulerParameters). If the PrologSlurmctld fails (returns a non-zero exit code), this will result in the job requeued to executed on another node if possible. Only batch jobs can be requeued. Interactive jobs (salloc and srun) will be cancelled if the PrologSlurmctld fails.


Based upon work by Jason Sollom, Cray Inc. and used by permission.

Last modified 9 January 2017