Metrics Guide

Contents

Metrics Overview

Slurm 25.11 introduced a comprehensive system for collecting and exposing metrics related to cluster resources, job states, and scheduler performance. The metrics system exposes real-time data about various Slurm entities through HTTP endpoints provided by the slurmctld daemon.

The metrics feature enables integration with popular monitoring systems like Prometheus, Grafana, and other observability tools.

Configuration

Prerequisites

The metrics feature requires specific configuration in slurm.conf:

  • PrivateData must not be set: The metrics feature is disabled if any PrivateData parameter is configured in slurm.conf. This is a security requirement to prevent exposure of sensitive information through metrics.
  • MetricsType parameter: Set the MetricsType parameter to specify which metrics plugin to use. Currently, only the OpenMetrics plugin is supported: 
    MetricsType=metrics/openmetrics

Plugin Loading

The metrics plugin is automatically loaded by slurmctld when the MetricsType parameter is configured.

HTTP Endpoints

Slurm exposes metrics through HTTP GET endpoints on the slurmctld daemon's listening port (default 6817). The following endpoints are available:

  • GET /metrics - Print available metric endpoints
  • GET /metrics/jobs - Job-related metrics including counts by state, resource allocation, and job statistics (examples)
  • GET /metrics/jobs-users-accts - User- and account-specific job metrics (examples)
  • GET /metrics/nodes - Node-related metrics including resource counts, states, and utilization (examples)
  • GET /metrics/partitions - Partition-related metrics including job counts per partition and resource allocation (examples)
  • GET /metrics/scheduler - Scheduler performance metrics including cycle times, backfill statistics, and queue lengths (examples)

All endpoints return data in UTF-8 text format making them compatible with Prometheus and other monitoring systems.

OpenMetrics Plugin

The OpenMetrics plugin implements the OpenMetrics 1.0 specification, ensuring compatibility with Prometheus and other monitoring systems that consume metrics in this format.

Metric Format

Each metric follows the OpenMetrics format with the following components:

  • Metric name: A descriptive name prefixed with "slurm_"
  • Metric type: Only "gauge" metrics are exposed
  • Metric value: The actual numeric value
  • Labels: Optional key-value pairs for additional context
  • Help text: Human-readable description of the metric

Metric Categories Provided by Slurm

Each endpoint provides a set of metrics related to the same general category. Numerous metrics are provided so they are not all documented on this page. A few examples are provided for each category in the following subsections.

Job Metrics

Job metrics provide information about job states, resource allocation, and job counts. Examples include:

  • slurm_jobs - Total number of jobs
  • slurm_jobs_running - Number of running jobs
  • slurm_jobs_pending - Number of pending jobs (see note)
  • slurm_jobs_cpus_alloc - Total CPUs allocated to jobs
  • slurm_jobs_memory_alloc - Total memory allocated to jobs

NOTE: In Slurm, pending jobs include both jobs waiting for resources and held jobs. Held jobs will not be scheduled until the hold is released.

User- and Account-Specific Job Metrics

Job metrics for user and accounts provide a count of jobs in each state for each active user and account in the system. It stores each entity under a key-value pair. Remember that every unique key-value pair represents a new time series, which can dramatically increase the amount of data stored. Examples include:

  • slurm_user_jobs_pending{username="john"} - Pending jobs for user "john"
  • slurm_account_jobs_pending{account="smith"} - Pending jobs for account "smith"

Node Metrics

Node metrics track resource availability, node states, and utilization. Examples include:

  • slurm_nodes - Total number of nodes
  • slurm_nodes_idle - Number of idle nodes
  • slurm_nodes_alloc - Number of allocated nodes
  • slurm_node_cpus{node="nodename"} - CPUs on the specified node
  • slurm_node_memory_bytes{node="nodename"} - Memory on the specified node (bytes)

Partition Metrics

Partition metrics show job distribution and resource allocation across partitions. Examples include:

  • slurm_partitions - Total number of partitions
  • slurm_partition_jobs{partition="name"} - Jobs on the specified partition
  • slurm_partition_nodes{partition="name"} - Nodes on the specified partition

The following metrics might be useful for Slinky or other systems which have an auto-scale feature. By knowing the maximum number of nodes that a job requested in a partition, the decision to extend the nodes of the partition by this number can be considered. Jobs which are held are not included in these metrics.

  • slurm_partition_jobs_max_job_nodes_nohold - Gives the maximum number of nodes requested by any job from all pending jobs that are not held in the partition.
  • slurm_partition_jobs_min_job_nodes_nohold - Gives the maximum of the minimum number of nodes requested by any job from all pending jobs that are not held in the partition.

Scheduler Metrics

Scheduler metrics provide insights into scheduling performance and behavior. Examples include:

  • slurm_sched_cycle_cnt - Scheduling cycle count
  • slurm_sched_cycle_last - Last scheduling cycle time
  • slurm_bf_cycle_cnt - Backfill cycle count
  • slurm_bf_active - Whether the backfill scheduler is currently running

Security Considerations

The metrics system has several important security implications:

  • No authentication: There is no built-in authentication mechanism for metrics endpoints. Anyone with network access to the slurmctld port can query metrics.
  • PrivateData dependency: The metrics feature is automatically disabled when any PrivateData parameter is set in slurm.conf to prevent exposure of sensitive information.
  • Network access: Metrics are exposed through the slurmctld network interface. Consider firewall rules and network segmentation to control access.
  • Information disclosure: Metrics may reveal information about cluster utilization, job patterns, and user activity that could be considered sensitive in some environments.

Performance Impact

Metrics collection and exposition can impact slurmctld performance:

  • Lock contention: Querying metrics requires acquiring various locks within slurmctld, which can impact scheduler performance during high-frequency queries.
  • Data collection overhead: Metrics are collected in real-time from slurmctld's internal data structures, which adds computational overhead.
  • Data processing overhead in the external monitoring system: We provide endpoints for unbounded entities like users and accounts metrics. A monitoring system may treat each entity as a new time series, which can dramatically increase the amount of data stored.
  • Network I/O: Frequent metric queries generate network traffic and consume slurmctld's network I/O capacity, especially on systems with thousands of jobs, users or accounts.

To minimize performance impact:

  • Configure appropriate scrape intervals in monitoring systems (e.g., 60-120 seconds)
  • Use caching mechanisms in monitoring systems when possible
  • Monitor slurmctld performance when enabling metrics
  • Do not use unbounded metric endpoints like /metrics/jobs-users-accts to store data in your monitoring system

Usage Examples

Basic curl Examples

Query job metrics:

$ curl http://slurmctld.example.com:6817/metrics/jobs
# HELP slurm_jobs Total number of jobs
# TYPE slurm_jobs gauge
slurm_jobs 42
# HELP slurm_jobs_running Number of jobs in Running state
# TYPE slurm_jobs_running gauge
slurm_jobs_running 15
# HELP slurm_jobs_pending Number of jobs in Pending state
# TYPE slurm_jobs_pending gauge
slurm_jobs_pending 27
...

Query node metrics:

$ curl http://slurmctld.example.com:6817/metrics/nodes
# HELP slurm_nodes Total number of nodes
# TYPE slurm_nodes gauge
slurm_nodes 100
# HELP slurm_nodes_idle Number of nodes in Idle state
# TYPE slurm_nodes_idle gauge
slurm_nodes_idle 85
# HELP slurm_nodes_alloc Number of nodes in Allocated state
# TYPE slurm_nodes_alloc gauge
slurm_nodes_alloc 15
...

Prometheus Configuration

Configure Prometheus to scrape Slurm metrics by adding the following to your prometheus.yml:

scrape_configs:
  - job_name: 'slurm_jobs'
    static_configs:
      - targets: ['slurm.example.com:6817']
    metrics_path: '/metrics/jobs'

  - job_name: 'slurm_nodes'
    static_configs:
      - targets: ['slurm.example.com:6817']
    metrics_path: '/metrics/nodes'

  - job_name: 'slurm_partitions'
    static_configs:
      - targets: ['slurm.example.com:6817']
    metrics_path: '/metrics/partitions'

  - job_name: 'slurm_scheduler'
    static_configs:
      - targets: ['slurm.example.com:6817']
    metrics_path: '/metrics/scheduler'

  - job_name: 'slurm_useracct'
    static_configs:
      - targets: ['slurm.example.com:6817']
    metrics_path: '/metrics/jobs-users-accts'