How to create an HPC solver#

Introduction#

An HPC solver runs its work as a batch job on a High-Performance Computing cluster instead of inside the solver container. From your run() function you submit a job to the cluster’s scheduler (Slurm), wait for it to finish, and return the results — all through a client that QCentroid provides for you.

This is the way to reach an HPC cluster that is wired to the platform through an HPC provider: your jobs run on the cluster’s compute nodes over the Slurm REST API.

Note

This page assumes you already know the basics of building a solver. If you don’t, start with How to create a solver first.

How it works#

You don’t manage the connection to the cluster yourself. When you select the HPC provider for your solver, the platform, before your run() is called:

Opens a secure VPN to the cluster (if the provider needs one).
Logs in and prepares the Slurm REST API access.
Injects the cluster credentials and defaults as environment variables, and makes the HPC client available to your code.

So your solver only has to focus on what job to run. Submitting it, waiting for it and reading the output is a handful of calls on the client:

Your run() → submit a script → HPC client → Slurm REST over the VPN → cluster runs it on a compute node → the job’s output comes back to your run().

Before you start#

Enable the HPC provider from the Providers section of the dashboard. See How to set-up providers.
When you register your solver, select that provider and choose the HPC language (python_hpc) in the programming language dropdown. This is what tells the platform to build your solver with the HPC client (and the VPN tooling, if the cluster needs one).

Runs on the platform only

HPC solvers cannot be run locally with python app.py. The VPN, the cluster credentials and the HPC client only exist inside the QCentroid runtime. Develop the job logic locally if you want, but test the full solver by running it on the platform.

The HPC client#

The client ships with QCentroid and is already installed in your solver at runtime — you do not add it to requirements.txt. Import it like this:

from qcentroid_solver_runner.hpc import QCentroidHPCClient

It handles the whole round trip: logging in to the cluster, talking to Slurm, submitting, polling and reading the job’s output back. It reads the cluster host, credentials and defaults from the environment the provider set up, so you never hardcode them.

Typical usage#

import logging
from qcentroid_solver_runner.hpc import QCentroidHPCClient

logger = logging.getLogger("qcentroid-user-log")   # so progress shows in the dashboard

with QCentroidHPCClient(logger=logger) as client:
    client.login()
    job_id = client.run_job(script, name="my-job", time_limit=5, environment="login")
    client.wait_for_completion(job_id, poll_interval=5)
    result = client.get_job_results(job_id)

Main methods#

Method	What it does
`login()`	Connect to the cluster and prepare Slurm access. The other methods call it for you if you forget
`run_job(script, **options)`	Submit a batch script and return its `job_id`. See the options below
`wait_for_completion(job_id, *, poll_interval=10, queue_timeout=None, run_timeout=None)`	Block until the job reaches a final state and return it. Logs progress on every poll (queue reason, node, queued/run times). Both timeouts default to no limit; reaching either one cancels the job and it comes back as `CANCELLED`
`get_job_results(job_id)`	A `JobResult` with `state`, `node`, `exit_code`, `succeeded`, `stdout`, `stderr`, and `info` (the full Slurm record)
`run_and_wait(script, **options)`	Submit + wait + fetch results in a single call
`get_job_status(job_id)`	Current state (`PENDING`, `RUNNING`, `COMPLETED`, …) without blocking
`cancel_job(job_id)` / `cancel_all()`	Cancel one job, or every active job
`list_partitions()` / `get_user_associations()`	Discover which partitions exist and which account/partition/qos combinations you may submit under — useful if a submit is rejected

Writing your `qcentroid.py`#

Put the client to work inside run(). This example submits a small job that loads a cluster module and runs a Python one-liner, then returns its output:

qcentroid.py

import logging
from qcentroid_solver_runner.hpc import QCentroidHPCClient

logger = logging.getLogger("qcentroid-user-log")

# The batch script to run on the cluster. Use "#!/bin/bash -l" (a login shell) when the
# script needs `module`. Module names depend on the cluster you are targeting.
BATCH_SCRIPT = """#!/bin/bash -l
module load python/3.11

echo "Running on $(hostname)"
python3 -c 'print("Hello world from the HPC")'
"""


def run(input_data: dict, solver_params: dict, extra_arguments: dict) -> dict:
    logger.info("Submitting the HPC job...")

    with QCentroidHPCClient(logger=logger) as client:
        client.login()
        job_id = client.run_job(
            BATCH_SCRIPT,
            name="my-hpc-job",
            time_limit=5,          # minutes
            environment="login",   # so `module load` works (see below)
        )
        client.wait_for_completion(job_id, poll_interval=5)
        result = client.get_job_results(job_id)

    logger.info("Job %s finished: %s (exit code %s)", result.job_id, result.state, result.exit_code)

    return {
        "job_id": result.job_id,
        "state": result.state,
        "succeeded": result.succeeded,
        "stdout": result.stdout,
        "stderr": result.stderr,
    }

Anything you build into input_data / solver_params can be used to shape the script or the job options, exactly like a normal solver.

`run_job()` options#

Scheduling options you don’t pass fall back to the defaults the provider configured, so in practice you only set what is specific to your job.

Option	Notes
`name`	Job name
`time_limit`	Wall-clock limit in minutes
`memory_per_node`	Memory in MB
`tasks`, `cpus_per_task`, `nodes`	Size of the allocation
`partition`, `account`, `qos`	Scheduling context
`gres`	Generic resources per node, e.g. `"gpu:1"`
`gpus`	GPUs per job
`exclusive`	Request the whole node
`environment`	`"login"` propagates the cluster login environment — this is what makes `module load` work. A dict or a `["KEY=value", ...]` list is also accepted
`working_directory`	Defaults to your home directory on the cluster
`job_overrides`	Raw Slurm REST job fields, for anything not covered above

#SBATCH directives are ignored

Unlike sbatch, the Slurm REST API does not read #SBATCH lines in the script. Pass scheduling options as arguments to run_job(), not as comments in the script.

Results and assets#

get_job_results() returns a JobResult:

Field	Meaning
`state`	Final Slurm state (`COMPLETED`, `FAILED`, `CANCELLED`, …)
`succeeded`	`True` only when the job completed with exit code 0
`exit_code`	The job’s exit code
`node`	The compute node it ran on
`stdout` / `stderr`	The job’s captured output
`info`	The full raw Slurm job record (timestamps, allocated resources, …)

The solver decides what a failed job means

A cluster job ending in FAILED or CANCELLED is not automatically a solver error — your run() still returns normally, reporting the state. Raise an exception yourself only if you want the whole execution marked as failed.

As with any solver, you can drop files into an additional_output folder to publish them as downloadable assets (raw stdout, a report, plots, …). See Generate output assets.

Good to know#

Memory usually must be set. On some clusters a job with no explicit memory never gets scheduled and sits in PENDING forever. If your jobs never start, set memory_per_node (or rely on the provider default).
Queue times can be long. The cluster is shared, so a job may sit queued for a while before it runs. wait_for_completion() logs the queue reason on each poll, and you can bound the wait with queue_timeout / run_timeout.
Never hardcode credentials. They arrive through the environment the provider set up; the client reads them for you.

What’s next#

Run your solver

Edit or update this solver