How to Submit Jobs to Slurm#

Some jobs can be time‑consuming, and we want to avoid reserving a powerful node solely for interactive work. Therefore, you can run the interactive part (the Tomwer node) with fewer resources (for example, without a GPU) and submit jobs to a node that will be released once processing is complete.

In Tomwer, resource‑consuming tasks can be executed on Slurm:

  • Slice / volume reconstruction:

    • nabu slice

    • nabu volume

    • multi-cor (formerly known as sa-axis)

    • multi-delta-beta (formerly known as sa-delta-beta)

  • volume casting

Warning: To launch jobs on Slurm you need to be on a slurm-client of the ESRF cluster. A simple way to check is to ensure you have access to the sbatch command.

Then you will need to handle job submission and the future object using the following widgets:

  • Slurm Cluster widget

  • Future Supervisor widget

Note: Both are part of the ‘cluster’ group.

slurm cluster Widget slurm cluster icon#

This widget is used to specify the partition and configuration for creating nodes.

Add it to the canvas and configure it as follows:

slurm cluster configuration

Then, connect it to one of the widgets that use it (as shown above) via the cluster_config input. In this example, it is connected to the nabu volume widget.

slurm cluster example

Tip: Since version 1.3, you can also select a specific GPU from the advanced settings. 2d685f06615d4e19937d746fa2be67ae

Warning: The available GPU types are updated once a partition is selected.

future supervisor Widget future supervisor icon#

This widget provides feedback on the various jobs currently being executed remotely.

Once a job is finished, it will be integrated back into your workflow by default.

tutorials/notebooks/canvas/img/future_supervisor_with_nabu_volume.png

Bash Script#

Each process compatible with Slurm will generate a bash script (.sh) in the ‘slurm_scripts’ folder.

The log files associated with these scripts will have the same name but with a different extension (.out).

If needed, you can relaunch these scripts from the command line using the sbatch command.

Demo Video of a Submission to Slurm#

Here is a video demonstration of this feature.

Note: The interface of the Slurm Cluster widget has been updated (simplified) since the video, but the underlying behavior and logic remain the same.

[1]:

from tomwer.resources.video import Video

Video("HYufV7Ya9v8", height=500, width=800)

[1]:

Hands‑on Exercise A#

Reconstruct a slice from the crayon.nx NXtomo (available from /data/projects/tomo-sample-data/training/workspaces/[date]/[user_id]/edition/crayon.nx).

Please connect to the interactive or nice partition and then submit your job to Slurm:

salloc --partition interactive --mem=32G --time=01:00:00 --x11 srun --pty bash -l
module load tomotools

Then, reconstruct one slice or a part of the volume (approximately 20 slices).

Warning: Slurm must be aware of the file location. If the processing involves a location other than /data/..., Slurm might not process it correctly.

Note: Some misalignments can exist between different versions of tomotools. It is recommended to use the same environment as the one you are currently in. 3cec6305fa424b3cb6d8d979295392c1

[ ]: