Running on a Cluster¶

When local laptop time isn't enough, you can take the same project to a SLURM HPC system. The same execution path supports interactive work inside an allocation and queued work submitted with lc run --async.

The big picture¶

lc run always dispatches through a Dask cluster. Three branches:

1. On your laptop → a LocalCluster sized to the machine.
2. Inside a SLURM allocation → an in-process scheduler bound to the driver's hostname, with one dask worker per allocated node launched via srun.
3. With DASK_SCHEDULER_ADDRESS set → connect to whatever scheduler you've pointed at.

You don't pick — lc run detects which case applies. The only thing you do differently on a cluster is request the nodes.

For longer work, lc run --async is an outer submission layer. It asks SLURM for an allocation and then executes ordinary lc run inside it, so recipe behavior and provenance stay identical.

Pre-flight: pick the right container runtime¶

On most HPC sites, docker isn't available on compute nodes. Most SLURM systems (including NERSC Perlmutter) provide podman-hpc. On a login node:

$EDITOR ~/.lightcone/config.yaml

container:
  runtime: podman-hpc
slurm:
  account: m1234
  time_padding: 1.5

slurm.account is required the first time you use --async; alternatively, pass --account m1234 for one submission. time_padding multiplies the serial sum of recipe time estimates and defaults to 1.5.

Then build and migrate the images for your project:

cd my-analysis
lc build

lc build runs podman-hpc build and then podman-hpc migrate, which copies the image into the per-node container cache. Compute nodes can read it without registry access.

If your site has only apptainer / singularity, the Lightcone toolchain doesn't ship explicit support for those today — you can run without containers (runtime: none) for the moment, with the caveat that the manifest's container_image field will record what was declared, not what executed. (See lc run for the provenance warning.)

A typical SLURM workflow¶

1. Choose interactive or queued execution¶

For short, agent-in-the-loop work, get an interactive allocation and run synchronously:

salloc -A m1234 -N 1 -t 02:00:00 -C gpu -q interactive
lc run heavy_fit -u baseline

For long or unattended work on Perlmutter, submit it from either a login or compute node:

lc run --async heavy_fit -u baseline
lc status

Lightcone chooses shared or regular, writes the script under .lightcone/jobs/, stores logs in scratch, and records the returned job id.

2. lc run inside the allocation¶

Once SLURM_JOB_ID is set in your environment, lc run does the rest:

• Starts an in-process Dask scheduler bound to the SLURM node hostname.
• Launches one dask worker per node via srun.
• Each worker advertises the node's CPU, memory, and GPU resources.
• Snakemake submits each rule via the Dask executor; rules with per-recipe resources: constraints land on workers that can hold them.

3. Per-recipe resources¶

Add resource hints in your astra.yaml recipe blocks:

outputs:
  - id: heavy_fit
    type: metric
    recipe:
      command: python scripts/fit.py --output {output}
      resources:
        cpus: 16
        memory: 64GB
        gpus: 1
        time_limit: 2h

Lightcone maps these portable ASTRA names to Snakemake/Dask resource requests for synchronous execution and uses the same declaration to size asynchronous jobs. Every recipe in an asynchronously submitted sub-DAG needs time_limit; the other fields have conservative zero/one defaults.

Interactive: agent-driven runs¶

During development you're usually iterating — ask the agent to build something, check the result, adjust the spec, repeat. For that loop you want to run the agent itself from inside a SLURM allocation so that lc run executes on the compute node rather than the login node.

salloc -A <your_project> -q interactive -C gpu --nodes=1 -t 00:30:00
# salloc drops you onto a compute node; from there:
cd /path/to/your-analysis
claude                   # or whichever agent CLI you prefer

Everything the agent triggers (lc run, scripts, etc.) now executes on the allocated node. When you're done iterating and want a hands-off work, use lc run --async. This is also valid while the agent is still on the compute node; sbatch queues an independent allocation.

What about login-node-only operations?¶

Build images, submit queued work, and look at status are all fine on a login node without an allocation:

lc build                       # build images (uses podman-hpc on login node)
lc run --async heavy_fit       # submits; does not execute on the login node
lc status                      # manifests + on-demand Slurm polling

Plain lc run must happen inside an allocation. On Perlmutter login nodes it fails with an allocation example and a hint to try --async.

External Dask schedulers¶

If you have a long-lived Dask cluster (Slurm jobqueue, k8s, etc.) that you'd rather attach to:

export DASK_SCHEDULER_ADDRESS=tcp://my-scheduler:8786
lc run

lc run notices the env var and connects rather than starting its own scheduler. It does not tear the scheduler down on exit.

NERSC Perlmutter: site-specific notes¶

Storage: keep Snakemake state on $SCRATCH¶

lc redirects state automatically when it detects Perlmutter, so this usually just works. To pin explicitly at project creation:

lc init your-analysis --scratch '$SCRATCH'   # kept verbatim, expanded at run time

Or, after the fact, edit <project>/.lightcone/lightcone.yaml:

scratch_root: $SCRATCH

Further reading¶

• NERSC interactive jobs — salloc patterns and reservation queues
• Perlmutter system overview — node types and partitions
• NERSC queue policy — QoS options for GPU and CPU partitions

Troubleshooting¶

• dask CLI is not on PATH inside the SLURM allocation. Install lightcone-cli into the venv that your sbatch script activates; dask ships with distributed, which is a transitive dep.
• Workers never register. Usually means the SLURM node hostnames aren't resolvable from each other; check SLURMD_NODENAME / gethostname() and confirm the workers can reach the scheduler.
• Image not found on compute nodes. Re-run lc build on the login node — the migrate step is the one that actually publishes the image to the per-node cache.
• Snakemake locking errors or silent rule-rerun loops on Perlmutter. .snakemake/ ended up on DVS-mounted storage — set scratch_root: $SCRATCH in the project's .lightcone/lightcone.yaml.
• pip install hangs or times out. Compute nodes have no public internet — always install from a login node.
• PermissionError reading another user's symlinked results/. Cross-user scratch path without group ACLs — request access from the data owner, or copy the manifests into your own scratch.

For the wiring detail, see engine/dask_cluster in the maintainer docs.