> ## Documentation Index > Fetch the complete documentation index at: https://docs.reasonos.com/llms.txt > Use this file to discover all available pages before exploring further. # Cluster jobs > Submit and manage batch work on an rbs cluster with rbs job — queues, priorities, dependencies, arrays, multi-node gangs, and content-addressed inputs and outputs. `rbs job` is **scheduler mode**: submit arbitrary commands or scripts to a cluster's worker fleet with Slurm-class semantics — queues, priorities, dependencies, job arrays, time limits, requeue on worker loss, live logs — and none of Slurm's setup. Jobs ride the same scheduler and workers as [remote build execution](/ci/remote-execution), but consume dedicated per-worker job slots, so a fleet full of jobs can still execute the build actions those jobs spawn. ## Pointing at a cluster Every `rbs job` command needs a cluster address, from `--server host:port` or the `RBS_REMOTE` environment variable (with `RBS_REMOTE_TOKEN` for authentication): ```bash theme={null} export RBS_REMOTE=cluster.example.com:8980 export RBS_REMOTE_TOKEN= ``` The cluster itself is just rbs: `rbs remote serve` runs the scheduler (and cache), and each machine that should execute work joins with `rbs remote worker`. `rbs remote status` shows workers, queue depth, and jobs by state — see [Remote Cache & Execution](/ci/remote-execution). ## Submitting work ```bash theme={null} rbs job submit [flags] -- command [args...] rbs job submit --script run.sh # or submit a shell script ``` ```bash theme={null} # A GPU training run with a time limit rbs job submit --name train --queue batch --gpus 2 --memory-gb 32 --time 4h \ -- python train.py --epochs 50 # 100-task array, at most 10 running at once rbs job submit --array "0-99%10" -- python process_shard.py ``` ### Submit flags | Flag | Meaning | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--name` | Job name (defaults to the command or script name). | | `--queue` | Queue to submit to (default `batch`). `rbs job queues` lists them. | | `--priority` | Higher runs first; queued age adds a bonus so nothing starves. | | `--cpus`, `--memory-gb`, `--gpus` | Resources to reserve on the worker. The scheduler bin-packs against them, and GPU jobs get concrete devices pinned so two jobs never see the same card. | | `--time` | Time limit (`30m`, `2h`); jobs that exceed it end as `TIMEOUT`. | | `--deps` | Dependencies: `afterok:ID,afterany:ID,afternotok:ID` (comma-separated). | | `--array` | Array spec: `0-99`, `1,3,7`, or `0-99%10` (`%N` throttles concurrency). Each task gets `RBS_ARRAY_INDEX`. | | `--env KEY=VALUE` | Environment for the payload (repeatable). | | `--inputs` | Content for the job's working directory (repeatable) — see below. | | `--outputs` | A file or directory the job produces, relative to its working directory (repeatable) — see below. | | `--nodes N` | Gang-allocate N workers for one job (MPI-style) — see multi-node jobs. | | `--container IMAGE` | Run the payload inside this OCI image (the worker needs docker or podman). | | `--no-requeue` | Fail on worker loss instead of requeuing. | | `--preemptible` | Allow higher-priority work to preempt this job. | | `--interactive` / `-i` | Attach stdin/stdout after submitting (Ctrl-D closes stdin). | | `--script FILE` | Submit a shell script instead of a command line. | | `--server` | Cluster address (default `$RBS_REMOTE`). | ## Getting code in and results out Traditional schedulers assume every node sees a shared filesystem, so getting code onto the cluster is your problem. rbs jobs move content instead, through the same content-addressed store the build cache uses. **`--inputs`** stages content into the directory the payload starts in. Each value is a path, a built `//pkg:target` label, or `DEST=SRC` to land content in a subdirectory: ```bash theme={null} rbs build //app:train # build once, locally rbs job submit \ --inputs . \ # the current directory (skips .git and .rbs) --inputs bin=//app:train \ # a built target's outputs, under bin/ --outputs checkpoints --outputs metrics.json \ -- python train.py ``` Inputs are staged **before** the job is queued, and only blobs the cluster does not already hold cross the wire — resubmitting a tree that changed by one file uploads one file, and a gang of N nodes costs one upload rather than N clones. A target label stages the outputs already built for `RBS_TARGET_PLATFORM`, so on a cross-platform cluster, build for the workers' platform first. **`--outputs`** declares what to collect back. The worker records the declared paths into the cluster's content store when the job ends — **including when it fails**, so a crashed run's checkpoints and logs are still recoverable — and you pull them down from wherever you are: ```bash theme={null} rbs job outputs 42 # → ./job-42-outputs/ rbs job outputs 42 --dest ./results ``` ## Managing jobs | Command | What it does | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | | `rbs job list` | Active jobs; `--state`, `--queue` filter, `--history` includes finished jobs. | | `rbs job show ` | Full detail: state and pending reason, priority breakdown, resources, dependencies, array position, attempts, peak memory / OOM, timestamps. | | `rbs job logs ` | Print the job's output; `-f` follows live until it ends. | | `rbs job wait ` | Block until the jobs finish; non-zero exit if any failed. | | `rbs job cancel ` | Cancel jobs. | | `rbs job outputs ` | Download the files the job declared with `--outputs`. | | `rbs job queues` | Queues with their priorities, time limits, and occupancy. | | `rbs job reserve ` | Create (`--in`, `--for`, `--reason`) or cancel (`--cancel`) a maintenance window: new work stops, running work drains. | ```bash theme={null} rbs job submit --name step1 -- ./prepare.sh rbs job submit --deps afterok:17 -- ./train.sh # runs only if job 17 succeeds rbs job wait 18 && rbs job outputs 18 ``` ## The environment a job sees The worker injects context into every payload: | Variable | When | Meaning | | -------------------------------------------------- | --------------- | ------------------------------------------------------ | | `RBS_JOB_ID` | always | The job's id. | | `RBS_ARRAY_INDEX` | array jobs | This task's index. | | `RBS_NODELIST`, `RBS_NNODES`, `RBS_NODE_RANK` | multi-node jobs | The gang's hosts, size, and this node's rank. | | `RBS_MASTER_ADDR`, `RBS_MASTER_PORT` | multi-node jobs | Rendezvous address (rank 0) and port. | | `RANK`, `WORLD_SIZE`, `MASTER_ADDR`, `MASTER_PORT` | multi-node jobs | The same values in the shape PyTorch/torchrun expects. | Multi-node jobs (`--nodes N`) gang-allocate N workers: the same command starts on every node with its rank injected, rank 0 drives, and distributed launchers like `torchrun` work out of the box: ```bash theme={null} rbs job submit --nodes 4 --gpus 8 \ --inputs . --outputs ckpt \ -- torchrun --nproc_per_node=8 train.py ``` ## Job targets in build files A job's shape — binary, resources, node count, retry policy — can also be declared as a target, keeping it versioned next to the code it runs: ```python theme={null} load("@rbs//job/rules.rbs", "ml_training_job") ml_training_job( name = "train_2node", binary = ":train_multinode", # a py_binary or other built binary nodes = 2, cpu = "4", memory = "8Gi", max_retries = 5, checkpoint_dir = "checkpoints/train", env = {"EPOCHS": "20"}, ) ``` `ml_training_job` accepts `binary`, `data`, `args`, `env`, `cpu` (default `"1"`), `memory` (default `"4Gi"`), `gpus` (default `0`), `partition`, `time_limit` (default `"1h"`), `nodes` (default `1`), plus fault-tolerance attributes: `elastic`, `min_nodes`, `max_nodes`, `max_retries` (default `3`), and `checkpoint_dir`. `batch_job` is the simpler shape (`binary`, `inputs`, `args`, `env`, `cpu`, `memory`), and `gpu_job(name, binary, gpus, ...)` wraps `ml_training_job` with a GPU partition default. Job targets are **experimental**. Building one produces a job manifest describing the work — it does not execute anything, and the scheduler does not yet consume these manifests directly. The supported execution path today is `rbs job submit`, which composes with built targets via `--inputs //pkg:target`.