Cluster jobs

rbs job is scheduler mode: submit arbitrary commands or scripts to a cluster’s worker fleet with Slurm-class semantics — queues, priorities, dependencies, job arrays, time limits, requeue on worker loss, live logs — and none of Slurm’s setup. Jobs ride the same scheduler and workers as remote build execution, but consume dedicated per-worker job slots, so a fleet full of jobs can still execute the build actions those jobs spawn.

Pointing at a cluster

Every rbs job command needs a cluster address, from --server host:port or the RBS_REMOTE environment variable (with RBS_REMOTE_TOKEN for authentication):

export RBS_REMOTE=cluster.example.com:8980
export RBS_REMOTE_TOKEN=<token>

The cluster itself is just rbs: rbs remote serve runs the scheduler (and cache), and each machine that should execute work joins with rbs remote worker. rbs remote status shows workers, queue depth, and jobs by state — see Remote Cache & Execution.

Submitting work

rbs job submit [flags] -- command [args...]
rbs job submit --script run.sh          # or submit a shell script

# A GPU training run with a time limit
rbs job submit --name train --queue batch --gpus 2 --memory-gb 32 --time 4h \
  -- python train.py --epochs 50

# 100-task array, at most 10 running at once
rbs job submit --array "0-99%10" -- python process_shard.py

Submit flags

Flag	Meaning
`--name`	Job name (defaults to the command or script name).
`--queue`	Queue to submit to (default `batch`). `rbs job queues` lists them.
`--priority`	Higher runs first; queued age adds a bonus so nothing starves.
`--cpus`, `--memory-gb`, `--gpus`	Resources to reserve on the worker. The scheduler bin-packs against them, and GPU jobs get concrete devices pinned so two jobs never see the same card.
`--time`	Time limit (`30m`, `2h`); jobs that exceed it end as `TIMEOUT`.
`--deps`	Dependencies: `afterok:ID,afterany:ID,afternotok:ID` (comma-separated).
`--array`	Array spec: `0-99`, `1,3,7`, or `0-99%10` (`%N` throttles concurrency). Each task gets `RBS_ARRAY_INDEX`.
`--env KEY=VALUE`	Environment for the payload (repeatable).
`--inputs`	Content for the job’s working directory (repeatable) — see below.
`--outputs`	A file or directory the job produces, relative to its working directory (repeatable) — see below.
`--nodes N`	Gang-allocate N workers for one job (MPI-style) — see multi-node jobs.
`--container IMAGE`	Run the payload inside this OCI image (the worker needs docker or podman).
`--no-requeue`	Fail on worker loss instead of requeuing.
`--preemptible`	Allow higher-priority work to preempt this job.
`--interactive` / `-i`	Attach stdin/stdout after submitting (Ctrl-D closes stdin).
`--script FILE`	Submit a shell script instead of a command line.
`--server`	Cluster address (default `$RBS_REMOTE`).

Getting code in and results out

Traditional schedulers assume every node sees a shared filesystem, so getting code onto the cluster is your problem. rbs jobs move content instead, through the same content-addressed store the build cache uses. --inputs stages content into the directory the payload starts in. Each value is a path, a built //pkg:target label, or DEST=SRC to land content in a subdirectory:

rbs build //app:train                        # build once, locally

rbs job submit \
  --inputs . \                               # the current directory (skips .git and .rbs)
  --inputs bin=//app:train \                 # a built target's outputs, under bin/
  --outputs checkpoints --outputs metrics.json \
  -- python train.py

Inputs are staged before the job is queued, and only blobs the cluster does not already hold cross the wire — resubmitting a tree that changed by one file uploads one file, and a gang of N nodes costs one upload rather than N clones. A target label stages the outputs already built for RBS_TARGET_PLATFORM, so on a cross-platform cluster, build for the workers’ platform first. --outputs declares what to collect back. The worker records the declared paths into the cluster’s content store when the job ends — including when it fails, so a crashed run’s checkpoints and logs are still recoverable — and you pull them down from wherever you are:

rbs job outputs 42                  # → ./job-42-outputs/
rbs job outputs 42 --dest ./results

Managing jobs

Command	What it does
`rbs job list`	Active jobs; `--state`, `--queue` filter, `--history` includes finished jobs.
`rbs job show <id>`	Full detail: state and pending reason, priority breakdown, resources, dependencies, array position, attempts, peak memory / OOM, timestamps.
`rbs job logs <id>`	Print the job’s output; `-f` follows live until it ends.
`rbs job wait <id...>`	Block until the jobs finish; non-zero exit if any failed.
`rbs job cancel <id...>`	Cancel jobs.
`rbs job outputs <id>`	Download the files the job declared with `--outputs`.
`rbs job queues`	Queues with their priorities, time limits, and occupancy.
`rbs job reserve <name>`	Create (`--in`, `--for`, `--reason`) or cancel (`--cancel`) a maintenance window: new work stops, running work drains.

rbs job submit --name step1 -- ./prepare.sh
rbs job submit --deps afterok:17 -- ./train.sh     # runs only if job 17 succeeds
rbs job wait 18 && rbs job outputs 18

The environment a job sees

The worker injects context into every payload:

Variable	When	Meaning
`RBS_JOB_ID`	always	The job’s id.
`RBS_ARRAY_INDEX`	array jobs	This task’s index.
`RBS_NODELIST`, `RBS_NNODES`, `RBS_NODE_RANK`	multi-node jobs	The gang’s hosts, size, and this node’s rank.
`RBS_MASTER_ADDR`, `RBS_MASTER_PORT`	multi-node jobs	Rendezvous address (rank 0) and port.
`RANK`, `WORLD_SIZE`, `MASTER_ADDR`, `MASTER_PORT`	multi-node jobs	The same values in the shape PyTorch/torchrun expects.

Multi-node jobs (--nodes N) gang-allocate N workers: the same command starts on every node with its rank injected, rank 0 drives, and distributed launchers like torchrun work out of the box:

rbs job submit --nodes 4 --gpus 8 \
  --inputs . --outputs ckpt \
  -- torchrun --nproc_per_node=8 train.py

Job targets in build files

A job’s shape — binary, resources, node count, retry policy — can also be declared as a target, keeping it versioned next to the code it runs:

load("@rbs//job/rules.rbs", "ml_training_job")

ml_training_job(
    name = "train_2node",
    binary = ":train_multinode",      # a py_binary or other built binary
    nodes = 2,
    cpu = "4",
    memory = "8Gi",
    max_retries = 5,
    checkpoint_dir = "checkpoints/train",
    env = {"EPOCHS": "20"},
)

ml_training_job accepts binary, data, args, env, cpu (default "1"), memory (default "4Gi"), gpus (default 0), partition, time_limit (default "1h"), nodes (default 1), plus fault-tolerance attributes: elastic, min_nodes, max_nodes, max_retries (default 3), and checkpoint_dir. batch_job is the simpler shape (binary, inputs, args, env, cpu, memory), and gpu_job(name, binary, gpus, ...) wraps ml_training_job with a GPU partition default.

Job targets are experimental. Building one produces a job manifest describing the work — it does not execute anything, and the scheduler does not yet consume these manifests directly. The supported execution path today is rbs job submit, which composes with built targets via --inputs //pkg:target.

Getting Started

Build

Languages

Containers & Deploy

Infrastructure

CI & Remote Builds

Team Workflow

Pointing at a cluster

Submitting work

Submit flags

Getting code in and results out

Managing jobs

The environment a job sees

Job targets in build files

​Pointing at a cluster

​Submitting work

​Submit flags

​Getting code in and results out

​Managing jobs

​The environment a job sees

​Job targets in build files

Pointing at a cluster

Submitting work

Submit flags

Getting code in and results out

Managing jobs

The environment a job sees

Job targets in build files