flowchart TB A["Recorded task dependencies"] --> B["CPU task A"] A --> C["CPU task B"] B --> D["Optional safe block slices"] C --> E["Supported device-backed kernel"] A --> F["Distributed coordinator"] F --> G["Remote worker processes"] G --> H["Committed shared storage"]
CPU, GPU and distributed execution
Parallel execution is constrained by dependencies, available work, memory and backend support. Increasing the number of workers does not automatically make a kernel parallel or turn a resident algorithm into an out-of-core one. The useful unit of parallelism differs between task scheduling, kernel internals, device execution and remote processes.
Between-block concurrency
Independent ready tasks can occupy CPU workers up to Hints::concurrency, subject to memory admission and operation-specific limits. A dataset cut into only two tasks cannot occupy sixteen workers through task-level parallelism alone. Conversely, thousands of tiny tasks may expose ample parallelism while spending too much time on halos, decoding and scheduling overhead.
BlockOp::max_in_flight can restrict a kernel’s simultaneous applications. This is useful for a model worker or scarce backend resource that should not be treated as one unrestricted resource per CPU task. The useful active set is therefore bounded by task availability, worker capacity, the memory budget and such per-phase limits together.
Within-block slicing has its own correctness contract
Slicing describes whether an operation’s output survives an internal cut. A finite pixel reach alone is insufficient: a scan can carry a state from the start of a line and give a different answer if that line is split. Operations that do not declare safe slicing retain ordinary whole-block execution.
For a sliceable stencil, each slab receives internal halos sufficient for the kernel and commits only its owned output. SlabPolicy controls whether this additional parallelism is used when the outer grid does not already provide enough work. SlabCut records the resulting slices. The extra halos repeat work, so dividing a small block among many threads can cost more than it saves.
A useful comparison holds the whole block’s output fixed, varies slab count, and checks the sliced result against the uncut kernel. Bitwise equality is appropriate when only work assignment changes and every voxel performs the same arithmetic in the same order. A generic tolerance should not hide a seam caused by an insufficient internal halo.
Some kernels instead parallelize their own rows, FFT tiles or model batches. Those internal workers can compete with outer task workers. Measure the actual pipeline’s CPU use and memory pressure before multiplying both thread counts.
GPU support is operation-specific
GpuKernel describes a supported device computation. Accelerated wraps a CPU operation with an accelerated alternative, and DevicePolicy determines the placement policy supported by that route. The presence of a GPU feature in the build does not mean every operation has a compatible kernel, dtype or boundary implementation.
Host and device memory are separate constraints. DeviceAdmission coordinates device task slots and workspace charges, while host buffers and source decoding still consume the process budget. An oversized device request can also be admitted alone; this is concurrency control rather than an allocation guarantee. Model inference can add weights, batch workspaces and runtime allocations beyond a simple input/output pair.
The current generic accelerated-operation route uploads its input and downloads its output for each GPU operation. A sequence of two such operations therefore can round-trip through host memory between kernels; generic chain composition does not imply persistent device-resident intermediates. Host widening to the device’s supported representation can also increase transfer bytes. These costs are reasons to measure end-to-end time, not only kernel execution.
Remote processes need committed data and explicit ownership
The distributed path represents a run using JobSpec and a WorkflowSpec. A Coordinator assigns ready tasks to workers; WorkerOptions configures the worker’s local execution. Processes read and commit through supported shared storage instead of assuming that another worker’s temporary buffer is visible. The numerical dependency graph remains the same, but transport and publication become explicit parts of completion.
A worker must finish the relevant output commit before reporting completion. Otherwise the coordinator could unlock a consumer whose source chunks or fragments are still missing. Backend caches also must not preserve a negative lookup after another process has published the requested object. These are visibility contracts, not merely bandwidth concerns.
Retries require an ownership protocol. A lost worker is not automatically recoverable under every job configuration. An explicit task lease allows the coordinator to return expired claims to the pending set; the default job has no such timeout. Lease duration must account for task execution and prefetched claims, or a healthy slow worker can be mistaken for a lost one. Reported failures and expired leases are different signals.
Limits that remain algorithmic
The generic iterative image executor carries private whole-volume state in one process and is not a distributed image-iteration implementation. A global solve or a reduction retaining all objects also does not become distributed merely because its block mapping stage uses multiple workers. Consult each operation’s route and workspace contract.
Good scaling comparisons preserve numerical conventions and include reads, transfers, commits and repeated work. Coordinator throughput, storage contention, model device contention and uneven task costs can each limit speedup. A perfectly balanced task count is not evidence of balanced runtime; progress and I/O events reveal where workers actually spend their time.