What Blockflow chooses automatically

Blockflow derives dependencies and execution geometry from an explicitly selected image computation. It also offers planners that can choose block grids and phase partitions. Those facilities are opt-in: constructing an operation does not ask an unrestricted optimizer to choose the scientific model, storage format and execution backend for an entire experiment.

The acquisition guide and execution story use the same concrete image as this page.

The useful question is therefore which component owns each decision, and when is that component asked? The distinction matters when diagnosing a plan that is correct but slow, or a convenient default that changes the intended analysis.

Five owners of a decision

flowchart TB
  C["Caller: images, algorithm, execution route"] --> O["Operation: support, types, laws and workspace"]
  C --> P["Explicit assembly or opt-in planning"]
  O --> P
  P --> B["Recorded phases, grids and source regions"]
  B --> S["Scheduler: readiness, priority and admission"]
  B --> E["Storage backend: reads, chunks and cache"]
  S <--> E
Figure 1: The caller selects the computation and planning route. Contracts constrain binding geometry; execution and storage make choices within those constraints.
Decision Owner and trigger What remains explicit
Time, channel, level and spatial axes Caller selects an OmeVolumePyramid view or attached image. Which signal to analyze, whether channels are registered and what the physical frame means.
PSF, noise model, threshold and stopping rule Caller constructs the algorithm and its parameters. Calibration, model validity and scientific acceptance criteria. No general automatic choice is implied.
Required support and output type The operation’s BlockOp contract. The caller chooses the operation; the planner cannot shrink its required neighborhood to fit memory.
Phase boundaries and grids Explicit PlanBuilder calls, or an explicitly invoked planner. Which planning search is permitted, its candidates, constraints and cost model.
Order and simultaneous work Scheduler uses the recorded graph and Hints. Resource policy, requested concurrency and backend-specific setup.
Chunk reads, compression and caching The selected environment/backend within its configured contracts. Existing input chunks, output publication requirements and whether prefetch/GPU/distributed paths are enabled.

A mistake in a cost estimate should affect speed, not the image’s mathematical definition. That separation is why algorithm parameters, binding plan geometry and advisory execution choices are different layers.

An explicit-grid example

Consider an acquisition shaped [4,2,64,512,512] in t,c,z,y,x order with stored chunks [1,1,8,64,64]. The caller chooses one time, one channel and one pyramid level, exposing a spatial volume [64,512,512]. Processing every time and channel requires explicit orchestration of eight such selections; it is not inferred merely from the presence of five axes.

Suppose the requested calculation is Gaussian smoothing followed by a supplied fixed threshold. The caller supplies sigma, truncation, edge convention and threshold value. SmoothOp derives the Gaussian support and produces f64 intensities; a ThresholdMask wrapped by VoxelwiseMaskOp then produces its declared mask carrier.

With an explicit BlockGrid::new shape [16,128,128], the selected volume has 4×4×4=64 cores. Calling PlanBuilder::pixels once with the composed Chain requests one pixel phase. Calling it twice requests two stored phase boundaries. The builder does not silently replace either request with a cheaper one. PlanBuilder::finish derives/checks support, types and source identities for that assembly.

At sigma one and truncation three, smoothing needs a three-voxel margin per axis. The threshold adds no spatial reach. An interior fused task therefore reads [22,134,134], not just [16,128,128]. An aligned core covers eight of the stored spatial chunks before adding its halo; the expanded read can intersect substantially more chunks. Core/chunk alignment does not remove halo reads, and a cache hit does not remove the algorithm’s dependency.

What an opt-in grid search changes

choose_grid takes a volume, Host, CostModel and a callback that builds the actual assembly on each proposed grid. It prices those assemblies under the host’s worker and active-memory budget, then returns one ChosenGrid with candidate outcomes and predicted active sets.

This route may select a different grid from [16,128,128]. It does not change the selected channel, choose sigma, invent a threshold or replace smoothing with another denoiser. The build callback still describes the requested pipeline.

Its grid_candidates family cuts each axis into power-of-two piece counts, retaining whole axes and using a minimum offered cut edge of eight. Candidates are visited in increasing block-count order. The search normally stops beyond four blocks per worker once it has a feasible answer; it can continue farther to find one that fits. Ties retain the coarser candidate. This is a bounded candidate search, not every possible integer block shape.

Importantly, that candidate generator receives the spatial volume, not the input Zarr chunk shape. The example’s power-of-two shapes may happen to align well with [8,64,64], but automatic input-chunk alignment is not its contract. If chunk geometry is essential, state it through the appropriate explicit grid and storage constraints, then inspect the plan’s actual source regions.

Grid search and partition search are different routes

PlanBuilder::partition asks a selected Strategy to split a supplied chain using Constraints. The builder’s current grid is not converted into that strategy’s candidate list. The constraints provide the candidate edges, split axes and permitted resources.

Greedy cuts at structural barriers, incompatible block mandates and changes in preferred traversal, then picks feasible blocks for its groups. Its heuristic is not a proof of the cheapest partition. Enumerating prices contiguous chain partitions, with a dynamic-programming search selected by default through PartitionSearch. Its exhaustive alternative is a comparison oracle; its single-group mode asks only for a block choice while preserving one phase. The search is optimal only under that candidate family and additive cost model; it does not optimize an unrestricted image DAG.

Partitioning preserves operation order and absolute source-image identities. It can choose materialisation points, not reorder scientific steps arbitrarily. The plan-wide grid search and chain partition search can be combined through a build callback, but they remain distinct choices. See fusion and planning for their tradeoffs.

Resource defaults and runtime adaptation

The caller/application establishes a cap using MemoryCap::resolve and, where appropriate, set_process_cap. Resolution considers explicit settings, environment, cgroup and machine evidence; a smaller binding cgroup limit wins. A reserve separates the cap from usable active-work capacity. Host::detect also detects worker availability; an application can override its pool width.

The planner can trade finer blocks against fewer simultaneously active coarse blocks. The scheduler then admits ready work as slots and memory become available. Default Hints request one concurrent task and phase-major priority; a chosen strategy or caller can supply other values. Near-cap measurements can refine runtime charges without changing the already recorded cores and source windows.

There is a deliberate limit to the protection: the local admission path can admit a lone oversized task when nothing else is active, avoiding a deadlock. It is therefore not an allocator-enforced hard RSS cap. In contrast, the grid search rejects candidates for which even one predicted block cannot fit its budget. Unpriced application allocations, resident helpers and codec/model workspaces still require explicit attention. See memory and storage.

Backends and optimizations require their own trigger

Traversal preferences and scheduler priority rank legal work; they do not remove phase images. ZarrEnvironment provides the chunk-backed storage path and constructs a shared cache by default. An application can disable it using without_cache or set capacity using with_cache. Prefetch is a separate opt-in with_prefetch setup requiring a cache. A nonzero advisory prefetch depth does not prove that every environment submits asynchronous future reads.

GPU use likewise requires a supported operation/backend and device policy, such as an Accelerated wrapper configured with an available kernel. CPU fallback versus required-GPU failure follows DevicePolicy. Transfers and device buffers still cost time/memory; this does not imply automatic fusion of arbitrary neighboring GPU operations. Distributed execution requires its coordinator, worker and shared-storage configuration. A local execution call does not automatically launch remote workers.

Skipping work also follows declared laws. Uniform-block shortcuts require the operation’s constant_maps_to contract; missing chunks represent background only where that background was declared. Iterative dirty-block skipping requires stationary update semantics. The executor cannot infer either property from a familiar algorithm name. Internal-image release follows reader lifetimes, with explicit retention/release overrides in Hints.

For newly written internal/output arrays, the backend derives chunks from the producing phase’s grid and dtype through chunk_for_grid. This is an automatic output-layout choice, not copying input chunks. It seeks roughly 256 KiB to 4 MiB chunks while honoring write ownership and divisibility; prime extents can limit that target. For the example’s [16,128,128] blocks, both an f64 work chunk (2 MiB) and a boolean chunk (256 KiB) can retain that shape. The input chunks remain [8,64,64]. A caller-dictated output layout adds constraints and can be refused if it would permit overlapping chunk writers.

Finally, operation produces contracts determine image dtype. Units, physical transforms, multiscale metadata, annotation names and publication destinations remain application responsibilities. A successfully executed image array is not automatically a correctly calibrated, fully published OME-Zarr result.