flowchart TB A["Select t=1, c=0, level=0"] --> B["Image 0: u16, 64 x 512 x 512"] B --> C["Phase 0: 64 smoothing tasks"] C --> D["Image 1: stored f64 intermediate"] D --> E["Phase 1: 64 pointwise threshold tasks"] E --> F["Image 2: stored boolean result"] F --> G["Publish with correct metadata"]
One image through the executor
This walkthrough follows one concrete smoothing-and-thresholding pipeline from an OME-Zarr acquisition to committed output. The numbers describe geometry and logical array bytes; they are not measured timings or compressed file sizes. The block grid and phase boundaries are chosen explicitly to make the behavior easy to inspect. Dimensions and image selection explains the acquisition axes; automatic and caller choices separates this explicit plan from cost-based selection.
Select one spatial image
Assume the acquisition declares axes t,c,z,y,x, shape [4,2,64,512,512], u16 pixels and storage chunks [1,1,8,64,64]. Select time index 1, channel index 0 and level zero. OmeVolumePyramid::open takes channel before time; its attached_level method produces an AttachedImage fixing those nonspatial indices. The resulting spatial view has shape [64,512,512] and source chunks [8,64,64].
Selection opens metadata and establishes an indexing rule; it does not decode the entire acquisition. Other time points and channels are not additional operands of this run. Axis names, physical transforms and the selected level must come from the metadata rather than being guessed from the five array extents. This story stipulates their values; a real application checks them.
Assemble two phases
Choose compute blocks [16,128,128]. The volume divides into 4 × 4 × 4 = 64 cores with no partial edge cores. Phase zero smooths with sigma one voxel on each axis, truncated at three standard deviations. SmoothOp declares reach [3,3,3], accepts the u16 input and produces f64. Phase one marks smoothed values strictly above a caller-supplied threshold, producing a boolean image through VoxelwiseMaskOp and ThresholdMask.
There are 64 tasks per phase and 128 tasks total. Image zero is the attached input, image one is the f64 intermediate, and image two is the boolean result. The two calls to PlanBuilder::pixels deliberately materialise the smoothing result. The following assembly sketch uses the actual APIs for this stipulated dataset:
use std::path::Path;
use blockflow::{
assemble::PlanBuilder,
budget::{set_process_cap, MemoryCap},
geometry::BlockGrid,
ome_zarr::OmeVolumePyramid,
op::Chain,
ops::{SmoothOp, voxelwise::{ThresholdMask, VoxelwiseMaskOp}},
strategy::{execute_phases, Hints},
zarr_env::ZarrEnvironment,
Dtype,
};
pub fn run(input: &Path, work: &Path, threshold: f64)
-> blockflow::error::Result<()>
{
set_process_cap(MemoryCap::resolve(Some(512 << 20)));
let pyramid = OmeVolumePyramid::open(input, 0, 1)?;
let source = pyramid.attached_level(0)?;
let shape = [64, 512, 512];
let grid = BlockGrid::new(shape, [16, 128, 128])?;
let mut builder = PlanBuilder::new(shape, Dtype::U16, grid);
builder.pixels(Chain::op(SmoothOp::isotropic("smooth", 1.0, 3.0)?))?;
builder.pixels(Chain::op(VoxelwiseMaskOp::from_mask(
"threshold", ThresholdMask::above(threshold))))?;
let assembly = builder.finish()?;
let environment = ZarrEnvironment::attach(work, &[source])?
.with_cache(64 << 20);
let hints = Hints { concurrency: 4, ..Hints::default() };
execute_phases("execution-story", &assembly.workflow,
&assembly.decomposition, &hints, &environment,
&[], &assembly.work())?;
Ok(())
}The threshold is in the input intensity scale after smoothing, not an inferred photon count or calibrated biological cutoff. The work path is scratch storage, separate from the input acquisition and the final publication destination.
Check the plan before moving pixels
PlanBuilder::finish builds an Assembly with a Decomposition and phase work. It folds element types, validates source references and checks the geometric contracts. The two phases retain their stated order. BlockGeometry records which region each task owns, reads and can validly write.
execute_phases checks that decomposition and work agree before running them. ZarrEnvironment::prepare also verifies that the attached input has the shape and dtype expected by image zero, creates the later arrays and validates chunk-exclusive primary-image writes. A mismatched acquisition is an error; it is not resized or converted implicitly to make the sketch fit.
Follow smoothing task index 1,1,1
This block’s owned core starts at [16,128,128] and ends at [32,256,256]. Every interval here is half-open. Its shape is [16,128,128], containing 262,144 output voxels. The radius-three Gaussian requires the raw source region [13,125,125] to [35,259,259]: shape [22,134,134], or 395,032 voxels.
This block is interior, so no real-image boundary clips the halo. The trustworthy smoothed region shrinks by three on every face and exactly covers the owned core. At outer image faces, the smoothing boundary convention applies to the true image edge; internal compute seams must use actual neighboring pixels instead.
| Quantity for this smoothing task | Logical size |
|---|---|
| u16 pixels corresponding to the owned core | 524,288 bytes = 0.5 MiB |
| Requested u16 halo-expanded input | 790,064 bytes ≈ 0.753 MiB |
| f64 array over the same halo-expanded extent | 3,160,256 bytes ≈ 3.014 MiB |
| Committed f64 output core | 2,097,152 bytes = 2 MiB |
Those are array payloads, not a complete task-memory estimate. In particular, gaussian_smooth_into_with performs separable axis passes and allocates two f64 scratch arrays over the supplied buffer. The u16 input can use its generic no-copy conversion path into f64 accumulation; that does not eliminate the output or scratch allocations. Decoder workspace, cache entries, temporary copies and allocator overhead must also be accounted for.
A requested region is not a storage chunk
On the selected source’s [8,64,64] chunk grid, the smoothing read intersects z chunk indices 1 through 4, y indices 1 through 4 and x indices 1 through 4. That is 4 × 4 × 4 = 64 storage chunks. Each full selected u16 chunk contains 32,768 values or 65,536 bytes. A cold read therefore intersects 4 MiB of logical decoded chunk data to provide the approximately 0.753 MiB requested region.
ZarrEnvironment::read routes the region through its cached source when one is configured, otherwise through the stored array’s subset reader. The storage backend decodes intersecting chunks as required and copies the requested subregions into the returned buffer. It does not ask the Gaussian kernel to understand compression or five-dimensional acquisition chunk keys.
The 4 MiB is the sum of full decoded chunk payloads touched, not a claim that all 64 chunks are simultaneously live or fetched from disk on every call. Compressed transport bytes depend on the codec and data. Neighboring tasks can reuse retained chunks, and a warm operating-system page cache can save I/O without necessarily saving decode work. Region-byte counters and chunk/cache statistics answer different questions.
Schedule ready work within the resources
All smoothing tasks read an already existing image zero, so they have no producer tasks in this run to wait for. TaskGraph does not create 64 extra producer tasks for the source chunks. Reading and decoding those chunks is work performed while a smoothing task executes.
The hints permit four concurrent tasks, subject to admission. The explicit 512 MiB process cap normally leaves 384 MiB usable after the current 128 MiB reserve; a lower cgroup limit can override the stated cap. The 64 MiB cache capacity is opportunistic retention within the shared process budget, not an additional allocation outside that budget.
For an arithmetic illustration, if each admitted task were charged 16 MiB, four active tasks would lease 64 MiB. If only 40 MiB remained available, two such tasks could fit at once. These charges are illustrative, not measured charges for the Gaussian above. Actual admission uses modeled live buffers, conservative margins, scratch contracts and near-cap observations. The current executor’s lone-oversized-task fallback also means admission is not an allocator- enforced hard RSS limit.
Ready tasks stream into freed slots instead of waiting for a complete batch of four to finish. SchedulePriority::PhaseMajor is the default preference, not a global barrier between these phases. A threshold task becomes eligible when its own required producer finishes, even though priority may favor remaining smoothing work.
Commit the core and unlock its consumer
After smoothing, the executor slices the valid core out of the halo-expanded result and writes image one at the core’s global coordinates. The halo is context, not extra output ownership. Only after the write completes can the producer be marked complete and its dependent threshold task become ready.
The environment does not copy the source chunk grid onto every output. Without an explicit output-layout override, chunk_plan derives writable chunk shapes using each phase’s grid and output dtype. For this exact block, chunk_for_grid gives [16,128,128] for the f64 intermediate and the boolean result: respectively 2 MiB and 256 KiB per full decoded chunk, both inside the current byte-size preference. Different blocks or dtypes can give different answers. Inspect ZarrEnvironment::chunk_at for the layout actually created.
Each primary output chunk has one writing task. That removes the lost-update hazard of simultaneous partial writes to the same compressed chunk. The write path also invalidates cached views around a write so a later reader cannot keep a stale unwritten version of an intermediate.
Threshold task [1,1,1] has zero reach, so its read is exactly its core. On this unchanged grid it depends on the corresponding smoothing task alone—not all 26 neighboring smoothing tasks and not the entire phase. Its f64 input is one intermediate chunk, it evaluates the chosen comparison, and it commits a boolean core to image two.
Finish, release and publish
The whole f64 intermediate has 16,777,216 voxels: 128 MiB of decoded payload. The boolean result has 16 MiB. These arrays can live on disk while the process holds only active buffers and retained chunks. Internal image one can be released once its final reader finishes; Hints::keep_images retains it when diagnostics or later publication explicitly need it.
The terminal work array is not yet a complete OME image group. Publishing must preserve the selected spatial axes, spacing, origin and units and choose a multiscale policy appropriate for a boolean mask. It must not relabel this mask as a uint64 instance-label image merely because the label-publication helper expects that type. Processing completion and publication completion are separate checks.
Compare a composed phase
Placing smoothing and thresholding in one Chain::sequence yields one phase with 64 tasks, no stored f64 intermediate and a boolean final image. The threshold adds no spatial reach, so the required raw halo stays three. The Gaussian still needs its local f64 result and scratch arrays; composition is not compilation into one pixel loop.
This removes the full intermediate write and later read: nominally 128 MiB of f64 payload in each direction across the volume, before compression and caching. It changes task boundaries, intermediate lifetime and block-buffer composition. For two finite-neighborhood kernels, unlike this pointwise second step, the combined raw halo generally grows by their sequential support.
Neither layout is chosen automatically by the two builder calls above. A caller can request priced partitioning through PlanBuilder::partition, or compare explicit alternatives. The same scientific output is the correctness control; traffic, scratch, cache reuse and release-build measurements determine which execution choice is better on the actual workload.
Continue with halos, scheduling, memory and storage and fusion for the contracts behind each step of this example.