API Reference
This section describes all available functions of this package.
Public API
DiskArrayEngine.GMDWop Type
GMDWop{N,I,O,F<:UserOp,SPL}A struct representing a generalized multi-dimensional windowed operation.
Fields
inars::I: Tuple or collection of input arrays, each wrapped asInputArray.outspecs::O: Tuple or collection of output array specifications.f::F: The user-supplied operation, typically a UserFunc returned bycreate_userfunction.windowsize::NTuple{N,Int}: The number of windows along each loop dimension.lspl::SPL: Loop splitter object ornothing, used for parallelization or block processing.
Description
GMDWop encapsulates all information required to perform a lazy windowed operation over multiple input arrays and produce one or more outputs. It manages the mapping between logical loop dimensions and physical array dimensions, window sizes, and any loop splitting for efficient computation.
Construct using GMDWop(inars, outspecs, f), where inars and outspecs are collections of input and output specifications, and f is the user operation.
DiskArrayEngine.interpolate_diskarray Method
interpolate_diskarray(a, conv)Function to interpolate a diskarray along one or more dimensions. The interpolation is specifed by the list conv consisting of pairs of dim_index => (source_axis,dest_axis) values. For example, to interpolate an input array with dimensions (x,y,t) to new coordinates (x2,y2,t2) you can do
#Old coordinates
a = [i+j+k for i in 1:4, j in 1:5, k in 1:6]
#source coordinates
x = 5.0:5.0:20.0
y = 2.0:3.0:14.0
#target coordinates
x2 = 5.0:0.5:20.0
y2 = 1.5:1.0:14.5
r = interpolate_diskarray(a,(1=>(x,x2),2=>(y,y2)))Internal API
DiskArrayEngine.EmptyInput Type
struct EmptyInputA kind of placeholder array generating an artificial input in the DAG graph, which only gets filled during the computation. Mainly used to reserve an input buffer.
sourceDiskArrayEngine.LoopWindows Type
Struct specifying the windows of a participating array along each dimension as well as the loop axes where this array participates in the loop
sourceDiskArrayEngine.ProductArray Type
Type very similar to Iterators.product, but for indexable arrays.
DiskArrayEngine.apparent_chunksize Method
Given the loop windows of an input array estimate the apparent chunks along an axis given the underlying chunks.
sourceDiskArrayEngine.bufferrepeat Method
Compute how often a buffer needs to be passed to the computation before it can be flushed to the output array
sourceDiskArrayEngine.clean_aggregator Method
Removes all outout buffers from an output aggregator that have been successfully put to disk
sourceDiskArrayEngine.domain_from_chunktype Method
Returns the full domain that a DiskArrays.ChunkVector object covers as a unit range
DiskArrayEngine.fix_output_overlap Method
If one of the outputs is a reduction it is important not to have overlapping loop ranges for a reduction group. This will try to correct loopranges to avoid the problems mentioned above.
sourceDiskArrayEngine.generate_outbuffers Method
Creates buffers for all outputs, results in a tuple of Dicts holding the collection for each output
sourceDiskArrayEngine.getbufsize Method
Determine the needed buffer size for a given Input array and loop ranges lr
sourceDiskArrayEngine.length_from_chunktype Method
Returns the length of a dimension covered by a DiskArrays.ChunkVector object
DiskArrayEngine.mustwrite Method
Check if maximum number of aggregations has happened for a buffer
sourceDiskArrayEngine.put_buffer Method
Checks if output buffers have accumulated to the end and exports to output array
sourceDiskArrayEngine.put_buffer Method
Function to finalize and remove buffers without writing to an array
sourceDiskArrayEngine.range_from_parentchunks Method
Tests that a supplied list of parent chunks covers the same domain and returns this
source