A sequence chunk is a group of elements processed together during sequential iteration. Chunking can improve performance by fetching or computing more elements than the caller has requested, then serving the remaining elements from an intermediate buffer. Vectors, vector-of collections, and ranges are among the inputs that benefit. section 10.12 section 9.0
The trade-off is less precise laziness: requesting one element may process others in its chunk. Code that depends on exactly one element being computed per request should not assume that a lazy sequence provides it. section 10.12 section 9.3.8
Construct a small chunk
(def b (chunk-buffer 10)) (chunk-append b 0) (chunk-append b 1) (chunk-append b 2) (chunk-cons (chunk b) ()) ;; (0 1 2) The buffer has capacity for ten elements, but this chunk contains the three appended elements. Once chunk has converted the buffer, appending to that buffer again throws a NullPointerException. section 10.12
For custom chunked lazy sequences:
- Use chunk-buffer, chunk-append, and chunk to assemble a chunk; treat the mutable buffer as an internal processing artifact. section 10.12
- Use chunk-cons to attach it to the remaining sequence. chunk-first, chunk-rest, and chunk-next provide chunk-oriented counterparts to first, rest, and next; chunked-seq? checks support for the abstraction. section 10.12
- Do not confuse chunks with seque: seque places an asynchronous queue between a producer and consumer, rather than setting iteration granularity. section 9.1.4 section 10.12
(defn map-chunked [f coll]
(lazy-seq
(when-let [s (seq coll)] ; #1
(let [cf (chunk-first s) ; #2
b (chunk-buffer (count cf))] ; #3
(.reduce cf (fn [b x] (chunk-append b (f x)) b) b) ; #4
(chunk-cons (chunk b) (map-chunked f (chunk-rest s))))))) ; #5
(take 10 (map-chunked inc (range 10000)))
;; (1 2 3 4 5 6 7 8 9 10)
A chunked map takes the first input chunk, maps its elements into a new buffer, and attaches that chunk to a lazy recursive continuation. Taking ten results from a range yields (1 2 3 4 5 6 7 8 9 10). section 10.12
Chunking depends on the sequence implementation. iterator-seq advances its input iterator in chunks of 32, while enumeration-seq is fully lazy. The chunk-* functions have been public since Clojure 1.1 but were labeled implementation details; use them deliberately when building custom sequences. section 9.3.8 section 10.12
