A chunked lazy sequence processes a group of elements when that group is needed, rather than realizing each element separately. This can improve sequential processing, but requesting one element may also evaluate others in its chunk. Ranges and vectors are common sources of chunked sequences. section 10.12 section 10.1
One look ahead can realize a whole chunk
After dropping 31 elements from a mapped range, `rest` causes 32 mapped elements to be evaluated in total; `next` causes 64. Both reach the chunk boundary, but `next` must establish whether another element exists, realizing the next chunk. With the non-chunked input in the same comparison, the totals are 32 and 33. section 10.1
The chunk API separates construction from traversal:
- `chunk-buffer` creates a buffer; `chunk-append` fills it; `chunk` turns its contents into a chunk; and `chunk-cons` joins that chunk to a sequential remainder. Treat the buffer as an internal, single-use artifact: appending after `chunk` throws. section 10.12
- `chunked-seq?` checks whether chunk operations apply. `chunk-first`, `chunk-rest`, and `chunk-next` traverse chunked sequences; using them on a non-chunked sequence throws a `ClassCastException`. section 10.12
Construct one chunk
`(def b (chunk-buffer 10))`, followed by `(chunk-append b 0)`, `(chunk-append b 1)`, and `(chunk-cons (chunk b) ())`, produces `(0 1)`. Capacity is available space, not a requirement to fill the buffer. section 10.12
(defn map-chunked [f coll]
(lazy-seq
(when-let [s (seq coll)] ; #1
(let [cf (chunk-first s) ; #2
b (chunk-buffer (count cf))] ; #3
(.reduce cf (fn [b x] (chunk-append b (f x)) b) b) ; #4
(chunk-cons (chunk b) (map-chunked f (chunk-rest s))))))) ; #5
(take 10 (map-chunked inc (range 10000)))
;; (1 2 3 4 5 6 7 8 9 10)
A chunked mapping function transforms every item in the current chunk into a new buffer, then lazily recurs on `chunk-rest`. section 10.12
Chunk size also matters with `pmap`: realizing a chunk can start work beyond the requested item. Most chunked inputs here use chunks of 32, while a non-chunked input limits that look-ahead differently. Larger custom chunks may create too many concurrent requests; benchmark before treating chunking or parallel mapping as a speedup. section 9.1.5
