psort sorts data too large to keep in memory by splitting the input into chunks, sorting those chunks in parallel, writing them to disk, and lazily merging them into one ordered sequence. section 8.3.5
Its operation has two practical phases:
- Partition an id-range, fetch each portion, sort it, and save the ordered chunk externally. section 8.3.5
- Compare the first value from each chunk, emit the winner, and lazily load subsequent values as needed. section 8.3.5
- Accept either the default compare comparator or a supplied comparator. section 8.3.5
Consume only the beginning
(take 10 (psort (IdRange. 0 10000))) ;; (0 1 2 3 4 5 6 7 8 9) The result exposes an ordered sequence while avoiding a full in-memory realization unless the caller requests it. section 8.3.5
(defn sort-all ; #1
([colls]
(sort-all compare colls))
([cmp colls]
(lazy-seq
(if (some identity (map first colls))
(let [[[win & lose] & xs] (sort-by first cmp colls)] ; #2
(cons win (sort-all cmp (if lose (conj xs lose) xs))))))))
(defn- load-chunk [fname]
(read-string (slurp fname)))
(defn psort ; #3
([id-range]
(psort compare id-range))
([cmp id-range]
(->> (r/fold 10000 concat (partial sort cmp) id-range)
(map load-chunk)
(sort-all cmp))))
(take 10 (psort (IdRange. 0 10000))) ; #4
;; (0 1 2 3 4 5 6 7 8 9)
Browse the implementation of sort-all, chunk loading, psort arities, and the lazy merge example. section 8.3.5
