Out-of-core computing keeps large-data work practical when the dataset exceeds available RAM. Instead of loading everything at once, it processes chunks stored on disk, reducing the memory barrier for tabular analysis and model training. section 2.3.1 section 5.4
The technique is a scaling strategy rather than a single algorithm. Dask extends the pandas DataFrame API for out-of-core work and can run across processors or clusters, while Vaex provides lazy out-of-core DataFrames for exploration. section 2.3.1
Choose an implementation according to the workload and deployment constraints:
- Use Dask when pandas-like operations must handle many rows through multiprocessing or cluster execution. section 2.3.1
- Use Vaex for lazy exploration and visualization of large tabular datasets on a workstation or server. section 2.3.1
- Use XGBoost when model training must support out-of-core processing alongside multicore or distributed execution. section 5.4
- Distinguish GPU acceleration from out-of-core scalability: GPU processing can improve computation time, but GPU memory access remains a constraint. section 2.3.1
Large-scale model training
A classification workflow using XGBClassifier can train on a large dataset that does not fit into memory, while also using multiple CPU cores or distributed training. The technique complements, rather than replaces, choices such as tree depth, estimator count, and evaluation strategy. section 5.4
