When a tabular dataset exceeds available memory, out-of-core learning trains a model from external storage in small portions called batches. The model updates its coefficients batch by batch instead of loading the complete dataset at once. This makes classical approaches such as linear and logistic regression usable for very large datasets, although storage latency and repeated processing can make training slow. section 4.3.5
A practical out-of-core workflow balances memory, convergence, and generalization:
- Keep the dataset in a warehouse or other storage system, extracting only the samples needed for each batch. section 4.3.5
- Estimate model coefficients on an initial batch, then update them with successive batches rather than restarting training. section 4.3.5
- Experiment with batch counts and repeated passes: unseen batches extend training, while reusing batches can improve learning but may overfit. section 4.3.5
- Consider online learning when examples arrive individually, and compare the approach with gradient boosting or deep learning when scale or data structure makes them competitive. section 4.3.6
Advertising interaction data
An advertising system may need to estimate many fixed interactions among display devices, websites, and advertisements. Instead of fitting every interaction simultaneously in memory, an out-of-core model can learn from successive batches. This is a case where the strategy may be preferable even when other algorithms are available. section 4.3.6
Listing 4.10 Out-of-core logistic regression with L2 regularization
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split
from sklearn.utils import gen_batches
from sklearn.metrics import accuracy_score
def generate_batches(X,
y,
batch_size,
random_state): #1
"""split dataset into batches """
examples = len(X)
batches = gen_batches(n=examples,
batch_size=batch_size) #2
sequence = np.arange(examples)
if random_state:
np.random.seed(random_state) #3
np.random.shuffle(sequence)
for batch in batches:
items = sequence[batch]
yield(X.iloc[items], y.iloc[items]) #4
model = SGDClassifier(loss="log_loss",
average=True,
penalty='l2',
alpha=0.001) #5
column_transform = ColumnTransformer(
[('categories', categorical_onehot_encoding, low_card_categorical),
('numeric', numeric_standardization, continuous)],
remainder='drop',
verbose_feature_names_out=False,
sparse_threshold=0.0)
X_train, X_test, y_train, y_test = train_test_split(data, target_median,
test_size=0.20,
random_state=0) #6
iterations = 10
for j in range(iterations): #7
generator = generate_batches(X_train, y_train, batch_size=256,
random_state=j)
for k, (Xt, yt) in enumerate(generator):
if k == 0:
column_transform.fit(Xt)
Xt = column_transform.transform(Xt)
if k == 0:
model.partial_fit(Xt, yt, classes=(0, 1)) #8
else:
model.partial_fit(Xt, yt) #9
predictions = model.predict(column_transform.transform(X_test))
score = accuracy_score(y_true=y_test, y_pred=predictions)
print(f"Accuracy on test set: {score:0.3f}") #10
A runnable scikit-learn listing demonstrates shuffled batch generation and out-of-core logistic regression with L2 regularization, followed by test-set accuracy. section 4.3.5
