Batching processes a large dataset in small subsets instead of loading everything into memory at once. It enables out-of-core learning when data remains in storage such as a data warehouse. section 4.3.5
Linear and logistic regression models can estimate coefficients from one batch, then update them with successive batches. With enough data, the final coefficients may closely resemble a full-memory fit. section 4.3.5
Choose batch reuse and size empirically:
- Use additional unseen batches to extend training, accepting storage-access latency and longer processing. section 4.3.5
- Reuse batches when more passes are necessary, but monitor overfitting from repeatedly seeing identical examples. section 4.3.5
Listing 4.10 Out-of-core logistic regression with L2 regularization
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split
from sklearn.utils import gen_batches
from sklearn.metrics import accuracy_score
def generate_batches(X,
y,
batch_size,
random_state): #1
"""split dataset into batches """
examples = len(X)
batches = gen_batches(n=examples,
batch_size=batch_size) #2
sequence = np.arange(examples)
if random_state:
np.random.seed(random_state) #3
np.random.shuffle(sequence)
for batch in batches:
items = sequence[batch]
yield(X.iloc[items], y.iloc[items]) #4
model = SGDClassifier(loss="log_loss",
average=True,
penalty='l2',
alpha=0.001) #5
column_transform = ColumnTransformer(
[('categories', categorical_onehot_encoding, low_card_categorical),
('numeric', numeric_standardization, continuous)],
remainder='drop',
verbose_feature_names_out=False,
sparse_threshold=0.0)
X_train, X_test, y_train, y_test = train_test_split(data, target_median,
test_size=0.20,
random_state=0) #6
iterations = 10
for j in range(iterations): #7
generator = generate_batches(X_train, y_train, batch_size=256,
random_state=j)
for k, (Xt, yt) in enumerate(generator):
if k == 0:
column_transform.fit(Xt)
Xt = column_transform.transform(Xt)
if k == 0:
model.partial_fit(Xt, yt, classes=(0, 1)) #8
else:
model.partial_fit(Xt, yt) #9
predictions = model.predict(column_transform.transform(X_test))
score = accuracy_score(y_true=y_test, y_pred=predictions)
print(f"Accuracy on test set: {score:0.3f}") #10
A runnable scikit-learn listing shuffles tabular examples, generates batches, and feeds them incrementally to an SGD classifier. section 4.3.5
