Stochastic gradient descent (SGD) is an optimization procedure for linear models that repeatedly adjusts model weights using gradients computed from small data batches or individual examples. It extends gradient descent to settings where data arrives in streams and may be seen only once. section 4.3.5
A practical out-of-core workflow typically:
- Reserves a test set because cross-validation can be infeasible when streamed examples are discarded. section 4.3.5
- Splits training data into batches and presents each batch to the optimizer. section 4.3.5
- Starts with guessed weights, measures error, computes its gradient, and updates weights toward lower error. section 4.3.5
Listing 4.10 Out-of-core logistic regression with L2 regularization
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split
from sklearn.utils import gen_batches
from sklearn.metrics import accuracy_score
def generate_batches(X,
y,
batch_size,
random_state): #1
"""split dataset into batches """
examples = len(X)
batches = gen_batches(n=examples,
batch_size=batch_size) #2
sequence = np.arange(examples)
if random_state:
np.random.seed(random_state) #3
np.random.shuffle(sequence)
for batch in batches:
items = sequence[batch]
yield(X.iloc[items], y.iloc[items]) #4
model = SGDClassifier(loss="log_loss",
average=True,
penalty='l2',
alpha=0.001) #5
column_transform = ColumnTransformer(
[('categories', categorical_onehot_encoding, low_card_categorical),
('numeric', numeric_standardization, continuous)],
remainder='drop',
verbose_feature_names_out=False,
sparse_threshold=0.0)
X_train, X_test, y_train, y_test = train_test_split(data, target_median,
test_size=0.20,
random_state=0) #6
iterations = 10
for j in range(iterations): #7
generator = generate_batches(X_train, y_train, batch_size=256,
random_state=j)
for k, (Xt, yt) in enumerate(generator):
if k == 0:
column_transform.fit(Xt)
Xt = column_transform.transform(Xt)
if k == 0:
model.partial_fit(Xt, yt, classes=(0, 1)) #8
else:
model.partial_fit(Xt, yt) #9
predictions = model.predict(column_transform.transform(X_test))
score = accuracy_score(y_true=y_test, y_pred=predictions)
print(f"Accuracy on test set: {score:0.3f}") #10
Python demonstrates shuffled batch generation and out-of-core logistic regression with scikit-learn’s SGDClassifier. section 4.3.5
