A parameterized function is a known function family whose behavior depends on adjustable values, such as weights and a bias. In machine learning, it approximates an unknown function: the architecture is chosen first, then training estimates parameter values that make predictions match observed outputs. section 3.5.1 section 1.3.3
For a linear model, the parameterized function is ϕ(x; w, b) = wᵀx + b = Σⱼwⱼxⱼ + b. The input x is fixed for a prediction; changing w or b changes the output and, geometrically, the decision boundary. Training searches for values that reduce prediction error across the data. section 3.1.3 section 3.5.1
A parameterized model separates two decisions:
- Choose the function family, such as a linear or nonlinear model, based on the problem. section 3.1.3
- Estimate its parameters from input-output pairs by repeatedly comparing predictions with known outputs and reducing aggregate error. section 1.3.5 section 3.5.1
Learning a line
Suppose observations are generated approximately by y = 1.5x + 2.73. Use ϕ(x; w, b) = wx + b, compute mean squared error, calculate gradients for w and b, and update both parameters repeatedly. The learned function can approach y_pred = 1.50059x + 2.746823. section 3.5.1
Listing 3.1 PyTorch linear model (closed-form gradient formula needed)
x = 10 * torch.randn(N) #1
= 1.5 * x + 2.73 _obs = y + (0.5 * torch.randn(N)) #2
for step in range(num_steps):
y_pred = w*x + b #3
mean_squared_error = torch.mean(
(y_pred - y_obs) ** 2) #4
w_grad = torch.mean(2 * ((y_pred - y_obs)* x))
b_grad = torch.mean(2 * (y_pred - y_obs)) #5
w = w - learning_rate * w_grad
b = b - learning_rate * b_grad #6
print("True function: y = 1.5*x + 2.73")
print("Learned function: y_pred = {}*x + {}".format(w[0], b[0]))
Inspect the PyTorch implementation linking the equation, squared-error objective, gradients, and iterative parameter updates. section 3.5.1
