Mean absolute error (MAE) is the average absolute difference between predicted and actual values. Because it uses the data’s units, its score is directly interpretable: lower values indicate smaller average errors. section 2.7
MAE is useful for comparing forecasting approaches because it is robust to outliers and penalizes all error sizes equally. Pairing it with sMAPE provides a broader evaluation rather than relying on one score. section 2.7 section 6.3.2
A practical MAE evaluation workflow is:
- Reserve historical observations as a test set so predictions can be compared with known actual values. section 3.3.3
- Align predictions and actual values in one DataFrame, preserving the relevant time and series identifiers. section 2.7 section 3.3.3
- Calculate MAE for each model with utilsforecast, then compare scores alongside sMAPE. section 2.7 section 10.2.9
- Interpret the score in the target’s units and avoid treating it as sufficient evidence without considering another metric or validation design. section 3.3.3 section 6.3.2
Comparing forecasting approaches
On monthly data, pretrained N-BEATS achieved MAE 1.59, compared with 1.99 after fine-tuning and 1.90 for a data-specific model. On daily data, the data-specific model achieved 1.34 versus 2.59 for zero-shot forecasting, showing that frequency affects model performance. section 2.7 section 2.8
Listing 2.13 Evaluating the performance of each method
from utilsforecast.losses import mae, smape
from utilsforecast.evaluation import evaluate
evaluation = evaluate(
test_df,
metrics=[mae, smape], #1
models=["NBEATS_pretrained", "NBEATS_finetuned",
➥ "NBEATS_trained"], #2
target_col="y", #3
)
evaluation = evaluation.drop(['unique_id'], axis=1)
evaluation = evaluation.set_index('metric')
evaluation
Run MAE and sMAPE together with utilsforecast.evaluate after combining each model’s predictions with the target values. section 2.7
