Symmetric mean absolute percentage error (sMAPE) measures forecast error as a percentage relative to predicted and actual values. Unlike mean absolute percentage error, it treats overforecasting and underforecasting symmetrically, making results easy to compare across forecasting approaches. section 2.7
Use sMAPE alongside mean absolute error rather than alone. sMAPE is percentage-based and easier to interpret when the data has a large scale, while MAE remains expressed in the original units. Avoid sMAPE when actual or predicted values are close or equal to zero. section 2.7 section 3.3.3
A practical sMAPE evaluation workflow is:
- Reserve a test set or create cross-validation windows containing actual future values. section 3.3.3 section 6.3.2
- Align each model’s predictions with the actual target values in one DataFrame. section 2.7
- Calculate sMAPE with MAE, then compare models across both metrics; agreement provides stronger evidence for choosing a method. section 2.7 section 10.2.9
Comparing forecasting approaches
On a monthly dataset, pretrained N-BEATS produced an sMAPE of 3.56%, compared with 4.63% after fine-tuning and 4.23% for a data-specific model. On daily data, the data-specific model achieved 4.87% versus 8.69% for zero-shot forecasting, illustrating that frequency affects comparative performance. section 2.7 section 2.8
Listing 2.13 Evaluating the performance of each method
from utilsforecast.losses import mae, smape
from utilsforecast.evaluation import evaluate
evaluation = evaluate(
test_df,
metrics=[mae, smape], #1
models=["NBEATS_pretrained", "NBEATS_finetuned",
➥ "NBEATS_trained"], #2
target_col="y", #3
)
evaluation = evaluation.drop(['unique_id'], axis=1)
evaluation = evaluation.set_index('metric')
evaluation
Evaluate multiple forecast columns with utilsforecast by supplying smape alongside mae, the model names, and the target column. section 2.7
