Results

Created Models

When creating an experiment, Studio tries to determine all possible situations that might benefit from the use of a specific model allowing the creation of simple models for simple situations or sophiscated models for complex scenarios. The identification and exploitaiton of the similiarites that arise between these sitations then results in reduction of complexity, leaving on the important, and useful details.

Backtesting Results

Backtesting plays a crucial role in evaluating how well the generated model performs within real-world applications. The model is built on a subset of the dataset, the Training set, and then is tested on the Validation set. Therefore, the created model is validating its accuracy based on previous historical data to make sure it can accurately forecast for unseen data - i.e. the forecasting horizion.

Backtesting Results

Predictor Importance

The Predictor Importance plot highlights the relative contribution of each predictor to overall model. This score is generated by aggregating its influence across all of the individual models created in the experiment offering a high-level of detail into which factors are the most influential. The image below is an example of a Predictor Importance plot.

Predictor Importance

Feature Importance

The Feature Importance plot extends upon the Predictor Importance by showing how each feature has contributed to the overall model. This showcases which features are the most important, and emphasises the importance of specific engineered transformations allowing for a micro-level view showing the specific temporal or statistical relationships that has informed the forecast result.

Feature Importance

Production Forecast

The Production Forecast provides a visual representation of the predicted values for your selected target variable over the specified time horizon. This section allows you to quickly understand the expected behavior of the target and gives insight into future trends.

Production Forecast

Model Metrics

Model metrics are calculated to evaluate the accuracy, reliability, and overall performance of your experiment. They provide quantitative measures of how closely the model’s predictions align with actual observed values, helping you understand both the magnitude and direction of errors with different metrics highlighting different aspects of the experiments performance. By reviewing these metrics, you can compare, identify areas for improvement, and make informed decisions about model parameters. Using multiple complementary metrics ensures a more complete and robust evaluation of model performance. By default, Studio uses RMSE, MAE, and MAPE as the chosen metrics but more metrics can be toggled on or off via the Select Metrics dropdown.

Model Metrics

The table below showcases all of the available metrics offered in Studio, with a guide on how to interpret the metrics.

Metric

How to Interpret It

MAE

Lower MAE indicates more accurate predictions. Due to it using absolute errors, values are easy to interpret in the same units as the target variable.

MASE

Values below one mean the model outperforms a naïve forecast; values above one mean it performs worse. Useful for comparing across series.

MAPE

Lower is better. Represents average error as a percentage. When actual values approach zero, be cautious, as percentages may become unstable.

MBE (Bias)

A value near zero indicates no systematic bias. Positive means consistent over-prediction; negative means consistent under-prediction.

MdAE

Lower is better. Robust to outliers and useful when the error distribution is skewed or contains extreme values.

RMSE

Lower is better. Strongly penalizes larger errors, making it useful when large mistakes are more important to avoid.

RMSSE

Values below one indicate performance better than a naïve scaled baseline. Like RMSE, it emphasizes large errors but supports cross-series comparisons.

SMAPE

Lower is better. A symmetric percentage-based measure, though it may behave inconsistently when values are close to zero.