Capping Variable Contributions

Harry He

2025-06-17

Purpose of Vignette

This vignette demonstrates how works in practice and how it affects model performance in different contexts. Traditional regularization methods like elastic net penalize coefficients, but in many real-world datasets, variables can have unstable variance or sporadic large magnitudes, meaning that a few predictors can dominate the fitted model. By explicitly capping each variable’s maximum contribution \(|x_j\beta_j|\), provides a way to directly control how much influence each predictor can exert. This vignette compares standard elastic net, the standard model, and a walk-forward model designed for time-series data and greater stability.

Standard Well-Behaved Data

The first example uses quick start example data from . Theoretically, we expect all models to perform similarly as all features are well-behaving and should rarely trigger large contribution penalties in the model.

ds_qs <- prepare_data("quickstart")
fits_qs <- fit_models(ds_qs$train_X, ds_qs$train_y, ds_qs$test_X, ds_qs$L, ds_qs$mu)

qs_enet_is <- mse_from_beta(ds_qs$train_X, ds_qs$train_y, fits_qs$enet$beta0, fits_qs$enet$beta)
qs_cap1_is <- mse_from_beta(ds_qs$train_X, ds_qs$train_y, fits_qs$cap1$beta0, fits_qs$cap1$beta)

qs_enet_oos <- mse_from_beta(ds_qs$test_X, ds_qs$test_y, fits_qs$enet$beta0, fits_qs$enet$beta)
qs_cap1_oos <- mse_from_beta(ds_qs$test_X, ds_qs$test_y, fits_qs$cap1$beta0, fits_qs$cap1$beta)
qs_cap2_oos <- mse_from_pred(fits_qs$cap2$predictions, ds_qs$test_y)

report_qs <- data.frame(
  Model = c("Elan", "Optimized Cap", "Walk Cap"),
  IS = c(qs_enet_is, qs_cap1_is, NA),
  OOS = c(qs_enet_oos, qs_cap1_oos, qs_cap2_oos)
)
kable(report_qs, digits = 6)
Model IS OOS
Elan 0.501179 0.591483
Optimized Cap 0.495869 0.580050
Walk Cap NA 0.537670

The in sample (IS) and out of sample (OOS) mean squared errors (MSE) show that the elastic net model and the standard model have similar IS and OOS performances. The walk forward model achieved better OOS performance, which can be attributed to it being more adapted to each out of sample observation. This comes at higher time complexity and the absence of a single trained model that best capture the relationship between the feature and target variables.

Synthetic Data (Unstable Variance Features)

To illustrate the strength of the model, I construct a synthetic dataset based on the latent data-generating process \(Z\rightarrow Y\). In this setup, the true predictors \(Z\) drive the outcome, but only a noisy proxy \(X\) is observed. Importantly, some variables in \(X\) exhibit unstable and high variance, occasionally taking on extreme values. Under these conditions, both the Optimized Cap and Walk Cap models outperform the standard elastic net, with the walk-forward implementation achieving the greatest improvement in generalization. These results demonstrate how ’s contribution-based constraints effectively limit the influence of high-variance features that might otherwise distort model predictions.

ds_px <- prepare_data("proxy")
fits_px <- fit_models(ds_px$train_X, ds_px$train_y, ds_px$test_X, ds_px$L, ds_px$mu)

px_enet_is <- mse_from_beta(ds_px$train_X, ds_px$train_y, fits_px$enet$beta0, fits_px$enet$beta)
px_cap1_is <- mse_from_beta(ds_px$train_X, ds_px$train_y, fits_px$cap1$beta0, fits_px$cap1$beta)

px_enet_oos <- mse_from_beta(ds_px$test_X, ds_px$test_y, fits_px$enet$beta0, fits_px$enet$beta)
px_cap1_oos <- mse_from_beta(ds_px$test_X, ds_px$test_y, fits_px$cap1$beta0, fits_px$cap1$beta)
px_cap2_oos <- mse_from_pred(fits_px$cap2$predictions, ds_px$test_y)

report_px <- data.frame(
  Model = c("Elan", "Optimized Cap", "Walk Cap"),
  IS = c(px_enet_is, px_cap1_is, NA),
  OOS = c(px_enet_oos, px_cap1_oos, px_cap2_oos)
)
kable(report_px, digits = 6)
Model IS OOS
Elan 0.860357 0.795325
Optimized Cap 0.803383 0.745417
Walk Cap NA 0.681158