Introduction to BOSO: Bilevel Optimization Selector Operator
10 April 2024
Abstract
BOSO is an R package to perform feature selection in linear regression models.
Contents
1 Installation
BOSO can be installed from CRAN repository:
install.packages("BOSO")Note that it is necessary to have IBM ILOG CPLEX installed, with a version greater than 12.1, and the cplexAPI package, which is used to invoke CPLEX from R environment.
Furthermore, check if packages are installed. Check if bestsubset is installed. If not, source necessary functions from bestsubset.
if (!requireNamespace('cplexAPI', quietly = TRUE)) {
testthat::skip('Package cplexAPI not installed (required for this vignette)!\n
Install it from CRAN: https://cran.r-project.org/web/packages/cplexAPI/index.html')
stop('Package cplexAPI not installed (required for this vignette)!\n
Install it from CRAN: https://cran.r-project.org/web/packages/cplexAPI/index.html', call. = FALSE)
}if (!requireNamespace('bestsubset', quietly = TRUE)) {
devtools::source_url("https://raw.githubusercontent.com/ryantibs/best-subset/master/bestsubset/R/fs.R")
devtools::source_url("https://raw.githubusercontent.com/ryantibs/best-subset/master/bestsubset/R/lasso.R")
devtools::source_url("https://raw.githubusercontent.com/ryantibs/best-subset/master/bestsubset/R/sim.R")
enlist <- function (...)
{
result <- list(...)
if ((nargs() == 1) & is.character(n <- result[[1]])) {
result <- as.list(seq(n))
names(result) <- n
for (i in n) result[[i]] <- get(i)
} else {
n <- sys.call()
n <- as.character(n)[-1]
if (!is.null(n2 <- names(result))) {
which <- n2 != ""
n[which] <- n2[which]
}
names(result) <- n
}
result
}
} else {require("bestsubset")}## ℹ SHA-1 hash of file is "cc12499ddd2cc538f318f3ee14ad4a3911945a27"
## ℹ SHA-1 hash of file is "e7c4bf157a3c33b0a11179541be100c0037a4fea"
## ℹ SHA-1 hash of file is "531fcc6ad06024eae37afaa3e15d83e0ea3ece89"2 Introduction
The package is designed to run the BOSO algorithm in one single function, which comprises two steps: block strategy and final problem.
The block strategy is a pre-processing step in which the BOSO MILP (Mixed-Integer Liner Programming) is applied to random subproblems of small dimension, aiming to discard variables for the final problem. The usual block size is 10 variables. This process is done iteratively until convergence.
Final problem applies the BOSO MILP with all the remaining variables from the block strategy.
Figure 1 summarizes the BOSO algorithm. An example dataset with 7 features is split into training and validation sets. Green boxes represent optimal selected features for a Specific K value. For example, for K=2, the subset of features that minimizes the validation error is {X3, X6}. The problem of selecting the best subset of features of length K is formulated via mixed-integer quadratic programming (MIQP) and solved using IBM ILOG CPLEX This process is repeated for each K value until an information criterion, in this case the extended Bayesian Information Criterion (eBIC), is not further improved. Minimal eBIC is found in this example for K=2. The final model is derived from Ridge regression with only these two selected variables.
3 Different parameters that can be changed in the BOSO algortihm.
Function BOSO has a number of parameters that can be changed:
- IC is the information criterion to be used. Options are AIC, BIC and eBIC
- IC.blocks is the information criterion to be used in the block strategy step. By default, it is the same as the IC, but it can be changed. Additionally, if IC is eBIC, IC.blocks will be BIC.
- nlambda The number of lambda values in the final problem. Default value 100.
- nlambda.blocks The number of lambda values in the blocks strategy. Default value 10, in order to alleviate the block strategy.
- TH_IC Threshold for the stopping criterion in K selection. Default value 1e-3. We have a worse model if the IC is at least 0.1% higher than the current solution.
On the other hand, there are some parameters that are identical to the ones found in the glmnet function: lambda.min.ratio, lambda, intercept, standardize and dfmax.
Finally, some of the parameters are solver-specific:
- Threads CPLEX parameter that defines the number of cores to be used. Default is 0 (automatic).
- timeLimit CPLEX parameter that defines the time limit per problem provided to CPLEX.Default value is 1e75 (infinite time).
- warmstart warmstart for CPLEX or use a different problem for each k. Default is False.
4 Example: High-5 scenario discussed in the BOSO article
Given a synthetic data set (see Methods section in the article), we evaluate the capacity of BOSO to extract the features that are related with the response variable. First o, we set the parameters for the generation of synthetic data:
# Set some overall simulation parameters
n = 100; # Size of training set
nval = n # Size of validation set
p = 10; # Number of variables
s = 5 # Number of nonzero coefficients
beta.type = 1 # Type of coefficiente structure
rho.vec = 0.35 # Variable autocorrelation level
snr.vec = exp(seq(log(0.05),log(6),length=10)) # Signal-to-noise ratios
nrep = 10 # Number of repetitions for a given setting
seed = 0 # Random number generator seedBenchmarked methods are defined: Forward Stepwise, Lasso, Relaxed lasso and BOSO.
# Regression functions: lasso, forward stepwise and BOSO
reg.funs = list()
reg.funs[["Forward stepwise"]] = function(x,y) fs(x,y,intercept=FALSE, max=min(n,p))
reg.funs[["Lasso"]] = function(x,y) lasso(x,y,intercept=FALSE,nlam=50)
reg.funs[["Relaxed lasso"]] = function(x,y) lasso(x,y,intercept=FALSE,
nrelax=10,nlam=50)
reg.funs[["BOSO - AIC"]] <- function(x,y,xval,yval) BOSO(x,y,xval,yval,
IC = "AIC",
nlambda = 100,
Threads = 4,
timeLimit = 60,
intercept = F,
standardize = F,
verbose = 0,
warmstart=T,
seed = 123456)
reg.funs[["BOSO - BIC"]] <- function(x,y,xval,yval) BOSO(x,y,xval,yval,
IC = "BIC",
nlambda = 100,
Threads = 4,
timeLimit = 60,
intercept = F,
standardize = F,
verbose = 0,
warmstart=T,
seed = 123456)
reg.funs[["BOSO - eBIC"]] <- function(x,y,xval,yval) BOSO(x,y,xval,yval,
IC = "eBIC",
nlambda = 100,
Threads = 4,
timeLimit = 60,
intercept = F,
standardize = F,
verbose = 0,
warmstart=T,
seed = 123456)We have adapted the sim.masterfunction from the bestsubset library function to assess the different methods.
sim.results <- list()
for (i in 1:length(snr.vec)) {
set.seed(i)
sim.results[[i]] <- sim.master(n = n, p = p, nval = nval,
rho=rho.vec, s=s, beta.type=beta.type,
snr = snr.vec[[i]],
reg.funs, nrep=nrep, seed=i,
verbose=T)
}4.1 Comparison of BOSO with other methods
Finally, the next part of the code shows the comparison of BOSO-eBIC with some of the most widely used algorithms in the literature: Lasso, Relaxed lasso and Forward Stepwise.
method.nums = c(6,1,2,3)
method.names = c("BOSO - eBIC","Forward stepwise","Lasso","Relaxed lasso")
plot.BOSO <- list(
"F" = plot.from.sim(sim.list = sim.results, what="F", rel.to=NULL, tuning="val",
method.nums=method.nums, method.names=method.names),
"nonzeros" = plot.from.sim(sim.list = sim.results, what="nonzero", rel.to=NULL, tuning="val",
method.nums=method.nums, method.names=method.names),
"fpos" = plot.from.sim(sim.list = sim.results, what="fpos", rel.to=NULL, tuning="val",
method.nums=method.nums, method.names=method.names),
"fneg" = plot.from.sim(sim.list = sim.results, what="fneg", rel.to=NULL, tuning="val",
method.nums=method.nums, method.names=method.names))## Coordinate system already present. Adding new coordinate system, which will
## replace the existing one.
## Coordinate system already present. Adding new coordinate system, which will
## replace the existing one.
## Coordinate system already present. Adding new coordinate system, which will
## replace the existing one.
## Coordinate system already present. Adding new coordinate system, which will
## replace the existing one.ggarrange(plotlist = plot.BOSO, ncol = 2, nrow = 2, common.legend = T, legend = "bottom")
4.2 Comparison of BOSO with different information criteria
The following part of code is designed to compare the performance of the BOSO algorithm using different information criteria. The options available are AIC, BIC and eBIC.
method.nums = c(4, 5, 6)
method.names = c("BOSO - AIC","BOSO - BIC","BOSO - eBIC")
plot.BOSO <- list(
"F" = plot.from.sim(sim.list = sim.results, what="F", rel.to=NULL, tuning="val",
method.nums=method.nums, method.names=method.names),
"nonzeros" = plot.from.sim(sim.list = sim.results, what="nonzero", rel.to=NULL, tuning="val",
method.nums=method.nums, method.names=method.names),
"fpos" = plot.from.sim(sim.list = sim.results, what="fpos", rel.to=NULL, tuning="val",
method.nums=method.nums, method.names=method.names),
"fneg" = plot.from.sim(sim.list = sim.results, what="fneg", rel.to=NULL, tuning="val",
method.nums=method.nums, method.names=method.names))## Coordinate system already present. Adding new coordinate system, which will
## replace the existing one.
## Coordinate system already present. Adding new coordinate system, which will
## replace the existing one.
## Coordinate system already present. Adding new coordinate system, which will
## replace the existing one.
## Coordinate system already present. Adding new coordinate system, which will
## replace the existing one.ggarrange(plotlist = plot.BOSO, ncol = 2, nrow = 2, common.legend = T, legend = "bottom")
5 References
Appendix
Luis V. Valcarcel, Edurne San José-Enériz, Xabier Cendoya, Ángel Rubio, Xabier Agirre, Felipe Prósper, Francisco J. Planes ‘BOSO: a novel feature selection algorithm for linear regression with high-dimensional data.’ submitted.
Hastie, Trevor, Robert Tibshirani, and Ryan Tibshirani. “Best Subset, Forward Stepwise or Lasso? Analysis and Recommendations Based on Extensive Comparisons.” Statistical Science 35.4 (2020): 579-592.
A Session Information
sessionInfo()## R version 4.3.3 (2024-02-29)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 22.04.4 LTS
##
## Matrix products: default
## BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so; LAPACK version 3.10.0
##
## locale:
## [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
## [4] LC_COLLATE=C LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
## [7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
## [10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
##
## time zone: Europe/Madrid
## tzcode source: system (glibc)
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] ggpubr_0.6.0 ggplot2_3.5.0 glmnet_4.1-8 Matrix_1.6-5
## [5] kableExtra_1.4.0 dplyr_1.1.4 BOSO_1.0.4 knitr_1.46
## [9] BiocStyle_2.30.0
##
## loaded via a namespace (and not attached):
## [1] tidyselect_1.2.1 viridisLite_0.4.2 farver_2.1.1
## [4] fastmap_1.1.1 promises_1.3.0 digest_0.6.35
## [7] mime_0.12 lifecycle_1.0.4 ellipsis_0.3.2
## [10] survival_3.5-8 magrittr_2.0.3 compiler_4.3.3
## [13] rlang_1.1.3 sass_0.4.9 tools_4.3.3
## [16] utf8_1.2.4 yaml_2.3.8 ggsignif_0.6.4
## [19] labeling_0.4.3 htmlwidgets_1.6.4 pkgbuild_1.4.4
## [22] curl_5.2.1 xml2_1.3.6 pkgload_1.3.4
## [25] abind_1.4-5 miniUI_0.1.1.1 withr_3.0.0
## [28] purrr_1.0.2 grid_4.3.3 fansi_1.0.6
## [31] urlchecker_1.0.1 profvis_0.3.8 xtable_1.8-4
## [34] colorspace_2.1-0 scales_1.3.0 iterators_1.0.14
## [37] tinytex_0.50 cli_3.6.2 rmarkdown_2.26
## [40] generics_0.1.3 remotes_2.5.0 rstudioapi_0.16.0
## [43] httr_1.4.7 sessioninfo_1.2.2 cachem_1.0.8
## [46] stringr_1.5.1 splines_4.3.3 BiocManager_1.30.22
## [49] vctrs_0.6.5 devtools_2.4.5 jsonlite_1.8.8
## [52] carData_3.0-5 bookdown_0.38 car_3.1-2
## [55] rstatix_0.7.2 systemfonts_1.0.6 foreach_1.5.2
## [58] tidyr_1.3.1 jquerylib_0.1.4 glue_1.7.0
## [61] codetools_0.2-20 cowplot_1.1.3 stringi_1.8.3
## [64] shape_1.4.6.1 gtable_0.3.4 later_1.3.2
## [67] munsell_0.5.1 tibble_3.2.1 pillar_1.9.0
## [70] htmltools_0.5.8.1 R6_2.5.1 evaluate_0.23
## [73] shiny_1.8.1.1 lattice_0.22-6 highr_0.10
## [76] backports_1.4.1 memoise_2.0.1 broom_1.0.5
## [79] httpuv_1.6.15 bslib_0.7.0 Rcpp_1.0.12
## [82] gridExtra_2.3 svglite_2.1.3 xfun_0.43
## [85] fs_1.6.3 usethis_2.2.3 pkgconfig_2.0.3