Simulation based calibration for OncoBayes2

Tue Feb 11 13:10:21 2020

This report documents the results of a simulation based calibration (SBC) run for OncoBayes2. TODO

The calibration data presented here has been generated at and with the OncoBayes git version as:

## Created:  2020-02-11 12:02:23 UTC
## git hash: a9d8c5adc623815556a35a069fa5573a5dd0d25c
## MD5:      c0ea3d87a4e045ba4eec23064260fea6

The MD5 hash of the calibration data file presented here must match the above listed MD5:

##                    calibration.rds 
## "c0ea3d87a4e045ba4eec23064260fea6"

Introduction

Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).

Self-consistency of any Bayesian analysis with a proper prior:

\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]

SBC procedure:

Repeat \(s=1, ..., S\) times:

  1. Sample from the prior \[\tilde{\theta} \sim p(\theta)\]

  2. Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]

  3. Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]

  4. Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]

The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]

Model description TODO

The fake data simulation function returns … TODO. Please refer to the sbc_tools.R and make_reference_rankhist.R R programs for the implementation details.

The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.

SBC results

Model 1: Single-agent logistic regression

Component intercept/slopes

Means

Standard deviations (tau’s)

Component intercept/slopes: group estimates

Group estimates components

Model 2: Double combination, fully exchangeable

Component intercept/slopes: exchangeable mixture component

Means

Standard deviations (tau’s)

Interaction parameters (from exchangeable part)

Mean

Standard deviation

Component intercept/slopes: group estimates

Group estimates components

Group estimates interaction(s)

Model 3: Double combination, EXchangeable/NonEXchangeable model

Component intercept/slopes: exchangeable mixture component

Means

Standard deviations (tau’s)

Interaction parameters (from exchangeable part)

Mean

Standard deviation (tau)

Component intercept/slopes: group estimates

Group estimates components

Group estimates interaction(s)

Model 4: Triple combination, EX/NEX model

Component intercept/slopes: exchangeable mixture component

Means

Standard deviations (tau’s)

Interaction parameters (means from exchangeable part)

Mean

Standard deviation (tau)

Component intercept/slopes: group estimates

Group estimates components

Group estimates interaction(s)

\(\chi^2\) Statistic, Model 1: Single-agent logistic regression

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 21.203 31 0.906
beta_group[A,I(log(drug_A/1)),log_slope] 47.488 31 0.029
beta_group[B,I(log(drug_A/1)),intercept] 20.141 31 0.933
beta_group[B,I(log(drug_A/1)),log_slope] 12.666 31 0.999
beta_group[C,I(log(drug_A/1)),intercept] 40.621 31 0.116
beta_group[C,I(log(drug_A/1)),log_slope] 29.440 31 0.546
mu_log_beta[I(log(drug_A/1)),intercept] 24.154 31 0.804
mu_log_beta[I(log(drug_A/1)),log_slope] 24.166 31 0.804
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] 27.219 31 0.661
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] 50.835 31 0.014

\(\chi^2\) Statistic, Model 2: Double combination, fully exchangeable

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 28.352 31 0.603
beta_group[A,I(log(drug_A/1)),log_slope] 28.992 31 0.570
beta_group[A,I(log(drug_B/1)),intercept] 37.728 31 0.189
beta_group[A,I(log(drug_B/1)),log_slope] 37.715 31 0.189
beta_group[B,I(log(drug_A/1)),intercept] 33.222 31 0.359
beta_group[B,I(log(drug_A/1)),log_slope] 17.267 31 0.978
beta_group[B,I(log(drug_B/1)),intercept] 29.050 31 0.567
beta_group[B,I(log(drug_B/1)),log_slope] 47.040 31 0.032
beta_group[C,I(log(drug_A/1)),intercept] 30.278 31 0.503
beta_group[C,I(log(drug_A/1)),log_slope] 42.950 31 0.075
beta_group[C,I(log(drug_B/1)),intercept] 34.080 31 0.322
beta_group[C,I(log(drug_B/1)),log_slope] 43.891 31 0.062
eta_group[A,I(drug_A/1 * drug_B/1)] 17.107 31 0.979
eta_group[B,I(drug_A/1 * drug_B/1)] 28.275 31 0.607
eta_group[C,I(drug_A/1 * drug_B/1)] 22.528 31 0.866
mu_eta[I(drug_A/1 * drug_B/1)] 43.405 31 0.069
mu_log_beta[I(log(drug_A/1)),intercept] 52.269 31 0.010
mu_log_beta[I(log(drug_A/1)),log_slope] 36.096 31 0.242
mu_log_beta[I(log(drug_B/1)),intercept] 35.130 31 0.279
mu_log_beta[I(log(drug_B/1)),log_slope] 46.720 31 0.035
tau_eta[STRAT,I(drug_A/1 * drug_B/1)] 19.776 31 0.940
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] 30.867 31 0.473
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] 24.288 31 0.799
tau_log_beta[STRAT,I(log(drug_B/1)),intercept] 33.811 31 0.333
tau_log_beta[STRAT,I(log(drug_B/1)),log_slope] 25.146 31 0.761

\(\chi^2\) Statistic, Model 3: Double combination, EXchangeable/NonEXchangeable model

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 31.917 31 0.421
beta_group[A,I(log(drug_A/1)),log_slope] 25.971 31 0.723
beta_group[A,I(log(drug_B/1)),intercept] 32.128 31 0.411
beta_group[A,I(log(drug_B/1)),log_slope] 36.032 31 0.245
beta_group[B,I(log(drug_A/1)),intercept] 38.477 31 0.167
beta_group[B,I(log(drug_A/1)),log_slope] 25.107 31 0.763
beta_group[B,I(log(drug_B/1)),intercept] 19.942 31 0.937
beta_group[B,I(log(drug_B/1)),log_slope] 33.363 31 0.353
beta_group[C,I(log(drug_A/1)),intercept] 21.811 31 0.889
beta_group[C,I(log(drug_A/1)),log_slope] 33.114 31 0.364
beta_group[C,I(log(drug_B/1)),intercept] 28.653 31 0.587
beta_group[C,I(log(drug_B/1)),log_slope] 26.573 31 0.693
eta_group[A,I(drug_A/1 * drug_B/1)] 34.918 31 0.287
eta_group[B,I(drug_A/1 * drug_B/1)] 35.616 31 0.260
eta_group[C,I(drug_A/1 * drug_B/1)] 36.480 31 0.229
mu_eta[I(drug_A/1 * drug_B/1)] 28.307 31 0.605
mu_log_beta[I(log(drug_A/1)),intercept] 36.480 31 0.229
mu_log_beta[I(log(drug_A/1)),log_slope] 30.195 31 0.507
mu_log_beta[I(log(drug_B/1)),intercept] 20.006 31 0.936
mu_log_beta[I(log(drug_B/1)),log_slope] 33.043 31 0.368
tau_eta[STRAT,I(drug_A/1 * drug_B/1)] 27.283 31 0.658
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] 17.690 31 0.973
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] 31.565 31 0.438
tau_log_beta[STRAT,I(log(drug_B/1)),intercept] 29.312 31 0.553
tau_log_beta[STRAT,I(log(drug_B/1)),log_slope] 25.728 31 0.734

\(\chi^2\) Statistic, Model 4: Triple combination, EX/NEX model

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 24.160 31 0.804
beta_group[A,I(log(drug_A/1)),log_slope] 20.141 31 0.933
beta_group[A,I(log(drug_B/1)),intercept] 28.678 31 0.586
beta_group[A,I(log(drug_B/1)),log_slope] 25.235 31 0.757
beta_group[A,I(log(drug_C/1)),intercept] 25.139 31 0.761
beta_group[A,I(log(drug_C/1)),log_slope] 41.638 31 0.096
beta_group[B,I(log(drug_A/1)),intercept] 30.176 31 0.508
beta_group[B,I(log(drug_A/1)),log_slope] 35.552 31 0.262
beta_group[B,I(log(drug_B/1)),intercept] 32.090 31 0.412
beta_group[B,I(log(drug_B/1)),log_slope] 28.461 31 0.597
beta_group[B,I(log(drug_C/1)),intercept] 32.506 31 0.393
beta_group[B,I(log(drug_C/1)),log_slope] 37.050 31 0.210
beta_group[C,I(log(drug_A/1)),intercept] 36.531 31 0.227
beta_group[C,I(log(drug_A/1)),log_slope] 17.152 31 0.979
beta_group[C,I(log(drug_B/1)),intercept] 45.850 31 0.042
beta_group[C,I(log(drug_B/1)),log_slope] 45.901 31 0.041
beta_group[C,I(log(drug_C/1)),intercept] 47.507 31 0.029
beta_group[C,I(log(drug_C/1)),log_slope] 22.496 31 0.867
eta_group[A,I(drug_A/1 * drug_B/1 * drug_C/1)] 30.918 31 0.470
eta_group[A,I(drug_A/1 * drug_B/1)] 19.091 31 0.953
eta_group[A,I(drug_A/1 * drug_C/1)] 31.021 31 0.465
eta_group[A,I(drug_B/1 * drug_C/1)] 39.590 31 0.139
eta_group[B,I(drug_A/1 * drug_B/1 * drug_C/1)] 31.962 31 0.419
eta_group[B,I(drug_A/1 * drug_B/1)] 17.882 31 0.971
eta_group[B,I(drug_A/1 * drug_C/1)] 39.878 31 0.132
eta_group[B,I(drug_B/1 * drug_C/1)] 34.099 31 0.321
eta_group[C,I(drug_A/1 * drug_B/1 * drug_C/1)] 35.475 31 0.265
eta_group[C,I(drug_A/1 * drug_B/1)] 26.880 31 0.678
eta_group[C,I(drug_A/1 * drug_C/1)] 39.738 31 0.135
eta_group[C,I(drug_B/1 * drug_C/1)] 27.104 31 0.667
mu_eta[I(drug_A/1 * drug_B/1 * drug_C/1)] 27.264 31 0.659
mu_eta[I(drug_A/1 * drug_B/1)] 40.115 31 0.126
mu_eta[I(drug_A/1 * drug_C/1)] 25.568 31 0.742
mu_eta[I(drug_B/1 * drug_C/1)] 39.725 31 0.135
mu_log_beta[I(log(drug_A/1)),intercept] 38.643 31 0.163
mu_log_beta[I(log(drug_A/1)),log_slope] 37.690 31 0.190
mu_log_beta[I(log(drug_B/1)),intercept] 21.862 31 0.887
mu_log_beta[I(log(drug_B/1)),log_slope] 41.965 31 0.090
mu_log_beta[I(log(drug_C/1)),intercept] 43.085 31 0.073
mu_log_beta[I(log(drug_C/1)),log_slope] 29.024 31 0.568
tau_eta[STRAT,I(drug_A/1 * drug_B/1 * drug_C/1)] 20.614 31 0.922
tau_eta[STRAT,I(drug_A/1 * drug_B/1)] 28.026 31 0.620
tau_eta[STRAT,I(drug_A/1 * drug_C/1)] 35.814 31 0.253
tau_eta[STRAT,I(drug_B/1 * drug_C/1)] 25.082 31 0.764
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] 29.638 31 0.536
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] 19.450 31 0.947
tau_log_beta[STRAT,I(log(drug_B/1)),intercept] 22.189 31 0.877
tau_log_beta[STRAT,I(log(drug_B/1)),log_slope] 23.494 31 0.831
tau_log_beta[STRAT,I(log(drug_C/1)),intercept] 22.630 31 0.862
tau_log_beta[STRAT,I(log(drug_C/1)),log_slope] 29.402 31 0.548

Session Info

## R version 3.6.1 (2019-07-05)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 16.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/libblas/libblas.so.3.6.0
## LAPACK: /usr/lib/lapack/liblapack.so.3.6.0
## 
## locale:
## [1] C
## 
## attached base packages:
## [1] tools     stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] mvtnorm_1.0-11     RBesT_1.4-0        tibble_2.1.3      
##  [4] rstan_2.19.2       StanHeaders_2.19.0 abind_1.4-5       
##  [7] Formula_1.2-3      checkmate_1.9.4    OncoBayes2_0.6-0  
## [10] testthat_2.2.1     Rcpp_1.0.2         devtools_2.2.1    
## [13] usethis_1.5.1      ggplot2_3.2.1      broom_0.5.2       
## [16] tidyr_1.0.0        dplyr_0.8.3        assertthat_0.2.1  
## [19] knitr_1.25         rmarkdown_1.16    
## 
## loaded via a namespace (and not attached):
##  [1] lattice_0.20-38    prettyunits_1.0.2  ps_1.3.0          
##  [4] zeallot_0.1.0      rprojroot_1.3-2    digest_0.6.21     
##  [7] plyr_1.8.4         R6_2.4.0           ggridges_0.5.1    
## [10] backports_1.1.5    stats4_3.6.1       evaluate_0.14     
## [13] highr_0.8          pillar_1.4.2       rlang_0.4.0       
## [16] lazyeval_0.2.2     rstudioapi_0.10    callr_3.3.2       
## [19] labeling_0.3       desc_1.2.0         stringr_1.4.0     
## [22] loo_2.1.0          munsell_0.5.0      compiler_3.6.1    
## [25] xfun_0.10          pkgconfig_2.0.3    pkgbuild_1.0.6    
## [28] rstantools_2.0.0   htmltools_0.4.0    tidyselect_0.2.5  
## [31] gridExtra_2.3      codetools_0.2-16   matrixStats_0.55.0
## [34] crayon_1.3.4       withr_2.1.2        grid_3.6.1        
## [37] nlme_3.1-141       gtable_0.3.0       lifecycle_0.1.0   
## [40] magrittr_1.5       scales_1.0.0       cli_1.1.0         
## [43] stringi_1.4.3      fs_1.3.1           remotes_2.1.0     
## [46] ellipsis_0.3.0     generics_0.0.2     vctrs_0.2.0       
## [49] glue_1.3.1         purrr_0.3.3        processx_3.4.1    
## [52] pkgload_1.0.2      parallel_3.6.1     yaml_2.2.0        
## [55] inline_0.3.15      colorspace_1.4-1   sessioninfo_1.1.1 
## [58] bayesplot_1.7.0    memoise_1.1.0