--- title: "Sample Size Planning for Sequential ANOVAs" author: "Meike Snijder-Steinhilber" date: "`r Sys.Date()`" output: rmarkdown::html_vignette: toc: true toc_depth: 4 css: customize.css description: > This vignette describes how to plan sample sizes for sequential ANOVAs using simulation-based approaches. vignette: > %\VignetteIndexEntry{Sample Size Planning for Sequential ANOVAs} %\VignetteEncoding{UTF-8}{inputenc} %\VignetteEngine{knitr::rmarkdown} bibliography: references.bib csl: "apa.csl" --- ```{r setup, include=FALSE} knitr::opts_chunk$set( echo = TRUE, eval = FALSE, message = TRUE, warning = FALSE, collapse = TRUE, comment = "#>" ) ``` ## Why Sample Size Planning Matters Sample size planning for sequential tests differs fundamentally from fixed-design studies. In sequential ANOVA, the final sample size is determined by the evidence in the data itself and consequently remains unknown beforehand -- data collection continues until either the upper or lower decision boundary is reached. **The challenge:** While this data-driven stopping rule is very efficient, it creates practical difficulties. Resource planning requires knowing whether you might need 100 observations or 1,000. Budget constraints, time limitations, and logistical considerations all demand some advance estimate of required resources. **The solution:** Although the exact final sample size cannot be known in advance, simulation-based planning bridges the gap between statistical theory and practical constraints. The `sprtt` package provides the `plan_sample_size()` function, which generates HTML reports summarizing simulation results for sequential ANOVAs. Researchers can obtain guidance on: - The typical amount of data required to reach a decision ($N_{\text{median}}$) - The upper limit of resources needed to achieve a specified decision rate ($N_{\text{max}}$) ### Resource Constraints and Decision Rates The decision boundaries of the sequential ANOVA control Type I ($\alpha$) and Type II ($\beta$) errors in the long run. However, introducing a maximum sample size $N_{\text{max}}$ for practical resource planning creates an important complication: it reduces the achievable power below the nominal $1-\beta$. When $N_{\text{max}}$ is reached before a decision boundary is crossed, this results in a **non-decision**. The non-decision rate depends directly on the chosen maximum sample size. This introduces a new metric: the **decision rate** (the probability of reaching a decision) given resource limitations. While non-decisions are undesirable, they represent a crucial conceptual distinction from accepting the null hypothesis. SPRTs like the sequential ANOVA differentiate between stopping data collection to accept the null hypothesis and stopping because more evidence would be required to make a decision but resources are exhausted. Importantly, as long as no decision has been reached, data collection can continue if additional resources become available. ## The `plan_sample_size()` Function The `plan_sample_size()` function generates interactive HTML reports for sample size planning based on a large simulation database. Reports include recommended maximum sample sizes, expected sample sizes, power curves, and comparisons to traditional ANOVA designs. ### Pre-computed Simulation Database To make sample size planning fast and accessible, `sprtt` includes access to extensive simulation results. These simulations were conducted by: 1. Generating 10,000 of datasets for each combination of parameters 2. Running sequential ANOVAs on each dataset 3. Recording when each test stopped 4. Aggregating these results to get key summary statistics that guide the sample size planning This simulation database is stored externally to keep the package installation size small. The data are downloaded automatically on first use of `plan_sample_size()` and cached locally for future sessions. ```{r, echo=FALSE, results='asis'} cat(sprintf( 'Source code of the simulation database: GitHub

' )) ``` ## Getting Started ### Your First Sample Size Report Let's walk through a practical example. Imagine you're planning a study to compare three groups. You want to detect medium-sized effects (Cohen's $f = 0.25$) or larger with specific error control. You set $\alpha = 0.05$ to control Type I errors at the standard 5% level, ensuring that rejections of the null hypothesis are trustworthy in the long run. To minimize Type II errors, you set $\beta = 0.05$, limiting false acceptances of $H_0$ also to 5%. However, given limited resources, you're willing to accept a 15% non-decision rate, meaning you'll reach a decision 85% of the time. Critically, this setup reflects a deliberate trade-off: by keeping both error rates as low as 5%, you accept that a decision will not always be reached — but when it is, it can be trusted. Non-decisions, by contrast, indicate that the available evidence was insufficient given your error constraints, and more data are required. Now let's see how to generate a sample size planning report for this scenario: ```{r first-report} plan_sample_size( f_expected = 0.25, # Expected effect size k_groups = 3, # Number of groups beta = 0.05, # beta error rate decision_rate = 0.85 # desired percentage of decisions ) ``` When you run this code for the first time, several things happen: 1. **Data download:** The simulation database (~70 MB) is downloaded from GitHub and saved to your local cache directory. This is a one-time operation. 2. **Report generation:** An HTML report is created in your temporary directory. 3. **Browser launch:** The report automatically opens in your default web browser (if running interactively). The entire process typically takes a couple of seconds for the initial download, then just a few seconds more for generating the subsequent report. ### Function Parameters | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `f_expected` | numeric | *required* | Expected standardized effect size (Cohen's *f*). Must be one of: 0.10, 0.15, 0.20, 0.25, 0.30, 0.35, or 0.40. | | `k_groups` | integer | *required* | Number of groups to compare. Must be 2, 3, or 4. | | `beta` | numeric | 0.05 | Beta error rate. Must be 0.20, 0.10, or 0.05. | | `output_dir` | character | `tempdir()` | Directory where the HTML report will be saved. | | `output_file` | character | `"sprtt-report-sample-size-planning.html"` | Filename for the generated report. | | `open` | logical | `interactive()` | Whether to open the report in your browser after generation. Set to `FALSE` for batch processing. | | `overwrite` | logical | FALSE | Whether to overwrite an existing file with the same name without prompting. | ### Input Validation The function validates all inputs before generating the report. If you specify a parameter value that doesn't exist in the simulation database, you'll receive an informative error message listing the available options. For example: ```{r error-example, eval=FALSE} # This will produce an error: plan_sample_size(f_expected = 0.22, k_groups = 3) #> Error: `f_expected` = 0.22 is not available. #> Please choose one of 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, or 0.4 ``` ## Practical Use Cases ### Case 1: Comparing Different Effect Sizes The expected effect size has a large impact on required sample size. Here's how to generate reports for different scenarios: ```{r example-effect-sizes} # report 1 plan_sample_size(f_expected = 0.15, k_groups = 3, beta = 0.05) # report 2 plan_sample_size(f_expected = 0.35, k_groups = 3, beta = 0.05) ``` ### Case 2: Saving Reports to a Specific Location By default, reports are saved to a temporary directory. For reports you want to keep, specify a custom location: ```{r example-custom-location} plan_sample_size( f_expected = 0.25, k_groups = 4, output_dir = "~/Documents/research/sample_size_planning", output_file = "study1_anova.html", open = TRUE ) ``` This is particularly useful when preparing documentation for grant applications, pre-registrations, or manuscript supplementary materials. ### Case 3: Preparing Multiple Scenarios When preparing grant applications or pre-registrations, you might want to explore multiple scenarios (e.g., different effect size assumptions): ```{r workflow-scenarios} # Define scenarios to compare scenarios <- data.frame( effect = c(0.15, 0.20, 0.25), label = c("conservative", "expected", "optimistic") ) # Generate reports for each scenario for (i in 1:(nrow(scenarios))) { plan_sample_size( f_expected = scenarios$effect[i], k_groups = 3, beta = 0.10, output_dir = "sample_size_reports", output_file = sprintf("plan_sample_size_%s.html", scenarios$label[i]), open = FALSE, # Don't open each one overwrite = TRUE ) } message("Generated ", nrow(scenarios), " sample size reports") ``` This approach creates a set of reports that document your planning across different assumptions. ## Managing the Simulation Data **Downloading Data Explicitly** While `plan_sample_size()` downloads data automatically when needed, you can also download it explicitly: ```{r download-explicit} # Download simulation data manually download_sample_size_data() ``` This is useful if you want to: - Pre-download data on a fast internet connection before traveling - Verify the download completed successfully - Troubleshoot download issues To force a re-download (for example, after a package update with new simulation data): ```{r download-force} download_sample_size_data(force = TRUE) ``` **Checking Cache Status** To see whether data are cached and how much disk space they occupy: ```{r cache-status} cache_info() ``` This displays: - The cache directory location on your system - Whether simulation data are currently cached - The file size **Clearing the Cache** If you need to free up disk space or suspect corrupted data, you can clear the cache: ```{r cache-clear} cache_clear() ``` The data will be re-downloaded automatically the next time you run `plan_sample_size()`. **Working with Simulation Data Directly** Advanced users may want to access the raw simulation data for custom analyses or visualizations. You can load the data directly into your R session: ```{r load-data-advanced} # Load the complete simulation dataset (downloads automatically if not yet cached) loaded <- load_sample_size_data() # Access the simulation data frame df_all <- loaded$data # Check which dataset version this report is based on loaded$description # short description loaded$version # e.g. "v0.1.0-data" loaded$created # date the dataset was created loaded$n_rep # number of simulation iterations per condition ``` The data frame `df_all` contains all simulation results and can be filtered, summarized, or visualized using standard R tools. See `?load_sample_size_data` for a full description of all available columns.