---
title: "The Generalizability-Theory Core"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{The Generalizability-Theory Core}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
set.seed(2026)
```

```{r setup}
library(aiEvalR)
```

This vignette covers the module that carries `aiEvalR`'s methodological
weight: multi-facet **generalizability theory** for evaluating AI
systems as measurement instruments. It goes beyond the single-number
reliability summaries in the getting-started tour.

## From one reliability number to a variance decomposition

Classical reliability gives you a single coefficient. But when you
evaluate an AI system, the "error" in a score has structure: some
variation comes from *which prompt* you used, some from *which model
version*, some from *stochastic run-to-run variation*. Generalizability
theory (Cronbach et al., 1972; Brennan, 2001) decomposes score variance
into these separate sources.

`aiEvalR` treats the object of measurement (a "case" -- the underlying
task) as what you want a generalizable score about, and treats prompt
formulation, model, and run as *facets* whose variation is error.

## A G-study

We simulate a fully crossed design: 20 cases, each evaluated under 3
prompt formulations, 2 models, and 2 stochastic runs. In real use, these
would be actual AI outputs; here we plant known variance components so
you can see them recovered.

```{r gstudy-data}
design <- expand.grid(case = 1:20, prompt = 1:3, model = 1:2, run = 1:2)

# plant variance: cases differ most, then prompts, then model, then run
case_eff   <- rnorm(20, sd = 1.0)
prompt_eff <- rnorm(3,  sd = 0.5)
model_eff  <- rnorm(2,  sd = 0.3)
run_eff    <- rnorm(2,  sd = 0.2)

design$score <- 5 +
  case_eff[design$case] + prompt_eff[design$prompt] +
  model_eff[design$model] + run_eff[design$run] +
  rnorm(nrow(design), sd = 0.5)
```

```{r gstudy}
g <- ai_generalizability(
  design,
  score  = "score",
  case   = "case",
  facets = c("prompt", "model", "run")
)
g
```

The variance components tell you *where* the noise lives. If prompt
variance is large, your evaluation is sensitive to prompt wording; if
run variance dominates, the system is stochastically unstable.

## A D-study: how many conditions do you need?

The G-study estimates variance components once. A **D-study** then
projects reliability for any hypothetical measurement design -- without
refitting -- so you can ask "how many prompts and runs do I need for a
dependable score?"

```{r dstudy}
# dependability using 3 prompts, 2 models, 2 runs (as observed)
ai_dstudy(g, n_prompt = 3, n_model = 2, n_run = 2)
```

Two coefficients are reported. The **generalizability coefficient**
(relative error) suits rank-ordering systems; **Phi** (absolute error,
"dependability") suits criterion-referenced decisions like "is this
system's score above a fixed bar." Reducing conditions lowers both:

```{r dstudy-compare}
full    <- ai_dstudy(g, n_prompt = 3, n_model = 2, n_run = 2)
minimal <- ai_dstudy(g, n_prompt = 1, n_model = 1, n_run = 1)

c(full_phi = full$phi, minimal_phi = minimal$phi)
```

Fewer conditions per facet means a less dependable score -- the D-study
quantifies exactly how much you lose.

## Conditional error and decision consistency

Measurement error is often not uniform: a system may be steady on
typical inputs and erratic near decision boundaries.
`ai_conditional_sem()` estimates error as a function of the score level.
When outputs drive a categorical decision via a threshold,
`ai_decision_consistency()` estimates how often repeated administrations
would agree.

```{r decision-consistency}
# rows = prompts, cols = repeated occasions; some prompts sit near a cut
responses <- rbind(
  c(9, 9, 10),   # clearly above a cut of 5 every time
  c(1, 0, 1),    # clearly below
  c(4, 6, 5)     # right on the boundary -> inconsistent
)
ai_decision_consistency(responses, cutpoint = 5)
```

A prompt whose repeated outputs straddle the cutpoint yields
inconsistent decisions even when the underlying score is only slightly
uncertain -- decision consistency captures that risk directly, which a
single reliability coefficient does not.

## References

Brennan, R. L. (2001). *Generalizability Theory*. Springer.

Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). *The
Dependability of Behavioral Measurements*. Wiley.