Model checking and validation

Model fitting and inference for infectious disease dynamics

Recap

Where we are

In the last session we asked whether the sampler worked: R̂, effective sample size, trace plots, divergences.

A converged chain tells you the algorithm did its job.

It says nothing about whether the model is any good.

You can have a perfectly converged posterior for a model that cannot possibly have produced your data.

Two different questions

	Question	Tools
Diagnostics	Did the algorithm work?	R̂, ESS, trace plots, divergences
Model checking	Does the model make sense?	Prior and posterior predictive checks, residuals, SBC

This session is about the second row.

The Bayesian workflow

Model building is a loop

Model checking is not a step at the end. Gelman et al. (2020) describe fitting as an iterative workflow:

Specify model Prior predictive check Fit Diagnose sampler Posterior predictive check

↺ if the checks fail, revise the model and go round again

So far the course has covered specifying, fitting and diagnosing. This session covers the two checking steps, and what to do when they fail.

Why bother?

• A model that fits badly can give you confidently wrong answers.
• Clean convergence diagnostics are not evidence that the model is right.
• Checking is cheap relative to the cost of publishing a wrong estimate.

The aim is not to prove the model is correct. Every model is wrong. The aim is to find the ways in which it is wrong that matter for the question you are asking.

The outbreak

Tristan da Cunha, 1971

The most remote inhabited island in the world: 284 people, one ship every few months.

An outbreak in a closed population

An influenza-like illness arrived with a visiting ship and infected most of the island over eight weeks.

Two features make it interesting for our purposes:

• Closed — nobody arrived or left, so there is no importation to explain anything away
• Two waves — the epidemic subsided and then came back, in a population with no new susceptibles

Prior predictive checks

Before you see the data

A prior predictive check simulates data from the model using parameters drawn from the prior alone.

$$\theta^{(i)} \sim p(\theta), \qquad y^{(i)} \sim p(y \mid \theta^{(i)})$$

Then ask: do these simulated epidemics look like something that could plausibly happen?

What our priors imply

Two hundred draws from this session’s priors, on an island of 284. Most burn through the population in a week.

What it catches

• Priors that put weight on impossible outcomes — attack rates above 100%, epidemics lasting decades, negative durations.

• Priors that are far too vague, so the model considers almost anything possible and the data have to do all the work.

• Priors that are accidentally informative, ruling out the truth before you start.

This needs no data and no fitting.

A warning about vague priors

“Uninformative” priors are rarely uninformative on the scale you care about.

You saw this in the first session: Uniform(1, 20) on $R_0$ sounds reasonable but most of its range produces infeasible epidemics.

A uniform says an $R_0$ of 19 is exactly as plausible as one of 3.

Posterior predictive checks

After fitting

A posterior predictive check simulates data using parameters drawn from the posterior:

$$\theta^{(i)} \sim p(\theta \mid y), \qquad y^{\text{rep},(i)} \sim p(y \mid \theta^{(i)})$$

If the model is adequate, the observed data should look like a typical draw from this distribution, not an outlier.

The SIR model on Tristan da Cunha

What to look for

• Does the timing of the peak match?
• Is the height of the peak captured?
• Is the shape right, or does the model systematically over- or under-shoot at particular times?
• Is the spread of the replicated data plausible, or is the observed series suspiciously smooth or suspiciously noisy by comparison?

Posterior predictive p-values

Pick a summary statistic $T(y)$ — peak height, epidemic duration, final size — and compare the observed value against its distribution under replication:

$$p_B = \Pr\big(T(y^\text{rep}) \geq T(y) \mid y\big)$$

• Values near 0.5 mean the observed data are typical for that statistic.
• Values near 0 or 1 mean the model systematically misses that feature.

These are not frequentist p-values and should not be thresholded at 0.05. They are a diagnostic for “which feature is my model missing”, not a test.

Total cases

Residuals: observed minus predicted, over time

Scattered about zero, spread the same throughout.

Runs above then below zero: the model is wrong, not just noisy.

Spread grows with the prediction: try a negative binomial.

Autocorrelation in the residuals breaks the conditional-independence assumption in the likelihood.

Simulation-based calibration

Checking the whole machinery

Predictive checks compare the model to one dataset. Simulation-based calibration (Talts et al. 2018) checks whether the inference is correct on average, across datasets:

1. Draw $\theta^\text{true} \sim p(\theta)$
2. Simulate data $y \sim p(y \mid \theta^\text{true})$
3. Fit the model to $y$ and obtain posterior samples
4. Record the rank of $\theta^\text{true}$ among those samples
5. Repeat many times

If the model and sampler are working, those ranks are uniformly distributed.

$\theta^\text{true}$ is itself a draw from the posterior it is ranked against, so its rank is uniform by construction. Nothing about the model being right enters it: SBC tests the inference, not the model.

Reading an SBC histogram

The grey band is where a uniform histogram should sit.

SBC is expensive: it means fitting the model hundreds of times.

When checks fail

What to do

Failure is information. In rough order of what to try:

1. Revisit the priors — are they excluding the region the data prefer?
2. Revisit the structure — is there a mechanism the model is missing? Waning immunity, age structure, time-varying transmission.
3. Revisit the data — are the observations really what you think they are?
4. Revisit the observation model — overdispersion, under-reporting, reporting delays (Session 8).

A model that fits everything is likely not to predict anything well.

When is a model good enough?

There is no threshold.

Good enough means

• it captures the features relevant to your question
• you understand the ways it does not

Document the checks you ran, including the ones that failed.

A reader who knows where the model breaks can judge how far to trust it. A reader shown only the final results cannot.

Your Turn

In the practical you will

• run prior predictive checks and see what vague priors actually imply
• generate posterior predictive distributions with predict and decondition
• compute posterior predictive p-values for several summary statistics
• inspect residuals for structure and autocorrelation

Simulation-based calibration is written up at the end, to work through afterwards.

Return to the session