Chapter 2 Applications of prediction models

Prediction models may have various applications in medical practice and research. We here present some additional material.

Fig 2.1 is from an Excel sheet available here.

In section 2.4.2, an example is presented of selection of patients for randomized controlled trials in traumatic brain injury (TBI). Another example is the selection of patients for therapy in differentiated thyroid cancer.

Inclusion criteria were considered for a randomized trial on adjuvant radio-iodine therapy in differentiated thyroid cancer 1). Randomized trials in this area were considered impossible due to anticipated formidable sample sizes in a disease with an overall excellent prognosis. The authors analyzed outcomes of 342 patients using Cox proportional hazards analysis. A model was created that distinguished risk categories of recurrence using commonly available baseline variables (modeling results in: Table extra).
A subset of patients was defined as those with a score of 3 or more (for example age 60+, score 60/20=3). These patients had a substantial risk for relapse (30–40%) and could be eligible for a randomized trial assessing the impact of high-dose therapy. Assuming a clinically relevant effect of 30% reduction of relapses, 290 patients would have to be entered in either arm (α 0•05, power 80%). The authors conclude that even though a randomized trial on this issue will be difficult to design and conduct, sample size is not the main problem.

Andrew Gelman started a great discussion about interaction testing in randomized controlled trials (RCTs), i.e. the search for subgroup effects of treatment.

I commented as follows:
Did anyone really simulate the case study as suggested? Admittedly, estimates of effect sizes are key; and these impact on the estimated SE in a model, correct?
Under the Null, main effect has SE 0.63; interaction has SE of 1.26, with N=1000, sigma 10.

With the given example, main effect 2.8*sigma; for x2== -.5, 2.1*sigma, and for x2== .5, 3.5*sigma.
This is the implication of the interaction being half the size of the main effect (1.4; -.7 and +.7 effect).
This results is 3 interesting findings:

1. SE of main effect is estimated larger (as 0.70)
2. SE of main effect is estimated as 0.66 if we adjust for x2
3. SE of interaction is estimated correctly as 1.26.

My reflections:

• The only correct model is the model with interaction; and there the SE is identical to what was derived under the Null (SE 1.26).
• In practice, we will start with the main effect model, and some variance is explained by adjusting for other covariates that are associated with the outcome. This is indeed what we observed for x2, even if x2 is interacting with x1. So, this confirms the recommendation to include covariates that are associated with the outcome, more than searching for subgroup effects.
• The inclusion of prognostic covariates is beneficial in linear models as well as in generalized linear models such as logistic or Cox regression, for example for mortality after an acute MI:

The R script:

library(“arm”)
N <- 1000
sigma <- 10
y <- rnorm(N, 0, sigma)
x1 <- sample(c(-0.5,0.5), N, replace=TRUE)
x2 <- sample(c(-0.5,0.5), N, replace=TRUE)
display(lm(y ~ x1))
display(lm(y ~ x1 + x2 + x1:x2))
# this was with y under the Null

# now with y under the alternative of separate x1 effects for x2 values
# specifically:
# overall effect of x1 = 2.8 * sigma; for x2==-.5: 2.1 * sigma; for x2==0.5: 3.5 * sigma
y[x1==.5 & x2== -.5] <- rnorm(length(y[x1==.5 & x2==-.5]), 2.1*sigma, sigma)
y[x1==.5 & x2== .5] <- rnorm(length(y[x1==.5 & x2== .5]), 3.5*sigma, sigma)
display(lm(y ~ x1)) # SE 0.72
display(lm(y ~ x1 + x2)) # SE 0.69
display(lm(y ~ x1 + x2 + x1:x2)) # SE interaction 1.31; 1.31/0.72 equals 1.82 rather than a factor 2