The validate() function in the rms library by Frank Harrell provides for an easy implementation of bootstrap validation of prediction models. Here is some R code to check on 2 issues re the estimation of the discriminatory ability (AUC, or Somers' D):
- Bias: Does bootstrap validation of the performance of logistic regression models appropriately correct for optimism bias?
- Variance: What is the standard error of the optimism corrected performance estimate? Can we simply use the standard error of the apparent performance estimate provided with the original model fit?
Findings:
- Bias: the optimism in the apparent estimate of performance can be substantial. With 10 predictors, n=200 (100 events), the average AUC drops from 0.78 (apparent) to 0.74 (validated). With n=100 (50 events), the optimism is larger (drop from 0.81 to 0.73). The AUC is reasonably well corrected for optimism by bootstrap validation in this simulation. With p=10, 50 events, the Events Per Variable (EPV) ratio is 5, with estimated optimism at 85% of what was needed (0.155 while 0.183 needed). Note that this is a simulation for a fully pre-specified model; random normal predictors, no correlations.
- Variance: the standard error of the optimism-corrected performance estimate is considerably larger than the standard error obtained with the apparent performance. With 10 predictors, n=200 (100 events, EPV 10), the apparent SE is around 85% of the true SE at validation; with n=100 (50 events, EPV 5), the apparent SE is around 75% (see Boxplots below). With 5 predictors and 50 events (EPV 10), the apparent SE is again 85% of the true SE at validation.
- –> More work is needed to provide methods for appropriate quantification of uncertainty in performance estimates after bootstrap validation.
