The validate() function in the rms library by Frank Harrell provides for an easy implementation of bootstrap validation of prediction models. Here is some R code to check on 2 issues re the estimation of the discriminatory ability (AUC, or Somers' D):

  1. Bias: Does bootstrap validation of the performance of logistic regression models appropriately correct for optimism bias?
  2. Variance: What is the standard error of the optimism corrected performance estimate? Can we simply use the standard error of the apparent performance estimate provided with the original model fit?

Findings:

  1. Bias: the optimism in the apparent estimate of performance can be substantial. With 10 predictors, n=200 (100 events), the average AUC drops from 0.78 (apparent) to 0.74 (validated). With n=100 (50 events), the optimism is larger (drop from 0.81 to 0.73). The AUC is reasonably well corrected for optimism by bootstrap validation in this simulation. With p=10, 50 events, the Events Per Variable (EPV) ratio is 5, with estimated optimism at 85% of what was needed (0.155 while 0.183 needed). Note that this is a simulation for a fully pre-specified model; random normal predictors, no correlations.
  2. Variance: the standard error of the optimism-corrected performance estimate is considerably larger than the standard error obtained with the apparent performance. With 10 predictors, n=200 (100 events, EPV 10), the apparent SE is around 85% of the true SE at validation; with n=100 (50 events, EPV 5), the apparent SE is around 75% (see Boxplots below). With 5 predictors and 50 events (EPV 10), the apparent SE is again 85% of the true SE at validation.
    • –> More work is needed to provide methods for appropriate quantification of uncertainty in performance estimates after bootstrap validation.

Boxplot for 10 predictors, 100 events
Boxplot for 10 predictors, 50 events
Boxplot for 5 predictors, 50 events

rcode_and_data/chapter17.txt · Last modified: 2018/07/12 15:03 by ewsteyerberg
chimeric.de = chi`s home Creative Commons License Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0