## Bootstrap and Jacknife

### Bootstrap and Jackknife

Bootstrap and Jackknife algorithms don’t really give you something for nothing. They give you something you previously ignored.

The jackknife is an algorithm for re-sampling from an existing sample to get estimates of the behavior of the single sample’s statistics. An example of the jackknife would be to omit the $$1^{st}, 2^{nd}, 3^{rd}$$ … observation from a sample of size $$n$$. Then compute the $$(n-1)$$ averages. The variance of these re-sampled averages is an estimate of the variance of the original sample mean.

The bootstrap is a generalization of the jackknife that re-samples, with replacement, some number of times (say 1000), and computes the statistic of interest from these re-samples, thereby providing an estimate of the original sample’s variability. (e.g. The $$95^{th}$$ percentile is estimated by the $$950^{th}$$ observation of the ordered 1000 re-samples). The name, of course, comes from its apparent ability to pull itself up by its own bootstraps. (In Rudolph Erich Raspe’s tale, Baron Munchausen had fallen to the bottom of a deep lake and just as he was to succumb to his fate he thought to pull himself up by his own bootstraps.)

##### Cautions:

While this simple algorithm works quite well in many situations, it is termed a “naive” bootstrap because indiscriminate use can produce bogus results.  For example, a simple resampling of a time-series would ignore any auto-correlation and lead to spurious conclusions.  If the data are in $$x, y$$ pairs, then a naive resampling would be from the joint distribution of $$x$$ and $$y$$ when for regression purposes we need the conditional distribution of $$y$$, given $$x$$. And it isn’t magic: If what you’re looking for isn’t in the original sample, then it cannot appear in any of the resamples.

##### Reference:

Bradley Efron and Robert J. Tibshirani, An Introduction to the Bootstrap, Chapman and Hall, 1993