Bootstrap resampling: confidence intervals from data
The bootstrap is a computational method for estimating the sampling distribution of a statistic
(like the mean, median, or variance) when the exact distribution is unknown or hard to derive.
It is especially useful for small datasets and complicated statistics.
1) Core idea: resample with replacement
Suppose you observe data \(\mathbf{x}=(x_1,\dots,x_n)\) and compute a statistic \(\hat{T}=T(\mathbf{x})\).
Bootstrap creates new “pseudo-datasets” by sampling from the observed data with replacement.
Each bootstrap sample has size \(n\), just like the original sample.
\[
\mathbf{x}^{*(b)}=(x_1^{*(b)},\dots,x_n^{*(b)}), \quad
x_i^{*(b)} \in \{x_1,\dots,x_n\}\ \text{drawn with replacement.}
\]
For each bootstrap sample, compute the same statistic:
\[
T^{*(b)} = T\!\big(\mathbf{x}^{*(b)}\big), \qquad b=1,\dots,B.
\]
Repeating this many times builds an empirical distribution of \(T^*\), which approximates how \(\hat{T}\) would vary under repeated sampling.
2) Bootstrap estimates: bias and standard error
The bootstrap mean and variance summarize the bootstrap distribution:
\[
\overline{T^*}=\frac{1}{B}\sum_{b=1}^B T^{*(b)},\qquad
\widehat{\mathrm{Var}}(T^*)=\frac{1}{B-1}\sum_{b=1}^B\big(T^{*(b)}-\overline{T^*}\big)^2.
\]
A common bias estimate is:
\[
\widehat{\mathrm{bias}}=\overline{T^*}-\hat{T}.
\]
3) Percentile confidence interval
If you want a \((1-\alpha)\) confidence interval, the percentile bootstrap uses quantiles of the bootstrap statistics:
\[
\mathrm{CI}_{1-\alpha}=\big[\,Q_{\alpha/2}(T^*),\ Q_{1-\alpha/2}(T^*)\,\big].
\]
This method is intuitive and widely used. It works well in many cases, but can be inaccurate for strongly biased or skewed statistics.
4) BCa interval (bias-corrected and accelerated)
The BCa method improves percentile intervals by adjusting the quantile levels using:
- Bias correction \(z_0\), which measures how often \(T^*\) falls below \(\hat{T}\).
- Acceleration \(a\), which measures how sensitive the statistic is to individual observations (estimated by jackknife).
Bias correction:
\[
z_0 = \Phi^{-1}\!\Big(\frac{\#\{T^*<\hat{T}\}}{B}\Big).
\]
Acceleration via jackknife leave-one-out values \(T_{(i)}\):
\[
a=\frac{\sum_{i=1}^n (\overline{T}_{(\cdot)}-T_{(i)})^3}{6\Big(\sum_{i=1}^n (\overline{T}_{(\cdot)}-T_{(i)})^2\Big)^{3/2}}.
\]
Adjusted quantile levels:
\[
\alpha_j=\Phi\!\Big(z_0+\frac{z_0+z_j}{1-a(z_0+z_j)}\Big),
\quad z_j=\Phi^{-1}(j),
\quad \mathrm{CI}=\big[Q_{\alpha_1}(T^*),Q_{\alpha_2}(T^*)\big].
\]
5) Interpreting the histogram
The histogram approximates the sampling distribution of your statistic. A narrow histogram suggests low variability (small standard error),
and a wide histogram suggests high uncertainty. The vertical lines show:
- Observed statistic \(\hat{T}\) computed from the original sample
- Lower and upper CI bounds (percentile or BCa)
6) Notes and limitations
- Bootstrap assumes the sample is representative of the population; resampling from a biased sample reproduces that bias.
- Small \(n\) can make BCa unstable (jackknife sensitivity); increasing \(B\) helps with Monte Carlo noise but not with representativeness.
- For dependent data (time series), you typically need block bootstrap variants.
