25  Plug-in estimation & Bootstrap I

Last time, we defined the empirical distribution of a dataset x=(x1,x2,,xm). We saw that the empirical distribution X has probability mass function p(x)=1mj=1mI{x}(xj), which is simply the proportion of datapoints taking the value xj.

25.1 The “plug-in” principle

We have one thing we didn’t get to last time, which is “plug-in estimation”.

Suppose now that X=(X1,X2,,Xm) is an IID sample from a distribution X that is either unknown or too difficult to work with directly. How can we find out things about this distribution X?

Well, we could form the empirical distribution X from these samples, and work with that instead.

We have to be a bit careful here, because there two levels of randomness here.

  1. First, there is the fact that the samples X=(X1,X2,,Xm) are random IID samples from X.

  2. Once we have the samples X, that fixes the empirical PMF p. Then X is itself a random variable, with PMF p.

We will write E, P, Var and so on for the first type of randomness – that is, randomness coming from the random variable X. We will write E, P, Var and so on for randomness coming from the empirical random variable X treating the samples X as fixed. So, for example, the expectation E(ϕ(X)) is really shorthand for the conditional expectation E(ϕ(X)X).

One way to estimate something about the random variable X is to take the formula involving X, then keep that same formula, but replace the true random variable X with the empirical random variable X. This is called the plug-in principle, and such an estimator is a plug-in estimator – the idea is that we simply “plug X in” to the existing formula.

This is easier to see if we take some examples.

Suppose we wanted to estimate the expectation EX of the true distribution X. To estimate this, we instead plug in the empirical distribution X in place of X and the empirical expectation E in place of the expectation over the random samples E. So our estimator is instead EX. We saw last time that EX is the sample mean X=1mj=1mXj. So the plug-in estimator for the expectation EX is the sample mean X.

Suppose we wanted to estimate the variance Var(X) of the true distribution. Again, we plug in X, to instead find Var(X), which we saw last time is Var(X)=1mj=1m(XjX)2, which is very similar to the sample variance of X.

Suppose wanted to estimate Eϕ(X) for some function ϕ. The plug-in estimator for this is Eϕ(X)=xϕ(x)p(x)=xϕ(x)1mj=1mI{x}(Xj)=1mj=1mxϕ(x)I{x}(Xj)=1mj=1mϕ(Xj), by the same logic we used for the expectation and variance last time. This is the Monte Carlo estimator from the beginning of this module – we have a sample X1,X2,,Xm and we have form the Monte Carlo estimator Eϕ(X). This shows there are deep connections between Monte Carlo estimation and the the empirical distribution and plug-in estimation.

25.2 The bootstrap set-up

OK, we’re now moving on from the empirical distribution to a slightly different but related topic: the bootstrap.

Suppose a statistician is interested in a statistic T=T(X1,X2,,Xn) of n IID samples from a random variable X. For example, this might be:

  • Suppose I pick a basketball squad of 12 players at random; what is their average height? Here, X is the distribution of basketball players’ heights, n=12, and the statistic is T=T(X1,X2,,X12)=112i=112Xi.

  • Suppose I visit The Edit Room cafe 5 times; what’s the longest queue I have to deal with. Here, X is the distribution of queue lengths at The Edit Room, n=5, and the statistic is T=T(X1,X2,X3,X4,X5)=max{X1,X2,X3,X4,X5}.

  • Suppose a supermarket distributor buys 1001 beef steaks; what is the median weight of the steaks? Here X is the distribution of weights of steaks, n=1001, and the statistic is T=T(X1,X2,,X1000)=median(X1,X2,,X1001).

The statistician is likely to be interested in properties of this statistic. For example, three of the most important things the statistician is likely to want to know are:

  • The expectation ET=ET(X1,,Xn) of the statistic.

  • The variance Var(T)=Var(T(X1,,Xn)) of the statistic – or related concepts like the standard deviation.

  • A prediction interval [U,V] for the statistic, such that P(T[U,V])=1α.

Now, if the statistician knew the true distribution X, and if it were simple enough to work with, then she could calculate the true values of these. But suppose the distribution is unknown (or too complicated to work with). Instead, the statistician just has m samples X=(X1,X2,,Xm). You could think of these as data measurements that are modelled as coming from the distribution X, or you could think of them as output from a computer program that can sample from X exactly.

Note that there’s two numbers here: n is the number of samples required to calculate the statistic T=T(X1,X2,,Xn) once, and m is the total number of samples we have available. The most common situation is “m is somewhat bigger than n, although not vastly bigger”, but the mathematical definitions are valid for any n and m.

The bootstrap method is the following idea:

  1. Take n samples from the empirical distribution X of X. This is equivalent to sampling n of the values X1,X2,,Xm with replacement. Let’s call these samples X1,X2,,Xn. Evaluate the statistic with these samples T=T(X1,X2,,Xn).

  2. Repeat step 1 many times; let’s say B times. Keep taking n of the samples with replacement and evaluating the statistic. We now have B versions of that statistic T1,T2,,TB.

  3. Use these B versions of the statistic to get a bootstrap estimate of the expectation, variance, or prediction interval. To estimate the expectation of the statistic ET=ET(X1,,Xn), use the sample mean of the evaluated statistics T=1Bk=1BTB. To estimate the variance Var(T) use the sample variance 1B1k=1B(TkT). We’ll come back to the prediction interval next time.

The bootstrap concept was discovered by the American statistician Bradley Efron in a hugely influential paper “Bootstrap methods: another look at the jackknife” in 1979. The name “bootstrap” comes from the phrase “to pull yourself up by your bootstraps”, which roughly means to make progress without any outside help, in a way that might initially seem impossible – similarly, the bootstrap manages to estimate properties of a statistic by just reusing the same set of samples over and over again. (The “jackknife” in the title of Efron’s paper was and earlier, simpler, less powerful idea along similar lines, named after the multipurpose tool the jackknife.)

25.3 Bootstrap for expectation and variance

Example 25.1 Let’s take the cafe example above. The statistic in question is T=T(X1,X2,X3,X4,X5)=min{X1,X2,X3,X4,X5}. A researcher wants to estimate the expectation of this statistic.

The researcher visits The Edit Room at 30 random occasion and notes the following data.

Queue length 0 1 2 3 4 5 7 11 Total
Number of occasions 11 5 7 3 1 1 1 1 30

 

We start by taking 5 samples from the empirical distribution – that is, we choose 5 of the datapoints uniformly at random with replacement. Let’s say these are (0,1,4,4,5). (It turns out we sampled the value 4 twice, even though it only occured once – that does happen sometimes when we’re sampling with replacement.) The value of the statistic for this sample is T1=T(0,1,4,4,5)=max{0,1,4,4,5}=5.

We keep doing this many times – we pick five samples with replacement, and calculate their maximum.

queues <- c(rep(0, 11), rep(1, 5), rep(2, 7), rep(3, 3), 4, 5, 7, 11)

boots <- 1000
maxes <- rep(0, boots)
for (k in 1:boots) {
  minisample <- sample(queues, 5, replace = TRUE)
  maxes[k] <- max(minisample)
}

This gives us 1000 realisations of the test statistic. We can use these to look at the distribution of the test statistic:

dist <- table(maxes) / boots
dist
maxes
    0     1     2     3     4     5     7    11 
0.009 0.034 0.223 0.227 0.103 0.107 0.124 0.173 
plot(dist)

We can also look at particular figures of interest. For example, the expectation of T should be close to the sample mean of our Ts, and the variance of T should be close to the sample variance of our Ts.

c(mean(maxes), var(maxes))
[1]  4.87900 10.50687

Next time, we’ll look into bootstrap methods in more detail.