BDA3 Chapter 14 Exercise 3

Here’s my solution to exercise 3, chapter 14, of Gelman’s Bayesian Data Analysis (BDA), 3rd edition. There are solutions to some of the exercises on the book’s webpage.

We need to reexpress (y−Xβ)T(y−Xβ) as (μ−β)TΣ−1(μ−β), for some μ, Σ. Using the QR-decomposition of X=QR, we see

(y−Xβ)T(y−Xβ)=(QT(y−Xβ))TQT(y−Xβ)=(QTy−QTXβ)T(QTy−QTXβ)=(QTy−Rβ)T(QTy−Rβ),

where Q is orthogonal and R an invertible upper triangular matrix. We can read off the minimum of this quadratic form as

ˆβ=R−1QTy,

which shows that μ=ˆβ=R−1QTy. Note that

(XTX)−1XT=(RTR)−1RTQT=R−1R−TRTQT=R−1QT

so that ˆβ=(XTX)−1XTy.

Expanding the brackets of both quadratic form expressions and comparing the quadratic coefficients, we see that

Σ−1=RTR=XTX,

which shows that Vβ=(XTX)−1, in the notation of page 355.