BDA3 Chapter 3 Exercise 1

Here’s my solution to exercise 1, chapter 3, of Gelman’s Bayesian Data Analysis (BDA), 3rd edition. There are solutions to some of the exercises on the book’s webpage.

Let y∣θ∼MultinomialJ(θ) with prior θ∼DirichletJ(α). We would like to find the marginal distribution of ϕ:=θ1θ1+θ2.

Marginal posterior of Dirichlet-multinomial

As shown in the book, the posterior is θ∣y∼Dirichlet(y+α). The marginal posterior of (θ1,θ2)∣y can be written as

p(θ1,θ2∣y)=∫10p(θ∣y)dθ3⋯dθJ−1∝θy1+α1−11θy2+α2−12∫10θy3+α3−13⋯θyJ−1+αJ−1−1J−1(1−J−1∑1θj)yJ+αJ−1dθ3⋯dθJ−1.

The tricky part is calculating the integral part, which we define

I:=∫10θy3+α3−13⋯θyJ−1+αJ−1−1J−1(1−J−1∑1θj)yJ+αJ−1dθ3⋯dθJ−1.

To calculate I, first note that

∫10θs(c−θ)tdθ=∫10θs(1−θc)tctdθ=∫10(θc)s(1−θc)tcs+tdθ=∫10ϕs(1−ϕ)tcs+t+1dϕ,ϕ:=θc=B(s+1,t+1)cs+t+1,

if c is not a function of θ. With c:=1−∑J−21θj,

I=∫10θy3+α3−13⋯θyJ−2+αJ−2−1J−2θyJ−1+αJ−1−1J−1(1−J−2∑1θj−θJ−1)yJ+αJ−1dθ3⋯dθJ−1=∫10θy3+α3−13⋯θyJ−2+αJ−2−1J−2(∫10θyJ−1+αJ−1−1J−1(c−θJ−1)yJ+αJ−1dθJ−1)dθ3⋯dθJ−2∝∫10θy3+α3−13⋯θyJ−2+αJ−2−1J−2(c)yJ−1+yJ+αJ−1+αJ−1dθ3⋯dθJ−2=∫10θy3+α3−13⋯θyJ−2+αJ−2−1J−2(1−J−2∑1θj)yJ−1+yJ+αJ−1+αJ−1dθ3⋯dθJ−2.

Continuing by induction,

I=(1−θ1−θ2)∑J3(yj+αj)−1.

Now that we have the integral part, the marginal posterior can be written

p(θ1,θ2∣y)∝θy1+α1−11θy2+α2−12(1−θ1−θ2)∑J3(yj+αj)−1.

This has the form of a Dirichlet distribution, so the marginal posterior is

(θ1,θ2,1−θ1−θ2)∣y∼Dirichlet(y1+α1,y2+α2,J∑3(yj+αj)).

Change of variables

Now define (ϕ1,ϕ2):=(θ1θ1+θ2,θ1+θ2), so that (θ1,θ2)=(ϕ1ϕ2,ϕ2−ϕ1ϕ2). The Jacobian of this transformation is

|∂θ1∂ϕ1∂θ1∂ϕ2∂θ2∂ϕ1∂θ2∂ϕ2|=|ϕ2ϕ1−ϕ21−ϕ1|=ϕ2.

Therefore, the probability distribution of the new variables is

p(ϕ1,ϕ2∣y)=(ϕ1ϕ2)y1+α1−1(ϕ2(1−ϕ1))y2+α2−1(1−ϕ2)∑J3(yj+αj)−11ϕ2=ϕy1+α1−11(1−ϕ1)y2+α2−1ϕy1+y2+α1+α2−32(1−ϕ2)∑J3(yj+αj)−1=p(ϕ1∣y)p(ϕ2∣y),

where

ϕ1∣y∼Beta(y1+α1,y2+α2)ϕ2∣y∼Beta(y1+y2+α1+α2−2,J∑3(yj+αj)).

The marginal posterior for ϕ1 is equivalent to the posterior obtained from a ϕ1∼Beta(α1,α2) prior with a y1∣ϕ1∼Binomial(y1+y2,ϕ1) likelihood.