BDA3 Chapter 2 Exercise 12

Here’s my solution to exercise 12, chapter 2, of Gelman’s Bayesian Data Analysis (BDA), 3rd edition. There are solutions to some of the exercises on the book’s webpage.

Suppose \(\theta\) has a Poisson likelihood so that \(\log p(y \mid \theta) \propto y \log(\theta) - \theta\). We will find Jeffrey’s prior for \(\theta\) and the gamma distribution that most closely approximates it.

The derivative of the log likelihood is \(\frac{y}{\theta} - 1\) and the second derivative is \(-\frac{y}{\theta^2}\). It follows that the Fisher information for \(\theta\) is

\[ J(\theta) = \mathbb E \left( \frac{y}{\theta^2} \right) = \frac{1}{\theta} , \]

so Jeffrey’s prior is \(p(\theta) \propto \frac{1}{\sqrt{\theta}}\). This is an improper prior because

\[ \int_0^\infty \theta^{-\frac{1}{2}} d\theta = \left[ 2\theta^{\frac{1}{2}} \right]_0^\infty = \infty . \]

Since Jeffrey’s prior is improper, we can try approximate it with a gamma prior. Let \(\alpha, \beta \in (0, \infty)\) be the shape and rate parameters of a gamma distribution. Then

\[ \dgamma(\theta \mid \alpha, \beta) \propto x^{\alpha - 1}e^{-\beta x} . \]

Choosing \(\alpha = \frac{1}{2}\) and \(\beta = 0\) yields Jeffrey’s prior. However, \(\beta\) must be positive for the gamma distribution to be proper, so we can choose \(\beta = \epsilon\) sufficiently small. We’ll use the smallest positive float representable in R.

epsilon <- .Machine$double.xmin

upper_limit <- 100000000
step <- upper_limit / 10000

prior <- tibble(theta = seq(0, upper_limit, step)) %>% 
  mutate(density = dgamma(theta, 0.5, epsilon))