BDA3 Chapter 5 Exercise 5

Here’s my solution to exercise 5, chapter 5, of Gelman’s Bayesian Data Analysis (BDA), 3rd edition. There are solutions to some of the exercises on the book’s webpage.

Suppose the joint distribution for parameters \(\theta = (\theta_1, \dotsc, \theta_J)\) can be written as a mixture of iid parameters

\[ p(\theta) = \int \prod_1^J p(\theta_j \mid \phi) p(\phi) d\phi . \]

We’d like to show that the pairwise covariance is always non-negative. Since the parameters \(\theta\) are exchangeable, it is sufficient to show that \(\theta_1, \theta_2\) have non-negative covariance. Using the law of total covariance, the fact that independent variables have zero covariance, and the fact that exchangeable variables have the same expectation, it follows that

\[ \begin{align} \cov(\theta_1, \theta_2) &= \mathbb E (\cov(\theta_1, \theta_2 \mid \phi)) + \cov(\mathbb E (\theta_1 \mid \phi), \mathbb E(\theta_2 \mid \phi)) \\ &= 0 + \mathbb E \left( \cov \left( \theta_1 \mid \phi, \theta_2 \mid \phi \right) \right) \\ &= \mathbb E \left( \mathbb E(\theta_1 \mid \phi) \mathbb E(\theta_2 \mid \phi) \right) - \mathbb E\left( \mathbb E(\theta_1 \mid \phi) \right) \mathbb E \left( \mathbb E(\theta_2 \mid \phi) \right) \\ &= \mathbb E \left( \mathbb E(\theta_1 \mid \phi)^2 \right) - \left( \mathbb E \mathbb E (\theta_1 \mid \phi) \right)^2 \\ &= \var(\mathbb E(\theta_1 \mid \phi)) \\ & \ge 0 . \end{align} \]