A well known fact about joint normally distributed random variables, is that they are independent if and only if their covariance is zero. In one direction, this statement is trivial. Any independent pair of random variables has zero covariance (assuming that they are integrable, so that the covariance has a well-defined value). The strength of the statement is in the other direction. Knowing the value of the covariance does not tell us a lot about the joint distribution so, in the case that they are joint normal, the fact that we can determine independence from this is a rather strong statement.
Theorem 1 A joint normal pair of random variables are independent if and only if their covariance is zero.
Proof: Suppose that X,Y are joint normal, such that and , and that their covariance is c. Then, the characteristic function of can be computed as
for all . It is standard that the joint characteristic function of a pair of random variables is equal to the product of their characteristic functions if and only if they are independent which, in this case, corresponds to the covariance c being zero. ⬜
To demonstrate necessity of the joint normality condition, consider the example from the previous post.
Example 1 A pair of standard normal random variables X,Y which have zero covariance, but is not normal.
As their sum is not normal, X and Y cannot be independent. This example was constructed by setting for some fixed , which is standard normal whenever X is. As explained in the previous post, the intermediate value theorem ensures that there is a unique value for K making the covariance equal to zero.
Theorem 1 generalizes in a straightforward manner to more than two random variables, even for infinitely many.
Theorem 2 Let be a joint normal collection of random variables. Then, the are independent if and only if their covariances are zero.
This result can be established by a straightforward extension of the proof given or theorem 1 above. However, it is also a special case of a more general independence result to be given further below, so I leave the proof until then. Using the equivalence of pairwise independence and zero covariances, theorem 2 can also be expressed without direct reference to covariances.
Theorem 3 Let be a joint normal collection of random variables. Then, the are independent if and only if they are pairwise independent.
Theorem 1 can also be extended in a different direction. Rather than replacing the pair of random variables by an arbitrary collection, all of which have zero covariance, they can instead be replaced by a pair of collections of variables.
Theorem 4 Let and be collections of random variables such that is joint normal. Then, and are independent if and only if X and Y have zero covariance for all and .
Just to clarify, this statement does not mean that random variables in are independent from each other but, rather, that the collection is independent of . As above, the equivalence of zero covariances and pairwise independence stated by theorem 1 allows to write this result without reference to covariances.
Theorem 5 Let and be collections of random variables such that is joint normal. Then, and are independent if and only if X and Y are independent for all and .
Although it is not difficult to generalize the proof of theorem 1 in order to directly prove theorems 4 and 5, they are also just special cases of a more general result which I prove further below. For now, the following example application is very useful in describing a Brownian motion X over the unit interval in terms of its endpoints and an independent Brownian bridge .
Example 2 (Brownian bridge) Let be a standard Brownian motion. Then, the process over is independent of over .
This example is a direct application of theorem 4 to the collections and . The covariances of the Brownian motion are given by . Hence, for any , we compute
The necessity of the joint normal condition in theorems 2, 3, 4, and 5 is demonstrated by the following example.
Example 3 A triple of standard normals which are pairwise independent (and, hence, pairwise joint normal), but are not all independent (hence, not all joint normal).
This shows that we cannot infer independence from pairwise independence for arbitrary collections of normal random variables, so that the joint normality of in theorems 2 and 3 is necessary. Furthermore, consider the collections and , both of which are joint normal. Then, X and Y are independent for all and even though is not independent of , demonstrating that joint normality of the union is necessary for theorems 4 and 5.
Example 3 can be constructed as follows. Let be independent random variables, where the are standard normal and have the Rademacher distribution, . Then, set , which also has the Rademacher distribution. As the are pairwise independent, then the random variables are also pairwise independent and, by symmetry of the standard normal distribution, are also standard normal. However, the product is equal to 1 and, hence,
Theorems 1, 2, 3, 4 and 5 are all special cases of the following `master’ theorem.
Theorem 6 Let be a collection of collections of random variables, whose union is joint normal.
Then, they are independent collections if and only if for all and for in I.
For clarity, I state precisely what the independence property means in this result. For each , let be the sigma-algebra generated by , which is just the smallest sigma-algebra on the underlying probability space with respect to which every is measurable. Then, independence of the collections is equivalent to independence of these sigma-algebras, so that
for all finite pairwise distinct sequences and .
We could prove theorem 6 directly by using the characteristic function for multivariate normals along similar lines to 1, although it can get a little messy. I will take an alternative approach and, instead, use theorem 1 proven above to imply the much more general result. This does not require making use of any further properties of joint normals beyond the basic statement that linear combinations of them remain joint normal. To make the leap from theorem 1 to 6, I use the following statement which applies to arbitrary collections of (real-valued) random variables. For any such collection , then denotes the collection of finite linear combinations from .
Lemma 7 Let be a collection of collections of random variables. The following are equivalent,
- are independent over .
- for any finite sequence and , then each pair and are independent.
Proof: The second statement immediately follows from the first, by the definitions, so I just concentrate on the proof that the second statement implies the first. Recall from the definition of independence, we need to show that (1) holds for any finite pairwise distinct sequence . We use induction on n, so suppose that the result holds for , and set and . We just need to show that
since, by the induction hypothesis, (1) follows immediately from this. We show that (2) holds for all A and B in the sigma-algebras generated by and respectively. Letting and be the union of the sigma-algebras generated by each finite subset of, respectively, and , these are pi-systems and, by the pi-system lemma, it is sufficient to prove the result for and . Hence, we suppose that A and B are, respectively, in the sigma-algebras of finite sub-collections
The characteristic function of the random vector is computed by
for any and , where and . Here, independence of and was used, and is guaranteed by the second statement of the lemma. Since the distribution of a random vector is uniquely determined by its characteristic function, this shows that Y and Z are independent. Hence, A and B are independent, giving (1) as required. ⬜
Finally, I use this to give a proof of the main independence result.
Proof of theorem 6: Choose a finite sequence and . Setting , by lemma 7 we just need to show that every and are independent. By assumption, we know that every and have zero covariance. By linearity of the covariance, this immediately extends to all and which, by theorem 1, are therefore independent, as required. ⬜