Thanks to convolution, we can obtain the probability distribution of a sum of independent random variables.
The Convolution Series
- Definition of convolution and intuition behind it
- Mathematical properties of convolution
- Convolution property of Fourier, Laplace, and z-transforms
- Identity element of the convolution
- Star notation of the convolution
- Circular vs. linear convolution
- Fast convolution
- Convolution vs. correlation
- Convolution in MATLAB, NumPy, and SciPy
- Deconvolution: Inverse convolution
- Convolution in probability: Sum of independent random variables
So far, we have looked into various aspects of convolution. One of its important applications is in probability: thanks to the convolution, we can obtain the probability density function (pdf) of a sum of two independent random variables (RVs). It turns out that the pdf of that sum is a convolution of pdfs of the two random variables.
In this article, we will show the proof of this theorem. This proof takes advantage of the convolution property of the Fourier transform.
Convolution Theorem in Probability
The probability density function of a sum of statistically independent random variables is the convolution of the contributing probability density functions.
Before we conduct the actual proof, we need to introduce the concept of the characteristic function.
The Characteristic Function
The characteristic function of a random variable is the Fourier transform of its probability density function with a negated argument :
Let us observe that
Another building block of the proof is the independence assumption which we examine next.
Independence of Random Variables
Two random variables are called statistically independent if their joint probability density function factorizes into the respective pdfs of the RVs.
If we have two RVs, and , they are independent if and only if
where is the joint pdf of and (probability density of all possible combinations of and values).
Sum of Two Independent Random Variables
Now to the main part of the proof!
We have two independent random variables, and , with probability density functions and respectively. We want to know what is the probability density function of the sum of and , i.e., what is the formula for . To discover that formula, we calculate the characteristic function of :
Note that we could separate the integrals only thanks to the independence of the two random variables: splitting into a product of and .
Convolution Property of the Fourier Transform
We found out that the characteristic function of a sum of two independent random variables is equal to the product of the individual characteristic functions of these random variables (Equation 4). Additionally, the characteristic function of a random variable with a negated argument is the Fourier transform of this RV’s probability density function (Equation 3). We thus have
The convolution property of the Fourier transform tells us that the multiplication in the Fourier domain is equivalent to convolution in the other domain (here: the domain of the random variable). Therefore,
what concludes the proof .
Note: is used instead of as the argument of in Equation 7 because it doesn’t matter what letter we use; , , and are all pdfs of one-dimensional random variables.
This proof can be extended to arbitrarily many random variables with the requirement that all of them are mutually independent.
In this article, we have proven that the probability distribution of a sum of independent random variables is a convolution of probability distributions of these random variables.
 Walter Kellermann, Statistical Signal Processing Lecture Notes, Winter Semester 2019/2020, University of Erlangen-Nürnberg.
Comments powered by Talkyard.