Category: Probability

Multivariate Normal Bookkeeping
For random variables $X_1,\dots, X_n$ , denote $\mathbf X := (X_1,\dots, X_n)^{\mathrm T}$ and define

$\mathbb E[\mathbf X] = (\mathbb E[X_1], \dots, \mathbb E[X_n])^{\mathrm T}.$

Define the covariance matrix $\boldsymbol{ \Sigma }_{\mathbf X} \in \mathcal M_{n \times n}( \mathbb R )$ by

$\displaystyle (\boldsymbol{ \Sigma }_{\mathbf X})_{i,j} = \mathrm{Cov}(X_i, X_j).$

Note that $\boldsymbol{ \Sigma }_{\mathbf X} = \mathbb E[(\mathbf X - \mathbb E[\mathbf X]) (\mathbf X - \mathbb E[\mathbf X])^{\mathrm T}]$ .

Problem 1. For i.i.d. $Z_1,\dots, Z_n \sim \mathcal N(0, 1)$ , define $\mathbf Z := (Z_1,\dots, Z_n)^{\mathrm T}$ . Calculate the p.d.f. $f_{\mathbf Z}$ of $\mathbf Z$ and evaluate $\boldsymbol{ \Sigma }_{\mathbf Z}$ .

(Click for Solution)

Solution. For each $I$ , the p.d.f. $f_{Z_i}$ of $Z_i$ is

$\displaystyle f_{Z_i}(z_i) = \frac{1}{\sqrt{2\pi}} e^{-\frac{z_i^2}{2}}.$

Since $Z_1,\dots, Z_n$ are independent,

$\begin{aligned} f_{\mathbf Z}(\mathbf z) &= \prod_{i=1}^n f_{Z_i}(z_i) = \frac 1{ \sqrt{ (2\pi)^{n} } } e^{-\frac 12 \sum_{i=1}^n z_i^2} = \frac 1{ \sqrt{ (2\pi)^{n} } } e^{-\frac 12 \mathbf z^{\mathrm T} \mathbf z}.\end{aligned}$

Furthermore,

${ ( \boldsymbol{ \Sigma }_{\mathbf Z} )}_{i,j} = \mathrm{Cov}(Z_i, Z_j) = \mathbb I_{\{0\}} (i-j) = \mathbf I_{i,j},$

where $\mathbf I \equiv \mathbf I_n$ denotes the $n \times n$ identity matrix.

Problem 2. For any invertible matrix $\mathbf A \in \mathcal M_{n \times n}(\mathbb R)$ and $\boldsymbol{\mu} \in \mathbb R^n$ , define $\mathbf X = \boldsymbol{\mu} + \mathbf A \mathbf Z$ . Calculate the p.d.f. $f_{\mathbf X}$ of $\mathbf X$ and evaluate $\boldsymbol{ \Sigma }_{\mathbf X}$ .

(Click for Solution)

Solution. We first assume the case $\boldsymbol{ \mu } = \mathbf 0$ . Define the bijective and continuous map. We observe that for $K \in \frak{B}(\mathbb R^n)$ ,

$\mathbb P_{\mathbf X}(K) = \mathbb P(\mathbf X \in K) = \mathbb P(\mathbf Z \in \mathbf A^{-1}(K)) = (\mathbb P_{\mathbf Z} \circ \mathbf A^{-1})(K).$

That is, $\mathbb P_{\mathbf X} = \mathbb P_{\mathbf Z} \circ \mathbf A^{-1}$ is the pushforward measure of $\mathbb P_{\mathbf Z}$ under $g$ . Letting $\lambda$ denote the $n$ -dimensional Lebesgue measure, we leave it as an exercise in linear algebra to verify that

$\displaystyle \frac{ \mathrm d\lambda(\mathbf A^{-1}\mathbf x) }{ \mathrm d\lambda(\mathbf x) } = |\mathbf A^{-1}| = \frac{1}{|\mathbf A|}.$

By a change of variables, for any $K \in \frak{B}(\mathbb R^n)$ ,

$\begin{aligned} \mathbb P_{\mathbf X} (K) &= \int_{\mathbb R^n} \mathbb I_K\, \mathrm d\mathbb P_{\mathbf X} \\ &= \int_{\mathbb R^n} \mathbb I_K\, \mathrm d\mathbb P_{\mathbf X} \\ &= \int_{\mathbb R^n} \mathbb I_K \circ \mathbf A \, \mathrm d\mathbb P_{\mathbf Z} \\ &= \int_{\mathbb R^n} \mathbb I_K( \mathbf A\mathbf z ) \cdot f_{\mathbf Z}(\mathbf z) \, \mathrm d\lambda(\mathbf z) \\ &= \int_{\mathbb R^n} \mathbb I_K(\mathbf x) \cdot f_{\mathbf Z}(\mathbf A^{-1} \mathbf x ) \, \mathrm d\lambda(\mathbf A^{-1} \mathbf x ) \\ &= \int_{\mathbb R^n} \mathbb I_K(\mathbf x) \cdot f_{\mathbf Z}(\mathbf A^{-1} \mathbf x ) \cdot \frac{1}{|\mathbf A|}\, \mathrm d\lambda(\mathbf x) \\ &= \int_{K} f_{\mathbf Z}(\mathbf A^{-1} \mathbf x ) \cdot \frac{1}{|\mathbf A|}\, \mathrm d\lambda(\mathbf x).\end{aligned}$

By the Radon-Nikodým theorem,

$\displaystyle \int_K f_{\mathbf X}(\mathbf x)\, \mathrm d\lambda(\mathbf x) = \mathbb P_{\mathbf X} (K) = \int_{K} f_{\mathbf Z}(\mathbf A^{-1} \mathbf x ) \cdot \frac{1}{|\mathbf A|}\, \mathrm d\lambda(\mathbf x),$

so that

$\begin{aligned} f_{\mathbf X}(\mathbf x) &= \frac{1}{|\mathbf A|} \cdot f_{\mathbf Z}(\mathbf A^{-1}\mathbf x) \\ &= \frac 1{ |\mathbf A| \cdot \sqrt{ (2\pi)^{n} } } \cdot e^{-\frac 12 \mathbf x^{\mathrm T} (\mathbf A\mathbf A^{\mathrm T})^{-1} \mathbf x } \\ &= \frac 1{ \sqrt{ (2\pi)^{n} \cdot |\mathbf A \mathbf A^{\mathrm T}| } } \cdot e^{-\frac 12 \mathbf x^{\mathrm T} (\mathbf A\mathbf A^{\mathrm T})^{-1} \mathbf x }.\end{aligned}$

Furthermore,

$\begin{aligned} \boldsymbol{\Sigma}_{\mathbf X} &= \mathbb E[(\mathbf A \mathbf Z - \mathbb E[\mathbf A \mathbf Z]) (\mathbf A \mathbf Z - \mathbb E[\mathbf A \mathbf Z])^{\mathrm T}] \\ &= \mathbb E[\mathbf A (\mathbf Z - \mathbb E[\mathbf Z]) (\mathbf Z - \mathbb E[\mathbf Z])^{\mathrm T}\mathbf A^{\mathrm T} ] \\ &= \mathbf A \mathbb E[(\mathbf Z - \mathbb E[\mathbf Z]) (\mathbf Z - \mathbb E[\mathbf Z])^{\mathrm T} ]\mathbf A^{\mathrm T} \\ &= \mathbf A \boldsymbol{\Sigma}_{\mathbf Z} \mathbf A^{\mathrm T} = \mathbf A \mathbf A^{\mathrm T} . \end{aligned}$

In the marginally more general case, $\mathbf X - \boldsymbol{\mu} = \mathbf A \mathbf Z$ . We leave it as an exercise to verify that $\mathbb E[\mathbf X] = \boldsymbol{ \mu }$ and $\boldsymbol{ \Sigma }_{\mathbf X} = \boldsymbol{ \Sigma }_{\mathbf A \mathbf Z} = \mathbf A\mathbf A^{\mathrm T}$ , so that finally

$\begin{aligned} f_{\mathbf X}(\mathbf x) = f_{\mathbf A \mathbf Z}(\mathbf x - \boldsymbol{\mu}) &= \frac 1{ \sqrt{ (2\pi)^{n} \cdot |\boldsymbol{ \Sigma }_{\mathbf A \mathbf Z}| } } \cdot e^{-\frac 12 (\mathbf x - \boldsymbol{\mu})^{\mathrm T} \boldsymbol{ \Sigma }_{\mathbf A \mathbf Z}^{-1} (\mathbf x - \boldsymbol{\mu}) }. \end{aligned}$

Definition 1. A random variable $\mathbf X$ is multivariate normal, denoted $\mathbf X \sim \mathcal N(\boldsymbol{ \mu }, \boldsymbol{ \Sigma })$ , if it has a p.d.f. $f_{\mathbf X}$ given by

$\displaystyle f_{\mathbf X}(\mathbf x) = \frac{1}{ \sqrt{ (2\pi)^n \cdot |\boldsymbol{ \Sigma } | } } \cdot e^{-\frac 12 (\mathbf x - \boldsymbol{\mu})^{\mathrm T} \boldsymbol{\Sigma}^{-1} (\mathbf x - \boldsymbol{\mu})}.$

In this case, we say that the mean of $\mathbf X$ is $\mathbb E[ \mathbf X ] = \boldsymbol{\mu}$ , and the covariance of $\mathbf X$ is $\boldsymbol{\Sigma}_{\mathbf{X}} = \boldsymbol{\Sigma}$ . For example, $\mathbf Z \sim \mathcal N(\mathbf 0, \mathbf I_n)$ .

Problem 3. Fix $\mathbf X \sim \mathcal N(\boldsymbol{ \mu }, \boldsymbol{\Sigma} )$ .
- Prove that if $\boldsymbol{\Sigma}$ is diagonal whose diagonal entires are positive $\sigma_i^2 > 0$ , then $X_1,\dots, X_n$ are independent with $X_i \sim \mathcal N( \mu_i, \sigma_i^2 )$ .
- Prove that for any invertible $\mathbf A \in \mathcal M_{n \times n}(\mathbb R)$ and $\boldsymbol{ \nu } \in \mathbb R^n$ , $\mathbf Y := \boldsymbol{\nu} + \mathbf A \mathbf X \sim \mathcal N(\boldsymbol{ \mu }_{\mathbf Y}, \boldsymbol{ \Sigma }_{\mathbf Y})$ for constants $\boldsymbol{ \mu }_{\mathbf Y}, \boldsymbol{ \Sigma }_{\mathbf Y}$ to be calculated.
(Click for Solution)

Solution. For the first point, if $\boldsymbol{\Sigma}$ is diagonal with $\boldsymbol{ \Sigma }_{i,i} = \sigma_i^2 > 0$ , then $\boldsymbol{\Sigma}^{-1}$ is diagonal with $\boldsymbol{ \Sigma }_{i,i} = 1/\sigma_i^2 > 0$ , and

$\displaystyle (\mathbf x - \boldsymbol{\mu})^{\mathrm T} \boldsymbol{\Sigma}^{-1} (\mathbf x - \boldsymbol{\mu}) = \sum_{i=1}^n \frac{ (x_i - \mu_i)^2 }{ \sigma_i^2 }.$

Some algebra then yields

$\begin{aligned} f_{\mathbf X}(\mathbf x) &= \frac{1}{ \sqrt{ (2\pi)^n \cdot |\boldsymbol{ \Sigma } | } } \cdot e^{-\frac 12 (\mathbf x - \boldsymbol{\mu})^{\mathrm T} \boldsymbol{\Sigma}^{-1} (\mathbf x - \boldsymbol{\mu})} \\ &= \prod_{i=1}^n \frac{1}{\sigma_i \sqrt{ 2\pi } } \cdot e^{-\frac 12 \sum_{i=1}^n \frac{ (x_i - \mu_i)^2 }{ \sigma_i^2 } } \\ &= \prod_{i=1}^n \frac{1}{\sigma_i \sqrt{ 2\pi } } \cdot \prod_{i=1}^n e^{-\frac{ (x_i - \mu_i)^2 }{ 2\sigma_i^2 } } \\ &= \prod_{i=1}^n \frac{1}{\sigma_i \sqrt{ 2\pi } } e^{-\frac{ (x_i - \mu_i)^2 }{ 2\sigma_i^2 } } = \prod_{i=1}^n f_{X_i}(x_i). \end{aligned}$

Therefore, $X_1,\dots, X_n$ are independent with $X_i \sim \mathcal N(\mu_i , \sigma_i^2)$ .

For the second point, follow the proof in Problem 2 with needful bookkeeping to obtain

$\begin{aligned} f_{\mathbf Y}(\mathbf y) &= f_{\mathbf A\mathbf X}(\mathbf y - \boldsymbol{\nu}) = \frac 1{|\mathbf A|} \cdot f_{\mathbf X}(\mathbf A^{-1}(\mathbf y - \boldsymbol{\nu})), \end{aligned}$

which simplifies to Definition 1 with $\mathbb E[\mathbf Y] = \boldsymbol{\nu} + \mathbf A \boldsymbol{\mu}$ and $\boldsymbol{\Sigma}_{\mathbf Y} = \mathbf A \boldsymbol{\Sigma}_{\mathbf X} \mathbf A^{\mathrm T}$ .

Problem 4. Let $\mathbf X \sim \mathcal N(\boldsymbol{ \mu } , \boldsymbol{ \Sigma })$ . Prove that there exists an invertible matrix $\mathbf A \in \mathcal M_{n \times n}(\mathbb R)$ such that $\mathbf X = \boldsymbol{ \mu } + \mathbf A \mathbf Z$ . In particular, each $X_i$ is normally distributed.

(Click for Solution)

Solution. By definition,

$\displaystyle f_{\mathbf X}(\mathbf x) = \frac{1}{ \sqrt{ (2\pi)^n \cdot |\boldsymbol{ \Sigma } | } } \cdot e^{-\frac 12 (\mathbf x - \boldsymbol{\mu})^{\mathrm T} \boldsymbol{\Sigma}^{-1} (\mathbf x - \boldsymbol{\mu})}.$

We first assume $\boldsymbol{\mu} = \mathbf 0$ . If we can find some invertible matrix $\mathbf B$ such that $\mathbf B \mathbf X = \mathbf Z$ , then we can set $\mathbf A := \mathbf B^{-1}$ . To that end, we note that $\boldsymbol{\Sigma}$ is symmetric since

$\displaystyle \boldsymbol{\Sigma}_{i,j} = \mathrm{Cov}(X_i, X_j) = \mathrm{Cov}(X_j, X_i) = \boldsymbol{\Sigma}_{j,i} > 0.$

By the spectral theorems, $\boldsymbol{\Sigma}$ is orthogonally diagonalisable. That is, there exists an orthogonal matrix $\mathbf P$ (i.e. $\mathbf P^{-1} = \mathbf P^{\mathrm T}$ ) and a diagonal matrix $\mathbf D$ such that

$\mathbf{P}^{\mathrm T} \boldsymbol{\Sigma} \mathbf{P} = \mathbf{D}\quad \Rightarrow \quad \boldsymbol{\Sigma} = \mathbf{P} \mathbf{D} \mathbf{P}^{\mathrm T}.$

Furthermore, the entries in the diagonal of $\mathbf D$ are all positive, denoted $\sigma_i^2 > 0$ . Define $\boldsymbol{\Delta}$ by

$\displaystyle \boldsymbol{\Delta}_{i,j} := \begin{cases} \sigma_i, & i = j, \\ 0, & i \neq j. \end{cases}$

Then $\boldsymbol{\Delta}\boldsymbol{\Delta}^{\mathrm T} = \mathbf{D}$ , so that

$\boldsymbol{\Sigma} = \mathbf{P} \mathbf{D} \mathbf{P}^{\mathrm T} = \mathbf{P} \boldsymbol{\Delta}\boldsymbol{\Delta}^{\mathrm T} \mathbf{P}^{\mathrm T} = (\mathbf{P} \boldsymbol{\Delta}) (\mathbf{P} \boldsymbol{\Delta})^{\mathrm T}.$

Now define $\mathbf B := ( \mathbf{P} \boldsymbol{\Delta})^{-1}$ . By Problem 3, $\mathbf B \mathbf X$ is multivariate normal. We calculate its expectation via

$\mathbb E[\mathbf B \mathbf X] = \mathbf B \mathbf E[\mathbf X] = \mathbf B \mathbf 0 = \mathbf 0$

and its covariance matrix via

$\begin{aligned}\boldsymbol{\Sigma_{\mathbf B \mathbf X}} &= \mathbb E[(\mathbf B \mathbf X)^{\mathrm T} (\mathbf B \mathbf X)] \\ &= \mathbf B \mathbb E[\mathbf X^{\mathrm T} \mathbf X] \mathbf B^{\mathrm T} \\ &= ( \mathbf{P} \boldsymbol{\Delta})^{-1} (\mathbf{P} \boldsymbol{\Delta}) (\mathbf{P} \boldsymbol{\Delta})^{\mathrm T} (( \mathbf{P} \boldsymbol{\Delta})^{-1})^{\mathrm T} = \mathbf I. \end{aligned}$

Therefore, $\mathbf B \mathbf X \sim \mathcal N(\mathbf 0, \mathbf I)$ . Since $\mathbf Z \sim \mathcal N(\mathbf 0, \mathbf I)$ , we conclude $\mathbf B \mathbf X = \mathbf Z$ , so that $\mathbf X = \mathbf B^{-1}\mathbf Z = \mathbf A \mathbf Z$ , as required.

For the general case, $\mathbf X - \boldsymbol{\mu}$ has expectation $\mathbf 0$ , and so by the first result, there exists some invertible matrix $\mathbf A$ such that

$\mathbf X - \boldsymbol{\mu} = \mathbf A \mathbf Z \quad \Rightarrow \quad \mathbf X = \boldsymbol{\mu} + \mathbf A \mathbf Z,$

as required.

Problem 5. Define $W_1 := \bar X_n$ and for each $i > 1$ , define $W_i := X_i - \bar X_n$ . If $X_1,\dots,X_n \sim \mathcal N(\mu, \sigma^2)$ are i.i.d., prove that $\bar X_n$ and $(W_2,\dots,W_n)$ are independent. Deduce that $\bar X_n$ and $S_n^2$ defined by

$\displaystyle S_n^2 := \frac 1{n-1} \sum_{i=1}^n (X_i - \bar X_n)^2$

are independent.

(Click for Solution)

Solution. For each $i > 1$ ,

$\displaystyle W_i = \sum_{j=1}^n \underbrace{ \left( \mathbb I_{\{ i \}}(j) -\frac{ 1 }{n} \right) }_{\mathbf A_{i,j}} X_j =: \sum_{j=1}^n \mathbf A_{i, j}X_j.$

Furthermore, define $\mathbf A_{1, j} = 1/n$ . Defining $\mathbf W := (W_1,\dots,W_n)^{\mathrm T}$ , $\mathbf W = \mathbf A \mathbf X$ . If $X_1,\dots,X_n$ are i.i.d. normally distributed with variance $\sigma^2$ , then by the converse of the first point in Problem 3 (which holds), $\mathbf X$ is multivariate normal. By the second point in Problem 3, $\mathbf W$ is multivariate normal.

We turn to evaluate its covariance matrix. Firstly, to shorten computations,

$\begin{aligned} \mathrm{Cov}(X_i, \bar X_n) &= \frac 1n \cdot \sum_{k=1}^n \mathrm{Cov}(X_i, X_k) \\ &= \frac 1n \cdot \mathrm{Cov}(X_i, X_i) = \frac{\sigma^2}{n}. \end{aligned}$

Furthermore,

$\begin{aligned} \mathrm{Cov}(\bar X_n, \bar X_n) &= \frac 1n \cdot \sum_{k=1}^n \mathrm{Cov}(X_i, \bar X_n) \\ &= \frac 1n \cdot \sum_{k=1}^n \frac{\sigma^2}{n} = \frac 1n \cdot n \cdot \frac{\sigma^2}n = \frac{\sigma^2}n. \end{aligned}$

Thus, for $j > 1$ ,

$\begin{aligned} (\boldsymbol{\Sigma}_{\mathbf W})_{1,j} &= \mathrm{Cov}(W_1, W_j) \\ &= \mathrm{Cov}(\bar X_n, X_j - \bar X_n) \\ &= \mathrm{Cov}(X_j , \bar X_n) - \mathrm{Cov}(\bar X_n,\bar X_n) \\ &= \frac{\sigma^2}{n} - \frac{\sigma^2}{n} = 0.\end{aligned}$

Therefore, using properties involving the exponential function,

$f_{\mathbf W}(\mathbf w) = f_{W_1}(w_1) \cdot f_{W_2,\dots,W_n}(w_2,\dots,w_n).$

Hence, $W_1$ and $(W_2,\dots,W_n)$ are independent. For the second claim, we first note that

$\displaystyle 0 = \sum_{i=1}^n X_i - n\bar X_n = \sum_{i=1}^n (X_i - \bar X_n) = X_1 - \bar X_n + \sum_{i=2}^n (X_i - \bar X_n),$

so that $X_1 - \bar X_n = -\sum_{i=2}^n (X_i - \bar X_n) = -\sum_{i=2}^n W_i$ . It follows that

$\begin{aligned} S_n^2 &= \frac 1{n-1} \left( (X_1 - \bar X_n)^2 + \sum_{i=2}^n (X_i - \bar X_n)^2 \right) \\ &= \frac 1{n-1} \left( (X_1 - \bar X_n)^2 + \sum_{i=2}^n W_i^2 \right) \\ &= \frac 1{n-1} \left( \left( \sum_{i=2}^n W_i \right)^2 + \sum_{i=2}^n W_i^2 \right). \end{aligned}$

Since $S_n^2$ can be written purely in terms of $(W_2,\dots, W_n)$ , it is independent of $W_1 = \bar X_n$ .

Problem 6. Using the definitions and notation in Problem 5, prove that for any $n \in \mathbb N_{>1}$ ,

$\displaystyle (n-1) \cdot S_n^2 = (n-2) \cdot S_{n-1}^2 + \frac{n-1}{n} \cdot (X_n - \bar X_{n-1})^2.$

Deduce that there exist i.i.d. $Z_1,\dots, Z_{n-1} \sim \mathcal N(0, 1)$ such that

$\displaystyle \frac{(n-1) \cdot S_n^2}{\sigma^2} = \sum_{i=1}^{n-1} Z_i^2.$

(Click for Solution)

Solution. We first observe that

$\begin{aligned} \bar X_n - \bar X_{n-1} &= \frac{1}{n} \cdot \sum_{i=1}^{n-1} X_i + \frac{1}{n} \cdot X_n - \frac 1{n-1} \cdot \sum_{i=1}^{n-1} X_i \\ &= -\frac{1}{n(n-1)} \cdot \sum_{i=1}^{n-1} X_i + \frac{1}{n} \cdot X_n = \frac 1n \cdot ( X_n - \bar X_{n-1} ). \end{aligned}$

Performing algebruh,

$\begin{aligned} &(n-1) \cdot S_n^2 \\ &= \sum_{i=1}^n \left( X_i - \bar X_{n-1} - \frac 1n \cdot ( X_n - \bar X_{n-1})\right)^2 \\ &= \sum_{i=1}^n \left[ (X_i - \bar X_{n-1})^2 - 2 \cdot (X_i - \bar X_{n-1}) \cdot \frac 1n \cdot ( X_n - \bar X_{n-1}) + \frac 1{n^2} \cdot ( X_n - \bar X_{n-1})^2 \right] \\ &= (n-2) \cdot S_{n-1}^2 + (X_n - \bar X_{n-1})^2 - \frac{2}{n} \cdot ( X_n - \bar X_{n-1})^2 + \frac 1{n} \cdot ( X_n - \bar X_{n-1})^2 \\ &= (n-2) \cdot S_{n-1}^2 + \frac{n-1}{n} \cdot (X_n - \bar X_{n-1})^2. \end{aligned}$

We now prove the final equation by induction. In the case $n = 2$ , we have

$\begin{aligned} \frac{S_2^2}{\sigma^2} &= \frac 1{\sigma^2} \cdot \frac 12 \cdot ((X_1 -\bar X_1)^2 + (X_2 -\bar X_1)^2) = \frac{(X_2 - X_1)^2}{2\sigma^2}. \end{aligned}$

Then $Z_1 := (X_2 - X_1)/(\sigma \sqrt 2) \sim \mathcal N(0, 1)$ , so that $S_2^2/\sigma^2 = Z_1^2$ , as required. For the induction step, suppose that the statement holds for $n = k$ . Then

$\displaystyle k \cdot S_{k+1}^2 = (k-1) \cdot S_k^2 + \frac{k}{k+1} \cdot (X_{k+1} - \bar X_k)^2.$

By relabelling the result in Problem 5, we have that $S_k^2$ is independent of $\bar X_k$ and $X_{k+1}$ . By the induction hypothesis, there exist i.i.d. $Z_1,\dots, Z_{k-1} \sim \mathcal N(0, 1)$ such that

$\displaystyle \frac{ (k-1) \cdot S_k^2 }{\sigma^2} := \sum_{i=1}^{k-1} Z_i^2.$

Furthermore, $X_{k+1} - \bar X_k \sim \mathcal N(0, \sigma^2 \cdot (k+1)/k)$ so that

$\displaystyle Z_{k+1} := \frac{\sqrt{k}}{\sqrt{k+1}} \cdot \frac{ X_{k+1} - \bar X_k }{\sigma} = \frac{X_{k+1} - \bar X_k}{ \sigma \cdot \sqrt{ \frac{k+1}{k} } } \sim \mathcal N(0, 1).$

Therefore,

$\displaystyle \displaystyle \frac{ k \cdot S_{k+1}^2 }{ \sigma^2 } = \frac{ (k-1) \cdot S_{k}^2 }{ \sigma^2 } + \frac{k}{k+1} \cdot \frac{ (X_{k+1} - \bar X_{k})^2 }{ \sigma^2 } = \sum_{i=1}^{k-1} Z_i^2 + Z_k^2 = \sum_{i=1}^{k} Z_i^2,$

as required.

—Joel Kindiak, 25 Jul 25, TBCH
December 12, 2025
The Central Limit Theorem

We are now ready to ascend the mountain of proving the central limit theorem, which we will take inspiration from this paper by Calvin Wooyoung Chin.

Lemma 1. Let $X$ be a real-valued random variable with finite expectation. For any $\delta > 0$ ,

$\displaystyle \mathbb P(|X| \geq \delta) \leq \frac{\mathbb E[|X|]}{\delta}.$

This is called Markov’s inequality. Furthermore, if $X$ has finite variance, then

$\displaystyle \mathbb P(|X - \mathbb E[X]| \geq \delta ) \leq \frac{\mathrm{Var}(X)}{\delta^2}.$

This is called Chebychev’s inequality.

Proof. For the first identity,

$\begin{aligned} \mathbb E[|X|] &= \mathbb E[|X| \cdot \mathbb I_{[\delta, \infty)}] + \mathbb E[|X| \cdot \mathbb I_{(-\infty, \delta)}] \\ &\geq \mathbb E[\delta \cdot \mathbb I_{[\delta, \infty)}] + 0\\ &= \delta \cdot \mathbb E[\mathbb I_{[\delta, \infty)} ] \\ &= \delta \cdot \mathbb P(|X| \geq \delta). \end{aligned}$

Dividing by $\delta$ yields the desired result. For Chebychev’s inequality, apply Markov’s inequality to the random variable $(X - \mathbb E[X])^2$ to get

$\begin{aligned} \mathbb P(|X - \mathbb E[X]| \geq \delta) &= \mathbb P( (X - \mathbb E[X])^2 \geq \delta^2 ) \\ &= \frac{\mathbb E[(X - \mathbb E[X])^2]}{\delta^2} \\ &\leq \frac{\mathrm{Var}(X)}{\delta^2}. \end{aligned}$

Chebychev’s inequality is also responsible for our intuition about probability arising from repeated experiments.

Corollary 1. Fix $K \subseteq \Omega$ and denote $p:= \mathbb P(K)$ . Define the i.i.d. random variables $\xi_i$ by $\xi_i = \mathbb I_K$ , so that $\xi_i \sim \mathrm{Ber}(p)$ . Define

$\displaystyle \bar \xi_n := \frac 1n \cdot \sum_{i=1}^n \xi_i.$

Then for any $\delta > 0$ , $\mathbb P(|\bar \xi_n - p| > \delta) \to 0$ . In this case, we say that $\bar \xi_n \to p$ in probability.

Proof. Fix $\delta > 0$ . Since $\mathbb E[\xi_n] = p$ and $\mathrm{Var}(\xi_n) = p(1-p)$ ,

$\displaystyle \mathbb E[\bar \xi_n] = p,\quad \mathrm{Var}(\bar \xi_n) = \frac{p(1-p)}{n}.$

By Chebychev’s inequality,

$\begin{aligned} \mathbb P(|\bar \xi_n - p| > \delta) &\leq \frac{\mathrm{Var}(\bar \xi_n)}{\delta^2} = \frac{p(1-p)}{n \cdot\delta^2}. \end{aligned}$

Taking $n \to \infty$ , $\mathbb P(|\bar \xi_n - p| > \delta) \to 0$ .

Denote $Z \sim \mathcal N(0, 1)$ and its c.d.f. by $\Phi$ .

Lemma 2. Let $X_1, X_2,\dots$ be real-valued random variables. Suppose for any thrice-differentiable $f : \mathbb R \to \mathbb R$ such that $f, f', f'', f'''$ are bounded,

$\displaystyle \lim_{n \to \infty} \mathbb E[f(X_n)] = \mathbb E[f(Z)].$

Then for any $z \in \mathbb R$ ,

$\displaystyle \lim_{n \to \infty} \mathbb P(X_n \leq z) = \Phi(z),\quad z \in \mathbb R.$

Proof. Fix $z \in \mathbb R$ and $\epsilon > 0$ . Since the p.d.f. of $Z$ is continuous, there exists $\delta > 0$ such that

$|\Phi(z \pm \delta) - \Phi(z)| < \epsilon.$

In particular, $\Phi(z+\delta) < \Phi(z) < \Phi(z-\delta) + \epsilon$ . Define thrice-differentiable bounded functions $f, F : \mathbb R \to [0, 1]$ with bounded first, second, and third derivatives such that

$f|_{(-\infty, z-\delta]} = 1,\quad f|_{[z, \infty)} = 0,\quad F|_{(-\infty, z]} = 1,\quad F|_{[z+\delta, \infty)} = 0.$

By hypothesis,

$\displaystyle \lim_{n \to \infty} \mathbb E[f(X_n)] = \mathbb P(Z \leq z -\delta) = \Phi(z-\delta) \geq \Phi(z) - \epsilon.$

Similarly, $\mathbb E[F(X_n)] < \Phi(z) + \epsilon$ . Since $f \leq \mathbb I_{(-\infty, z]} \leq F$ point-wise, for sufficiently large $n$ ,

$\Phi(z) - \epsilon < \mathbb E[f(X_n)] \leq \underbrace{ \mathbb E[ \mathbb I_{(-\infty, z]} ] }_{ \mathbb P(X_n \leq z)} \leq \mathbb E[F(X_n)] < \Phi(z) + \epsilon,$

so that $\displaystyle \lim_{n \to \infty} \mathbb P(X_n \leq z) = \Phi(z)$ , as required.

Theorem 1 (Central Limit Theorem). Let $X_1,X_2,\dots$ be i.i.d. with mean $0$ and variance $1$ . Then for any $z \in \mathbb R$ ,

$\displaystyle \lim_{n \to \infty} \mathbb P(\sqrt n \cdot \bar X_n \leq x) = \Phi(z).$

In this case, we say that $\sqrt n \cdot \bar X_n \to Z$ in distribution.

Proof. Fix i.i.d. $Y_1,Y_2,\dots \sim \mathcal N(0, 1)$ that are independent of $X_1,X_2,\dots$ . For each $n \in \mathbb N$ and $0 \leq i \leq n$ , define

$\displaystyle Z_{n,i} := \frac 1{\sqrt{n}} \cdot \left( \sum_{j=1}^i X_j + \sum_{j=i+1}^n Y_j \right),\quad S_{n,i} := Z_{n,i} - \frac 1{\sqrt{n}} \cdot X_i.$

By direct computations, $Z_{n,n} = \sqrt{n} \cdot \bar X_n$ and $Z_{n,0} = Z \sim \mathcal N(0, 1)$ . The key idea is using Lemma 2 to conclude our proof. Fix any thrice-differentiable $f : \mathbb R \to \mathbb R$ such that $f, f', f'', f'''$ are bounded. We claim that

$\displaystyle \lim_{n \to \infty} \mathbb E[f(Z_{n,n})] = \mathbb E[f(Z_{n,0})].$

Fix $\epsilon > 0$ . We aim to find $N \in \mathbb N$ such that for $n > N$ ,

$\displaystyle |\mathbb E[f(Z_{n,n})] - \mathbb E[f(Z_{n,0})]| < \epsilon.$

Applying Taylor’s theorem for the interval containing $S_{n,i}$ and $Z_{n,i}$ , there exists $C_{n,i}$ between them such that

$\displaystyle f(Z_{n_i}) = f(S_{n,i}) + f'(S_{n,i}) \cdot (Z_{n,i} - S_{n,i}) + \frac{f''(C_{n,i})}{2!} \cdot (Z_{n,i} - S_{n,i})^2.$

Performing algebra yields

$\displaystyle f(Z_{n,i}) - f(S_{n,i}) - f'(S_{n,i}) \cdot \frac{X_i}{\sqrt n} - f''(S_{n,i}) \cdot \frac{X_i^2}{2 n} = \underbrace{ \frac{ ( f''(C_{n,i}) - f''(S_{n,i}) ) X_i^2}{2n} }_{R_{n,i}}.$

Since $f'''$ is bounded, there exists $\delta > 0$ such that

$\delta \cdot \| f''' \|_\infty \leq \epsilon.$

By the mean value theorem, whenever $|x - y| \leq \delta$ ,

$|f''(x) - f''(y)| \leq \| f''' \|_\infty \cdot |x - y| \leq \delta \cdot \| f''' \|_\infty \leq \epsilon.$

Consider the “good event” $G := \{ |X_i| \leq \delta \sqrt{n} \}$ and its complement

$G^c := \Omega \backslash G = \{ |X_i| > \delta \sqrt{n} \}.$

By our bound on the second derivative via the mean value theorem, we can bound $R_{n,i}$ in the “good event” $G$ by

$\displaystyle |f''(C_{n,i}) - f''(S_{n,i})| \leq \epsilon \quad \Rightarrow \quad \mathbb E[|R_{n,i}| \cdot \mathbb I_G] \leq \frac{\epsilon}{2n} \cdot \mathbb E[X_i^2] = \frac{\epsilon}{2n}.$

On the other hand, in the “bad event” $G^c$ , since $\| f'' \|_\infty \leq M$ for some $M > 0$ ,

$\displaystyle \mathbb E[|R_{n,i}| \cdot \mathbb I_{G^c}] \leq \frac{M}{n} \cdot \mathbb E[X_i^2 \cdot \mathbb I_{G^c}] \leq \frac Mn \cdot \mathbb E[\mathbb I_{G^c}].$

By Chebychev’s inequality,

$\begin{aligned}\mathbb E[\mathbb I_{G^c}] &= \mathbb P(|X_i| \geq \delta \sqrt n) \leq \frac{1}{\delta \cdot \sqrt{n}^2} = \frac{1}{n \delta}.\end{aligned}$

Therefore,

$\displaystyle \mathbb E[|R_{n,i}|] = \mathbb E[|R_{n,i}| \cdot \mathbb I_G] + \mathbb E[|R_{n,i}| \cdot \mathbb I_{G^c}] \leq \left( \frac{\epsilon}{2} + \frac{M}{n\delta} \right) \cdot \frac 1n.$

Thus, selecting $N > 2M/(\epsilon \delta)$ yields $\mathbb E[|R_{n,i}|] \leq \epsilon/n$ . Plugging in the left-hand side of $R_{n,i}$ ,

$\displaystyle |\mathbb E[f(Z_{n,i})] - \mathbb E[f(S_{n,i})] - \mathbb E[f''(S_{n,i}) ] / 2n |\leq \frac{\epsilon}{n}.$

The argument remains almost unchanged with $Z_{n,i}$ replaced with $Z_{n,i-1}$ , so that we have the bound

$\displaystyle |\mathbb E[f(Z_{n,i-1})] - \mathbb E[f(S_{n,i})] - \mathbb E[f''(S_{n,i}) ] / 2n |\leq \frac{\epsilon}{n}.$

Applying the triangle inequality,

$\displaystyle |\mathbb E[f(Z_{n,i})] - \mathbb E [f(Z_{n,i-1})]| \leq \frac{\epsilon}{n} + \frac{\epsilon}{n} = \frac{2\epsilon}{n}.$

Finally, applying a telescoping series,

$\begin{aligned} |\mathbb E[f(Z_{n,n})] - \mathbb E [f(Z_{n,0})]| &\leq \sum_{i=1}^n |\mathbb E[f(Z_{n,i})] - \mathbb E [f(Z_{n,i-1})]| \leq \sum_{i=1}^n \frac{2\epsilon}n = 2\epsilon. \end{aligned}$

Replace $\epsilon$ with $\epsilon/3$ to complete the argument.

With that, we are done with probability…for now. There are many directions we can take from this point on. We could have proven this result using techniques involving characteristic functions or even Brownian motion, but those topics will require their entire blog sections in order to properly discuss. We conclude with the famous normal approximation to the binomial and Poisson distributions.

Corollary 1. For $X_n \sim \mathrm{Bin}(n, p)$ ,

$\displaystyle \lim_{n \to \infty} \mathbb P \left( \frac{X_n - np}{\sqrt{np(1-p)}} \leq z \right) = \Phi(z).$

This is the normal approximation to the binomial distribution.

Proof. For each $n$ , write $X_n := \sum_{i=1}^n \xi_i$ for i.i.d. $\xi_i \sim \mathrm{Ber}(p)$ and

$\displaystyle Y_n := \frac{ X_n - np }{ \sqrt{p(1-p)} } .$

Then

$\displaystyle \bar Y_n := \frac 1n \cdot Y_n = \frac 1n \cdot \sum_{i=1}^n \frac{ \xi_i - p }{\sqrt{p(1-p)}} = \frac{X_n - np}{n \sqrt{p(1-p)}}.$

Since the $(\xi_i - p)/\sqrt{p(1-p)}$ are i.i.d. with mean $0$ and variance $1$ , by the central limit theorem,

$\displaystyle \lim_{n \to \infty} \mathbb P \left( \frac{X_n - np}{\sqrt{np(1-p)}} \leq z \right) = \lim_{n \to \infty} \mathbb P ( \sqrt n \cdot \bar Y_n \leq z) = \Phi(z).$

Corollary 2. We have

$\displaystyle \lim_{\lambda \to \infty} \mathbb P \left( \frac{X - \lambda}{\sqrt{\lambda}} \leq z \right) = \Phi(z), \quad X \sim \mathrm{Pois}(\lambda).$

This is the normal approximation to the Poisson distribution.

Proof. Assume $\lambda \in \mathbb N^+$ . Write $X_{\lambda} := \sum_{i=1}^{\lambda} \xi_i$ for i.i.d. $\xi_i \sim \mathrm{Pois}(1)$ and

$\displaystyle Y_\lambda := X_{\lambda} - \lambda .$

Then

$\displaystyle \bar Y_{\lambda} = \frac{X_{\lambda} - \lambda}{\lambda } = \frac 1{\lambda} \cdot \sum_{i=1}^{\lambda} (\xi_i - 1).$

Since the $\xi_i - 1$ are i.i.d. with mean $0$ and variance $1$ , by the central limit theorem,

$\displaystyle \lim_{\lambda \to \infty} \mathbb P \left( \frac{X_{\lambda} - \lambda}{ \sqrt{\lambda} } \leq z \right) = \lim_{n \to \infty} \mathbb P ( \sqrt{ \lambda } \cdot \bar Y_\lambda \leq z) = \Phi(z).$

—Joel Kindiak, 31 Jul 25, 1359H

December 9, 2025
Adding Random Variables…Revisited

We are now justified in adding random variables. For instance, if $X, Y \sim \mathcal N(0, 1)$ , what is the distribution of $X + Y$ ? Unfortunately, in the most extreme cases, this answer is trivial.

Lemma 1. Let $X \sim \mathcal N(0, 1)$ and $Y = -X$ . Then $Y \sim \mathcal N(0, 1)$ and $X + Y = 0$ .

Proof. We observe that for any $K \in \frak{B}(\mathbb R)$ , since $f_X(\cdot) = f_X(-\cdot)$ , by a change of variables,

$\begin{aligned} \mathbb P_Y(K) = \mathbb P_X(-K) &= \int_{-K} f_X(x)\, \mathrm dx \\ &= \int_K f_X(-x)\, \mathrm dx \\ &= \int_K f_X(x)\, \mathrm dx = \mathbb P_X(K).\end{aligned}$

Therefore, for any $K \in \frak{B}(\mathbb R)$ ,

$\displaystyle \int_K f_Y\, \mathrm d\lambda = \int_K f_X\, \mathrm d\lambda.$

Therefore, $f_Y = f_X$ so that $Y \sim \mathcal N(0, 1)$ .

Here, it is clear that $Y$ is entirely dependent— $100\%$ correlated even—to the random variable $X$ . We got a rather useless answer to our question in this case. If instead we swung to the other extreme, namely $X,Y$ being independent, we don’t have a general formula for any $X, Y$ . Nevertheless, the independent case proves to be far more useful in reality—for instance, the exam scores of two students are effectively independent barring (sufficiently drastic) externalities.

Therefore, let’s discuss what it means for two random variables $X,Y : \Omega \to \mathbb R$ to be independent. We have previously seen that two events $K,L$ are called independent if $\mathbb P(K \cap L) = \mathbb P(K) \cdot \mathbb P(L)$ .

Lemma 2. The set $\sigma(X) := \{X^{-1}(K) : K \in \frak{B}(\mathbb R)\} \subseteq \mathcal F$ forms a $\sigma$ -algebra, called the $\sigma$ -algebra generated by $X$ .

Given two $\sigma$ -algebras $\mathcal F_1, \mathcal F_2 \subseteq \mathcal F$ , also known as sub- $\sigma$ -algebras of $\mathcal F$ , we say that $\mathcal F_1, \mathcal F_2$ are independent if for any $K \in \mathcal F_1, L \in \mathcal F_2$ , $K, L$ are independent.

Definition 1. Two random variables $X, Y$ are said to be independent if $\sigma(X), \sigma(Y)$ are independent.

Suppose $\mathbb P_X \ll \mu$ and $\mathbb P_Y \ll \mu$ , where $\mu$ denotes either the counting measure $|\cdot|$ or the Lebesgue measure $\lambda$ . Let $\mu^2$ be the product of two copies of $\mu$ . Suppose $(X,Y) \ll \mu^2$ .

Theorem 1. $X, Y$ are independent and $(X,Y) \ll \mu^2$ if and only if

$\displaystyle \mathbb P((X, Y) \in K) = \int_K f_X \cdot f_Y\, \mathrm d\mu^2,$

where the integrand in the right-hand side is interpreted to mean

$f_X(\cdot , y) = f_X,\quad f_Y(x, \cdot) = f_Y, \quad x, y \in \mathbb R.$

Proof. Consider the cumulative distribution functions

$F_X(x) := \mathbb P(X \in (-\infty, x]) \equiv \mathbb P(X \leq x),\quad F_Y(y) := \mathbb P( Y \leq y).$

By the Radon-Nikodým theorem,

$\displaystyle F_X(x) = \int_{(-\infty, x]} f_X\, \mathrm d\mu,\quad F_Y(y) = \int_{(-\infty, y]} f_Y\, \mathrm d\mu.$

In the direction $(\Leftarrow)$ , by the Fubini-Tonelli theorem,

$\begin{aligned} \mathbb P(X^{-1}((-\infty, x]) \cap Y^{-1}((-\infty, y])) &= \mathbb P((X,Y) \in (-\infty, x] \times (-\infty, y]) \\ &= \int_{(-\infty, x] \times (-\infty, y]} f_{X,Y}\, \mathrm d\mu^2 \\ &= \int_{(-\infty, x]} \int_{(-\infty, y]} f_{X,Y}\, \mathrm d \mu\, \mathrm d \mu \\ &= \int_{(-\infty, x]} \int_{(-\infty, y]} f_X \cdot f_Y\, \mathrm d \mu\, \mathrm d \mu \\ &= \int_{(-\infty, x]} f_X \, \mathrm d \mu \cdot \int_{(-\infty, y]} f_Y\, \mathrm d \mu \\ &= \mathbb P(X^{-1}(-\infty, x]) \cdot \mathbb P(Y^{-1}(-\infty, y]), \end{aligned}$

establishing independence.

In the direction $(\Rightarrow)$ , we similarly use the Fubini-Tonelli theorem to obtain

$\begin{aligned} \iint_{(-\infty, x] \times (-\infty, y]} f_{X,Y}\, \mathrm d\mu^2 &= \int_{(-\infty, x]} f_X\, \mathrm d\mu \cdot \int_{(-\infty, y]} f_Y\, \mathrm d\mu \\ &= \int_{(-\infty, x]} \int_{(-\infty, y]} f_X \cdot f_Y\, \mathrm d \mu\, \mathrm d \mu \\ &= \iint_{(-\infty, x] \times (-\infty, y]} f_X \cdot f_Y\, \mathrm d\mu^2, \end{aligned}$

so that $f_{X,Y} = f_X \cdot f_Y$ .

Henceforth, when $X,Y$ are independent and $X \ll \mu$ and $Y \ll \mu$ , we assume $(X,Y) \ll \mu$ so that $f_{X,Y}$ is a meaningful quantity.

Corollary 1. Define

$\displaystyle F_{X,Y}(x,y) = \mathbb P((X,Y) \in (-\infty, x] \times (-\infty, y]) \equiv \mathbb P(X \leq x, Y \leq y).$

Then $X, Y$ are independent and $(X,Y) \ll \mu^2$ if and only if for any $x, y \in \mathbb R$ ,

$\displaystyle F_{X,Y}(x,y) = F_X(x)\cdot F_Y(y).$

Finally, let’s add these random variables together.

Theorem 1. If $X, Y$ are independent $\mathbb R$ -valued random variables with density functions $f_X, f_Y$ , then $X+Y$ is a random variable with density function

$f_{X+Y} = f_X * f_Y,$

where the right-hand side denotes convolution with respect to $\mu$ :

$\displaystyle (f * g)(u) := \int_{\mathbb R} f \cdot g(u- \cdot )\, \mathrm d\mu \equiv \int_{\mathbb R} f(x) \cdot g(u- x )\, \mathrm d\mu(x).$

If $\mu$ is the counting measure, we get

$\displaystyle (f * g)(u) = \sum_{n \in \mathbb Z} f(n) \cdot g(u-n).$

If $\mu$ is the Lebesgue measure, we get

$\displaystyle (f * g)(u) = \int_{-\infty}^{\infty} f(t) \cdot g(u-t)\, \mathrm dt.$

Proof. Denote $g(x,y) = x+y$ so that for any fixed $x$ , $g(x,y) \leq u$ if and only if $y \leq u - x$ . For any $K \in \frak{B}(\mathbb R)$ ,

$\displaystyle \begin{aligned} \int_{K} f_{X+Y}\, \mathrm d\mu &= \mathbb P(X+Y \in K) \\ &= \mathbb P((X,Y) \in g^{-1}(K)) \\ &= \int_{g^{-1}(K)} f_{X,Y}\, \mathrm d\mu^2 \\ &= \int_{g^{-1}(K)} f_{X,Y}(x,y)\, \mathrm d\mu^2(x,y) \\ &= \int_{\mathbb R} \int_{K-x} f_{X,Y}(x,y)\, \mathrm d\mu(y)\, \mathrm d\mu(x) \\ &= \int_{\mathbb R} \int_{K} f_{X,Y}(x,y-x)\, \mathrm d\mu(y)\, \mathrm d\mu(x) \\ &= \int_{K} \int_{\mathbb R} f_{X,Y}(x,y-x)\, \mathrm d\mu(x)\, \mathrm d\mu(y) \\ &= \int_{K} \underbrace{ \int_{\mathbb R} f_{X}(x) \cdot f_Y(y-x)\, \mathrm d\mu(x) }_{(f_X * f_Y)(y)} \, \mathrm d\mu(y) = \int_{K} f_X * f_Y\, \mathrm d\mu. \end{aligned}$

Finally, let’s concretely add two independently normally distributed random variables.

Theorem 2. If $X \sim \mathcal N(\mu_1, \sigma_1^2)$ and $Y \sim \mathcal N(\mu_2, \sigma_2^2)$ are independent, then $X + Y \sim \mathcal N(\mu_1 + \mu_2, \sigma_1^2 + \sigma_2^2)$ .

Proof. Write $X = \mu_1 + \sigma_1 Z_1$ and $Y = \mu_2 + \sigma_2 Z_2$ . Then

$X + Y = (\mu_1 + \mu_2) + \sigma_1 Z_1 + \sigma_2 Z_2,$

where $Z_1, Z_2 \sim \mathcal N(0, 1)$ are independent. It suffices to prove that

$\sigma_1 Z_1 + \sigma_2 Z_2 \sim \mathcal N(0, \sigma_1^2 + \sigma_2^2).$

By construction, for any $\sigma > 0$ , if $U \sim \mathcal N(0, \sigma^2)$ , then

$\displaystyle f_U(u) = \frac{1}{ \sigma \sqrt{2\pi} } e^{-\frac{u^2}{2\sigma^2} }.$

Defining $W := \sigma_1 Z_1 + \sigma_2 Z_2$ ,

$\begin{aligned} f_W(w) &= (f_{\sigma_1 Z_1} * f_{\sigma_2 Z_2})(w) \\ &= \int_{-\infty}^{\infty} \frac{1}{ \sigma_1 \sqrt{2\pi} } e^{-\frac{t^2}{2\sigma_1^2} } \cdot \frac{1}{ \sigma_2 \sqrt{2\pi} } e^{-\frac{(w-t)^2}{2\sigma_2^2} }\, \mathrm dt \\ &= \frac{1}{\sigma_1 \sigma_2 \cdot 2\pi}\int_{-\infty}^{\infty} \exp\left( - \frac 12 \left( \frac{t^2}{\sigma_1^2} + \frac{(w-t)^2}{\sigma_2^2} \right)\right)\, \mathrm dt. \end{aligned}$

By algebruh,

$\displaystyle \frac{t^2}{\sigma_1^2} + \frac{(w-t)^2}{\sigma_2^2} = \left( \frac{1}{\sigma_1^2} + \frac{1}{\sigma_2^2} \right) t^2 - \frac{2w}{\sigma_2^2} \cdot t + \frac{w^2}{\sigma_2^2} = A(t - h)^2 + k.$

for carefully calculated constants $A, h, k$ , in particular, with

$\displaystyle A = \frac{1}{\sigma_1^2} + \frac{1}{\sigma_2^2},\quad k = \frac{w}{A \sigma_2^2} = -\frac 14 \cdot \frac{4w^2}{\sigma_2^4} \cdot \frac{\sigma_1^2 \cdot \sigma_2^2}{\sigma_1^2 + \sigma_2^2} + \frac{w^2}{\sigma_2^2} = -\frac{w^2}{\sigma_1^2 + \sigma_2^2}.$

Denoting $\sigma_3^2 := 1/A$ , the integral then simplifies to

$\begin{aligned}\int_{-\infty}^{\infty} e^{-\frac 12 (A(t-h)^2 + k)}\, \mathrm dt &= e^{-\frac 12 k} \cdot \sigma_3 \cdot \sqrt{2\pi} \cdot \underbrace{ \int_{-\infty}^{\infty} \frac{1}{\sigma_3 \sqrt{2\pi} }e^{-\frac {(t-h)^2}{2\sigma_3^2}} \, \mathrm dt }_1 \\ &= e^{-\frac 12 k} \cdot \sigma_3 \cdot \sqrt{2\pi}.\end{aligned}$

Therefore, denoting $\sigma^2 := \sigma_1^2 + \sigma_2^2$ so that $k = -w^2/\sigma^2$ ,

$\begin{aligned} f_W(w) &= \frac{\sigma_3}{\sigma_1 \sigma_2} \cdot \frac{1}{\sqrt{2\pi}} \cdot e^{-\frac 12 k} = \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{w^2}{2\sigma^2}}. \end{aligned}$

Therefore, $\sigma_1 Z_1 + \sigma_2 Z_2 = W \sim \mathcal N(0, \sigma^2) = \mathcal N(0, \sigma_1^2 + \sigma_2^2)$ , as required.

In practice, to compute probabilities involving normal distributions, we define

$\displaystyle \Phi(z) := \mathbb P(Z \leq z) = \frac 12 + \frac{1}{\sqrt{2\pi}} \int_0^z e^{-t^2/2}\, \mathrm dt$

and tabulate commonly used approximate values of $\Phi(z)$ for $z > 0$ , known in the statistics community as a $z$ -table. By symmetry, $\Phi(-z) = 1- \Phi(z)$ . For any $X \sim \mathcal N(\mu, \sigma^2)$ , since $X = \mu + \sigma Z$ , we can reduce the computation to

$\displaystyle \mathbb P(X \leq x) = \mathbb P\left( Z \leq \frac{x - \mu}{\sigma} \right) = \Phi \left( \frac{x - \mu}{\sigma}\right).$

By induction, we have the following sampling distribution $\bar X_n$ .

Corollary 2. For independent, identically distributed random variables $X_1,\dots, X_n \sim \mathcal N(\mu, \sigma^2)$ ,

$\displaystyle \bar X_n := \frac{1}{n} \sum_{i=1}^n X_i \sim \mathcal N \left( \mu, \frac{\sigma^2}{n} \right).$

The central limit theorem claims that even if $X_i$ are not normally distributed, $\bar X_n$ converges in distribution to $\mathcal N(\mu, \sigma^2/n)$ . For the limiting distribution to be constant, the equivalent claim is that $(\bar X_n-\mu) / (\sigma / \sqrt{n})$ converges in distribution to $Z \sim \mathcal N(0, 1)$ .

We have previously stated the special case $(\mu, \sigma) = (0, 1)$ and aim to prove it properly using techniques in stochastic calculus. If we have this result, we are emboldened to carry out hypothesis tests, which are commonplace in the STEM fields as well as the data-driven social sciences. We will digress to this application before we dive right back into our ascent toward the central limit theorem.

—Joel Kindiak, 23 Jul 25, 2257H

December 5, 2025
Legitimate Integral Swapping

Given two random variables $X, Y$ , what is the distribution of $X + Y$ ? We could define the discrete random variables, and perhaps continuous ones as well. But let’s go back to measure theory and define the joint distribution $(X, Y)$ as rigorously as possible.

Let $(\Omega, \mathcal F, \mu)$ be a measure space (or a probability space if $\mu$ is a probability measure). Equip $\mathbb R^n$ is equipped with the Borel $\sigma$ -algebra $\frak{B}(\mathbb R^n)$ , generated by open balls under the Euclidean metric.

Lemma 1. Let $X, Y : \Omega \to \mathbb R$ be a random variable. Then $(X,Y) : \Omega \to \mathbb R^2$ defined by $(X,Y)(\omega) := (X(\omega), Y(\omega))$ is a random variable.

Proof. The map $(X,Y)$ is continuous, and therefore, has open sets as pre-images of open sets. Therefore, $(X,Y)$ is $\mathcal F$ / $\frak{B}(\mathbb R^2)$ -measurable.

What would be a reasonable measure on $\mathbb R^2$ ? Intuitively, we should have a measure $\lambda_2$ on $\mathbb R^2$ such that $\lambda_2(K \times L) = \lambda_1(K) \cdot \lambda_1(L)$ , where $\lambda_1$ denotes the usual Lebesgue measure that we painstakingly constructed. In fact, more generally, given measure spaces $(\Omega_i, \mathcal F_i, \mu_i)$ , we would like to define a reasonable $\sigma$ -algebra $\mathcal F$ on $\Omega := \prod_{i=1}^n \Omega_i$ and a measure $\mu$ on $\mathcal F$ such that

$\displaystyle \mu\left( \prod_{i=1}^n K_i \right) = \prod_{i=1}^n \mu_i(K_i),\quad K_i \in \mathcal F_i.$

It turns out that with the help of Carathéodory’s extension theorem, this task isn’t as Sisyphean as it seems.

Theorem 1. Given measure spaces $(\Omega_i, \mathcal F_i, \mu_i)$ , there exists a $\sigma$ -algebra $\mathcal F$ on $\Omega := \prod_{i=1}^n \Omega_i$ and a measure $\mu$ on $\mathcal F$ such that $\prod_{i=1}^n K_i \in \mathcal F$ and

$\displaystyle \mu\left( \prod_{i=1}^n K_i \right) = \prod_{i=1}^n \mu_i(K_i),\quad K_i \in \mathcal F_i.$

Proof. We will prove the special case $n = 2$ for simplicity. Define the algebra

$\mathcal F^0 := \{ K \times L : K \in \mathcal F_1, L \in \mathcal F_2\}.$

For any $K \in \mathcal F_1$ and $x \in \Omega_1$ , define the $x$ -section $K_x \subseteq \Omega_2$ by

$K_x := \{ y \in \Omega_2 : (x, y) \in K\}.$

Define the $y$ -section $K^y$ similarly. Now given $M := \bigcup_{i=1}^n K_i \times L_i \in \mathcal F^0$ , for any $x \in \Omega_1$

$\displaystyle M_x = \bigcup_{i : x \in K_i} L_i \in \mathcal F_2.$

Therefore, the quantity $\mu_2(M_x)$ is well-defined. Similarly, $\mu_1(M^y)$ is well-defined for any $y \in \Omega_2$ . Hence, define the function $f_M(x) := \mu_2(M_x)$ , which is non-negative and simple since in the special case $M = (K_1 \times L_1) \cup (K_2 \times L_2)$ ,

$f_M = \mu_2(L_1 \cup L_2) \cdot \mathbb I_{K_1 \cap K_2} + \mu_2(L_1) \cdot \mathbb I_{K_1 \backslash K_2} + \mu_2(L_2) \cdot \mathbb I_{K_2 \backslash K_1}.$

We can similarly define $g_M(y) := \mu_1(M^y)$ , and define

$\displaystyle \mu_0(M) := \int_{\Omega_1} f_M\, \mathrm d\mu_1 = \int_{\Omega_2} g_M\, \mathrm d\mu_2.$

To see this in the simplest case when $M$ is a disjoint union (the rest follows by careful bookkeeping),

$\displaystyle \int_{\Omega_1} f_M\, \mathrm d\mu_1 = \sum_{i=1}^n \mu_2(L_i) \cdot \mu_1(K_i) = \int_{\Omega_2} g_M\, \mathrm d\mu_2.$

In particular, $\mu_0(K \times L) = \mu_1(K) \cdot \mu_2(L)$ . We claim that $\mu_0$ is countably additive. Fix $M = \bigsqcup_{i=1}^\infty M_i \in \mathcal F^0$ . Then for any $x \in \Omega_1$ ,

$\displaystyle f_M(x) = \mu_2(M_x) = \mu_2 \left( \bigsqcup_{i=1}^\infty (M_i)_x \right) = \sum_{i=1}^\infty \mu_2((M_i)_x) = \sum_{i=1}^\infty f_{M_i}(x).$

Therefore, the function $\sum_{i=1}^n f_{M_i}$ converges monotonically to $f_M$ , and by the monotone convergence theorem,

$\displaystyle \mu_0(M) = \int_{\Omega_1} f_M\, \mathrm d\mu_1 = \sum_{i=1}^\infty \int_{\Omega_1} f_{M_i}\, \mathrm d\mu_1 = \sum_{i=1}^\infty \mu_0(M_i).$

Now apply Carathéodory’s extension theorem to obtain a $\sigma$ -algebra $\mathcal F \supseteq \mathcal F^0$ and a measure $\mu : \mathcal F \to [0, \infty]$ such that $\mu|_{\mathcal F^0} = \mu_0$ .

Theoretically, we could just start defining random variables on the product space $\Omega_1 \times \Omega_2$ and go on our merry way. But we still need to answer a key question: given the distributions $\mathbb P_X$ and $\mathbb P_Y$ , how do we compute $\mathbb P_{X+Y}$ ? In a more abstract manner, we need to integrate with respect to our newly minted measure $\mu$ in a computationally consistent manner with integrals with respect to our old measures $\mu_1,\mu_2$ respectively. Surprisingly, answering this question leads us to one of the most important theorems in multivariable calculus, which is Fubini’s theorem, as it allows us to rigorously swap integrals—a key tool in any reasonable calculation.

Denote the base measure spaces by $\Omega_1 \equiv (\Omega_1,\mathcal F_1,\mu_1)$ and $\Omega_2 \equiv (\Omega_2,\mathcal F_2,\mu_2)$ , and their product space by $\Omega \equiv (\Omega, \mathcal F, \mu)$ . By construction, $\mathcal F \supseteq \mathcal F_1 \times \mathcal F_2$ . We remark that for any $M \in \mathcal F_1 \times \mathcal F_2$ and $x \in \Omega_1, y \in \Omega_2$ , $M_x \in \mathcal F_2$ and $M^y \in \mathcal F_1$ , since the $\sigma$ -algebra

$\{M \in \mathcal F : \forall x \in \Omega_1\ \forall y \in \Omega_2\ M_x \in \mathcal F_1 \wedge M^y \in \mathcal F_2\}$

contains $\mathcal F_1 \times \mathcal F_2$ .

Now we observe that $\mathbb R = \bigcup_{n \in \mathbb Z} [n, n+1)$ and each $[n, n+1)$ has Lebesgue measure $1$ .

Definition 1. A measure space $(\Omega, \mathcal F, \mu)$ is $\sigma$ -finite if there exist $K_1,K_2\dots$ with $\mu(K_i) < \infty$ such that $\Omega = \bigcup_{i=1}^\infty K_i$ . For instance, $\mathbb R^n$ is $\sigma$ -finite.

Lemma 2. Suppose $\Omega_1,\Omega_2$ are $\sigma$ -finite. For any $M \in \mathcal F_1 \times \mathcal F_2$ , the non-negative functions $f_M(x) := \mu_2(M_x)$ and $g_M(y) := \mu_1(M^y)$ are measurable, and define the predicate $\phi$ by

$\displaystyle \phi(M) := \left( \int_{\Omega_1} f_M\, \mathrm d\mu_1 = \mu(M) = \int_{\Omega_2} g_M\, \mathrm d\mu_2 \right).$

Then $\phi(M)$ holds for any $M \in \mathcal F_1 \times \mathcal F_2$ . Note that this result is an extension from that of Theorem 1.

Proof. We first prove the case that $\mu_1(\Omega_1) < \infty$ and $\mu_2(\Omega_2) < \infty$ . It is straightforward that $\phi(M)$ holds if $M \in \mathcal F^0$ or even a disjoint union of sets in $\mathcal F^0$ . If $\phi(M_1)$ and $\phi(M_2)$ , then $\phi(M_1 \backslash M_2)$ . Finally, if $M_1 \subseteq M_2 \subseteq \dots$ and $\phi(M_i)$ , then defining $M = \bigcup_{i=1}^\infty M_i$ , $f_M = \lim_{n \to\infty} f_{M_n}$ is measurable, and by the monotone convergence theorem,

$\displaystyle \int_{\Omega_1} f_M\, \mathrm d\mu_1 = \lim_{n \to \infty} \int_{\Omega_1} f_{M_n}\, \mathrm d\mu_1 = \lim_{n \to \infty} \mu(M_n) = \mu(M).$

Therefore, $\phi(M)$ . Let $\mathcal F' \supseteq \mathcal F_0$ denote the smallest subset of $\mathcal F$ such that these two properties are satisfied. We can verify that $\mathcal F'$ is a $\sigma$ -algebra, and hence contains $\mathcal F_1 \times \mathcal F_2$ , as required.

We now generalise to the $\sigma$ -finite case. Suppose $K_1 \subseteq K_2 \subseteq \dots$ such that $\bigcup_{i=1}^\infty K_i = \Omega_1$ , and $\bigcup_{i=1}^\infty L_i = \Omega_2$ similarly. For each $I$ , define $M_i := M \cap (K_i \times L_i)$ so that $M = \bigcup_{i=1}^\infty M_i$ . The result follows by the monotone convergence theorem.

Lemma 3. For any map $f : \Omega_1 \times \Omega_2 \to [0, \infty]$ , all of its sections

$f_x := f(x, \cdot) : \Omega_2 \to [0, \infty],\quad f^y := f( \cdot, y) : \Omega_1 \to [0, \infty]$

are measurable.

Proof. Apply Lemma 2 to the result $f_x^{-1}([a, \infty]) = (f^{-1}([a, \infty]))_x \in \mathcal F_2$ .

We can now discuss the Fubini-Tonelli theorem. The Fubini theorem is the special case when all integrals therein are finite. Here, a function $f : \Omega \to [-\infty, \infty]$ is integrable if $K := f^{-1}(\{-\infty, \infty\})$ has measure zero and $f \cdot \mathbb I_{\Omega \backslash K}$ is integrable.

Theorem 2 (Fubini-Tonelli Theorem). Suppose $\Omega_1,\Omega_2$ are $\sigma$ -finite. If $f:\Omega_1 \times \Omega_2 \to [-\infty, \infty]$ is either non-negative and measurable (resp. integrable), then the functions $g, h$ defined by

$\displaystyle g(x) := \int_{\Omega_2} f(x, \cdot)\, \mathrm d\mu_2,\quad h(y) := \int_{\Omega_1} f(\cdot, y)\, \mathrm d\mu_1$

are measurable (resp. integrable) and

$\displaystyle \int_{\Omega_1} \int_{\Omega_2} f\, \mathrm d\mu_2\, \mathrm d\mu_1 = \int_{\Omega_2} \int_{\Omega_1} f\, \mathrm d\mu_1\, \mathrm d\mu_2 = \int_{\Omega} f\, \mathrm d\mu.$

Proof. We return to the usual simple $\Rightarrow$ non-negative measurable $\Rightarrow$ integrable strategy. If $f = \mathbb I_M$ , then we obtain this result by Lemma 2. The result extends by linearity to non-negative simple functions.

If $f$ is non-negative, find a sequence of non-negative simple functions $\Phi_n$ that monotonically converge to $f$ . By the monotone convergence theorem,

$\displaystyle \int_{\Omega} f\, \mathrm d\mu = \lim_{n \to \infty} \int_{\Omega} \Phi_n\, \mathrm d\mu.$

For each $n$ , define $\varphi_n$ by setting for each $x \in \Omega_1$ , $\varphi_n(x) := \int_{\Omega_2} \Phi_n(x,\cdot)\, \mathrm d\mu_2$ . Then $\varphi_n$ monotonically increases to $g$ . By the monotone convergence theorem again, since $\Phi_n$ are all step functions,

$\displaystyle \int_{\Omega_1} g\, \mathrm d\mu_1 = \lim_{n \to \infty} \int_{\Omega_1} \varphi_n\, \mathrm d \mu_1 = \lim_{n \to \infty} \int_{\Omega} \Phi_n\, \mathrm d\mu = \int_{\Omega} f\, \mathrm d\mu.$

Finally, in the case $f$ is integrable, write $f = f^+ - f^-$ and perform needful bookkeeping.

As much as we feel somewhat justified to add distributions in general, there is one more measure-theoretic machinery we need to discuss—the technical density function known as the Radon-Nikodým derivative. In doing so, we can be justified in letting $f_X$ denote the density function for any sufficiently nice random variable $X$ .

—Joel Kindiak, 21 Jul 25, 2313H

December 2, 2025
Density Functions on Steroids

We have seen that a random variable $X$ is said to be (absolutely) continuous if there exists a continuous, non-negative, integrable function $f_X$ such that for any Borel-measurable $K$ ,

$\displaystyle \mathbb P_X(K) = \int_K f_X\, \mathrm d\lambda.$

In this case, $\mathbb P_X \ll \lambda$ in the sense that for any Borel-measurable $L$ ,

$\lambda(L) = 0 \quad \Rightarrow \quad \mathbb P_X(L) = 0.$

It turns out that the reverse is true.

Theorem 1 (Radon-Nikodým Theorem). Let $(\Omega, \mathcal F)$ be a measurable space. If $\mu, \nu$ are $\sigma$ -finite measures on $(\Omega, \mathcal F)$ such that $\nu \ll \mu$ , then there exists a measurable function $f : \Omega \to [0, \infty]$ such that

$\displaystyle \nu(K) = \int_K f\, \mathrm d\mu.$

Furthermore, $f$ is unique $\mu$ -a.e., called the Radon-Nikodým derivative, denoted $\displaystyle \frac{\mathrm d \nu}{\mathrm d\mu}$ .

In particular, for any measure $\mathbb P_X$ on $(\mathbb R^n, \frak{B}(\mathbb R^n), \lambda)$ , where $\lambda$ denotes the $n$ -dimensional Lebesgue measure,

$\displaystyle f_X := \frac{\mathrm d\mathbb P_X}{\mathrm d\lambda} \quad \iff \quad \mathrm d\mathbb P_X \equiv f_X\, \mathrm d\lambda$

is called the probability density function of $X$ , where the right-hand side follows a legitimate abuse of notation.

Our goal in this post is to prove the Radon-Nikodým theorem. To achieve that goal, we will need to introduce the notion of a signed measure, and a lemma known as Hahn’s decomposition theorem.

Let $(\Omega, \mathcal F)$ be a measurable space.

Definition 1. The map $\mu : \mathcal F \to \mathbb R$ (excluding infinities) is called a signed measure if $\mu(\emptyset) = 0$ and $\mu$ is countably additive.

Theorem 2 (Hahn Decomposition Theorem). If $\mu$ is a signed measure on $(\Omega, \mathcal F)$ , then there exist disjoint measurable sets $P, N \in \mathcal F$ such that

$\displaystyle \mu( \cdot \cap P) \in [0,\infty),\quad \mu( \cdot \cap N) \in (-\infty, 0],\quad P \sqcup N = \Omega.$

In this case, we call the set $P$ a positive set, denoted $P \geq 0$ , and $N$ a negative set, denoted $N \leq 0$ . A set $M$ is null if it is positive and negative, i.e. $\mu( \cdot \cap M) \in \{0\}$ .

Proof of Theorem 2. We first check that $\alpha := \sup \{\mu(K) : K \geq 0\} < \infty$ . If not, then for any $n \in \mathbb N^+$ , there exists $K_n \in \mathcal F$ with $K_n \geq 0$ such that $\mu(K_n) \geq n$ . Then $K := \bigcup_{n=1}^\infty K_n$ has measure $\mu(K) \geq n$ for any $n$ , leading to $\mu(K) = \infty$ , a contradiction.

Therefore, for any $n \in \mathbb N^+$ , there exists $P_n \geq 0$ such that $\alpha - 1/n < \mu(P_n) \leq \alpha$ . It follows by bookkeeping that $P := \bigcup_{n=1}^\infty P_n$ is positive. We claim that $N:= \Omega \backslash P$ is negative.

Suppose otherwise. Then there exists $K \in \mathcal F$ such that $\mu(K \cap N) > 0$ . Assume $K \subseteq N$ so that $\mu(K) > 0$ . If $K$ is positive, then so is $P \cup K$ , thus $\mu(P \cup K) = \mu(P) + \mu(K) > \alpha$ , a contradiction. Therefore, there exists $K_1 \subseteq K$ such that $\mu(K_1) = \mu(K_1 \cap K) < 0$ . Furthermore, $\mu(K \backslash K_1) = \mu(K) - \mu(K_1) > 0$ .

Define $n_1 := \min \{k \in \mathbb N : \mu(K_1) < -1/k\}$ , which exists by the Archimedean property of $\mathbb R$ . Repeating the argument inductively on the set $K \backslash \bigcup_{j=1}^{i-1} K_j$ , we obtain $K \supseteq K_1 \supseteq K_2 \supseteq \dots$ such that $K_i \subseteq K \backslash \bigcup_{j=1}^{i-1} K_j$ and $\mu(K_i) < -1/{n_i}$ , where $n_i$ is the smallest possible positive integer. Define $L := K \backslash \bigsqcup_{i=1}^\infty K_i$ , which has positive finite measure. Since $\mu$ is countably additive,

$\displaystyle \mu(K) = \mu(L) + \sum_{i=1}^\infty \mu(K_i).$

Hence, $\mu(K_i) \to 0 \iff 1/n_i \to 0$ as $i \to \infty$ .

If we can show that $L \geq 0$ , then $\mu(L \cup P) > \alpha$ , yielding the desired contradiction. To that end, suppose for a contradiction that there exists $M \subseteq L$ such that $\mu(M) = \mu(M \cap L) < 0$ . Then $\mu(M) < -1/(n_i - 1)$ for some $n_i$ , contradicting the construction of $K_i$ , as required.

Now we prove the Radon-Nikodým theorem.

Proof of Theorem 1. We first assume that $\mu, \nu$ are finite. For any $n, k$ , define the signed measure $\lambda_{n,k} := \nu - \frac{k}{2^n} \cdot \mu : \mathcal F \to \mathbb R$ . By the Hahn decomposition theorem, there exist disjoint positive and negative sets $P_{n,k}, N_{n,k}$ with respect to $\lambda_{n,k}$ such that $P_{n,k} \sqcup N_{n,k} = \Omega$ .

Since $k\cdot 2^{-n} \geq l \cdot 2^{-m}$ implies $P_{n,k} \subseteq P_{m,l}$ , for any $n$ , $( P_{n,k} )$ is decreasing in $k$ . Define the set $P_n := \bigcap_{k=1}^\infty P_{n,k}$ , which is positive with respect to all $\lambda_{n,k}$ . Since this set is positive with respect to $\lambda_{n,k}$ ,

$\displaystyle \lambda_{n,k}(P_{n,k}) \geq 0 \quad \Rightarrow \quad \mu(P_n) \leq \mu(P_{n,k}) \leq \frac{2^n}{k} \cdot \nu(P_{n,k}) \leq \frac{2^n}{k} \cdot \nu(\Omega).$

Taking $k \to \infty$ , we obtain $\mu(P_n) = 0$ for any $n$ . Define the simple functions

$\begin{aligned} f_{n}^- &:= \sum_{k=1}^\infty \frac{k}{2^n} \cdot \mathbb I_{P_{n,k} \backslash P_{n,k+1}} + \infty \cdot \mathbb I_{P_n}, \\ f_{n}^+ &:= \sum_{k=1}^\infty \frac{k+1}{2^n} \cdot \mathbb I_{P_{n,k} \backslash P_{n,k+1}} + \infty \cdot \mathbb I_{P_n}. \end{aligned}$

Defining $M_{n,k} := M \cap P_{n,k} \backslash P_{n,k+1}$ ,

$\displaystyle \lambda_{n,k}(M_{n,k}) \geq 0 \quad \Rightarrow \quad \frac{k}{2^n} \cdot \mu(M_{n,k}) \leq \nu(M_{n,k}).$

On the other hand, $M_{n,k} \subseteq \Omega \backslash P_{n, k+1} = N_{n,k+1}$ , so that

$\displaystyle \lambda_{n, k+1}(M_{n,k}) \leq 0 \quad \Rightarrow \quad \nu(M_{n,k}) \leq \frac{k+1}{2^n} \cdot \mu(M_{n,k}).$

Combining both estimates,

$\displaystyle \frac{k}{2^n} \cdot \mu(M_{n,k}) \leq \nu(M_{n,k}) \leq \frac{k+1}{2^n} \cdot \mu(M_{n,k}).$

Summing over $k$ , since $M = \bigsqcup_{k=1}^\infty M_{n,k} \sqcup ( M \cap P_n)$ ,

$\displaystyle \int_{M} f_n^-\, \mathrm d\mu \leq \nu(M \backslash P_n) \leq \int_{M} f_n^+\, \mathrm d\mu .$

Since $\mu(P_n) = 0$ implies $\nu(P_n) = 0$ ,

$\displaystyle \int_{M} f_n^-\, \mathrm d\mu \leq \nu(M) \leq \int_{M} f_n^+\, \mathrm d\mu .$

Therefore, $( f_n^- )$ is monotonically non-decreasing in $n$ , and so converges to some $f^- := \limsup_{n \to \infty} f_n^-$ by the monotone convergence theorem, and

$\begin{aligned} \int_M f^-\, \mathrm d\mu &= \lim_{n \to \infty} \int_M f_n^-\, \mathrm d\mu \leq \nu(M) \leq \int_M f^+\, \mathrm d\mu = \int_M f^-\, \mathrm d\mu + \frac {\mu(\Omega)}{2^n} . \end{aligned}$

Taking $n \to \infty$ , $\displaystyle \nu(M) = \int_M f^-\, \mathrm d\mu$ , so we set $f := f^-$ , as required.

Now suppose that $\mu, \nu$ are $\sigma$ -finite. Write $\Omega = \bigsqcup_{i=1}^\infty K_i$ with each $\mu(K_i) < \infty$ . For each $i$ , obtain a non-negative measurable function $f_i$ such that

$\displaystyle \nu(M \cap K_i) = \int_{M \cap K_i} f_i\, \mathrm d\mu ,\quad M \in \mathcal F.$

Define the non-negative measurable function $f := \sum_{i=1}^\infty f_i \cdot \mathbb I_{K_i}$ . Then for any $M \in \mathcal F$ ,

$\displaystyle \nu(M) = \sum_{i=1}^\infty \nu (M \cap K_i) = \sum_{i=1}^\infty \int_{M} f_i \cdot \mathbb I_{K_i}\, \mathrm d\mu = \int_{M} \sum_{i=1}^\infty f_i \cdot \mathbb I_{K_i}\, \mathrm d\mu = \int_{M} f\, \mathrm d\mu.$

Finally, $\mu$ -a.e. uniqueness holds since

$\displaystyle \left( \int_{M} f\, \mathrm d\mu = 0,\ M \in \mathcal F \right) \quad \Rightarrow \quad f = 0\ \mu\text{-a.e.}.$

We can now finally turn our attention to adding two continuous random variables $X, Y$ properly, since $\mathbb P_{X,Y} \ll \lambda$ so that by the Radon-Nikodým theorem, there exists a density function $f_{X,Y}$ for the joint distribution $\mathbb P_{X,Y}$ .

—Joel Kindiak, 23 Jul 25, 0113H

November 25, 2025
Formalising the Dirac Delta

The Dirac delta “function” $\delta$ has been a bane in my teaching since I needed to start teaching it in engineering mathematics, since it is, really, not actually a function. Many of its results seem miraculously correct, and yet we don’t have any reasonably accessible introduction to its formal definition.

We will adopt a measure-theoretic interpretation of $\delta$ and regard it as a measure. We will then define the relevant Laplace transform notions on measures that extend the classical setting. Finally, we assert that

$\displaystyle \int_{(-\infty, t]} \mathrm d\delta = U(t) \equiv \mathbb I_{[0,\infty)}(t),$

so that, in some limited sense, $U' = \delta$ .

For any measure $\mu$ on $(\mathbb R, \frak{B}(\mathbb R))$ , if the measurable function $f$ is $\mu$ -integrable, make the definition

$\displaystyle \langle \mu, f \rangle := \int_{\mathbb R} f\, \mathrm d\mu.$

whenever the right-hand side is well-defined. Using the Radon-Nikodým theorem, for suitably-defined functions $f, g$ ,

$\langle \lambda_g, f\rangle = \langle f \cdot g, \lambda \rangle = \langle \lambda_f, g \rangle,$

where the measure $\lambda_f$ is defined by $\lambda_f(S) := \int_S f\, \mathrm d\lambda$ . We remark that by the same theorem, the map $f \to \lambda_f$ is injective $\lambda$ -a.e. in the following sense: if $\lambda_f = 0$ , then $f = 0$ $\lambda$ -a.e.. Furthermore, it is bona-fide injective when restricted to the space of continuous functions (i.e. $f$ is continuous).

Define the map $K : \mathbb R^2 \to \mathbb R$ by $K(s,t) := e^{-st} \cdot U(t)$ .

Definition 1. For any measure $\mu$ on $(\mathbb R, \frak{B}(\mathbb R))$ , if $f$ is $\mu$ -integrable and $K(s, \cdot) \cdot f$ is $\mu$ -integrable, we define the $\mu$ –Laplace transform by

$\displaystyle \mathcal L_{\mu}\{f \} (s) := \langle \mu ,f \cdot K(s,\cdot)\rangle \equiv \int_{\mathbb R} K(s, \cdot) \cdot f\, \mathrm d\mu,$

whenever the right-hand side is well-defined. In particular, define $\mathcal L\{\mu\} := \mathcal L_{\mu}\{1\}$ .

Problem 1. If $K(s,\cdot)\cdot f$ is $\lambda$ -integrable, prove that $\mathcal L_{\lambda}\{f \} = \mathcal L\{f\}$ , where the right-hand side refers to the classical Laplace transform on sub-exponential functions.

(Click for Solution)

Solution. By definition, for any $s$ ,

$\begin{aligned} \mathcal L_{\lambda}\{f \}(s) &= \langle \lambda, f \cdot K(s,\cdot), \rangle \\ &= \int_{\mathbb R} K(s, \cdot) \cdot f\, \mathrm d\lambda \\ &= \int_{-\infty}^{\infty} K(s, t) \cdot f(t)\, \mathrm d \lambda( t ) \\ &= \int_{-\infty}^{\infty} e^{-st} \cdot f(t)\, \mathrm dt = \mathcal L\{f\}(s). \end{aligned}$

Hence, $\mathcal L\{\lambda_f\} \equiv \mathcal L_{\lambda_f}\{1\} \equiv \mathcal L_{\lambda}\{f\} = \mathcal L\{f\}$ .

Problem 2. Prove that the Laplace transform on measures is linear in the measures: for suitably defined measures $\mu,\nu$ , a suitably-defined function $f$ , and $\alpha \in \mathbb R$ ,

$\mathcal L_{\mu + \nu} \{f\} = \mathcal L_{\mu}\{f\} + \mathcal L_{\nu}\{f\},\quad \mathcal L_{\mu}\{\alpha f \} = \alpha \cdot \mathcal L_{\mu}\{f\}.$

Furthermore, if $\mu$ is signed, then $\mathcal L_{\alpha \mu}\{f \} = \alpha \cdot \mathcal L_{\mu}\{f\}$ .

(Click for Solution)

Solution. For additivity, we first prove the additivity of measures: for $S \in \frak B(\mathbb R)$ and suitably defined functions $f$ , we first claim that

$\begin{aligned} \langle \mu + \nu, f\rangle &= \int_{\mathbb R} f\, \mathrm d(\mu + \nu) \\ &= \int_{\mathbb R} f\, \mathrm d\mu + \int_{\mathbb R} f\, \mathrm d\nu \\ &= \langle \mu,f \rangle + \langle \nu, f\rangle, \end{aligned}$

so that for suitably chosen $s$ ,

$\begin{aligned}\mathcal L_{\mu + \nu}\{f\}(s) &= \langle \mu + \nu, K(s, \cdot) \cdot f\rangle \\ &= \langle \mu, K(s, \cdot) \cdot f \rangle + \langle \nu, K(s, \cdot) \cdot f\rangle \\ &= \mathcal L_{\mu }\{f\}(s) + \mathcal L_{ \nu}\{f\}(s) \\ &= (\mathcal L_{\mu }\{f\} + \mathcal L_{\nu }\{f\})(s).\end{aligned}$

To that end, we prove the result for the non-negative simple function $f = \sum_{i=1}^n a_i \cdot \mathbb I_{S_i}$ and leave the extensions via the monotone convergence theorem and decomposition $f = f^+ - f^-$ as an exercise:

$\begin{aligned} \int_{\mathbb R} f\, \mathrm d(\mu + \nu) &= \sum_{i=1}^n a_i (\mu + \nu)(S_i) \\ &= \sum_{i=1}^n a_i \cdot (\mu(S_i) + \nu(S_i)) \\ &= \sum_{i=1}^n a_i \cdot \mu(S_i) + \sum_{i=1}^n a_i \cdot \nu(S_i) \\ &= \int_{\mathbb R} f\, \mathrm d\mu + \int_{\mathbb R} f\, \mathrm d\nu, \end{aligned}$

as required. Scalar multiplication follows similarly.

Problem 3. Prove that for any $a \in \mathbb R$ , the map $\delta_a$ defined by $\delta_a(S) := \mathbb I_{S}(a)$ is a probability measure. We call $\delta := \delta_0$ the Dirac measure and observe that that $\delta_a = \delta ( \cdot - a)$ .

(Click for Solution)

Solution. We verify the three properties of a probability measure. Firstly,

$\delta(\emptyset) = \mathbb I_{\emptyset}(0) = 0.$

Secondly,

$\begin{aligned} \delta \left( \bigsqcup_{i=1}^\infty S_i \right) &= \mathbb I_{\bigsqcup_{i=1}^\infty S_i }(0) = \sum_{i=1}^\infty \mathbb I_{ S_i }(0) = \sum_{i=1}^\infty \delta(S_i).\end{aligned}$

Finally, $\delta(\mathbb R) = \mathbb I_{\mathbb R}(0) = 1$ .

Problem 4. Prove the sifting property:

$\displaystyle \langle \delta_a, f \rangle := \int_{\mathbb R} f\, \mathrm d\delta_a = f(a).$

Deduce that $\mathcal L_{\delta_a}\{f\}= K(\cdot, a) \cdot f$ . In particular, $\mathcal L_{\delta_a}\{1\}(s) = e^{-as}$ .

(Click for Solution)

Solution. We can use the same non-negative simple $\Rightarrow$ non-negative $\Rightarrow$ integrable technique for the sifting property, which we will carry out for completeness.

Let $f = \sum_{i=1}^n a_i \cdot \mathbb I_{S_i}$ be a non-negative simple function. We observe that

$\delta_a(S_i) = 1 \quad \iff \quad a \in S_i \quad \iff \quad f(a) = a_i.$

Then

$\begin{aligned} \int_{\mathbb R} f\, \mathrm d\delta_a &= \sum_{i=1}^n a_i \cdot \delta_a(S_i) = a_i \cdot 1 = f(a).\end{aligned}$

Now suppose $f$ is non-negative. Find a sequence $\varphi_n$ of simple functions such that $\varphi_n \to f$ monotonically. By the monotone convergence theorem,

$\begin{aligned} \int_{\mathbb R} f\, \mathrm d\delta_a &=\lim_{n \to \infty} \int_{\mathbb R} \varphi_n\, \mathrm d\delta_a = \lim_{n \to \infty} \varphi_n(a) = f(a).\end{aligned}$

Finally, suppose $f$ is integrable. Writing $f = f^+ - f^-$ ,

$\begin{aligned}\int_{\mathbb R} f\, \mathrm d\delta_a &= \int_{\mathbb R} f^+\, \mathrm d\delta_a - \int_{\mathbb R} f^-\, \mathrm d\delta_a \\ &= f^+(a) - f^-(a) \\ &= (f^+ - f^-)(a) = f(a).\end{aligned}$

In particular, for any fixed $s$ ,

$\displaystyle \mathcal L_{\delta_a}\{f\}(s) = \langle \delta_a, K(s,\cdot) \cdot f\rangle = \int_{\mathbb R} K(s, \cdot) \cdot f\ \mathrm d\delta_a = K(s, a) \cdot f(a) .$

In particular, $\mathcal L_{\delta_a}\{1\} = K(\cdot, a)$ .

Remark 1. We have shown that $\mathcal L_{\mu}\{f\}$ is a meaningful generalisation of both the usual Laplace transform $\mathcal L\{f\}$ and the Laplace transform of the measure $\mathcal L\{ {\delta_a} \}:=\mathcal L_{\delta_a}\{1\}$ :

$\begin{aligned} \mathcal L \{ \alpha \cdot \lambda_f + \beta \cdot \delta_a \} &= \mathcal L_{ \alpha \cdot \lambda_f + \beta \cdot \delta_a}\{1\} \\ &= \mathcal L_{ \alpha \cdot \lambda_f }\{1\} + \mathcal L_{ \beta \cdot \delta_a}\{1\}\\ &= \alpha \cdot \mathcal L_{\lambda_f}\{1\} + \beta \cdot \mathcal L_{\delta_a}\{1\} \\ &= \alpha \cdot \mathcal L \{\lambda_f\} + \beta \cdot \mathcal L \{\delta_a \} \\ &= \alpha \cdot \mathcal L \{f\} + \beta \cdot \mathcal L \{\delta_a \}. \end{aligned}$

In this manner and usage, we embed $f \to \lambda_f$ as a (signed-)measure, and define $f + \beta \cdot \delta_a := \lambda_f + \beta \cdot \delta_a$ via said embedding, finally permitting engineers to write the following guiltlessly:

$\begin{aligned} \mathcal L\{\alpha \cdot f + \beta \cdot \delta_a\} &\equiv \mathcal L \{ \alpha \cdot \lambda_f + \beta \cdot \delta_a \} \\ &= \alpha \cdot \mathcal L \{f\} + \beta \cdot \mathcal L \{\delta_a \}. \end{aligned}$

Problem 5. Prove that $\displaystyle \int_{(-\infty, t]} \mathrm d\delta_a = U_a(t)$ . Deduce that for $a > 0$ and valid $s$ ,

$\displaystyle \int_{\mathbb R} \mathrm d\delta_a = 1 \quad \text{and} \quad \mathcal L\{\delta_a\}(s) = s \cdot \mathcal L\{U_a\}(s),$

where $U_a = U( \cdot - a)$ .

(Click for Solution)

Solution. For the first claim,

$\begin{aligned} \int_{(-\infty, t]} \mathrm d\delta_a &= \int_{\mathbb R} \mathbb I_{(-\infty, t]}\, \mathrm d\delta_a = \delta_a((-\infty, t]) = \begin{cases} 1, & t \geq a, \\ 0, & t < a. \end{cases}\end{aligned}$

Taking $T \to \infty$ and $T > a$ in particular, by the monotonicity of measures,

$\begin{aligned} \int_{\mathbb R} \mathrm d\delta_a &= \lim_{T \to \infty} \int_{(-\infty, T]} \mathrm d\delta_a = \lim_{T \to \infty} U_a(T) = \lim_{T \to \infty} 1 = 1.\end{aligned}$

For the final result,

$\displaystyle \mathcal L\{U_a\}(s) = \frac{e^{-as}}{s} = \frac{\mathcal L\{\delta_a\}(s)}{s}.$

Remark 2. In classical formulas, if $f$ is differentiable such that $f' \cdot \mathbb I_{(-\infty, t]}$ is $\lambda$ -integrable for any $t \in \mathbb R$ , $\displaystyle \lim_{N \to \infty}f(-N) = 0$ , and has a Laplace transform, we have

$\displaystyle \int_{(-\infty, t]} f'\, \mathrm d\lambda = f(t),\quad \mathcal L\{f'\}(s) = s \cdot \mathcal L\{f\}(s) - f(0).$

Since Problems 3 and 4 agree with classical setups, engineers sloppily write $U' = \delta$ and use these results on their happy merry way, leaving mathematicians infuriated in having to clean up after their…stuff.

This measure-theoretic formalisation only helps us with the integration aspect of the Dirac delta, since measure theory is, in some sense a “completion” of integral calculus. Yet, a full understanding requires us to develop a “completion” of differential calculus—this generalisation builds on measure theory and discusses distribution theory.

We hinted at these objects via the notation $\langle \mu, f \rangle$ . The map

$\langle \mu, \cdot \rangle : \{\text{good functions}\} \to \mathbb R$

is called a distribution. In a sense, then we could define $U := \langle U, \cdot \rangle$ as a distribution (though this requires much more effort), and by defining differentiation of distributions, we get the true equality $U' = \delta$ . But all that work requires a lot more pain and suffering.

As completions, they become indispensable tools in solving many more differential equations than a usual post-secondary or even undergraduate course would present.

—Joel Kindiak, 16 Jul 25, 1150H

November 11, 2025
Integrable Sanity Checks
Having developed much technology for Lebesgue integration, let’s do a quick sanity check that we do, in fact, recover the usual Riemann integral.

Theorem 1. Let $f : \mathbb R \to \mathbb R$ be a bounded measurable function. If $f|_{[a, b]}$ is Riemann-integrable, then $f|_{[a, b]}$ is Lebesgue-integrable, and

$\displaystyle \int_{[a, b]} f\, \mathrm d\lambda = \int_a^b f \equiv \int_a^b f(x)\, \mathrm dx$

where the integral on the left-hand side denotes the Lebesgue integral (here $\lambda$ denotes the Lebesgue measure), and the integral on the right-hand side denotes the usual Riemann integral.

Proof. Suppose $f \geq 0$ for simplicity. We first note that all step functions $\sum_{i=1}^n a_i \cdot \mathbb I_{[x_{i-1},x_i)}$ are simple functions. By the definition of the lower integral $\mathcal L_a^b(f)$ and upper integral $\mathcal R_a^b(f)$ ,

$\begin{aligned} \mathcal L_a^b(f) &= \sup_P \sum_{i=1}^n m_i(f, P) \Delta x_i \\ &= \sup_P \int_{[a,b]} m_i(f, P) \cdot \mathbb I_{[x_{i-1},x_i)} \\ &\leq \sup_{\substack{\varphi\ \text{simple} \\ 0 \leq \varphi \leq f \cdot \mathbb I_{[a, b]}} } \int_{\mathbb R} \varphi \, \mathrm d\lambda \\ &\leq \inf_P \sum_{i=1}^n M_i(f, P) \Delta x_i \leq \mathcal R_a^b(f). \end{aligned}$

Since $f$ is Riemann-integrable, both left-hand side and right-hand side equal to $\int_a^b f$ , so that

$\displaystyle \int_a^b f\, \mathrm d\lambda = \int_{\mathbb R} f \cdot \mathbb I_{[a, b]}\, \mathrm d\lambda = \sup_{\substack{\varphi\ \text{simple} \\ 0 \leq \varphi \leq f \cdot \mathbb I_{[a, b]}} } \int_{\mathbb R} \varphi \, \mathrm d\lambda = \int_a^b f.$

Since the right-hand side is finite, so is the left-hand side, so that $f$ is Lebesgue-integrable.

For the general case, define $-m := \inf_{x \in [a, b]} f(x) < 0$ . Now $(f + m)|_{[a,b]}$ is bounded, measurable, and Riemann-integrable. Since it is nonnegative, by the first result, it is Lebesgue-integrable. Therefore, $f = (f+m) -m$ when restricted to $[a, b]$ is also Lebesgue-integrable, and

$\begin{aligned} \int_{[a, b]} f\, \mathrm d\lambda &= \int_{[a, b]} ((f + m) - m)\, \mathrm d\lambda \\ &= \int_{[a, b]} (f + m)\, \mathrm d\lambda - \int_{[a, b]} m\, \mathrm d\lambda\\ &= \int_a^b f + \int_a^b m -\int_a^b m = \int_a^b f.\end{aligned}$

Theorem 1 therefore tells us that the Lebesgue integral generalises the Riemann integrable, at least when $f$ is a bounded function. If $f \geq 0$ and measurable (but not necessarily integrable), more is true.

Let $(\Omega, \mathcal F, \mu)$ be a measure space.

Lemma 1. Let $f : \Omega \to [0, \infty]$ be measurable. The map $\nu : \mathcal F \to [0, \infty]$ defined by

$\displaystyle \nu(K) := \int_K f\, \mathrm d\mu$

is a measure. Furthermore, $\lambda(K) = 0$ implies that $\nu(K) = 0$ .

Proof. For the empty set condition,

$\displaystyle \nu(\emptyset) = \int_{\emptyset}f\, \mathrm d\lambda = \int_{\Omega} f \cdot \mathbb I_\emptyset\, \mathrm d\lambda = \int_{\Omega} 0\, \mathrm d\lambda = 0.$

For the countable additivity condition, fix a pairwise-disjoint sequence $\{K_i\} \subseteq \frak{B}(\mathbb R)$ . Define $K := \bigsqcup_{i=1}^\infty K_i$ . The sequence $f \cdot \sum_{i=1}^n \mathbb I_{K_i}$ of measurable functions monotonically increases to the measurable function $f \cdot \mathbb I_K$ . By the monotone convergence theorem,

$\begin{aligned} \nu\left(\bigsqcup_{i=1}^\infty K_i \right) &= \nu(K) = \int_K f\, \mathrm d\lambda = \int_{\Omega} f \cdot \mathbb I_K \, \mathrm d\lambda \\ &= \int_{\Omega} \lim_{n \to \infty} f \cdot \sum_{i=1}^n \mathbb I_{K_i} \, \mathrm d\lambda = \lim_{n \to \infty} \int_{\Omega} f \cdot \sum_{i=1}^n \mathbb I_{K_i} \, \mathrm d\lambda \\ &= \lim_{n \to \infty} \sum_{i=1}^n \int_{\Omega} f \cdot \mathbb I_{K_i} \, \mathrm d\lambda = \lim_{n \to \infty} \sum_{i=1}^n \int_{K_i} f \, \mathrm d\lambda \\ &= \lim_{n \to \infty} \sum_{i=1}^n \nu(K_i) = \sum_{i=1}^\infty \nu(K_i). \end{aligned}$

Therefore, $\nu$ is countably additive. Finally, suppose $\lambda (K) = 0$ . Fix any simple function $0 \leq \varphi \leq f$ , $\varphi = \sum_{i=1}^n a_i \cdot \mathbb I_{K_i}$ , where $K = \bigsqcup_{i=1}^n K_i \supseteq K_i$ . By the monotonicity of $\lambda$ , $0 \leq \lambda(K_i) \leq \lambda(K) = 0$ . Hence,

$\displaystyle \int_{K} \varphi\, \mathrm d\lambda = \sum_{i=1}^n a_i \cdot \lambda(K_i) = \sum_{i=1}^n a_i \cdot 0 = 0.$

Therefore,

$\displaystyle \nu(K) = \int_K f\, \mathrm d\lambda = \sup_{\varphi} \int_K \varphi\, \mathrm d\lambda = \sup_{\varphi} 0 = 0,$

as required.

Lemma 2. Let $f : \Omega \to \mathbb R$ be a non-negative measurable function. Then $\int_{\Omega} f\, \mathrm d\mu = 0$ if and only if there exists some $K \in \mathcal F$ with $\mu(K) = 0$ such that $f|_{\Omega \backslash K} = 0$ . In this case, we say that $f=0$ $\mu$ –almost everywhere (abbreviated: $\mu$ -a.e.).

Proof. For the direction $(\Leftarrow)$ , the proof in Lemma 1 yields

$\begin{aligned} \int_{\Omega} f\, \mathrm d\mu = \int_{\Omega\backslash K} f\, \mathrm d\mu + \int_{K} f\, \mathrm d\mu = \int_{\Omega\backslash K} f\, \mathrm d\mu + 0 = \int_{\Omega\backslash K} f\, \mathrm d\mu. \end{aligned}$

Hence,

$\displaystyle \int_{ \Omega } f\, \mathrm d\mu = \int_{ \Omega \backslash K } f\, \mathrm d\mu = \int_{ \Omega \backslash K } g\, \mathrm d \mu = \int_{ \Omega } g\, \mathrm d\mu.$

For the direction $(\Rightarrow)$ , we will prove by contrapositive. Fix $K \in \mathcal F$ with $\mu(K) = 0$ and $f|_{\Omega \backslash K} > 0$ . Then

$\displaystyle \int_{\Omega} f\, \mathrm d\mu = \int_{K} f\, \mathrm d\mu + \int_{\Omega \backslash K} f\, \mathrm d\mu \geq \int_{\Omega \backslash K}f\, \mathrm d\mu > 0.$

Lemma 3. Let $f, g : \Omega \to \mathbb R$ be integrable functions. Then $\int_{K} f\, \mathrm d\mu = \int_{K} g\, \mathrm d\mu$ for any $K \in \mathcal F$ if and only if there exists some $L \in \mathcal F$ with $\mu(L) = 0$ such that $f|_{\Omega \backslash L} = g|_{\Omega \backslash L}$ . In this case, we say that $f=g$ $\mu$ –a.e..

Proof. By linearity of the Lebesgue integral, it suffices to prove the case $g = 0$ . For the direction $(\Leftarrow)$ , for any $K \in \mathcal F$ ,

$\displaystyle \int_K f\, \mathrm d\mu = \int_{K \backslash L} f\, \mathrm d\mu = \int_{K \backslash L} 0\, \mathrm d\mu = 0.$

For the direction $(\Rightarrow)$ , define $\Omega^+ := f^{-1}(\mathbb R^+)$ and $\Omega^- := f^{-1}(\mathbb R^-)$ . Then $f^+, f^-$ are nonnegative functions and by Lemma 2,

$\displaystyle \int_{\Omega} f^+\, \mathrm d\mu = \int_{\Omega^+} f\, \mathrm d\mu = 0 \quad \Rightarrow \quad f^+ = 0\quad \mu\ \text{a.e.}.$

Thus, there exists $L^+ \in \mathcal F$ with $\mu(L^+) = 0$ such that $f^+|_{\Omega \backslash L^+} = 0$ . Similarly, $f^- = 0$ $\mu$ -a.e., and there exists $L^- \in \mathcal F$ with $\mu(L^-) = 0$ such that $f^+|_{\Omega \backslash L^+} = 0$ . Observe that

$\begin{aligned} \mu(L^+ \cup L^-) &= \mu(L^+ \sqcup L^- \backslash L^+) \\ &= \mu(L^+) + \mu(L^- \backslash L^+) \\ &\leq \mu(L^+) + \mu(L^-) = 0 + 0 = 0. \end{aligned}$

Hence, writing $f = f^+ - f^-$ ,

$\begin{aligned} f|_{\Omega \backslash (L^+ \cup L^-)} &= (f^+ - f^-)|_{\Omega \backslash (L^+ \cup L^-)} \\ &= f^+|_{\Omega \backslash (L^+ \cup L^-)} - f^-|_{\Omega \backslash (L^+ \cup L^-)} \\ &= 0 - 0 = 0.\end{aligned}$

Therefore, $f = 0$ $\mu$ -a.e..

Lemma 4. For any non-negative measurable function $f : \Omega \to [0, \infty]$ , there exists a sequence $\{f_n\}$ of simple functions $f_n : \Omega \to [0, \infty]$ such that $f_n \to f$ monotonically.

Proof. For each $\mathbb N$ , define for $k = 1,2,\dots,2^{2n}$

$\displaystyle I_{n,k} := 2^{-n} \cdot [k-1, k).$

Furthermore, define $I_{n,2^n+1} := [2^n, \infty)$ . Define the non-negative simple functions $f_n : \Omega \to [0, \infty]$ by

$\displaystyle f_n := \sum_{k=1}^{2^{2n}+1} \frac{k-1}{2^n} \cdot \mathbb I_{f^{-1}(I_{n,k})}.$

Here are the functions for $n=0,1$ for illustration:

$\begin{aligned} f_0 := \sum_{k=1}^2 (k-1) \cdot \mathbb I_{f^{-1}(I_{0,k})} &= \mathbb I_{f^{-1}([1, 2))}, \\ f_1 := \sum_{k=1}^5 \frac{k-1}{2} \cdot \mathbb I_{f^{-1}(I_{1,k})} &= \frac{1}{2} \cdot \mathbb I_{f^{-1}([1/2, 1))} + \mathbb I_{f^{-1}([1, 3/2))} \\ &\phantom{--} + \frac{3}{2} \cdot \mathbb I_{f^{-1}([3/2, 2))} + 2 \cdot \mathbb I_{f^{-1}([2, 5/2))}. \end{aligned}$

Each $f_n$ is made up of $2^{2^n}$ pieces. By performing necessary real-analysis calculations, it’s not hard to verify that $f_n \to f$ monotonically as $n \to \infty$ .

Theorem 2 (Change-of-Variables). Let $(\Psi, \mathcal G)$ be a measurable space, and $f : \Omega \to \Psi$ be a measurable function that induces the pushforward measure $\nu \equiv \mu(f^{-1}(\cdot))$ on $\mathcal G$ . Then for any integrable $g : \Psi \to \mathbb R$ ,

$\displaystyle \int_{\Omega} g \circ f\, \mathrm d\mu = \int_{\Psi} g\, \mathrm d\nu .$

Proof. We first prove the result for measurable $g : \Psi \to [0, \infty]$ . Firstly, suppose $g = \sum_{i=1}^n a_i \cdot \mathbb I_{K_i}$ is a simple function. We observe that

$(g \circ f)(\omega) = a_i \quad \iff \quad f(\omega) \in K_i \quad \iff \quad \omega \in f^{-1}(K_i).$

Therefore, $g \circ f = \sum_{I=i}^n a_i \cdot \mathbb I_{f^{-1}(K_i)}$ , so that

$\displaystyle \int_{\Omega} g \circ f\, \mathrm d\mu= \sum_{i=1}^n a_i \cdot \mu(f^{-1}(K_i)) = \sum_{i=1}^n a_i \cdot \nu(K_i) = \int_{\Psi} g\, \mathrm d\nu.$

Now suppose $g$ is nonnegative and measurable. Use Lemma 4 to find a sequence $\{g_n\}$ of non-negative simple functions $g_n \leq g$ that monotonically converge to $g$ . It is obvious that $g_n \circ f \to g \circ f$ monotonically as well. By the monotone convergence theorem,

$\displaystyle \int_{\Omega} g \circ f\, \mathrm d\mu = \lim_{n \to \infty} \int_{\Omega} g_n \circ f\, \mathrm d\mu = \lim_{n \to \infty} \int_{\Psi} g_n\, \mathrm d\mu = \int_{\Psi} g\, \mathrm d\nu .$

Finally suppose $g : \Psi \to \mathbb R$ is integrable. Write $g = g^+ - g^-$ . Applying relevant linearity properties,

$\begin{aligned}\int_{\Omega} g \circ f\, \mathrm d\mu &= \int_{\Omega} (g^+ - g^-) \circ f\, \mathrm d\mu \\ &= \int_{\Omega} (g^+ \circ f - g^- \circ f)\, \mathrm d\mu \\ &= \int_{\Omega} g^+ \circ f\, \mathrm d\mu - \int_{\Omega} g^- \circ f\, \mathrm d\mu \\ &= \int_{\Psi} g^+\, \mathrm d\nu - \int_{\Psi} g^-\, \mathrm d\nu \\ &= \int_{\Psi} (g^+\, - g^-)\, \mathrm d\nu = \int_{\Psi} g\, \mathrm d \nu.\end{aligned}$

Let $\Psi \equiv (\Psi, \mathcal G, \nu)$ denote either of the measure spaces $\mathbb Z \equiv (\mathbb Z, \mathcal P(\mathbb Z), |\cdot|)$ or $\mathbb R \equiv (\mathbb R, \frak B(\mathbb R), \lambda)$ .

Definition 2. Let $X : \Omega \to \Psi$ be a measurable map. Suppose there exists a non-negative integrable function $f_X : \Psi \to \mathbb R$ such that its distribution $\mathbb P_X$ satisfies

$\displaystyle \mathbb P_X(K) = \int_K f_X\, \mathrm d \nu,\quad K \in \mathcal F.$

If $\Psi = \mathbb Z$ , then we call $X$ a discrete random variable so that $f_X = \sum_{i \in \mathbb Z} f_X(i) \cdot \mathbb I_{\{ i \}}$ and for any $K \subseteq \mathbb Z$ ,

$\begin{aligned} \mathbb P_X( K ) &= \int_{ K } f_X\, \mathrm d |\cdot| \\ &= \int_{ \mathbb Z} f_X \cdot \mathbb I_{K} \, \mathrm d |\cdot| \\ &= \sum_{x \in K} f_X(x) \cdot |\{x \}| = \sum_{x \in K} f_X(x). \end{aligned}$

If $\Psi = \mathbb R$ , then we call $X$ a continuous random variable, and for any $a \leq b$ ,

$\displaystyle \mathbb P_X([a, b]) = \int_a^b f_X \equiv \int_a^b f_X(x)\, \mathrm dx.$

A random variable $X : \Omega \to \mathbb R$ is said to be continuous if there exists a non-negative integrable function $f_X : \mathbb R \to \mathbb R$ such that its distribution $\mathbb P_X$ satisfies

$\displaystyle \mathbb P_X(K) = \int_K f_X\, \mathrm d \lambda,\quad K \in \mathcal F \quad \Rightarrow \quad \mathbb P_X([a, b]) = \int_a^b f_X.$

In these special cases, we call $f_X$ the probability density function of $X$ , which is $\lambda$ -almost everywhere unique by Lemma 3, and define the cumulative distribution function by $F_X(x) := \mathbb P_X((-\infty, x])$ .

We see therefore the unification that measure theory offers our study of probability theory—in fact we can just take these results for granted when eventually talking about (absolutely) continuous random variables.

Theorem 3. Suppose $\mu = \mathbb P$ is a probability measure. Then for any continuous random variable $X : \Omega \to \Psi$ and continuous $g : \Psi \to \mathbb R$ ,

$\displaystyle \mathbb E[g(X)] = \int_{\Omega} g \circ X\, \mathrm d \mathbb P = \int_{\Psi} g\, \mathrm d \mathbb P_X = \int_{\Psi} g \cdot f_X\, \mathrm d\nu.$

Proof. We first claim that for any $\mathbb P_X$ -integrable $g : \Psi \to \mathbb R$ and $K \in \mathcal F$ ,

$\begin{aligned} \int_{\Psi} g\, \mathrm d \mathbb P_X &= \int_{\Psi} g \cdot f_X\, \mathrm d\nu. \end{aligned}$

It suffices to prove the case when $g = \sum_{i=1}^n a_i \cdot \mathbb I_{K_i} : \Psi \to [0, \infty]$ is simple, and the rest follows by the monotone convergence theorem (for non-negative $g$ ) and linearity arguments on the decomposition $g = g^+ - g^-$ (for integrable $g$ ). To that end,

$\begin{aligned} \int_{\Psi} g\, \mathrm d \mathbb P_X &= \sum_{i=1}^n a_i \cdot \mathbb P_X(K_i) \\ &= \sum_{i=1}^n a_i \cdot \int_{K_i} f_X \, \mathrm d\nu \\ &= \sum_{i=1}^n a_i \cdot \int_{\Psi} f_X \cdot \mathbb I_{K_i} \, \mathrm d\nu \\ &= \int_{\Psi} \sum_{i=1}^n a_i \cdot \mathbb I_{K_i} \cdot f_X \cdot \mathrm d\nu = \int_{\Psi} g \cdot f_X \, \mathrm d\nu. \end{aligned}$

The result is then obvious.

Theorem 3 helps us recover the usual formula for expectation when we particularise $X$ to discrete or continuous random variables in the spirit of Definition 2:
- for discrete $X$ , $\mathbb E[g(X)] = \sum_{x \in \mathbb Z} g(x) f_X(x)$ ,
- for continuous $X$ , $\mathbb E[g(X)] = \int_{-\infty}^{\infty} g(x) f_X(x)\, \mathrm dx$ .
Furthermore, by using $(\Psi, \mathcal G)$ rather than their specific implementations, our arguments remain valid when we particularise to higher-dimensional spaces like $(\mathbb R^n, \frak{B}(\mathbb R^n), \lambda)$ so that we can discuss the distributions of combinations of random variables like $g(X,Y)$ or even more specifically $X + Y$ .

We turn to cumulative distribution functions and continuous random variables next time.

—Joel Kindiak, 13 Jul 25, 2349H
November 4, 2025
The Measure-Theoretic Trifecta
Any self-respecting discussion in measure theory cannot ignore the three pillars of convergence: the monotone convergence theorem, Fatou’s lemma, and the dominated convergence theorem.

These results aim to answer the question: given measurable functions $f : \Omega \to \mathbb R$ , what conditions on $\{f_n\}$ do we need in order to guarantee the following interchange of limits?

$\displaystyle \int_{\Omega} \lim_{n \to \infty} f_n\, \mathrm d\mu = \lim_{n \to \infty} \int_{\Omega} f_n\, \mathrm d\mu.$

Recall the monotone convergence theorem for real numbers.

Lemma 1. Let $\{x_n\}$ be a non-decreasing sequence of real numbers that is bounded above. Then $\{x_n\}$ converges to $\sup\{x_n\}$ .

We want to prove an analogous result for measurable functions. Let $(\Omega, \mathcal F, \mu)$ be any measure space.

Lemma 2. For measurable $f, g : \Omega \to [0, \infty]$ , if $f \leq g$ , then

$\displaystyle \int_{\Omega} f\, \mathrm d\mu \leq \int_{\Omega} g\, \mathrm d\mu.$

Proof. Fix $\epsilon > 0$ . Then there exists a simple function $0 \leq \varphi \leq f \leq g$ such that

$\displaystyle \int_{\Omega} f\, \mathrm d\mu - \epsilon \leq \int_{\Omega} \varphi\, \mathrm d\mu \leq \int_{\Omega} g\, \mathrm d\mu.$

Taking $\epsilon \to 0^+$ yields the desired result.

Theorem 1 (Monotone Convergence Theorem). Let $\{f_n\}$ be a sequence of measurable functions $f_n : \Omega \to [0, \infty]$ . Then the map $f : \Omega \to \mathbb R$ defined by

$\displaystyle f(\omega) := \sup_n f_n(\omega)$

is measurable. Furthermore, if $\{f_n\}$ is increasing in $n$ (that is, $f_i \leq f_j$ whenever $i < j$ ), then $f_n \to f$ and

$\displaystyle \int_{\Omega} f\, \mathrm d\mu = \lim_{n \to \infty} \int_{\Omega} f_n\, \mathrm d\mu.$

Proof. For any $a \in [0, \infty]$ , we have $[a, \infty] \in \frak{B}([0,\infty])$ . Since each $f_n$ is measurable, $f_n^{-1}([a, \infty]) \in \mathcal F$ . Therefore,

$\displaystyle f^{-1}([a, \infty]) = \bigcap_{n = 1}^\infty f_n^{-1}([a, \infty]) \in \mathcal F.$

Now if $\{f_n\}$ is increasing in $n$ , then for any $\omega \in \Omega$ , $f_n(\omega) \to f(\omega)$ by Lemma 1. By Lemma 2,

$\displaystyle \lim_{n \to \infty} \int_{\Omega} f_n\, \mathrm d\mu \leq \lim_{n \to \infty} \int_{\Omega} f\, \mathrm d\mu = \int_{\Omega} f\, \mathrm d\mu.$

Thus, we claim that the reverse inequality

$\displaystyle \int_{\Omega} f\, \mathrm d\mu \leq \lim_{n \to \infty} \int_{\Omega} f_n\, \mathrm d\mu$

holds. Firstly, assume $\int_{\Omega} f\, \mathrm d\mu = \infty$ . We claim that the right hand side equals $\infty$ . Fix $N > 0$ . By definition, there exists a simple function $\varphi = \sum_{i=1}^n a_i \cdot \mathbb I_{K_i}$ , $a_i > 0$ , such that $0 \leq \varphi \leq f$ and

$\displaystyle \sum_{i=1}^n a_i \cdot \mu(K_i) = \int_{\Omega} \varphi\, \mathrm d\mu > N.$

Define $\epsilon := \min\{a_1,\dots,a_n\}/2$ . so that $\epsilon^{-1} > a_i > \epsilon$ for each $i$ . Define $K := \bigsqcup_{i=1}^n K_i$ . There are two cases: $\mu(K) < \infty$ or $\mu(K) = \infty$ .

Suppose $\mu(K) < \infty$ . Let $\delta > 0$ be a constant to be tuned. Define

$L_n := \{\omega \in K : f_n(\omega) > f(\omega) - \delta\}.$

Due to the monotonicity of $\{f_n\}$ , $L_1 \subseteq L_2 \subseteq \dots$ and $K = \bigcup_{n=1}^\infty L_n$ . By continuity of measures, $\mu(L_n) \to \mu(K)$ . Hence, there exists $m \in \mathbb N$ such that $\mu(L_m) > \mu(K) - \delta$ . Therefore,

$\begin{aligned} \int_{L_m} \varphi\, \mathrm d\mu &= \int_{L_m} \sum_{i=1}^n a_i \cdot \mathbb I_{K_i}\, \mathrm d\mu \\ &= \sum_{i=1}^n a_i \cdot \mu (K_i \cap L_m) \\ &= \sum_{i=1}^n a_i \cdot \mu(K_i) - \sum_{i=1}^n a_i \cdot \mu(K_i \backslash L_m) \\ &> \sum_{i=1}^n a_i \cdot \mu(K_i) - \max\{a_1,\dots,a_n\} \cdot \sum_{i=1}^n \mu(K_i \backslash L_m) \\ &> N - \epsilon^{-1} \cdot \delta. \end{aligned}$

By the construction of $L_m$ ,

$\begin{aligned} \int_{L_m} f_m \, \mathrm d\mu + \delta \cdot \mu(L_m) &= \int_{L_m} (f_m + \delta) \, \mathrm d\mu \\ &> \int_{L_m} f \, \mathrm d\mu\\ & \geq \int_{L_m} \varphi \, \mathrm d\mu \\ &> N - \epsilon^{-1} \cdot \delta. \end{aligned}$

Therefore,

$\begin{aligned} \int_{\Omega} f_m\, \mathrm d\mu &\geq \int_{L_m} f_m \, \mathrm d\mu \\ &> N - \epsilon^{-1} \cdot \delta - \delta \cdot \mu(L_m) \\ &> N - \delta \cdot (\epsilon^{-1} + \mu(K)) > N - 1,\end{aligned}$

where we set $\delta := 1/(\epsilon^{-1}+\mu(K))$ . That way, $\int_{\Omega} f_m\, \mathrm d\mu \to \infty$ , as required.

Now suppose $\mu(K) = \infty$ . It suffices to assume $\mu(K_1) = \infty$ without loss of generality. Define

$M_n := \{\omega \in K_1 : f_n(\omega) \geq \epsilon\}.$

Similar to the previous case, $M_1 \subseteq M_2 \subseteq \dots$ and $\bigcup_{n=1}^\infty M_n = K_1$ . Thus, there exists $\ell \in \mathbb N$ such that $\mu(M_{\ell}) > N/\epsilon$ . Hence,

$\displaystyle \int_{\Omega} f_{\ell}\, \mathrm d\mu \geq \int_{M_{\ell}} f_{\ell}\, \mathrm d\mu \geq \int_{\Omega} \epsilon \cdot \mathbb I_{M_{\ell}} \, \mathrm d\mu = \epsilon \cdot \mu(M_{\ell}) > N.$

That way, $\int_{\Omega} f_{\ell}\, \mathrm d\mu \to \infty$ , as required.

Finally, we work on the case where $\int_{\Omega} f\, \mathrm d\mu < \infty$ . Fix $\delta > 0$ . By definition, there exists a simple function $\varphi = \sum_{i=1}^n a_i \cdot \mathbb I_{K_i}$ , $a_i > 0$ , such that $0 \leq \varphi \leq f$ and

$\displaystyle \sum_{i=1}^n a_i \cdot \mu(K_i) = \int_{\Omega} \varphi\, \mathrm d\mu > \int_{\Omega} f\, \mathrm d\mu - \delta.$

Define $\epsilon := \min\{a_1,\dots,a_n\}/2$ . so that $\epsilon^{-1} > a_i > \epsilon$ for each $i$ . Define $K := \bigsqcup_{i=1}^n K_i$ . Then

$\displaystyle \mu(K) = \sum_{i=1}^n \mu(K_i) < \epsilon^{-1} \cdot \sum_{i=1}^n a_i \cdot \mu(K_i) \leq \epsilon^{-1} \cdot \int_{\Omega} f\, \mathrm d\mu < \infty.$

Fix $\eta > 0$ to be tuned. Define

$R_n := \{\omega \in K : f_n(\omega) > \varphi(\omega) - \eta\},$

so that $R_1 \subseteq R_2 \subseteq \dots$ and $\bigcup_{n=1}^\infty R_n = K$ . Hence for any $\theta > 0$ , there exists $k \in \mathbb N$ such that $\mu(R_k) > \mu(K) - \theta$ . Hence,

$\begin{aligned} \int_{\Omega} f_k\, \mathrm d\mu &\geq \int_{R_k} f_k\, \mathrm d\mu \\ &\geq \int_{R_k} (\varphi - \eta)\, \mathrm d\mu \\ &= \int_{R_k} \varphi\, \mathrm d\mu - \eta \cdot \mu(R_k) \\ &> \int_{\Omega} \varphi\, \mathrm d\mu - \epsilon^{-1} \cdot \theta - \eta \cdot \mu(K). \end{aligned}$

Choose $\theta = \epsilon \cdot \delta$ and $\eta := \delta/\mu(K)$ to obtain the inequality

$\begin{aligned} \int_{\Omega} f_k\, \mathrm d\mu &> \int_{\Omega} \varphi\, \mathrm d\mu - 2\delta &> \int_{\Omega} f\, \mathrm d\mu - 3\delta. \end{aligned}$

That way, $\int_{\Omega} f_k\, \mathrm d\mu \to \int_{\Omega} f\, \mathrm d\mu$ , as required.

The monotone convergence theorem therefore tells us that if $\{f_n\}$ is monotonically increasing in $n$ , then

$\displaystyle \lim_{n \to \infty} \int_{\Omega} f_n\, \mathrm d\mu = \int_{\Omega} \lim_{n \to \infty} f_n\, \mathrm d\mu.$

What happens if we do not have such a nice feature like montonicity? While equality feels far-fetched, we do get a useful partial result.

Theorem 2 (Fatou’s Lemma). Let $\{f_n\}$ be a sequence of measurable functions $f_n : \Omega \to [0, \infty]$ . Then the map $f : \Omega \to \mathbb R$ defined by

$\displaystyle f(\omega) := \liminf_{n \to \infty} f_n(\omega) \equiv \lim_{n \to \infty} \left\{ \inf_{k \geq n} f_k(\omega) \right\}$

is measurable. Furthermore,

$\displaystyle \int_{\Omega} f\, \mathrm d\mu \leq \lim_{n \to \infty} \int_{\Omega} f_n\, \mathrm d\mu.$

Proof. We know that $f$ is measurable because

$\displaystyle f^{-1}([a, \infty]) = \bigcup_{n = 1}^\infty \left( \bigcap_{k=n}^\infty f_k^{-1}([0, \infty]) \right).$

Define $\tilde f_n := \inf_{k \geq n} f_k$ , which is measurable. By construction, $\{ \tilde f_n\}$ is monotonically increasing and $\tilde f_n \leq f_n$ . By the monotone convergence theorem,

$\displaystyle \int_{\Omega} \liminf_{n \to \infty} f_n\, \mathrm d\mu = \int_{\Omega} \lim_{n \to \infty} \tilde f_n\, \mathrm d\mu = \lim_{n \to \infty} \int_{\Omega}\tilde f_n\, \mathrm d\mu \leq \lim_{n \to \infty} \int_{\Omega} f_n\, \mathrm d\mu.$

Remark 1. We usually abbreviate Fatou’s lemma using the inequality

$\displaystyle \int_{\Omega} \liminf_{n \to \infty}f_n\, \mathrm d\mu \leq \liminf_{n \to \infty} \int_{\Omega} f_n\, \mathrm d\mu.$

To ensure the correct inequality direction, the example $f_n = \mathbb I_{[n,n+1)}$ yields $0$ on the left-hand side and $1$ on the right-hand side. Thus, the integral of the point-wise limit is usually more “stringent” than the limit of the integral.

In a sense, the monotone convergence theorem requires a “strictest” requirement on the sequence $\{f_n\}$ of measurable functions in order to interchange limits. Fatou’s lemma, on the other hand, requires almost “no” requirement on $\{f_n\}$ , at the price of losing equality. Do we have some “middle ground” between the two?

Yes we do, and it is called Lebesgue’s dominated convergence theorem. In fact, we require the underlying sequence to be integrable (which is usually what we are interested in) and have some form of “boundedness” in order to interchange limits. This theorem turns out to more often than not be the most useful out of the three convergence theorems.

Theorem 3 (Dominated Convergence Theorem). Let $\{f_n\}$ be a sequence functions $f_n : \Omega \to \mathbb R$ satisfying the following properties:
- each $f_n$ is measurable,
- there exists a measurable function $f : \Omega \to \mathbb R$ such that $f_n \to f$ ,
- there exists an integrable function $g : \Omega \to [0, \infty)$ such that $|f_n| \leq g$ .
Then $f_n, f$ are integrable, and

$\displaystyle \int_{\Omega} f\, \mathrm d\mu = \lim_{n \to \infty} \int_{\Omega} f_n\, \mathrm d\mu.$

Proof. We first suppose each $f_n \geq 0$ , which implies that $f \geq 0$ . Since $|f_n| \leq g$ ,

$\displaystyle \int_{\Omega} f_n \, \mathrm d\mu \leq \int_{\Omega} g \, \mathrm d\mu < \infty.$

Thus, each $f_n$ is integrable. By Fatou’s lemma,

$\displaystyle \int_{\Omega} f\, \mathrm d\mu = \int_{\Omega} \liminf_{n \to \infty} f_n\, \mathrm d\mu \leq \lim_{n \to \infty} \int_{\Omega} f_n\, \mathrm d\mu \leq \int_{\Omega} g\, \mathrm d\mu < \infty,$

since $g$ is integrable, so that $f$ is integrable.

On the other hand, applying Fatou’s lemma to the nonnegative sequence $g - f_n \to g - f$ , since $g-f$ is integrable,

$\displaystyle \int_{\Omega} (g-f)\, \mathrm d\mu \leq \lim_{n \to \infty} \int_{\Omega} (g-f_n)\, \mathrm d\mu.$

By the linearity of integration and algebruh,

$\begin{aligned} \lim_{n \to \infty} \int_{\Omega} f_n\, \mathrm d\mu &= \int_{\Omega} g\, \mathrm d\mu-\lim_{n \to \infty} \int_{\Omega} (g-f_n)\, \mathrm d\mu \\ &\leq \int_{\Omega} g\, \mathrm d\mu-\int_{\Omega} (g-f)\, \mathrm d\mu = \int_{\Omega} f\, \mathrm d\mu. \end{aligned}$

For the general case, decompose $f_n = f_n^+ - f_n^-$ and $f = f^+ - f^-$ . Apply the nonnegative case to $f_n^+ \to f^+$ and $f_n^- \to f^-$ respectively since $|f_n^+| \leq g$ and $|f_n^-| \leq g$ . Combine the integrals to yield the desired result.

There are other measure-theoretic tools that can help us make sense of probability, but we will leave them to the next post. Here, we established the trifecta of measure theory up and front, ready-to-use for any future use case.

—Joel Kindiak, 13 Jul 25, 1439H
October 28, 2025
A Student’s Nightmare
Let $X_1,X_2,\dots$ be i.i.d. random variables that denote the scores on a recent exam, with mean $\mu$ and standard deviation $\sigma$ . Suppose students used to score $\mu_0$ marks on an end-year exam. This time, however, students seem rather dejected after leaving the exam hall, lamenting that the exam was a lot more challenging than before.

We can use hypothesis testing to determine with some reasonably small error $\alpha \in (0, 1)$ whether or not their claim holds weight; that is, whether the exam was harder, and thus, the population mean score $\mu$ has decreased. The default case $\mu = \mu_0$ is called the null hypothesis, denoted $\mathrm H_0$ , and the proposed change $\mu < \mu_0$ is called the alternative hypothesis, denoted $\mathrm H_1$ . Usually, we abbreviate as follows

$\mathrm H_0 : \mu = \mu_0 \quad \text{vs.} \quad \mathrm H_1 : \mu < \mu_0.$

How do we go about testing this hypothesis? We will presume the innocence of the defendant $\mathrm H_0$ until we have sufficient evidence to convict it guilty, concluding $\mathrm H_1$ . Under $\mathrm H_0$ , which states that $\mu = \mu_0$ , we have $\bar X_n \sim \mathcal N(\mu_0, \sigma^2/n)$ approximately by the central limit theorem.

We then go and sample $n$ students, obtaining the sample points $X_1 = x_1, \dots, X_n = x_n$ , and we can compute

$\displaystyle \bar x := \frac 1n \cdot \sum_{i=1}^n x_i,$

since $\bar X_n := \frac 1n \sum_{i=1}^n X_i$ is an unbiased estimator for $\mu$ . But how do we estimate $\sigma^2$ ? We use the unbiased estimator $S_n^2 := \frac{n}{n-1} \cdot \tilde S_n^2$ , where

$\displaystyle \tilde S_n^2 = \frac 1n \cdot \sum_{i=1}^n (X_i - \bar X_n)^2 \quad \Rightarrow \quad S_n^2 = \frac 1{n-1} \cdot \sum_{i=1}^n (X_i - \bar X_n)^2.$

If each $X_i$ are normally distributed, so is $\bar X_n$ . What would be the distribution of $S_n^2$ ?

Theorem 1. For $n \in \mathbb N^+$ , if $X_1,\dots, X_n \sim \mathcal N(\mu, \sigma^2)$ are i.i.d., then there exist i.i.d. $Z_1,\dots, Z_n \sim \mathcal N(0, 1)$ such that

$\displaystyle \frac{(n-1) \cdot S_n^2}{\sigma^2} = \sum_{i=1}^{n-1} Z_i^2.$

Proof. See Exercise 6 on multivariate normal distributions.

The right-hand side is often abbreviated as the chi-squared distribution with $n-1$ degrees of freedom. We slowly formalise it as follows.

Lemma 1. Suppose $X = Z^2$ , where $Z \sim \mathcal N(0, 1)$ . Then the p.d.f. of $X$ is given by

$\displaystyle f_X(x) = \frac 1{2^{1/2} \cdot \Gamma(1/2)} \cdot x^{-1/2} e^{-x/2},$

where $\Gamma(\cdot)$ denotes the gamma function. Recall that $\Gamma(1/2) = \sqrt{\pi}$ .

Proof. We first remark that $\mathbb P(X < 0) = \mathbb P(Z^2 < 0) = 0$ . Therefore, we restrict our attention to $x \geq 0$ . Then

$\begin{aligned} \mathbb P(X \leq x) &= \mathbb P(Z^2 \leq x) \\ &= \mathbb P(-\sqrt{x} \leq Z \leq \sqrt{x}) \\ &= \mathbb P(Z \leq \sqrt{x}) - \mathbb P(Z \leq -\sqrt{x}). \end{aligned}$

Differentiating and applying the p.d.f. of a standard normal distribution,

$\begin{aligned} f_X(x) &= \frac{\mathrm d}{\mathrm dx} \mathbb P(X \leq x) \\ &= \frac{\mathrm d}{\mathrm dx} (\mathbb P(Z \leq \sqrt{x}) - \mathbb P(Z \leq -\sqrt{x})) \\ &= f_Z(\sqrt{x}) \cdot \frac{1}{2\sqrt x} - f_Z(-\sqrt{x}) \cdot -\frac 1{2\sqrt x}) \\ &= \frac 1{\sqrt x} \cdot f_Z(\sqrt{x}) = \frac 1{\sqrt x} \cdot \frac{1}{\sqrt{2\pi}} \cdot e^{-(\sqrt x)^2/2} \\ &= \frac 1{2^{1/2} \cdot \Gamma(1/2)} \cdot x^{-1/2} e^{-x/2}. \end{aligned}$

Lemma 2. For any $\alpha, \beta > 0$ ,

$\displaystyle \int_0^{1} u^{\alpha-1} \cdot (1-u)^{\beta-1} \, \mathrm du = \frac{\Gamma(\alpha) \cdot \Gamma(\beta)}{\Gamma(\alpha+\beta)}.$

Proof. Assuming all integrals are finite, we first prove that

$\displaystyle \int_{\mathbb R^n} f * g\, \mathrm d\lambda = \left( \int_{\mathbb R^n} f \, \mathrm d\lambda \right) \cdot \left(\int_{\mathbb R^n} g \, \mathrm d\lambda \right),$

where $\lambda$ denotes the Lebesgue measure on $\mathbb R^n$ . Including the dummy variables for readability, Fubini’s theorem and the change-of-variables $\mathbf w = \mathbf x - \mathbf y$ reduces the left-hand side reduces to

$\begin{aligned} \int_{\mathbb R^n} f * g\, \mathrm d\lambda &= \int_{\mathbb R^n} (f * g)(\mathbf x)\, \mathrm d\lambda(\mathbf x) \\ &= \int_{\mathbb R^n} \int_{\mathbb R^n} f(\mathbf y) g(\mathbf x - \mathbf y)\, \mathrm d\lambda(\mathbf y)\, \mathrm d\lambda(\mathbf x) \\ &= \int_{\mathbb R^n} f(\mathbf y) \int_{\mathbb R^n} g(\mathbf x - \mathbf y)\, \mathrm d\lambda(\mathbf x)\, \mathrm d\lambda(\mathbf y) \\ &= \int_{\mathbb R^n} f(\mathbf y) \int_{\mathbb R^n} g(\mathbf w)\, \mathrm d\lambda(\mathbf w)\, \mathrm d\lambda(\mathbf y) \\ &= \left( \int_{\mathbb R^n} f(\mathbf y)\, \mathrm d\lambda(\mathbf y) \right) \cdot \left( \int_{\mathbb R^n} g(\mathbf w)\, \mathrm d\lambda(\mathbf w) \right) \\ &= \left( \int_{\mathbb R^n} f \, \mathrm d\lambda \right) \cdot \left(\int_{\mathbb R^n} g \, \mathrm d\lambda \right). \end{aligned}$

Now define $f_\alpha(t) = t^{\alpha-1} e^{-t} \cdot \mathbb I_{(0,\infty)}(t)$ , so that applying this result to the definition of the gamma function defined by

$\displaystyle \Gamma(z) = \int_{-\infty}^{\infty} t^{z-1} e^{-t} \cdot \mathbb I_{(0,\infty)}(t)\, \mathrm dt$

yields

$\begin{aligned} \Gamma(\alpha) \cdot \Gamma(\beta) &= \left( \int_{-\infty}^{\infty} f_\alpha\, \mathrm d\lambda \right) \cdot \left( \int_{-\infty}^{\infty} f_\beta\, \mathrm d\lambda \right) = \int_{-\infty}^{\infty} (f_\alpha * f_\beta)\, \mathrm d\lambda.\end{aligned}$

Evaluating the convolution, and using the change of variables $s = ut$ ,

$\begin{aligned} (f_\alpha * f_\beta)(t) &= \int_{-\infty}^{\infty} f_\alpha(s) \cdot f_\beta(t-s)\, \mathrm ds \\ &= \int_{0}^{t} s^{\alpha-1} e^{-s} \cdot (t-s)^{\beta-1} e^{-(t-s)} \, \mathrm ds \\ &= e^{-t} \cdot \int_{0}^{t} s^{\alpha-1} \cdot (t-s)^{\beta-1} \, \mathrm ds \\ &= e^{-t} \cdot \int_{0}^{1} (ut)^{\alpha-1} \cdot (t-ut)^{\beta-1} \cdot t \, \mathrm du \\ &= e^{-t} \cdot \int_{0}^{1} t^{\alpha-1} \cdot u^{\alpha - 1} \cdot t^{\beta - 1} \cdot (1-u)^{\beta-1} \cdot t \, \mathrm du \\ &= t^{(\alpha + \beta) - 1} \cdot e^{-t} \cdot\int_{0}^{1} u^{\alpha - 1} \cdot (1-u)^{\beta-1} \, \mathrm du. \end{aligned}$

Integrating on both sides, and applying Fubini’s theorem,

$\begin{aligned}\Gamma(\alpha) \cdot \Gamma(\beta) &= \int_{-\infty}^{\infty} \left( t^{(\alpha + \beta) - 1} \cdot e^{-t} \cdot \int_{0}^{1} u^{\alpha - 1} \cdot (1-u)^{\beta-1} \, \mathrm du \right) \, \mathrm dt \\ &= \left( \int_{-\infty}^{\infty} t^{(\alpha + \beta) - 1} \cdot e^{-t} \, \mathrm dt \right) \cdot \left( \int_{0}^{1} u^{\alpha - 1} \cdot (1-u)^{\beta-1} \, \mathrm du \right) \\ &= \Gamma(\alpha + \beta) \cdot \left( \int_{0}^{1} u^{\alpha - 1} \cdot (1-u)^{\beta-1} \, \mathrm du \right), \end{aligned}$

and the result follow by algebruh.

Definition 1. The random variable $W$ is said to follow a chi-squared distribution with $\nu > 0$ degrees of freedom, denoted $W \sim \chi_\nu^2$ , if its density function is given by

$\displaystyle f_W(w) = \frac 1{2^{\nu/2} \cdot \Gamma(\nu/2)} \cdot w^{\nu/2-1} \cdot e^{- w/2}.$

Lemma 3. For any $\mu, \nu$ , if $X \sim \chi_\mu^2$ and $Y \sim \chi_\nu^2$ are independent, then $W := X +Y \sim \chi_{\mu + \nu}^2$ .

Proof. Denoting $\alpha := \mu/2$ , $\beta := \nu/2$ , taking convolutions and applying Lemma 2,

$\begin{aligned} f_W(w) &= \int_0^{w} f_X(x) f_Y(w-x)\, \mathrm dx \\ &= \int_0^{w} \frac 1{2^{\alpha} \cdot \Gamma(\alpha)} \cdot x^{\alpha-1} \cdot \frac 1{ \Gamma(\beta)} \cdot (w-x)^{\beta-1} e^{- w/2}\, \mathrm dy \\ &= \frac 1{2^{\alpha+\beta} \cdot \Gamma(\alpha+\beta)} \cdot e^{- w/2} \cdot \frac {\Gamma(\alpha+\beta)}{\Gamma(\alpha) \cdot \Gamma(\beta)} \cdot \int_0^{w} x^{\alpha - 1} \cdot (w-x)^{\beta - 1}\, \mathrm dx \\ &= \frac 1{2^{\alpha +\beta} \cdot \Gamma(\alpha +\beta)} \cdot e^{- w/2} \cdot \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha) \cdot \Gamma(\beta)} \int_0^{1} (uw)^{\alpha-1} \cdot (w-uw)^{\beta-1} \cdot w \, \mathrm du \\ &= \frac 1{2^{\alpha +\beta} \cdot \Gamma(\alpha+\beta)} \cdot w^{\alpha +\beta-1} \cdot e^{- w/2} \cdot \underbrace{ \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha) \cdot \Gamma(\beta)} \cdot \int_0^{1} u^{\alpha-1} \cdot (1-u)^{\beta-1} \, \mathrm du }_1 \\ &= \frac 1{2^{(\mu + \nu)/2} \cdot \Gamma((\mu + \nu)/2)} \cdot w^{(\mu + \nu)/2-1} \cdot e^{- w/2}. \end{aligned}$

Lemma 4. Fix $n \in \mathbb N^+$ . Then $W \sim \chi_n^2$ if and only if there exist i.i.d. $Z_1, \dots, Z_{n} \sim \mathcal N(0, 1)$ such that $W = \sum_{i=1}^{n} Z_i^2$ .

Proof. The direction $(\Leftarrow)$ is the content of Lemma 3, and the direction $(\Rightarrow)$ follows an argument by induction applied to Lemma 3 again.

Example 1. As defined in Theorem 1, $(n-1) \cdot S_n^2 / \sigma^2 \sim \chi_{n-1}^2$ .

Let $X_1,\dots, X_n$ be i.i.d. with mean $\mu$ and known variance $\sigma^2$ , The central limit theorem (which we aim to eventually prove) tells us that regardless of the underlying distribution of $X_1,\dots, X_n$ ,

$\displaystyle \bar X_n \sim \mathcal N \left( \mu , \frac{\sigma^2}{n} \right) \quad \text{approximately}.$

Equivalently, we have that

$\displaystyle \frac{\bar X_n - \mu}{\sigma/\sqrt n} \sim \mathcal N(0, 1)\quad \text{approximately}.$

Of course, if the $X_i$ terms are normally distributed, then this statement holds exactly.

If $\sigma^2$ is unknown, however, then such a conclusion isn’t very useful, since our estimate $(\bar x_n - \mu)/(\sigma / \sqrt n)$ can only be expressed in terms of $\sigma$ . However, the quantity $S_n^2$ is an unbiased estimate of $\sigma^2$ . Furthermore, if the $X_i$ terms are normally distributed, then Example 1 characterises the distribution of $S_n^2$ via

$\displaystyle \frac{(n-1) S_n^2}{\sigma^2} \sim \chi_{n-1}^2.$

What would be the distribution of the modified random variable $T_n$ defined below?

$\displaystyle T_n := \frac{\bar X_n - \mu}{S_n/\sqrt n}$

By algebraic manipulation,

$\displaystyle T_n = \frac{\bar X_n - \mu}{\sigma/\sqrt n} \cdot \frac{\sigma}{S_n} = Z \cdot \frac 1{\sqrt{(n-1) \cdot \frac{S_n^2}{\sigma^2} / (n-1)}}.$

Definition 2. A random variable $T$ is said to follow a Student’s $t$ -distribution with $\nu$ degrees of freedom, denoted $T \sim t(\nu)$ , if there exist independent $Z \sim \mathcal N(0, 1)$ and $W \sim \chi_\nu^2$ such that

$\displaystyle T := \frac{Z}{\sqrt{W / \nu}}.$

In particular, $\displaystyle \frac{\bar X_n - \mu}{S_n/\sqrt n} \sim t(n-1)$ . As such, we recover the classic $t$ -test for statistical hypothesis testing by modelling the data, assuming the underlying data follows a normal distribution.

To use the $t$ -distribution well, we would need its p.d.f., so that we can use numerical integration techniques to estimate various crucial probabilities that we can tabulate nicely in a table.

Theorem 2. If $T \sim t(\nu)$ , then $T$ has a p.d.f. $f_T$ given by

$\displaystyle f_T(t) = \frac{\Gamma((\nu+1)/2)}{\Gamma (\nu/2) \cdot \sqrt{\nu \pi}} \cdot \left( 1 + \frac{t^2}{\nu} \right)^{-(\nu+1)/2}.$

Proof. By definition, find i.i.d. $Z \sim \mathcal N(0, 1)$ and $W \sim \chi_\nu^2$ such that $T = Z/\sqrt{W / \nu}$ . Denote $F_T(t) := \mathbb P(T \leq t)$ . By Fubini’s theorem,

$\displaystyle \begin{aligned} F_T(t) &= \mathbb P\left( \frac{Z}{\sqrt{W / \nu}} \leq t \right) \\ &= \int_{\mathbb R^2} \mathbb I_{z \leq t\sqrt{w/\nu}}(z,w) \cdot f_Z(z) \cdot f_W(w)\, \mathrm dz\, \mathrm dw \\ &= \int_{0}^\infty \int_{-\infty}^{t\sqrt{w/\nu}} f_Z(z) \cdot f_W(w)\, \mathrm dz \, \mathrm dw. \end{aligned}$

Differentiating under the integral sign,

$\displaystyle \begin{aligned} f_T(t) = \frac{\mathrm d}{\mathrm dt} ( F_T(t) ) &= \int_{0}^\infty \frac{\partial}{\partial t} \int_{-\infty}^{t\sqrt{w/\nu}} f_Z(z) \cdot f_W(w)\, \mathrm dz \, \mathrm dw \\ &= \int_{0}^\infty f_Z(t\sqrt{w/\nu}) \cdot f_W(w) \cdot \sqrt{\frac w\nu} \, \mathrm dw. \end{aligned}$

We leave it as an exercise to verify that

$\displaystyle f_Z(t\sqrt{w/\nu}) \cdot f_W(w) \cdot \sqrt{\frac w\nu} = w^{(\nu-1)/2} \cdot e^{-\frac w2\left( 1 + \frac{t^2}{\nu} \right)},$

where $c = (2^{(\nu+1)/2} \cdot \Gamma (\nu/2) \cdot \sqrt{\nu \pi})^{-1}$ , so that by the substitution $u = \frac w2\left( 1 + \frac{t^2}{\nu} \right)$ ,

$\begin{aligned} f_T(t) &= c \cdot \int_{0}^\infty \left( \frac 2{ 1 + \frac{t^2}{\nu}} \cdot u \right)^{(\nu-1)/2} \cdot e^{-u} \cdot \frac 2{ 1 + \frac{t^2}{\nu}} \, \mathrm du \\ &= c \cdot \left( \frac 2{ 1 + \frac{t^2}{\nu}} \right)^{(\nu+1)/2} \int_{0}^\infty u^{(\nu+1)/2 - 1} \cdot e^{-u}\, \mathrm du \\ &= c \cdot 2^{(\nu+1)/2} \cdot \left( 1 + \frac{t^2}{\nu} \right)^{-(\nu+1)/2} \cdot \Gamma((\nu+1)/2) \\ &= \frac{\Gamma((\nu+1)/2)}{\Gamma (\nu/2) \cdot \sqrt{\nu \pi}} \cdot \left( 1 + \frac{t^2}{\nu} \right)^{-(\nu+1)/2}. \end{aligned}$

Recall our original question: we wanted to analyse if students’ scores have decreased, by testing the hypothesis $\mathrm H_0$ against $\mathrm H_1$ given by

$\mathrm H_0 : \mu = \mu_0,\quad \text{vs.}\quad \mathrm H_1 : \mu < \mu_0.$

Assuming our students’ scores follow a normal distribution (which happens frequently enough to be an acceptable assumption), we can obtain a random sample $X_1,\dots, X_n$ and compute the $t$ -statistic $T \sim t(n-1)$ given by the expressions

$\displaystyle \bar X_n := \frac 1n \cdot \sum_{i=1}^n X_i,\quad S_n^2 := \frac 1{n-1} \cdot \sum_{i=1}^n (X_i - \bar X_n)^2, \quad T:= \frac{\bar X_n - \mu_0 }{ S_n / \sqrt n }.$

By collecting data to evaluate $\bar X_n = \bar x_n$ and $S_n = s_n$ , we obtain a $t$ -value of

$\displaystyle t := \frac{\bar x_n - \mu_0 }{s_n / \sqrt{n}}.$

The instance $t < 0$ inches towards our suspicions of marks having decreased being correct. Since $T \sim t(n-1)$ , we can compute the $p$ -value $p := \mathbb P(T < t)$ .

If instead we don’t know that student’s scores follow a normal distribution but we do know what the standard deviation of their scores is, then the central limit theorem tells us the standardized random variable of $\bar X_n$ is approximately normally distributed (that is, it converges in distribution to the standard normal distribution), so that we can regard

$\displaystyle \frac{ \bar X_n - \mu_0 }{\sigma/ \sqrt n} \approx Z \sim \mathcal N(0, 1).$

In this case, our $p$ -value is computed using the $z$ -value:

$\displaystyle z := \frac{\bar x_n - \mu_0}{\sigma / \sqrt{n}},\quad p := \mathbb P(Z < z).$

How do we reject $\mathrm H_0$ ? Notice that in the $t$ -value and $z$ -value calculations, we assume $\mathrm H_0$ holds, that is, $\mu = \mu_0$ , and substituted accordingly. The larger that $|t|$ or $|z|$ is away from $0$ , the greater the normalised change, and so the smaller the value of $p$ .

How small is sufficiently small for us to reject $\mathrm H_0$ ? No one knows. If you require $p \leq 0$ , then there is no way that we can reject $\mathrm H_0$ . Otherwise, if you require $p \leq \alpha$ for some predetermined $\alpha \in (0, 1)$ of your choice, commonly $\alpha = 0.05$ (or in physicists’ case, $\alpha \approx 5.7 \times 10^{-7}$ , known as the 5-sigma-rule), there is a chance of rejecting $\mathrm H_0$ . We call $\alpha$ the level of significance, and reject $\mathrm H_0$ if and only if $p \leq \alpha$ .

This form of statistical hypothesis testing is commonly used in sciences—physical and social—by interpreting collected data and the statistics it suggests about the underlying distribution. Yet, our conclusions may be wrong.
- If we rejected $\mathrm H_0$ when we shouldn’t have, we call it a type-I error.
- If we did not reject $\mathrm H_0$ when we should have, we call it the type-II error.
We note that $\alpha$ refers to our maximum chosen probability of unintentionally committing a type-I error.

Finally, hypothesis tests pertaining an existing population mean commonly take on three flavours:
- Left-tailed: $\mathrm H_0 : \mu = \mu_0$ vs. $\mathrm H_1 : \mu < \mu_0$ ,
- Right-tailed: $\mathrm H_0 : \mu = \mu_0$ vs. $\mathrm H_1 : \mu > \mu_0$ ,
- Two-tailed: $\mathrm H_0 : \mu = \mu_0$ vs. $\mathrm H_1 : \mu \neq \mu_0$ .
Now we return to our original quest: proving the central limit theorem. We will need to revisit distribution functions and characteristic functions for a not-too-challenging proof of the central limit theorem, at least for the undergraduate context.

—Joel Kindiak, 26 Jul 25, 2103H
October 23, 2025
The Lebesgue Integral

Now that we have properly constructed the Lebesgue measure $\lambda$ on a suitably defined $\sigma$ -algebra $\mathcal F$ on $\mathbb R$ that effectively calculates lengths of intervals (for instance, $\lambda([a, b]) = b-a$ ). Just like constructing the real numbers, we constructed this measure formally for purely logical consistency reasons—the fun starts when we use these constructions to solve problems.

Continuous random variables $X$ are often modelled using continuous probability density functions $f_X$ that define their distributions $\mathbb P_X$ by

$\displaystyle \mathbb P_X(K) := \int_K f_X(x)\, \mathrm dx \equiv \int_{\mathbb R} (f_X \cdot \mathbb I_{K})(x)\, \mathrm dx.$

By construction, we require $\mathbb P_X(\mathbb R) = 1$ , so that

$\displaystyle \int_{\mathbb R} f_X(x)\, \mathrm dx = 1.$

This construction suggests a need for a robust notion of integration that accounts for Lebesgue measurable sets $K$ (i.e. $K \in \mathcal F$ ), that is even stronger than usual Riemann integration.

Recall the definition of a measurable function.

Definition 1. Let $(\Omega, \mathcal F)$ and $(\Psi, \mathcal G)$ be measurable spaces. We call $f : \Omega \to \Psi$ $\mathcal F/\mathcal G$ -measurable if $f^{-1}(K) \in \mathcal F$ for any $K \in \mathcal G$ , and omit the prefix when there is no ambiguity.

Unless stated otherwise, we abbreviate $\mathbb R \equiv (\mathbb R, \frak{B}(\mathbb R))$ , where $\frak{B}(\mathbb R)$ refers to the Borel $\sigma$ -algebra on $\mathbb R$ . Similarly, we equip $[-\infty, \infty]$ with the Borel $\sigma$ -algebra $\frak{B}([-\infty, \infty])$ . Henceforth, let $(\Omega, \mathcal F)$ be a measurable space.

Lemma 1. For any $K \subseteq \Omega$ , $\mathbb I_K : \Omega \to \mathbb R$ is measurable if and only if $K$ is measurable.

Let’s now equip $(\Omega, \mathcal F)$ with a measure $\mu$ . Using this measure, we can define integration properly. Rather intuitively, for any measurable $K$ , we should define

$\displaystyle \int_{\Omega} \mathbb I_K\, \mathrm d\mu := \mu(K).$

The idea is to define $\int_{\mathbb R} f\, \mathrm d\mu$ using these indicator functions by linear extensions, and we will do so slowly.

Definition 2. A measurable function $f : \Omega \to [-\infty, \infty]$ is simple if $f(\Omega)$ is a finite set. Denote $K_i := f^{-1}(\{ x_i \})$ , so that

$\displaystyle f = \sum_{i=1}^n x_i \cdot \mathbb I_{K_i}.$

If $f$ is non-negative, then we define by linearity

$\displaystyle \int_{\Omega} f\, \mathrm d\mu := \sum_{i=1}^n x_i \cdot \mu(\mathbb I_{K_i}).$

Lemma 2. Let $f , g: \Omega \to [-\infty, \infty]$ be simple functions. Then $f+g$ is simple, and for any $\alpha \in \mathbb R$ , $\alpha f$ is also simple. Furthermore, if $f,g$ are non-negative, so is $f+g$ , as well as $\alpha f$ whenever $\alpha \geq 0$ . In these cases,

$\begin{aligned} \int_{\Omega} (f+g)\, \mathrm d\mu = \int_{\Omega} f\, \mathrm d\mu + \int_{\Omega} g\, \mathrm d\mu,\quad \int_{\Omega} \alpha f\, \mathrm d\mu = \alpha \cdot \int_{\Omega} f\, \mathrm d\mu . \end{aligned}$

Proof. Write

$\displaystyle f = \sum_{i=1}^n x_i \cdot \mathbb I_{K_i}, \quad g = \sum_{j=1}^m y_j \cdot \mathbb I_{L_j}.$

Defining $M_{i,j} := K_i \cap L_j$ , we observe that

$\begin{aligned} \mathbb I_{K_i} &= \mathbb I_{K_i} \cdot \mathbb I_{\Omega} = \mathbb I_{K_i} \cdot \sum_{j=1}^m \mathbb I_{L_j} = \sum_{j=1}^m \mathbb I_{K_i}\mathbb I_{L_j} = \sum_{j=1}^m \mathbb I_{K_i \cap L_j} = \sum_{j=1}^m \mathbb I_{M_{i,j}}. \end{aligned}$

Similarly, $\mathbb I_{L_j} = \sum_{i=1}^n \mathbb I_{M_{i,j}}$ . Hence,

$\begin{aligned} f + g &= \sum_{i=1}^n x_i \cdot \mathbb I_{K_i}+ \sum_{j=1}^m y_j \cdot \mathbb I_{L_j} \\ &= \sum_{i=1}^n x_i \cdot \sum_{j=1}^m \mathbb I_{M_{i,j}} + \sum_{j=1}^m y_j \cdot \sum_{i=1}^n \mathbb I_{M_{i,j}} \\ &= \sum_{i=1}^n \sum_{j=1}^m (x_i + y_j) \cdot \mathbb I_{M_{i,j}}. \end{aligned}$

Since $\{x_i + y_j : i, j\}$ is finite, $f+g$ is a simple function. To compute its integral, we remark that

$\displaystyle \bigsqcup_{i=1}^n M_{i,j} = \bigsqcup_{i=1}^n (K_i \cap L_j) = L_j \cap \bigsqcup_{i=1}^n K_i = L_j \cap \Omega = L_j.$

Similarly, $\bigsqcup_{j=1}^m M_{i,j} = K_i$ , so that

$\displaystyle \sum_{i=1}^n \mu( M_{i,j} ) = \mu(K_i), \quad \sum_{j=1}^m \mu( M_{i,j} ) = \mu(L_j).$

Hence,

$\begin{aligned} \int_{\Omega} (f+g)\, \mathrm d\mu &= \sum_{i=1}^n \sum_{j=1}^m (x_i + y_j) \mu(M_{i,j}) \\ &= \sum_{i=1}^n \sum_{j=1}^m x_i \mu(M_{i,j}) + \sum_{i=1}^n \sum_{j=1}^m y_j \mu(M_{i,j}) \\ &= \sum_{i=1}^n x_i \sum_{j=1}^m \mu(M_{i,j}) + \sum_{j=1}^m y_j \sum_{i=1}^n \mu(M_{i,j}) \\ &= \sum_{i=1}^n x_i \mu(K_i) + \sum_{j=1}^m y_j \mu(L_j) \\ &= \int_{\Omega} f\, \mathrm d\mu + \int_{\Omega} g\, \mathrm d\mu. \end{aligned}$

The proof of the other result is similar, and simpler (pun intended).

Having defined (possibly infinite) integrals of non-negative simple functions, we shall extend our ideas a little bit to encompass non-negative functions.

Definition 3. Let $f : \Omega \to [0, \infty]$ be a measurable function. Clearly, $\varphi = 0$ is a simple function that satisfies the inequality $0 \leq \varphi \leq f$ . Thus, we define

$\displaystyle \int_{\Omega} f\, \mathrm d\mu := \sup \left\{ \int_{\Omega} \varphi\, \mathrm d\mu : \varphi\ \text{simple}, 0\leq \varphi \leq f \right\}.$

Do we recover the same properties in Lemma 2? The answer is yes.

Lemma 3. Let $f, g : \Omega \to [0, \infty]$ be measurable functions. Then $f+g : \Omega \to [0, \infty]$ is measurable, and

$\displaystyle \int_{\Omega} (f + g)\, \mathrm d\mu = \int_{\Omega} f\, \mathrm d\mu + \int_{\Omega} g\, \mathrm d\mu.$

Likewise, for $\alpha \in [0, \infty]$ , $\alpha f : \Omega \to [0, \infty]$ is measurable and

$\displaystyle \int_{\Omega} \alpha f \, \mathrm d\mu = \alpha \int_{\Omega} f\, \mathrm d\mu .$

Proof. Fix $\epsilon > 0$ . By definition, there exists a simple function $\varphi : \Omega \to [0, \infty]$ such that $\varphi \leq f$ and

$\displaystyle \int_{\Omega} f\, \mathrm d\mu - \epsilon \leq \int_{\Omega} \varphi\, \mathrm d\mu \leq \int_{\Omega} f\, \mathrm d\mu.$

Similarly, there exists a simple function $\psi : \Omega \to [0, \infty]$ such that $\psi \leq g$ and

$\displaystyle \int_{\Omega} g\, \mathrm d\mu - \epsilon \leq \int_{\Omega} \psi\, \mathrm d\mu \leq \int_{\Omega} g\, \mathrm d\mu.$

Since $\varphi + \psi : \Omega \to [0,\infty]$ is a simple function,

$\displaystyle \left( \int_{\Omega} f\, \mathrm d\mu + \int_{\Omega} g\, \mathrm d\mu \right) - 2\epsilon \leq \int_{\Omega} (\varphi + \psi)\, \mathrm d\mu \leq \int_{\Omega} f\, \mathrm d\mu + \int_{\Omega} g\, \mathrm d\mu.$

Furthermore, since $\varphi + \psi \leq f + g$ ,

$\displaystyle \left( \int_{\Omega} f\, \mathrm d\mu + \int_{\Omega} g\, \mathrm d\mu \right) - 2\epsilon \leq \int_{\Omega} (f + g)\, \mathrm d\mu \leq \int_{\Omega} f\, \mathrm d\mu + \int_{\Omega} g\, \mathrm d\mu.$

Taking $\epsilon \to 0^+$ yields the desired result.

Lemma 4. For any measurable $K$ , define

$\displaystyle \int_K f\, \mathrm d\mu := \int_{\Omega} f \cdot \mathbb I_K\, \mathrm d\mu.$

Then for measurable $f \geq 0$ and measurable $K_1, \dots, K_n$ , if $K= \bigsqcup_{i=1}^n K_i$ , then

$\displaystyle \int_K f\, \mathrm d\mu = \sum_{i=1}^n \int_{K_i} f\, \mathrm d\mu.$

Proof. We observe that $\mathbb I_K = \sum_{i=1}^n \mathbb I_{K_i}$ , so that Lemma 3 yields

$\begin{aligned} \int_K f \, \mathrm d\mu &= \int_{\Omega} f \cdot \mathbb I_K \, \mathrm d\mu \\ &= \int_{\Omega} f \cdot \sum_{i=1}^n \mathbb I_{K_i} \, \mathrm d\mu = \int_{\Omega} \sum_{i=1}^n f \cdot \mathbb I_{K_i} \, \mathrm d \mu \\ &= \sum_{i=1}^n \int_{\Omega} f \cdot \mathbb I_{K_i} \, \mathrm d \mu = \sum_{i=1}^n \int_{K_i} f \, \mathrm d\mu. \end{aligned}$

Definition 4. Let $f : \Omega \to \mathbb R$ be measurable. Define the non-negative functions

$f^+ := \max\{0, f\}, f^{-} := \max\{0, -f\} : \Omega \to [0,\infty).$

We say that $f$ is $\mu$ –integrable if $\int_{\Omega} f^+\, \mathrm d\mu$ and $\int_{\Omega} f^-\, \mathrm d\mu$ are finite, and define its integral by their difference:

$\displaystyle \int_{\Omega} f\, \mathrm d\mu := \int_{\Omega} f^+\, \mathrm d\mu - \int_{\Omega} f^-\, \mathrm d\mu.$

We omit the prefix when there is no ambiguity. We say that a function is Lebesgue-integrable if it is $\lambda$ -integrable, where $\lambda$ denotes the Lebesgue measure.

Using Lemma 3, it should be obvious that the integral is linear.

Theorem 1. If $f, g : \Omega \to \mathbb R$ are integrable, then so is $f+g$ , and

$\displaystyle \int_{\Omega} ( f + g ) \, \mathrm d\mu = \int_{\Omega} f \, \mathrm d\mu + \int_{\Omega} g \, \mathrm d\mu.$

Furthermore, for any $\alpha \in \mathbb R$ ,

$\displaystyle \int_{\Omega} \alpha f \, \mathrm d\mu = \alpha \int_{\Omega} f\, \mathrm d\mu.$

Proof. We leave the scalar multiplication case as a relatively routine exercise in case-splitting. It turns put that we will need the special case $\alpha = -1$ for additivity. The idea is to find a useful disjoint union $\Omega = \bigsqcup_{i=1}^n K_i$ , so that

$\begin{aligned} \int_{\Omega} (f+g) \, \mathrm d\mu &= \sum_{i=1}^n \int_{K_i} (f+g)\, \mathrm d\mu \\ &= \sum_{i=1}^n \left( \int_{K_i} f\, \mathrm d\mu + \int_{K_i} g\, \mathrm d\mu \right) \\&= \sum_{i=1}^n \int_{K_i} f\, \mathrm d\mu + \sum_{i=1}^n \int_{K_i} g\, \mathrm d\mu \\ &= \int_{\Omega} f \, \mathrm d\mu + \int_{\Omega} g \, \mathrm d\mu.\end{aligned}$

Now consider $(f+g)^+ = \max\{0, f+g\}$ and $(f+g)^- = \max\{0, -(f+g)\}$ . Define $K_f^+ := \{ \omega \in \Omega: f(\omega) \geq 0\}$ . Similarly define $K_f^-, K_g^+, K_g^-$ . By observation,

$\begin{aligned} K_f^+ \cap K_g^+ \subseteq K_{f+g}^+,\quad K_f^- \cap K_g^- \subseteq K_{f+g}^-. \end{aligned}$

On the other hand for any $\omega \in K_f^+ \cap K_g^-$ , $\omega \in K_{f+g}^+$ if and only if

$f(\omega)+g(\omega) \geq 0 \iff |f(\omega)| = f (\omega)\geq -g(\omega) = |g(\omega)|.$

Hence, define $L_{\geq} := \{ \omega \in \Omega : |f(\omega)| \geq |g(\omega)| \}$ and $L_{\leq}$ similarly. Then

$\begin{aligned} K_{f+g}^+ &= ( \underbrace{ K_f^+ \cap K_g^+ }_{K_1} ) \sqcup (\underbrace{ K_f^+ \cap K_g^- \cap L_{\geq} }_{K_2}) \sqcup ( \underbrace{ K_f^- \cap K_g^+ \cap L_{\leq} }_{K_3} ), \\ K_{f+g}^- &= ( \underbrace{ K_f^- \cap K_g^- }_{K_4} ) \sqcup (\underbrace{ K_f^- \cap K_g^+ \cap L_{\geq} }_{K_5}) \sqcup (\underbrace{ K_f^+ \cap K_g^- \cap L_{\leq} }_{K_6}). \end{aligned}$

We remark that $(f+g)|_{K_2}, -g|_{K_2}$ are non-negative and hence

$\displaystyle \begin{aligned} \int_{K_2} f\, \mathrm d\mu = \int_{K_2} ((f+g) + (-g))\, \mathrm d\mu &= \int_{K_2} (f+g) \, \mathrm d\mu + \int_{K_2} (-g) \, \mathrm d\mu, \end{aligned}$

so that

$\begin{aligned} \int_{K_2} (f+g) \, \mathrm d\mu &= \int_{K_2} f \, \mathrm d\mu - \int_{K_2} (-g) \, \mathrm d\mu \\ &= \int_{K_2} f \, \mathrm d\mu - (-1)\int_{K_2} g \, \mathrm d\mu \\ &= \int_{K_2} f \, \mathrm d\mu + \int_{K_2} g \, \mathrm d\mu. \end{aligned}$

This result follows similarly for $K_3,K_5,K_6$ . Since $\Omega = \bigsqcup_{i=1}^6 K_i$ , the result follows.

Definition 5. Let $\mathbb P$ be a probability measure on $(\Omega, \mathcal F)$ . For any random variable $X : \Omega \to \mathbb R$ , the expectation of $X$ is defined by

$\displaystyle \mathbb E[X] := \int_{\Omega} X\, \mathrm d\mathbb P,$

whenever the integral is finite.

The immediate application of proving linearity therefore is to recover the famous additivity of expectation:

$\begin{aligned} \mathbb E[X+Y] &= \int_{\Omega} (X+Y)\, \mathrm d\mathbb P \\ &= \int_{\Omega} X\, \mathrm d\mathbb P + \int_{\Omega} Y\, \mathrm d\mathbb P \\ &= \mathbb E[X] + \mathbb E[Y]. \end{aligned}$

In fact, if $X(\Omega)$ is finite, then we have already proven this result. But we need to think bigger, and explore the three crucial convergence theorems in measure theory.

—Joel Kindiak, 12 Jul 25, 1256H

October 21, 2025