KindiakMath

Category: Probability

The Gaussian Integral

Problem 1. Evaluate the Gaussian integral $\displaystyle \int_{-\infty}^{\infty} e^{-x^2}\, \mathrm dx$ .

(Click for Solution)

Solution. Since $e^{-x^2}$ is even in $x$ ,

$\displaystyle \int_{-\infty}^{\infty} e^{-x^2}\, \mathrm dx = 2 \cdot \int_{0}^{\infty} e^{-x^2}\, \mathrm dx.$

For the right-hand side,

$\begin{aligned}\left( \int_{0}^{\infty} e^{-x^2}\, \mathrm dx \right)^2 &= \left( \int_{0}^{\infty} e^{-x^2}\, \mathrm dx \right) \cdot \left( \int_{0}^{\infty} e^{-y^2}\, \mathrm dy \right) \\ &= \int_0^\infty e^{-x^2} \cdot \left( \int_0^\infty e^{-y^2}\, \mathrm dy \right)\, \mathrm dx \\ &= \int_0^\infty e^{-x^2} \cdot \left( \int_0^\infty e^{-u^2x^2} \cdot x\, \mathrm du \right)\, \mathrm dx \\ &= \int_0^\infty \int_0^\infty xe^{-x^2(u^2+1)} \, \mathrm du\, \mathrm dx. \end{aligned}$

By Fubini’s theorem,

$\begin{aligned}\left( \int_{0}^{\infty} e^{-x^2}\, \mathrm dx \right)^2 &= \int_0^\infty \int_0^\infty xe^{-x^2(u^2+1)} \, \mathrm du\, \mathrm dx\\ &= \int_0^\infty \int_0^\infty xe^{-x^2(u^2+1)} \, \mathrm dx\, \mathrm du \\ &= \int_0^\infty \left[ -\frac 1{2(u^2+1)} \cdot e^{-x^2(u^2+1)} \right]_0^\infty\, \mathrm du \\ &= \int_0^\infty \frac 1{2(u^2+1)}\, \mathrm du \\ &= \frac 12 \cdot [\tan^{-1}(u)]_0^{\infty} \\ &= \frac 12 \cdot \frac{\pi}2 = \frac{\pi}{4}. \end{aligned}$

Taking square roots,

$\displaystyle \int_{0}^{\infty} e^{-x^2}\, \mathrm dx = \frac{\sqrt{\pi}}{2} \quad \Rightarrow \quad \int_{-\infty}^{\infty} e^{-x^2}\, \mathrm dx = \sqrt{\pi}.$

This calculation was authored by Hirokazu Iwasawa.

—Joel Kindiak, 26 Jul 25, 2156H

January 2, 2026
The Poisson Distribution

Definition 1. The random variable $X : \Omega \to \mathbb N_0$ follows a Poisson distribution with rate parameter $\lambda > 0$ , denoted $X \sim \mathrm{Pois}(\lambda)$ , if

$\displaystyle \mathbb P(X=x) = c_\lambda \cdot \frac{\lambda^x}{x!}$

for some $c_\lambda > 0$ .

Problem 1. Evaluate $c_\lambda$ .

(Click for Solution)

Solution. We require $\sum_{x = 0}^\infty \mathbb P(X=x) = 1$ :

$\begin{aligned}1 = \sum_{x = 0}^\infty \mathbb P(X=x) &= \sum_{x =0}^\infty c_\lambda \cdot \frac{\lambda^x}{x!} = c_\lambda \cdot \sum_{x = 0}^\infty \frac{\lambda^x}{x!} = c_\lambda \cdot e^{\lambda}. \end{aligned}$

Therefore, $c_\lambda = e^{-\lambda}$ .

Problem 2. Evaluate $\mathbb E[X]$ and $\mathrm{Var}(X)$ .

(Click for Solution)

Solution. By definition of the expectation,

$\begin{aligned} \mathbb E[X] &= \sum_{x = 0}^\infty x \cdot \mathbb P(X=x) \\ &= \sum_{x = 1}^\infty x \cdot c_\lambda \cdot \frac{\lambda^x}{x!} \\ &= \lambda \cdot \sum_{x = 1}^\infty c_\lambda \cdot \frac{\lambda^{x-1}}{(x-1)!} \\ &= \lambda \cdot \sum_{x = 0}^\infty c_\lambda \cdot \frac{\lambda^{x}}{x!} = \lambda \cdot 1 = \lambda. \end{aligned}$

For the variance, we compute the term

$\begin{aligned}\mathbb E[X(X-1)] &= \sum_{x = 0}^\infty x(x-1) \cdot \mathbb P(X=x) \\ &= \sum_{x = 2}^\infty x(x-1) \cdot c_\lambda \cdot \frac{\lambda^x}{x!} \\ &= \lambda^2 \cdot \sum_{x = 2}^\infty c_\lambda \cdot \frac{\lambda^{x-2}}{(x-2)!} \\ &= \lambda^2 \cdot 1 = \lambda^2. \end{aligned}$

Therefore,

$\begin{aligned} \mathrm{Var}(X) &= \mathbb E[X^2] - \mathbb E[X]^2 \\ &= \mathbb E[X(X-1)] + \mathbb E[X] - \mathbb E[X]^2 \\ &= \lambda^2 + \lambda - \lambda^2 = \lambda.\end{aligned}$

Problem 3. Given that $X \sim \mathrm{Pois}(\lambda)$ and $Y \sim \mathrm{Pois}(\mu)$ are independent, determine the distribution of $X + Y$ .

(Click for Solution)

Solution. Denoting $W := X+Y$ , we take the discrete convolution of the p.d.f.s of $X,Y$ to obtain

$\begin{aligned}f_W (w) &= \sum_{x = 0}^w f_X(x) \cdot f_Y(w-x) \\ &= \sum_{x=0}^w c_\lambda \cdot \frac{\lambda^x}{x!} \cdot c_\mu \cdot \frac{\lambda^{w-x}}{(w-x)!} \\ &= \sum_{x=0}^w c_\lambda \cdot \frac{\lambda^x}{x!} \cdot c_\mu \cdot \frac{\mu^{w-x}}{(w-x)!} \\ &= c_\lambda \cdot c_\mu \cdot \sum_{x=0}^w \frac{\lambda^x}{x!} \cdot \frac{\mu^{w-x}}{(w-x)!} \\ &= c_\lambda \cdot c_\mu \cdot \frac 1{w!} \cdot \sum_{x=0}^w \frac{w!}{x! \cdot (w-x)!} \cdot \lambda^x \cdot \mu^{w-x} \\ &= c_\lambda \cdot c_\mu \cdot \frac 1{w!} \cdot \sum_{x=0}^w {w \choose x} \cdot \lambda^x \cdot \mu^{w-x} \\ &= c_\lambda \cdot c_\mu \cdot \frac {(\lambda + \mu)^w}{w!}.\end{aligned}$

Furthermore, $c_\lambda \cdot c_\mu = e^{-\lambda} \cdot e^{\mu} = e^{-(\lambda + \mu)} = c_{\lambda + \mu}$ . Hence,

$\displaystyle \mathbb P(X+Y = w) = c_{\lambda + \mu} \cdot \frac{(\lambda + \mu)^w}{w!},$

so that $X +Y \sim \mathrm{Pois}(\lambda + \mu)$ .

Problem 4. Fix $\lambda > 0$ . Suppose $X_n \sim \mathrm{Bin}(n, \lambda/n)$ and $Y \sim \mathrm{Pois}(\lambda)$ . Prove that $f_{X_n} \to f_Y$ .

(Click for Solution)

Solution. Fix $y \in \mathbb N_0$ . For each $n$ ,

$\begin{aligned} f_{X_n}(y) &= {n \choose y} \left(1 - \frac{\lambda}{n}\right)^{n-y} \left( \frac{\lambda}n \right)^y \\ &= \frac{n(n-1) \cdot \dots \cdot (n-y+1)}{y!} \cdot \frac{\lambda^y }{n^y } \cdot \left(1 - \frac{\lambda}{n}\right)^{n-y} \\ &= \frac{n}{n} \cdot \frac{n-1}{n} \cdot \cdots \cdot \frac{n-y+1}n \cdot \left(1 - \frac{\lambda}{n}\right)^{-y} \cdot \left(1 - \frac{\lambda}{n}\right)^{n} \cdot \frac{\lambda^y }{y! } \end{aligned}$

Taking $n \to \infty$ ,

$\displaystyle \lim_{n \to \infty} f_{X_n}(y) = 1 \cdot 1 \cdot e^{-\lambda} \cdot \frac{\lambda^y}{y!} = e^{-\lambda} \cdot \frac{\lambda^y}{y!} = f_Y(y).$

—Joel Kindiak, 1 Aug 25, 1751H

December 30, 2025
Proving Feynman’s Trick
Feynman’s trick in differentiating under the integral sign has been creatively wielded to evaluate otherwise intractable integrals. In this exercise, we prove Feynman’s trick and use it to evaluate the seemingly intractable Dirichlet integral

$\displaystyle \int_{-\infty}^{\infty} \frac{\sin x}{x}\, \mathrm dx.$

Let $(\Omega, \mathcal F, \mathbb R)$ be a measure space and $f : \Omega \times [a, b] \to \mathbb R$ be a function such that for each $t \in [a, b]$ , $f( \cdot, t)$ is measurable.

Problem 1. Suppose the following conditions:
- For any $\omega \in \Omega$ , $f(\omega, \cdot)$ is continuous.
- There exists some non-negative integrable $g : \Omega \to \mathbb R$ such that for any $(\omega, t) \in \Omega \times [a, b]$ , $|f( \omega ,t)| \leq g(\omega)$ .
Prove that the map $F : [a, b] \to \mathbb R$ defined by $\displaystyle F(t) : = \int_{\Omega} f(\cdot , t)\, \mathrm d\mu$ is continuous.

(Click for Solution)

Solution. Fix $t_n \to t$ . For any $\omega \in \Omega$ , since $f(\omega, \cdot)$ is continuous,

$f(\omega, t_n) \to f(\omega, t),$

so that $f(\cdot, t_n) \to f(\cdot, t)$ pointwise. Furthermore,

$\displaystyle \int_{\Omega} |f(\cdot, t)|\, \mathrm d\mu \leq \int_{\Omega} g\, \mathrm d\mu < \infty,$

so that $f(\cdot, t_n)$ and $f(\cdot, t)$ are all integrable.

Since $g$ is integrable, by Lebesgue’s dominated convergence theorem,

$\displaystyle F(t_n) = \lim_{n \to \infty} \int_{\Omega} f(\cdot , t_n)\, \mathrm d\mu= \int_{\Omega} f(\cdot , t)\, \mathrm d\mu = F(t),$

so that $F$ is continuous, as required.

Problem 2. Suppose the following conditions:
- There exists some $t_0 \in [a, b]$ such that $f(\cdot, t_0)$ is integrable.
- For each $\omega \in \Omega$ , $f(\omega, \cdot)$ is differentiable with derivative at $t_0$ denoted by $\frac{\partial f}{\partial t}(\omega,t_0)$ .
- There exists some non-negative integrable $g : \Omega \to \mathbb R$ such that for any $(\omega, t) \in \Omega \times [a, b]$ , $\left|\frac{\partial f}{\partial t} (\omega,t) \right| \leq g( \omega )$ .
Prove that the map $F : [a, b] \to \mathbb R$ defined by $\displaystyle F(t) : = \int_{\Omega} f(\cdot, t)\, \mathrm d\mu$ is differentiable on $[a, b]$ and

$\displaystyle F'(t) = \frac{\mathrm d}{\mathrm dt} \int_{\Omega} f(\omega, t)\, \mathrm d\mu(\omega) = \int_{\Omega} \frac{\partial f}{\partial t}(\omega, t)\, \mathrm d\mu(\omega).$

(Click for Solution)

Solution. We first check that $F$ is well-defined. By hypothesis, $F(t_0)$ is well-defined. Fix $t \in [a, b]$ . By the mean value theorem, there exists $c$ between $t_0$ and latex t$ such that

$\displaystyle \frac{ |f(\omega, t) - f(\omega, t_0)| }{ |t - t_0| } = \left| \frac{ \partial f }{ \partial t} (\omega, c) \right| \leq g(\omega).$

By performing more analysis, $f(\cdot, t)$ is integrable, so that $F(t)$ is well-defined.

Now fix $\omega \in \Omega$ . For any $t_n \to t$ , since each $f(\cdot, t_n) - f(\cdot, t)$ is measurable,

$\displaystyle \frac{\partial f}{\partial t}(\omega, t) := \lim_{n \to \infty} \underbrace{ \frac{f(\omega, t_n) - f(\omega, t)}{ t_n - t} }_{\varphi_n(\omega)}$

is measurable. Furthermore, $\varphi_n \to \frac{\partial f}{\partial t} (\cdot, t)$ pointwise. We claim that $|\varphi_n| \leq g$ , since the mean value theorem gives $c$ between $t_n$ and $t$ such that

$\displaystyle \frac{ | f(\omega, t_n) - f(\omega, t) |}{ | t_n - t | } = \left| \frac{\partial f}{\partial t}(\omega, c) \right| \leq g(\omega).$

By algebruh and the triangle inequality, each $\varphi_n$ is integrable. Hence, by Lebesgue’s dominated convergence theorem,

$\begin{aligned} \lim_{t_n \to t} \int_{\Omega} \frac{f(\omega, t_n) - f(\omega, t)}{t_n - t}\, \mathrm d\mu(\omega) &= \lim_{n \to \infty} \int_{\Omega} \varphi_n\, \mathrm d\mu \\ &= \int_{\Omega} \lim_{n \to \infty} \varphi_n\, \mathrm d\mu = \int_{\Omega} \frac{\partial f}{\partial t} (\cdot, t)\, \mathrm d\mu. \end{aligned}$

On the other hand, by bookkeeping

$\begin{aligned} \frac{ \mathrm d }{ \mathrm dt } F(t) &= \lim_{ t_n \to t } \frac{ F(t_n) - F(t) }{ t_n - t } \\ &= \lim_{t_n \to t} \int_{\Omega} \frac{f(\omega, t_n) - f(\omega, t)}{t_n - t}\, \mathrm d\mu(\omega) \\ &= \int_{\Omega} \frac{\partial f}{\partial t} (\omega, t)\, \mathrm d\mu(\omega). \end{aligned}$

Therefore,

$\displaystyle F'(t) = \frac{\mathrm d}{\mathrm dt} \int f(\omega, t)\, \mathrm d\mu(\omega) = \int \frac{\partial f}{\partial t}(\omega, t)\, \mathrm d\mu(\omega).$

Remark 1. Thanks to Problem 2, our proof that $\mathcal L\{tf'(t)\} = -\mathcal L\{f\}'(s)$ in the study of differential equations becomes a logically correct one.

Problem 3. Use Problem 2 to evaluate $\displaystyle \int_{-\infty}^{\infty} \frac{\sin x}{x}\, \mathrm dx$ .

(Click for Solution)

Solution. Define the function $I$ by

$\displaystyle I(y) := \int_{0}^{\infty} \frac{\sin x}{x} \cdot e^{-xy}\, \mathrm dy$

that satisfies the hypotheses of Problem 2, and our goal is to evaluate $2 \cdot I(0)$ . Applying Problem 2 and integrating by parts,

$\begin{aligned} I'(y) &= \int_{0}^{\infty} \frac{\partial}{\partial y} \left( \frac{\sin x}{x} \cdot e^{-xy} \right) \, \mathrm dx \\ &= \int_{0}^{\infty} \frac{\sin x}{x} \cdot -xe^{-xy} \, \mathrm dx \\ &= -\int_{0}^{\infty} e^{-xy} \cdot \sin x \, \mathrm dx \\ &= -\left[ \frac{e^{-xy}}{(-y)^2 + 1^2} \cdot ((-y) \sin(x) - \cos(x)) \right]_{0}^{\infty} \\ &= -\frac{1}{1+y^2}. \end{aligned}$

Integrating and applying the first fundamental theorem of calculus,

$\displaystyle I(y) = I(0) - \int_0^y \frac{1}{1+y^2}\, \mathrm dy = I(0) - \tan^{-1}(y).$

Since $I$ is continuous,

$\displaystyle \lim_{y \to \infty} I(y) = \int_0^\infty \lim_{y \to \infty} \frac{\sin x}{x} \cdot e^{-xy}\, \mathrm dx = \int_0^\infty 0\, \mathrm dx = 0.$

Taking $y \to \infty$ on all sides, therefore,

$\displaystyle 0 = I(0) - \frac{\pi}{2} \quad \Rightarrow \quad I(0) = \frac{\pi}{2}.$

Therefore, the Dirichlet integral evaluates to

$\displaystyle \int_{-\infty}^{\infty} \frac{\sin x}{x}\, \mathrm dx = 2 \cdot I(0) = \pi.$

—Joel Kindiak, 29 Jul 25, 1319H
December 26, 2025
Binomial Theorem Corollaries

Recall the binomial theorem, which states that for any $n \in \mathbb N$ and $x \in \mathbb R$ ,

$\displaystyle (1 + x)^n = \sum_{k=0}^n {n \choose k} x^k,$

where we define $0^0 := 1$ by convention.

Problem 1. Prove that for any $a, b \in \mathbb R$ ,

$\displaystyle (a + b)^n = \sum_{k=0}^n {n \choose k} a^{n-k} b^k.$

(Click for Solution)

Solution. If $a = 0$ then the result is trivial. If $a > 0$ , then

$\begin{aligned} (a+b)^n &= a^n \cdot \left(1 + \frac ba\right)^n = a^n \cdot \sum_{k=0}^n \left( \frac ba \right)^k \\ &= a^n \cdot \sum_{k=0}^n a^{-k} \cdot b^k = \sum_{k=0}^n a^{n-k} \cdot b^k. \end{aligned}$

Problem 2. Evaluate the sums $\displaystyle \sum_{k=0}^n {n \choose k}$ and $\displaystyle \sum_{k=0}^n (-1)^k {n \choose k}$ .

(Click for Solution)

Solution. By Problem 1,

$\displaystyle (1 + (\pm 1))^n = \sum_{k=0}^n (\pm 1)^k {n \choose k}.$

Therefore, $\displaystyle \sum_{k=0}^n {n \choose k} = (1+1)^n = 2^n$ and $\displaystyle \sum_{k=0}^n (-1)^k {n \choose k} = (1-1)^n = 0$ .

Problem 3. Prove that $\displaystyle \sum_{k \in \mathbb N_0 :\, 2k \leq n} {n \choose 2k} = \sum_{k \in \mathbb N_0 :\, 2k+1 \leq n} {n \choose 2k+1}$ .

(Click for Solution)

Solution. By Problem 2,

$\begin{aligned} 0=\sum_{k=0}^n (-1)^k {n \choose k} &= \sum_{k \in \mathbb N_0 :\, 2k \leq n} (-1)^{2k} {n \choose 2k} + \sum_{k \in \mathbb N_0 :\, 2k+1 \leq n} (-1)^{2k+1} {n \choose 2k+1} \\ &= \sum_{k \in \mathbb N_0 :\, 2k \leq n} {n \choose 2k} - \sum_{k \in \mathbb N_0 :\, 2k+1 \leq n} {n \choose 2k+1}. \end{aligned}$

Therefore,

$\displaystyle \sum_{k \in \mathbb N_0 :\, 2k \leq n} {n \choose 2k} = \sum_{k \in \mathbb N_0 :\, 2k+1 \leq n} {n \choose 2k+1}.$

Problem 4. Evaluate the sums $\displaystyle \sum_{k= 1}^n k {n \choose k}$ and $\displaystyle \sum_{k= 1}^n k^2 {n \choose k}$ .

(Click for Solution)

Solution. Recall the vanilla binomial theorem

$\displaystyle (1 + x)^n = \sum_{k=0}^n {n \choose k} x^k.$

Differentiating on both sides twice,

$\begin{aligned} n(1+x)^{n-1} &= \sum_{k=1}^{n} k {n \choose k} x^{k-1}, \\ n(n-1)(1+x)^{n-2} &= \sum_{k=2}^{n} k(k-1) {n \choose k} x^{k-2}. \end{aligned}$

Setting $x = 1$ in the first identity,

$\displaystyle \sum_{k=1}^{n} k {n \choose k} = n \cdot 2^{n-1}.$

Setting $x = 1$ in the second identity,

$\begin{aligned} n(n-1)\cdot 2^{n-2} &= \sum_{k=2}^{n} k(k-1) {n \choose k} = \sum_{k=2}^{n} k^2 {n \choose k} - \sum_{k=2}^{n} k {n \choose k}. \end{aligned}$

By algebruh, since $\displaystyle 1^r {n \choose 1} = n$ for any $r$ ,

$\displaystyle \sum_{k=1}^{n} k^2 {n \choose k} = n(n-1)\cdot 2^{n-2} + n \cdot 2^{n-1} = n \cdot 2^{n-1} \cdot (2n-1).$

Problem 5. Evaluate $\displaystyle \sum_{k= 0}^r {m \choose k} {n \choose r-k}$ .

(Click for Solution)

Solution. Using the binomial theorem on the product

$\displaystyle (1+x)^{m+n} = (1+x)^m (1+x)^n,$

we have

$\begin{aligned} \sum_{r=0}^{m+n} {m+n \choose r} x^r &= \left(\sum_{k=0}^m {m \choose k} x^k\right) \cdot \left(\sum_{l=0}^n {n \choose l} x^l\right) \\ &= \sum_{k=0}^m \sum_{l=0}^n {m \choose k} {n \choose l} x^{k+l} \\ &= \sum_{r=0}^{m+n} \sum_{k+l=r} {m \choose k} {n \choose l} x^r \\ &= \sum_{r=0}^{m+n} \sum_{k=0}^r {m \choose k} {n \choose r-k} x^r. \end{aligned}$

Comparing the coefficients of $x^r$ ,

$\displaystyle \sum_{k= 0}^r {m \choose k} {n \choose r-k} = {m+n \choose r}.$

This result is known as Vandermonde’s identity.

Problem 6. Evaluate $\displaystyle \sum_{k= 0}^n {n \choose k}^2$ .

(Click for Solution)

Solution. Setting $m = n$ and $r = 2k$ in Problem 5,

$\displaystyle \sum_{k= 0}^n {n \choose k}^2 = \sum_{k= 0}^r {n \choose k} {n \choose 2k-k} = {n + n \choose k} = {2n \choose k}.$

—Joel Kindiak, 1 Aug 25, 1520H

December 23, 2025
Several Probability Puzzles

Problem 1. Let $X_1,\dots, X_n \sim \mathcal U(0, 1)$ be i.i.d.. Let $g : [0,1]^n \to [0,1]^n$ denote the permutation

$g(x_1,\dots, x_n) = (y_1,\dots, y_n)$

such that $y_1 \leq \dots \leq y_n$ . Denoting $(Y_1,\dots, Y_n) = g(X_1, \dots, X_n)$ , evaluate $\mathbb E[Y_i]$ for each $i$ .

(Click for Solution)

Solution. Since $\mathbb P (X_i = X_j) = 0$ whenever $i \neq j$ , we can assume $Y_1 < \dots < Y_n$ .

We will obtain the distribution of $Y_i$ . Fix $x \in [0, 1]$ . Let $V_x \sim \mathrm{Bin}(n-1, x)$ denote the number of sample points that are less than $x$ , which follows a binomial distribution. It follows that $\{ Y_i = x \} = \{ V_x = i-1 \}$ , so that

$\displaystyle f_{Y_i} (x) = \mathbb P(V_x = i-1) = {n-1 \choose i-1} x^{i-1} (1-x)^{n-i}.$

Hence, by recalling the properties of the Beta distribution,

$\begin{aligned} \mathbb E[Y_i] &= \int_0^1 x \cdot f_{Y_i} (x)\, \mathrm dx \\ &= \int_0^1 x \cdot {n-1 \choose i-1} x^{i-1} (1-x)^{n-i}\, \mathrm dx \\ &= \int_0^1 {n-1 \choose i-1} x^i (1-x)^{n-i}\, \mathrm dx \\ &= {n-1 \choose i-1} \int_0^1 x^i (1-x)^{n-i}\, \mathrm dx \\ &= \frac{\Gamma(n)}{\Gamma(i) \cdot \Gamma(n-i+1)} \cdot \frac{\Gamma(i+1) \cdot \Gamma(n-i+1)}{ \Gamma(n+1) } \\ &= \frac{\Gamma(n)}{\Gamma(i) \cdot \Gamma(n-i+1)} \cdot \frac{i \cdot \Gamma(i) \cdot \Gamma(n-i+1)}{ (n+1) \cdot \Gamma(n) } = \frac{i}{n+1}. \end{aligned}$

Problem 2. Calculate the average number of rolls of a fair six-sided die that you need to roll in order for the sum of all rolls to be a multiple of $6$ .

(Click for Solution)

Solution. Let $\xi_i$ denote the $i$ -th roll and $X_n := \sum_{i=1}^n \xi_i$ denote the sum of the first $n$ rolls. Define the stopping time $N$ by

$latex\displaystyle N := \inf_{n \in \mathbb N} \{6 \mid X_n\}.$

We claim that $N \sim \mathrm{Geom}(1/6)$ . For any $n \in \mathbb N$ ,

$\{N = n\} = \{6 \nmid X_1, \dots, 6 \nmid X_{n-1}, 6 \mid X_n\}.$

For each $i$ ,

$\{ 6 \nmid X_{i-1}, 6 \mid X_i\} = \{ \xi_i = X_i - X_{i-1} \},$

which is one of the six possible numbers with equal probability:

$\mathbb P(N = n) = \mathbb P(\{6 \nmid X_1, \dots, 6 \nmid X_{n-1}, 6 \mid X_n\}) = (5/6)^{n-1} \cdot (1/6).$

Therefore, $N \sim \mathrm{Geom}(1/6)$ so that $\mathbb E[N] = 1/(1/6) = 6$ as well.

Problem 3. What is the probability of getting an odd number of heads out of $n$ independent flips of a fair coin?

(Click for Solution)

Solution. Let $X \sim \mathrm{Bin}(n, 1/2)$ denote the number of heads out of $n$ independent flips of a fair coin. Then the required probability is

$\begin{aligned} p_{\text{odd}} &= \sum_{k\, \text{odd}}^{n} \mathbb P(X = k) = \sum_{ k\, \text{odd} }^{ n } {n \choose k} \frac 1{2^n}. \end{aligned}$

Using properties involving the binomial coefficient,

$\begin{aligned} 0 = (1+(-1))^n &= \sum_{k=0}^n {n \choose k} (-1)^k \\ &= \sum_{k\, \text{odd}}^n {n \choose k} (-1)^k + \sum_{k\, \text{even}}^n {n \choose k} (-1)^k \\ &= \sum_{k\, \text{odd}}^n {n \choose k} \cdot (-1) + \sum_{k\, \text{even}}^n {n \choose k} \cdot 1 \\ &= -\sum_{k\, \text{odd}}^n {n \choose k} + \sum_{k\, \text{even}}^n {n \choose k}. \end{aligned}$

Therefore,

$\displaystyle \sum_{k\, \text{odd}}^n {n \choose k} = \sum_{k\, \text{even}}^n {n \choose k}.$

In particular,

$\begin{aligned} p_{\text{even}} &= \sum_{ k\, \text{even} }^{ n } {n \choose k} \frac 1{2^n} = p_{\text{odd}}. \end{aligned}$

Since $p_{\text{odd}} + p_{\text{even}} = 1$ , we must have $p_{\mathrm{odd}} = 1/2$ , as required.

Problem 4. Given $X_1, X_2 \sim \mathcal U(0, 1)$ , calculate $\mathbb E[\min\{X_1,X_2\}]$ .

(Click for Solution)

Solution. Denoting $Y = \min\{X_1, X_2\}$ , we observe that

$\displaystyle \mathbb P(Y > y) = \mathbb P(X_1 > y, X_2 > y) = \mathbb P(X_1 > y) \cdot \mathbb P(X_2 > y) = (1-y)^2.$

Therefore, by the tail integral for expectation,

$\begin{aligned} \mathbb E[Y] &= \int_0^1 \mathbb P(Y > y) \cdot \mathbb I_{[0, 1]}(y) \, \mathrm dy = \int_0^1 (1-y)^2\, \mathrm dy = \int_0^1 y^2\, \mathrm dy = 1/3. \end{aligned}$

Problem 5. You’re the second-best player in a single-elimination tournament with $2^n$ players. Assume the brackets are randomly seeded, and the better player always wins each match. What is the probability you reach the finals?

(Click for Solution)

Solution. Each tournament will have $n$ stages, and at stage $i$ , there will be $2^{n+1-i}$ players. In order to reach the final stage, we need to be in a different “bracket” with the best player. At stage $1$ , there are two “brackets”, and each bracket has $2^{n-1}$ players. Therefore, the required probability is

$\displaystyle \frac{2^{n-1}}{2^n - 1}.$

Problem 6. Consider the sample space $\{0, 1\}$ and the sequence of random variables $X_0, X_1,\dots$ with the property that

$\mathbb P(X_{n+1} = 1 \mid X_n = 1) = p,\quad \mathbb P(X_{n+1} = 0 \mid X_n = 0) = q.$

Assuming that $X_n$ has identical distribution, evaluate $\mathbb P(X_n = 1)$ .

(Click for Solution)

Solution. Denote $\mathbb P(X_n = 1) = s$ . By the law of total probability,

$\begin{aligned} \mathbb P(X_{n+1} = 1) &= \mathbb P(X_{n+1} = 1 \mid X_n = 1) \cdot \mathbb P(X_n = 1) \\ &\phantom{==} + \mathbb P(X_{n+1} = 1 \mid X_n = 0) \cdot \mathbb P(X_n = 0) \\ s &= p \cdot s + (1 - q) \cdot (1-s) \\ (1- p) \cdot s &= (1-q) - (1-q) \cdot s \\ (2 - p - q) \cdot s &= 1 - q \\ s &= \frac{1-q}{2-p-q}. \end{aligned}$

—Joel Kindiak, 17 Oct 25, 1947H

December 19, 2025
Counting Arguments

Recall that the quantity $\displaystyle {n \choose r}$ was defined inductively using Pascal’s identity

$\displaystyle {n \choose r} = {n-1 \choose r} + {n-1 \choose r-1}.$

and denotes the number of $r$ -subsets of a set of size $n$ (i.e. $n$ distinct objects).

Problem 1. Prove that $\displaystyle {n \choose r} = {n \choose n-r}$ .

(Click for Solution)

Solution. Fix a set of $n$ distinct objects. There are $\displaystyle {n \choose r}$ possible $r$ -subsets of items that we can remove from that set. Therefore, every $(n-r)$ -subset of items left behind is obtained by exactly one corresponding $r$ -subset of items removed. Therefore, the number of $(n-r)$ -subsets (left behind) equals the number of $r$ -subsets (removed), yielding

$\displaystyle {n \choose r} = {n \choose n-r}.$

Problem 2. Prove that $\displaystyle {n \choose m} {m \choose r} = {n \choose r} {n-r \choose m-r}$ .

(Click for Solution)

Solution. We can interpret the identity as counting the number of $m$ -committeees out of $n$ persons, and among the $m$ persons, we choose $r$ persons in a “core team”. This is the quantity counted by the left-hand side:

$\displaystyle {n \choose m} {m \choose r}.$

On the right-hand side, we count the same quantity differently: first choose the $r$ “core team” members, then choose the remaining $(m-r)$ members out of the remaining $(n-r)$ persons:

$\displaystyle {n \choose r} {n-r \choose m-r}.$

Since both types of counting give the same total,

$\displaystyle {n \choose m} {m \choose r} = {n \choose r} {n-r \choose m-r}.$

Problem 3. Prove that $\displaystyle {n \choose r} = \frac nr {n-1 \choose r-1}$ .

(Click for Solution)

Solution. Replacing $(n,m,r)$ with $(n,r,1)$ in Problem 2,

$\displaystyle r \cdot {n \choose r} = {n \choose r} {r \choose 1} = {n \choose 1} {n-1 \choose r-1} = n \cdot {n-1 \choose r-1}.$

Problem 4. Prove that $\displaystyle {n \choose r} = \frac {n}{n-r} {n-1 \choose r}$ .

(Click for Solution)

Solution. Using Problems 1 and 3,

$\displaystyle {n \choose r} = {n \choose n-r} = \frac{n}{n-r} \cdot {n-1 \choose n-r-1} = \frac{n}{n-r} \cdot {n-1 \choose r}.$

Problem 5. Prove that $\displaystyle {n \choose r} = \frac {n-r+1}r {n \choose r-1}$ .

(Click for Solution)

Solution. Replacing $(n,m,r)$ with $(n,r,r-1)$ in Problem 2,

$\begin{aligned} r \cdot {n \choose r} ={n \choose r} {r \choose r-1} &= {n \choose r-1} {n-(r-1) \choose r-(r-1)} \\ &= (n-r+1) \cdot {n \choose r-1}. \end{aligned}$

Problem 6. For any $n \in \mathbb N^+, r \in \mathbb N_0$ , count the number of $n$ -tuples $(x_1,\dots,x_n) \in \mathbb N_0^n$ such that

$\displaystyle \sum_{k=1}^n x_k = r.$

(Click for Solution)

Solution. Consider a row of $r$ stars and $(n-1)$ bars. Let $x_1$ denote the number of stars before the 1st bar, $x_2$ denote the number of stars after the 1st bar and before the 2nd bar, and so on and so forth. Each arrangement of the $r$ stars and $(n-1)$ bars then corresponds to each desired $n$ -tuple. Thus, the required number is the number of places to place the bars:

$\displaystyle {r + (n-1) \choose (n-1)} = {n+r-1 \choose n-1} ={n+r-1 \choose r}.$

Problem 7. For any $n \in \mathbb N^+, p \in \mathbb N_0$ , suppose $p$ is prime. Count the number of $(2n)$ -tuples $(x_1,\dots,x_n,y_1,\dots, y_n) \in \mathbb N_0^n$ such that

$\displaystyle \left( \sum_{i=1}^n x_i \right) \cdot \left( \sum_{j=1}^n y_j \right) = p.$

(Click for Solution)

Solution. Since $p$ is prime, we require one of the sums to equal $1$ , and the other to equal $p$ . If the $x_i$ terms sum to $1$ , then there are a total of $n$ possible options of $(x_1,\dots, x_n)$ . By Problem 6, there are a total of

$\displaystyle {n+p-1 \choose p}$

options of $(y_1,\dots, y_n)$ . By symmetry, the required total is

$\displaystyle 2n \cdot {n+p-1 \choose p}.$

—Joel Kindiak, 31 Jul 25, 2114H

December 16, 2025
Multivariate Normal Bookkeeping
For random variables $X_1,\dots, X_n$ , denote $\mathbf X := (X_1,\dots, X_n)^{\mathrm T}$ and define

$\mathbb E[\mathbf X] = (\mathbb E[X_1], \dots, \mathbb E[X_n])^{\mathrm T}.$

Define the covariance matrix $\boldsymbol{ \Sigma }_{\mathbf X} \in \mathcal M_{n \times n}( \mathbb R )$ by

$\displaystyle (\boldsymbol{ \Sigma }_{\mathbf X})_{i,j} = \mathrm{Cov}(X_i, X_j).$

Note that $\boldsymbol{ \Sigma }_{\mathbf X} = \mathbb E[(\mathbf X - \mathbb E[\mathbf X]) (\mathbf X - \mathbb E[\mathbf X])^{\mathrm T}]$ .

Problem 1. For i.i.d. $Z_1,\dots, Z_n \sim \mathcal N(0, 1)$ , define $\mathbf Z := (Z_1,\dots, Z_n)^{\mathrm T}$ . Calculate the p.d.f. $f_{\mathbf Z}$ of $\mathbf Z$ and evaluate $\boldsymbol{ \Sigma }_{\mathbf Z}$ .

(Click for Solution)

Solution. For each $I$ , the p.d.f. $f_{Z_i}$ of $Z_i$ is

$\displaystyle f_{Z_i}(z_i) = \frac{1}{\sqrt{2\pi}} e^{-\frac{z_i^2}{2}}.$

Since $Z_1,\dots, Z_n$ are independent,

$\begin{aligned} f_{\mathbf Z}(\mathbf z) &= \prod_{i=1}^n f_{Z_i}(z_i) = \frac 1{ \sqrt{ (2\pi)^{n} } } e^{-\frac 12 \sum_{i=1}^n z_i^2} = \frac 1{ \sqrt{ (2\pi)^{n} } } e^{-\frac 12 \mathbf z^{\mathrm T} \mathbf z}.\end{aligned}$

Furthermore,

${ ( \boldsymbol{ \Sigma }_{\mathbf Z} )}_{i,j} = \mathrm{Cov}(Z_i, Z_j) = \mathbb I_{\{0\}} (i-j) = \mathbf I_{i,j},$

where $\mathbf I \equiv \mathbf I_n$ denotes the $n \times n$ identity matrix.

Problem 2. For any invertible matrix $\mathbf A \in \mathcal M_{n \times n}(\mathbb R)$ and $\boldsymbol{\mu} \in \mathbb R^n$ , define $\mathbf X = \boldsymbol{\mu} + \mathbf A \mathbf Z$ . Calculate the p.d.f. $f_{\mathbf X}$ of $\mathbf X$ and evaluate $\boldsymbol{ \Sigma }_{\mathbf X}$ .

(Click for Solution)

Solution. We first assume the case $\boldsymbol{ \mu } = \mathbf 0$ . Define the bijective and continuous map. We observe that for $K \in \frak{B}(\mathbb R^n)$ ,

$\mathbb P_{\mathbf X}(K) = \mathbb P(\mathbf X \in K) = \mathbb P(\mathbf Z \in \mathbf A^{-1}(K)) = (\mathbb P_{\mathbf Z} \circ \mathbf A^{-1})(K).$

That is, $\mathbb P_{\mathbf X} = \mathbb P_{\mathbf Z} \circ \mathbf A^{-1}$ is the pushforward measure of $\mathbb P_{\mathbf Z}$ under $g$ . Letting $\lambda$ denote the $n$ -dimensional Lebesgue measure, we leave it as an exercise in linear algebra to verify that

$\displaystyle \frac{ \mathrm d\lambda(\mathbf A^{-1}\mathbf x) }{ \mathrm d\lambda(\mathbf x) } = |\mathbf A^{-1}| = \frac{1}{|\mathbf A|}.$

By a change of variables, for any $K \in \frak{B}(\mathbb R^n)$ ,

$\begin{aligned} \mathbb P_{\mathbf X} (K) &= \int_{\mathbb R^n} \mathbb I_K\, \mathrm d\mathbb P_{\mathbf X} \\ &= \int_{\mathbb R^n} \mathbb I_K\, \mathrm d\mathbb P_{\mathbf X} \\ &= \int_{\mathbb R^n} \mathbb I_K \circ \mathbf A \, \mathrm d\mathbb P_{\mathbf Z} \\ &= \int_{\mathbb R^n} \mathbb I_K( \mathbf A\mathbf z ) \cdot f_{\mathbf Z}(\mathbf z) \, \mathrm d\lambda(\mathbf z) \\ &= \int_{\mathbb R^n} \mathbb I_K(\mathbf x) \cdot f_{\mathbf Z}(\mathbf A^{-1} \mathbf x ) \, \mathrm d\lambda(\mathbf A^{-1} \mathbf x ) \\ &= \int_{\mathbb R^n} \mathbb I_K(\mathbf x) \cdot f_{\mathbf Z}(\mathbf A^{-1} \mathbf x ) \cdot \frac{1}{|\mathbf A|}\, \mathrm d\lambda(\mathbf x) \\ &= \int_{K} f_{\mathbf Z}(\mathbf A^{-1} \mathbf x ) \cdot \frac{1}{|\mathbf A|}\, \mathrm d\lambda(\mathbf x).\end{aligned}$

By the Radon-Nikodým theorem,

$\displaystyle \int_K f_{\mathbf X}(\mathbf x)\, \mathrm d\lambda(\mathbf x) = \mathbb P_{\mathbf X} (K) = \int_{K} f_{\mathbf Z}(\mathbf A^{-1} \mathbf x ) \cdot \frac{1}{|\mathbf A|}\, \mathrm d\lambda(\mathbf x),$

so that

$\begin{aligned} f_{\mathbf X}(\mathbf x) &= \frac{1}{|\mathbf A|} \cdot f_{\mathbf Z}(\mathbf A^{-1}\mathbf x) \\ &= \frac 1{ |\mathbf A| \cdot \sqrt{ (2\pi)^{n} } } \cdot e^{-\frac 12 \mathbf x^{\mathrm T} (\mathbf A\mathbf A^{\mathrm T})^{-1} \mathbf x } \\ &= \frac 1{ \sqrt{ (2\pi)^{n} \cdot |\mathbf A \mathbf A^{\mathrm T}| } } \cdot e^{-\frac 12 \mathbf x^{\mathrm T} (\mathbf A\mathbf A^{\mathrm T})^{-1} \mathbf x }.\end{aligned}$

Furthermore,

$\begin{aligned} \boldsymbol{\Sigma}_{\mathbf X} &= \mathbb E[(\mathbf A \mathbf Z - \mathbb E[\mathbf A \mathbf Z]) (\mathbf A \mathbf Z - \mathbb E[\mathbf A \mathbf Z])^{\mathrm T}] \\ &= \mathbb E[\mathbf A (\mathbf Z - \mathbb E[\mathbf Z]) (\mathbf Z - \mathbb E[\mathbf Z])^{\mathrm T}\mathbf A^{\mathrm T} ] \\ &= \mathbf A \mathbb E[(\mathbf Z - \mathbb E[\mathbf Z]) (\mathbf Z - \mathbb E[\mathbf Z])^{\mathrm T} ]\mathbf A^{\mathrm T} \\ &= \mathbf A \boldsymbol{\Sigma}_{\mathbf Z} \mathbf A^{\mathrm T} = \mathbf A \mathbf A^{\mathrm T} . \end{aligned}$

In the marginally more general case, $\mathbf X - \boldsymbol{\mu} = \mathbf A \mathbf Z$ . We leave it as an exercise to verify that $\mathbb E[\mathbf X] = \boldsymbol{ \mu }$ and $\boldsymbol{ \Sigma }_{\mathbf X} = \boldsymbol{ \Sigma }_{\mathbf A \mathbf Z} = \mathbf A\mathbf A^{\mathrm T}$ , so that finally

$\begin{aligned} f_{\mathbf X}(\mathbf x) = f_{\mathbf A \mathbf Z}(\mathbf x - \boldsymbol{\mu}) &= \frac 1{ \sqrt{ (2\pi)^{n} \cdot |\boldsymbol{ \Sigma }_{\mathbf A \mathbf Z}| } } \cdot e^{-\frac 12 (\mathbf x - \boldsymbol{\mu})^{\mathrm T} \boldsymbol{ \Sigma }_{\mathbf A \mathbf Z}^{-1} (\mathbf x - \boldsymbol{\mu}) }. \end{aligned}$

Definition 1. A random variable $\mathbf X$ is multivariate normal, denoted $\mathbf X \sim \mathcal N(\boldsymbol{ \mu }, \boldsymbol{ \Sigma })$ , if it has a p.d.f. $f_{\mathbf X}$ given by

$\displaystyle f_{\mathbf X}(\mathbf x) = \frac{1}{ \sqrt{ (2\pi)^n \cdot |\boldsymbol{ \Sigma } | } } \cdot e^{-\frac 12 (\mathbf x - \boldsymbol{\mu})^{\mathrm T} \boldsymbol{\Sigma}^{-1} (\mathbf x - \boldsymbol{\mu})}.$

In this case, we say that the mean of $\mathbf X$ is $\mathbb E[ \mathbf X ] = \boldsymbol{\mu}$ , and the covariance of $\mathbf X$ is $\boldsymbol{\Sigma}_{\mathbf{X}} = \boldsymbol{\Sigma}$ . For example, $\mathbf Z \sim \mathcal N(\mathbf 0, \mathbf I_n)$ .

Problem 3. Fix $\mathbf X \sim \mathcal N(\boldsymbol{ \mu }, \boldsymbol{\Sigma} )$ .
- Prove that if $\boldsymbol{\Sigma}$ is diagonal whose diagonal entires are positive $\sigma_i^2 > 0$ , then $X_1,\dots, X_n$ are independent with $X_i \sim \mathcal N( \mu_i, \sigma_i^2 )$ .
- Prove that for any invertible $\mathbf A \in \mathcal M_{n \times n}(\mathbb R)$ and $\boldsymbol{ \nu } \in \mathbb R^n$ , $\mathbf Y := \boldsymbol{\nu} + \mathbf A \mathbf X \sim \mathcal N(\boldsymbol{ \mu }_{\mathbf Y}, \boldsymbol{ \Sigma }_{\mathbf Y})$ for constants $\boldsymbol{ \mu }_{\mathbf Y}, \boldsymbol{ \Sigma }_{\mathbf Y}$ to be calculated.
(Click for Solution)

Solution. For the first point, if $\boldsymbol{\Sigma}$ is diagonal with $\boldsymbol{ \Sigma }_{i,i} = \sigma_i^2 > 0$ , then $\boldsymbol{\Sigma}^{-1}$ is diagonal with $\boldsymbol{ \Sigma }_{i,i} = 1/\sigma_i^2 > 0$ , and

$\displaystyle (\mathbf x - \boldsymbol{\mu})^{\mathrm T} \boldsymbol{\Sigma}^{-1} (\mathbf x - \boldsymbol{\mu}) = \sum_{i=1}^n \frac{ (x_i - \mu_i)^2 }{ \sigma_i^2 }.$

Some algebra then yields

$\begin{aligned} f_{\mathbf X}(\mathbf x) &= \frac{1}{ \sqrt{ (2\pi)^n \cdot |\boldsymbol{ \Sigma } | } } \cdot e^{-\frac 12 (\mathbf x - \boldsymbol{\mu})^{\mathrm T} \boldsymbol{\Sigma}^{-1} (\mathbf x - \boldsymbol{\mu})} \\ &= \prod_{i=1}^n \frac{1}{\sigma_i \sqrt{ 2\pi } } \cdot e^{-\frac 12 \sum_{i=1}^n \frac{ (x_i - \mu_i)^2 }{ \sigma_i^2 } } \\ &= \prod_{i=1}^n \frac{1}{\sigma_i \sqrt{ 2\pi } } \cdot \prod_{i=1}^n e^{-\frac{ (x_i - \mu_i)^2 }{ 2\sigma_i^2 } } \\ &= \prod_{i=1}^n \frac{1}{\sigma_i \sqrt{ 2\pi } } e^{-\frac{ (x_i - \mu_i)^2 }{ 2\sigma_i^2 } } = \prod_{i=1}^n f_{X_i}(x_i). \end{aligned}$

Therefore, $X_1,\dots, X_n$ are independent with $X_i \sim \mathcal N(\mu_i , \sigma_i^2)$ .

For the second point, follow the proof in Problem 2 with needful bookkeeping to obtain

$\begin{aligned} f_{\mathbf Y}(\mathbf y) &= f_{\mathbf A\mathbf X}(\mathbf y - \boldsymbol{\nu}) = \frac 1{|\mathbf A|} \cdot f_{\mathbf X}(\mathbf A^{-1}(\mathbf y - \boldsymbol{\nu})), \end{aligned}$

which simplifies to Definition 1 with $\mathbb E[\mathbf Y] = \boldsymbol{\nu} + \mathbf A \boldsymbol{\mu}$ and $\boldsymbol{\Sigma}_{\mathbf Y} = \mathbf A \boldsymbol{\Sigma}_{\mathbf X} \mathbf A^{\mathrm T}$ .

Problem 4. Let $\mathbf X \sim \mathcal N(\boldsymbol{ \mu } , \boldsymbol{ \Sigma })$ . Prove that there exists an invertible matrix $\mathbf A \in \mathcal M_{n \times n}(\mathbb R)$ such that $\mathbf X = \boldsymbol{ \mu } + \mathbf A \mathbf Z$ . In particular, each $X_i$ is normally distributed.

(Click for Solution)

Solution. By definition,

$\displaystyle f_{\mathbf X}(\mathbf x) = \frac{1}{ \sqrt{ (2\pi)^n \cdot |\boldsymbol{ \Sigma } | } } \cdot e^{-\frac 12 (\mathbf x - \boldsymbol{\mu})^{\mathrm T} \boldsymbol{\Sigma}^{-1} (\mathbf x - \boldsymbol{\mu})}.$

We first assume $\boldsymbol{\mu} = \mathbf 0$ . If we can find some invertible matrix $\mathbf B$ such that $\mathbf B \mathbf X = \mathbf Z$ , then we can set $\mathbf A := \mathbf B^{-1}$ . To that end, we note that $\boldsymbol{\Sigma}$ is symmetric since

$\displaystyle \boldsymbol{\Sigma}_{i,j} = \mathrm{Cov}(X_i, X_j) = \mathrm{Cov}(X_j, X_i) = \boldsymbol{\Sigma}_{j,i} > 0.$

By the spectral theorems, $\boldsymbol{\Sigma}$ is orthogonally diagonalisable. That is, there exists an orthogonal matrix $\mathbf P$ (i.e. $\mathbf P^{-1} = \mathbf P^{\mathrm T}$ ) and a diagonal matrix $\mathbf D$ such that

$\mathbf{P}^{\mathrm T} \boldsymbol{\Sigma} \mathbf{P} = \mathbf{D}\quad \Rightarrow \quad \boldsymbol{\Sigma} = \mathbf{P} \mathbf{D} \mathbf{P}^{\mathrm T}.$

Furthermore, the entries in the diagonal of $\mathbf D$ are all positive, denoted $\sigma_i^2 > 0$ . Define $\boldsymbol{\Delta}$ by

$\displaystyle \boldsymbol{\Delta}_{i,j} := \begin{cases} \sigma_i, & i = j, \\ 0, & i \neq j. \end{cases}$

Then $\boldsymbol{\Delta}\boldsymbol{\Delta}^{\mathrm T} = \mathbf{D}$ , so that

$\boldsymbol{\Sigma} = \mathbf{P} \mathbf{D} \mathbf{P}^{\mathrm T} = \mathbf{P} \boldsymbol{\Delta}\boldsymbol{\Delta}^{\mathrm T} \mathbf{P}^{\mathrm T} = (\mathbf{P} \boldsymbol{\Delta}) (\mathbf{P} \boldsymbol{\Delta})^{\mathrm T}.$

Now define $\mathbf B := ( \mathbf{P} \boldsymbol{\Delta})^{-1}$ . By Problem 3, $\mathbf B \mathbf X$ is multivariate normal. We calculate its expectation via

$\mathbb E[\mathbf B \mathbf X] = \mathbf B \mathbf E[\mathbf X] = \mathbf B \mathbf 0 = \mathbf 0$

and its covariance matrix via

$\begin{aligned}\boldsymbol{\Sigma_{\mathbf B \mathbf X}} &= \mathbb E[(\mathbf B \mathbf X)^{\mathrm T} (\mathbf B \mathbf X)] \\ &= \mathbf B \mathbb E[\mathbf X^{\mathrm T} \mathbf X] \mathbf B^{\mathrm T} \\ &= ( \mathbf{P} \boldsymbol{\Delta})^{-1} (\mathbf{P} \boldsymbol{\Delta}) (\mathbf{P} \boldsymbol{\Delta})^{\mathrm T} (( \mathbf{P} \boldsymbol{\Delta})^{-1})^{\mathrm T} = \mathbf I. \end{aligned}$

Therefore, $\mathbf B \mathbf X \sim \mathcal N(\mathbf 0, \mathbf I)$ . Since $\mathbf Z \sim \mathcal N(\mathbf 0, \mathbf I)$ , we conclude $\mathbf B \mathbf X = \mathbf Z$ , so that $\mathbf X = \mathbf B^{-1}\mathbf Z = \mathbf A \mathbf Z$ , as required.

For the general case, $\mathbf X - \boldsymbol{\mu}$ has expectation $\mathbf 0$ , and so by the first result, there exists some invertible matrix $\mathbf A$ such that

$\mathbf X - \boldsymbol{\mu} = \mathbf A \mathbf Z \quad \Rightarrow \quad \mathbf X = \boldsymbol{\mu} + \mathbf A \mathbf Z,$

as required.

Problem 5. Define $W_1 := \bar X_n$ and for each $i > 1$ , define $W_i := X_i - \bar X_n$ . If $X_1,\dots,X_n \sim \mathcal N(\mu, \sigma^2)$ are i.i.d., prove that $\bar X_n$ and $(W_2,\dots,W_n)$ are independent. Deduce that $\bar X_n$ and $S_n^2$ defined by

$\displaystyle S_n^2 := \frac 1{n-1} \sum_{i=1}^n (X_i - \bar X_n)^2$

are independent.

(Click for Solution)

Solution. For each $i > 1$ ,

$\displaystyle W_i = \sum_{j=1}^n \underbrace{ \left( \mathbb I_{\{ i \}}(j) -\frac{ 1 }{n} \right) }_{\mathbf A_{i,j}} X_j =: \sum_{j=1}^n \mathbf A_{i, j}X_j.$

Furthermore, define $\mathbf A_{1, j} = 1/n$ . Defining $\mathbf W := (W_1,\dots,W_n)^{\mathrm T}$ , $\mathbf W = \mathbf A \mathbf X$ . If $X_1,\dots,X_n$ are i.i.d. normally distributed with variance $\sigma^2$ , then by the converse of the first point in Problem 3 (which holds), $\mathbf X$ is multivariate normal. By the second point in Problem 3, $\mathbf W$ is multivariate normal.

We turn to evaluate its covariance matrix. Firstly, to shorten computations,

$\begin{aligned} \mathrm{Cov}(X_i, \bar X_n) &= \frac 1n \cdot \sum_{k=1}^n \mathrm{Cov}(X_i, X_k) \\ &= \frac 1n \cdot \mathrm{Cov}(X_i, X_i) = \frac{\sigma^2}{n}. \end{aligned}$

Furthermore,

$\begin{aligned} \mathrm{Cov}(\bar X_n, \bar X_n) &= \frac 1n \cdot \sum_{k=1}^n \mathrm{Cov}(X_i, \bar X_n) \\ &= \frac 1n \cdot \sum_{k=1}^n \frac{\sigma^2}{n} = \frac 1n \cdot n \cdot \frac{\sigma^2}n = \frac{\sigma^2}n. \end{aligned}$

Thus, for $j > 1$ ,

$\begin{aligned} (\boldsymbol{\Sigma}_{\mathbf W})_{1,j} &= \mathrm{Cov}(W_1, W_j) \\ &= \mathrm{Cov}(\bar X_n, X_j - \bar X_n) \\ &= \mathrm{Cov}(X_j , \bar X_n) - \mathrm{Cov}(\bar X_n,\bar X_n) \\ &= \frac{\sigma^2}{n} - \frac{\sigma^2}{n} = 0.\end{aligned}$

Therefore, using properties involving the exponential function,

$f_{\mathbf W}(\mathbf w) = f_{W_1}(w_1) \cdot f_{W_2,\dots,W_n}(w_2,\dots,w_n).$

Hence, $W_1$ and $(W_2,\dots,W_n)$ are independent. For the second claim, we first note that

$\displaystyle 0 = \sum_{i=1}^n X_i - n\bar X_n = \sum_{i=1}^n (X_i - \bar X_n) = X_1 - \bar X_n + \sum_{i=2}^n (X_i - \bar X_n),$

so that $X_1 - \bar X_n = -\sum_{i=2}^n (X_i - \bar X_n) = -\sum_{i=2}^n W_i$ . It follows that

$\begin{aligned} S_n^2 &= \frac 1{n-1} \left( (X_1 - \bar X_n)^2 + \sum_{i=2}^n (X_i - \bar X_n)^2 \right) \\ &= \frac 1{n-1} \left( (X_1 - \bar X_n)^2 + \sum_{i=2}^n W_i^2 \right) \\ &= \frac 1{n-1} \left( \left( \sum_{i=2}^n W_i \right)^2 + \sum_{i=2}^n W_i^2 \right). \end{aligned}$

Since $S_n^2$ can be written purely in terms of $(W_2,\dots, W_n)$ , it is independent of $W_1 = \bar X_n$ .

Problem 6. Using the definitions and notation in Problem 5, prove that for any $n \in \mathbb N_{>1}$ ,

$\displaystyle (n-1) \cdot S_n^2 = (n-2) \cdot S_{n-1}^2 + \frac{n-1}{n} \cdot (X_n - \bar X_{n-1})^2.$

Deduce that there exist i.i.d. $Z_1,\dots, Z_{n-1} \sim \mathcal N(0, 1)$ such that

$\displaystyle \frac{(n-1) \cdot S_n^2}{\sigma^2} = \sum_{i=1}^{n-1} Z_i^2.$

(Click for Solution)

Solution. We first observe that

$\begin{aligned} \bar X_n - \bar X_{n-1} &= \frac{1}{n} \cdot \sum_{i=1}^{n-1} X_i + \frac{1}{n} \cdot X_n - \frac 1{n-1} \cdot \sum_{i=1}^{n-1} X_i \\ &= -\frac{1}{n(n-1)} \cdot \sum_{i=1}^{n-1} X_i + \frac{1}{n} \cdot X_n = \frac 1n \cdot ( X_n - \bar X_{n-1} ). \end{aligned}$

Performing algebruh,

$\begin{aligned} &(n-1) \cdot S_n^2 \\ &= \sum_{i=1}^n \left( X_i - \bar X_{n-1} - \frac 1n \cdot ( X_n - \bar X_{n-1})\right)^2 \\ &= \sum_{i=1}^n \left[ (X_i - \bar X_{n-1})^2 - 2 \cdot (X_i - \bar X_{n-1}) \cdot \frac 1n \cdot ( X_n - \bar X_{n-1}) + \frac 1{n^2} \cdot ( X_n - \bar X_{n-1})^2 \right] \\ &= (n-2) \cdot S_{n-1}^2 + (X_n - \bar X_{n-1})^2 - \frac{2}{n} \cdot ( X_n - \bar X_{n-1})^2 + \frac 1{n} \cdot ( X_n - \bar X_{n-1})^2 \\ &= (n-2) \cdot S_{n-1}^2 + \frac{n-1}{n} \cdot (X_n - \bar X_{n-1})^2. \end{aligned}$

We now prove the final equation by induction. In the case $n = 2$ , we have

$\begin{aligned} \frac{S_2^2}{\sigma^2} &= \frac 1{\sigma^2} \cdot \frac 12 \cdot ((X_1 -\bar X_1)^2 + (X_2 -\bar X_1)^2) = \frac{(X_2 - X_1)^2}{2\sigma^2}. \end{aligned}$

Then $Z_1 := (X_2 - X_1)/(\sigma \sqrt 2) \sim \mathcal N(0, 1)$ , so that $S_2^2/\sigma^2 = Z_1^2$ , as required. For the induction step, suppose that the statement holds for $n = k$ . Then

$\displaystyle k \cdot S_{k+1}^2 = (k-1) \cdot S_k^2 + \frac{k}{k+1} \cdot (X_{k+1} - \bar X_k)^2.$

By relabelling the result in Problem 5, we have that $S_k^2$ is independent of $\bar X_k$ and $X_{k+1}$ . By the induction hypothesis, there exist i.i.d. $Z_1,\dots, Z_{k-1} \sim \mathcal N(0, 1)$ such that

$\displaystyle \frac{ (k-1) \cdot S_k^2 }{\sigma^2} := \sum_{i=1}^{k-1} Z_i^2.$

Furthermore, $X_{k+1} - \bar X_k \sim \mathcal N(0, \sigma^2 \cdot (k+1)/k)$ so that

$\displaystyle Z_{k+1} := \frac{\sqrt{k}}{\sqrt{k+1}} \cdot \frac{ X_{k+1} - \bar X_k }{\sigma} = \frac{X_{k+1} - \bar X_k}{ \sigma \cdot \sqrt{ \frac{k+1}{k} } } \sim \mathcal N(0, 1).$

Therefore,

$\displaystyle \displaystyle \frac{ k \cdot S_{k+1}^2 }{ \sigma^2 } = \frac{ (k-1) \cdot S_{k}^2 }{ \sigma^2 } + \frac{k}{k+1} \cdot \frac{ (X_{k+1} - \bar X_{k})^2 }{ \sigma^2 } = \sum_{i=1}^{k-1} Z_i^2 + Z_k^2 = \sum_{i=1}^{k} Z_i^2,$

as required.

—Joel Kindiak, 25 Jul 25, TBCH
December 12, 2025
The Central Limit Theorem

We are now ready to ascend the mountain of proving the central limit theorem, which we will take inspiration from this paper by Calvin Wooyoung Chin.

Lemma 1. Let $X$ be a real-valued random variable with finite expectation. For any $\delta > 0$ ,

$\displaystyle \mathbb P(|X| \geq \delta) \leq \frac{\mathbb E[|X|]}{\delta}.$

This is called Markov’s inequality. Furthermore, if $X$ has finite variance, then

$\displaystyle \mathbb P(|X - \mathbb E[X]| \geq \delta ) \leq \frac{\mathrm{Var}(X)}{\delta^2}.$

This is called Chebychev’s inequality.

Proof. For the first identity,

$\begin{aligned} \mathbb E[|X|] &= \mathbb E[|X| \cdot \mathbb I_{[\delta, \infty)}] + \mathbb E[|X| \cdot \mathbb I_{(-\infty, \delta)}] \\ &\geq \mathbb E[\delta \cdot \mathbb I_{[\delta, \infty)}] + 0\\ &= \delta \cdot \mathbb E[\mathbb I_{[\delta, \infty)} ] \\ &= \delta \cdot \mathbb P(|X| \geq \delta). \end{aligned}$

Dividing by $\delta$ yields the desired result. For Chebychev’s inequality, apply Markov’s inequality to the random variable $(X - \mathbb E[X])^2$ to get

$\begin{aligned} \mathbb P(|X - \mathbb E[X]| \geq \delta) &= \mathbb P( (X - \mathbb E[X])^2 \geq \delta^2 ) \\ &= \frac{\mathbb E[(X - \mathbb E[X])^2]}{\delta^2} \\ &\leq \frac{\mathrm{Var}(X)}{\delta^2}. \end{aligned}$

Chebychev’s inequality is also responsible for our intuition about probability arising from repeated experiments.

Corollary 1. Fix $K \subseteq \Omega$ and denote $p:= \mathbb P(K)$ . Define the i.i.d. random variables $\xi_i$ by $\xi_i = \mathbb I_K$ , so that $\xi_i \sim \mathrm{Ber}(p)$ . Define

$\displaystyle \bar \xi_n := \frac 1n \cdot \sum_{i=1}^n \xi_i.$

Then for any $\delta > 0$ , $\mathbb P(|\bar \xi_n - p| > \delta) \to 0$ . In this case, we say that $\bar \xi_n \to p$ in probability.

Proof. Fix $\delta > 0$ . Since $\mathbb E[\xi_n] = p$ and $\mathrm{Var}(\xi_n) = p(1-p)$ ,

$\displaystyle \mathbb E[\bar \xi_n] = p,\quad \mathrm{Var}(\bar \xi_n) = \frac{p(1-p)}{n}.$

By Chebychev’s inequality,

$\begin{aligned} \mathbb P(|\bar \xi_n - p| > \delta) &\leq \frac{\mathrm{Var}(\bar \xi_n)}{\delta^2} = \frac{p(1-p)}{n \cdot\delta^2}. \end{aligned}$

Taking $n \to \infty$ , $\mathbb P(|\bar \xi_n - p| > \delta) \to 0$ .

Denote $Z \sim \mathcal N(0, 1)$ and its c.d.f. by $\Phi$ .

Lemma 2. Let $X_1, X_2,\dots$ be real-valued random variables. Suppose for any thrice-differentiable $f : \mathbb R \to \mathbb R$ such that $f, f', f'', f'''$ are bounded,

$\displaystyle \lim_{n \to \infty} \mathbb E[f(X_n)] = \mathbb E[f(Z)].$

Then for any $z \in \mathbb R$ ,

$\displaystyle \lim_{n \to \infty} \mathbb P(X_n \leq z) = \Phi(z),\quad z \in \mathbb R.$

Proof. Fix $z \in \mathbb R$ and $\epsilon > 0$ . Since the p.d.f. of $Z$ is continuous, there exists $\delta > 0$ such that

$|\Phi(z \pm \delta) - \Phi(z)| < \epsilon.$

In particular, $\Phi(z+\delta) < \Phi(z) < \Phi(z-\delta) + \epsilon$ . Define thrice-differentiable bounded functions $f, F : \mathbb R \to [0, 1]$ with bounded first, second, and third derivatives such that

$f|_{(-\infty, z-\delta]} = 1,\quad f|_{[z, \infty)} = 0,\quad F|_{(-\infty, z]} = 1,\quad F|_{[z+\delta, \infty)} = 0.$

By hypothesis,

$\displaystyle \lim_{n \to \infty} \mathbb E[f(X_n)] = \mathbb P(Z \leq z -\delta) = \Phi(z-\delta) \geq \Phi(z) - \epsilon.$

Similarly, $\mathbb E[F(X_n)] < \Phi(z) + \epsilon$ . Since $f \leq \mathbb I_{(-\infty, z]} \leq F$ point-wise, for sufficiently large $n$ ,

$\Phi(z) - \epsilon < \mathbb E[f(X_n)] \leq \underbrace{ \mathbb E[ \mathbb I_{(-\infty, z]} ] }_{ \mathbb P(X_n \leq z)} \leq \mathbb E[F(X_n)] < \Phi(z) + \epsilon,$

so that $\displaystyle \lim_{n \to \infty} \mathbb P(X_n \leq z) = \Phi(z)$ , as required.

Theorem 1 (Central Limit Theorem). Let $X_1,X_2,\dots$ be i.i.d. with mean $0$ and variance $1$ . Then for any $z \in \mathbb R$ ,

$\displaystyle \lim_{n \to \infty} \mathbb P(\sqrt n \cdot \bar X_n \leq x) = \Phi(z).$

In this case, we say that $\sqrt n \cdot \bar X_n \to Z$ in distribution.

Proof. Fix i.i.d. $Y_1,Y_2,\dots \sim \mathcal N(0, 1)$ that are independent of $X_1,X_2,\dots$ . For each $n \in \mathbb N$ and $0 \leq i \leq n$ , define

$\displaystyle Z_{n,i} := \frac 1{\sqrt{n}} \cdot \left( \sum_{j=1}^i X_j + \sum_{j=i+1}^n Y_j \right),\quad S_{n,i} := Z_{n,i} - \frac 1{\sqrt{n}} \cdot X_i.$

By direct computations, $Z_{n,n} = \sqrt{n} \cdot \bar X_n$ and $Z_{n,0} = Z \sim \mathcal N(0, 1)$ . The key idea is using Lemma 2 to conclude our proof. Fix any thrice-differentiable $f : \mathbb R \to \mathbb R$ such that $f, f', f'', f'''$ are bounded. We claim that

$\displaystyle \lim_{n \to \infty} \mathbb E[f(Z_{n,n})] = \mathbb E[f(Z_{n,0})].$

Fix $\epsilon > 0$ . We aim to find $N \in \mathbb N$ such that for $n > N$ ,

$\displaystyle |\mathbb E[f(Z_{n,n})] - \mathbb E[f(Z_{n,0})]| < \epsilon.$

Applying Taylor’s theorem for the interval containing $S_{n,i}$ and $Z_{n,i}$ , there exists $C_{n,i}$ between them such that

$\displaystyle f(Z_{n_i}) = f(S_{n,i}) + f'(S_{n,i}) \cdot (Z_{n,i} - S_{n,i}) + \frac{f''(C_{n,i})}{2!} \cdot (Z_{n,i} - S_{n,i})^2.$

Performing algebra yields

$\displaystyle f(Z_{n,i}) - f(S_{n,i}) - f'(S_{n,i}) \cdot \frac{X_i}{\sqrt n} - f''(S_{n,i}) \cdot \frac{X_i^2}{2 n} = \underbrace{ \frac{ ( f''(C_{n,i}) - f''(S_{n,i}) ) X_i^2}{2n} }_{R_{n,i}}.$

Since $f'''$ is bounded, there exists $\delta > 0$ such that

$\delta \cdot \| f''' \|_\infty \leq \epsilon.$

By the mean value theorem, whenever $|x - y| \leq \delta$ ,

$|f''(x) - f''(y)| \leq \| f''' \|_\infty \cdot |x - y| \leq \delta \cdot \| f''' \|_\infty \leq \epsilon.$

Consider the “good event” $G := \{ |X_i| \leq \delta \sqrt{n} \}$ and its complement

$G^c := \Omega \backslash G = \{ |X_i| > \delta \sqrt{n} \}.$

By our bound on the second derivative via the mean value theorem, we can bound $R_{n,i}$ in the “good event” $G$ by

$\displaystyle |f''(C_{n,i}) - f''(S_{n,i})| \leq \epsilon \quad \Rightarrow \quad \mathbb E[|R_{n,i}| \cdot \mathbb I_G] \leq \frac{\epsilon}{2n} \cdot \mathbb E[X_i^2] = \frac{\epsilon}{2n}.$

On the other hand, in the “bad event” $G^c$ , since $\| f'' \|_\infty \leq M$ for some $M > 0$ ,

$\displaystyle \mathbb E[|R_{n,i}| \cdot \mathbb I_{G^c}] \leq \frac{M}{n} \cdot \mathbb E[X_i^2 \cdot \mathbb I_{G^c}] \leq \frac Mn \cdot \mathbb E[\mathbb I_{G^c}].$

By Chebychev’s inequality,

$\begin{aligned}\mathbb E[\mathbb I_{G^c}] &= \mathbb P(|X_i| \geq \delta \sqrt n) \leq \frac{1}{\delta \cdot \sqrt{n}^2} = \frac{1}{n \delta}.\end{aligned}$

Therefore,

$\displaystyle \mathbb E[|R_{n,i}|] = \mathbb E[|R_{n,i}| \cdot \mathbb I_G] + \mathbb E[|R_{n,i}| \cdot \mathbb I_{G^c}] \leq \left( \frac{\epsilon}{2} + \frac{M}{n\delta} \right) \cdot \frac 1n.$

Thus, selecting $N > 2M/(\epsilon \delta)$ yields $\mathbb E[|R_{n,i}|] \leq \epsilon/n$ . Plugging in the left-hand side of $R_{n,i}$ ,

$\displaystyle |\mathbb E[f(Z_{n,i})] - \mathbb E[f(S_{n,i})] - \mathbb E[f''(S_{n,i}) ] / 2n |\leq \frac{\epsilon}{n}.$

The argument remains almost unchanged with $Z_{n,i}$ replaced with $Z_{n,i-1}$ , so that we have the bound

$\displaystyle |\mathbb E[f(Z_{n,i-1})] - \mathbb E[f(S_{n,i})] - \mathbb E[f''(S_{n,i}) ] / 2n |\leq \frac{\epsilon}{n}.$

Applying the triangle inequality,

$\displaystyle |\mathbb E[f(Z_{n,i})] - \mathbb E [f(Z_{n,i-1})]| \leq \frac{\epsilon}{n} + \frac{\epsilon}{n} = \frac{2\epsilon}{n}.$

Finally, applying a telescoping series,

$\begin{aligned} |\mathbb E[f(Z_{n,n})] - \mathbb E [f(Z_{n,0})]| &\leq \sum_{i=1}^n |\mathbb E[f(Z_{n,i})] - \mathbb E [f(Z_{n,i-1})]| \leq \sum_{i=1}^n \frac{2\epsilon}n = 2\epsilon. \end{aligned}$

Replace $\epsilon$ with $\epsilon/3$ to complete the argument.

With that, we are done with probability…for now. There are many directions we can take from this point on. We could have proven this result using techniques involving characteristic functions or even Brownian motion, but those topics will require their entire blog sections in order to properly discuss. We conclude with the famous normal approximation to the binomial and Poisson distributions.

Corollary 1. For $X_n \sim \mathrm{Bin}(n, p)$ ,

$\displaystyle \lim_{n \to \infty} \mathbb P \left( \frac{X_n - np}{\sqrt{np(1-p)}} \leq z \right) = \Phi(z).$

This is the normal approximation to the binomial distribution.

Proof. For each $n$ , write $X_n := \sum_{i=1}^n \xi_i$ for i.i.d. $\xi_i \sim \mathrm{Ber}(p)$ and

$\displaystyle Y_n := \frac{ X_n - np }{ \sqrt{p(1-p)} } .$

Then

$\displaystyle \bar Y_n := \frac 1n \cdot Y_n = \frac 1n \cdot \sum_{i=1}^n \frac{ \xi_i - p }{\sqrt{p(1-p)}} = \frac{X_n - np}{n \sqrt{p(1-p)}}.$

Since the $(\xi_i - p)/\sqrt{p(1-p)}$ are i.i.d. with mean $0$ and variance $1$ , by the central limit theorem,

$\displaystyle \lim_{n \to \infty} \mathbb P \left( \frac{X_n - np}{\sqrt{np(1-p)}} \leq z \right) = \lim_{n \to \infty} \mathbb P ( \sqrt n \cdot \bar Y_n \leq z) = \Phi(z).$

Corollary 2. We have

$\displaystyle \lim_{\lambda \to \infty} \mathbb P \left( \frac{X - \lambda}{\sqrt{\lambda}} \leq z \right) = \Phi(z), \quad X \sim \mathrm{Pois}(\lambda).$

This is the normal approximation to the Poisson distribution.

Proof. Assume $\lambda \in \mathbb N^+$ . Write $X_{\lambda} := \sum_{i=1}^{\lambda} \xi_i$ for i.i.d. $\xi_i \sim \mathrm{Pois}(1)$ and

$\displaystyle Y_\lambda := X_{\lambda} - \lambda .$

Then

$\displaystyle \bar Y_{\lambda} = \frac{X_{\lambda} - \lambda}{\lambda } = \frac 1{\lambda} \cdot \sum_{i=1}^{\lambda} (\xi_i - 1).$

Since the $\xi_i - 1$ are i.i.d. with mean $0$ and variance $1$ , by the central limit theorem,

$\displaystyle \lim_{\lambda \to \infty} \mathbb P \left( \frac{X_{\lambda} - \lambda}{ \sqrt{\lambda} } \leq z \right) = \lim_{n \to \infty} \mathbb P ( \sqrt{ \lambda } \cdot \bar Y_\lambda \leq z) = \Phi(z).$

—Joel Kindiak, 31 Jul 25, 1359H

December 9, 2025
Adding Random Variables…Revisited

We are now justified in adding random variables. For instance, if $X, Y \sim \mathcal N(0, 1)$ , what is the distribution of $X + Y$ ? Unfortunately, in the most extreme cases, this answer is trivial.

Lemma 1. Let $X \sim \mathcal N(0, 1)$ and $Y = -X$ . Then $Y \sim \mathcal N(0, 1)$ and $X + Y = 0$ .

Proof. We observe that for any $K \in \frak{B}(\mathbb R)$ , since $f_X(\cdot) = f_X(-\cdot)$ , by a change of variables,

$\begin{aligned} \mathbb P_Y(K) = \mathbb P_X(-K) &= \int_{-K} f_X(x)\, \mathrm dx \\ &= \int_K f_X(-x)\, \mathrm dx \\ &= \int_K f_X(x)\, \mathrm dx = \mathbb P_X(K).\end{aligned}$

Therefore, for any $K \in \frak{B}(\mathbb R)$ ,

$\displaystyle \int_K f_Y\, \mathrm d\lambda = \int_K f_X\, \mathrm d\lambda.$

Therefore, $f_Y = f_X$ so that $Y \sim \mathcal N(0, 1)$ .

Here, it is clear that $Y$ is entirely dependent— $100\%$ correlated even—to the random variable $X$ . We got a rather useless answer to our question in this case. If instead we swung to the other extreme, namely $X,Y$ being independent, we don’t have a general formula for any $X, Y$ . Nevertheless, the independent case proves to be far more useful in reality—for instance, the exam scores of two students are effectively independent barring (sufficiently drastic) externalities.

Therefore, let’s discuss what it means for two random variables $X,Y : \Omega \to \mathbb R$ to be independent. We have previously seen that two events $K,L$ are called independent if $\mathbb P(K \cap L) = \mathbb P(K) \cdot \mathbb P(L)$ .

Lemma 2. The set $\sigma(X) := \{X^{-1}(K) : K \in \frak{B}(\mathbb R)\} \subseteq \mathcal F$ forms a $\sigma$ -algebra, called the $\sigma$ -algebra generated by $X$ .

Given two $\sigma$ -algebras $\mathcal F_1, \mathcal F_2 \subseteq \mathcal F$ , also known as sub- $\sigma$ -algebras of $\mathcal F$ , we say that $\mathcal F_1, \mathcal F_2$ are independent if for any $K \in \mathcal F_1, L \in \mathcal F_2$ , $K, L$ are independent.

Definition 1. Two random variables $X, Y$ are said to be independent if $\sigma(X), \sigma(Y)$ are independent.

Suppose $\mathbb P_X \ll \mu$ and $\mathbb P_Y \ll \mu$ , where $\mu$ denotes either the counting measure $|\cdot|$ or the Lebesgue measure $\lambda$ . Let $\mu^2$ be the product of two copies of $\mu$ . Suppose $(X,Y) \ll \mu^2$ .

Theorem 1. $X, Y$ are independent and $(X,Y) \ll \mu^2$ if and only if

$\displaystyle \mathbb P((X, Y) \in K) = \int_K f_X \cdot f_Y\, \mathrm d\mu^2,$

where the integrand in the right-hand side is interpreted to mean

$f_X(\cdot , y) = f_X,\quad f_Y(x, \cdot) = f_Y, \quad x, y \in \mathbb R.$

Proof. Consider the cumulative distribution functions

$F_X(x) := \mathbb P(X \in (-\infty, x]) \equiv \mathbb P(X \leq x),\quad F_Y(y) := \mathbb P( Y \leq y).$

By the Radon-Nikodým theorem,

$\displaystyle F_X(x) = \int_{(-\infty, x]} f_X\, \mathrm d\mu,\quad F_Y(y) = \int_{(-\infty, y]} f_Y\, \mathrm d\mu.$

In the direction $(\Leftarrow)$ , by the Fubini-Tonelli theorem,

$\begin{aligned} \mathbb P(X^{-1}((-\infty, x]) \cap Y^{-1}((-\infty, y])) &= \mathbb P((X,Y) \in (-\infty, x] \times (-\infty, y]) \\ &= \int_{(-\infty, x] \times (-\infty, y]} f_{X,Y}\, \mathrm d\mu^2 \\ &= \int_{(-\infty, x]} \int_{(-\infty, y]} f_{X,Y}\, \mathrm d \mu\, \mathrm d \mu \\ &= \int_{(-\infty, x]} \int_{(-\infty, y]} f_X \cdot f_Y\, \mathrm d \mu\, \mathrm d \mu \\ &= \int_{(-\infty, x]} f_X \, \mathrm d \mu \cdot \int_{(-\infty, y]} f_Y\, \mathrm d \mu \\ &= \mathbb P(X^{-1}(-\infty, x]) \cdot \mathbb P(Y^{-1}(-\infty, y]), \end{aligned}$

establishing independence.

In the direction $(\Rightarrow)$ , we similarly use the Fubini-Tonelli theorem to obtain

$\begin{aligned} \iint_{(-\infty, x] \times (-\infty, y]} f_{X,Y}\, \mathrm d\mu^2 &= \int_{(-\infty, x]} f_X\, \mathrm d\mu \cdot \int_{(-\infty, y]} f_Y\, \mathrm d\mu \\ &= \int_{(-\infty, x]} \int_{(-\infty, y]} f_X \cdot f_Y\, \mathrm d \mu\, \mathrm d \mu \\ &= \iint_{(-\infty, x] \times (-\infty, y]} f_X \cdot f_Y\, \mathrm d\mu^2, \end{aligned}$

so that $f_{X,Y} = f_X \cdot f_Y$ .

Henceforth, when $X,Y$ are independent and $X \ll \mu$ and $Y \ll \mu$ , we assume $(X,Y) \ll \mu$ so that $f_{X,Y}$ is a meaningful quantity.

Corollary 1. Define

$\displaystyle F_{X,Y}(x,y) = \mathbb P((X,Y) \in (-\infty, x] \times (-\infty, y]) \equiv \mathbb P(X \leq x, Y \leq y).$

Then $X, Y$ are independent and $(X,Y) \ll \mu^2$ if and only if for any $x, y \in \mathbb R$ ,

$\displaystyle F_{X,Y}(x,y) = F_X(x)\cdot F_Y(y).$

Finally, let’s add these random variables together.

Theorem 1. If $X, Y$ are independent $\mathbb R$ -valued random variables with density functions $f_X, f_Y$ , then $X+Y$ is a random variable with density function

$f_{X+Y} = f_X * f_Y,$

where the right-hand side denotes convolution with respect to $\mu$ :

$\displaystyle (f * g)(u) := \int_{\mathbb R} f \cdot g(u- \cdot )\, \mathrm d\mu \equiv \int_{\mathbb R} f(x) \cdot g(u- x )\, \mathrm d\mu(x).$

If $\mu$ is the counting measure, we get

$\displaystyle (f * g)(u) = \sum_{n \in \mathbb Z} f(n) \cdot g(u-n).$

If $\mu$ is the Lebesgue measure, we get

$\displaystyle (f * g)(u) = \int_{-\infty}^{\infty} f(t) \cdot g(u-t)\, \mathrm dt.$

Proof. Denote $g(x,y) = x+y$ so that for any fixed $x$ , $g(x,y) \leq u$ if and only if $y \leq u - x$ . For any $K \in \frak{B}(\mathbb R)$ ,

$\displaystyle \begin{aligned} \int_{K} f_{X+Y}\, \mathrm d\mu &= \mathbb P(X+Y \in K) \\ &= \mathbb P((X,Y) \in g^{-1}(K)) \\ &= \int_{g^{-1}(K)} f_{X,Y}\, \mathrm d\mu^2 \\ &= \int_{g^{-1}(K)} f_{X,Y}(x,y)\, \mathrm d\mu^2(x,y) \\ &= \int_{\mathbb R} \int_{K-x} f_{X,Y}(x,y)\, \mathrm d\mu(y)\, \mathrm d\mu(x) \\ &= \int_{\mathbb R} \int_{K} f_{X,Y}(x,y-x)\, \mathrm d\mu(y)\, \mathrm d\mu(x) \\ &= \int_{K} \int_{\mathbb R} f_{X,Y}(x,y-x)\, \mathrm d\mu(x)\, \mathrm d\mu(y) \\ &= \int_{K} \underbrace{ \int_{\mathbb R} f_{X}(x) \cdot f_Y(y-x)\, \mathrm d\mu(x) }_{(f_X * f_Y)(y)} \, \mathrm d\mu(y) = \int_{K} f_X * f_Y\, \mathrm d\mu. \end{aligned}$

Finally, let’s concretely add two independently normally distributed random variables.

Theorem 2. If $X \sim \mathcal N(\mu_1, \sigma_1^2)$ and $Y \sim \mathcal N(\mu_2, \sigma_2^2)$ are independent, then $X + Y \sim \mathcal N(\mu_1 + \mu_2, \sigma_1^2 + \sigma_2^2)$ .

Proof. Write $X = \mu_1 + \sigma_1 Z_1$ and $Y = \mu_2 + \sigma_2 Z_2$ . Then

$X + Y = (\mu_1 + \mu_2) + \sigma_1 Z_1 + \sigma_2 Z_2,$

where $Z_1, Z_2 \sim \mathcal N(0, 1)$ are independent. It suffices to prove that

$\sigma_1 Z_1 + \sigma_2 Z_2 \sim \mathcal N(0, \sigma_1^2 + \sigma_2^2).$

By construction, for any $\sigma > 0$ , if $U \sim \mathcal N(0, \sigma^2)$ , then

$\displaystyle f_U(u) = \frac{1}{ \sigma \sqrt{2\pi} } e^{-\frac{u^2}{2\sigma^2} }.$

Defining $W := \sigma_1 Z_1 + \sigma_2 Z_2$ ,

$\begin{aligned} f_W(w) &= (f_{\sigma_1 Z_1} * f_{\sigma_2 Z_2})(w) \\ &= \int_{-\infty}^{\infty} \frac{1}{ \sigma_1 \sqrt{2\pi} } e^{-\frac{t^2}{2\sigma_1^2} } \cdot \frac{1}{ \sigma_2 \sqrt{2\pi} } e^{-\frac{(w-t)^2}{2\sigma_2^2} }\, \mathrm dt \\ &= \frac{1}{\sigma_1 \sigma_2 \cdot 2\pi}\int_{-\infty}^{\infty} \exp\left( - \frac 12 \left( \frac{t^2}{\sigma_1^2} + \frac{(w-t)^2}{\sigma_2^2} \right)\right)\, \mathrm dt. \end{aligned}$

By algebruh,

$\displaystyle \frac{t^2}{\sigma_1^2} + \frac{(w-t)^2}{\sigma_2^2} = \left( \frac{1}{\sigma_1^2} + \frac{1}{\sigma_2^2} \right) t^2 - \frac{2w}{\sigma_2^2} \cdot t + \frac{w^2}{\sigma_2^2} = A(t - h)^2 + k.$

for carefully calculated constants $A, h, k$ , in particular, with

$\displaystyle A = \frac{1}{\sigma_1^2} + \frac{1}{\sigma_2^2},\quad k = \frac{w}{A \sigma_2^2} = -\frac 14 \cdot \frac{4w^2}{\sigma_2^4} \cdot \frac{\sigma_1^2 \cdot \sigma_2^2}{\sigma_1^2 + \sigma_2^2} + \frac{w^2}{\sigma_2^2} = -\frac{w^2}{\sigma_1^2 + \sigma_2^2}.$

Denoting $\sigma_3^2 := 1/A$ , the integral then simplifies to

$\begin{aligned}\int_{-\infty}^{\infty} e^{-\frac 12 (A(t-h)^2 + k)}\, \mathrm dt &= e^{-\frac 12 k} \cdot \sigma_3 \cdot \sqrt{2\pi} \cdot \underbrace{ \int_{-\infty}^{\infty} \frac{1}{\sigma_3 \sqrt{2\pi} }e^{-\frac {(t-h)^2}{2\sigma_3^2}} \, \mathrm dt }_1 \\ &= e^{-\frac 12 k} \cdot \sigma_3 \cdot \sqrt{2\pi}.\end{aligned}$

Therefore, denoting $\sigma^2 := \sigma_1^2 + \sigma_2^2$ so that $k = -w^2/\sigma^2$ ,

$\begin{aligned} f_W(w) &= \frac{\sigma_3}{\sigma_1 \sigma_2} \cdot \frac{1}{\sqrt{2\pi}} \cdot e^{-\frac 12 k} = \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{w^2}{2\sigma^2}}. \end{aligned}$

Therefore, $\sigma_1 Z_1 + \sigma_2 Z_2 = W \sim \mathcal N(0, \sigma^2) = \mathcal N(0, \sigma_1^2 + \sigma_2^2)$ , as required.

In practice, to compute probabilities involving normal distributions, we define

$\displaystyle \Phi(z) := \mathbb P(Z \leq z) = \frac 12 + \frac{1}{\sqrt{2\pi}} \int_0^z e^{-t^2/2}\, \mathrm dt$

and tabulate commonly used approximate values of $\Phi(z)$ for $z > 0$ , known in the statistics community as a $z$ -table. By symmetry, $\Phi(-z) = 1- \Phi(z)$ . For any $X \sim \mathcal N(\mu, \sigma^2)$ , since $X = \mu + \sigma Z$ , we can reduce the computation to

$\displaystyle \mathbb P(X \leq x) = \mathbb P\left( Z \leq \frac{x - \mu}{\sigma} \right) = \Phi \left( \frac{x - \mu}{\sigma}\right).$

By induction, we have the following sampling distribution $\bar X_n$ .

Corollary 2. For independent, identically distributed random variables $X_1,\dots, X_n \sim \mathcal N(\mu, \sigma^2)$ ,

$\displaystyle \bar X_n := \frac{1}{n} \sum_{i=1}^n X_i \sim \mathcal N \left( \mu, \frac{\sigma^2}{n} \right).$

The central limit theorem claims that even if $X_i$ are not normally distributed, $\bar X_n$ converges in distribution to $\mathcal N(\mu, \sigma^2/n)$ . For the limiting distribution to be constant, the equivalent claim is that $(\bar X_n-\mu) / (\sigma / \sqrt{n})$ converges in distribution to $Z \sim \mathcal N(0, 1)$ .

We have previously stated the special case $(\mu, \sigma) = (0, 1)$ and aim to prove it properly using techniques in stochastic calculus. If we have this result, we are emboldened to carry out hypothesis tests, which are commonplace in the STEM fields as well as the data-driven social sciences. We will digress to this application before we dive right back into our ascent toward the central limit theorem.

—Joel Kindiak, 23 Jul 25, 2257H

December 5, 2025
Legitimate Integral Swapping

Given two random variables $X, Y$ , what is the distribution of $X + Y$ ? We could define the discrete random variables, and perhaps continuous ones as well. But let’s go back to measure theory and define the joint distribution $(X, Y)$ as rigorously as possible.

Let $(\Omega, \mathcal F, \mu)$ be a measure space (or a probability space if $\mu$ is a probability measure). Equip $\mathbb R^n$ is equipped with the Borel $\sigma$ -algebra $\frak{B}(\mathbb R^n)$ , generated by open balls under the Euclidean metric.

Lemma 1. Let $X, Y : \Omega \to \mathbb R$ be a random variable. Then $(X,Y) : \Omega \to \mathbb R^2$ defined by $(X,Y)(\omega) := (X(\omega), Y(\omega))$ is a random variable.

Proof. The map $(X,Y)$ is continuous, and therefore, has open sets as pre-images of open sets. Therefore, $(X,Y)$ is $\mathcal F$ / $\frak{B}(\mathbb R^2)$ -measurable.

What would be a reasonable measure on $\mathbb R^2$ ? Intuitively, we should have a measure $\lambda_2$ on $\mathbb R^2$ such that $\lambda_2(K \times L) = \lambda_1(K) \cdot \lambda_1(L)$ , where $\lambda_1$ denotes the usual Lebesgue measure that we painstakingly constructed. In fact, more generally, given measure spaces $(\Omega_i, \mathcal F_i, \mu_i)$ , we would like to define a reasonable $\sigma$ -algebra $\mathcal F$ on $\Omega := \prod_{i=1}^n \Omega_i$ and a measure $\mu$ on $\mathcal F$ such that

$\displaystyle \mu\left( \prod_{i=1}^n K_i \right) = \prod_{i=1}^n \mu_i(K_i),\quad K_i \in \mathcal F_i.$

It turns out that with the help of Carathéodory’s extension theorem, this task isn’t as Sisyphean as it seems.

Theorem 1. Given measure spaces $(\Omega_i, \mathcal F_i, \mu_i)$ , there exists a $\sigma$ -algebra $\mathcal F$ on $\Omega := \prod_{i=1}^n \Omega_i$ and a measure $\mu$ on $\mathcal F$ such that $\prod_{i=1}^n K_i \in \mathcal F$ and

$\displaystyle \mu\left( \prod_{i=1}^n K_i \right) = \prod_{i=1}^n \mu_i(K_i),\quad K_i \in \mathcal F_i.$

Proof. We will prove the special case $n = 2$ for simplicity. Define the algebra

$\mathcal F^0 := \{ K \times L : K \in \mathcal F_1, L \in \mathcal F_2\}.$

For any $K \in \mathcal F_1$ and $x \in \Omega_1$ , define the $x$ -section $K_x \subseteq \Omega_2$ by

$K_x := \{ y \in \Omega_2 : (x, y) \in K\}.$

Define the $y$ -section $K^y$ similarly. Now given $M := \bigcup_{i=1}^n K_i \times L_i \in \mathcal F^0$ , for any $x \in \Omega_1$

$\displaystyle M_x = \bigcup_{i : x \in K_i} L_i \in \mathcal F_2.$

Therefore, the quantity $\mu_2(M_x)$ is well-defined. Similarly, $\mu_1(M^y)$ is well-defined for any $y \in \Omega_2$ . Hence, define the function $f_M(x) := \mu_2(M_x)$ , which is non-negative and simple since in the special case $M = (K_1 \times L_1) \cup (K_2 \times L_2)$ ,

$f_M = \mu_2(L_1 \cup L_2) \cdot \mathbb I_{K_1 \cap K_2} + \mu_2(L_1) \cdot \mathbb I_{K_1 \backslash K_2} + \mu_2(L_2) \cdot \mathbb I_{K_2 \backslash K_1}.$

We can similarly define $g_M(y) := \mu_1(M^y)$ , and define

$\displaystyle \mu_0(M) := \int_{\Omega_1} f_M\, \mathrm d\mu_1 = \int_{\Omega_2} g_M\, \mathrm d\mu_2.$

To see this in the simplest case when $M$ is a disjoint union (the rest follows by careful bookkeeping),

$\displaystyle \int_{\Omega_1} f_M\, \mathrm d\mu_1 = \sum_{i=1}^n \mu_2(L_i) \cdot \mu_1(K_i) = \int_{\Omega_2} g_M\, \mathrm d\mu_2.$

In particular, $\mu_0(K \times L) = \mu_1(K) \cdot \mu_2(L)$ . We claim that $\mu_0$ is countably additive. Fix $M = \bigsqcup_{i=1}^\infty M_i \in \mathcal F^0$ . Then for any $x \in \Omega_1$ ,

$\displaystyle f_M(x) = \mu_2(M_x) = \mu_2 \left( \bigsqcup_{i=1}^\infty (M_i)_x \right) = \sum_{i=1}^\infty \mu_2((M_i)_x) = \sum_{i=1}^\infty f_{M_i}(x).$

Therefore, the function $\sum_{i=1}^n f_{M_i}$ converges monotonically to $f_M$ , and by the monotone convergence theorem,

$\displaystyle \mu_0(M) = \int_{\Omega_1} f_M\, \mathrm d\mu_1 = \sum_{i=1}^\infty \int_{\Omega_1} f_{M_i}\, \mathrm d\mu_1 = \sum_{i=1}^\infty \mu_0(M_i).$

Now apply Carathéodory’s extension theorem to obtain a $\sigma$ -algebra $\mathcal F \supseteq \mathcal F^0$ and a measure $\mu : \mathcal F \to [0, \infty]$ such that $\mu|_{\mathcal F^0} = \mu_0$ .

Theoretically, we could just start defining random variables on the product space $\Omega_1 \times \Omega_2$ and go on our merry way. But we still need to answer a key question: given the distributions $\mathbb P_X$ and $\mathbb P_Y$ , how do we compute $\mathbb P_{X+Y}$ ? In a more abstract manner, we need to integrate with respect to our newly minted measure $\mu$ in a computationally consistent manner with integrals with respect to our old measures $\mu_1,\mu_2$ respectively. Surprisingly, answering this question leads us to one of the most important theorems in multivariable calculus, which is Fubini’s theorem, as it allows us to rigorously swap integrals—a key tool in any reasonable calculation.

Denote the base measure spaces by $\Omega_1 \equiv (\Omega_1,\mathcal F_1,\mu_1)$ and $\Omega_2 \equiv (\Omega_2,\mathcal F_2,\mu_2)$ , and their product space by $\Omega \equiv (\Omega, \mathcal F, \mu)$ . By construction, $\mathcal F \supseteq \mathcal F_1 \times \mathcal F_2$ . We remark that for any $M \in \mathcal F_1 \times \mathcal F_2$ and $x \in \Omega_1, y \in \Omega_2$ , $M_x \in \mathcal F_2$ and $M^y \in \mathcal F_1$ , since the $\sigma$ -algebra

$\{M \in \mathcal F : \forall x \in \Omega_1\ \forall y \in \Omega_2\ M_x \in \mathcal F_1 \wedge M^y \in \mathcal F_2\}$

contains $\mathcal F_1 \times \mathcal F_2$ .

Now we observe that $\mathbb R = \bigcup_{n \in \mathbb Z} [n, n+1)$ and each $[n, n+1)$ has Lebesgue measure $1$ .

Definition 1. A measure space $(\Omega, \mathcal F, \mu)$ is $\sigma$ -finite if there exist $K_1,K_2\dots$ with $\mu(K_i) < \infty$ such that $\Omega = \bigcup_{i=1}^\infty K_i$ . For instance, $\mathbb R^n$ is $\sigma$ -finite.

Lemma 2. Suppose $\Omega_1,\Omega_2$ are $\sigma$ -finite. For any $M \in \mathcal F_1 \times \mathcal F_2$ , the non-negative functions $f_M(x) := \mu_2(M_x)$ and $g_M(y) := \mu_1(M^y)$ are measurable, and define the predicate $\phi$ by

$\displaystyle \phi(M) := \left( \int_{\Omega_1} f_M\, \mathrm d\mu_1 = \mu(M) = \int_{\Omega_2} g_M\, \mathrm d\mu_2 \right).$

Then $\phi(M)$ holds for any $M \in \mathcal F_1 \times \mathcal F_2$ . Note that this result is an extension from that of Theorem 1.

Proof. We first prove the case that $\mu_1(\Omega_1) < \infty$ and $\mu_2(\Omega_2) < \infty$ . It is straightforward that $\phi(M)$ holds if $M \in \mathcal F^0$ or even a disjoint union of sets in $\mathcal F^0$ . If $\phi(M_1)$ and $\phi(M_2)$ , then $\phi(M_1 \backslash M_2)$ . Finally, if $M_1 \subseteq M_2 \subseteq \dots$ and $\phi(M_i)$ , then defining $M = \bigcup_{i=1}^\infty M_i$ , $f_M = \lim_{n \to\infty} f_{M_n}$ is measurable, and by the monotone convergence theorem,

$\displaystyle \int_{\Omega_1} f_M\, \mathrm d\mu_1 = \lim_{n \to \infty} \int_{\Omega_1} f_{M_n}\, \mathrm d\mu_1 = \lim_{n \to \infty} \mu(M_n) = \mu(M).$

Therefore, $\phi(M)$ . Let $\mathcal F' \supseteq \mathcal F_0$ denote the smallest subset of $\mathcal F$ such that these two properties are satisfied. We can verify that $\mathcal F'$ is a $\sigma$ -algebra, and hence contains $\mathcal F_1 \times \mathcal F_2$ , as required.

We now generalise to the $\sigma$ -finite case. Suppose $K_1 \subseteq K_2 \subseteq \dots$ such that $\bigcup_{i=1}^\infty K_i = \Omega_1$ , and $\bigcup_{i=1}^\infty L_i = \Omega_2$ similarly. For each $I$ , define $M_i := M \cap (K_i \times L_i)$ so that $M = \bigcup_{i=1}^\infty M_i$ . The result follows by the monotone convergence theorem.

Lemma 3. For any map $f : \Omega_1 \times \Omega_2 \to [0, \infty]$ , all of its sections

$f_x := f(x, \cdot) : \Omega_2 \to [0, \infty],\quad f^y := f( \cdot, y) : \Omega_1 \to [0, \infty]$

are measurable.

Proof. Apply Lemma 2 to the result $f_x^{-1}([a, \infty]) = (f^{-1}([a, \infty]))_x \in \mathcal F_2$ .

We can now discuss the Fubini-Tonelli theorem. The Fubini theorem is the special case when all integrals therein are finite. Here, a function $f : \Omega \to [-\infty, \infty]$ is integrable if $K := f^{-1}(\{-\infty, \infty\})$ has measure zero and $f \cdot \mathbb I_{\Omega \backslash K}$ is integrable.

Theorem 2 (Fubini-Tonelli Theorem). Suppose $\Omega_1,\Omega_2$ are $\sigma$ -finite. If $f:\Omega_1 \times \Omega_2 \to [-\infty, \infty]$ is either non-negative and measurable (resp. integrable), then the functions $g, h$ defined by

$\displaystyle g(x) := \int_{\Omega_2} f(x, \cdot)\, \mathrm d\mu_2,\quad h(y) := \int_{\Omega_1} f(\cdot, y)\, \mathrm d\mu_1$

are measurable (resp. integrable) and

$\displaystyle \int_{\Omega_1} \int_{\Omega_2} f\, \mathrm d\mu_2\, \mathrm d\mu_1 = \int_{\Omega_2} \int_{\Omega_1} f\, \mathrm d\mu_1\, \mathrm d\mu_2 = \int_{\Omega} f\, \mathrm d\mu.$

Proof. We return to the usual simple $\Rightarrow$ non-negative measurable $\Rightarrow$ integrable strategy. If $f = \mathbb I_M$ , then we obtain this result by Lemma 2. The result extends by linearity to non-negative simple functions.

If $f$ is non-negative, find a sequence of non-negative simple functions $\Phi_n$ that monotonically converge to $f$ . By the monotone convergence theorem,

$\displaystyle \int_{\Omega} f\, \mathrm d\mu = \lim_{n \to \infty} \int_{\Omega} \Phi_n\, \mathrm d\mu.$

For each $n$ , define $\varphi_n$ by setting for each $x \in \Omega_1$ , $\varphi_n(x) := \int_{\Omega_2} \Phi_n(x,\cdot)\, \mathrm d\mu_2$ . Then $\varphi_n$ monotonically increases to $g$ . By the monotone convergence theorem again, since $\Phi_n$ are all step functions,

$\displaystyle \int_{\Omega_1} g\, \mathrm d\mu_1 = \lim_{n \to \infty} \int_{\Omega_1} \varphi_n\, \mathrm d \mu_1 = \lim_{n \to \infty} \int_{\Omega} \Phi_n\, \mathrm d\mu = \int_{\Omega} f\, \mathrm d\mu.$

Finally, in the case $f$ is integrable, write $f = f^+ - f^-$ and perform needful bookkeeping.

As much as we feel somewhat justified to add distributions in general, there is one more measure-theoretic machinery we need to discuss—the technical density function known as the Radon-Nikodým derivative. In doing so, we can be justified in letting $f_X$ denote the density function for any sufficiently nice random variable $X$ .

—Joel Kindiak, 21 Jul 25, 2313H

December 2, 2025