KindiakMath

The Exponential Family

Recall that if \lambda > 0 and X_n \sim \mathrm{Geom}(\lambda/n), for any x \geq 0,

\displaystyle \lim_{n \to \infty} \mathbb P\left( \frac {X_n}n > x \right) = e^{-\lambda x}.

Definition 1. A continuous random variable is said to follow an exponential distribution with rate parameter \lambda > 0, denoted X \sim \mathrm{Exp}(\lambda), if

\mathbb P(X > x) = e^{-\lambda x}.

Suppose X \sim \mathrm{Exp}(\lambda).

Problem 1. Prove the following properties:

  • f_X(x) = \lambda e^{-\lambda x} \cdot \mathbb I_{[0,\infty)}(x),
  • \mathbb E[X] = 1/\lambda,
  • \mathrm{Var}(X) = 1/\lambda^2,
  • X satisfies the memoryless property.
(Click for Solution)

Solution. The c.d.f. F_X of X for x > 0 is given by

F_X(x) = \mathbb P(X \leq x) = 1 - \mathbb P(X > x) = 1 - e^{-\lambda x}.

Hence,

\displaystyle f_X(x) = \frac{\mathrm d}{\mathrm dx}(F_X(x)) = \frac{\mathrm d}{\mathrm dx} (1- e^{-\lambda x}) = \lambda e^{-\lambda x}.

For the second result, we use the tail-probability characterisation of the expectation, where the interchange of integrals is valid by Fubini’s theorem:

\begin{aligned} \mathbb E[X] &= \int_{-\infty}^{\infty} x f_X(x)\, \mathrm dx \\ &= \int_{0}^{\infty} \int_0^x f_X(x)\, \mathrm dy\, \mathrm dx \\ &= \int_{0}^{\infty} \int_{y}^\infty f_X(x) \, \mathrm dx \, \mathrm dy \\ &= \int_{0}^{\infty} \mathbb P(X > y) \, \mathrm dy. \end{aligned}

Hence, for X \sim \mathrm{Exp}(\lambda),

\begin{aligned} \mathbb E[X] &= \int_{0}^{\infty} e^{-\lambda y}\, \mathrm dy = \frac 1{\lambda} \cdot [-e^{-\lambda y}]_0^\infty = \frac 1{\lambda} \cdot (0 - (-1)) = \frac 1{\lambda}. \end{aligned}

For the variance, we adopt a similar approach:

\begin{aligned} \mathbb E[X^2] &= \int_{-\infty}^{\infty} 2y \cdot \mathbb P(X > y) \, \mathrm dy \\ &= \int_{0}^{\infty} 2y \cdot e^{-\lambda y} \, \mathrm dy \\ &= \frac 2{\lambda} \int_0^\infty y e^{-\lambda y}\, \mathrm dy \\ &= \frac 2{\lambda} \cdot \mathbb E[X] = \frac 2{\lambda^2}. \end{aligned}

Therefore,

\displaystyle \mathrm{Var}(X) = \mathbb E[X^2] - \mathbb E[X]^2 = \frac{2}{\lambda^2} - \frac 1{\lambda^2} = \frac{1}{\lambda^2}.

For the memoryless property,

\begin{aligned} \mathbb P(X > s + t \mid X > t) &= \frac{\mathbb P(X > s + t, X > t)}{\mathbb P(X > t)} \\ &= \frac{\mathbb P(X > s+t)}{\mathbb P(X > t)} = \frac{e^{-\lambda(s+t)}}{e^{-\lambda t}} \\ &= e^{-\lambda s} = \mathbb P(X > s).\end{aligned}

Problem 2. Suppose Y \sim \mathrm{Exp}(\mu) is independent to X.

  • Calculate the distribution of \min\{X, Y\}.
  • If \lambda = \mu, evaluate the p.d.f. of X + Y.
(Click for Solution)

Solution. Denoting W := \min\{X, Y\},

\begin{aligned} \mathbb P(W > w) &= \mathbb P(X > w, Y > w) \\ &= \mathbb P(X > w) \cdot \mathbb P(Y > w)\\&= e^{-\lambda w} \cdot e^{-\mu w} \\ &= e^{-(\lambda + \mu) w}. \end{aligned}

Hence, \min\{X, Y\} = W \sim \mathrm{Exp}(\lambda + \mu). To evaluate the p.d.f. of U:= X+ Y, we compute the convolution of their individual p.d.f.s:

\begin{aligned} f_U(u) &= (f_X * f_Y)(u) \\ &= \int_0^u f_X(x) \cdot f_Y(u-x)\, \mathrm dx \\ &= \int_0^u \lambda e^{-\lambda x} \cdot \mu e^{-\mu(u - x)}\, \mathrm dx \\ &= \lambda \cdot \mu \cdot e^{-\mu u} \cdot \int_0^u e^{-(\lambda - \mu) x} \, \mathrm dx \\ &= \lambda^2 \cdot e^{-\lambda u} \cdot \int_0^u 1 \, \mathrm dx \\ &= \lambda^2 \cdot u \cdot e^{-u}. \end{aligned}

Definition 2. A continuous random variable X is said to follow a gamma distribution with shape parameter \alpha > 0 and rate parameter \lambda > 0, denoted X \sim \Gamma( \alpha, \lambda) if it has a p.d.f. given by

\displaystyle f_X(x) = \frac{\lambda^\alpha}{\Gamma(\alpha)} \cdot x^{\alpha - 1} \cdot e^{-\lambda x}.

Problem 3. Prove the following properties:

  • if X \sim \Gamma(\alpha, \lambda), then \mathbb E[X] = \alpha/\lambda, \mathrm{Var}(X) = \alpha/\lambda^2,
  • if X_i \sim \Gamma(\alpha_i, \lambda) are i.i.d., then \sum_{i=1}^n X_i \sim \Gamma(\sum_{i=1}^n \alpha_i, \lambda),
  • if X \sim \Gamma(\alpha, \lambda) and c > 0, then cX \sim \Gamma(\alpha, \lambda/c).
(Click for Solution)

Solution. Suppose Y \sim \Gamma(\alpha + 1,\lambda). By definition of the expectation,

\begin{aligned} \mathbb E[X^n] &= \int_0^\infty x^n \cdot \frac{\lambda^\alpha}{\Gamma(\alpha)} \cdot x^{\alpha - 1} \cdot e^{-\lambda x}\, \mathrm dx \\ &= \frac {\alpha \cdot (\alpha+1) \cdot \cdots \cdot (\alpha +n-1)}{\lambda^n} \cdot \int_0^\infty \frac{\lambda^{\alpha+n}}{\Gamma(\alpha+1)} \cdot x^{(\alpha + n) - 1} \cdot e^{-\lambda x}\, \mathrm dx \\ &= \frac{\alpha \cdot (\alpha+1) \cdot \cdots \cdot (\alpha +n-1)}{\lambda^n} \cdot \int_0^\infty f_Y(x)\, \mathrm dx \\ &= \frac{\alpha \cdot (\alpha+1) \cdot \cdots \cdot (\alpha +n-1)}{\lambda^n} . \end{aligned}

Hence, \mathbb E[X] = \alpha/\lambda, and

\begin{aligned} \mathrm{Var}(X) &= \mathbb E[X^2] - \mathbb E[X]^2 = \frac{\alpha \cdot (\alpha+1)}{\lambda^2} - \frac{\alpha^2}{\lambda^2} = \frac{\alpha}{\lambda^2}. \end{aligned}

We prove the second result by induction. Suppose X \sim \Gamma(\alpha, \lambda) and Y \sim \Gamma (\beta, \lambda) are independent. To evaluate the p.d.f. of U:= X+ Y, we compute the convolution of their individual p.d.f.s:

\begin{aligned} f_U(u) &= (f_X * f_Y)(u) \\ &= \int_0^u f_X(x) \cdot f_Y(u-x)\, \mathrm dx \\ &= \int_0^u \frac{\lambda^\alpha}{\Gamma(\alpha)} \cdot x^{\alpha - 1} \cdot e^{-\lambda x} \cdot \frac{\lambda^\beta}{\Gamma(\beta)} \cdot (u-x)^{\beta - 1} \cdot e^{-\lambda (u-x)}\, \mathrm dx \\ &= \frac{\lambda^{\alpha + \beta}}{\Gamma(\alpha) \cdot \Gamma(\beta)} \cdot \int_0^u x^{\alpha - 1} \cdot (u-x)^{\beta - 1} \cdot e^{-\lambda u}\, \mathrm dx \\ &= \frac{\lambda^{\alpha + \beta}}{\Gamma(\alpha) \cdot \Gamma(\beta)} \cdot \int_0^1 (ut)^{\alpha - 1} \cdot (u-ut)^{\beta - 1} \cdot e^{-\lambda u}\cdot u\, \mathrm dt \\ &= \frac{\lambda^{\alpha + \beta}}{\Gamma(\alpha + \beta)} \cdot u^{(\alpha+\beta) -1}\cdot e^{-\lambda u} \cdot \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha) \cdot \Gamma(\beta)} \cdot \int_0^1 t^{\alpha - 1} \cdot (1-t)^{\beta - 1} \, \mathrm dt \\ &= \frac{\lambda^{\alpha + \beta}}{\Gamma(\alpha + \beta)} \cdot u^{(\alpha+\beta) -1}\cdot e^{-\lambda u}. \end{aligned}

Therefore, W \sim \Gamma(\alpha +\beta, \lambda). Inductively, if X_i \sim \Gamma(\alpha_i, \lambda) are i.i.d.,

\displaystyle \sum_{i=1}^{k+1} X_i = \sum_{i=1}^{k} X_i + X_{k+1} \sim \Gamma \left( \sum_{i=1}^{k} \alpha_i + \alpha_{k+1}, \lambda \right) = \Gamma \left( \sum_{i=1}^{k+1} \alpha_i, \lambda \right).

For the final property, denoting V := cX,

\begin{aligned} f_{V}(v) = f_{cX}(v) &= \frac 1c \cdot f_X\left( \frac vc \right) \\ &= \frac 1c \cdot \frac{\lambda^\alpha}{\Gamma(\alpha)} \cdot \left( \frac vc \right)^{\alpha - 1} \cdot e^{-\lambda v/c} \\ &= \frac{(\lambda /c)^\alpha}{\Gamma(\alpha)} \cdot v^{\alpha - 1} \cdot e^{-(\lambda /c) v}. \end{aligned}

Hence, cX = V \sim \Gamma(\alpha, \lambda / c).

Given probability distributions \mathbb Q_1, \mathbb Q_2, write \mathbb Q_1 =\mathbb Q_2 if there exists a random variable X such that X \sim \mathbb Q_1 and X \sim \mathbb Q_2.

Problem 4. Prove the following properties:

  • \mathrm{Exp}(\lambda) = \Gamma(1, \lambda),
  • \Gamma(\nu/2, 1/2) = \chi^2(\nu),
  • for i.i.d. X_1,\dots, X_n \sim \mathrm{Exp}(\lambda), \sum_{i=1}^n X_i \sim \Gamma(n, \lambda),\bar X \sim \Gamma(n, \lambda/n),
  • for any fixed c > 0, if W \sim \chi^2(\nu), then cW \sim \Gamma(\nu/2, 1/(2c)).
(Click for Solution)

Solution. We note that if X \sim \Gamma( 1,\lambda), since \Gamma(1) = 0! = 1,

\displaystyle f_X(x) = \frac{\lambda}{\Gamma(1)} \cdot x^{1 - 1} \cdot e^{-\lambda x} = \lambda e^{-\lambda x},

so that X \sim \mathrm{Exp}(1, \lambda). If X \sim \Gamma( \nu/2, 1/2), then

\displaystyle f_X(x) = \frac{(1/2)^{\nu/2}}{\Gamma(\nu/2)} \cdot x^{\nu/2 - 1} \cdot e^{-x/2} = \frac{1}{2^{\nu/2} \cdot \Gamma(\nu/2)} \cdot x^{\nu/2 - 1} \cdot e^{-x/2}.

The last two results are immediate corollaries of Problem 3.

These probability distributions are examples of the exponential family of probability distributions.

—Joel Kindiak, 4 Aug 25, 1356H

, ,

Published by

joelkindiak

Leave a comment