KindiakMath

Category: Theory

Diagonalization for Legends
Introductory mathematics undergraduates do not know what diagonalization is. Those introduced to linear algebra learn that not all matrices can be diagonalised—not even all square matrices. Those who learn a bit more recognise that square matrices over $\mathbb C$ can be diagonalised in a generalised sense into the Jordan normal form.

Legends suggest that any matrix over $\mathbb K = \mathbb R$ or $\mathbb C$ can be diagonalised. In a modified sense, of course. This is called singular value decomposition.

Theorem 1 (Singular Value Decomposition). For any matrix $\mathbf A \in \mathcal M_{m \times n}(\mathbb K)$ , there exist unitary matrices $\mathbf U \in \mathrm{GL}_m(\mathbb K)$ and $\mathbf V \in \mathrm{GL}_n(\mathbb K)$ and a diagonal matrix $\mathbf{\Sigma} \in \mathcal M_{m \times n}(\mathbb K)$ such that

$\mathbf A = \mathbf U \mathbf{\Sigma} \mathbf V^*.$

Here, we say that $\mathbf{\Sigma} \in \mathcal M_{m \times n}(\mathbb K)$ is diagonal to mean that $\mathbf{\Sigma} _{ij} = 0$ whenever $i \neq j$ .

Lemma 1. The matrix $\mathbf A^* \mathbf A \in \mathcal M_{n \times n}(\mathbb K)$ is unitarily diagonalisable with nonnegative eigenvalues.

Proof. By conjugate-transpose properties,

$(\mathbf A^* \mathbf A)^* = \mathbf A^* (\mathbf A^*)^* = \mathbf A^* \mathbf A.$

Therefore, the matrix $\mathbf A^* \mathbf A$ is self-adjoint, and thus normal. By the spectral theorems, it is unitarily diagonalisable, i.e. there exists an orthonormal basis of $\mathbb K^n$ consisting of eigenvectors of $\mathbf A^* \mathbf A$ .

Now suppose $(\mathbf A^* \mathbf A) \mathbf v = \lambda \mathbf v$ . Then

$\begin{aligned} \langle \mathbf A \mathbf v , \mathbf A \mathbf v \rangle &= \langle (\mathbf A^* \mathbf A) \mathbf v , \mathbf v \rangle = \langle \lambda \mathbf v, \mathbf v \rangle = \lambda \langle \mathbf v , \mathbf v \rangle. \end{aligned}$

By the non-degeneracy of $\langle \cdot , \cdot \rangle$ ,

$\displaystyle \lambda = \frac{ \langle \mathbf A \mathbf v , \mathbf A \mathbf v \rangle }{\langle \mathbf v , \mathbf v \rangle} \geq \frac{ 0 }{ \langle \mathbf v , \mathbf v \rangle } = 0.$

Lemma 2. Let $\mathbf A \in \mathcal M_{k \times m}(\mathbb K)$ and $\mathbf B \in \mathcal M_{m \times n}(\mathbb K)$ be matrices.
- $\mathrm{rank}(\mathbf A) = \mathrm{rank}(\mathbf A^*)$ .
- $\mathrm{rank}(\mathbf A) \leq \min\{k, m\}$ .
- $\mathrm{rank}(\mathbf A \mathbf B) \leq \min\{\mathrm{rank}(\mathbf A), \mathrm{rank}(\mathbf B)\}$ .
- $\mathrm{rank}(\mathbf A^* \mathbf A) = \mathrm{rank}(\mathbf A) = \mathrm{rank}( \mathbf A \mathbf A^*)$ .
Proof. For the first claim, we leave it as an exercise to verify that $\mathrm{ker}(\mathbf A^*) = \mathbf A(\mathbb K^m)^{\perp}$ . By orthogonal decomposition,

$\mathbb K^k = \mathbf A(\mathbb K^m) \oplus \mathbf A(\mathbb K^m)^{\perp}.$

Hence,

$\begin{aligned} \mathrm{nullity}(\mathbf A^*) &= \mathrm{dim}(\mathbf A(\mathbb K^m)^{\perp}) \\ &= k - \mathrm{dim}(\mathbf A(\mathbb K^m)) \\ &= k - \mathrm{rank}(\mathbf A). \end{aligned}$

By the rank-nullity theorem,

$\begin{aligned} \mathrm{rank}(\mathbf A) &= k - \mathrm{nullity}(\mathbf A^*) = \mathrm{rank}(\mathbf A^*). \end{aligned}$

The second result follows from the inequalities $\mathrm{rank}(\mathbf A) \leq m$ and $\mathrm{rank}(\mathbf A^*) \leq k$ . The third result follows from the containment $\mathbf A\mathbf B(\mathbb K^n) \subseteq \mathbf A(\mathbb K^m)$ and

$\mathrm{rank}(\mathbf A \mathbf B) = \mathrm{rank}(\mathbf B^* \mathbf A^*) \leq \mathrm{rank}(\mathbf B^*) = \mathrm{rank}(\mathbf B).$

The direction $\leq$ in the fourth result is an obvious corollary. For the direction $\geq$ , we leave it as an exercise to verify that $\mathrm{ker}(\mathbf A^* \mathbf A) \subseteq \mathrm{ker}(\mathbf A)$ , from which the result follows by the rank-nullity theorem.

Proof of Theorem 1. We first note that it suffices to establish the equation

$\mathbf A \mathbf V = \mathbf U \mathbf{\Sigma}.$

Letting $\sigma_i := \mathbf{\Sigma}_{ii}$ for $1 \leq i \leq r$ , if $\{\mathbf u_1,\dots,\mathbf u_m\}$ is any orthonormal basis for $\mathbb K^m$ and $\{\mathbf v_1,\dots,\mathbf v_n\}$ is any orthonormal basis for $\mathbb K^n$ , defining

$\mathbf U := \begin{bmatrix} \mathbf u_1 & \cdots & \mathbf u_m \end{bmatrix},\quad \mathbf V := \begin{bmatrix} \mathbf v_1 & \cdots & \mathbf v_n \end{bmatrix},$

we obtain for any $1 \leq r \leq \min\{m , n \}$ ,

$\begin{aligned} \mathbf A \mathbf V &= \mathbf A \begin{bmatrix} \mathbf v_1 & \cdots & \mathbf v_n \end{bmatrix} \\ &= \begin{bmatrix} \mathbf A \mathbf v_1 & \cdots & \mathbf A \mathbf v_r & \mathbf A \mathbf v_{r+1} & \cdots & \mathbf A \mathbf v_n \end{bmatrix} \end{aligned}$

and

$\begin{aligned} \mathbf U \mathbf{\Sigma} &= \begin{bmatrix} \mathbf u_1 & \cdots & \mathbf u_m \end{bmatrix} \begin{bmatrix} \sigma_1 & 0 & \cdots & 0 & \cdots & 0 \\ 0 & \ddots & \ddots & \vdots & & \vdots \\ \vdots & \ddots & \sigma_r & 0 & \cdots & 0 \\ 0 & \cdots & 0 & 0 & \cdots & 0 \\ \vdots & & \vdots & \vdots & & \vdots \\ 0 & \cdots & 0 & 0 & \cdots & 0 \end{bmatrix} \\ &= \begin{bmatrix} \sigma_1 \mathbf u_1 & \cdots & \sigma_r \mathbf u_r & \mathbf 0 & \cdots & \mathbf 0 \end{bmatrix}. \end{aligned}$

This means we need to find suitable orthonormal vectors $\mathbf u_1,\dots,\mathbf u_m$ and $\mathbf v_1,\dots,\mathbf v_n$ , as well as suitably defined numbers $\sigma_1, \dots, \sigma_r$ such that

$\mathbf A \mathbf v_i = \sigma_i \mathbf u_i,\quad 1 \leq i \leq r.$

In particular,

$\displaystyle \sigma_i \neq 0 \quad \iff \quad \mathbf u_i = \frac{1}{\sigma_i} \mathbf A \mathbf v_i .$

By Lemma 2, suppose $\mathrm{rank}(\mathbf A) = r \leq \min\{m, n\}$ . By Lemma 1, the matrix $\mathbf A^* \mathbf A \in \mathcal M_{n \times n}(\mathbb K)$ is unitarily diagonalisable with nonnegative eigenvalues

$\lambda_1 \geq \dots \geq \lambda_r > 0 = \lambda_{r+1} = \cdots = \lambda_n,$

some possibly repeated, and orthonormal basis $\{\mathbf v_1,\dots,\mathbf v_n\}$ . Furthermore, for $i, j$ ,

$\langle \mathbf A \mathbf v_i, \mathbf A \mathbf v_j \rangle =\langle \mathbf v_i, \mathbf A^* \mathbf A \mathbf v_j \rangle = \langle \mathbf v_i, \lambda_j \mathbf v_j \rangle = \lambda_j \langle \mathbf v_i, \mathbf v_j \rangle.$

In particular, $\{\mathbf A \mathbf v_i : 1 \leq i \leq r\}$ is orthogonal, and for each $i$ , $\| \mathbf A \mathbf v_i \| = \sqrt{\lambda_i} =: \sigma_i$ . Define the corresponding unit vectors

$\displaystyle \mathbf u_i := \frac{ \mathbf A \mathbf v_i }{\| \mathbf A \mathbf v_i \|} = \frac{1}{\sigma_i} \mathbf A \mathbf v_i,\quad 1 \leq i \leq r.$

Then $\{\mathbf u_1,\dots,\mathbf u_r\}$ is clearly orthonormal. Extend it to an orthonormal basis for $\mathbb K^m$ via a modified form of the Gram-Schmidt process to yield the desired result.

Remark 1. The constants $\sigma_i$ are called the singular values of $\mathbf A$ .

Therefore, any matrix, given effort, can be diagonalised, albeit rather creatively using our arsenal of linear algebraic tools.

—Joel Kindiak, 26 Mar 25, 1920H
October 13, 2025
Fourier Orthogonality

Let’s discuss Fourier series using orthogonality—a fundamental approximation tool when studying periodic functions. Let $\mathbb K = \mathbb R$ or $\mathbb C$ .

Definition 1. For any $K \subseteq \mathbb R$ , define the subspace $\mathcal C(K, \mathbb K) \subseteq \mathcal F(K, \mathbb K)$ of functions that are continuous on $K$ .

Remark 1. The relevant continuity properties in the case $\mathbb K = \mathbb R$ are mostly preserved in the case $\mathbb K = \mathbb C$ due to the same topological ideas in real and complex analysis. Products of functions remain continuous, and in particular, Riemann-integrable in both cases, with the latter being Riemann-integrable if and only if the real and imaginary parts $\mathrm{Re}(f), \mathrm{Im}(f)$ are Riemann-integrable, in which can we define

$\displaystyle \int_a^b f(t)\, \mathrm dt := \int_a^b \mathrm{Re}(f)(t)\, \mathrm dt + i \int_a^b \mathrm{Im}(f)(t)\, \mathrm dt.$

Recall the usual the inner product on $\mathbb K^n \cong \mathcal F(\{1,\dots,n\}, \mathbb K)$ by

$\displaystyle \langle f, g \rangle = \sum_{k=1}^n f(k) \overline{g(k)}.$

Definition 2. Define the subspace

$\mathcal L^2([a, b], \mathbb K) := \{f \in \mathcal F([a, b], \mathbb K : |f|^2\ \text{is Riemann-integrable}\} \subseteq \mathcal F([a, b], \mathbb K)$

of functions that are square-integrable on $K$ . For simplicity, we make the identification in $\mathcal L^2([a, b], \mathbb K)$ :

$\displaystyle f = g\quad \iff \quad \int_a^b |(f-g)(t)|\, \mathrm dt = 0.$

This is certainly true if $f,g$ are continuous.

Remark 2. The more technical formulation is that the relation $\sim$ on $\mathcal L^2([a, b], \mathbb K)$ is an equivalence relation defined by

$\displaystyle f \sim g\quad \iff \quad \int_a^b |(f-g)(t)|\, \mathrm dt = 0,$

and that we will regard without ambiguity $\mathcal L^2([a, b], \mathbb K)/{\sim} = \mathcal L^2([a, b], \mathbb K)$ . In any case, we have

$\mathcal C([a, b], \mathbb K) \subseteq \mathcal L^2([a, b], \mathbb K) \subseteq \mathcal F([a, b], \mathbb K)$

as subspaces.

We can extend the inner product idea to $\mathcal L^2([a, b], \mathbb K) \subseteq \mathcal F(K, \mathbb K)$ .

Lemma 1. The map $\langle \cdot, \cdot \rangle : \mathcal L^2([a, b], \mathbb K) \times \mathcal L^2([a, b], \mathbb K) \to \mathbb K$ defined by

$\displaystyle \langle f, g \rangle := \int_a^b f(t) \overline{g(t)}\, \mathrm dt$

is a well-defined inner product.

Proof. To prove that $\langle \cdot, \cdot \rangle$ is well-defined in general requires some advanced analytic machinery known as Hölder’s inequality, and so we will relegate that discussion elsewhere in advanced real analysis. However, in the case $f, g$ are bounded, we also have $f \bar g$ being bounded, so that the integral is well-defined. This certainly holds when $f,g$ are continuous on $[a, b]$ due to the extreme value theorem.

We now verify the rest of the inner product axioms. For any $f$ ,

$\displaystyle \langle f, f \rangle = \int_a^b f(t) \overline{f(t)}\, \mathrm dt = \int_a^b |f(t)|^2\, \mathrm dt \geq 0,$

where equality holds if and only if $f = 0$ . The remaining two properties follow from the linearity of integration.

Corollary 1. A function $\alpha : [a, b] \to \mathbb R$ is a weight if it is nonnegative, continuous on $(a, b)$ , and Riemann-integrable on $[a, b]$ . For any weight $\alpha$ , the map $\langle \cdot, \cdot \rangle_\alpha : \mathcal L^2([a, b], \mathbb K) \times \mathcal L^2([a, b], \mathbb K) \to \mathbb K$ defined by

$\displaystyle \langle f, g \rangle_\alpha := \langle f, \alpha g\rangle$

is a well-defined inner product, in particular with the weight

$\displaystyle \alpha := \frac{1}{\langle 1, 1 \rangle} = \frac 1{b-a}.$

Henceforth, we will regard $\mathcal L^2([a, b]) = (\mathcal L^2([a, b]), \langle \cdot, \cdot \rangle_{\alpha})$ as an inner-product space. Furthermore, we denote $\langle \cdot, \cdot \rangle = \langle \cdot, \cdot \rangle_{\alpha_{[a, b]}}$ when there is no ambiguity.

Finally, we are most interested in periodic functions, such as $\sin$ and $\cos$ , which have a period of $2\pi$ . More precisely, $\sin = \sin(\cdot + 2\pi)$ , and for any $t < 2\pi$ , $\sin \neq \sin(\cdot + t)$ . In fact, since $e^{it} = \cos(t) + i \sin(t)$ by Euler’s formula, the function $f : \mathbb R \to \mathbb C$ defined by $f(t) = e^{it}$ also has a period of $2\pi$ .

Definition 3. For any $T > 0$ , we call a function $f \in \mathcal F(\mathbb R, \mathbb K)$ $T$ –periodic if $f = f(\cdot + T)$ and for $t < T$ , $f \neq f(\cdot + t)$ . Define the subspace

$\mathcal F_T := \{f \in \mathcal F(\mathbb R, \mathbb K) : f\ \text{is}\ T\text{-periodic}\},$

as well as the subspace

$\mathcal L_T^2 := \{f \in \mathcal F_T : f|_{[-T/2,T/2]} \in \mathcal L^2([-T/2,T/2])\}.$

Finally, define the set of periodic functions by $\displaystyle \bigcup_{T \in \mathbb R_{> 0}} \mathcal F_T$ .

Lemma 2. For any $a \in \mathbb R$ ,

$\mathcal L_T^2 := \{f \in \mathcal F_T : f|_{[a,a+T]} \in \mathcal L^2([a,a+T])\}.$

Proof. Suppose $a \geq 0$ without loss of generality. The proof follows from an integration by substitution:

$\begin{aligned} \int_a^{a+T}f(t)\, \mathrm dt &= \int_a^{T/2}f(t)\, \mathrm dt + \int_{T/2}^{a+T}f(t)\, \mathrm dt \\ &= \int_a^{T/2}f(t)\, \mathrm dt + \int_{T/2}^{a+T}f(t-T)\, \mathrm dt \\ &= \int_a^{T/2}f(t)\, \mathrm dt + \int_{-T/2}^{a}f(t)\, \mathrm dt \\ &= \int_{-T/2}^{a}f(t)\, \mathrm dt + \int_a^{T/2}f(t)\, \mathrm dt = \int_{-T}^{T}f(t)\, \mathrm dt.\end{aligned}$

Replacing $f$ with $f^2$ yields the desired results.

An inner product space isn’t much good unless we have a subspace with an orthonormal basis. For brevity, we shall abuse the notation $f \equiv f(t)$ when context is clear.

Theorem 1. For any $n \in \mathbb N$ , the set $\mathcal E_n := \{e^{ikt} : -n \leq k \leq n\} \subseteq \mathcal L_{2\pi}^2$ is orthonormal. Define the subspace $E_n := \mathrm{span}(\mathcal E_n) \subseteq \mathcal L_{2\pi}^2$ .

Proof. For any $k, m \in [-n, n]$ ,

$\begin{aligned}\langle e^{i k t}, e^{i m t} \rangle &= \frac 1{2\pi} \int_0^{2\pi} e^{i k t } \overline{e^{i m t}}\, \mathrm dt \\ &= \frac 1{2\pi} \int_0^{2\pi} e^{i k t } e^{-i m t}\, \mathrm dt \\ &= \frac 1{2\pi} \int_0^{2\pi} e^{i (k-m) t } \, \mathrm dt.\end{aligned}$

Therefore, $\langle e^{i k t}, e^{i k t} \rangle = 1$ while for $k \neq m$ ,

$\begin{aligned}\langle e^{i k t}, e^{i m t} \rangle &= \frac 1{2\pi} \int_0^{2\pi} e^{i (k-m) t } \, \mathrm dt =\frac 1{2\pi} \left[ \frac{e^{i(k-m)t}}{i(k-m)} \right]_0^{2\pi} = 0\end{aligned}$

by the $(2\pi)/k$ -periodicity of $e^{ikt}$ .

Corollary 2. For any $f \in \mathcal L_{2\pi}^2$ and any $n \in \mathbb N$ , the $n$ -th Fourier sum $S_n(f)$ of $f$ is given by

$\displaystyle S_n(f):= \mathrm{proj}_{E_n}(f) = \sum_{k=-n}^n \hat f(k) e^{ikt},$

where the Fourier coefficients are given by

$\displaystyle \hat f(k) := \frac 1{2\pi} \int_{-\pi}^{\pi} f(t) e^{-ikt}\, \mathrm dt.$

Proof. By the definition of orthogonal projection,

$\displaystyle \mathrm{proj}_{E_n}(f) = \sum_{k=-n}^n \langle f, e^{ikt} \rangle e^{ikt}.$

By definition of the inner product (on $[ -\pi, \pi ]$ ),

$\begin{aligned} \langle f, e^{ikt} \rangle &= \frac 1{\pi-(-\pi)} \int_{-\pi}^{\pi} f(t) \overline{e^{ikt}}\, \mathrm dt \\ &= \frac 1{2\pi} \int_{-\pi}^{\pi} f(t) e^{-ikt}\, \mathrm dt = \hat f(k). \end{aligned}$

This formula is, in fact, the motivation for the Fourier transform

$\displaystyle \hat f(\xi) = \frac 1{2\pi} \int_{-\infty}^{\infty} f(t) e^{-i\xi t}\, \mathrm dt,$

which we may explore in the future. A more pressing issue arises. If $f : \mathbb R \to \mathbb R$ , then how do we know that $S_n(f)$ is real-valued?

Corollary 3. If $\mathbb K = \mathbb R$ , then for any $f \in \mathcal L_{2\pi}^2$ and any $n \in \mathbb N$ ,

$\displaystyle (S_n(f))(t) = \frac{a_0}{2} + \sum_{k=1}^n (a_k \cos(kt) + b_k \sin(kt)),$

where

$\begin{aligned} a_n = \frac 1{\pi} \int_{-\pi}^{\pi} f(t) \cos(kt)\,\mathrm dt, \quad b_n = \frac 1{\pi} \int_{-\pi}^{\pi} f(t) \sin(kt)\,\mathrm dt. \end{aligned}$

Proof. By Euler’s formula,

$\displaystyle \hat f(k) = \frac 1{2\pi} \int_{-\pi}^{\pi} f(t) \cos(kt)\,\mathrm dt- i\cdot \frac 1{2\pi} \int_{-\pi}^{\pi} f(t) \sin(kt)\,\mathrm dt = \frac{a_k}{2} - i \frac{b_k}{2}.$

Furthermore, $\overline{\hat f(k) e^{ikt}} = \hat f(-k) e^{-ikt}$ . Hence,

$\displaystyle \begin{aligned} \hat f(-k) e^{-ikt} + \hat f(k) e^{ikt} &= \mathrm{Re}(2 \hat f(k) e^{ikt}) \\ &= \mathrm{Re}( ( a_k - i b_k ) (\cos(kt) + i \sin(kt) ) \\ &= \mathrm{Re}(a_k \cos(kt) + b_k \sin(kt) + i ( {\cdots} )) \\ &= a_k \cos(kt) + b_k \sin(kt).\end{aligned}$

Since $c_0 = a_0/2$ , the result follows.

Remark 2. Since the results hold for any $n$ , we usually write

$\displaystyle f(t) = \frac{a_0}{2} + \sum_{k=1}^\infty (a_k \cos(kt) + b_k \sin(kt)),$

where we assume suitable notions convergence (either pointwise or square-integrable) for sufficiently nicely-defined functions, so that we can evaluate several surprising infinite series.

Corollary 3. If $\mathbb K = \mathbb R$ , then for any $T > 0$ , $f \in \mathcal L_{T}^2$ and any $n \in \mathbb N$ ,

$\displaystyle (S_n(f))(t) = \frac{a_0}{2} + \sum_{k=1}^n (a_k \cos(kt) + b_k \sin(kt)),$

where

$\begin{aligned} a_k &= \frac 2{T} \int_{-T/2}^{T/2} f(t) \cos \left(\frac{2\pi k t}{T}\right)\,\mathrm dt, \\ b_k &= \frac 2{T} \int_{-T/2}^{T/2} f(t) \sin\left(\frac{2\pi k t}{T}\right)\,\mathrm dt. \end{aligned}$

Proof. Define the differentiable bijection $h : t \mapsto t T/(2\pi)$ so that $h' = T/(2\pi)$ , the function $g \in \mathcal L_{2\pi}^2$ defined by $g(t) = (f \circ h)(t)$ and the orthonormal set $\mathcal E_n' := \{h^{-1}(\phi) : \phi \in \mathcal E_n\}$ . By Lemma 1 and a change-of-variable,

$(S_n(f))(t) = \mathrm{proj}_{\mathrm{span}(\mathcal E_n')}(f) = \mathrm{proj}_{\mathrm{span}(\mathcal E_n)}(f \circ h) = (S_n(g))(t).$

Applying Corollary 3,

$(S_n(g))(t) = \displaystyle \frac{a_0}{2} + \sum_{k=1}^n (a_k \cos(kt) + b_k \sin(kt)),$

where we make the change-of-variable $u = h(t)$ to get

$\displaystyle \begin{aligned} a_n &= \frac 1{\pi} \int_{-\pi}^{\pi} g(t) \cos(kt)\, \mathrm dt \\ &= \frac 1{\pi} \int_{-\pi}^{\pi} f(h(t)) \cos(kt)\, \mathrm dt \\ &= \frac 1{\pi} \int_{h(-\pi)}^{h(\pi)} f(u) \cos(kh^{-1}(u)) \cdot \frac{1}{h'(t)}\, \mathrm du \\ &= \frac 1{\pi} \int_{-T/2}^{T/2} f(u) \cos \left( \frac{2\pi k u}{T} \right) \cdot \frac{2\pi}{T}\, \mathrm du,\end{aligned}$

and $b_n$ is derived similarly.

Using Remark 2, we can obtain rather surprising values for infinite series.

Theorem 2. $\displaystyle \frac 1{1^2} + \frac 1{2^2} + \frac 1{3^2} + \cdots = \frac{\pi^2}{6}$ .

Proof. Define $f : [-\pi,\pi] \to \mathbb R$ by $f(t) = t^2$ , and extend it to a $2\pi$ -periodic function via $f = f(\cdot + 2\pi)$ . Then

$\displaystyle a_0 = \frac{1}{\pi} \int_{-\pi}^{\pi} t^2\, \mathrm dt = \frac 1{\pi} \cdot 2 \cdot \frac{\pi^3}{3} = \frac{2\pi^2}{3}.$

Furthermore, for any $k \neq 0$ , two applications of integration by parts yields

$\displaystyle \begin{aligned} a_k &= \frac 1{\pi}\int_{-\pi}^{\pi} t^2 \cos(kt)\,\mathrm dt \\ &= \frac 1{\pi} \left[ \frac{t^2 \sin(kt)}{k} + \frac{2t \cos(kt)}{k^2} - \frac{2 \sin(kt)}{k^3}\right]_{-\pi}^{\pi} \\ &= \frac 1{\pi} \cdot 4\left[ \frac{t \cos(kt)}{k^2} \right]_{0}^{\pi} = \frac 1{\pi} \cdot \frac{4\pi \cos(k\pi)}{k^2} = {(-1)}^k \cdot \frac{4}{k^2}. \end{aligned}$

We leave it as an exercise to verify that for any $k$ , $b_k = 0$ . By Remark 2,

$\displaystyle f(t) = \frac{\pi^2}{3} + \sum_{k=1}^\infty {(-1)}^k \cdot \frac{4}{k^2} \cos(kt).$

Setting $t = \pi$ ,

$\displaystyle \pi^2 = f(\pi) = \frac{\pi^2}{3} + 4 \cdot \sum_{k=1}^\infty {(-1)}^k \cdot \frac{1}{k^2} (-1)^k = \frac{\pi^2}{3} + 4 \cdot \sum_{k=1}^\infty \frac{1}{k^2}.$

By algebruh,

$\displaystyle \frac 1{1^2} + \frac 1{2^2} + \frac 1{3^2} + \cdots = \sum_{k=1}^\infty \frac{1}{k^2} = \frac 14 \cdot \left( \pi^2 - \frac{\pi^2}{3} \right) = \frac{\pi^2}{6}.$

Finally, we will discuss one more crucial application—the singular value decomposition—which shall be our capstone result in applied linear algebra.

—Joel Kindiak, 22 Mar 25, 1300H

October 6, 2025
Creating Euclidean Geometry
Last time, we discussed symmetric bilinear forms and its applications into geometry, in particular, to rational trigonometry, which posits a computationally tractable alternative to classical trigonometry. The rational analog of distance $d$ would be the quadrance $Q$ , and the rational analog of angle $\theta$ would be the spread $s$ , connected in the case $V = \mathbb R^2$ by the equations

$Q = d^2,\quad s = \sin^2\theta.$

In this post, we see how we can use linear algebra and ideas in rational trigonometry to define a broad framework for geometry, of which we obtain Euclid’s five postulates in the special case $V = \mathbb R^2$ .

Let $V$ be a vector space over a field $\mathbb K$ .

Definition 1. An affine space $\mathbb A$ over $V$ is a set $\mathbb A$ , whose elements are called points, equipped with an addition map $+ : \mathbb A \times V \to \mathbb A$ that satisfies the following properties:
- For any $A \in \mathbb A$ , $A + \mathbf 0 = A$ .
- For any $A \in \mathbb A$ and $\mathbf v , \mathbf w \in V$ , $(A + \mathbf v) + \mathbf w = A + (\mathbf v + \mathbf w)$ .
- For any $A \in \mathbb A$ , the map $V \to \mathbb A, \mathbf v \mapsto A + \mathbf v$ is bijective.
Example 1. Any vector space $V$ is an affine space over itself. Defining $\mathbb A = V$ , for each point $A$ in $\mathbb A$ , we will define the position vector $\overrightarrow{OA} = A \in V$ .

Lemma 1. For points $A, B \in \mathbb A$ , there exists a unique vector $\overrightarrow{AB} \in V$ such that $A + \overrightarrow{AB} = B$ .

Proof. Fix $A, B \in \mathbb A$ . Since $\mathbf v \mapsto A + \mathbf v$ is bijective, there exists a unique vector $\mathbf v \in V$ such that $A + \mathbf v = B$ . Denote $\mathbf v = \overrightarrow{AB}$ .

Hence, we will denote $\overrightarrow{AB} = B - A = \overrightarrow{OB} - \overrightarrow{OA}$ without loss of ambiguity.

Lemma 2. For points $A, B, C, D \in \mathbb A$ , $\overrightarrow{AB} = \overrightarrow{CD}$ precisely when $A + \overrightarrow{CD} = B$ and $C + \overrightarrow{AB} = D$ .

Henceforth, let $\mathbb A$ denote any affine space over $V$ . Therefore, points are positions determined by vectors that start from the origin. Each point $R \in \mathbb A$ can be described by a position vector $\mathbf r = \overrightarrow{OR}$ . Let $\mathbb K$ be any field that contains $\mathbb Q$ .

Definition 1. A line $l$ containing the point $A \in \mathbb A$ and parallel to the vector $\mathbf d \in V$ is the set

$l := \{A + t \mathbf d : t \in \mathbb K\} \subseteq \mathbb A.$

Equivalently, we say that $R$ belongs in the line if its position vector $\mathbf r = \mathbf r(t)$ satisfies the parametric equation of a line, i.e. $l : \mathbf r(t) = \mathbf a + t \mathbf d$ .

Theorem 1. For any two distinct points $A, B \in \mathbb A$ , there exists a unique line that contains $A$ and $B$ .

Proof. For existence, define the line $l_{AB}$ by the equation $\mathbf r(t) = \mathbf a + t(\mathbf b - \mathbf a)$ . Since $\mathbf r(0) = \mathbf a$ and $\mathbf r(1) = \mathbf b$ , $A,B$ lie on $l_{AB}$ .

For uniqueness, let $m$ be any other line that passes through $A, B$ . Then there exists a point $C \in \mathbb A$ and a vector $\mathbf u \in V$ such that $m : \mathbf r(s) = \mathbf c + s\mathbf u$ . Find distinct scalars $s_0, s_1 \in \mathbb K$ such that $\mathbf c + s_0 \mathbf u = \mathbf a$ and $\mathbf c + s_1 \mathbf u = \mathbf b$ . Then

$\displaystyle \mathbf b - \mathbf a = (s_1 - s_0)\mathbf u \quad \Rightarrow \quad \mathbf u = \frac 1{s_1-s_0} (\mathbf b - \mathbf a).$

We need to prove that $m = l_{AB}$ . For any $F \in m$ , find $s_3 \in \mathbb K$ such that

$\displaystyle \mathbf f = \mathbf c + s_3 \mathbf u = \mathbf a + (s_3 - s_0)\mathbf u = \mathbf a + \frac{s_3 - s_0}{s_1 - s_0} (\mathbf b - \mathbf a)\quad \Rightarrow \quad F \in l_{AB}.$

Hence, $m \subseteq l_{AB}$ . We can prove $l_{AB} \subseteq m$ similarly since

$\mathbf a + t(\mathbf b - \mathbf a) = \mathbf c + (s_0 + ts_1 - ts_0)\mathbf u.$

In particular, when $\mathbf A = V = \mathbb R^2$ and $\mathbb K = \mathbb R$ , we obtain Euclid’s first two postulates:

Corollary 1. A straight line segment can be drawn joining any two points. Furthermore, any straight line segment can be extended indefinitely in a straight line.

The rest of Euclid’s postulates will require our previous discussion on symmetric bilinear forms $[ \cdot , \cdot ]$ , which basically characterise perpendicularity, since the usual inner product $\langle \cdot, \cdot \rangle$ on $\mathbb R^n$ yields the desired properties.

Recall that for any symmetric bilinear form $[ \cdot , \cdot ] : V \times V \to \mathbb K$ , we can define the quadrance $Q : V \to \mathbb K$ by $Q(\mathbf v) = [ \mathbf v , \mathbf v ]$ . In particular, for points $A , B \in \mathbb A$ , we can define the quadrance between two points $Q : \mathbb A \times \mathbb A \to \mathbb K$ by $Q(A,B) := Q(\overrightarrow{AB})$ .

Definition 3. A sphere $S \subseteq \mathbb A$ with centre $C$ and quadrance $Q$ is defined to be the set

$S := \{R \in \mathbb A : Q(C,R) = Q\} \subseteq \mathbb A.$

A sphere in $\mathbb A = V = \mathbb R^2$ equipped with the usual inner product is called a circle.

Theorem 2. For any two distinct points $A, B \in \mathbb A$ , there exists a unique circle with centre $A$ and quadrance $Q(A,B)$ .

Corollary 2. Given any straight line segment, a circle can be drawn having the segment as radius and one endpoint as center.

Definition 4. For $\mathbf u, \mathbf v \in V$ , we say that $\mathbf u$ and $\mathbf v$ are perpendicular or orthogonal, denoted $\mathbf u \perp \mathbf v$ , if $[\mathbf u, \mathbf v] = 0$ .

Lemma 3. If $\mathbf u, \mathbf v$ have nonzero quadrances, then $\mathbf u \perp \mathbf v \iff s(\mathbf u, \mathbf v) = 1$ .

Theorem 3. All right angles are congruent to one another.

We are left with the infamous parallel postulate, which seems nontrivial to state, but turns out to be definable using the notions we discussed in affine spaces. We remark that to debunk the parallel postulate, we will need a broader notion of space, possibly projective space, to account for the varied notions of parallelism.

Definition 5. Two nonzero vectors $\mathbf u, \mathbf v$ are parallel, denoted $\mathbf u \parallel \mathbf v$ , if the set $\{\mathbf u, \mathbf v\}$ is linearly dependent i.e. there exists some $\alpha \in\mathbb K$ such that $\mathbf v = \alpha \mathbf u$ . We say that the lines $l_{AB}$ and $l_{CD}$ are parallel if $\overrightarrow{AB} \parallel \overrightarrow{CD}$ .

Theorem 4. For any line $l$ and point $A$ , there exists at most one line passing through $A$ that is parallel to $l$ .

Proof. Let $l_1$ and $l_2$ be lines that pass through $A$ and are parallel to $l$ . Let $B,C$ be points on $l$ and define $\mathbf d := \overrightarrow{BC}$ . For any point $R$ on $l_1$ , $\overrightarrow{AR} \parallel \mathbf d$ . Likewise, for any point $S$ on $l_2$ , $\overrightarrow{AS} \parallel \mathbf d$ . Thus, $\overrightarrow{AR} \parallel \overrightarrow{AS}$ . Some bookkeeping yields $l_1 = l_2$ .

Corollary 3 (Playfair’s Axiom). In a plane, given a line and a point not on it, at most one line parallel to the given line can be drawn through the point.

Corollary 4. If two lines are drawn which intersect a third in such a way that the sum of the inner angles on one side is less than two right angles, then the two lines inevitably must intersect each other on that side if extended far enough.

Proof. Corollary 3 implies Corollary 4 via the following geometric argument, which only requires Corollary 1 and Corollary 2 to establish.

With that, we have successfully created a working model for Euclidean geometry, defining quadrances and spreads in general fields, then take square roots in the case $\mathbb K = \mathbb R$ and even employing trigonometric functions to obtain our classical notions of distances and angles (in radians). We now list three common starting points for elementary school students’ learning of Euclidean geometry.

Corollary 5. Define $1^{\circ} := \pi/180$ so that $180^{\circ} = \pi$ . The following proposition holds:
- Adjacent angles on a straight line sum to $180^{\circ}$ .
- Interior angles on the same side of a transversal sum to $180^{\circ}$ .
Interestingly, we do not have purely mechanistic tools to draw an angle of $1^{\circ}$ , but can define the concept mathematically and obtain close approximations to that. However, using constructions in Euclidean geometry, we can draw the obvious angle ${180}^\circ$ and the special angle $60^{\circ}$ , bisections of these angles, sums and differences, and integer multiples of said angles.

In the next post, we take advantage of inner product spaces to discuss orthogonal functions, and derive the usual formulas involved in discussing Fourier series.

—Joel Kindiak, 20 Mar 25, 1523H
September 22, 2025
Defending Rational Trigonometry
Let $\mathbb K$ be a field. In the case $\mathbb K = \mathbb Q \subseteq \mathbb R \subseteq \mathbb C$ , we can define the usual inner product $\langle \cdot, \cdot \rangle$ on $\mathbb K^n$ via

$\displaystyle \langle \mathbf u, \mathbf v \rangle = \sum_{i=1}^n u_i \bar{v}_i = \mathbf v^* \mathbf u = \mathbf v^* \mathbf I_n \mathbf u \in \mathbb K.$

Furthermore, in the case $\mathbb K = \mathbb Q \subseteq \mathbb R$ , $\mathbf v^* \mathbf u = \mathbf u^* \mathbf v$ due to symmetry. In particular, setting $\mathbf u = \mathbf v$ , we get

$\displaystyle \langle \mathbf v, \mathbf v \rangle = \sum_{i=1}^n v_i \bar{v}_i = \sum_{i=1}^n |v_i|^2 \in \mathbb K.$

It is the square that motivates us to call the map $Q(\mathbf v) = \langle \mathbf v, \mathbf v \rangle$ a quadratic form, that satisfies the following properties:

Definition 1. Let $V$ be any vector space over $\mathbb K$ . Let $Q : V \to \mathbb K$ be a map and define the map $[\cdot, \cdot] : V \times V \to \mathbb K$ by

$[\mathbf u,\mathbf v] := \frac 12 (Q(\mathbf u + \mathbf v) - Q(\mathbf u) - Q(\mathbf v)).$

We call $Q$ a quadratic form if
- For any $\alpha \in \mathbb K$ , $Q(\alpha \mathbf v) = \alpha^2 Q(\mathbf v)$ .
- The map $[ \cdot , \cdot ]$ is symmetric and bi-linear.
Lemma 1. For any matrix $\mathbf A \in \mathcal M_{n \times n}(\mathbb K)$ , the map $Q_{\mathbf A} : \mathbb K^n \to \mathbb K$ defined by $Q_{\mathbf A}(\mathbf v) = \mathbf v^{\mathrm T} \mathbf A \mathbf v$ is a quadratic form. Furthermore, there exists a unique symmetric matrix $\mathbf B$ such that $Q_{\mathbf A} = Q_{\mathbf B}$ .

Proof. For the quadratic form claim, we first note that

$\begin{aligned} 2[\mathbf u, \mathbf v]_{\mathbf A} &= Q_{\mathbf A}(\mathbf u + \mathbf v) - Q_{\mathbf A}(\mathbf u) - Q_{\mathbf A}(\mathbf v) \\ &= (\mathbf u + \mathbf v)^{\mathrm T} \mathbf A (\mathbf u + \mathbf v) - \mathbf u^{\mathrm T} \mathbf A \mathbf u - \mathbf v^{\mathrm T} \mathbf A \mathbf v \\ &= (\mathbf u^{\mathrm T} + \mathbf v^{\mathrm T}) (\mathbf A \mathbf u + \mathbf A \mathbf v) - \mathbf u^{\mathrm T} \mathbf A \mathbf u - \mathbf v^{\mathrm T} \mathbf A \mathbf v \\ &= \mathbf u^{\mathrm T}\mathbf A \mathbf u + \mathbf u^{\mathrm T}\mathbf A \mathbf v + \mathbf v^{\mathrm T}\mathbf A \mathbf u + \mathbf v^{\mathrm T}\mathbf A \mathbf v - \mathbf u^{\mathrm T} \mathbf A \mathbf u - \mathbf v^{\mathrm T} \mathbf A \mathbf v \\ &= \mathbf u^{\mathrm T}\mathbf A \mathbf v + \mathbf v^{\mathrm T}\mathbf A \mathbf u \\ &= \mathbf u^{\mathrm T}\mathbf A \mathbf v + (\mathbf v^{\mathrm T}\mathbf A \mathbf u)^{\mathrm T} \\ &= \mathbf u^{\mathrm T}\mathbf A \mathbf v + \mathbf u^{\mathrm T} \mathbf A^{\mathrm T}\mathbf v \\ &= \mathbf u^{\mathrm T}(\mathbf A + \mathbf A^{\mathrm T})\mathbf v\end{aligned}$

Therefore, by bookkeeping, $[ \cdot , \cdot ]_{\mathbf A}$ is bilinear. For the symmetric matrix claim, we first note that for any $\mathbf v$ ,

$\begin{aligned} Q_{\mathbf A}(\mathbf v) &= [\mathbf v, \mathbf v]_{\mathbf A} = \mathbf v^{\mathrm T} \left( \frac 12 (\mathbf A + \mathbf A^{\mathrm T}) \right) \mathbf v = \mathbf v^{\mathrm T} \mathbf B \mathbf v = Q_{\mathbf B}(\mathbf v), \end{aligned}$

where $\mathbf B := \frac 12 (\mathbf A + \mathbf A^{\mathrm T})$ is a desired symmetric matrix. Furthermore,

$[\mathbf e_i, \mathbf e_j]_{\mathbf A} = a_{ij} + a_{ji},$

so that $[\mathbf e_i, \mathbf e_j]_{\mathbf B} = 2b_{ij}$ . Finally, if there exists another symmetric matrix $\mathbf C$ such that $Q_{\mathbf A} = Q_{\mathbf C}$ , then

$b_{ij} = \frac 12 [\mathbf e_i, \mathbf e_j]_{\mathbf B} = \frac 12 [\mathbf e_i, \mathbf e_j]_{\mathbf C} = c_{ij},$

so that $\mathbf B = \mathbf C$ , as required.

This means that any symmetric matrix defines a quadratic form. It turns out that the reverse happens to be true too: any quadratic form can be defined in terms of a symmetric matrix $\mathbf A$ .

Lemma 3. For any quadratic form $Q : \mathbb K^n \to \mathbb K$ , there exists a unique symmetric matrix $\mathbf A$ such that $Q = Q_{\mathbf A}$ .

Proof. Since $Q$ is a quadratic form, the map $[\cdot, \cdot] : V \times V \to \mathbb K$ defined by

$[\mathbf u,\mathbf v] := \frac 12 (Q(\mathbf u + \mathbf v) - Q(\mathbf u) - Q(\mathbf v))$

is symmetric and bilinear. Define $\mathbf A = [a_{ij}]$ by $a_{ij} = [\mathbf e_i, \mathbf e_j]$ so that $[\mathbf u, \mathbf v] = \mathbf u^{\mathrm T}\mathbf A \mathbf v$ . Then $Q(\mathbf v) = [\mathbf v, \mathbf v] = \mathbf v^{\mathrm T} \mathbf A \mathbf v = Q_{\mathbf A}(\mathbf v)$ .

In fact, more is true.

Lemma 4. Let $V$ be any vector space over $\mathbb K$ . Then any symmetric and bilinear form $[\cdot, \cdot] : V \times V \to \mathbb K$ induces a unique quadratic form $Q : V \to \mathbb K$ .

Proof. Define $Q : V \to \mathbb K$ by $Q(\mathbf v) = [\mathbf v, \mathbf v]$ . By the symmetry and bilinearity of $[\cdot, \cdot]$ ,

$\begin{aligned} Q(\mathbf u + \mathbf v) &= [\mathbf u + \mathbf v, \mathbf u + \mathbf v] \\ &= [\mathbf u , \mathbf u] + [\mathbf u , \mathbf v] + [\mathbf v , \mathbf u] + [\mathbf v , \mathbf v] \\ &= Q(\mathbf u) + Q(\mathbf v) + 2 [\mathbf u, \mathbf v], \end{aligned}$

as required.

This leads us to a rather fascinating discussion on rational trigonometry, which aims to discuss trigonometry and geometry on a much more computationally tractable foundation, introduced by UNSW Professor Norman Wildberger.

For the rest of this post, suppose $[\cdot, \cdot]$ is a symmetric and bilinear form on $V$ . If $\mathbb K = \mathbb R$ , then any inner product is a symmetric and bilinear form on $V$ .

Definition 2. Define the quadrance of $\mathbf v \in V$ by the quadratic form $Q(\mathbf v) = [\mathbf v,\mathbf v]$ . For vectors $\mathbf u, \mathbf v \in V$ with nonzero quadrance, define the cross of the vectors by

$\displaystyle c(\mathbf u, \mathbf v) := \frac{[\mathbf u, \mathbf v]^2}{Q(\mathbf u) Q(\mathbf v)}.$

Finally, define the spread of the vectors by $s := 1 - c$ .

Remark 1. Three nonzero vectors $\mathbf u$ , $\mathbf v$ , $\mathbf w$ form a triangle precisely when $\mathbf u + \mathbf v + \mathbf w = \mathbf 0$ . Subsequent discussions apply to that of a triangle since $Q(-\mathbf w) = Q(\mathbf w)$ .

Theorem 1 (Cross Law). For $\mathbf u, \mathbf v \in V$ ,

$(Q(\mathbf u) + Q(\mathbf v) - Q(\mathbf u + \mathbf v))^2 = 4 Q(\mathbf u) Q(\mathbf v) c(\mathbf u, \mathbf v).$

Proof. The left-hand side simplifies to $4[\mathbf u, \mathbf v]^2$ .

Theorem 2 (Pythagoras’ Theorem). For $\mathbf u, \mathbf v \in V$ , we have $[\mathbf u, \mathbf v] = 0$ if and only if

$Q(\mathbf u + \mathbf v) = Q(\mathbf u) + Q(\mathbf v).$

Theorem 3 (Triple Quad Formula). For $\mathbf u, \mathbf v \in V$ , we have $s(\mathbf u, \mathbf v) = 0$ if and only if

$(Q(\mathbf u) + Q(\mathbf v) - Q(\mathbf u + \mathbf v))^2 = 4 Q(\mathbf u) Q(\mathbf v).$

Equivalently,

$(Q(\mathbf u) + Q(\mathbf v) + Q(\mathbf u + \mathbf v))^2 = 2(Q(\mathbf u)^2 + Q(\mathbf v)^2 + Q(\mathbf u + \mathbf v)^2).$

Proof. The result is immediate from the equivalence $s = 1 - c$ . For the equivalent statement, denote $Q_1 = Q(\mathbf u)$ , $Q_2 = Q(\mathbf v)$ , $Q_3 = Q(\mathbf u + \mathbf v)$ . Then

$\begin{aligned} (Q_1 + Q_2 + Q_3)^2 &= (Q_1 + Q_2 - Q_3 + 2 Q_3)^2 \\ &= (Q_1 + Q_2 - Q_3)^2 + 2(Q_1 + Q_2 - Q_3)\cdot 2 Q_3 + 4Q_3^2 \\ &= 4Q_1 Q_2 + 2(Q_1 + Q_2 - Q_3)\cdot 2 Q_3 + 4Q_3^2 \\ &= 2 \cdot 2(Q_1 Q_2 + Q_1Q_3 + Q_2Q_3) \\ &= 2 \cdot ((Q_1 + Q_2 + Q_3)^2 - (Q_1^2 + Q_2^2 + Q_3^2)) \\ &= 2 (Q_1 + Q_2 + Q_3)^2 - 2 (Q_1^2 + Q_2^2 + Q_3^2), \end{aligned}$

from which algebra yields the desired result.

Theorem 4 (Spread Law). For $\mathbf u, \mathbf v \in V$ , suppose $\mathbf u, \mathbf v, \mathbf u +\mathbf v$ have nonzero quadrances $Q_1, Q_2,Q_3$ respectively. Define

$\begin{aligned} s_1 &:= s(\mathbf v, \mathbf u + \mathbf v) ,\quad s_2 := s(\mathbf u, \mathbf u + \mathbf v), \quad s_3 := s(\mathbf u, \mathbf v).\end{aligned}$

Then $\displaystyle \frac{s_1}{Q_1} = \frac{s_2}{Q_2} = \frac{s_3}{Q_3}$ .

Proof. By the Cross law,

$\begin{aligned} (Q_1 + Q_2 - Q_3)^2 &= 4 Q_1 Q_2 (1 - s_3). \end{aligned}$

Recalling the proof of Theorem 3,

$\begin{aligned} (Q_1 + Q_2 + Q_3)^2 &= (Q_1 + Q_2 - Q_3 + 2 Q_3)^2 \\ &= (Q_1 + Q_2 - Q_3)^2 + 2(Q_1 + Q_2 - Q_3)\cdot 2 Q_3 + 4Q_3^2 \\ &= 4Q_1 Q_2 + 2(Q_1 + Q_2 - Q_3)\cdot 2 Q_3 + 4Q_3^2 - 4 Q_1 Q_2 s_3 \\ &= 2 (Q_1 + Q_2 + Q_3)^2 - 2 (Q_1^2 + Q_2^2 + Q_3^2) - 4 Q_1 Q_2 s_3, \end{aligned}$

so that $4 Q_1 Q_2 s_3 = (Q_1 + Q_2 + Q_3)^2-2 (Q_1^2 + Q_2^2 + Q_3^2)$ . By the symmetry of the right-hand side,

$4 Q_1 Q_2 s_3 = 4 Q_2 Q_3 s_1 = 4 Q_3 Q_1 s_2.$

Dividing by $4Q_1 Q_2 Q_3$ on all sides,

$\displaystyle \frac{s_1}{Q_1} = \frac{s_2}{Q_2} = \frac{s_3}{Q_3} =: \frac 1D.$

Theorem 5 (Triple Spread Formula). Following the notation in Theorem 4,

$\begin{aligned} (s_1 + s_2 + s_3)^2 = 2(s_1^2 + s_2^2 + s_2^2) + 4 s_1 s_2 s_3.\end{aligned}$

Proof. For each $i$ , $Q_i = D s_i$ . Substituting into the equivalent of the Cross law,

$4 Ds_1 Ds_2 s_3 = (Ds_1 + Ds_2 + Ds_3)^2- 2 (D^2 s_1^2 + D^2 s_2^2 + D^2 s_3^2) .$

Dividing by $D^2$ yields

$4 s_1 s_2 s_3 = (s_1 + s_2 + s_3)^2-2 ( s_1^2 + s_2^2 + s_3^2) ,$

as required.

For more fascinating details about rational trigonometry, check out Prof Wildberger’s YouTube channel. In particular, we want to use rational trigonometry to create geometry in the next post.

—Joel Kindiak, 20 Mar 25, 1024H
September 15, 2025
The Spectral Theorems

Let $V$ be a finite-dimensional inner product space over $\mathbb K$ , and $T : V \to V$ be a linear transformation (or if you’d prefer, linear operator on $V$ ). Recall that $T$ is diagonalisable if and only if there exists a linearly independent set $K$ of eigenvectors of $T$ such that $\mathrm{span}(K) = V$ .

Not all operators are diagonalisable (though all operators over $\mathbb C$ will have a Jordan normal form, details here). But diagonalisable operators are convenient since we can compute $T^n$ with relative ease. What’s fascinating, however, is that in the case $\mathbb K = \mathbb C$ , if $T$ is normal, then $T$ is diagonalisable, i.e. there exists a basis $K$ of $V$ such that each vector in $K$ is an eigenvector of $T$ . Something nicer, in fact, happens in this case. It turns out that $K$ must be orthonormal. These two conditions turn out to be equivalent, and proving this surprising fact is our goal for this post.

Lemma 1. Multiplication in the subset of $\mathcal M_{n \times n}(\mathbb K)$ of diagonal matrices is commutative.

Now suppose $\mathbb K = \mathbb R$ or $\mathbb C$ .

Theorem 1. Suppose $c_T$ is a product of linear factors in $\mathbb K$ . $T$ is normal if and only if there exists an orthonormal (ordered) set $K$ of eigenvectors of $T$ such that $\mathrm{span}(K) = V$ .

Proof. We first prove the direction $(\Leftarrow)$ . Suppose there exists an orthonormal set $K$ of eigenvectors of $T$ such that $\mathrm{span}(K) = V$ . Then $T$ is similar to the matrix

$[T]_K := \begin{bmatrix} \lambda_1 & 0 & \cdots & 0 \\ 0 & \ddots & \ddots & \vdots \\ \vdots & \ddots & \ddots & 0 \\ 0 & \cdots & 0 & \lambda_n \end{bmatrix}.$

By definition of the conjugate-transpose,

$[T]_K^* := \begin{bmatrix} \bar{\lambda}_1 & 0 & \cdots & 0 \\ 0 & \ddots & \ddots & \vdots \\ \vdots & \ddots & \ddots & 0 \\ 0 & \cdots & 0 & \bar{\lambda}_n \end{bmatrix}.$

Hence, since both matrices are diagonal, $[T]_K [T]_K^* = [T]_K^* [T]_K$ . We leave it as an exercise to verify that $[T^*]_K = [T]_K^*$ . Hence, $[T \circ T^*]_K = [T^* \circ T]_K$ implies that $T \circ T^* = T^* \circ T$ , i.e. $T$ is normal.

For the direction $(\Rightarrow)$ , we will prove using induction on $\dim(V)$ . The case $n = 1$ is left as an obvious exercise. Assume that $T$ is a normal operator and suppose $\dim(V) = k+1$ . By hypothesis, the characteristic polynomial $c_T$ has at least one complex root $\lambda$ , which means that $T$ has at least one unit eigenvector $\mathbf u$ such that $T(\mathbf v) = \lambda \mathbf u$ .

Define the subspace $W := \mathrm{span}\{\mathbf u\}$ . Since $V$ is finite-dimensional, we can make the direct sum decomposition $V = W \oplus W^\perp$ . By considering dimensions,

$k+1 = \dim(V) = \dim(W) + \dim(W^{\perp}) = 1 + \dim(W^{\perp}),$

so that $\dim(W^{\perp}) = k$ . We claim that $S := T|_{W^{\perp}} : W^{\perp} \to W$ is a normal operator. Firstly, we need to check that $S (W^{\perp}) \subseteq W^{\perp}$ , i.e. that $S$ is $W^{\perp}$ -invariant, so that $S$ is indeed a linear operator. Secondly, we need to prove that $S \circ S^* = S^* \circ S$ , so that $S$ is indeed normal.

For the first claim, fix $\mathbf v \in W^{\perp}$ , so that $\langle \mathbf u, \mathbf v \rangle = 0$ . We need to check that $S(\mathbf v) \in W^{\perp}$ . Since $T$ is normal,

$\langle \mathbf u, S(\mathbf v) \rangle = \langle \mathbf u, T(\mathbf v) \rangle = \langle T^*(\mathbf u), \mathbf v\rangle = \langle \bar{\lambda}\mathbf u, \mathbf v\rangle = \bar{\lambda}\langle \mathbf u, \mathbf v\rangle = 0,$

so that we do have $S(\mathbf v) \in W^{\perp}$ , as required.

For the second claim, we first observe that for any $\mathbf v, \mathbf w \in W^{\perp}$ ,

$\langle \mathbf v, S^*(\mathbf w) \rangle = \langle S(\mathbf v), \mathbf w) = \langle T(\mathbf v), \mathbf w)\rangle = \langle \mathbf v, T^*(\mathbf w) \rangle.$

Hence, $S^*|_{W^{\perp}} = T^*|_{W^{\perp}}$ . By a similar argument, as before, for any $\mathbf v \in W^{\perp}$ ,

$\langle \mathbf u, S^*(\mathbf v) \rangle = \langle \mathbf u, T^*(\mathbf v) \rangle = \lambda \langle \mathbf u, \mathbf v \rangle = 0,$

so that $S^*$ is $W^{\perp}$ -invariant. Hence,

$S \circ S^* = T|_{W^{\perp}} \circ T^* |_{W^{\perp}} = (T \circ T^*)|_{W^{\perp}}.$

Therefore, $S$ is normal since $T$ is normal. Similarly, $c_S$ is a product of linear factors in $\mathbb K$ . By the induction hypothesis, there exists an orthonormal basis $K$ of eigenvectors of $S$ for $W^{\perp}$ . Then $\{\mathbf u\} \cup K$ forms an orthonormal basis for $W \oplus W^{\perp} = V$ , as required.

Corollary 1. $T$ is normal if and only if there exists an orthonormal basis $K$ of $V$ consisting of eigenvectors of $T$ such that $[T]_K$ is diagonal.

Corollary 2. The square matrix $\mathbf A : \mathbb C^n \to \mathbb C^n$ is normal if and only if there exists a unitary matrix $\mathbf P$ such that $\mathbf P^* \mathbf A \mathbf P$ is diagonal.

Theorem 2. Suppose $\mathbb K = \mathbb R$ . $T$ is self-adjoint if and only if there exists an orthonormal (ordered) set $K$ of eigenvectors of $T$ such that $\mathrm{span}(K) = V$ .

Proof. For the direction $(\Leftarrow)$ , the proof in Theorem 1 yields that

$[T^*]_K = [T]_K^* = [T]_K^{\mathrm T} = [T]_K,$

so that $T^* = T$ , as required.

For the direction $(\Rightarrow)$ , suppose $T$ is self-adjoint. Since $T$ is normal, by Theorem 1, it suffices to check that $c_T$ is a product of linear factors in $\lambda \in \mathbb R$ . Let $\lambda \in \mathbb C$ be any root of $c_T$ . Then there exists $\mathbf v \in V$ such that

$\bar{\lambda}\mathbf v =T^*(\mathbf v) = T(\mathbf v) = \lambda \mathbf v.$

Since $\mathbf v \neq \mathbf 0$ , we must have $\lambda = \bar{\lambda}$ , yielding $\lambda \in \mathbb R$ . Hence, $c_T$ can be written as a product of linear factors, as required.

Corollary 3. The square matrix $\mathbf A : \mathbb R^n \to \mathbb R^n$ is symmetric if and only if there exists an orthogonal matrix $\mathbf P$ such that $\mathbf P^{\mathrm T} \mathbf A \mathbf P$ is diagonal.

These climactic results are known as the spectral theorems, and we will revisit them in the following crucial result: the singular value decomposition. These results cap off a theoretical study in undergraduate linear algebra.

In subsequent posts, we will explore various applications of what we have learned in other areas of mathematics.

—Joel Kindiak, 18 Mar 25, 1812H

September 8, 2025
Angle-Preserving Transformations
Recall that for any field $\mathbb K$ , to construct the complex numbers $\mathbb C_{\mathbb K}$ , we needed to define $i = \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix} =: \sqrt{-1}$ . Geometrically, this means for any $u_1,u_2 \in \mathbb K$ , $i \begin{bmatrix} u_1 \\ u_2\end{bmatrix} = \begin{bmatrix} u_2 \\ -u_1 \end{bmatrix}$ .

Something curious happens in the case $\mathbb K = \mathbb R$ . For any $\mathbf u := \begin{bmatrix} u_1 \\ u_2 \end{bmatrix}, \mathbf v := \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} \in \mathbb R^2$ , we have

$\begin{aligned} \langle i \mathbf u, i \mathbf v\rangle &= \left\langle \begin{bmatrix} -u_2 \\ u_1 \end{bmatrix}, \begin{bmatrix} -v_2 \\ v_1 \end{bmatrix} \right\rangle \\ &= (-u_2)(-v_2) + u_1 v_1 \\ &= u_1 v_1 + u_2 v_2 \\ &= \left\langle \begin{bmatrix} u_1 \\ u_2 \end{bmatrix}, \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} \right\rangle = \langle \mathbf u, \mathbf v\rangle. \end{aligned}$

In other words, $i : \mathbb R^2 \to \mathbb R^2$ preserves angles. We generalise to other linear transformations.

Let $V$ be an inner product space over $\mathbb K$ , where $\mathbb K$ equals $\mathbb R$ or $\mathbb C$ .

Definition 1. We say that a linear transformation $T : V \to V$ , also known as a linear operator on $V$ , is:
- angle-preserving if for any $\mathbf u, \mathbf v \in V$ , $\langle T(\mathbf u), T(\mathbf v) \rangle = \langle \mathbf u, \mathbf v \rangle$ ,
- norm-preserving if for any $\mathbf v \in V$ , $\|T(\mathbf v)\| = \|\mathbf v\|$ .
It is clear that if $T$ is angle-preserving, then it is norm-preserving. Does that hold in the opposite direction? More is true, in fact, as long as $V$ has an orthonormal basis.

Theorem 1. Suppose $V$ has an orthonormal basis $\{\mathbf e_\alpha : \alpha \in I\}$ . The following are equivalent:
- $T$ is angle-preserving,
- $T$ is norm-preserving,
- $\{T (\mathbf e_\alpha) : \alpha \in I\}$ is orthonormal,
- $T^* = T^{-1}$ , i.e. $T$ is unitary.
Proof. Suppose $T$ is norm preserving. Fix $\gamma \in \mathbb K$ to be tuned. Consider the expression

$\| T(\mathbf e_\alpha + \gamma \mathbf e_\beta)\|^2 = \| \mathbf e_\alpha + \gamma \mathbf e_\beta\|^2.$

Writing the norm as an inner product, the left-hand simplifies to

$\begin{aligned} \|T(\mathbf e_\alpha)\|^2 + \bar{\gamma} \langle T(\mathbf e_\alpha), T(\mathbf e_\beta) \rangle +\gamma \overline{\langle T(\mathbf e_\alpha), T(\mathbf e_\beta) \rangle} + |\gamma|^2 \|T(\mathbf e_\beta)\|^2. \end{aligned}$

Since $\langle \mathbf e_\alpha, \mathbf e_\beta \rangle = 0$ , the right-hand side simplifies to

$\begin{aligned} \|\mathbf e_\alpha\|^2 + |\gamma|^2 \|\mathbf e_\beta \|^2. \end{aligned}$

Since $T$ is norm-preserving, equating the expressions yields

$\bar{\gamma} \langle T(\mathbf e_\alpha), T(\mathbf e_\beta) \rangle +\gamma \overline{\langle T(\mathbf e_\alpha), T(\mathbf e_\beta) \rangle} = 0.$

By algebra, setting $\gamma = \langle T(\mathbf e_\alpha), T(\mathbf e_\beta) \rangle$ and simplify

$| \langle T(\mathbf e_\alpha), T(\mathbf e_\beta) \rangle |^2 = |\gamma|^2 = 0$

so that $\langle T(\mathbf e_\alpha), T(\mathbf e_\beta) \rangle = 0$ , as required. Furthermore,

$\begin{aligned} (T^* \circ T)(\mathbf e_\alpha) &= \sum_{\beta} \langle (T^* \circ T)(\mathbf e_\alpha), \mathbf e_\beta\rangle \mathbf e_\beta \\ &= \sum_{\beta} \langle T^*(T(\mathbf e_\alpha)), \mathbf e_\beta\rangle \mathbf e_\beta \\ &= \sum_{\beta} \langle T(\mathbf e_\alpha), T(\mathbf e_\beta) \rangle \mathbf e_\beta \\ &= 1 \cdot \mathbf e_\alpha + \sum_{\alpha \neq \beta} 0 \cdot \mathbf e_\beta = \mathbf e_\alpha. \end{aligned}$

Therefore, $T^* = T^{-1}$ . Finally, under this assumption, for any $\mathbf u, \mathbf v \in V$ ,

$\begin{aligned} \langle T(\mathbf u),T(\mathbf v)\rangle &= \langle \mathbf u, T^* ( T(\mathbf v) )\rangle \\ &= \langle \mathbf u, T^{-1} ( T(\mathbf v) ) \rangle \\ &= \langle \mathbf u, \mathbf v \rangle. \end{aligned}$

In particular, $T^* \circ T = I_V = T \circ T^*$ . In this case, we call $T$ a normal operator.

Lemma 1. Unitary operators are normal. Furthermore, if $T^* = T$ (i.e. $T$ is self-adjoint), then $T$ is normal.

Theorem 2. Let $T : V \to V$ be a normal operator. The following propositions hold:
- For any $\mathbf u, \mathbf v$ , $\langle T(\mathbf u),T(\mathbf v)\rangle = \langle T^*(\mathbf u),T^*(\mathbf v)\rangle$ .
- If $T(\mathbf v) = \lambda \mathbf v$ , then $T^*(\mathbf v) = \bar{\lambda} \mathbf v$ .
- If $T(\mathbf u) = \lambda \mathbf u$ and $T(\mathbf v) = \mu \mathbf v$ and $\lambda \neq \mu$ , then $\mathbf u \perp \mathbf v$ .
Proof. If $T \circ T^* = T^* \circ T$ , then for any $\mathbf u, \mathbf v$ ,

$\begin{aligned} \langle T(\mathbf u),T(\mathbf v)\rangle &= \langle \mathbf u,(T^* \circ T)(\mathbf v)\rangle \\ &= \langle \mathbf u,(T \circ T^*)(\mathbf v)\rangle \\ &= \langle T^*(\mathbf u), T^*(\mathbf v)\rangle .\end{aligned}$

We leave it as an exercise to check that $T - \lambda$ is normal. Then

$\begin{aligned} \langle (T^* - \bar\lambda)(\mathbf v), (T^* - \bar\lambda)(\mathbf v)\rangle &= \langle (T - \lambda)^* (\mathbf v), (T -\lambda)^*(\mathbf v)\rangle \\ &= \langle (T - \lambda) (\mathbf v), (T -\lambda)(\mathbf v)\rangle = 0. \end{aligned}$

Finally,

$\lambda \langle \mathbf u, \mathbf v \rangle = \langle T(\mathbf u), \mathbf v\rangle = \langle \mathbf u, T^*(\mathbf v)\rangle =\mu \langle \mathbf u, \mathbf v \rangle$

yields the desired result since $\lambda - \mu \neq 0$ .

You may now be wondering: what on earth are we doing, and why? We have started from the notion of angle-preserving transformations, and ended up at normal operators—for what? Well, we want to prove a rather elegant result: any real symmetric matrix $\mathbf A$ can be diagonalised! Furthermore, the matrix we use to diagonalise $\mathbf A$ can be comprised of our favorite type of basis—an orthonormal basis.

—Joel Kindiak, 15 Mar 25, 1554H
September 1, 2025
Constructing the Cross Product

Previously, our discussion on the dot product began with a simple question: What is the angle between two vectors? Our study on the cross product shall also begin with a simple question: What is a vector that is perpendicular to two vectors?

One possible approach is to give the answer, and leave it as it is. However, part of this blog’s goal is to journey from the known to the unknown, and in this context, it means properly constructing the cross product. Just like with determinants, we are going to impose several conditions on the cross product, and the result from these conditions will bring us to one and only one definition for the cross product.

Just like determinants, we are going to build the cross product $\times : \mathbb K^3 \times \mathbb K^3 \to \mathbb K^3$ one step at a time. Here, we denote $\mathbf u \times \mathbf v := \times(\mathbf u, \mathbf v)$ for readability. We will let $(\mathbf i, \mathbf j , \mathbf k) = (\mathbf e_1, \mathbf e_2, \mathbf e_3)$ denote the usual standard ordered basis on $\mathbb K^3$ .

The first observation is that, as a basic normalising step, we want $\mathbf k$ to be the vector perpendicular to $\mathbf i$ and $\mathbf j$ . They are all unit vectors (orthonormal, in fact). Actually, both $\pm \mathbf k$ would be perpendicular to $\mathbf i$ and $\mathbf j$ .

Lemma 1. In the case $\mathbb R = \mathbb K$ , we have $\mathbf i \cdot \mathbf k = 0$ , $\mathbf j \cdot \mathbf k = 0$ . Equivalently, $\mathbf i \perp \mathbf k$ and $\mathbf j \perp \mathbf k$ .

Almost by custom, we will define $\mathbf i \times \mathbf j := \mathbf k$ and $\mathbf j \times \mathbf i := -\mathbf i \times \mathbf j = -\mathbf k$ .

Definition 1. Define $\mathbf i \times \mathbf j := \mathbf k$ , $\mathbf j \times \mathbf k := \mathbf i$ , and $\mathbf k \times \mathbf i := \mathbf j$ . For $\mathbf u, \mathbf v \in \{\mathbf i, \mathbf j, \mathbf k \}$ , define $\mathbf u \times \mathbf v := -\mathbf v \times \mathbf u$ whenever $\mathbf u \neq \mathbf v$ .

By Lemma 1, we obtain the property

$(\mathbf u \times \mathbf v) \cdot \mathbf u = (\mathbf u \times \mathbf v) \cdot \mathbf v = \mathbf 0.$

Now, the stipulation $\mathbf u \neq \mathbf v$ is rather arbitrary. If instead we allow $\mathbf u \times \mathbf v := -\mathbf v \times \mathbf u$ for any $\mathbf u, \mathbf v \in \mathbb K^3$ , we actually obtain an alternating map, which yields $\mathbf u \times \mathbf u = \mathbf 0$ .

Theorem 1. For any $\mathbf u \in \mathbb K^3$ , $\mathbf u \times \mathbf u = \mathbf 0$ . In particular,

$\mathbf i \times \mathbf i = \mathbf j \times \mathbf j = \mathbf k \times \mathbf k = \mathbf 0.$

Next, and rather understandably, we would like $\times$ to be multi-linear (since there are two arguments, we say that $\times$ is bi-linear). It turns out that we are forced into one formula for $\mathbf u \times \mathbf v$ .

Theorem 2. Suppose furthermore that $\times$ is linear in both arguments. Then for any $\mathbf u = \begin{bmatrix} u_1 \\ u_2 \\ u_3 \end{bmatrix} , \mathbf v = \begin{bmatrix} v_1 \\ v_2 \\ v_3 \end{bmatrix} \in \mathbb K^3$ ,

$\mathbf u \times \mathbf v = \begin{bmatrix} u_2 v_3 - u_3 v_2 \\ -(u_1 v_3 - u_3 v_1) \\ u_1 v_2 - u_2 v_1 \end{bmatrix}.$

Proof. We first exploit the linearity of $\mathbf e_i \times (\cdot)$ . Writing $\mathbf v = v_1 \mathbf e_1 + v_2 \mathbf e_2 + v_3 \mathbf e_3$ ,

$\begin{aligned}\mathbf e_i \times \mathbf v &= \mathbf e_i \times (v_1 \mathbf e_1 + v_2 \mathbf e_2 + v_3 \mathbf e_3) \\ &= v_1 \mathbf e_i \times \mathbf e_1 + v_2 \mathbf e_i \times \mathbf e_2 + v_3 \mathbf e_i \times \mathbf e_3 \\ &= \sum_{n \neq i} v_n \mathbf e_i \times \mathbf e_n.\end{aligned}$

In particular,

$\mathbf e_1 \times \mathbf v = \begin{bmatrix} 0 \\ -v_3 \\ v_2\end{bmatrix}, \quad \mathbf e_2 \times \mathbf v = \begin{bmatrix} v_3 \\ 0 \\ -v_1\end{bmatrix}, \quad \mathbf e_3 \times \mathbf v = \begin{bmatrix} -v_2 \\ v_1 \\ 0\end{bmatrix}.$

Hence, exploiting the linearity of $(\cdot)\times \mathbf v$ , writing $\mathbf u = u_1 \mathbf e_1 + u_2 \mathbf e_2 + u_3 \mathbf e_3$ ,

$\begin{aligned} \mathbf u \times \mathbf v &= (u_1 \mathbf e_1 + u_2 \mathbf e_2 + u_3 \mathbf e_3) \times \mathbf v \\ &= u_1 \mathbf e_1 \times \mathbf v + u_2 \mathbf e_2 \times \mathbf v + u_3 \mathbf e_3 \times \mathbf v \\ &= u_1 \begin{bmatrix} 0 \\ -v_3 \\ v_2\end{bmatrix} + u_2 \begin{bmatrix} v_3 \\ 0 \\ -v_1\end{bmatrix} + u_3 \begin{bmatrix} -v_2 \\ v_1 \\ 0\end{bmatrix} \\ &= \begin{bmatrix} u_2 v_3 - u_3 v_2 \\ -(u_1 v_3 - u_3 v_1) \\ u_1 v_2 - u_2 v_1 \end{bmatrix}. \end{aligned}$

Corollary 1. For vectors $\mathbf u, \mathbf v, \mathbf w \in \mathbb K^3$ , $(\mathbf u \times \mathbf v) \cdot \mathbf w = \det(\mathbf u, \mathbf v, \mathbf w)$ and identifying

$\mathbf u \times \mathbf v \equiv ( \mathbf u \times \mathbf v )^{\mathrm T} = (\mathbf u \times \mathbf v) \cdot (\cdot ) = \det(\mathbf u, \mathbf v, \cdot)$ ,

we have

$\mathbf u \times \mathbf v \equiv \det(\mathbf u, \mathbf v, \cdot) =: \left|\begin{matrix} \mathbf i & \mathbf j & \mathbf k \\ u_1 & u_2 & u_3 \\ v_1 & v_2 & v_3 \end{matrix}\right|.$

Proof. By expanding each component,

$\mathbf u \times \mathbf v = \left| \begin{matrix} u_2 & u_3 \\ v_2 & v_3 \end{matrix} \right| \mathbf i - \left| \begin{matrix} u_1 & u_3 \\ v_1 & v_3 \end{matrix} \right| \mathbf j + \left| \begin{matrix} u_1 & u_2 \\ v_1 & v_2 \end{matrix} \right| \mathbf k.$

Hence, taking the dot product with $\mathbf w = w_1 \mathbf e_1 + w_2 \mathbf e_2 + w_3 \mathbf w_3$ ,

$\begin{aligned} (\mathbf u \times \mathbf v) \cdot \mathbf w &= \left| \begin{matrix} u_2 & u_3 \\ v_2 & v_3 \end{matrix} \right| w_1 - \left| \begin{matrix} u_1 & u_3 \\ v_1 & v_3 \end{matrix} \right| w_2 + \left| \begin{matrix} u_1 & u_2 \\ v_1 & v_2 \end{matrix} \right| w_3 \\ &= \left|\begin{matrix} w_1 & w_2 & w_3 \\ u_1 & u_2 & u_3 \\ v_1 & v_2 & v_3 \end{matrix}\right| = \det(\mathbf w, \mathbf u, \mathbf v) = -\det(\mathbf u, \mathbf w, \mathbf v) \\ &= -(-\det(\mathbf u, \mathbf v, \mathbf w)) = \det(\mathbf u, \mathbf v, \mathbf w). \end{aligned}$

Interestingly however, if we began with this definition of the cross product, then we will recover all of our previous properties.

Corollary 2. For any $\mathbf u,\mathbf v \in \mathbb K^3$ , define

$\mathbf u \times \mathbf v = \begin{bmatrix} u_2 v_3 - u_3 v_2 \\ -(u_1 v_3 - u_3 v_1) \\ u_1 v_2 - u_2 v_1 \end{bmatrix} \equiv \det(\mathbf u, \mathbf v, \cdot).$

Then we recover the hypotheses in Definition 1, Theorem 1 and Theorem 2.

Proof. Since $\det$ is alternative, $(\mathbf u, \mathbf v) \in \{\mathbf i, \mathbf j , \mathbf k\}^2$ recovers our hypotheses in Definition 1 and Theorem 1. Since the determinant is multilinear, we recover Theorem 2.

Using the usual dot product (which still works as a computation in $\mathbb K^3$ , barring some technical interpretive caveats), we have that $\mathbf u \times \mathbf v$ is orthogonal to both $\mathbf u$ and $\mathbf v$ .

Corollary 3. For vectors $\mathbf u, \mathbf v, \mathbf w \in \mathbb K^3$ ,

$(\mathbf u \times \mathbf v) \cdot \mathbf u = (\mathbf u \times \mathbf v) \cdot \mathbf v = \mathbf 0.$

Proof. The results follow since the determinant is alternative.

Now, the discussion on angles only really makes sense when $\mathbb K = \mathbb R$ , in that

$\mathbf u \cdot \mathbf v = 0 \quad \iff \quad \mathbf u \perp \mathbf v,$

so let’s zoom in on this situation. We observe that for any $k \in \mathbb R$ , $k(\mathbf u \times \mathbf v) \perp \mathbf u$ . So what is it about $\mathbf u \times \mathbf v$ that makes it special? Perhaps $\|\mathbf u \times \mathbf v \|$ is intimately connected with the “degree” of separation, no pun intended, between $\mathbf u$ and $\mathbf v$ . Obviously, due to linearity, $\| \mathbf u \times \mathbf v\| = 0$ whenever either $\mathbf u$ or $\mathbf v$ are the zero vector $\mathbf 0$ .

Assume therefore that $\mathbf u, \mathbf v$ are nonzero vectors. Due to bilinearity, if we can decode the case for $\| \mathbf u \| = \| \mathbf v \| = 1$ , then using the property that $\mathbf u = \|\mathbf u \| \hat{\mathbf u}$ , we obtain

$\| \mathbf u \times \mathbf v\| = \| (\|\mathbf u \| \hat{\mathbf u}) \times (\|\mathbf v \| \hat{\mathbf v})\| = \| \mathbf u \| \| \mathbf v \| \| \hat{\mathbf u} \times \hat{\mathbf v}\|.$

Therefore, let’s work with the simplified case $\| \mathbf u \| = \| \mathbf v \| = 1$ . At this point, we could bash the algebra out. However, taking inspiration from this thread, we will adopt a much cleaner approach. Define the matrix

$\mathbf A = \begin{bmatrix} \mathbf u & \mathbf v & \mathbf u \times \mathbf v \end{bmatrix}.$

Then

$\begin{aligned} \det(\mathbf A^{\mathrm T} \mathbf A) &= \det(\mathbf A^{\mathrm T}) \det(\mathbf A) = (\det \mathbf A)^2. \end{aligned}$

On the left-hand side,

$\begin{aligned} \det(\mathbf A^{\mathrm T} \mathbf A) &= \left| \begin{matrix} \mathbf u^{\mathrm T} \mathbf u & \mathbf u^{\mathrm T} \mathbf v & \mathbf u^{\mathrm T} (\mathbf u \times \mathbf v) \\ \mathbf v^{\mathrm T} \mathbf u & \mathbf v^{\mathrm T} \mathbf v & \mathbf v^{\mathrm T} (\mathbf u \times \mathbf v) \\ (\mathbf u \times \mathbf v)^{\mathrm T} \mathbf u & (\mathbf u \times \mathbf v)^{\mathrm T} \mathbf v & (\mathbf u \times \mathbf v)^{\mathrm T} (\mathbf u \times \mathbf v) \end{matrix} \right| \\ &= \left| \begin{matrix} \| \mathbf u \|^2 & \mathbf u \cdot \mathbf v & 0 \\ \mathbf u \cdot \mathbf v & \| \mathbf v \|^2 & 0 \\ 0 & 0 & \| \mathbf u \times \mathbf v \|^2 \end{matrix} \right| \\ &= \| \mathbf u \times \mathbf v \|^2\left| \begin{matrix} \| \mathbf u \|^2 & \mathbf u \cdot \mathbf v \\ \mathbf u \cdot \mathbf v & \| \mathbf v \|^2 \end{matrix} \right| \\ &= \| \mathbf u \times \mathbf v \|^2 (\|\mathbf u\|^2 \|\mathbf v\|^2 - (\mathbf u \cdot \mathbf v)^2).\end{aligned}$

On the right-hand side,

$(\det \mathbf A)^2 = (\det(\mathbf u, \mathbf v, \mathbf u \times \mathbf v))^2 = ((\mathbf u \times \mathbf v) \cdot (\mathbf u \times \mathbf v))^2 = \|\mathbf u \times \mathbf v\|^4.$

Therefore, dividing by $\|\mathbf u \times \mathbf v\|^2$ on both sides,

$\begin{aligned} \|\mathbf u \times \mathbf v\|^2 &= \|\mathbf u\|^2 \|\mathbf v\|^2 - (\mathbf u \cdot \mathbf v)^2 \\ &= 1^2 \cdot 1^2 - (1 \cdot 1 \cdot \cos \theta)^2 \\ &= 1 - \cos^2 (\theta) = \sin^2(\theta). \end{aligned}$

This equation can only mean one thing: $\| \mathbf u \times \mathbf v \| = \sin(\theta)$ , where we don’t even need to worry about $| \cdot |$ on the right-hand side because $\sin(\theta) \geq 0$ whenever $\theta \in [0, \pi]$ . This yields a rather elegant geometric interpretation for the cross product:

Corollary 4. For $\mathbf u, \mathbf v \in \mathbb R^3$ with angle $\theta$ between them as computed by the dot product (and consider $\theta$ undefined when either $\mathbf u$ or $\mathbf v$ are the zero vector $\mathbf 0$ ),

$\|\mathbf u \times \mathbf v\| = \|\mathbf u\|\|\mathbf v \|\sin(\theta).$

Letting $\mathbf n$ denote the unit vector of $\mathbf u \times \mathbf v$ , we recover the classic cross product formula:

$\mathbf u \times \mathbf v = \|\mathbf u\|\|\mathbf v \|\sin(\theta) \mathbf n.$

—Joel Kindiak, 13 Mar 25, 2236H

August 25, 2025
We Finally Discuss Transposes
Previously, we wanted to find a best approximation to the usually unsolvable equation $\mathbf A\mathbf x =\mathbf b$ . Using the idea of projections, we know that $\mathbf u$ will give us the best approximation if and only if

$\mathbf A\mathbf u = \mathrm{proj}_{\mathbf A(\mathbb K^n)} (\mathbf b).$

With a bit more work, we show that this equation holds if and only if

$\langle \mathbf A \mathbf e_j, \mathbf A \mathbf u \rangle = \langle \mathbf A \mathbf e_j, \mathbf b \rangle.$

But where do we go from here?

Now, let’s recall the usual inner product on $\mathbb K^n$ . Given $\mathbf u, \mathbf v \in \mathbb K^n$ , the usual inner product is defined by

$\displaystyle \langle \mathbf u, \mathbf v \rangle := \sum_{j=1}^n u_i \bar{v}_i = \begin{bmatrix} \bar{v}_1 & \cdots & \bar{v}_n \end{bmatrix} \mathbf u.$

Thus, we have the definition $\mathbf v^* :=\begin{bmatrix} \bar{v}_1 & \cdots & \bar{v}_n \end{bmatrix}$ if and only if

$\mathbf v^* := \langle \cdot ,\mathbf v \rangle : \mathbb K^n \to \mathbb K.$

In this case, $\langle \mathbf u, \mathbf v \rangle = \mathbf v^* \mathbf u$ . To adapt to more general settings, we adopt the left-hand definition, called an adjoint.

Henceforth, let $V$ be an inner product space over a field $\mathbb K$ .

Definition 1. For any $\mathbf v \in V$ , define its adjoint by $\mathbf v^* := \langle \cdot, \mathbf v\rangle \in \mathcal L(V, \mathbb K) =: V^*$ .

Given $\mathbf A : \mathbb K^n \to \mathbb K^m$ , a reasonable definition for $\mathbf A^*$ would be

$\mathbf A^* := \begin{bmatrix} (\mathbf A \mathbf e_1)^* \\ \vdots \\ (\mathbf A \mathbf e_n)^* \end{bmatrix} : \mathbb K^m \to \mathbb K^n.$

In particular, by writing in terms of the usual standard basis,

$\displaystyle \mathbf A^* = \sum_{j=1}^n \langle \cdot , \mathbf A \mathbf e_j \rangle \mathbf e_j : \mathbb K^m \to \mathbb K^n.$

where we adopted the inner product notation for aesthetic purposes. When $\mathbb K = \mathbb R$ , we denote $\mathbf A^{\mathrm T} = \mathbf A^*$ .

Theorem 1. We have $[a_{ij}]^* = [\bar{a}_{ji}]$ . If $\mathbb K = \mathbb R$ , then $[a_{ij}]^* = [a_{ji}]$ . Thus, we call $\mathbf A^*$ the conjugate transpose of $\mathbf A$ , and $\mathbf A^{\mathrm T}$ the transpose of $\mathbf A$ .

Proof. Let $\mathbf A = \begin{bmatrix} [a_{1,i}] & \cdots & [a_{n,i}] \end{bmatrix}$ . By definition,

$\displaystyle (\mathbf A^*(\mathbf e_i))(j) = \sum_{k=1}^n \langle \mathbf e_i , \mathbf A \mathbf e_k \rangle \mathbf e_k(j) = \langle \mathbf e_i , \mathbf A\mathbf e_j \rangle = \bar{a}_{ji}.$

To generalise this idea to the linear transformation $T : V \to W$ between two inner product spaces would take considerably much more effort, but there are instances where we might be able to make a sufficiently reasonable definition.

In particular, if $B := \{\mathbf e_\alpha : \alpha \in I\}$ and $C := \{\mathbf f_\beta : \beta \in J\}$ form bases for $V$ and $W$ respectively (this is certainly true in the finite-dimensional setting), then assuming that $B$ is finite, we can make the definition

$\displaystyle T^* := \sum_{\alpha \in I} \langle \cdot , T(\mathbf e_\alpha) \rangle \mathbf e_\alpha : W \to V,$

where the sum of the right-hand side is finite and thus well-defined. In particular, for each $\beta \in J$ ,

$\displaystyle T^*(\mathbf f_\beta) := \sum_{\alpha \in I} \langle \mathbf f_\beta , T(\mathbf e_\alpha) \rangle \mathbf e_\alpha.$

If furthermore that $B$ is orthonormal, then applying $\langle \mathbf e_\gamma, \cdot \rangle$ on both sides,

$\displaystyle \begin{aligned} \langle \mathbf e_\gamma, T^*(\mathbf f_\beta) \rangle &= \left\langle \mathbf e_\gamma, \sum_{\alpha \in I} \langle \mathbf f_\beta , T(\mathbf e_\alpha) \rangle \mathbf e_\alpha \right\rangle \\ &= \sum_{\alpha \in I} \overline{\langle \mathbf f_\beta , T(\mathbf e_\alpha) \rangle} \langle \mathbf e_\gamma, \mathbf e_\alpha \rangle \\ &= \sum_{\alpha \in I} \langle T(\mathbf e_\alpha), \mathbf f_\beta \rangle \langle \mathbf e_\gamma, \mathbf e_\alpha \rangle \\ &= \langle T(\mathbf e_\gamma), \mathbf f_\beta \rangle + \sum_{\alpha \neq \gamma} \langle T(\mathbf e_\alpha), \mathbf f_\beta \rangle \cdot 0 \\ &= \langle T(\mathbf e_\gamma), \mathbf f_\beta \rangle. \end{aligned}$

Extending by linearity, we obtain the crucial identity

$\langle T(\mathbf v), \mathbf w \rangle = \langle \mathbf v, T^*(\mathbf w)\rangle$

that serves as a working definition of $T^*$ , which we have properly constructed at least in the case $V$ is finite-dimensional.

Let $V$ be an inner product space over $\mathbb K$ .

Lemma 1. Let $T : V \to W$ be a linear transformation such that there exists a linear transformation $T^* : W \to V$ that satisfies the crucial property

$\langle T(\mathbf v), \mathbf w \rangle = \langle \mathbf v, T^*(\mathbf w)\rangle,\quad \mathbf v \in V,\quad \mathbf w \in W.$

Then $T^*$ is unique. We call $T^*$ the adjoint of $T$ .

Proof. We first remark that for any $\mathbf v \in V$ , $\langle \cdot ,\mathbf v \rangle = O_V$ implies

$\langle \mathbf v ,\mathbf v \rangle = O_V(\mathbf v) = 0 \quad \Rightarrow \quad \mathbf v = \mathbf 0.$

Now suppose there exist two such transformations $T_1, T_2$ that satisfy the crucial property. Fix $\mathbf w \in W$ . Then linearity yields

$\langle \cdot, T_1(\mathbf w) - T_2(\mathbf w) \rangle = O_V \quad \Rightarrow \quad T_1(\mathbf w) = T_2(\mathbf w).$

Since $\mathbf w \in W$ is arbitrary, we have $T_1 = T_2$ , as required.

Corollary 1. For any $\mathbf v \in \mathbb K^n$ , $\mathbf w \in \mathbb K^m$ , and $\mathbf A : \mathbb K^n \to \mathbb K^m$ ,

$\langle \mathbf A\mathbf v, \mathbf w\rangle = \langle \mathbf v, \mathbf A^* \mathbf w \rangle.$

Furthermore, the map $\mathcal M_{m \times n}(\mathbb K) \to \mathcal M_{n \times m}(\mathbb K), \mathbf A \mapsto \mathbf A^*$ is an isomorphism.

Corollary 2. Let $U , V , W$ be inner product spaces over $\mathbb K$ and $S : U \to V$ , $T : V \to W$ where their adjoints exist. Then $(T \circ S)^* = S^* \circ T^*$ .

Proof. Fix $\mathbf u \in U, \mathbf w \in W$ . Then

$\begin{aligned}\langle (T \circ S)(\mathbf u), \mathbf w \rangle &= \langle T (S(\mathbf u)), \mathbf w \rangle \\ &= \langle S(\mathbf u), T^*(\mathbf w) \rangle \\ &= \langle \mathbf u, S^*(T^*(\mathbf w)) \rangle \\ &= \langle \mathbf u, (S^* \circ T^*)(\mathbf w) \rangle. \end{aligned}$

By uniqueness, $(T \circ S)^* = S^* \circ T^*$ .

Corollary 3. For $\mathbf A : \mathbb K^m \to \mathbb K^k$ and $\mathbf B : \mathbb K^n \to \mathbb K^m$ . Then $(\mathbf A \mathbf B)^* = \mathbf B^* \mathbf A^*$ . If $\mathbb K = \mathbb R$ , then $(\mathbf A \mathbf B)^{\mathrm T} = \mathbf B^{\mathrm T} \mathbf A^{\mathrm T}$ .

Corollary 4. Define $\overline{[a_{ij}]} := [\bar{a}_{ij}]$ for appropriately sized matrices. For $\mathbf A : \mathbb K^m \to \mathbb K^k$ , $\mathbf A^{\mathrm T} = {\overline{\mathbf A}}^*$ , even when $\mathbb K = \mathbb C$ . Hence, for $\mathbf B : \mathbb K^n \to \mathbb K^m$ , $(\mathbf A \mathbf B)^{\mathrm T} = \mathbf B^{\mathrm T} \mathbf A^{\mathrm T}$ .

Finally, we can return to our original problem in $\mathbb R^n$ . We have derived the following identity using best approximations:

$\langle \mathbf A \mathbf e_j, \mathbf A \mathbf u \rangle = \langle \mathbf A \mathbf e_j, \mathbf b \rangle.$

In the context $\mathbb K = \mathbb R$ , ${\mathbf A}^* = {\mathbf A}^{\mathrm T}$ , so that taking adjoints,

$\langle \mathbf e_j, {\mathbf A}^{\mathrm T} \mathbf A \mathbf u \rangle = \langle \mathbf e_j, {\mathbf A}^{\mathrm T} \mathbf b \rangle.$

Since $\mathbf e_j$ is arbitrary, by the injectivity of $\mathbf v \mapsto \langle \cdot ,\mathbf v \rangle$ , ${\mathbf A}^{\mathrm T} \mathbf A \mathbf u = {\mathbf A}^{\mathrm T} \mathbf b$ . Therefore, we can solve our best approximation problem. In fact, we can solve this problem even if $\mathbb K = \mathbb C$ .

Theorem 2. Let $V, W$ be inner product spaces over $\mathbb K$ , $T : V \to W$ a linear transformation.
- If $W$ is finite-dimensional, then for any $\mathbf y \in W$ , the equation $T(\mathbf x) = \mathbf y$ has a unique best approximation.
- If the equation $T(\mathbf x) = \mathbf y$ has a unique best approximation then $T(\mathbf u)$ is the best approximation if and only if $T(\mathbf u) = \mathrm{proj}_{T(V)}(\mathbf y)$ .
- Furthermore, if $T^*$ exists, then $T(\mathbf u)$ is the best approximation if and only if $(T^* \circ T)(\mathbf u)=T^*(\mathbf y)$ .
Corollary 5. Fix $\mathbf A : \mathbb K^n \to \mathbb K^m$ and $\mathbf b \in \mathbb K^m$ . The equation $\mathbf A \mathbf x = \mathbf b$ has a unique best approximation. Furthermore, $\mathbf A \mathbf u$ is the best approximation if and only if ${\mathbf A}^* \mathbf A \mathbf u = {\mathbf A}^* \mathbf b$ . In particular, if $\mathbb K =\mathbb R$ , $\mathbf u$ must satisfy the equation ${\mathbf A}^{\mathrm T} \mathbf A \mathbf u = {\mathbf A}^{\mathrm T} \mathbf b$ .

And so I used my graphing calculator to solve the equation ${\mathbf A}^{\mathrm T} \mathbf A \mathbf u = {\mathbf A}^{\mathrm T} \mathbf b$ , and with 5 minutes of somewhat tedious typing, computed the theoretically perfect best-fit line (up to a precision of 3 significant figures). Then 3 months later, I learned that the calculator had an in-built function to compute this line in 3 seconds. Bummer.

In our next post, we will examine the adjoint transformation $T^*$ a bit more, and discover something peculiar about matrices that satisfy the innocent-looking equality $\mathbf A^* = \mathbf A$ . These tools will help us explore the spectral theorems, which will empower us with one very useful concrete application—singular value decomposition.

—Joel Kindiak, 13 Mar 25, 1716H
August 18, 2025
Baby Approximation Theory

When I took my A-Level examinations, I was required to take a Chemistry practical assessment. In this assessment, I collected data and needed to draw a line of best-fit. About that time, I learned linear algebra, and thought I’d apply what I learned to compute the perfect best-fit line. I got an ‘A’ for Chemistry in the end. It helped me quite a bit.

Given the dataset $\{(x_i, y_i) : 1 \leq n \leq N\} \subseteq \mathbb R \times \mathbb R$ , we want to find a line of best fit, usually denoted

$\hat y_i = \beta_0 + \beta_1 x_i,\quad 1 \leq i \leq N,$

that minimises the total squared-error $\displaystyle \sum_{i=1}^N (y_i - \hat y_i )^2$ . How do we achieve this goal? Let’s try to solve the equation directly, which gives the matrix equation

$\begin{bmatrix}1 & x_1 \\ \vdots & \vdots \\ 1 & x_N\end{bmatrix} \begin{bmatrix}\beta_0 \\ \beta_1 \end{bmatrix} = \begin{bmatrix} y_1 \\ \vdots \\ y_N \end{bmatrix}.$

The probability that we can even solve this won’t be high, since we have too much data! For instance, the simple data set $\{ (1, 1), (2, 3), (3, 4) \}$ will be enough to confound this equation. With this question at hand, let’s ask a more generically formable question: What would be a “best-fit” solution to the usually-unsolvable equation $\mathbf A \mathbf x = \mathbf b$ ?

Even more generally, if $V$ is an inner product space over $\mathbb K$ , and $T : V \to V$ is a linear transformation, what would be a “best-fit” solution to the equation $T(\mathbf x) = \mathbf y$ ? By definition, a solution exists if and only if $\mathbf y \in T(V) \subseteq V$ . What if $\mathbf y \notin T(V)$ ?

Lemma 1. For any subspace $W \subseteq V$ , the orthogonal complement

$W^{\perp} := \{ \mathbf u \in V : \mathbf u \perp W \}$

is also a subspace of $V$ . Furthermore, if $V$ is finite-dimensional, then

$V = W \oplus W^{\perp}.$

Proof. If $V$ is finite dimensional, then $W$ is finite-dimensional and contains an orthonormal basis, and

$W = \ker(I_V - \mathrm{proj}_W), W^{\perp} = \ker(\mathrm{proj}_W).$

The decomposition $I_V = (I_V - \mathrm{proj}_W) + \mathrm{proj}_W$ with the property

$\begin{aligned} \mathrm{proj}_W \circ (I_V - \mathrm{proj}_W) &= \mathrm{proj}_W - \mathrm{proj}_W^2 \\ &= \mathrm{proj}_W - \mathrm{proj}_W = O_V\end{aligned}$

yields the desired result.

Theorem 1. Let $V$ be finite-dimensional and $W \subseteq V$ a subspace. Then for any $\mathbf y \in V$ , there exists unique vectors $\mathbf w \in W$ , $\mathbf n \in W^\perp$ , such that

$\mathbf y = \mathbf w + \mathbf n.$

We call this expression the orthogonal decomposition of $\mathbf y$ . In fact, by Lemma 1, we must have

$\mathbf y = \mathrm{proj}_W(\mathbf y) + (\mathbf y - \mathrm{proj}_W(\mathbf y)).$

Proof. We have established existence by Lemma 1, and it remains to prove uniqueness. Consider two possible decompositions

$\mathbf y = \mathbf w + \mathbf n = \mathbf w' + \mathbf n'.$

Then

$\mathbf w - \mathbf w' = \mathbf n' - n \in W \cap W^{\perp} = \{\mathbf 0\},$

yielding $\mathbf w = \mathbf w'$ and $\mathbf n = \mathbf n'$ .

So the projection is indeed a powerful tool, since we can always write $\mathbf y$ in terms of its projection. Furthermore, the orthogonal component $\mathbf y - \mathrm{proj}_{T(V)}(\mathbf y)$ will definitely be orthogonal to $\mathbf y$ . But this raises the question: is $\mathrm{proj}_{T(V)}(\mathbf y)$ really our best-fit solution?

Definition 1. For any $\mathbf v \in V$ and subspace $W \subseteq V$ , we call $\mathbf u \in W$ a best approximation of $\mathbf v$ onto $W$ if

$\| \mathbf v - \mathbf u \| \leq \| \mathbf v - \mathbf w \|,\quad \mathbf w \in W.$

Theorem 2. For any $\mathbf y \in V$ and subspace $W \subseteq V$ , $\mathrm{proj}_W(\mathbf v)$ is the best approximation of $\mathbf y$ onto $W$ .

Proof. Firstly, for vectors $\mathbf u, \mathbf v \in V$ ,

$\begin{aligned} \| \mathbf u + \mathbf v \|^2 &= \langle \mathbf u + \mathbf v, \mathbf u + \mathbf v \rangle \\ &= \langle \mathbf u, \mathbf u \rangle + \langle \mathbf u, \mathbf v \rangle + \langle \mathbf v, \mathbf u \rangle + \langle \mathbf v, \mathbf v \rangle \\ &= \| \mathbf u \|^2 + \| \mathbf v \|^2 + \langle \mathbf u, \mathbf v \rangle + \langle \mathbf v, \mathbf u \rangle. \end{aligned}$

Thus, we have Pythagoras’ theorem for inner product spaces if $\langle \mathbf u, \mathbf v \rangle = 0$ , then

$\| \mathbf u + \mathbf v \|^2 = | \mathbf u \|^2 + \| \mathbf v \|^2.$

Therefore, for any $\mathbf w \in W$ ,

$\begin{aligned} \| \mathbf v -\mathbf w \|^2 &= \| (\mathbf v - \mathrm{proj}_W(\mathbf v)) + (\mathrm{proj}_W(\mathbf v) - \mathbf w)\|^2 \\ &= \| \mathbf v - \mathrm{proj}_W(\mathbf v) \|^2 + \| \mathrm{proj}_W(\mathbf v) - \mathbf w \|^2 \\ &\geq \| \mathbf v - \mathrm{proj}_W(\mathbf v) \|^2 + 0 \\ &= \| \mathbf v - \mathrm{proj}_W(\mathbf v) \|^2. \end{aligned}$

For uniqueness, suppose $\mathbf u_1, \mathbf u_2$ are best approximations. Define $\mathbf u_3 := \frac 12 (\mathbf u_1 + \mathbf u_2)$ . Then

$\begin{aligned} 0 < \| \mathbf v - \mathbf u_3 \| &= \textstyle \| \frac 12 (\mathbf v - \mathbf u_1) + \frac 12 (\mathbf v - \mathbf u_2) \| \\ &\leq \textstyle \frac 12 \| \mathbf v - \mathbf u_1 \| + \frac 12 \| \mathbf v - \mathbf u_2 \| \\ &= \| \mathbf v - \mathbf u_1 \|.\end{aligned}$

Hence, $\mathbf u_3$ is also a best approximation. Defining $\mathbf z_i := \mathbf v - \mathbf u_i$ , we have the equality

$\textstyle \| \mathbf z_1 \| = \| \mathbf z_2 \| = \| \frac 12 (\mathbf z_1 + \mathbf z_2) \| =: 1,$

where we set them equal $1$ without loss of generality. We can prove the parallelogram identity via

$\|\mathbf z_1 - \mathbf z_2 \|^2 + \|\mathbf z_1 + \mathbf z_2 \|^2 = 2 \|\mathbf z_1\|^2 + 2 \|\mathbf z_2 \|^2.$

Substituting the various values,

$\|\mathbf z_1 - \mathbf z_2 \|^2 + 2^2 = 2 \cdot 1^2 + 2 \cdot 1^2.$

Hence, $\|\mathbf z_1 - \mathbf z_2 \| = 0$ yields $\mathbf z_1 = \mathbf z_2 = \mathbf z_3$ . Hence $\mathbf u_1 = \mathbf u_2 = \mathbf u_3$ are the one and the same best approximation.

The key take-away is this: the best approximation of $\mathbf v$ onto $W$ is its projection $\mathrm{proj}_W(\mathbf v)$ . But how do we answer our semi-original question: finding a best-fit solution to $T(\mathbf x) = \mathbf y$ ?

Corollary 1. Let $V, W$ be finite-dimensional vector spaces and $W$ be an inner product space. Let $T : V \to W$ be a linear transformation and $\mathbf y \in W$ . Then the equation $T(\mathbf x) = \mathbf y$ has a best approximation given by $\mathbf u \in V$ such that $T(\mathbf u) = \mathrm{proj}_W(\mathbf y)$ .

Proof. Apply Theorem 2 to the subspace $T(V) \subseteq W$ .

This result helps us partly solve our original equation. Let $\mathbf A : \mathbb K^n \to \mathbb K^m$ . By Theorem 2, a best approximation solution to the equation $\mathbf A \mathbf x = \mathbf b$ , we need to find $\mathbf u \in \mathbb K^n$ such that $\mathbf A \mathbf u = \mathrm{proj}_{\mathbf A(\mathbb K^n)} (\mathbf b)$ . But how do we actually compute such a $\mathbf u$ ?

The key property that we want to exploit is that $(\mathbf b - \mathbf A \mathbf u) \perp \mathbf A(\mathbb K^n)$ . This means that for any $\mathbf e_i$ ,

$\langle \mathbf A \mathbf e_j, \mathbf b - \mathbf A \mathbf u \rangle = 0 \quad \iff \quad \langle \mathbf A \mathbf e_j, \mathbf A \mathbf u \rangle = \langle \mathbf A \mathbf e_j, \mathbf b \rangle.$

It turns out that in solving this problem, we are finally forced to introduce the idea of the matrix transpose, which we will properly discuss in the next post. Remember, we haven’t technically answered our best-fit line question: find meaningful constants $\beta_0,\beta_1$ such that

$\hat y_i = \beta_0 + \beta_1 x_i,\quad 1 \leq i \leq N.$

As a side-note, we can generalise this problem to a more realistic setting involving machine learning datasets $\{(\mathbf x_i, y_i) : 1 \leq i \leq N\} \subseteq \mathbb R^n \times \mathbb R$ , where we seek sufficiently meaningful parameters $\theta_0 \in \mathbb R$ , $\boldsymbol{\theta} \in \mathbb R^n$ such that

$\hat y_i = \theta_0 + \langle \boldsymbol{\theta}, \mathbf x_i \rangle,\quad 1 \leq i \leq N.$

More on these ideas in future posts.

—Joel Kindiak, 12 Mar 25, 2102H

August 11, 2025
Generalised Perpendicularity
Recall that an inner product $\langle \cdot , \cdot \rangle$ on a vector slave $V$ over a field $\mathbb K$ ( $\mathbb R$ or $\mathbb C$ ) gives us the generalised formulation of the dot product on $\mathbb R^2$ . In particular, if $\mathbb K = \mathbb R$ , then two nonzero vectors $\mathbf u, \mathbf v \in V$ have angle $\theta$ defined by

$\langle \mathbf u , \mathbf v \rangle = \| \mathbf u \| \| \mathbf v \| \cos(\theta).$

In particular, $\langle \mathbf u, \mathbf v \rangle = 0$ if and only if $\cos(\theta) = 0$ , which means $\theta = \pi/2$ . In this case, we say that $\mathbf u, \mathbf v$ are perpendicular. In the usual two-dimensional space,

$\mathbf u \perp \mathbf v \quad \iff \quad \mathbf u \cdot \mathbf v = 0.$

In particular, we obtain a class result from coordinate geometry.

Theorem 1. Two lines $a_1 x + b_1 y = c_1$ and $a_2 x + b_2 y = c_2$ are perpendicular if and only if $a_1 a_2 + b_1 b_2 = 0$ . In particular, if $m_{ \text{tangent} } = a_1$ , $m_{ \text{normal} } = a_2$ and $b_1 = b_2 = 1$ , then we obtain the more familiar formula

$m_{ \text{tangent} } \cdot m_{ \text{normal} } = -1$

via $m_{ \text{tangent} } = a_1$ , $m_{ \text{normal} } = a_2$ .

Proof. The first line is parallel to the direction vector $b_1 \mathbf e_1 - a_1 \mathbf e_2$ and the second line is parallel to the direction vector $b_2 \mathbf e_1 - a_2 \mathbf e_2$ .

While this interpretation only makes sense when $\mathbf u, \mathbf v$ are nonzero vectors, we still have the property $\langle \mathbf u, \mathbf v \rangle = 0$ . This lends us to the more general (and objectively more veracious) notion of right (i.e. ortho) angles (i.e. gonal)—orthogonality.

Henceforth, let $(V, \langle \cdot , \cdot \rangle)$ be an inner product space over $\mathbb K$ .

Definition 1. Two vectors $\mathbf u, \mathbf v$ are orthogonal if $\langle \mathbf u, \mathbf v \rangle = 0$ .
- A set $K$ of vectors is orthogonal if any distinct vectors $\mathbf u, \mathbf v \in K$ are orthogonal.
- We say that $\mathbf v$ is perpendicular to the subset $U \subseteq V$ if for any $\mathbf u \in U$ , $\mathbf u,\mathbf v$ are orthogonal. In this case, we denote $\mathbf v \perp U$ .
- Two subsets $U, W \subseteq V$ are orthogonal if for any $\mathbf u \in U$ and $\mathbf v \in V$ , $\{ \mathbf u, \mathbf v \}$ is orthogonal. In this case, we denote $U \perp W$ .
Lemma 1. The zero vector $\mathbf 0$ is orthogonal to any $\mathbf v \in V$ . Furthermore, if $U,W \subseteq V$ are orthogonal subspaces, then $U \cap W = \{\mathbf 0\}$ .

Proof. For any $\mathbf v \in U \cap W$ , $\langle \mathbf v, \mathbf v \rangle = 0$ .

Lemma 2. Any orthogonal set $K$ of nonzero vectors is linearly independent.

Proof. Assume $K = \{\mathbf v_1,\dots,\mathbf v_k\}$ without loss of generality. Consider the equation

$\displaystyle \sum_{i=1}^k c_i \mathbf v_k = \mathbf 0.$

For each $j$ , apply $\langle \cdot , \mathbf v_j\rangle$ on both sides to obtain

$\begin{aligned}\langle \mathbf 0, \mathbf v_j\rangle &= \left\langle \sum_{i=1}^k c_i \mathbf v_i, \mathbf v_j \right\rangle = \sum_{i=1}^k c_i \langle \mathbf v_i, \mathbf v_j \rangle \\ &= \sum_{i \neq j} c_i \cdot 0 + c_j \langle \mathbf v_j , \mathbf v_j \rangle = c_j \cdot \|\mathbf v_j \|^2. \end{aligned}$

Since $\|\mathbf v_j \| \neq 0$ , $c_j = 0$ . Since this argument holds for any $j$ , we get $c_1 = \cdots = c_k = 0$ , and the result holds.

Lemma 3. The map $\langle \cdot, \cdot \rangle: \mathbb K^n \times \mathbb K^n \to \mathbb K$ defined by

$\displaystyle \langle \mathbf u, \mathbf v \rangle := \sum_{i=1}^n u_i \bar v_i$

is an inner product on $\mathbb K^n$ . In particular, $\langle \mathbf e_i, \mathbf e_j \rangle = \mathbb I_{\{i\}}(j)$ . Hence, we call the standard basis $\{\mathbf e_1,\dots,\mathbf e_n\}$ for $\mathbb K^n$ orthonormal.

Definition 2. An orthogonal set $K \subseteq V$ is orthonormal if $\| \mathbf v \| = 1$ for each $\mathbf v \in K$ . We call $K$ an orthonormal basis for $V$ if $\mathrm{span}(K) = V$ .

Why are orthonormal sets so special? That is because decomposing any vector in $\mathrm{span}(K)$ becomes trivial.

Theorem 2. Let $I$ be an index set and $K = \{\mathbf u_\alpha : \alpha \in I\}$ be an orthonormal set. For any vector $\mathbf v \in \mathrm{span}(K)$ , there exist unique vectors $\mathbf u_1,\dots,\mathbf u_k$ such that

$\displaystyle \mathbf v = \sum_{i=1}^n \langle \mathbf v, \mathbf u_i \rangle \mathbf u_i.$

Proof. Since $K$ is linearly independent, it forms an orthonormal basis for $\mathrm{span}(K)$ . Hence, for any $\mathbf v \in \mathrm{span}(K)$ , there exist unique vectors $\mathbf u_i \in K$ and unique scalars $c_i \in \mathbb K$ such that

$\displaystyle \mathbf v = \sum_{i=1}^n c_i \mathbf u_i.$

For each $j$ , apply $\langle \cdot, \mathbf u_j\rangle$ on both sides,

$\displaystyle \langle \mathbf v, \mathbf u_j\rangle = \sum_{i=1}^n c_i \langle \mathbf u_j, \mathbf u_i\rangle = \sum_{i \neq j} c_i \cdot 0 + c_j \cdot 1 = c_j,$

as required.

In this regard, the standard basis $\{\mathbf e_1,\dots,\mathbf e_n\}$ plays a special role in being a sufficiently “preferred” basis than others.

That’s great, but what if we only have a basis $K$ for $V$ , and not an orthonormal basis for $V$ . Can we make such a conversion? For finite-dimensional vector spaces, the answer is ‘Yes’!

But first, we will need an “orthogonalising” tool. Given an orthonormal set $K$ , if $\mathbf v \notin \mathrm{span}(K)$ , then can we find some vector $\mathbf u \in \mathrm{span}(K)$ such that $(\mathbf v - \mathbf u) \perp K$ ?

Lemma 4. Let $K := \{ \mathbf u_1,\dots,\mathbf u_n \}$ be an orthonormal set and $U := \mathrm{span}(K)$ . Motivated by Theorem 2, define the projection map by

$\displaystyle \mathrm{proj}_U := \sum_{i=1}^n \langle \cdot, \mathbf u_i \rangle \mathbf u_i : V \to U.$

Clearly, $(\mathrm{proj}_U)|_U = \mathrm{id}_U$ . Then for any $\mathbf v \notin U$ , $(\mathbf v - \mathrm{proj}_U(\mathbf v)) \perp U$ .

Proof. For any $j$ ,

$\begin{aligned} \langle \mathrm{proj}_U(\mathbf v), \mathbf u_j \rangle &= \sum_{i=1}^n \langle \mathbf v, \mathbf u_i \rangle \langle \mathbf u_i , \mathbf u_j\rangle = \langle \mathbf v , \mathbf u_j \rangle. \end{aligned}$

Hence,

$\langle \mathbf v - \mathrm{proj}_U(\mathbf v), \mathbf u_j \rangle = \langle \mathbf v , \mathbf u_j \rangle -\langle \mathrm{proj}_U(\mathbf v), \mathbf u_j \rangle = 0.$

The result follows by the conjugate-linearity of $\langle \mathbf v - \mathrm{proj}_U(\mathbf v), \cdot \rangle$ .

Theorem 3 (Gram-Schmidt Process). Every finite-dimensional vector space $V$ will have an orthonormal basis.

We may think that the isomorphism $\mathbf v_k \mapsto \mathbf e_k \in \mathbb K^n$ suffices. However, we need this bijection to preserve angles, which isn’t a trivial matter to check at all.

Instead, we will slowly construct this orthonormal basis. Let $\{\mathbf v_1,\dots,\mathbf v_n\}$ be a basis (not necessarily orthonormal!) for $V$ . Define $\mathbf u_1 := \hat{\mathbf w}_1 = \hat{\mathbf v}_1$ . Now $\{\mathbf v_1, \mathbf v_2\}$ is obviously linearly independent. This means $\mathbf v_2 \notin \mathrm{span}\{\mathbf u_1\}$ .

Define

$\mathbf w_2 := \mathbf v_2 - \mathrm{proj}_{\mathrm{span}\{\mathbf u_1\}}(\mathbf v_2),$

which is orthogonal to $\mathbf u_1$ . Hence, define $\mathbf u_2 := \hat{\mathbf w}_2$ . Inductively, given $K_i := \{\mathbf u_1,\dots,\mathbf u_i\}$ , define,

$\mathbf w_{i+1} := \mathbf v_{i+1} - \mathrm{proj}_{\mathrm{span}(K_i)}(\mathbf v_{i+1})$

and $\mathbf u_{i+1} := \hat{\mathbf w}_{i+1}$ . Then we will obtain the desired orthonormal basis $K_n$ , since by construction we have for each $i$ , $\mathbf v_i \in \mathrm{span}(K_i) \subseteq \mathrm{span}(K_n)$ .

The infinite-dimensional cases aren’t easy to analyse, but can be reasonably studied in the context of Hilbert spaces, which features surprisingly prominently in modern physics. To do that will require us to study functional analysis, since we need to scrutinise convergence issues more closely than the finite-dimensional setting.

For now, let’s discuss the crucial finite-dimensional application in best-fit approximations, one of my favourite areas of study in applied mathematics—approximation theory.

—Joel Kindiak, 12 Mar 25, 1711H
August 4, 2025