Formulas

Formulas and Theorem

---

Expansion in Orthogonal Basis

If we have an Orthogonal Basis $\{{\vec{u}}_1,\dots ,{\vec{v}}_n\}$ in ${\mathbb{R}}^n$ then for any $\vec{w}\in {\mathbb{R}}^n$,

$$\vec{w}=c_1{\vec{u}}_1+\cdots +c_n{\vec{v}}_n$$

$C_q$ can be found using, $c_q=\frac{\vec{w}\cdot {\vec{u}}_q}{{\vec{u}}_q\cdot {\vec{u}}_q}$

---

Orthogonal Projection

Let non-zero $\vec{u}\in {\mathbb{R}}^n$, and $\vec{y}\in {\mathbb{R}}^n$. The orthogonal projection of $\vec{y}$ onto $\vec{u}$ is the vector in the span of $\vec{u}$ that is closest to $\vec{y}$.

\text{proj}_\vec{u}\vec{y}=\frac{\vec{y}\cdot\vec{u}}{\vec{u} \cdot \vec{u}}\vec{u}

Also, $\vec{y}=\hat{y}+\vec{z}$ and, $\vec{z}\in W^{\perp }$

---

Best Approximation Theorem

Let $W$ be a subspace of ${\mathbb{R}}^n,\vec{y}\in {\mathbb{R}}^n$, and $\hat{y}$ is the orthogonal projection of $\vec{y}$ onto $W$. Then for any $\vec{v}\ne \hat{y},\vec{v}\in W$, we have

$$||\vec{y}-\hat{y}||<||\vec{y}-\vec{v}||$$

---

Orthogonal Decomposition Theorem

Let $W$ be a subspace of ${\mathbb{R}}^n$. Then, each $\vec{y}\in {\mathbb{R}}^n$ has a unique decomposition.

$$\vec{y}=\hat{y}+z,\hat{y}\in W,z\in W^{\perp }$$

If $\overrightarrow{u_1},\dots ,\overrightarrow{u_n}$ is the orthogonal basis for $W$,

$$\hat{y}=\frac{\vec{y}\cdot \overrightarrow{u_1}}{\overrightarrow{u_1}\cdot \overrightarrow{u_1}}\overrightarrow{u_1}+\cdots +\frac{\vec{y}\cdot \overrightarrow{u_n}}{\overrightarrow{u_n}\cdot \overrightarrow{u_n}}\overrightarrow{u_n}$$

$\hat{y}$ is the orthogonal projection of $\vec{y}$ onto $W$

---

QR Factorization

For a $m\times n$ matrix $A$ linearly independent columns,

$$A=QR$$

$Q$ is an $m\times n$, with columns are an orthonormal basis for $\text{Col}A$. $R$ is $n\times n$, upper triangular, with positive entries on its diagonal.

---

Normal Equation

$$A^{T}A\hat{x}=A^T\vec{b}$$

Manipulating this we can get this,

$$\hat{x}=(A^{T}A{)}^{-1}{A}^T\vec{b}$$

---

Eigenvectors orthogonality

If $A$ is a symmetric matrix, with eigenvectors $\overrightarrow{v_1}$ and $\overrightarrow{v_2}$ corresponding to two distinct eigenvalues, then $\overrightarrow{v_1}$ and $\overrightarrow{v_2}$ are orthogonal.

Proof

---