Mathematical Background

Vector space

A vector space $V$ over a field $K$ is a set with operations $+$ and $\cdot$ , where $(\forall a, b, c \in V) (\forall x, y \in K)$ :

Axiom	Formula
Associativity of addition	$(a + b) + c = a + (b + c)$
Commutativity	$a + b = b + a$
Neutral element wrt. addition	$(\exists0 \in V) (a + 0 = a)$
Inverse element wrt. addition	$(\exists d \in V) (a + d = 0)$
—	—
Distributivity of multiplying a vector by a scalar wrt. to vector addition	$x \cdot (a + b) = x \cdot a + x \cdot b$
Distributivity of multiplying a vector by a scalar wrt. to scalar addition	$(x + y) \cdot a = x \cdot a + y \cdot a$
Associativity of multiplication	$(a \cdot b) \cdot c = a \cdot (b \cdot c)$
Neutral element wrt. multiplication	$(\exists1 \in K) (1 \cdot a = a)$

Axiom

Formula

Associativity of addition

$(a + b) + c = a + (b + c)$

Commutativity

$a + b = b + a$

Neutral element wrt. addition

$(\exists0 \in V) (a + 0 = a)$

Inverse element wrt. addition

$(\exists d \in V) (a + d = 0)$

—

Distributivity of multiplying a vector by a scalar wrt. to vector addition

$x \cdot (a + b) = x \cdot a + x \cdot b$

Distributivity of multiplying a vector by a scalar wrt. to scalar addition

$(x + y) \cdot a = x \cdot a + y \cdot a$

Associativity of multiplication

$(a \cdot b) \cdot c = a \cdot (b \cdot c)$

Neutral element wrt. multiplication

$(\exists1 \in K) (1 \cdot a = a)$

Vector

Vectors are elements of a vector space $V$ .

Dot product (inner product)

Geometrically, it’s a projection of $u$ into $v$ . A number that describes the similarity of $u$ and $v$ .

⟨ u, v ⟩ = i = 1 \sum n u_{i} \cdot \overline{v_{i}} \in K

Two vectors are orthogonal iff $⟨ u, v ⟩ = 0$ .
With complex numbers, we have to conjugate $v$ (hence the overline).
$⟨ v, v ⟩ = ∣ v ∣$ (length of vector $v$ ).
A vector $v$ is normalized (or unit) if $∣ v ∣ = 1$ .

Basis

A Euclidian vector space $V$ can be generated using a finite set of vectors that form the basis ${u_{1}, u_{2}, ..., u_{n}}$ . For every $v \in V$ , $v = a_{1} u_{1} + a_{2} u_{2} + ... + a_{n} u_{n}$ , where $a_{i} \in K$ .

A basis is orthonormal if all of its vectors are both normalized and orthogonal to each other.
Each orthonormal basis can be expressed as a square matrix.
Cartesian coordinate system (standard basis) is, for example in 3D, ${e_{1}, e_{2}, e_{3}}$ where:

$e_{1} = (1, 0, 0) e_{2} = (0, 1, 0) e_{3} = (0, 0, 1)$

Projecting vectors to a different basis

Projection is a change of coordinate system. A vector $v \in V$ can be projected into basis ${u_{1}, u_{2}, ..., u_{n}}$ :

v = a_{1} u_{1} + a_{2} u_{2} + ... + a_{n} u_{n} (\forall i = {1, ..., n} : a_{i} = \frac{⟨ v , u _{i} ⟩}{⟨ u _{i} , u _{i} ⟩})

Projection can be realized using matrix multiplication $v_{{u_{1}, u_{2}}} = A v_{{e_{1}, e_{2}}}$ .
Transform from one basis to another is a linear mapping.
An orthonormal transform matrix is unitary: $A^{- 1} = \overline{A}^{T}$ (conjugate transpose).
- If $y = A x$ is forward transform, then $x = A^{- 1} y = \overline{A}^{T} y$ is inverse transform.
"A vector is projected into a basis." is the same as "A vector is expanded into linear combination of basis vectors.".
An $N$ -dimensional vector is the same as a discrete function with length $N$ .

Function space

A set of functions between two sets.

Inner product (in function spaces)

Discerete:

$⟨ f, g ⟩ = n = 0 \sum \infty f (n) \overline{g (n)}$
Continuous:

$⟨ f, g ⟩ = \int_{a}^{b} f (x) \overline{g (x)} d x$
Both products perform projection of one function into another.
Orthogonal has the same meaning as in vector spaces.
For $cos (x)$ and $sin (x)$ , we use $a = - π$ and $b = π$ .
$sin$ and $cos$ are orthogonal to one another, but neither is "normalized".

Expansion of discrete functions

Each discrete function can be decomposed into a linear combination of some simpler, well-known functions.
Any discrete 1D function $f$ of $N$ samples can be uniquely expressed as a linear combination of mutually orthogonal functions $φ_{1}, φ_{2}, ..., φ_{N}$ :

$f (m) = k = 1 \sum N a_{k} φ_{k} (m)$
- Where $a_{k}$ is a scalar and is found using inner product $a_{k} = \frac{⟨ f , φ _{k} ⟩}{⟨ φ _{k} , φ _{k} ⟩}$ .
The set of functions $φ_{k}$ is predefined.
We want the decomposition to be unique, fast, and numerically stable.
It’s very similar to decomposition of vectors, except we have an orthogonal basis of functions instead of vectors.
Ideally we want a perfect decomposition, so that we can go back, but sometimes approximation is enough (e.g., compression).

Expansion of continuous functions

Any continous function $f$ can be uniquely expanded into mutually orthogonal functions $φ_{k}, k \in N$ :

f (x) = k = 0 \sum \infty a_{k} φ_{k} (x)

where scalar coefficients $a_{k} = \frac{⟨ f , φ _{k} ⟩}{⟨ φ _{k} , φ _{k} ⟩}$ .

Even and odd functions

For even functions $f (- x) = f (x)$ (e.g., $cos$ ).
For odd functions $f (- x) = - f (x)$ (e.g., $sin$ ).
Any function can be split into an even and odd function:

$E (m) O (m) f (m) = \frac{1}{2} [f (m) + f (- m)] = \frac{1}{2} [f (m) - f (- m)] = E (m) + O (m)$

Sinc

sinc (x) = \frac{sin x}{x}

Kronecker delta

δ_{i, j} = {0, 1, i \neq = j i = j

Dirac delta

Contiguous equivalent to the Kronecker delta. Also known as the unit impulse:

δ (x) = {0, \infty, x \neq = 0 x = 0 such that \int_{\infty}^{\infty} δ (x) d x = 1

Euler’s formula (Euler-Moivre equation)

e^{i x} = cos x + i \cdot sin x

Questions

What is a vector space generated by the given basis?: A set of all linear combinations of the basis vectors.
What are orthogonal vectors?: Geometrically, vectors that are perpendicular. Mathematically, those whose inner product is zero.
What is the orthonormal basis?: Basis formed of unit vectors that are orthogonal to one another.
How can we simply convert a vector from one basis to another basis?: Simply by multiplying it with a transform matrix (possibly going through the standard basis in the middle).
What is the unitary/orthogonal matrix?: A matrix whose inversion is the same as its conjugate transpose.
What is the difference between basis vector and basis function?: A discrete function is a N-dimensional vector. A continous function is a (possibly) (uncountably) infinite vector.
Explain the meaning of expansion coefficients.: They measure the "significance" of the respective basis function used in the expansion.