Linear Algebra

Linear Equation

Linear Space

Linear Subspace

Affine Space

Topological Linear Space

Normed Linear Space

(1,0)-Tensor/Vector

\begin{matrix} a, b : F^{n} c : F \\ {(a + b)}_{i} \equiv a_{i} + b_{i} & , \\ {(c a)}_{i} \equiv c a_{i} & . \end{matrix}

Example

a₁ = 1, a₂ = 0, a₃ = 3 or a = (1,0, 3).

Basis

\begin{matrix} e_{i} : F^{n}, a_{i} : F & , \\ a \equiv (a_{1}, \dots, a_{n}) \equiv a_{i} e_{i} & , \\ e_{i} \cdot e_{j} \equiv case (i = j) (1) (0) \equiv {(1_{n})}_{i, j} \equiv δ_{i, j} & . \end{matrix}

Example

e_{i_{1}, i_{2}} e_{i_{2}, i_{3}} e_{i_{3}, i_{1}} = e_{i_{1}, i_{3}} e_{i_{3}, i_{1}} = e_{i_{1}, i_{1}}

Orthonormal Basis

Gram–Schmidt Process

Linear Map

\begin{matrix} A, B \subseteq f, c : f, f : A \to B \\ f (a + b) = f (a) + f (b) (additive), \\ f (c a) = c f (a) (homogeneous). \end{matrix}

Antilinear Map

f (a x + b y) = a^{*} f (x) + b^{*} f (y) .

Semilinear Map

2-Tensor/Matrix

\begin{matrix} a & , & b & : & F^{n, m} & , & c & : & f & , \\ a_{i, j} & \equiv & {(a_{i})}_{j} & , \\ ( & a & = & b & ) & \equiv & ( & i & , & j & ) & \to & ( & a_{i, j} & = & b_{i, j} & ) & . \end{matrix}

Properties

{(a + b)}_{i, j} \equiv {({(a + b)}_{i})}_{j} \equiv {(a_{i} + b_{i})}_{j} \equiv {(a_{i})}_{j} + {(b_{i})}_{j} \equiv a_{i, j} + b_{i, j} (additive).

{(c a)}_{i, j} \equiv {({(c a)}_{i})}_{j} \equiv {(c a_{i})}_{j} \equiv c {(a_{i})}_{j} \equiv c a_{i, j} (homogeneous).

a − b \equiv a + (- 1) b .

{(a - b)}_{i, j} \equiv {(a + (- 1) b)}_{i, j} \equiv a_{i, j} + {((- 1) b)}_{i, j} \equiv a_{i, j} + (- 1) b_{i, j} \equiv a_{i, j} − b_{i, j} .

Example

a₁ = (3,1) and a₂ = (0,6)or a = ((3,1), (0,6)).

Vector-Matrix Multiplication

\begin{matrix} a & : & F^{n, m} & , & x & : & F^{n} & , & y & : & F^{m} & , \end{matrix}

Left Vector Multiplication

{(x a)}_{i} \equiv x {a^{t}}_{i} .

Right Vector Multiplication

{(a y)}_{i} \equiv a_{i} y .

Matrix-Matrix Multiplication

\begin{matrix} a & : & F^{n, m} & , & b & : & F^{m, l} & , \\ a & b & : & F^{n, l} & , \\ {(a b)}_{i, j} & \equiv & a_{i, k} & b_{k, j} & \equiv & a_{i} & \cdot & {b^{t} j}_{*} & . \end{matrix}

Properties

a (b c) = (a b) c (associative) .

∵

{(a (b c))}_{i, j} \equiv a_{i, k} {(b c)}_{k, j} \equiv a_{i, k} b_{k, l} c_{l, j} \equiv {(a b)}_{i, l} c_{l, j} \equiv {((a b) c)}_{i, j} .

(a, b) × (a b ≠ b a) (non-commutative),

a (b + c) = a b + a c (left-distributive) .

∵

{(a (b + c))}_{i, j} \equiv a_{i, k} {(b + c)}_{k, j} \equiv a_{i, k} (b_{k, j} + c_{k, j}) \equiv a_{i, k} b_{k, j} + a_{i, k} c_{k, j} \equiv {(a b)}_{i, j} + {(a c)}_{i, j} \equiv {(a b + a c)}_{i, j} .

(a + b) c = a c + b c (right-distributive) .

∵

{((a + b) c)}_{i, j} \equiv {(a + b)}_{i, k} c_{k, j} \equiv (a_{i, k} + b_{i, k}) c_{k, j} \equiv a_{i, k} c_{k, j} + b_{i, k} c_{k, j} \equiv {(a c)}_{i, j} + {(b c)}_{i, j} \equiv {(a c + b c)}_{i, j} .

Matrix Exponential

Similarity

b = p^{- 1} a p

Column Matrix/Vector

\begin{matrix} a & : & F^{n, 1} & , \\ a & \equiv & ( & ( & a_{1, 1} & ) & , ⋯, & ( & a_{n, 1} & ) & ) \end{matrix}

Row Matrix/Vector

\begin{matrix} a & : & F^{1, n} & , \\ a \equiv ((a_{1, 1}, ⋯, a_{1, n}),) \end{matrix}

Transposition

{(a^{t})}_{i, j} \equiv a_{j, i} .

Identity Matrix

\begin{matrix} 1_{n} & : & F^{n, n} & , \\ {(1_{n})}_{i, j} & \equiv & case & ( & i & = & j & ) & ( & 1 & ) & ( & 0 & ) & . \end{matrix}

Zero Matrix

\begin{matrix} 0_{n} & : & F^{n, n} & , \\ {(0_{n})}_{i, j} & \equiv & 0. \end{matrix}

Scalar Matrix

\begin{matrix} c & : & f & , \\ c & 1_{n} & . \end{matrix}

Diagonal

(i ≠ j) \to (a_{i, j} = 0) .

Properties

((i ≠ j) \to (a_{i, j} = 0) × (b_{i, j} = 0)) \to ((i ≠ j) \to ({(a b)}_{i, j} = 0)) .

(i = j)→(j ≠ k)→a_i,j b_j,k = 0(j = k)→(i ≠ j)→a_i,j b_j,k = 0

Symmetric Matrix

\begin{matrix} a & : & F^{n, n} & , \\ ( & i & , & j & ) & \to & ( & a_{i, j} & = & a_{j, i} & ) & , \end{matrix}

a = a^{t} .

Properties

((i, j) \to (a_{i, j} = a_{j, i}) × (b_{i, j} = b_{j, i})) \to (i, j) \to ({(a + b)}_{i, j} = {(a + b)}_{j, i}) .

∵

{(a + b)}_{i, j} \equiv a_{i, j} + b_{i, j} = a_{j, i} + b_{j, i} \equiv {(a + b)}_{j, i}

((i, j) \to (a_{i, j} = a_{j, i})) \to (i, j) \to ({(c a)}_{i, j} = {(c a)}_{j, i}) .

((i, j) \to ({(a b)}_{i, j} = {(a b)}_{j, i})) \to (a b = b a) .

((i, j) \to (a_{i, j} = a_{j, i})) \to (i, j) \to ({a^{n}}_{i, j} = {a^{n}}_{j, i}) .

a^{- 1} = {(a^{- 1})}^{t} ≃ (a = a^{t}) × a^{- 1} exists.

Lemma

(a+a^t)^t=a^t+(a^t)^t=a^t+a=a+a^t.

Antisymmetric/skew-symmetric

\begin{matrix} a & : & F^{n, n} & , \\ ( & i & , & j & ) & \to & a_{i, j} & = & {−a}_{j, i} & , \\ or \\ a & = & a^{t} & . \end{matrix}

[a, b] \equiv a b - b a .

{[a, b]}^{t} \equiv {(a b - b a)}^{t} \equiv b^{t} a^{t} − a^{t} b^{t} = (−b) (−a) − (−a) (−b) \equiv b a - a b \equiv −[a, b].

exp (a) \equiv {(n!)}^{- 1} a^{n} .

Lemma

{(a - a^{t})}^{t} = a^{t} - {a^{t}}^{t} = a^{t} - a = - (a - a^{t}) .

Conjugate Transpose

{(a^{*})}_{i, j} \equiv {(a_{j, i})}^{*} .

Hermitian Matrix

\begin{matrix} a & : & F^{n, n} & , \\ ( & i & , & j & ) & \to & ( & a_{i, j} & = & {(a_{j, i})}^{*} & ) & , \\ or & a & = & a^{*} \end{matrix}

Orthogonal Matrix

\begin{matrix} a & : & F^{n, n} \\ a_{i, j} & a_{i, j} & = & a_{j, i} & a_{j, i} & = & {(1_{n})}_{i, j} & , \\ or & a & a^{t} & = & a^{t} & a & = & 1_{n} & , \end{matrix}

a_{j, i} = {(a^{- 1})}_{i, j} or a^{t} = a^{- 1}

Normal Matrix

\begin{matrix} a & : & F^{n, n} \\ a_{i, j} & {(a_{i, j})}^{*} & = & {(a_{j, i})}^{*} & a_{j, i} & , \\ or & a & a^{*} & = & a^{*} & a & . \end{matrix}

Unitary Matrix

\begin{matrix} a & : & F^{n, n} \\ a_{i, j} & {(a_{i, j})}^{*} & = & {(a_{j, i})}^{*} & a_{j, i} & = & {(1_{n})}_{i, j} & , \\ or & a & a^{*} & = & a^{*} & a & = & 1_{n} & . \end{matrix}

Lower/Left Triangular Matrix

\begin{matrix} a & : & F^{n, m} & , \\ ( & i & , & j & ) & \to & ( & i & < & j & ) & \to & ( & a_{i, j} & = & 0 & ) & . \end{matrix}

Upper/Right Triangular Matrix

\begin{matrix} a & : & F^{n, m} & , \\ ( & i & , & j & ) & \to & ( & i & > & j & ) & \to & ( & a_{i, j} & = & 0 & ) & . \end{matrix}

Miner

m_{i, j} \equiv \det (a_{p ≠ i, q ≠ j}) .

Cofactor

c_{i, j} \equiv {(- 1)}^{i + j} m_{i, j} .

Adjugate Matrix

{a'}_{i, j} \equiv c_{j, i} or a' \equiv c^{t} .

Inverse Matrix

{(a^{- 1})}_{i, j} \equiv {\det (a)}^{- 1} {a'}_{i, j} .

Generalised Inverse

Moore–Penrose Inverse

Trace

tr (a) \equiv a_{i, i} .

Properties

tr (a^{t}) = tr (a) .

tr (a b) = tr (b a) .

tr (c a + d b) = c tr (a) + d tr (b) .

tr (a b^{t}) = a : b .

Eigenvalues and Eigenvectors

\begin{matrix} a & : & F^{n, n} & , & x & : & F^{n} & , & x & ≠ 0, & c & : & f & , \\ a_{i, j} & x_{j} & = & c & x_{i} & , \\ or & a & x & = & c & x & . \end{matrix}

Eigenspace

Jordan Normal Form

Levi-Civita Symbol

ε_{i_{1}, i_{2}, i_{3}} \equiv caseevenodd ((i_{1}, i_{2}, i_{3})) (1) (- 1) (0) .

Alternatively

ε_{i_{1}, i_{2}, i_{3}} \equiv 2^{- 1} (i_{1} − i_{2}) (i_{2} − i_{3}) (i_{3} − i_{1}) .

See .

ε_{i_{1}, i_{2}, i_{3}} ε_{i_{4}, i_{5}, i_{1}} = e_{i_{1}, i_{4}}

ε_{i₁, i₂, i₃} ε_{i₄, i₅, i₁}=e_{i₁, i₄} e_{i₂, i₅} − e_{i₁, i₅} e_{i₂, i₄}.

Inner Product

a \cdot b \equiv a_{i} b_{i} .

Frobenius Inner Product

a⋅b=a_i,j b_i,j=vec(a)^t vec(b).

Exterior Product

\begin{matrix} e_{i_{1}} & , & e_{i_{2}} & , & e_{i_{3}} & : & F^{n} & , \\ e_{i_{1}} & \land & e_{i_{2}} & \equiv & ε_{i_{1}, i_{2}, i_{3}} & e_{i_{3}} & , \\ a & \land & b & = & a_{i} & b_{j} & ε_{i, j, k} & e_{k} & = & ε_{i, j, k} & a_{i} & b_{j} & e_{k} \end{matrix}

Lemma

a ∧ (b ∧ c)=a_i₁ e_i₁ ∧ (ε_{i₂, i₃, i₄} b_i₂ c_i₃ e_i₄)=ε_{i₂, i₃, i₄} a_i₁ b_i₂ c_i₃ (e_i₁ ∧ e_i₄)=ε_{i₂, i₃, i₄} ε_{i₁, i₄, i₅} a_i₁ b_i₂ c_i₃ e_i₅=(e e−e e) ... ...=(a⋅c) b − (a⋅b) c

a,b:R³ ⊢(a⊗b)⋅a=(a⊗b)⋅b=0 ||a ⊗ b||=||a|| ||b||

Outer Product

\begin{matrix} a & : & F^{n} & , & b & : & F^{m} & , \\ {(a \otimes b)}_{i, j} & \equiv & a_{i} & b_{j} & . \end{matrix}

\begin{matrix} i & \equiv & ( & i_{1} & , ⋯, & i_{n} & ) & , & j & \equiv & ( & j_{1} & , ⋯, & j_{m} & ) & , & a & : & F^{i} & , & b & : & F^{j} & , & a & ⊗ & b & : & F^{i, j} & , \\ {(a \otimes b)}_{i, j} & \equiv & a_{i} & b_{j} & . \end{matrix}

Example

a=(a₁, a₂, a₃), b=(b₁, b₂),a ⊗ b=((a₁ b₁, a₁ b₂), (a₂ b₁, a₂ b₂), (a₃ b₁, a₃ b₂)).

Properties

{(a \otimes b)}^{t} = b ⊗ a .

∵

^t=b ⊗ a(a⊗b)^t_i,j=(a⊗b)_j,i=a_j b_i=b_i a_j=(b ⊗ a)_i,j.

a ⊗ (b + c) = a ⊗ b + a ⊗ c (left-distributive).

∵

(a ⊗ (b+c))_i,j=a_i (b+c)_j=a_i (b_j + c_j)=a_i b_j + a_i c_j=(a⊗b)_i,j + (a ⊗ c)_i,j=(a ⊗ b+a ⊗ c)_i,j.

(a + b) ⊗ c = a ⊗ c + b ⊗ c (right-distributive).

∵

((a+b) ⊗ c)_i,j=(a+b)_i c_j=(a_i + b_i) c_j=a_i c_j + b_i c_j=(a ⊗ c)_i,j + (b⊗c)_i,j=(a ⊗ c+b ⊗ c)_i,j.

c (a \otimes b) = (c a) ⊗ b = a ⊗ (c b) .

∵

(c (a⊗b))_i,j=c (a⊗b)_i,j=c a_i b_j=(c a_i) b_j=a_i (c b_j)

Outer product of tensors satisfies:

(a⊗b) ⊗ c=a ⊗ (b⊗c) (associative).tr(a⊗b)=(a⊗b)_i,i=a_i b_ia ⊗_Kron b=vec(b ⊗_outer a)a ⊗_Kron b^t=a ⊗_outer b

Kronecker Product

a:f^n,m, b:f^l,k ⊢(a⊗b)_i,j≡a_i,j b

Example

(a₁, a₂) ⊗ (b₁, b₂)=(a₁ (b₁, b₂), a₂ (b₁, b₂))=((a₁ b₁, a₁ b₂), (a₂ b₁, a₂ b₂))

Properties

(a⊗b)_{p (r−1) + v,q (s−1) + w}=a_r,s b_v,w

Elementwise Product

Also called Hadamard product.

{(a \circ b)}_{i, j} \equiv a_{i, j} b_{i, j} .

Properties

{(a \circ b)}_{i, j} = a_{i, j} b_{i, j} = b_{i, j} a_{i, j} = {(b \circ a)}_{i, j} (commutative),

(a ∘ (b∘c))_i,j=a_i,j (b∘c)_i,j=a_i,j b_i,j c_i,j=(a∘b)_i,j c_i,j=((a∘b) ∘ c)_i,j(associative),(a ∘ (b+c))_i,j=a_i,j (b+c)_i,j=a_i,j (b_i,j + c_i,j)=a_i,j b_i,j + a_i,j c_i,j=(a ∘ b+a ∘ c)_i,j(distributive),((c a) ∘ b)_i,j=(c a)_i,j b_i,j=c a_i,j b_i,j=a_i,j (c b)_i,j=c (a_i,j b_i,j),a_i,j 0=0 a_i,j=0 or a ∘ 0_n=0_n ∘ a=0.

Khatri–Rao Product

a * b \equiv {(a_{i, j} ⊗ b_{i, j})}_{i, j} .

Example

a=((a_1,1, a_1,2), (a_2,1, a_2,2))a_1,1=((a_{1,1, 1,1}, a_{1,1, 1,2}), (a_{1,1, 2,1}, a_{1,1, 2,2}))a_1,2=((a_{1,2, 1,1},), (a_{1,2, 2,1},))a_2,1=((a_{2,1, 1,1}, a_{2,1, 1,2}),)a_2,2=((a_{2,2, 1,1},),)b=((b_1,1, b_1,2), (b_2,1, b_2,2))b_1,1=((b_{1,1, 1,1},),)b_1,2=((b_{1,2, 1,1}, b_{1,2, 1,2}),)b_2,1=((b_{2,1, 1,1},), (b_{2,1, 2,1},))b_2,2=((b_{2,2, 1,1}, b_{2,2, 1,2}), (b_{2,2, 2,1}, b_{2,2, 2,2}))a * b=((a_1,1 ⊗ b_1,1, a_1,2 ⊗ b_1,2),(a_2,1 ⊗ b_2,1, a_2,2 ⊗ b_2,2))

Tracy–Singh Product

a ∘ b=(a_i,j ∘ b)_i,j=((a_i,j ⊗ b_k,l)_k,l)_i,j

Determinant

\det (a) \equiv ε_{*} a_{1, i} a_{n, i} .

\det (a) = ε_{1_{1}, \dots, 1_{n}} a_{1, i_{1}} a_{n, i_{n}}

Properties

\det (1_{n}) = 1 .

\det (a^{t}) = \det (a) .

\det (c a) = c^{3} \det (a) .

\det (a b) = \det (a) \det (b) .

\det (u ⊗ v) = 0 .

Characteristic Polynomial

\begin{matrix} a & : & F^{n, n} & , \\ \det & ( & x & 1_{n} & - & a & ) \end{matrix}

Rank

Matrix Decomposition

Eigendecomposition/Spectral Decomposition

\begin{matrix} a & , & q & : & F^{n, n} & , \\ ( & i & ) & \to & a_{j, k} & x_{i, k} & = & c_{i, i} & x_{i, j} & , \\ or & a & = & q & c & q^{- 1} & . \end{matrix}

a,q:f^n,n, c_i,i is eigenvalue, c_i,j=0 if i ≠ j,a_j,k x_i,k=c_i,i x_i,j for each i. or a=q c q⁻¹,

Singular Value Decomposition

Dual Space

\begin{matrix} A & \subseteq & f & , \\ A^{*} & \equiv & A & \to & f & . \end{matrix}

\begin{matrix} f & , & g & : & A^{*} & , \\ ( & f & + & g & ) & ( & a & ) & = & f & ( & a & ) & + & g & ( & a & ) & , \\ ( & c & f & ) & ( & a & ) & = & c & f & ( & a & ) & . \end{matrix}

Transpose of a linear map

Linear Form

Integration

\begin{matrix} I & ( & f & + & g & ) & = & I & ( & f & ) & + & I & ( & g & ) & , \\ I & ( & a & f & ) & = & a & I & ( & f & ) & . \end{matrix}

Bilinear Form

\begin{matrix} A & , & B & , & C & \subseteq & f & , & f & : & A & × & B & \to & C & , \\ f & ( & a & + & b & , & c & ) & = & f & ( & a & , & c & ) & + & f & ( & b & , & c & ) & , \\ f & ( & c & a & , & b & ) & = & c & f & ( & a & , & b & ) & , \\ f & ( & a & , & b & + & c & ) & = & f & ( & a & , & b & ) & + & f & ( & a & , & c & ) & , \\ f & ( & a & , & c & b & ) & = & c & f & ( & a & , & b & ) & . \end{matrix}

Antisymmetric Bilinear Form

f (a, b) = −f (a, b) .

Bilinear Map

\begin{matrix} A & \subseteq & f & , & f & : & A & × & A & \to & f \\ f & ( & a & + & b & , & c & ) & = & f & ( & a & , & c & ) & + & f & ( & b & , & c & ) & , & f & ( & c & a & , & b & ) & = & c & f & ( & a & , & b & ) & , \\ f & ( & a & , & b & + & c & ) & = & f & ( & a & , & b & ) & + & f & ( & a & , & c & ) & , & f & ( & a & , & c & b & ) & = & c & f & ( & a & , & b & ) & . \end{matrix}

Alternating Bilinear Map

Submatrix/block matrix

a_{i,j, k,l}=(a_i,j)_k,l

Change of Bases

Covariance and Contravariance

Minimal Polynomial

\begin{matrix} a & : & F^{n, n} & , \\ μ_{a} & ( & a & ) & = & 0. \end{matrix}

\begin{matrix} μ_{a} & ( & a & ) & = & 0 \\ ≃ & \det & ( & λ & 1_{n} & − & a & ) & = & 0 \\ ≃ & a & x & = & λ & x & . \end{matrix}

Cayley–Hamilton Theorem

Householder Transformation

Numerical Linear Algebra

QR Decomposition

Jacobian Matrix

Hessian Matrix

Covariance and Contravariance

Active and passive transformation

Covariant transformation

Gramian matrix

Vectorisation

vec:f_m,n→f_{m n}vec(a)_i=a_{i % m,i / m}

i	⋯	m−1	m	⋯	m+m − 1	⋯	(n−1) m	⋯	(n−1) m+m − 1
i % m	⋯	m−1	0	⋯	m−1	⋯	0	⋯	0
i / m	⋯	0	1	⋯	1	⋯	n−1	⋯	n−1

Principle Invariants

\begin{matrix} a & : & F^{n, m} & , \\ I_{a} & \equiv & tr & ( & a & ) & \equiv & a_{i, i} & , \\ {II}_{a} & \equiv & ( & {tr (a)}^{2} & − & tr & ( & a^{2} & ) & ) & / 2, \\ {III}_{a} & \equiv & \det & ( & a & ) & , \end{matrix}

D_a I_a=1_n,D_a II_a=I_a 1_n − a,D_a III_a=III_a a^−t.