Advanced Maths for Machine Learning and Data Science

Linear algebra is the language of space manipulation.

Machine learning is controlled geometric transformation.

Learning Path

deeplearning.ai Mathematics for ML and Data Science Specialization

1. Advanced Linear Algebra ∑

• Week 1: Systems of linear equations
• Week 2: Solving systems of linear equations
• Week 3: Vectors and Linear Transformations
• Week 4: Determinants and Eigenvectors

2. Calculus ∫

• Week 1: Derivatives and Optimization
• Week 2: Gradients and Gradient Descent
• Week 3: Optimization in Neural Networks and Newton's Method

3. Probability & Statistics 🎲

• Week 1: Introduction to Probability and Probability Distributions
• Week 2: Probability distributions with multiple variables
• Week 3: Sampling and Point estimation
• Week 4: Confidence Intervals and Hypothesis testing

---

Why Linear Algebra Matters in ML

Machine learning deals with multiple features, large datasets, efficient computation, and vectorized operations. Instead of writing nested loops, we use matrix operations to compute predictions and updates efficiently.

Think of ML as:

• Data → points in high-dimensional space

• Features → axes

• Models → transformations

• Training → adjusting geometry

• Loss → distance between vectors

• Optimization → walking downhill in space

• Gradient descent becomes directional movement

• Regularization becomes shrinking vector norms

• Overfitting becomes high-dimensional distortion

• RAG embeddings become spatial similarity

• Attention becomes weighted projection

---

Mathematical Objects

A mathematical object is an abstract concept that can be assigned to a symbol and involved in formulas — numbers, expressions, shapes, functions, and sets.

Tensor

An algebraic object describing a multilinear relationship between sets of algebraic objects associated with a vector space.

---

Scalar ($s \in \mathbb{R}$)

A single number — a 0-dimensional tensor.

Examples: $5$, $-3.14$, $\pi$, $e$

---

Matrices ($A \in \mathbb{R}^{n \times m}$)

A matrix is a 2D array of numbers.

$$A = \begin{bmatrix} 85 & 76 & 66 & 5 \\ 94 & 75 & 18 & 28 \\ 68 & 40 & 71 & 5 \end{bmatrix}$$

In mathematics: $A \in \mathbb{R}^{3 \times 4}$ — real-valued matrix, 3 rows, 4 columns.

In programming:

A = np.array([[85, 76, 66, 5],
              [94, 75, 18, 28],
              [68, 40, 71, 5]])

In theory:

• Uppercase letters (A, B, X) → Matrices
• Lowercase letters (x, y, z) → Vectors or scalars

Matrix Size

A ∈ ℝᵐˣⁿ → Matrix A has m rows and n columns

• Square matrix — same number of rows and columns ($m = n$)

Element Notation

$A_{ij}$ — element in the $i$-th row and $j$-th column.

• $A_{11} = 85$ → Row 1, Column 1
• $A_{32} = 40$ → Row 3, Column 2

Use in ML

The data matrix $X \in \mathbb{R}^{m \times n}$ where each row is a training example and each column is a feature.

---

Vectors ($\vec{x}$)

A vector is a matrix with 1 column — a point in $n$-dimensional space with direction and magnitude.

$$x = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}$$

In math: $y \in \mathbb{R}^4$

In programming: y = np.array([460, 232, 315, 178])

Dimension: $n \times 1$

In ML, vectors represent:

• A data point
• A feature column (direction)
• A model weight vector (direction of best fit)

Element Indexing

$y_i$ — the $i$-th element. Linear algebra uses 1-indexed notation by default.

Vector Norms

L1 norm: $\|v\|_1 = \sum |v_i|$

L2 norm (Euclidean): $\|v\|_2 = \sqrt{\sum v_i^2}$

---

Transpose ($\mathbf{A}^T$)

Transpose swaps rows and columns.

If $A \in \mathbb{R}^{m \times n}$ then $A^T \in \mathbb{R}^{n \times m}$

$(A^T)_{ij} = A_{ji}$

$$A = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix} \quad \Rightarrow \quad A^T = \begin{bmatrix} 1 & 4 \\ 2 & 5 \\ 3 & 6 \end{bmatrix}$$

Used in: Normal Equation, gradient derivations.

---

Identity Matrix ($I$)

The matrix equivalent of the number 1 — 1s on the diagonal, 0s everywhere else.

$$I = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \qquad AI = IA = A$$

---

Inverse Matrix ($A^{-1}$)

Matrix inverse is like division. Only square matrices can have inverses.

$A^{-1}A = AA^{-1} = I$

Used in the Normal Equation: $\theta = (X^T X)^{-1} X^T y$

Invertible vs Non-Invertible

	Type	Condition
Invertible / non-singular	Has an inverse	Full rank — rows and columns linearly independent
Non-invertible / singular	No inverse	Rows or columns linearly dependent

Common causes of non-invertibility:

• Redundant features (e.g., size in feet and meters — $x_2 = kx_1$)
• More features than training examples ($m \leq n$) — use regularization

MATLAB/Octave:

• pinv(A) — pseudo-inverse (works even if singular)
• inv(A) — standard inverse

---

Determinant ($\det(A)$)

Tells us whether a matrix is invertible.

For a 2×2 matrix: $\det(A) = ad - bc$

• $\det(A) \neq 0$ → invertible
• $\det(A) = 0$ → singular (no unique solution or infinitely many)

Regularization (Ridge Regression) adds a small value to the diagonal to ensure invertibility in practice.

---

Matrix as a Transformation

A matrix is a transformation of space. All ML models are compositions of transformations.

$y = Ax$

$A$ transforms vector $x$ into a new vector $y$. Geometrically, a matrix can stretch, compress, rotate, reflect, shear, or project.

T : ℝ² → ℝ³ → T maps 2D vectors to 3D space

---

Matrix Addition / Subtraction

Addition is element-wise. Both matrices must have the same dimensions.

$$(A + B)_{ij} = A_{ij} + B_{ij}$$ $$\begin{bmatrix}1 & 2 \\ 3 & 4\end{bmatrix} + \begin{bmatrix}5 & 6 \\ 7 & 8\end{bmatrix} = \begin{bmatrix}6 & 8 \\ 10 & 12\end{bmatrix}$$

---

Matrix Multiplication ($AB$)

Given $A \in \mathbb{R}^{m \times n}$ and $B \in \mathbb{R}^{n \times p}$:

$C = AB \in \mathbb{R}^{m \times p} \qquad C_{ij} = \sum_k A_{ik} B_{kj}$

Inner dimensions must match: $(m \times n)(n \times p) \rightarrow (m \times p)$

Properties:

• Not commutative: $AB \ne BA$
• Associative: $(AB)C = A(BC)$

Use in ML

Everything in deep learning is matrix multiplication:

• Inputs × Weights → activations
• Neural net forward pass: $Z = WX + b$
• Backpropagation is matrix calculus

---

Vectorization

If $X$ is $m \times n$ and $\theta$ is $n \times 1$:

$h = X\theta \qquad (m \times 1 \text{ predictions})$

This computes predictions for all training examples in one operation — optimized for CPU/GPU hardware.

Linear regression hypothesis for all examples: $h = X\theta \qquad \text{where } h_\theta(x) = \theta^T x$

---

Dot Product ($a \cdot b$)

The dot product of two $n$-dimensional vectors:

$x^T y = x_1 y_1 + x_2 y_2 + \dots + x_n y_n$

Produces a scalar.

---

Summary

Concept	Role in ML
Vectors	Features, parameters, predictions
Matrices	Datasets, weight tensors, transformations
Transpose	Normal equation, gradient math
Inverse	Closed-form solutions
Matrix multiplication	Forward pass, vectorized predictions
Dot product	Similarity, attention scores
Determinant	Check invertibility