Advance MultiVariant Linear Algebra

When Geometry Becomes Dangerous

---

Orthogonality = Independence

Orthogonality representing perpendicularity, where elements (vectors, functions, or data) are independent and have a dot product or inner product of zero.

Two vectors are orthogonal if:

$$v^T w = 0$$

Or their dot product is zero. ie they form a 90 angle or are perpendicular.

$$v.w =0$$

In ML, orthogonality means:

• No linear dependence
• No shared directional information

This is why:

• PCA finds orthogonal principal components
• QR decomposition builds orthogonal bases
• SVD decomposes into orthogonal directions

Orthogonality reduces redundancy.

Eigenvectors

Non-zero vectors associated with a linear transformation (represented by a square matrix) that do not change their direction when that transformation is applied.

Instead, they are only

• scaled—stretched
• compressed
• reversed

by a scalar factor known as the eigenvalue $\lambda$

If:

$$Av = \lambda v$$

That means:

• Applying transformation $A$
• Does not change the direction of $v$
• Only scales it by $\lambda$

Eigenvectors are:

• Directions that remain stable under transformation.
• Natural Directions of a Transformation

In ML:

• PCA uses eigenvectors of the covariance matrix
• These represent directions of maximum variance
• Each eigenvector = principal axis of data

Eigenvalues tell you how important that direction is.

Singular Value Decomposition (SVD)

Factorization of a real or complex matrix into a rotation, followed by a rescaling followed by another rotation.

SVD states:

$$X = U \Sigma V^T$$

Geometrically:

1. $V^T$ rotates the space
2. $Sigma$ scales each axis
3. $U$ rotates again

So any matrix transformation can be seen as:

Rotate → Stretch → Rotate

This is why SVD is foundational in:

• Dimensionality reduction
• Embeddings
• LLM weight compression
• Recommender systems

Neural Networks as Layered Transformations

Each layer computes:

$$z = Wx + b$$

Which is:

1. Linear transformation ( W )
2. Shift ( b )
3. Nonlinear activation

Geometrically:

• Linear layers reshape space
• Activations bend space
• Deep networks progressively warp geometry

Training adjusts ( W ) so that:

• Classes become linearly separable
• Desired outputs align with target directions

Deep learning is geometry engineering.

---

A matrix is a transformation of space.

All machine learning models are compositions of transformations.

If:

$$y = Ax$$

Then $A$ transforms vector $x$ into a new vector $y$.

Geometrically, a matrix can:

• Stretch
• Compress
• Rotate
• Reflect
• Shear
• Project

Linear Regression = Projection

When solving linear regression:

$$\hat{\theta} = (X^T X)^{-1} X^T y$$

You are not just solving equations.

You are projecting vector ( y ) onto the column space of ( X ).

Meaning:

• ( X ) defines a subspace (all linear combinations of features)
• ( y ) may not lie in that space
• We find the closest point in that space

This closest point is the orthogonal projection.

The residual error is perpendicular to the feature space.

Mathematically:

$$X^T (y - X\hat{\theta}) = 0$$

Geometric meaning:

The error vector is orthogonal to every feature direction.

---