What is singular value decomposition?

Singular value decomposition (SVD) is a mathematical technique that breaks down a matrix into three separate matrices, allowing us to understand the underlying structure of the original matrix.

Given a matrix $A$ of size $m \times n$, it's SVD is represented as:

where:

• $U$ is an $m \times m$, orthogonal matrixA matrix for which its inverse is its transpose. (columns are orthogonal unit vectors).

• $\sum$ is a diagonal matrix with non-negative real numbers on the diagonal. The elements on the diagonal are the square roots of the positive eigenvalues of $A.A^{T}$.

• $V^{T}$ is the transpose of an orthogonal matrix (rows are orthogonal unit vectors).

Example

Suppose we have the following matrix $A$:

And we want to calculate its SVD.

Calculating $V$

1. To find a matrix of the eigenvectors of $A \times A^{T}$, we'll first find $A \times A^{T}$:

2. After getting $A \times A^{T}$, we'll find its eigenvalues as follows:

Hence, the eigenvalues are $20$ and $80$.

3. Now we have the eigenvalues, we need to find the corresponding eigenvectors. Let's suppose $v_{1}$ and $v_{2}$ are two corresponding eigenvectors:

4. Now, we can say that:

Calculating $\sum$

The eigenvalues we got are $20$ and $80$. Hence, our ∑ matrix will be:

Calculating $U$

To calculate $U$, we can use the following relation:

Combining calculations

The final form of SVD after combining all the calculations is:

You can verify the calculations as follows:

Conclusion

SVD has vast applications in machine learning techniques such as dimensionality reduction, signal processing, or image compression. It can also be used to determine the transformation of an image matrix.

Note: Read about eigen value decomposition.