The Eckart-Young Theorem
The Eckart-Young Theorem: Given an n by m matrix X of rank r Â£ m Â£ n, and its singular value decomposition, ULV', where U is an n by m matrix, L is an m by m diagonal matrix of singular values, and V is an m by m matrix such that
U'U = In and V'V = VV' = Im
with the singular values arranged in decreasing sequence
l1Â³l2Â³l3Â³ ... Â³lmÂ³ 0
then there exists an n by m matrix B of rank s, s Â£ r, which minimizes the sum of the squared error between the elements of X and the corresponding elements of B when
B = ULsV'
where the diagonal elements of the m by m diagonal matrix Ls are
l1Â³l2Â³l3Â³ ... Â³ls > ls+1 = ls+2 = ... = lm = 0
Geometrically, the Eckart-Young Theorem states that the least squares approximation
in s dimensions of a matrix X can be found by replacing the smallest m-s roots of L with zeroes and remultiplying ULV'.
For example, given:
the rank 2 approximation is:
svd_example.r -- R Program to illustrate simple SVD.
svd_example.r looks like this:
# file: svd_example.r
# Purpose: Simple R program showing Singular
# Value Decomposition