The Eckart-Young Theorem

The Eckart-Young Theorem: Given an n by m matrix X of rank r £ m £ n, and its singular value decomposition, ULV', where U is an n by m matrix, L is an m by m diagonal matrix of singular values, and V is an m by m matrix such that

U'U = In and V'V = VV' = Im

with the singular values arranged in decreasing sequence

l1³l2³l3³ ... ³lm³ 0

then there exists an n by m matrix B of rank s, s £ r, which minimizes the sum of the squared error between the elements of X and the corresponding elements of B when

B = ULsV'

where the diagonal elements of the m by m diagonal matrix Ls are

l1³l2³l3³ ... ³ls > ls+1 = ls+2 = ... = lm = 0

Geometrically, the Eckart-Young Theorem states that the least squares approximation in s dimensions of a matrix X can be found by replacing the smallest m-s roots of L with zeroes and remultiplying ULV'.

For example, given:

the rank 2 approximation is:

svd_example.r -- R Program to illustrate simple SVD.
svd_matrix.txt -- Simple 4 by 3 matrix.
svd_example.r looks like this:
# file: svd_example.r
# Purpose: Simple R program showing Singular
#          Value Decomposition