The singular values \(\{\sigma_1, \sigma_2, \dots, \sigma_m\}\) of a matrix \(A\) can be defined in several ways:

• the square roots of the eigenvalues of \(B=A^\star A\text{,}\)
• the lengths of the semiaxes of the hyperellipse that is the image of the unit circle under the linear transformation defined by \(A\text{.}\)
• The diagonal entries of the diagonal matrix \(\Sigma\) when we write \(A\) has a factorisation \(A=U \Sigma V^\star\) where $U$ and $V$ are unitary matrices, and $\Sigma$ is a diagonal matrix. (We'll be a little more precise below).

Please read Lecture 4 of Trefethen and Bau. T+B make the important observation in their introduction: the image of the unit sphere under any \(m \times n\) matrix is a hyperellipse.

Suppose that \(\{u_i\}_{i=1}^m\) is a set of orthonormal vectors, such that principal semiaxes of the hyperellipse are \(\{\sigma_i u_i\}\text{.}\) These scalars \(\sigma_1, \sigma_2, \dots, \sigma_m\) are the singular values of \(A\), and encapsulate many important properties of \(A\text{.}\)

The rest of the exposition followed Lecture 4 of Trefethen and Bau pretty closely. In summary, the SVD of \(A \in \Cmn\) is

where

• \(U=(u_1 | u_2 | u_3 | \dots | u_m) \text{,}\) the matrix that has as its columns the orthoronormal vectors in the direction of the principal semiaxes of the image of the unit sphere \(S \in \Rn \text{.}\)
• \(\Sigma = \diag(\sigma_1, \sigma_2, \dots, \sigma_n)\) is the diagonal matrix containing the lengths of the principal semiaxes. We order them as \begin{equation*} \sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_n \geq 0. \end{equation*}
• Then \(V=(v_1 | v_2 | v_3 | \dots | v_n)\) is the matrix whose columns are the preimages of the principal semiaxes, i.e., \(A v_i = \sigma_i u_i.\) It should be clear that the \(\{v_i\}\) form an orthonormal set, so \(V^{-1}=V^\star.\)