Skip to content

Operators on Inner Product Space

Orthogonal Complements

Theorem for null space and range of \(T^*\)

Suppose \(T\in \mathcal{L}(V, W)\), then

(i) \(N T^*=(R T)^\perp\).

(ii) \(R T^*=(NT)^\perp\).

\(\square\)

Singular value decomposition

For a linear map \(T \in \mathcal{L}(V,W)\), we could decompose it as we have for self-adjoint operator or normal operator.

Recall the important Riesz representation theorem in inner product space.

Riesz representation theorem

Assume \(V\) is finite-dimensional and \(\phi\) is a linear functional on \(V\), then there exists a unique vector \(v\in V\) such that

\[ \phi(u)=\langle v,u\rangle,\quad \forall v\in V. \]

\(\square\)

In functional analysis, we have actually a similar result for infinite-dimensional spaces.

The following lemma of \(T^*T\) is necessary.

Lemma

Properties of \(T^*T\). Suppose \(T \in \mathcal{L}(V,W)\)

(i) \(T^*T\) is a self-adjoint operator on \(V\). We could also check \(TT^*\) is a self-adjoint operator on \(W\).

(ii) \(N T^*T=N T\).

(iii) \(R T^*T = R T^*\).

(iv) dimension. \(\text{Dim} R T=\text{Dim} R T^*=\text{Dim}R T^*\).

(i) by definition.

\[ \langle T^*Tv, w\rangle=\langle Tv, Tw\rangle=\langle v, T^*T w\rangle\Rightarrow T^*T=(T^*T)^*. \]

(ii) \(N T\subset N T^*T\) is apparent. Assume \(v\in N T^*T\), \(T^*T v=0\), so \(\langle v, T^*Tv\rangle=0\), so \(\langle Tv, Tv\rangle=\|Tv\|=0\), which means \(Tv=0\).

(iii) \(R T^*T \subset R T^*T\) is apparent. For another direction, we use (ii) \(R T^*T=(N T^*T)^\perp=(N T)^\perp=R T^*\).

(iv) Use fundamental theorem of linear maps.

Definition of singular value

Assume a linear operator \(T\in \mathcal{L}(V, W)\), the singular values of \(T\) are defined as the nonnegative square roots of the eigenvalues of \(T^*T\), listed in decreasing order.

(SVD) Singular value decomposition

Assume a linear operator \(T\in \mathcal{L}(V, W)\), with its positive singular values \(s_1,\cdots, s_r\). Then there exists orthonormal lists \(e_1,\cdots, e_r\subset v\), \(f_1,\cdots, f_r\subset W\), such that

\[ Tv=\sum_{k=1}^r s_k \langle v, e_k\rangle f_k. \]

Here we denote that \(V\) and \(W\) is finite-dimensional. And the proof is constructive. This method also gives info about the eigenvectors construction.

Let \(s_1,\cdots,s_n\) to be the singular value of \(T(\text{Dim}V=n)\), where \(s_{r+1},\cdots, s_n\) are zero singular values.

  • Apply spectral thoerem to \(T^*T\) and there exists orthonormal basis \(e_1,\cdots, e_n\subset V\), such that
\[ T^*T e_k=s_k^2 e_k,\quad k=1,\cdots, n. \]
  • Define \(f_k=\frac{Te_k}{s_k}\) for \(k=1,\cdots, r\).

this is actually orthonormal basis in \(W\). This is also inspired by \(T=W\Sigma V^T\), so \(TV=W\Sigma\), so \(TV\Sigma^{-1}=W\), which shows a relationship of basis from \(V\) to \(W\) space.

  • Prove the proposition by expressing \(v\) in the constructed orthonormal basis
\[ \begin{align*} Tv&=T\left(\sum_{k=1}^n \langle v, e_k\rangle e_k\right)\\ &=\sum_{k=1}^n \langle v, e_k\rangle T e_k\\ &=\sum_{k=1}^r \langle v, e_k\rangle s_k f_k\\ \end{align*} \]

for \(k\geq r\), \(Te_k=0\) because \(T^*T e_k =0\cdot e_k\) and Property of self-adjoint \(T^*T\) (ii).

We could also check that the matrix with respect to basis \(\{e_k\}_{1\leq k\leq r}\) and \(\{f_k\}_{1\leq k\leq r}\) which should be extended.

Note we have \(\{e_k\}_{1\leq k\leq n}\), and from the above proof we have \(Te_k=s_k f_k\) for \(k\leq r\) and \(0\) for \(k >r\). We shall extend \(\{f_k\}_{1\leq k\leq r}\) to \(\{f_k\}_{1\leq k\leq m} (\text{Dim} W=m)\) by utilizing \(N T^*\). This is because we want to solve \(R(T)^\perp\), which equals \(NT^*\) by Theorem for null space and range of \(T^*\). (Readers should double check the dimension of \(N T^*\), which is \(m-r\), for \(\text{Dim} RT=r\).)

\(\square\)

Matrix version of SVD, a compact SVD form

Assume \(A\) is an \(m\)-by-\(n\) matrix of rank \(r\geq 1\). Then there exists an \(m\)-by-\(r\) matrix \(W\) with orthogonal columns, an \(r\)-by-\(r\) diagonal matrix \(\Sigma\) with positive numbers on the diagonal, and an \(n\)-by-\(r\) matrix \(V\) with orthonormal columns such that

\[ A=W\Sigma V^*. \]

Let \(T: \mathbb{F}^n\rightarrow \mathbb{F}^m\) whose matrix with respect to the standard basis equals \(A\). From the above proof of the SVD theorem, we have \(\text{Dim}RT=r\) and

\[ Tv=\sum_{k=1}^r s_k \langle v, e_k\rangle f_k. \]

we make use of the above structure. Let

\(W\) to be the \(m\)-by-\(r\) matrix whose columns are \(f_1,\cdots,f_r\),

\(\Sigma\) to be the \(r\)-by-\(r\) diagonal matrix \(\Sigma\) with entries \(s_1,\cdots,s_r\),

\(V\) to be the \(n\)-by-\(r\) matrix whose columns are \(e_1,\cdots,e_r\).

Choose \(u_k\), a standard base of \(\mathbb{F}^m\), then apply this matrix

\[ (AV-W\Sigma)u_k=Ae_k-Ws_k u_k=s_kf_k-s_kf_k=0. \]

so \(AV=W\Sigma\), multiply both sides by \(V^*\) and we have \(A=W\Sigma V^*\). But we have to be careful.

Here actually \(VV^*= I\) does not hold absolutely. We have to argue as follows. If \(k\leq r\), \(V^*e_k=u_k\), so \(VV^*e_k=e_k\). Thus \(AVV^*v=Av\) for all \(v\in \text{span}(e_1,\cdots,e_m)\). For \(v\in \text{span}(e_1,\cdots,e_m)^\perp\), we have \(Av=0\) and \(V^*v=0\), so we also have \(AVV^*v=Av=0\).

\(\square\)