Hilbert Space

We first introduce inner product in abstract space.

Inner Product

Assume $\mathcal{U}$ is a linear space on number field $\mathbb{K}$ (i.e. $\mathbb{R}$ or $\mathbb{C}$). A mapping

$$(\cdot ,\cdot ):\mathcal{U}\times \mathcal{U}\mapsto \mathbb{K}$$

is said to be a Inner Product, if it satisfies that, for all $x,y,z\in \mathcal{U}$, for all $\alpha \in \mathbb{K}$,

(i) Homogeneous on the first element. $(\alpha x,y)=\alpha (x,y)$.

(ii) Linearity on the first element. $(x+z,y)=(x,y)+(z,y)$.

(iii) Conjugate inverse. $(x,y)=\overline{(y,x)}$.

(iv) Positive-definite. $(x,x)\ge 0$ and $(x,x)=0\iff x=\theta$.

And A linear space $\mathcal{U}$ with inner product is called Inner Product Space.

With the above definition, we have the following properties of inner product.

Properties of inner product

Assume $\mathcal{U}$ is an inner product space on number field $\mathbb{K}$. Then $x,y,z\in \mathcal{U}$, for all $\alpha \in \mathbb{K}$,

(i) Conjugate Homogeneous on the second element. $(x,\alpha y)=\bar{\alpha }(x,y)$.

(ii) Linearity on the second element. $(x,y+z)=(x,y)+(x,z)$.

(iii) Zero for null vector. $(x,\theta )=(x,0\cdot x)=0(x,x)=0$.

(iv) Normed linear space. Define $\Vert x\Vert =\sqrt{(x,x)}$, then it is a norm.

(v) $(x,y)$ is a continuous function with respect to both $x,y$.

ProofProof for (v)

Easy to show that (i), (ii), (iii).

We prove for (iv) and the triangle inequation. This is based on Schwarz Inequation.

$$|(x,y){|}^2\le \Vert x{\Vert }^2\Vert y{\Vert }^2$$

We still make use of $\mathrm{\forall }\lambda \in \mathbb{K}$

$$\begin{array}{rl}0& \le \Vert x+\lambda y{\Vert }^2\\ & =(x+\lambda y,x+\lambda y)\\ & =\Vert x{\Vert }^2+\bar{\lambda }(x,y)+\lambda (y,x)+|\lambda {|}^2\Vert y{\Vert }^2\\ & =\Vert x{\Vert }^2+\lambda (y,x)+\bar{\lambda }[(x,y)+\lambda \Vert y{\Vert }^2]\end{array}$$

Assume $(y,y)\ne 0$, and choose $\lambda =-\frac{(x,y)}{\Vert y{\Vert }^2}$, we have

$$\begin{array}{rl}0& \le \Vert x{\Vert }^2-\frac{(x,y)}{\Vert y{\Vert }^2}(y,x)\\ |(x,y){|}^2& \le \Vert x{\Vert }^2\Vert y{\Vert }^2\end{array}$$

With Schwarz inequation, we have

$$\begin{array}{rl}\Vert x+y{\Vert }^2& =|(x+y,x+y)|\\ & =|(x+y,y)+(x+y,x)|\\ & \le |(x+y,y)|+|(x+y,x)|\\ & \le \Vert x+y\Vert \Vert y\Vert +\Vert x+y\Vert \Vert x\Vert \text{Schwarz Inequation}\end{array}$$

and we are done.

Polarization Identity

(i) For $\mathbb{K}=\mathbb{R}$, we have

$$(x,y)=\frac{1}{4}(\Vert x+y{\Vert }^2-\Vert x-y{\Vert }^2).$$

(ii) For $\mathbb{K}=\mathbb{C}$, we have

$$(x,y)=\frac{1}{4}(\Vert x+y{\Vert }^2-\Vert x-y{\Vert }^2+i\Vert x+iy{\Vert }^2-i\Vert x-iy{\Vert }^2).$$

Proof

(i) Just expand the righthand item in inner product.

(ii) Still the same logic, but

$$\Vert x+y{\Vert }^2-\Vert x-y{\Vert }^2=2(x,y)+2\overline{(x,y)}=4Re[(x,y)].$$

We notice $Re[(x,iy)]=Re[-i(x,y)]=Im[(x,y)]$, so apply $y=iy$ in the above equation and we have

$$4Im[(x,y)]=4Re[(x,iy)]=\Vert x+iy{\Vert }^2-\Vert x-iy{\Vert }^2.$$

$◻$

Hilbert Space

If a inner product space $\mathcal{U}$ is complete, then it is called Hilbert Space.

We have known that a inner product space must be a normed linear space, then how about the inverse? Generally speaking, a normed linear space could not be a inner product space, i.e. use the inner product to deduce a norm, unless we add the following condition.

Parallelogram Law

If a norm satisfies

$$\Vert x+y{\Vert }^2+\Vert x-y{\Vert }^2=2(\Vert x{\Vert }^2+\Vert y{\Vert }^2)$$

then we could define a inner product using polarization identity.

Inversely speaking, a norm introduced by a norm satisfy the above condition, which is called Parallelogram Law.

ProofLemmaProof for Necessary

• Sufficient.

Just expand lefthand of the equation using inner product and we have the result.

• Necessary.

We first show that the inner product defined by Polar equation satisfies linearity.

$$\begin{array}{rl}(x,z)+(y,z)& =\frac{1}{4}(\Vert x+z{\Vert }^2-\Vert x-z{\Vert }^2+i\Vert x+iz{\Vert }^2-i\Vert x-iz{\Vert }^2)\\ & +\frac{1}{4}(\Vert y+z{\Vert }^2-\Vert y-z{\Vert }^2+i\Vert y+iz{\Vert }^2-i\Vert y-iz{\Vert }^2)\\ & =\frac{1}{8}(\Vert x+y+2z{\Vert }^2-\Vert x+y-2z{\Vert }^2+i\Vert x+y+i2z{\Vert }^2-i\Vert x+y-i2z{\Vert }^2)\\ & =\frac{1}{2}(\Vert \frac{x+y}{2}+z{\Vert }^2-\Vert \frac{x+y}{2}-z{\Vert }^2+i\Vert \frac{x+y}{2}+iz{\Vert }^2-i\Vert \frac{x+y}{2}-iz{\Vert }^2)\\ & =2(\frac{x+y}{2},z)\\ & =\frac{1}{2}(x+y,2z)\text{this is not good}.\end{array}$$

Let $y=\theta$, we have $(x,z)=2(\frac{x}{2},z)$. Choose $x=x+y$, we have $(x+y,z)=2(\frac{x+y}{2},z)$. Comparing with before and we have

$$(x,z)+(y,z)=(x+y,z).$$

So let $y=y-x$, we have

$$(x,z)+(y-x,z)=(y,z)\Rightarrow (y-x,z)=(y,z)-(x,z)$$

which means "+" and "-" satisfy.

This is a little tricky, and we have to show a lemma.

For a continuous function $f(\alpha )$ defined on $\mathbb{R}$, which satisfies

$$f({\alpha }_1+{\alpha }_2)=f({\alpha }_1)+f({\alpha }_2),\mathrm{\forall }{\alpha }_1,{\alpha }_2\in \mathbb{R},$$

then $\mathrm{\forall }\alpha \in \mathbb{R}$, we have

$$f(\alpha )=\alpha f(1).$$

Let

$$f(\alpha )=(\alpha x,z).$$

it is continuous because

$$\begin{array}{rl}|f(\alpha )-f({\alpha }_0)|& =|((\alpha -{\alpha }_0)x,z)|\text{subtraction}\end{array}$$

For $\alpha \to {\alpha }_0$, $(\alpha -{\alpha }_0)x\to 0x=\theta$, so the lefthand of the above equation tends to $0$.

And

$$f({\alpha }_1+{\alpha }_2)=(({\alpha }_1+{\alpha }_2)x,z)=({\alpha }_{1}x+{\alpha }_{2}x,z)=({\alpha }_{1}x,z)+({\alpha }_{2}x,z),$$

so we have

$$(\alpha x,z)=\alpha (x,z),\mathrm{\forall }\alpha \in \mathbb{R},$$

which is homogeneity.

As for conjugate property and positive-definitivity, just follow the definition.

Orthogonal System

Orthogomal System

Assume $\mathcal{U}$ is a inner product space, $x,y\in \mathcal{U}$. If $(x,y)=0$, then we call $x$ is orthogamal to $y$, denoted by $x\perp y$. If a subset $M\subset \mathcal{U}$, $\mathrm{\forall }y\in M$, $x\perp y$, then we call $x$ is orthogonal to $M$, denoted by $x\perp M$. If $N\subset \mathcal{U}$, $\mathrm{\forall }x\in N$, $x\perp M$, then we call $N$ is orthogonal to $M$, denoted by $N\perp M$.

An orthogonal complement of $M$ is denoted by

$$M^{\perp }=\{x\in \mathcal{U}:x\perp M\}.$$

Property of Orthogomal

(i) Pythagorean Theorem. For $x_1,\cdots ,x_n$ which are mutually orthogonal, then $x:=x_1+\cdots +x_n$, we have

$$\Vert x{\Vert }^2=\sum _{i=1}^n\Vert x_i{\Vert }^2.$$

(ii) Dense set with Zero Vector. Assume $L\subset \mathcal{U}$ is a dense subset, if $x\in \mathcal{U}$ is orthogonal to $L$, then $x=0$.

(iii) For $M\subset \mathcal{U}$, $M^{\perp }$ is a closed subset.

Orthogomal Projection

We introudce orthogonal projection using the best approximation element.

Definition of best Approximation element

Assume $M\subset \mathcal{U}$, $x\in \mathcal{U}$, $y\in \mathcal{U}$ is called the best approximation element of $x$ in $M$, if

$$\Vert x-y\Vert =\underset{z\in M}{inf}\Vert x-z\Vert$$

Here we give a sufficient condition for the existence and uniqueness of best approximation element.

Theorem for existence and uniqueness of best approximation element

Assume $M\subset \mathcal{U}$, $x\in \mathcal{U}$. If $M$ is a closed convex set, then there exists a unique best approximation element $y$ of $x$ in $M$.

Proof

This proof leverages the convex property.

From the above theorem, A subspace is naturally convex, thus adding closedness could satisfy the above condition. So

Orthogomal decomposition and projection

Assume $M\subset \mathcal{U}$ is a closed subspace, then $\mathrm{\forall }x\in \mathcal{U}$, $\mathrm{\exists }!y\in M,z\in M^{\perp }$, such that

$$x=y+z.$$

the above equation is called orthogonal decomposition, and $y$ is called the orthogonal projection of $x$ in $M$.

Normalized Orthogomal System

Definition of Normalized Orthogomal System

System $\{e_n{\}}_{n\ge 1}\subset \mathcal{U}$ is called Normalized Orthogomal System, if it satisfies

$$(e_n,e_m)=\{\begin{array}{l}1,n=m,\\ 0,n\ne m.\end{array}$$

For a element $x\in \mathcal{U}$, define

$$c_n=(x,e_n)$$

to be the Fourier coefficient of $x$. They also form a system $\{c_n\}$.

Now we talk a little deeper into this one.

Explicite expression of orthogonal decomposition

Assume $n$ is given, $\{e_1,\cdots ,e_n\}$ is a finite-number normalized orthogonal system. Let $M=\text{span}(e_1,\cdots ,e_n)$, then $\mathrm{\forall }x\in \mathcal{U}$, we have the orthogonal projection of $x$ on $M$

$$x=\sum _{i=1}^n(x,e_n)e_n$$

Proof

First use Orthogomal decomposition and projection to show that

$$x=y+z,y\in \text{space}(e_1,\cdots ,e_n).$$

So by property of base, $y=\sum _{i=1}^n{a}_n{e}_n$. Making a inner product with $e_j$ on both sides gives

$$a_j=(y,e_j)$$

So we are done.

Bessel Inequation

For a normalized orthogonal system $\{e_n\}\subset \mathcal{U}$, $x\in \mathcal{U}$ with its Fourier coefficients $\{c_n\}$, we have

$$\sum _{n=1}^{\mathrm{\infty }}\Vert (x,e_n){\Vert }^2\le \Vert x{\Vert }^2.$$

or $\{e_n\}\subset l^2$.

Proof

Use finite many elements and then let $n\to \mathrm{\infty }$.

Inversely speaking, given a sequence in $\{c_n\}\subset l^2$, could we find a element in $x\in \mathcal{U}$ such that $c_n$ is its fourier coefficient? It turns out that we need to add another condition.

Riesz-Fischer Theorem

Assume $\mathcal{U}$ is a Hilbert space, $\{e_n\}$ are its normalized orthogonal system, number sequence $\{c_n\}\subset l^2$. Then there exists $x\in \mathcal{U}$, such that for each $n$, we have

$$c_n=(x,e_n)$$

and Parseval equation holds

$$\sum _{i=1}^{\mathrm{\infty }}\Vert c_n{\Vert }^2=\sum _{i=1}^{\mathrm{\infty }}\Vert (x,e_n){\Vert }^2=\Vert x{\Vert }^2$$

Proof

$◻$

Now we pause here. It is interesting to see that Bessel inequation holds for all inner product space, but Parseval equation does not also hold. We dig deeper into this and this is actually related to the number of $\{e_n\}$. For a Hilbert space, its number is enough to let "=" hold.

Completeness

Definition for completeness of orthogonal system

Assume $\mathcal{U}$ is a inner product space. An normalized orthogonal system is said to be complete, if for all $x\in \mathcal{U}$, Parseval equation holds; it is said to be total, if it is dense in $\mathcal{U}$.

Relationship

Assume $\mathcal{U}$ is a Hilbert space. $\{e_n\}$ is a normalized orthogonal system. Then we have the following equivalence.

(i) $\{e_n\}$ is complete.

(ii) $\mathrm{\forall }x\in \mathcal{U}$, series $\sum _{i=1}^n(x,e_i)e_i$ converges to $x$, i.e.

$$x=\sum _{n=1}^{\mathrm{\infty }}(x,e_n)e_n.$$

(iii) $\mathrm{\forall }x,y\in \mathcal{U}$,

$$(x,y)=\sum _{n=1}^{\mathrm{\infty }}(x,e_n)\overline{(y,e_n)}.$$

(iv) $\{e_n\}$ is total.

(i) $\Rightarrow$ (ii)(ii) $\Rightarrow$ (iii)(iii) $\Rightarrow$ (iv)(iv) $\Rightarrow$ (i)

(i) and (ii) are equivalent because

$${\Vert x-\sum _{n=1}^N(x,e_n)e_n\Vert }^2=\Vert x{\Vert }^2-\sum _{n=1}^N|(x,e_n){|}^2$$

Using equation from (ii) we only get infinite series. We start from finite-numbers. Define

$$x_n=\sum _{i=1}^n(x,e_i)e_i,y_n=\sum _{j=1}^n(y,e_j)e_j,$$

then

$$\begin{array}{rl}(x_n,y_n)& =(\sum _{i=1}^n(x,e_i)e_i,\sum _{j=1}^n(y,e_j)e_j)\\ & =\sum _{i=1}^n(x,e_n)\overline{(y,e_n)}.\end{array}$$

Let $n\to \mathrm{\infty }$, since by hölder inequation

$${(\sum _{i=1}^{\mathrm{\infty }}|(x,e_n)\overline{(y,e_n)}|)}^2\le (\sum _{i=1}^{\mathrm{\infty }}|(x,e_n){|}^2)(\sum _{i=1}^{\mathrm{\infty }}|(y,e_n){|}^2)$$

by (ii) the above converges. By continuity of inner product, we are done.

Assume $\mathrm{\forall }x\in \mathcal{U}$, $(x,e_n)=0$. So $\mathrm{\forall }y\in \mathcal{U}$

$$(x,y)=\sum _{n=1}^{\mathrm{\infty }}(x,e_n)\overline{(y,e_n)}=0.$$

Choose $y=x$ and $(x,x)=0$ implies $x=\theta$.

$\mathrm{\forall }x\in \mathcal{U}$, by Bessel ineuqation, $\sum _{n=1}^{\mathrm{\infty }}|(x,e_n){|}^2<\mathrm{\infty }$. For number sequence $\{(x,e_n)\}\subset l^2$, by Riesz-Fischer Theorem, $\mathrm{\exists }y\in \mathcal{U}$, such that $(x,e_n)=(y,e_n)$ for each $n$. So $(x-y,e_n)=0$ for each $n$, by (iv) assumption, we have $x-y=\theta$, i.e. $x=y$ and $x$ satisfies

$$\Vert x{\Vert }^2=\Vert y{\Vert }^2=\sum _{i=1}^{\mathrm{\infty }}|(x,e_n){|}^2.$$