An Application of Linear Algebra

Kevin McGoldrick - Math 345, Fall 2009

Least Squares Estimation

The Necessary Linear Algebra

Here we present some results allowing us to derive a formula for χ.

Theorem 1: Suppose V is a finite dimensional inner product space, with subspace W.  Let u be any vector in V.  The “best approximation” from u to W is the orthogonal projection of u onto W, denoted projW u.  In other words, for every w in W such that w is not projW u, we have

||u-projW u|| < ||u-w||.

Proof: For any w in W, we have

u-w = (u-(projW u)) + ((projW u)-w).

Note that ((projW u)-w) is a vector in W, and that

(u-(projW u)) is orthogonal to any vector in W.  Now,

||u-w||2 = ||(u-(projW u)) + ((projW u)-w)||2

= ||(u-(projW u))||2 + ||((projW u)-w)||2


Since ||((projW u)-w)||2 > 0, then

||u-w||2 > ||(u-(projW u))||2 ,


||u-w|| > ||(u-(projW u))||

as desired.

Here, P is the orthogonal projection of V onto the plane.

(P = projW v)