An Application of Linear Algebra

Kevin McGoldrick - Math 345, Fall 2009

Least Squares Estimation

The Problem Defined

Suppose we have the following points of data, believed to be linearly related:

(2,5), (3,1), (-4,7), (-1,2),

 

Let m denote the slope of such a line and b denote the intercept.  For any point (x,y) we wish to have     y = mx + b.  Hence, we have the system depicted to the right.  Then, we have a system of the form

Ax = b.

 

In general, suppose we are given an inconsistent solution to the system of equations Ax = b.  Define the error vector ε = b— Ax.  It seems natural that the best solution to our system will minimize the norm, or magnitude of the  error, ||ε|| (Note: norms are computed with the Euclidean inner product, e.g. ||(e1,…,en)|| = (e12 + … + en2)(1/2))

 

As such, let χ be the solution which makes ||ε|| as small as possible.  The vector χ is known as the least squares solution.

The vertical distance between a point and the pictured best fit line is the magnitude of each individual error.