An Application of Linear Algebra

Kevin McGoldrick - Math 345, Fall 2009

Least Squares Estimation

The Key Result

Theorem 3: Given that χ is the least squares solution to the system Ax = b, then χ is also a solution to the system ATAx = ATb.  Thus, if A has linearly independent columns, the unique least squares solution is

χ = (ATA)-1ATb



Proof: Suppose χ is a least squares solution to the system Ax = b.  Then, multiplying both sides by AT gives ATAχ = ATb.  By Theorem 2, ATA is invertible, which implies that χ is a unique solution.  Multiplying both sides by (ATA)-1 gives the desired result.


With Theorem 3, we now have an efficient method of computing the least squares solution to an inconsistent system.  The inconsistency guarantees that the columns of A are linearly independent.  Thus, we need only compute the product (ATA)-1ATb to find χ.

The line above represents the least squares solution for the given points.