Interpolation and Extrapolation

|  This page has been nominated for cleanup for the following reason: needs to be in course format. This course is still very much in the rough! .
Please edit this page to improve it. See this module's talk page for discussion.

|

This course belongs to the track Numerical Algorithms in the Department of Scientific Computing in the School of Computer Science.

In this course, students will learn how to approximate discrete data. Different representations and techniques for the approximation are discussed.

## Contents

* 1 Discrete Data
* 2 Polynomial Interpolation
* 3 The Vandermonde Matrix
* 4 Lagrange Interpolation
* 5 Newton Interpolation
* 6 Interpolation error
* 6.1 Proof
* 7 Switching Between Representations
* 8 Arbitrary Basis Functions
* 9 Piecewise Interpolation / Splines
* 10 Radial Basis Functions
* 11 Extrapolation
* 12 Exercises

## Discrete Data[edit | edit source]

Discrete data arises from measurements of  f ( x ) {\displaystyle f(x)} or when  f ( x ) {\displaystyle f(x)} is difficult or expensive to evaluate.

In general, discrete data is given by a set of nodes  x i {\displaystyle x_{i}} ,  i = 1 … n {\displaystyle i=1\dots n} and the function  f ( x ) {\displaystyle f(x)} evaluated at these points:  f i = f ( x i ) {\displaystyle f_{i}=f(x_{i})} ,  i = 1 … n {\displaystyle i=1\dots n} .

In the following, we will refer to the individual nodes and function values as  x i {\displaystyle x_{i}} and  f i {\displaystyle f_{i}} respectively. The vectors of the  n {\displaystyle n} nodes or function values are written as  x _ {\displaystyle {\underline {\mathbf {x} }}} and  f _ {\displaystyle {\underline {\mathbf {f} }}} respectively.

We will also always assume that the  x i {\displaystyle x_{i}} are distinct.

## Polynomial Interpolation[edit | edit source]

Although there are many ways of representing a function  f ( x ) {\displaystyle f(x)} , polynomials are the tool of choice for interpolation. As the Taylor series expansion of a function  f ( x ) {\displaystyle f(x)} around  0 {\displaystyle 0} shows, every continuous, differentiable function has a polynomial representation

f ( x ) {\displaystyle f(x)} |  = {\displaystyle =} |  f ( 0 ) + f ′ ( 0 ) x + 1 2 f ″ ( 0 ) x 2 + 1 6 f ‴ ( 0 ) x 3 + … {\displaystyle f(0)+f'(0)x+{\frac {1}{2}}f''(0)x^{2}+{\frac {1}{6}}f'''(0)x^{3}+\dots }

|
|

|  = {\displaystyle =} |  ∑ n = 0 ∞ 1 n ! f ( n ) ( 0 ) ⏟ = a n x n {\displaystyle \sum _{n=0}^{\infty }\underbrace {{\frac {1}{n!}}f^{(n)}(0)} _{=a_{n}}x^{n}}
|  = {\displaystyle =} |  ∑ n = 0 ∞ a n x n {\displaystyle \sum _{n=0}^{\infty }a_{n}x^{n}}

in which the  a n = 1 n ! f ( n ) ( 0 ) {\displaystyle a_{n}={\frac {1}{n!}}f^{(n)}(0)} are the exact weights.

If a function  f ( x ) {\displaystyle f(x)} is assumed to be continuous and differentiable and we assume that it's derivatives  f ( n ) ( 0 ) {\displaystyle f^{(n)}(0)} are zero or of negligible magnitude for increasing  n {\displaystyle n} , it is a good idea to approximate the function with a polynomial.

Given  n {\displaystyle n} distinct nodes  x i {\displaystyle x_{i}} ,  i = 1 … n {\displaystyle i=1\dots n} at which the function values  f i = f ( x i ) {\displaystyle f_{i}=f(x_{i})} are known, we can construct an interpolating polynomial g ( x ) {\displaystyle g(x)} such that  g ( x i ) = f i {\displaystyle g(x_{i})=f_{i}} for all  i = 1 … n {\displaystyle i=1\dots n} . Such an interpolating polynomial with degree  ≤ n {\displaystyle \leq n} is unique. However infinitely many such polynomials can be found with degree  > n {\displaystyle >n\;} .

Any nice way of deriving the interpolation error?

## The Vandermonde Matrix[edit | edit source]

By definition as given above, our polynomial

g ( x ) = ∑ i = 0 n − 1 a i x i {\displaystyle g(x)=\sum _{i=0}^{n-1}a_{i}x^{i}}

must interpolate the function  f ( x ) {\displaystyle f(x)} at the  n {\displaystyle n} nodes  x i {\displaystyle x_{i}} . This leads to the following system of  n {\displaystyle n} linear equations in the coefficients  a i {\displaystyle a_{i}} :

f ( x 1 ) {\displaystyle f(x_{1})} |  = {\displaystyle =} |  a 0 + a 1 x 1 + a 2 x 1 2 + ⋯ + a n − 1 x 1 n − 1 {\displaystyle a_{0}+a_{1}x_{1}+a_{2}x_{1}^{2}+\dots +a_{n-1}x_{1}^{n-1}}

|
|

f ( x 2 ) {\displaystyle f(x_{2})} |  = {\displaystyle =} |  a 0 + a 1 x 2 + a 2 x 2 2 + ⋯ + a n − 1 x 2 n − 1 {\displaystyle a_{0}+a_{1}x_{2}+a_{2}x_{2}^{2}+\dots +a_{n-1}x_{2}^{n-1}}
|  ⋮ {\displaystyle \vdots } |
f ( x n ) {\displaystyle f(x_{n})} |  = {\displaystyle =} |  a 0 + a 1 x n + a 2 x n 2 + ⋯ + a n − 1 x n n − 1 {\displaystyle a_{0}+a_{1}x_{n}+a_{2}x_{n}^{2}+\dots +a_{n-1}x_{n}^{n-1}}

We can re-write this system of linear equation in matrix notation as

M a = f {\displaystyle \mathbf {M} \mathbf {a} =\mathbf {f} }

where  a {\displaystyle \mathbf {a} } is the vector of the  a i {\displaystyle a_{i}} ,  i = 0 … n − 1 {\displaystyle i=0\dots n-1} coefficients and  M {\displaystyle \mathbf {M} } is the so-called moment matrix or Vandermonde matrix with

M = [ 1 x 1 x 1 2 … x 1 n − 1 1 x 2 x 2 2 … x 2 n − 1 ⋮ ⋮ ⋮ ⋱ ⋮ 1 x n x n 2 … x n n − 1 ] {\displaystyle \mathbf {M} ={\begin{bmatrix}1&x_{1}&x_{1}^{2}&\dots &x_{1}^{n-1}\\\1&x_{2}&x_{2}^{2}&\dots &x_{2}^{n-1}\\\\\vdots &\vdots &\vdots &\ddots &\vdots \\\1&x_{n}&x_{n}^{2}&\dots &x_{n}^{n-1}\end{bmatrix}}}

M i , j = x i j − 1 {\displaystyle \mathbf {M} _{i,j}=x_{i}^{j-1}}

This linear system of equations can be solved for  a {\displaystyle \mathbf {a} } using Gauss elimination or other methods.

The following shows how this can be done in Matlab or GNU Octave.

% declare our function f
f = inline('sin(pi*2*x)','x')

% order of the interpolation?
n = 5

% declare the nodes x_i as random in [0,1]
x = rand(n,1)

% create the moment matrix
M = zeros(n)
for i=1:n
M(:,i) = x.^(i-1);
end

% solve for the coefficients a
a = M \ f(x)

% compute g(x)
g = ones(n,1) * a(1)
for i=2:n
g = g + x.^(i-1)*a(i);
end;

% how good was the approximation?
g - f(x)

However practical it may seem, the Vandermonde matrix is not especially suited for more than just a few nodes. Due to it's special structure, the Condition Number of the Vandermode matrix increases with  n {\displaystyle n} .

If in the above example we set  n = 10 {\displaystyle n=10} and compute  a {\displaystyle \mathbf {a} } with

% solve for the coefficients a
a = inv(M) * f(x)

the approximation is only good up to 10 digits. For  n = 20 {\displaystyle n=20} it is accurate only to one digit.

We must therefore consider other means of computing and/or representing  g ( x ) {\displaystyle g(x)} .

## Lagrange Interpolation[edit | edit source]

Given a set of  n {\displaystyle n} nodes  x i {\displaystyle x_{i}} ,  i = 1 … n {\displaystyle i=1\dots n} , we define the  k {\displaystyle k} th Lagrange Polynomial as the polynomial of degree  n − 1 {\displaystyle n-1} that satisifies

ℓ k ( x i ) = { 0 i ≠ k 1 i = k {\displaystyle \ell _{k}(x_{i})={\begin{cases}0\quad &i\neq k\\\1&i=k\end{cases}}}

Maybe add a nice plot of a Lagrange polynomial over a few nodes?

The first condition, namely that  ℓ k ( x i ) = 0 {\displaystyle \ell _{k}(x_{i})=0} for  i ≠ k {\displaystyle i\neq k} can be satisfied by construction. Since every polynomial of degree  n − 1 {\displaystyle n-1} can be represented as the product

g ( x ) = c ∏ i = 1 n − 1 ( x − ξ i ) {\displaystyle g(x)=c\prod _{i=1}^{n-1}(x-\xi _{i})}

where the  ξ i {\displaystyle \xi _{i}} are the roots of  g ( x ) {\displaystyle g(x)} and  c {\displaystyle c} is some constant, we can construct  ℓ k ⋆ ( x ) {\displaystyle \ell _{k}^{\star }(x)} as

ℓ k ⋆ ( x ) = ∏ i = 1 , i ≠ k n ( x − x i ) {\displaystyle \ell _{k}^{\star }(x)=\prod _{i=1,i\neq k}^{n}(x-x_{i})} .

The polynomial  ℓ k ⋆ ( x ) {\displaystyle \ell _{k}^{\star }(x)} is of order  n − 1 {\displaystyle n-1} and satisifies the first condition, namely that  ℓ k ⋆ ( x i ) = 0 {\displaystyle \ell _{k}^{\star }(x_{i})=0} for  i ≠ k {\displaystyle i\neq k} .

Since, by construction, the  n − 1 {\displaystyle n-1} roots of  ℓ k ⋆ ( x ) {\displaystyle \ell _{k}^{\star }(x)} are at the nodes  x i {\displaystyle x_{i}} ,  i ≠ k {\displaystyle i\neq k} , we know that the value of  ℓ k ⋆ ( x k ) {\displaystyle \ell _{k}^{\star }(x_{k})} cannot be zero. This value, however, should be  1 {\displaystyle 1} by definition. We can enforce this by scaling  ℓ k ⋆ ( x ) {\displaystyle \ell _{k}^{\star }(x)} by  ℓ k ⋆ ( x k ) − 1 {\displaystyle \ell _{k}^{\star }(x_{k})^{-1}} :

ℓ k ( x ) {\displaystyle \ell _{k}(x)} |  = {\displaystyle =} |  ℓ k ⋆ ( x ) ℓ k ⋆ ( x k ) {\displaystyle {\frac {\ell _{k}^{\star }(x)}{\ell _{k}^{\star }(x_{k})}}}

|
|

|  = {\displaystyle =} |  ∏ i = 1 , i ≠ k n ( x − x i ) ∏ i = 1 , i ≠ k n ( x k − x i ) {\displaystyle {\frac {\prod _{i=1,i\neq k}^{n}(x-x_{i})}{\prod _{i=1,i\neq k}^{n}(x_{k}-x_{i})}}}

If we insert any  x i {\displaystyle x_{i}} ,  i ≠ k {\displaystyle i\neq k} into the above equation, the nominator becomes  0 {\displaystyle 0} and the first condition is satisified. If we insert  x k {\displaystyle x_{k}} , the nominator and the denominator are the same and we get  1 {\displaystyle 1} , which means that the second condition is also satisfied.

We can now use the Lagrange polynomials  ℓ k ( x ) {\displaystyle \ell _{k}(x)} to construct a polynomial  g ( x ) {\displaystyle g(x)} of degree  n − 1 {\displaystyle n-1} which interpolates the  n {\displaystyle n} function values  f i {\displaystyle f_{i}} at the points  x i {\displaystyle x_{i}} .

Consider that, given the definition of  ℓ k ( x ) {\displaystyle \ell _{k}(x)} ,

f k ℓ k ( x i ) = { f k i = k 0 i ≠ k {\displaystyle f_{k}\ell _{k}(x_{i})={\begin{cases}f_{k}\quad &i=k\\\0&i\neq k\end{cases}}} .

Therefore, if we define  g ( x ) {\displaystyle g(x)} as

g ( x ) = ∑ k = 1 n f k ℓ k ( x ) {\displaystyle g(x)=\sum _{k=1}^{n}f_{k}\ell _{k}(x)}

then  g ( x i ) = f i {\displaystyle g(x_{i})=f_{i}} for all  i = 1 … n {\displaystyle i=1\dots n} . Since the  ℓ k ( x ) {\displaystyle \ell _{k}(x)} are all of degree  n − 1 {\displaystyle n-1} , the weighted sum of these polynomials is also of degree  n − 1 {\displaystyle n-1} .

Therefore,  g ( x ) {\displaystyle g(x)} is the polynomial of degree  n − 1 {\displaystyle n-1} which interpolates the  n {\displaystyle n} values  f i {\displaystyle f_{i}} at the nodes  x i {\displaystyle x_{i}} .

The representation with Lagrange polynomials has some advantages over the monomial representation  g ( x ) = ∑ i = 0 n − 1 a i x i {\displaystyle g(x)=\sum _{i=0}^{n-1}a_{i}x^{i}} described in the previous section.

First of all, once the Lagrange polynomials have been computed over a set of nodes  x i {\displaystyle x_{i}} , computing the interpolating polynomial for a new set of function values  f i {\displaystyle f_{i}} is trivial, since these only need to be multiplied with the Lagrange polynomials, which are independent of the  f i {\displaystyle f_{i}} .

Another advantage is that the representation can easily be extended if additional points are added. The Lagrange polynomial  ℓ k + ( x ) {\displaystyle \ell _{k}^{+}(x)} , obtained by adding a point  x n + 1 {\displaystyle x_{n+1}} to the interpolation, is

ℓ k + ( x ) = ℓ k ( x ) x − x n + 1 x k − x n + 1 {\displaystyle \ell _{k}^{+}(x)=\ell _{k}(x){\frac {x-x_{n+1}}{x_{k}-x_{n+1}}}} .

Only the new polynomial  ℓ n + 1 + ( x ) {\displaystyle \ell _{n+1}^{+}(x)} would need to be re-computed from scratch.

In the monomial representation, adding a point would involve adding an extra line and column to the Vandermonde matrix and re-computing the solution.

* Add a word on Barycentric interpolation to facilitate adding points.

## Newton Interpolation[edit | edit source]

Given a set of  n {\displaystyle n} function values  f i {\displaystyle f_{i}} evaluated at the nodes  x i {\displaystyle x_{i}} , the Newton polynomial is defined as the sum

g ( x ) = ∑ i = 1 n a i n i ( x ) {\displaystyle g(x)=\sum _{i=1}^{n}a_{i}n_{i}(x)}

where the  n i ( x ) {\displaystyle n_{i}(x)} are the  n {\displaystyle n} Newton basis polynomials defined by

n 1 ( x ) {\displaystyle n_{1}(x)} |  = {\displaystyle =} |  1 {\displaystyle 1}

|
|

n i ( x ) {\displaystyle n_{i}(x)} |  = {\displaystyle =} |  ∏ k = 1 i − 1 ( x − x k ) for i > 1 {\displaystyle \prod _{k=1}^{i-1}(x-x_{k})\quad {\mbox{for }}i>1} .

An interpolation polynomial of degree n+1 can be easily obtained from that of degree n by just adding one more node point  x n + 1 {\displaystyle x_{n+1}} and adding a polynomial of degree n+1 to  g ( x ) {\displaystyle g(x)} .

The Newton form of the interpolating polynomial is particularly suited to computations by hand, and underlies Neville's algorithm for polynomial interpolation. It is also directly related to the exact formula for the error term  f i − g ( x i ) {\displaystyle f_{i}-g(x_{i})} . Any good text in numerical analysis will prove this formula and fully specify Neville's algorithm, usually doing both via divided differences.

(Should link to error formula, divided differences and Neville's algorithm, adding articles where they don't exist)

## Interpolation error[edit | edit source]

When interpolating a given function f by a polynomial of degree n at the nodes x0,...,xn we get the error

f ( x ) − p n ( x ) = f [ x 0 , … , x n , x ] ∏ i = 0 n ( x − x i ) {\displaystyle f(x)-p_{n}(x)=f[x_{0},\ldots ,x_{n},x]\prod _{i=0}^{n}(x-x_{i})}

where

f [ x 0 , … , x n , x ] {\displaystyle f[x_{0},\ldots ,x_{n},x]}

is the notation for divided differences.

If f is n + 1 times continuously differentiable on a closed interval I and  p n ( x ) {\displaystyle p_{n}(x)} be a polynomial of degree at most n that interpolates f at n + 1 distinct points {xi} (i=0,1,...,n) in that interval. Then for each x in the interval there exists  ξ {\displaystyle \xi } in that interval such that

f ( x ) − p n ( x ) = f ( n + 1 ) ( ξ ) ( n + 1 ) ! ∏ i = 0 n ( x − x i ) {\displaystyle f(x)-p_{n}(x)={\frac {f^{(n+1)}(\xi )}{(n+1)!}}\prod _{i=0}^{n}(x-x_{i})}

The above error bound suggests choosing the interpolation points xi such that the product | Π (x − xi) | is as small as possible.

### Proof[edit | edit source]

Let's set the error term is

R n ( x ) = f ( x ) − p n ( x ) {\displaystyle R_{n}(x)=f(x)-p_{n}(x)}

and set up a auxiliary function  Y ( t ) {\displaystyle Y(t)} and the function is

Y ( t ) = R n ( t ) − R n ( x ) W ( x ) W ( t ) {\displaystyle Y(t)=R_{n}(t)-{\frac {R_{n}(x)}{W(x)}}\ W(t)}

where

W ( t ) = ∏ i = 0 n ( t − x i ) {\displaystyle W(t)=\prod _{i=0}^{n}(t-x_{i})}

and

W ( x ) = ∏ i = 0 n ( x − x i ) {\displaystyle W(x)=\prod _{i=0}^{n}(x-x_{i})}

Since  x i {\displaystyle x_{i}} are roots of function f and  p n ( x ) {\displaystyle p_{n}(x)} , so we will have

Y ( x ) = R n ( x ) − R n ( x ) W ( x ) W ( x ) = 0 {\displaystyle Y(x)=R_{n}(x)-{\frac {R_{n}(x)}{W(x)}}\ W(x)=0}

and

Y ( x i ) = R n ( x i ) − R n ( x ) W ( x ) W ( x i ) = 0 {\displaystyle Y(x_{i})=R_{n}(x_{i})-{\frac {R_{n}(x)}{W(x)}}\ W(x_{i})=0}

Then  Y ( t ) {\displaystyle Y(t)} has n+2 roots. From Rolle's theorem,  Y ′ ( t ) {\displaystyle Y^{\prime }(t)} has n+1 roots, then  Y ( n + 1 ) ( t ) {\displaystyle Y^{(n+1)}(t)} has one root  ξ {\displaystyle \xi } , where  ξ {\displaystyle \xi } is in the interval I.

So we can get

Y ( n + 1 ) ( t ) = R n ( n + 1 ) ( t ) − R n ( x ) W ( x ) ( n + 1 ) ! {\displaystyle Y^{(n+1)}(t)=R_{n}^{(n+1)}(t)-{\frac {R_{n}(x)}{W(x)}}\ (n+1)!}

Since  p n ( x ) {\displaystyle p_{n}(x)} is a polynomial of degree at most n, then

R n ( n + 1 ) ( t ) = f ( n + 1 ) ( t ) {\displaystyle R_{n}^{(n+1)}(t)=f^{(n+1)}(t)}

Thus

Y ( n + 1 ) ( t ) = f ( n + 1 ) ( t ) − R n ( x ) W ( x ) ( n + 1 ) ! {\displaystyle Y^{(n+1)}(t)=f^{(n+1)}(t)-{\frac {R_{n}(x)}{W(x)}}\ (n+1)!}

Since  ξ {\displaystyle \xi } is the root of  Y ( n + 1 ) ( t ) {\displaystyle Y^{(n+1)}(t)} , so

Y ( n + 1 ) ( ξ ) = f ( n + 1 ) ( ξ ) − R n ( x ) W ( x ) ( n + 1 ) ! = 0 {\displaystyle Y^{(n+1)}(\xi )=f^{(n+1)}(\xi )-{\frac {R_{n}(x)}{W(x)}}\ (n+1)!=0}

Therefore

R n ( x ) = f ( x ) − p n ( x ) = f ( n + 1 ) ( ξ ) ( n + 1 ) ! ∏ i = 0 n ( x − x i ) {\displaystyle R_{n}(x)=f(x)-p_{n}(x)={\frac {f^{(n+1)}(\xi )}{(n+1)!}}\prod _{i=0}^{n}(x-x_{i})} .

## Switching Between Representations[edit | edit source]

## Arbitrary Basis Functions[edit | edit source]

## Piecewise Interpolation / Splines[edit | edit source]

B Splines

## Radial Basis Functions[edit | edit source]

## Extrapolation[edit | edit source]

Interpolation is the technique of estimating the value of a function for any intermediate value of the independent variable while the process of computing the value of the function outside the given range is called extrapolation.

Caution must be used when extrapolating, as assumptions outside the data region (linearization, general shape and slope) may breakdown.

## Exercises[edit | edit source]