Introduction to Radial Basis Function

1
Introduction to Radial Basis Function

• Mark J. L. Orr

2
Radial Basis Function Networks

• Linear model

3
Radial functions

• Gassian RBFc center, r radius

• monotonically decreases with distance from center

• Multiquadric RBF

• monotonically increases with distance from center

4
Gaussian RBF
multiqradric RBF
5
Least Squares

• model

• training data (x1, y1), (x2, y2), , (xp, yp)
• minimize the sum-squared-error

6
Example

• Sample points (noisy) from the curve y x
  (1, 1.1), (2, 1.8), (3, 3.1)
• linear model f(x) w1h1(x) w2h2(x),where
  h1(x) 1, h2(x) x
• estimate the coefficient w1, w2

7

• f(x) x

8

• New model f(x) w1h1(x) w2h2(x)
  w3h3(x)where h1(x) 1, h2(x) x, h3(x) x2

9

• absorb all the noise overfit
• If the model is too flexible, it will fit the
  noise
• If it is too inflexible, it will miss the target

10
The optimal weight vector

• model

• sum-squared-error

• cost function weight penalty term is added

11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
Example

• Sample points (noisy) from the curve y x
  (1, 1.1), (2, 1.8), (3, 3.1)
• linear model f(x) w1h1(x) w2h2(x),where
  h1(x) 1, h2(x) x
• estimate the coefficient w1, w2

16
(No Transcript)
17
The projection matrix

• At the optimal weightthe value of cost function
  C yTPythe sum-squared-error S yTP2y

18
Model selection criteria

• estimates of how well the trained model will
  perform on future input
• standard tool cross validation
• error variance

19
Cross validation

• leave-one-out (LOO) cross-validation

• generalized cross-validation

20
Ridge regression

• mean-squared-error

21
Global ridge regression

• Use GCV

• re-estimation formula
• initialize ?
• re-estimate ?, until convergence

22
Local ridge regression

• research problem

23
Example
24
Selection the RBF

• forward selection
• starts with an empty subset
• added one basis function at a time
• most reduces the sum-squared-error
• until some chosen criterion stops
• backward elimination
• starts with the full subset
• removed one basis function at a time
• least increases the sum-squared-error
• until the chosen criterion stops decreasing