Regression

1
Regression

• CS294 Practical Machine Learning
• Romain Thibaux
• 02/07

TexPoint fonts used in EMF. Read the TexPoint
manual before you delete this box. AAAAAAAAAAAA
2
Regression

• A new perspective on freedom

TexPoint fonts used in EMF. Read the TexPoint
manual before you delete this box. AAAAAAAAAAAA
3
Outline

• Nearest Neighbor, Kernel Regression
• Linear regression
• Derivation from minimizing the sum of squares
• Probabilistic interpretation
• Online version (LMS)
• Overfitting and Regularization
• L1 Regression
• Spline Regression

4
Where are we?
5
Classification
6
Cat
Dog
7
Cleanliness
Size
8




?
9
Regression
10
Price


y


Top speed
x
11
Regression
Data Goal given , predict i.e. find a
prediction function
12
Examples

• Voltage ! Temperature
• Processes, memory ! Power consumption
• Protein structure ! Energy
• Robot arm controls ! Torque at effector
• Location, industry, past losses ! Premium

13
4
14
1
15
Nearest neighbor
15
10
5
0
-5
-10
-5
0
5
10
15
20
25
16
Voronoi Diagram
Wikipedia
17
Voronoi Diagram
http//www.qhull.org/html/qvoronoi.htm
18
Nearest neighbor

• To predict x
• Find the data point xi closest to x
• Choose y yi
• No training
• Finding closest point can be expensive
• Overfitting

19
Kernel Regression
e.g.

• To predict X
• Give data point xi weight
• Normalize weights
• Let

20
Kernel Regression
15
10
5
0
-5
-10
-5
0
5
10
15
20
25
matlab demo
21
Kernel Regression

• No training
• Smooth prediction
• Slower than nearest neighbor
• Must choose width of

22
2
23
Linear regression
24
Linear regression
26
24
Temperature
22
20
30
40
20
30
20
10
10
0
0
start Matlab demo lecture2.m
25
Linear regression
26
Linear Regression
Error or residual
Observation
Prediction
Sum squared error
27
Learning as Optimization
28
Learning as Optimization
29
Learning as Optimization
30
Learning as Optimization
31
Linear Regression
n
d
Solve the system (its better not to invert the
matrix)
32
Minimize the sum squared error
Sum squared error
Linear equation
Linear system
33
LMS Algorithm(Least Mean Squares)
where
Online algorithm
34
Online Learning
35
Beyond lines and planes
still linear in
everything is the same with
36
Geometric interpretation
20
10
400
0
300
200
-10
100
0
10
20
0
Matlab demo
37
Linear Regression summary
Given examples
Let
For example
Let
n
d
by solving
Minimize
Predict
38
Probabilistic interpretation
Likelihood
39
Assumptions vs. Reality
Voltage
Temperature
Intel sensor network data
40
Assumptions vs. Reality
Requests per minute
requests per minute
5000
0
0
1
2
Time (days)
41
Overfitting
30
25
20
15
10
5
0
-5
-10
-15
0
2
4
6
8
10
12
14
16
18
20
Matlab demo
42
Ridge Regression(Regularization)
Minimize
with small
Effect of regularization (degree 19)
15
10
5
0
-5
-10
0
2
4
6
8
10
12
14
16
18
20
43
Probabilistic interpretation
Likelihood
Prior
Posterior
44
3
45
Locally Linear Regression
46
Global temperature increase
0.6
0.4
0.2
0
-0.2
-0.4
-0.6
-0.8
1840
1860
1880
1900
1920
1940
1960
1980
2000
2020
source http//www.cru.uea.ac.uk/cru/data/tempera
ture
47
Locally Linear Regression
e.g.

• To predict X
• Give data point xi weight
• Let
• Let

48
Locally Linear Regression
To minimize
where
Solve
Predict

• Good even at the boundary (more important in
  high dimension)
• Solve linear system for each new prediction
• Must choose width of

49
Locally Linear Regression Gaussian kernel
180
source http//www.cru.uea.ac.uk/cru/data/tempera
ture
50
Locally Linear Regression Laplacian kernel
180
source http//www.cru.uea.ac.uk/cru/data/tempera
ture
51
4
52
L1 Regression
53
Sensitivity to outliers
High weight given to outliers
Influence function
54
L1 Regression
Linear program
Influence function
55
Spline RegressionRegression on each interval
70
60
50
5200
5400
5600
5800
56
Spline RegressionWith equality constraints
57
Spline RegressionWith L1 cost
58
Summary

• Nearest Neighbors
• Kernel Regression
• Locally Linear Regression / Spline Regression
• Linear Regression
• Prevent overfitting regularization
• Robustness to outliers L1 regression

59
To learn more

• The Elements of Statistical Learning, Hastie,
  Tibshirani, Friedman, Springer

60
Further topics

• Feature Selection future lecture
• Generalized Linear Models
• Gaussian process regression