Class

1

• Class 7 Chapter 6
• The LMS Algorithm
• Derivation of the Algorithm
• Convergence of the Weight Vector
• An Example
• Learning Curve
• Noise in the Weight Vector Solution
• Misadjustment
• Performance
• Exercises

2
The Least-Mean Square (LMS) Algorithm

• We have discussed two methods for descending the
  performance surface toward the minimum
• Newtons Method
• Steepest Descent
• Both algorithms require an estimate of the
  gradient at each iteration
• The gradient estimation methods are general since
  they are based on taking differences between
  estimated points on the performance surface.

3
The Least-Mean Square (LMS) Algorithm (Cont)

• The LMS algorithm uses a special estimate of the
  gradient that is valid for the adaptive linear
  combiner (it can be extended to recursive
  adaptive filters).
• It is more restricted in its use than previous
  algorithms
• It is important because of its simplicity and
  ease of computation
• It does not require off-line gradient estimates
  or repetitions of the data.

4
The Least-Mean Square (LMS) Algorithm (Cont)

• If the adaptive system is an adaptive linear
  combiner (ALC) and if the input vector Xk and the
  desired response dk are available at each
  iteration, the LMS algorithm is generally the
  best choice for many applications.

5
Derivation of the LMS Algorithm
Remember the two basic ways of applying the ALC
Desired response
Input
Error
Output
6
Derivation of the LMS Algorithm (Cont)
Desired response
Error
Output
7
Derivation of the LMS Algorithm (Cont)
8
Derivation of the LMS Algorithm (Cont)
9
Derivation of the LMS Algorithm (Cont)
?
10
Derivation of the LMS Algorithm (Cont)
11
Convergence of the Weight Vector
12
Convergence of the Weight Vector (Cont)
13
Convergence of the Weight Vector (Cont)
14
Convergence of the Weight Vector (Cont)
15
An Example of Convergence
16
Example(Cont)
17
Example(Cont)
18
Example(Cont)
19
Example(Cont)
Random Process
20
Example(Cont)
Input Process
21
Example(Cont)
22
Example(Cont)
Weight Convergence Upper Track (Black)
Weight value
Iteration number, k
23
Example(Cont)
Weight Convergence Lower Track (Red)
Weight value
Iteration number, k
24
Example(Cont)
Error Signal Upper Track (Black)
Error Signal
Iteration number, k
25
Example(Cont)
Error Signal Lower Track (Red)
Error Signal
Iteration number, k
26
Example(Cont)
27
Example(Cont)
adsp Figure 6.3 11/4/99 Saved as
adspcl6_1 clear N16 phi0.01 LMS
Algorithm mu0.1 r.1randn
(1,250) k1250 xkrsin(2pik/N) Xxk0
xk(1249) dk2cos(2pik/N) W00 for
n1249 e(n)dk(n)-X(,n)'W(,n)
tempW(,n)2mue(n)X(,n) WW
temp end e(250)dk(250)-X(,250)'W(,250) W(,
250)
28
Example(Cont)
mu20.05 r2.1randn(1,500) k21500 xk2r2sin
(2pik2/N) X2xk20 xk2(1499) dk22cos(2pi
k2/N) W24-10 for n1499
e2(n)dk2(n)-X2(,n)'W2(,n)
temp2W2(,n)2mu2e2(n)X2(,n) W2W2
temp2 end e2(500)dk2(500)-X2(,500)'W2(,500)
W2(,500)
v.41 .78 1.9 6.3 13.6 23.8
37 w0-2.058 w1-10.050 w0star2sin(2pi/
N)cos(2pi/N)/((12phi)2-(cos(2pi/N)2)) w1sta
r-2(12phi)sin(2pi/N)/((12phi)2-(cos(2pi/
N)2)) W0,W1meshgrid(w0,w1) c(.5phi)(W0.2
W1.2)cos(2pi/N)W0.W12sin(2pi/N)W12Figu
re 6.3 cmin(.5phi)(w0star2w1star2)cos(2pi/
N)w0starw1star2sin(2pi/N)w1star2 cw(.5phi
)(W(1,).2W(2,).2)cos(2pi/N)W(1,).W(2,)
2sin(2pi/N)W(2,)2 cw2(.5phi)(W2(1,).2
W2(2,).2)cos(2pi/N)W2(1,).W2(2,)2sin(2p
i/N)W2(2,)2
29
Example(Cont)
figure(1)clf contour(w0,w1,c,v) axis('square') gr
id title('Mean-Square-Error') ylabel('w1') xlabel(
'w0') hold on plot(W(1,),W(2,),'k') plot(W2(1,)
,W2(2,),'r') plot(w0star,w1star,'rx') hold
off figure(2)clf plot(k,r)grid figure(3)clf
plot(k,xk)grid
figure (4)clf plot(k,W)grid hold on plot(0
250,w0star w0star,'r') plot(0 250,w1star
w1star,'r') hold off figure (5)clf plot(k2,W2)
grid hold on plot(0 500,w0star
w0star,'r') plot(0 500,w1star
w1star,'r') hold off figure(6)clf plot(k,e,'k')
grid figure(7)clf plot(k2,e2,'r')grid
30
Example(Cont)
31
Learning Curve
32
Learning Curve (Cont)
eig(0.51 .5cos(pi/8).5cos(pi/8)
0.51) ans 0.0481 0.9719
33
Learning Curve (Cont)
34
Noise in the Weight-Vector Solution
35
Noise in the Weight-Vector Solution (Cont)
36
Noise in the Weight-Vector Solution (Cont)
37
Misadjustment
38
Misadjustment (Cont)
39
Misadjustment (Cont)
40
Misadjustment (Cont)
41
Performance
42
Performance (Cont)
43
Performance (Cont)
Steepest Descent
LMS
44
Performance (Cont)
adsp Figure 6.6 11/16/99 Adaptation time
perforamnce comparison Saved as
adsp6_3 clear nw110500 sd50nw.2 lms2.5nw
figure(1)clf semilogy(nw,sd,nw,lms)grid title
('Adaptation Time Comparison') xlabel('Number of
Weights, L1') ylabel('Time constant (data
samples)')
45
Exercise (To be Handed In)
46
Examples
47
Examples
48
Examples (Cont)
49
Examples (Cont)
50
Examples (Cont)
P0 -sin(2pi/N) -sin(4pi/N) P
0 -0.3827 -0.7071 WstarR\P Wstar
2.0006 -0.2807 -2.5192
N16 R.51 .5cos(2pi/N)
.5cos(4pi/N) .5cos(2pi/N) .51
.5cos(2pi/N) .5cos(4pi/N) .5cos(2pi/N)
.51 R 0.5100 0.4619 0.3536
0.4619 0.5100 0.4619 0.3536 0.4619
0.5100
51
Examples (Cont)
cmin2Wstar'RWstar-2P'Wstar cmin
0.1113
52
Examples (Cont)
53
Examples (Cont)
54
Examples (Cont)
55
Examples (Cont)
56
Examples (Cont)
57
Examples (Cont)
eig(R) ans 0.0100 0.1564 1.3636
mu.1.05 T.25./(mulambda') mu
T ans 0.1000 250.0000 15.9799
1.8334 0.0500 500.0000 31.9598 3.6669
58
Criteria for Adaptive Algorithm Selection
Rate of convergence. This is the number of
iterations required for the algorithm, in
response to stationary inputs, to converge
close enough to the optimum Wiener solution in
the mean-square sense. A fast rate of
convergence allows the algorithm to adapt rapidly
to a stationary environment of unknown statistics.
59
Criteria for Adaptive Algorithm Selection (Cont)
Misadjustment. This parameter provides a
quantitative measure of the amount by which the
final value of the mean-square error, averaged
over an ensemble of adaptive filters, deviates
from the minimum mean-square error produced by
the Wiener filter.
60
Criteria for Adaptive Algorithm Selection (Cont)
Tracking. When an adaptive filtering algorithm
operates in a nonstationary environment, the
algorithm is required to track statistical
variations in the environment. The tracking
performance of the algorithm is influenced by two
contradictory freatures (1) rate of
convergence, and (2) steady-state fluctuation due
to algorithm noise.
61
Criteria for Adaptive Algorithm Selection (Cont)
Robustness. An adaptive filter is robust when
small disturbances (i.e., disturbances with small
energy) can only result in small estimation
errors. The disturbances may arise from a
variety of factors, internal or external to the
filter.
62
Criteria for Adaptive Algorithm Selection (Cont)
Computational requirements. The issues of
concern are(a) the number of operations (i.e.,
multiplications, divisions, and
additions/subtractions) required to make one
complete iteration of the algorithm, (b) the size
of memory required, (c) the investment required
to program the algorithm on a computer (DSP).
63
Criteria for Adaptive Algorithm Selection (Cont)
Structure. This refers to the structure of
information flow in the algorithm, determining
the manner in which it is implemented in hardware
form. For example, analgorithm whose structure
exhibits high modularity, parallelism, or
concurrency is well suited for implementation
using modern component technology.
64
Criteria for Adaptive Algorithm Selection (Cont)
Numerical properties. When an algorithm is
implemented, inaccuracies are produced due to
quantization errors, (due to A/D and internal
data formats). There are two basic issues
numerical stability and numerical accuracy.
Numerical stability is an inherent characteristic
of the adaptive filtering algorithm. Numerical
accuracy is determined by the data and
coefficient formats. An adaptive filtering
algorithm is numerically robust if it is
insensitive to variations in wordlength used in
its digital implementation.