A Fast-Nonegativity-Constrained Least Squares Algorithm

1
A Fast-Nonegativity-Constrained Least Squares
Algorithm

• R. Bro, S. D. Jong
• J. Chemometrics,11,393-401, 1997

By Maryam Khoshkam
2

• Introduction
• Algorithm of classical non-negative least square
  (nnls)
• A numerical example
• Fast nonegative least square (Fnnls)
• Results and discussion

3
introduction
Estimation of models subject to non-negativity
constraints is of practical importance in
chemistry.
The time required for estimating true least
squares non-negativity constrained model is
typicaly many times longer than for estimating
unconstrained model.
Approximations procedures
Unconstrained estimation Setting negative value
to zero
What is the problem of force to zero algorithm????
4

1. There is no guarantee whatsoever for the quality
   of the model.
2. When included in multiway algorithm, it can cause
   the algorithm to diverge. (specially, in noisy
   data or difficult models)

Non-negativity constrained linear least square
5
NNLS
NNLS will be stated using the following
nomenclature
For all m
lsqnonneg command in MATLAB is based on this
algorithm
Lawson, C.L. and R.J. Hanson, Solving
Least-Squares Problems, Prentice-Hall, Chapter
23, p. 161, 1974.
6
Some important aspects of the algorithm

• This is an active set algorithm.

Thus the nnls first find the true passive and
active sets, then perform the least square on
corresponding columns of Z.
HOW????
First of all, it suppose that all elements of d,
are in active set.
If the all elements of d are active, estimate
that how the form of initial d.?????
R1,2,.,M
P
7

• Initial solution vector d is feasible and set
  equal to a M1 zero vector

• Then it removes the passive sets elements of d,
  one by one, and STEP by STEP

HOW????

• Using the vector w

8
wZT ( x Z d )
One necessary condition for optimally of a
solution , is that the derivatives with respect
to the parameters of the passive set be zero. WHY
??
How is the f (dm) if m is an active set ????
9
Thus at optimal solution, we expect that
When we are being at optimal condition, wm gt0.
wht is the meaning of positive value for wm?
10

• positive wm shows that by increasing dm to a more
  positive value, the change in residual is
  negative (f ', which is the slope is negative).
• It means that the residual becomes less, and more
  close to zero, when dm goes toward positive.

11
Algorithm nnls

1. PØ
2. R1,2,,M
3. d0
4. wZT(x-Zd)

loop A
No
Opt. sol. Of d
yes
Loop B Build sp
ds wZT(x-Zd)
yes
No
Loop C
all spgt0
12
A simple numerical example
loop A
Max(wn)gt1e-15 R?Ø
Loop B
13
Loop B
No
Max(wn)gt1e-15 R?Ø
Opt. sol. Of d
14
Graphical representation of nonlinear least
square algorithm
H3A ? H H2A- pka12.6 H2A- ? H
HA2- pka24.0 HA2- ? H A3-
pka36.3
D
Z
DX.Z (ILS)
15
x10x1X(,1)
Z(10x4)
D(4x100)
X(10x100)

d(4x1)
Step1

1. P
2. R1,2,3,4

0.005 x D(,1)
w
16
nnls
Force to zero
Variation of w during nonnegative least square
17
w
Residual matrix
wZT(x-Zd)

wi in each ?, is the contribution of ith species
in in residual vector in each ?.
w(41)
r(10x1)
Z (10x4)T
18
(No Transcript)
19
PARAFAC-ALS
Least square
20
If the size of X is 10x200x5 in a 3 component
system, the size of Z is 500x3 And so on.
Computation of Z, can be computationally costly
for large arrayes Excessive memory is required
to calculate X(IxJK), X(JxIK) and X(K,IJ)
21
And in similar way for estimation of B and C
22
Nnls can not used for this simplified version of
PARAFAc, Why??
23
In estimating A, and using non-negativity
constraint on it,
A(IxF)
X(IxJK)

Z
SLOW!!!!!!!

• A modification

24
Fast non-negativity least square
1. Accept the cross product (xZ and ZZ) instead
of the row data
wZT ( x Z d ) ? wZTx (ZTZ)d
SP(ZP)TZP-1(ZP)Tx? SP(ZTZ)P-1(ZTx)
2. Set the passive and active variables, before
enter to loop B
d is not the zero vector in this case
25
Thanks
26
Ex.
dZ\x
Force to zero
nnlsq
RMS20
RMS103
27
In unconstrained solution of d, the mth negative
element is active set
3th element of d
In constrained solution of d, the mth zero
element is active set
2th and 3th elements of d
Thus it is not possible to realize the true
active set form unconstraint least square
solution.
What happen if the true active set is known?
28
For example, if we know that 2th and 3th elements
of d are true active sets,
Then perform the simple unconstrained least
square with the 1st column of Z. (columns
corresponding to passive sets. selectivity
constraint !!!!
29
Loop C
1)
2)
3)
Update R and P
4)
SP(ZP)TZP-1(ZP)Tx
5)
SR0