Open Issues in Constrained Blind Source Separation

1
Open Issues in Constrained Blind Source
Separation

• Jonathon Chambers
• Cardiff Professorial Research Fellow
• Cardiff School of Engineering
• Cardiff University, Wales, U.K.
• E-mail chambersj_at_cf.ac.uk

2
Summary of Talk

• Acknowledgement
• Historical background motivation
• BSS with matrix constraints
• Penalty functions in FD-BSS
• Exploiting periodicity in BSS
• Future application-driven challenges

3
Acknowledgements

• Jonathon Chambers wishes to express his
  sincere thanks for the support of Professor
  Andrzej Cichocki, Riken Brain Science Institute,
  Japan
• The invitation from the organising committee
  of the workshop to give this talk.
• His co-researchers Drs Saeid Sanei, Maria
  Jafari and Wenwu Wang.

4
LMS Algorithm

• B. Widrow, and M.E. Hoff, Jr.,
• Adaptive switching circuits, IRE Wescon
  Conv. Rec., pt. 4, pp. 96-104, 1960.
• LMS Update

5
Historical Background

• The field of conventional adaptive signal
  processing has been greatly enhanced by the
  exploitation of constrained optimisation
• Constraints on the error, and/or structure or
  some norm of the weights via, for example,
  Lagrange multipliers and/or Karush-Khun-Tucker
  conditions

6
Historical Background

• Certain key papers
• O.L. Frost, III, An algorithm for linearly
  constrained adaptive array processing, Proc.
  IEEE, Vol. 60(8), pp. 926-925, 1972
• R.P. Gitlin et al. The tap-leakage algorithm an
  algorithm for the stable operation of a digitally
  implemented fractionally spaced equalizer, Bell
  Sys. Tech. Journal, Vol. 61(8), pp. 1817-1839,
  1982.
• D.T.M. Slock, Convergence behavior of the LMS
  and Normalised LMS Algorithms, IEEE Trans.
  Signal Processing, Vol. 41(9), pp. 2811-2825,
  1993.

7
Historical Background Cont.

• S.C. Douglas, A family of normalized LMS
  algorithms, IEEE Signal Processing Letters, Vol.
  1(3), pp. 49-51, 1994.
• S.C. Douglas, and M. Rupp, A posteriori updates
  for adaptive filters, Asilomar Conference on
  Signals, Systems and Computers, Vol. 2, pp
  1641-1645, 1997.
• T. Gänsler, et al., A robust proportionate
  affine projection algorithm for network echo
  cancellation, Proc. ICASSP 2000, Vol. 2, pp.
  793-796, 2000.
• O. Vainia, Polynomial constrained LMS adaptive
  algorithm for measurement signal processing,
  Proc. IECON 2002, Vol. 2, pp. 1479-1482, 2002.

8
Motivation

• In many applications of Independent Component
  Analysis (ICA) and Blind Source Separation (BSS)
  estimated source signals and the mixing or
  separating matrices have some special structure
  or some constraints are imposed for the
  matrices, Cichocki and Georgiev, 2003

9
Fundamental Model for Instantaneous Blind Source
Separation
10
Certain BSS Books

• Andrzej Cichocki and Shun-Ichi Amari, Adaptive
  Blind Signal and Image Processing, Wiley, 2002
• Simon Haykin Unsupervised Adaptive Filtering,
  Vols. I and II, Wiley, 2000
• Aapo Hyvärinen, Juha Karhunen and Erkki Oja,
  Independent Component Analysis, Wiley, 2001
• Te-Won Lee, Independent component analysis
  theory and applications, Kluwer, 1998

11
BSS References

• A. Mansour and M. Kawamoto, ICA Papers
  Classified According to their Applications and
  Performances, IEICE Trans. Fundamentals, Vol.
  E86-A, No. 3, March 2003, pp. 620-633.
• In 2002, 800 different papers have been
  published, these are downloadable at
  http//ali.mansour.free/REF.htm

12
BSS With Matrix Constraints
With a symmetric mixing matrix CG,2003-
13
BSS With Matrix Consts. Cont.
Stable Frobenius norm of the separating matrix
Theorem CG 2003 The learning rule
where ß gt 0 is a scaling factor and ?(t)
trace(WT(t)F(y(t))W(t)) gt 0, stabilizes the
Frobenius norm of W(t) such that
14
BSS With Matrix Consts. Cont.
Consequence The modified NG descent learning
algorithm, with a forgetting factor, described
as
with ?(t) -trace(WT(t)?J(W)/ ? WWT(t)W(t)) gt
0 has a W(t) with bounded Frobenius norm
throughout the learning process.
15
BSS With Matrix Consts. Cont.
Prof. Amaris Leaky NG Algorithm becomes
where 0 ltlt (1-ß?(t)?(t)) lt 1 is the leakage
factor
16
BSS With Matrix Consts. Cont.
Introducing a semi-orthogonality constraint so
that it is possible to extract an arbitrary group
of sources, say e, 1 ? e ? N. Assuming
pre-whitened data
and the mixing matrix A QH, the demixing
matrix We should satisfy WeA Ie,0N-e
17
BSS With Matrix Consts. Cont.
A natural gradient algorithm to find We
becomes-
With initial conditions which satisfy
18
Real Convolutive Mixing Env. Cocktail Party
Problem
19
Convolutive BSS Model
Convolution
Compact form
Expansion form
20
Taxonomy of Existing Sols. To Convolutive BSS

• Performing blind separation in the time domain
  by extending the existing instantaneous methods
  to conv. case
• Transforming the convolutive BSS problem into
  multiple instantaneous (complex) problems in the
  frequency domain
• Decomposing the system into smaller problems
  using, for example, a subband approach
• Hybrid frequency and time domain approaches

21
Transform Convolutive BSS into the Frequency
Domain
DFT
Convolutive BSS problem
Multiple complex-valued instantaneous BSS
problems
22
Mathematical Formulation
In the frequency domain-
23
De-mixing Operation
24
Constrained Optimisation and Joint Diagonalisation
25
Joint Diagonalisation Criterion
Exploiting the non-stationarity of speech signals
measured by the cross-spectrum of the output
signals,
26
Exterior Penalty Function Approach
27
Exterior Penalty Function Approach
Typical exterior penalty functions, and the
shadow area represents the feasible set.
28
Proposed General Cost Function
With a factor vector ? to incorporate exterior
penalty functions, our cost function becomes-
29
Numerical Experiments

• Use an exterior penalty function

• Employ a variant of gradient adaptation

• Utilize the filter length constraint to address
  the permutation problem (Parra Spence)
• System with two inputs and two outputs (TITO!)
• H(z) 1 1.9 -0.75, z-50.5 0.3 0.2
  z-5-0.7 -0.3 -0.2, 0.8 -0.1 D 7, T
  1024, K 5.

30
Convergence Performance of the New Criterion as a
function of ?
31
Room Environment Experiment

• Use roommix function due to Westner

• Room 10x10x10m3 cube

• Wall reflections calculated up to fifth order,
  atten. factor 0.5
• SIR is plotted as a function of length of the
  separating system

32
Room Environment
33
Room Environment SIR
34
Permutation Problem in FD-CBSS
35
Summary of Existing Solutions to Permut. Problem
in FD-CBSS

• Constraints on the filter models in the
  frequency domain
• Using special structure contained in signals
• Merging beamforming view to align solutions
• Exploiting the continuity of the spectra of the
  recovered signals could coupled hidden Markov
  Models be used?
• What happens when the sources move,
  enter/re-enter the environment? What is the way
  forward?

36
Exploiting Source (Pseudo) -Periodicity

• W. Wang, M.G. Jafari, S. Sanei, and J.A.
  Chambers, Blind source separation of convolutive
  mixtures of cyclostationarity, to appear in the
  Special Issue on BSS, International Journal of
  Adaptive Control and Signal Processing, Guest
  Editor Mike Davies, Queen Marys College,
  University of London
• H. Swada, R. Mukai, S. Araki, and S. Makino, A
  robust and precise method for solving the
  permutation problem of frequency-domain blind
  source separation, ICA 2003, Nara, Japan, 2003,
  pp. 505-510.

37
A natural gradient update exploiting
cyclostationarity

• The Cyclostationary NGA uses the update equation
• where
• and ?p is the cycle frequency of the p-th source

38
A natural gradient update exploiting periodicity

• The Periodic NGA type update equation
• where

39
Emerging Applications
Biomedical- ECG, EEG, MEG and their
integration Microarray time courses
Measurements from the nano-lab http//www.nmrc.i
e/research/transducers-group/trends.htmlhttp//ww
w.nanospace.systems.org/ns_2000/NS00_Sessions.htm
http//nanomed.ncl.ac.uk/m2l.htm
Star Trek The Tri-corder
40
Emerging Applications

• T. Bowles, J. Chambers, and A. Jakobsson,
  Advanced spectral estimation for the
  identification of cell-cycle regulated genes,
  IEEE EMBS UK and RI Postgraduate Conf in
  Biomedical Engineering and Medical Physics, 2003.
• X. Liao, and L. Carin, Constrained independent
  component analysis of DNA microarray signals,
  IEEE Workshop on Genomic Signal Processing and
  Statistics, 2002.
• S-I, Lee, and S. Batzoglou, Discovering
  biological processes from microarray data using
  independent component analysis, Dept EE/CS,
  Stanford Univ.

41
Summary

• The exploitation of constrained optimisation
  has been fundamental to the development and
  application of adaptive signal processing this
  process is, however, very much in its infancy in
  blind source separation (BSS).
• Utilisation of certain a priori knowledge on the
  mixing matrices and the properties of the sources
  is likely to yield solutions to real-life SP
  problems.
• As such, the challenge for DSP engineers in the
  21st Century, is to advance the application of
  BSS methods in line with methods from adaptive
  signal processing.

42
Other References

• A. Cichocki, and P. Georgiev, Blind source
  separation algorithms with matrix constraints,
  IEICE Trans. Fundamentals, Vol. E86-A(3), March
  2003, pp. 522-531.
• J.G. McWhirter, Mathematics and signal
  processing, Mathematics Today, April 2003, pp
  47-54.
• W. Wang, S. Sanei, and J. Chambers, Penalty
  function based joint diagonalization approach for
  convolutive blind source separation, submitted
  to IEEE T-SP, Sept 2003.

43
Close ???
Mark Twain A man who swings a cat by its tail
learns things he can learn no other way