Miroslav Krn

1

• Miroslav Kárný
• Department of Adaptive Systems ÚTIA AVCR
  school_at_utia.cas.cz
• http//www.utia.cas.cz/AS_dept

2
Aims of the talk

• Put an engineering perspective on decision making
  (DM)
• Stress the dynamic character of DM
• Share experience
• List some open problems worth of research effort

3
Guide

• Motivation
• Bayesian set up recalled
• Consequences of Bayesian set up
• Modeling
• Multiple-participants modeling
• Design of DM strategies
• Concluding remarks

4
DM Traffic control scenario
aims restrictions
?
advised model
advices
data
learned model
prior
data
5
DM in traffic control scenario
?
6
Features of DM under uncertainty
COMMON
VARIANTS
theory makes sense if the best DM is attempted !
7
Guide

• Motivation
• Bayesian set up recalled
• Consequences of Bayesian set up
• Modeling
• Multiple-participants modeling
• Design of DM strategies
• Concluding remarks

8
Design of DDM under uncertainty
Design finds strategy R(T) minimizing the
aim-expressing loss Z(d(T), ?(T)), using
available information under given restrictions
unseen ?t
System S(T) is a world part, St d(t-1), ?(t-1),
at yt, ?t
Actions at
Innovations yt
Data dt(yt, at )
Strategy R(T) generates actions by rules Rt
d(t-1) at ? at
information, complexity, range
Restrictions
9
Aim loss
?
Aim orders strategies
R? better than R? iff behavior Q? (d(T),
?(T))? of S-R? closer to the aim than
behavior Q? (d(T), ?(T))? of S-R?
Loss Z quantifies the order
Z(Q?) ? Z(Q?) ? R? better than R?
loss unsuitable for design if the dependence of
Q on R is unknown!
10
Experience model domains
?
provides
System S(T) a world part, St d(t-1),
?(t-1), at yt, ?t ,
Model M(T) a world image, Mt
prediction of consequences of the action choice
? design of R(T)
approximation of S(T) theory based
approximation of S(T) theory based
Model M(T)
white grey black
field knowledge universal approximation
field knowledge universal approximation
box
Unseen ?(T)
meaning domain important
loss unsuitable for design if experience does
not remove uncertainty!
11
Uncertainty U
?
Z?(U) ?Z(Q?) Z((d(T), ?(T))?) for S-R?
?
Unseen ?(T)
?
Loss Z(U)
Non-manageable complexity
?
Modeling errors (modeling noise)
?
Non-encountered influences (noise)
Uncertainty
anything unknown in design when S-R fixed
E Z(U) 0,? orders R irrespectively
of U
WHAT E?
12
E requirements consequences
?
?
?
?
?
?
?
?
Loss Z(U)
?
?
U
U
?
?
E? E? E?
?
?
Neutral ?(Z) Z
E mathematical expectation of utility function
?(Z) of loss Z!
13
Guide

• Motivation
• Bayesian set up recalled
• Consequences of Bayesian set up
• Modeling
• Multiple-participants modeling
• Design of DM strategies
• Concluding remarks

14
E Bayesian calculus
min EZ Emin EZa,? R ? a
a
Basic DM lemma
?(d(T)) EZ(d(T), ?(T))d(T) t T, T-1,
,1 ?(d(t-1)) min E?(d(t))at, d(t-1)
at
Sequential DM
Optimal R(T) randomized
supp f (at d(t-1)) Arg min E?(d(t))at,
d(t-1)
f (?(T)d(T)) Bayesian filtering
Needed pdfs
f (ytat, d(t-1)) Bayesian prediction
how to filter predict ?
15
Observation evolution models
NC DM
f(at, ?t d(t-1)) f(at d(t-1), ?t ) f(?t
d(t-1))
Observation model
f(yt at, d(t-1), ?t)
relates seen to unseen
models unseen
f(?t1 at1, d(t), ?t)
Evolution model
Predictive pdf
f (yt at, d(t-1)) ? f(yt at, d(t-1), ?t)
f(?t at, d(t-1))d?t
f(?t d(t)) ?
data update
f(yt at, d(t-1), ?t) f(?t d(t-1))
Filtering
time update
f(?t1 d(t)) ? f(?t1 at1,d(t), ?t) f(?t
d(t))d?t
Prior pdf f(?1 d(0)) f(?1) belief in
possible ?1
why f(?1) gt 0 when S(T) ? M(T) for any ?1?
16
Bayesian paradigm reality
World part generating d(T)
Prior pdf belief that ?1 is
the best projection
S(T)
Nearest model
unknown as S(T) unknown
Posterior pdf belief in ?1 corrected by data
d(T)
Model set indexed by ?(T)
any practical consequence ?
17
Projection consequences

• World model is a subjective thing

• Bayes rule learns the best projection ?
  minimizes KL distance

• KL distance is an inherent Bayesian discrepancy
  measure

• Quality of the best projection depends on M(T)
• careful modeling of reality pays back

• Projection error cannot be measured

any chance to get information about better
models ?
18
Model comparison
f(d(T)M?)? f(d(T),??(T)) d??(T)
f(d(T)M?)? f(d(T),??(T)) d??(T)
Model set M? indexed by ??(T)
Model set M? indexed by ??(T)
The best M ? M? M? uncertain
Point estimation
Model combination
f(M d(T)) ? f(d(T)M) f(M)
f(d(T)) ?M f(d(T)M) f(M)
avoids unnecessary DM
preserves complexity
lesson ?
19
Lesson from model comparison

• Values of predictive pdf serve for model
  comparison

• Compound hypothesis testing is straightforward
  if alternative
• model sets are specified (modeling!)

• Unnecessary DM making should be avoided (valid
  generally!)

• No new techniques are needed

just modeling !
where we are ?
20
Guide

• Motivation
• Bayesian set up recalled
• Consequences of Bayesian set up
• Modeling
• Multiple-participants modeling
• Design of DM strategies
• Concluding remarks

21
Modeling
Sub-Guide

• Practical modeling
• - finite memory
• - exponential family

• Quantification of prior knowledge
• Mixtures universal approximation
• Mixed-data modeling

• Multiple-participants modeling
• - relationships of data spaces
• - non-closure problem of mixture set
  

lets go!
22
Practical modeling
What yt I can expect knowing something?
deterministic regression obeying conservation
laws
Eytsomething, ?Eyt?t, ? y(?t, ?)
Everybody has finite memory
?t at, , at-m, yt-1, , yt-n
If conditional variance r constant model
closest to uniform
f(yt at, d(t-1), ?) f(yt ?t, ?) Ny (y(?t
,?), r),
? (?,r)
t
source of richness
Estimation feasible iff y(?t, ?) ? G(?t)
linear in ?, nonlinear in ?t
why Ny dominates?
23
Dynamic exponential family
?t yt, G(?t) data vector
?
f(yt?t, ?) A(?) exp B(?t),C(?)
linear in array B(?t) B(dt, ?t-1)
?
f(?) ?A (?)exp V,C(?)
self-reproducing pdf ? conjugate pdf
Evolution of sufficient statistics V V B(?),
? ? 1
Continuous y, ?? y,1
N A(2?r)-0.5, B -0.5? ?, C r -1-1,?-1,?
?
?
?
?
?
A1, B? ?(?,?), C? log(Pr(y ?))
Discrete ? MC
N conjugate pdf GiW, data updating ? least
squares
MC Di,
? counting ys in ? groups
prior knowledge mapped on structure of ?, form
G (?t) V, ?
24
Sources of prior knowledge

• obsolete analogical data
• designed values
• experiments in non-standard setting
• simulation results
• expert knowledge opinion

Sources

• nature
• precision
• reliability
• degree of repetitions
• degree of mutual compatibility

Differences in
a methodology of handling ?
25
Prior knowledge quantification
V? B(?) select ?ki expressing knowledge Ki
set Vi?kB(?ki)
how to get ?ki yki ?ki ?
Given ?ki, uncertain function H(yki) its
expectation h(?ki), find
the flattest conjugate f(? Vi, ?i) meeting
EH(yki) ?kih(?ki)
Ki consistent iff ? Vi, ?i meeting the
restrictions ? k
Merging of K(I) into posterior pdf given V(I),
?(I) real data d(T)
VT ? t B(?t) ?i f( Ki d(T))Vi , ?T T?i f(
Ki d(T))?i with
f( Ki d(T)) ? f(d(T) Vi, ?i)
where we are?
26
Modeling
Sub-Guide

• Practical modeling
• - finite memory
• - exponential family

• Quantification of prior knowledge
• Mixtures universal approximation
• Mixed-data modeling

• Multiple-participants modeling
• - relationships of data spaces
• - non-closure problem of mixture set
  

lets continue!
27
Mixture modeling estimation
Mixtures universal asymptotic approximation used
if EF insufficient
? piece-wise EF on a rich finite covering d(t)
?c suppc
f(ytat,d(t-1),?)??c Pr(d(t)? suppc
at,d(t-1),?)A(?c)exp B(?ct ),C(?c)

?ct a.s. ?c
?(ct,c)
Marginal of f(yt,ct at,d(t-1),?) ? c
?cA(?c)exp B(?t ),C(?c)
QB estimation prior pdf conjugated to
individual components
?(ct,c) ? wct Pr(d(t) ? suppc at, d(t-1))
(no ?)
wct ? c-th predictive pdf
Vct Vct-1 wct B(?ct ), ?ct ?ct-1
wct
mixed data ?
28
Mixed data modeling
Model of yt with continuous quantitative
discrete qualitative parts
f(yt?t ,?)?i f(yit yi1t,, ynt,?t ,?i)?i
f(yit ?it ,?i)
, ?ityi1t ?i1t
Factors f(yit ?it ,?i) from EF
simple (Q)B estimation
?i
Discrete entries Continuous entries
y D C
Variants
No EF!
EF MC
logit, probit ?
EF N regression
EF N regression
one of still neglected topics !
29
Guide

• Motivation
• Bayesian set up recalled
• Consequences of Bayesian set up
• Modeling
• Multiple-participants modeling
• Design of DM strategies
• Concluding remarks

30
Multiple-participants modeling
Participant X with
Participant Y with
common XY quantities


QX
QXY
QY

surplus quantities
surplus quantities
X, Y learnt independently
f(QX, QXY) f(QY, QXY)
marginal
?
f (QXY ) ? f(QXY) (1-?) f(QXY)
Shared model is mixture
? past predictive pdfs
?
?
Extension by chain rule
f(QX,QXY,QY) f(QX QXY)f(QY QXY)f(QXY)
conditional
solution minimizes expected KL distance its
choice version were motivated by asymptotic
Bayes rule!
31
Use of multiple modeling
Desired d, ?
Predicted d
Estimated ?
Common QXY
Common prediction
Polled estimate
Compromise in ideals
Shared mixture
model or ideal for global DM
Extended model
sensors or multiple information fusion
the only reasonable one?
Is the chosen extension
order independent with more participants?
a sample of open important problems !
32
Non-closure problem
universal approximation

Set MN of mixtures with N terms
shared model
marginal evaluation


MN MN
normalization



MN x MO MN O
sharing
Operations on pdfs



chain rule
MN x MO MN x O
curse of dimensionality


conditioning

MN x MO M
unsolved non-closure problem
even worse in design part !
33
Guide

• Motivation
• Bayesian set up recalled
• Consequences of Bayesian set up
• Modeling
• Multiple-participants modeling
• Design of DM strategies
• Concluding remarks

34
Modeling of aims loss
Sub-Guide

• Practical design
• - LQ
• - horizon length
• - state what you want
• Specification of set-points revised
• Fully probabilistic design

35
Practical LQ design
Model loss in DDM restricted by curse of
dimensionality
Feasible low dimensional MC (mixtures of)
yt ? Nyt(?0 at ?i ?ia at-i ?iy yt-i, r)
Linear ARX
Quadratic loss
Z(d(T),w(T)) ? t ? T dt wt2
Horizon T how far ahead you
optimize
given by
Kernel of norm how sensitive is Z to
deviations
Set-point w(T) where you would like to
drive d(T)
let us comment them !
36
State what you want !
T
Planning time qualified by Z
1
T
Real time
Optimization for Tgt1 hard why not repeat it with
T 1 ?
up to instability ? min E ?
min E ? ? ? min E
Short-sighted policy wasteful even dangerous
optimal ? may be unacceptable
? 0 for ? ? 0
Try Z ? t ? T yt2 for T ? , yt
at 1.1 at-1 yt-1 , y0 ? 0
something else ?
37
Modeling of aims loss
Sub-Guide

• Practical design
• - LQ
• - horizon length
• - state what you want
• Specification of set-points revised
• Fully probabilistic design

38
Set-points w(T)
Minimized EZ(d(T),w(T)) Ed(T) w(T)2

Needed joint model f(d(T),w(T)) f(d(T)w(T))
f(w(T))
f(d(T),w(T)) ?t f(yt at,d(t-1),w(t))
f(atd(t-1),w(t)) f(wtw(t-1),d(t-1))
f(d(T),w(T)) ?t f(yt at, d(t-1)) ?
f(atd(t-1),w(t)) f(wtw(t-1))
known values pre-program strategy
R(d(t-1),w(T)) at
Models of w(T)
random walk, trend etc. full feedback
R(d(t-1),w(t-1)) at
lady following wt predicted position R
full feedback
man escaping wt - predicted position R full
feedback
what about qualitative variables ?
39
Specification of w(T) revised
Joint pdf of influenced variables d(T), ?(T)
f(d(T), ?(T)) ?t f(yt, ?t at, d(t-1), ?(t-1))
f(at d(t-1))
optimized R(T)
model of S-R
best model of S(T)
Strategy influences pdf of S-R in known
deterministic way
Choose R(T) to make this pdf close to ideal pdf
around desired w(T)
f I(d(T), ?(T)) ?t f I(yt, ?t at, d(t-1),
?(t-1)) f I(at d(t-1))
Choose R(T) to make KL distance of f(d(T),?(T))
to f I(d(T),?(T)) small
Fully probabilistic design
solvability properties ?
40
Fully probabilistic design

• Suitable for mixed-data case with qualitative
  variables
• Conceptually plausible common language with
  modeling
• lady following f I model of her
  behavior
• Ideal pdf can be related to the data
  distribution
• for f(d) Nd(d,r) define f I(y) Nd(w,
  r)
• Democratic common ideal constructed by sharing
  extending
• Optimal strategy found explicitly
• Approximate feasible solution for normal mixtures

open man escaping, improvements for normal
mixtures
41
Guide

• Motivation
• Bayesian set up recalled
• Consequences of Bayesian set up
• Modeling
• Multiple-participants modeling
• Design of DM strategies
• Concluding remarks
• - lessons
• - unsaid

42
Lessons

• Bayesian set up is well justified well working
  methodology
• Everything has to be modeled as carefully as
  possible
• No new methods are needed just modeling
• Approximations coping with complexity are
  necessary
• Good unified software is decisive
• Lot of open problems exist

43
Non-discussed points

• Joint learning and design (duality)
• Approximations (functional, equivalence based,)
• Software (research, analysis, communication,
  GUI,)

44
THE END

• Motivation
• Bayesian set up recalled
• Consequences of Bayesian set up
• Modeling
• Multiple-participants modeling
• Design of DM strategies
• Concluding remarks
• - lessons
• - unsaid