Connectionist Computing CS4018

1
Connectionist ComputingCS4018

• Gianluca Pollastri
• office CS A1.07
• email gianluca.pollastri_at_ucd.ie

2
Credits

• Geoffrey Hinton, University of Toronto.
• borrowed some of his slides for Neural Networks
  and Computation in Neural Networks courses.
• Ronan Reilly, NUI Maynooth.
• slides from his CS4018.
• Paolo Frasconi, University of Florence.
• slides from tutorial on Machine Learning for
  structured domains.

3
Lecture notes

• http//gruyere.ucd.ie/2007_courses/4018/
• Strictly confidential...

4

• 8 Mar 2007 Read four short descriptions
  (borrowed from a course by G.Hinton) on
  generalization, overfitting, early stopping,
  weight decay (links on the course web site).
• Email me a max 100x4 word summary by Apr 5th
  (midnight, any time zone). You are fully
  responsible for making sure I get the email
  before the deadline. Worth 5 of the final mark.
  1 off for every day you are late.

5
Bayesian Networks

• Networks that compactly represent a probability
  distribution and use the Bayesian machinery to
  update degrees of belief based on evidence.

6
Marginalising in BN

• What we want to do is
• Which means

7
distributing

• In
• we want to distribute the sums so that we are
  making the smallest possible number of operations.

8
example
A1,A2,A3,A4
A3, A4, A5
A4,A5
A4,A5
A3,A4
9
marginalisation by elimination

• We now know that we can marginalise a probability
  distribution wrt a variable (or set of variables)
  by successive eliminations.
• Not all elimination sequences carry the same
  complexity.
• Now the task is finding the best elimination
  sequence.

10
Domain graph

• We say that two variables are members of the same
  domain if they appear in the same conditional
  probability table of a BN.
• The domain graph for a set of variables is a
  graph with one node for each variable and an
  undirected edge between any two variables that
  are members of the same domain.
• This is sometimes also called moralised graph for
  the BN.

11
example
A
B
C
E
D
F
G
12
elimination from a domain graph

• We eliminate a variable A from a domain graph G
  by the following procedure
• add a link (fill-in) between any two neighbours
  of A
• remove A
• The new domain graph is called G-A. It can be
  shown that G-A is the domain graph for P(U\A).
• This means that we can perform variable
  elimination on the domain graph.

13
example eliminate A
A
B
B
C
C
E
D
E
D
F
G
F
G
14
example eliminate B
B
C
C
E
D
E
D
F
G
F
G
15
example eliminate C
C
E
D
E
D
F
G
F
G
16
example eliminate C first
A
B
A
B
C
E
D
E
D
F
G
F
G
order matters!
17
induced graph

• We call induced graph of G and an elimination
  order s (or s-completion of G) the graph Gs
  obtained by augmenting G with all the fill-ins
  associated with s.

18
triangulated graph

• An undirected graph G is triangulated if every
  cycle with more than three links has a chord (a
  link connecting two nodes not being neighbours in
  the cycle).
• A graph G is said to be a triangulation of G if
  G is triangulated, and G is a subgraph of G
  over the same nodes.
• Any s-completion of G is a triangulation of G.
• A graph is triangulated if, and only if, it has
  an elimination sequence without fill-ins.

19
induced graph and cliques

• Every time we eliminate a node A (eliminate a
  variable A) we create a clique, i.e. a fully
  connected subgraph of G containing all neighbours
  of A.
• Every maximal clique in an induced
  graphcorresponds to a intermediate factor in the
  computations
• Every factor stored during the process is a
  subset of some maximal clique in the graph

20
example
Elimination order A, B, C, D, E, F, G. No
fill-ins needed
A
B
C
E
D
F
G
21
induced width

• The size of the largest clique in the induced
  graph is an indicator for the complexity of
  variable elimination
• This quantity is called the induced width of a
  graph according to the specified ordering
• Finding a good ordering for a graph is equivalent
  to finding the minimal induced width of the graph

22
Elimination on Trees

• Suppose we have a tree
• A network where each variable has at most one
  parent
• All the factors involve at most two variables
• Thus, the domain graph is also a tree

23
Elimination on Trees

• We can maintain the tree structure by eliminating
  extreme variables in the tree

A
C
B
A
E
D
C
B
F
G
D
E
F
G
24
Elimination on Trees

• Formally, for any tree, there is an elimination
  ordering with induced width 1
• Inference on trees is linear in number of
  variables

25
PolyTrees

• A polytree is a network where there is at most
  one path from one variable to another
• Inference in a polytree is linear in the
  representation size of the network
• This assumes tabular CPT representation
• Check it if you wish..

26
General Networks

• What do we do when the network is not a polytree?
• If network has a cycle, the induced width for any
  ordering is greater than 1

27
Example

• Eliminating A, B, C, D, E,.

28
Example

• Eliminating H,G, E, C, F, D, E, A

A
A
B
C
D
E
F
G
H
H
29
General Networks

• It can be shown that finding an ordering that
  minimises the induced width is NP-Hard
• However,
• There are reasonable heuristics for finding good
  orderings
• There are provable approximations to the best
  induced width
• If the graph has a small induced width, there are
  algorithms that find it in polynomial time

30
Junction tree

• If G is a triangulated graph, and C1 .. Ck are
  its cliques, a junction tree for G is a graph
• whose nodes are C1 .. Ck
• such that each node on the path between Ci and Cj
  contains CinCj.
• The edge between two nodes is labelled with the
  intersection between the nodes.

31
example from triangulated graph to junction tree
T,V
T
A,L,T
B,L,S
B,L
A,L
A,L,B
X,A
A
A,B
A,B,D
32
Junction trees and triangulated graphs

• A graph is triangulated if and only if it has a
  junction tree.
• This means that for any BN, for each elimination
  order there is a junction tree.