Learning the structure of Deep sparse Graphical Model

1
Learning the structure of Deep sparse Graphical
Model

• Ryan Prescott Adams
• Hanna M Wallach
• Zoubin Ghahramani

Presented by Zhengming Xing
Some pictures are directly copied from the paper
and Hanna Wallachs slides
2
outline

• Introduction
• Finite belief network
• Infinite belief network
• Inference
• Experiment

3
Introduction
Main contribution combine deep belief network
and nonparametric bayesian together.
Main idea use IBP to learn the structure of the
network
Structure of the network include Depth Width Conn
ectivity
4
Single layer network
Use Binary matrix to represent the network. Black
refer to 1(two unit were connected) White refer
to 0 (two unit were not connected)
Z
IBP can be used as the prior for infinite columns
binary matrix
5
Review IBP
1.First customer tries dishes. 2.
Nth customer tries Tasked dishes K with
probability
new dishes
6
Multi-layer network
7
Cascading IBP
Also parameterize by Each dishes in the
restaurant is also a customer in another Indian
buffet process Each matrix is exchangeable both
rows and columns This chain can reach the state
with probability one ( number
of unit in layer m) Properties For unit in layer
m1 Expected number of parents Expected number
of children
8
Sample from the CIBP prior
9
model
m refer to the layers and increase upto M.
weights
bias
Place layer wise Gaussian prior on weights and
bias, Gamma prior on noise precision
10
Inference
Weights, bias, noise variance can be sampled with
Gibbs sampler.
11
Inference( sample Z)
Two step 1. 2.
Sample existing dishes
MH-sample
Add a new unit and, and insert connection to this
unit with
MH ratio
For a exist unit remove the connection to this
unit with
MH ratio
12
Experiment result
Olivetti faces
Remove bottom halves of the test image.
13
Experiment result
MNIST Digits
14
Experiment result
Frey Faces