Information Flow in Networks: Beyond Network Coding

1
Information Flow in Networks Beyond Network
Coding
Babak Hassibi Department of Electrical
Engineering California Institute of Technology
First Year Review, August 27, 2009
2
Theory
Data Analysis
Numerical Experiments
Lab Experiments
Field Exercises
Real-World Operations

• First principles
• Rigorous math
• Algorithms
• Proofs

• Correct statistics
• Only as good as underlying data

• Simulation
• Synthetic, clean data

• Stylized
• Controlled
• Clean, real-world data

• Semi-Controlled
• Messy, real-world data

• Unpredictable
• After action reports in lieu of data

3
Overview of Work Done

• Network information theory
• wired and wireless networks, entropic vectors,
  Stam vectors, groups, matroids, Ingleton
  inequality, Cayleys hyperdeterminant, entropy
  power inequality
• Estimation over lossy networks
• asymptotic analysis of random matrix recursions,
  universal laws for networks
• Distributed adaptive consensus
• Distance-dependent Kronecker graphs
• allow searchability

4
Network Information Theory

• Network information theory deals studies the
  limits of information flow in networks. Unlike
  point-to-point problems (solved by Shannon in
  1948) almost all network information theory
  problems are open.

relay
transmitter
receiver
y1
x1
P(y1x1,x2)
s1
y1
P(y1x)
x
y2
P(y2x)
y2
x2
s2
P(y2x1,x2)
transmitter
5
A General Network Problem
Network
Suppose each source wants to communicate with its
corresponding destination at rate

The problem with the above formulation is that it
is infinite letter, and that for each T it is a
highly non-convex optimization problem (both in
the input distributions and the network
operations).
6
Entropy Vectors and Wired Networks

• Consider n discrete random variables
  of alphabet size N and define the
  normalized entropy of as
• This defines a dimensional vector call an
  entropy vector
• The space of all entropic vectors is denoted by
  and can be shown to be a compact convex set
• Main Result Network information theory problems
  for wired networks can be cast as linear
  optimization over the set
• Problem A characterization of is not known
  for ngt3

7
Stam Vectors and Wireless Networks

• Consider n continuous random vectors
  of dimension N and define the Stam
  entropy of as
• This defines a dimensional vector call a
  Stam vector
• The space of all Stam vectors is denoted by
  and can be shown to be a compact convex set
• Main Result Network information theory problems
  for wireless networks can be cast as linear
  optimization over
• For n2, is characterized by the entropy
  power inequality

8
Network Coding

• Allows for combining information packets
• Yields significant improvement over routing
• Schemes have been linear, i.e., packets are
  combined using linear operations
• How to do optimal network coding is generally not
  known
• However, it is known that linear schemes are not
  generally optimal---more on this later

9
Entropy and Groups

• Given a finite group and n subgroups
  the dimensional vector
  with entries
• where is entropic
• Conversely, any entropic vector for some
  collection of n random variables, corresponds to
  some finite group and n of its subgroups
• Abelian groups are not sufficient to characterize
  all entropic vectors---they satisfy a so-called
  Ingleton inequality, which entropic vectors can
  violate
• this is why linear network coding is not
  sufficient---linear codes form an Abelian group

10
Codes from Non-Abelian Groups
If a and b are chosen from a non-Abelian group,
one may be able to infer them from ab and ba.
There is also a larger set of signals that one
may transmit.
11
The Group PGL(2,p)

• We have performed computer search to find the
  smallest finite group that violates the Ingleton
  inequality
• It is the projective linear group PGL(2,5) with
  120 elements
• It can be used to construct codes stronger than
  linear network codes

12
Entropy and Matroids

• A matroid is a set of objects along with a rank
  function that satisfies submodularity
• Entropy satisfies submodularity and therefore
  defines a matroid
• However, not all matroids are entropic
• A matroid is called representable if it can be
  represented by a collection of vectors over some
  (finite) field.
• All representable matroids are entropic, but not
  all entropic matroids are representable
• When an entropic matroid is representable, the
  corresponding network problem has an optimal
  solution which is a linear network code (over the
  finite field which represents the matroid)

13
The Fano Matroid
The Fano matroid has a representation only over
GF(2)
14
The Fano Network

• The souces are a,b,c
• Each link has unit capacity
• The sinks desire c,b,a, respectively
• What is the maximum rate?

15
The Fano Network Solution
Therefore the capacity is 3 The network only has
a solution on GF(2)
16
The Non-Fano Matroid
The Non-Fano matroid has a representation over
every field except GF(2)
17
The Non-Fano Network

• The souces are a,b,c
• Each link has unit capacity
• The sinks desire c,b,a, respectively
• What is the maximum rate?

18
The Non-Fano Network Solution
Therefore the capacity is 4 Network has a
solution except on GF(2)
19
A Network with no Linear Solution
h
j
k

• This network has no linear coding solution with
  capacity 7
• The linear coding capacity can be shown to be
  70/11 lt 7

20
Capacity is 7

• A non-Abelian group solution can be given
• Alternatively, view a,b,c,d,e,f,g on the LHS as
  elements of , and a,b,c,h,i,j,k on
  the RHS as elements of , such
  that

• The resulting capacity is

21
Matroid Representations

• Unfortunately, determining whether a general
  matroid is representable is a classical open
  problem in matroid theory
• However, the question of whether a matroid is
  binary representable has a relatively simple
  answer
• the matroid must have no 4-element minor such
  that all pairs are independent and all triples
  dependent---see matrix below
• Proposal Use the above fact to decompose an
  arbitrary network into two components a binary
  representable component, and a component
  involving U(2,4) minors
• the binary component can be solved since the
  space of binary entropic vectors can be computed,
  the nonbinary components have trivial solutions
  over any field (they are just U(2,4)s)---however,
  the two solutions must be glued at the end

22
Entropy and Cayleys Hyperdeterminant

• A possible candidate to study entropic vectors is
  the family of jointly Gaussian random vectors
• This question becomes related to the question of
  the principal minors of a given covariance matrix
• It can be shown that the principal minors of a
  matrix satisfy certain hyperdeterminantal
  relations
• We have used these facts to the show that
  Gaussian random vectors generate the space of
  entropic vectors for n3 and violate the Ingleton
  bound for n4
• We conjecture that they generate the entropic
  region for all n

23
Estimation and Control over Lossy Networks

• There is a great deal of recent interest in
  estimation and control over lossy networks.
• While in many cases (especially in estimation)
  determining the optimal algorithms is
  straightforward, determining the performance
  (stability, mean-square-error, etc.) can be quite
  challenging (see, e.g., Sinopoli et al).
• The main reason is that the system performance is
  governed by a random matrix Riccati recursion,
  which is incredibly difficult to analyze.

24
Large System Analysis

• When the dynamical system being estimated or
  controlled has a large state space dimension, we
  have proposed a method of analysis based on large
  random matrix theory.
• The contention is that when the system dimension
  and the network are large, the performance of the
  system exhibits universal laws that depend only
  on the macroscopic properties of the system and
  network.
• The main tool for the analysis is the Stieltjes
  transform of a random matrix A

s(z) E ( trace (zI A)-1 )/n
From which the marginal eigendistribution of A
can be found via
p(?) lim Im( s(?j?) )/2p
??0
25
Example

• Consider a MIMO linear time-invariant system with
  random output measurements that are randomly
  dropped across some lossy network

26
Consensus

• What is Consensus?
• Given a network where nodes have different
  values, update over time to converge on a single
  value
• In many cases, we would like convergence to the
  sample average
• Simple local averaging often works
• Motivation
• Sensor network application
• Synchronizing distributed agents
• Agree on one value to apply to all agents

27
Distributed Adaptive Consensus

• How to quickly reach consensus in a network is
  important in many applications
• Local weighted averaging often allows consensus
  across a network
• for example, Metropolis weighting (which requires
  only the knowledge of the degree of ones node)
  works
• However, if global knowledge of the network
  topology is available, optimal weights (to
  minimize the consensus time) can be found using
  semi-definite programming
• However, the semi-definite program cannot be made
  distributed, since the sub-gradient of the second
  largest eigenvalue requires global knowledge
• We have developed an algorithm that
  simultaneously updates both the local averages
  and the weights using only local information
• it computes the gradient of a certain quadratic
  cost
• It can be shown to reach consensus faster than
  Metropolis weighting

28
(No Transcript)
29
Overview of Work Done

• Network information theory
• wired and wireless networks, entropic vectors,
  Stam vectors, groups, matroids, Ingleton
  inequality, Cayleys hyperdeterminant, entropy
  power inequality
• Estimation over lossy networks
• asymptotic analysis of random matrix recursions,
  universal laws for networks
• Distributed adaptive consensus
• Distance-dependent Kronecker graphs
• allow searchability