Cooperative Systems and Related Topics in

1
Cooperative Systems and Related Topics in
Decentralized Control, Communications, and Game
Theory
Robert Murphey Air Force Research
Laboratory Guidance and Control
Branch AFRL/MNGN Eglin AFB, FL
murphey_at_eglin.af.mil [phone] ext.
3453
2
Outline of Talk

• Definition of cooperative systems
• Issues with cooperative systems
• Classical team theory
• Existing Information structures
• New information / precedence structures
• Efficiency in organizations (cooperative
  systems)
• Minimal communications in organizations
• Passive communication
• Stability

3
What Are Cooperative Systems?
Multiple dynamic entities that share constraints,
information, or tasks to accomplish a common ,
though perhaps not singular, objective.
4
What Are Cooperative Systems?
Multiple dynamic entities that share constraints,
information, or tasks to accomplish a common ,
though perhaps not singular, objective.
Aspects of Cooperative Systems

• Continuous and discrete dynamics (states).
• States may be interlinked (joint) between
  dynamical entities.
• Communication between entities or ability to
  observe joint states.
• Authority (hierarchy) may exist in the system.
• May exist conflicting objectives, not all
  shared globally.
• Uncertainty abounds.

5
Performance Criteria For Cooperative Systems

• Solution Quality
• Global and local
• Performance bounds
• Graceful degradation with a loss of agents
• Stability
• Robust to local pertubations in decisions and
  communication/observation
• Global convergence
• Solution Rates
• Inherently on-line algorithms
• Decomposition/separation of larger problem
• Examples
• Event driven - continuous dynamics
• Resource
  allocation with conflicting objectives

6
Fundamental Assumptions For Cooperative
Systems (The Three Commandments of Cooperative
Systems)

• 1. Total cost added (radio and processing) must
  provide a greater increase in expected system
  effectiveness than like cost being used to add
  additional (non-cooperative) agents.
• 2. Performance lower bound of a cooperative
  system as links degrade (noise) should never be
  worse than performance of same agents without
  cooperative algorithms.
• 3. System performance may degrade as agents are
  lost (through allocation or attrition). Total
  effectiveness should remain better than
  non-cooperative system.

7
Team Theory and the Theory of Organizations
Introduced by Marschak 55 developing a theory
of organization from game theory. Definition
A team is a "group of people each with access to
different information, each deciding about
something different, but participating in a
common payoff as a result of their joint
decision." Radner 62. Equivalently, "one
individual making different decisions in
successive time periods, the payoff being a
function of all the decisions over the total time
period. Implies 1. Team decision problems
can always be decomposed into a single person
sequential decision problem (stochastic control)
with zero memory. 2. Team decision problems are
static in nature, i.e. you can't have multiple
decision makers each acting simultaneously in
multiple time periods.
R. Radner, Team decision problems, Ann. Math
Statistics, Vol 33, No. 3, pp. 857-881 (1962).
J. Marschak, Elements for a theory of
teams, Management Science, Vol. 1, pp. 127-137
(1955).
8
Formulation of Team Problems
Radner 62 N decision makers (DM) A decision
is an N-tuple u (u1, u2, ..., uN) s.t.
individual decision variables, ui, ui ? Di
where u ? D ? D1 x D2 x ... x DN. The payoff c
is a function of the decision, u, and the "state
of the world z (z1, z2, ..., zN)
Assumes z is a random vector (random
events). Implication state of the world, as
observed by any single DM never depends on the
actions of other DMs. OK for modeling static
teams but not for all dynamic teams. Radner
finds that if payoff is quadratic and linear
w.r.t. zi which are distributed normal, then
optimal decision is linear w.r.t. zi.
9
Information Structures
Based upon work by Marschak, Radner 72 and Ho,
Chu 72. Information at DM i assumed to be a
known linear function of a random vector ? and a
subset of the decisions ui of other
DMs Causality requires Define (1) j is
related to i, jRi if (2) j
precedes i, iff jRi or jRr, rRs, kRi
i 1
Precedence line
i 3
i 2
Memory line
i 4
Precedence Diagram
Y.C. Ho and K.C. Chu, Team decision theory and
information structures in optimal
control problems-part 1, IEEE Trans. Automatic
Control, Vol. AC-17, No. 1, pp 15-20, (1972).
J. Marschak and R. Radner, Economic Theory of
Teams, Yale University Press, New Haven CT.
Y.C. Ho and K.C. Chu, Information structure in
dynamic multi-person control problems,
Automatica, Vol. 10, pp 341-351 (1974).
10
Information Structures Static Pattern
Simply the Radner team theoretic model (1) N
decision makers without time or 1 decision maker
over N intervals (2) Each DM independent
informationally or no memory From Radners
result,
i 1
i 2
i 3
i N
11
Information Structures Classical Pattern
Derived from stochastic control formulation.
Single decision maker making a sequence of
decisions in successive time periods based on
accumulating information. Assumes infinite
perfect memory for decisions and state of the
world. Where xi is the system state at
stage i, yi is the observation at stage i, wi and
vi and x1 are random variables, independently and
normally distributed. Solution if
unconstrained with a quadratic performance index
optimal solution is affine.
i 1
i 2
dynamics
observations
i 3
i N
12
Information Structures Partially Nested Pattern
Generalizes the classical pattern. For each DM i
and all j preceding i, the information zi infers
zj . Result action of precedents always
determined at i. Solution if unconstrained
with a quadratic performance index optimal
solution is affine. Why? Partially nested
structures are always equivalent to a static
structure.
i 1
i 3
i 2
i 4
i 5
i 6
13
Other Information Structures

• Witsenhausen 68 showed that for
  non-classical information structures (limited
  memory) that affine solutions cannot be
  guaranteed. This extends to non-partially nested
  structures as well.
• Geanakoplos, Milgrom 91 point out
• (1) DMs typically have finite attention therefore
  assumption of partial nested structures, i.e.
  that zi infers zj , is not always merited.
• (2) DMs in a management hierarchy may direct
  subordinates constrain actions or supply payoff
  function (not necessary for infinite attention).
• gt A cost exists for infering deep
  information from precedents, efficiency
• is also a factor.
• gt Existing information structure theory
  breaks down for passing constraints
  and payoff functions.
• Results for regular hierarchies,
• (1) Optimal organizations find DMs specializing
  (information selective).
• (2) Limit to depth of hierarchies.
• (3) More prior information gt refined
  information systems permit less efficient DMs.

J. Geanakoplos, P. Milgrom 91, A theory of
hierarchies based on limited managerial
attention, J. of the Japanese and International
Economies, Vol. 5, No. 3, pp 205-225 (1991)
H.S. Witsenhausen, A counterexample in stochastic
optimum control, Siam J. Control, Vol. 6, No. 1,
pp 131-147 (1968).
14
New Information Structures

• Constraining subordinate actions or supplying
  them payoff functions requires new information
  structure theory.
• Complexity indicated by cooperative
  N-person non-zero sum games in extensive form.
• Causality requirements due to assumptions
• (1) zi always explicitly a function only of other
  DMs actions
• (2) xi can always be recovered from ui for fixed
  payoff functions, known
• dynamics and infinite attention.
• New information structure.
• Use converse assumption uj, j ? i,
  recoverable from knowledge of xj with fixed
  payoff and known dynamics.
• Actually explicit knowledge of uj often
  unnecessary.
• Causality no longer an issue.

15
Two Stage, Two Decision Maker Example

• Ho Chu Model.
• DM i acts on t 1 information from DM j at t
  2.
• Latency an issue depends on recursion rate of
  t.
• New Model.
• DM i acts on t 1 information from DM j at t
  1 ?.
• Latency not a function of recursion rate, only
  of transmission and processing time ?.

u1(1)
u2(1)
u1(2)
u2(2)
State link
x2(1)
x1(1)
x2(2)
x1(2)
16
T Stage, Two Decision Maker Partially Nested
u1(1)
u2(1)
u1(2)
u2(2)

• Ho Chu Model.
• DM i acts on t information from DM j at t1.
• Perfect memory.
• New Model.
• DM i acts on t information from DM j at t ?.
• Perfect memory.

u1(3)
u2(3)
u1(T)
u2(T)
x2(1)
x1(1)
x2(2)
x1(2)
x1(2)
x2(2)
x2(2)
x1(2)
17
New Information Structure Model For Dynamic,
Multi-DM Systems
Bidirectional State link

• Eliminate bad features of precedence in
  information
• Latency not tied to recursion rate
• Model finite attention of DMs, ability to
  obtain deep information
• Limit to depth of inference (memory)
• Efficiency of DM (type and depth)
• Still have need for precedence
• Constrain actions of subordinates
• Supply payoff function to subordinates

x1(1)
x2(1)
x3(1)
x2(2)
x3(2)
x1(2)
Information Graph
Multi-Graph
x1(1)
x2(1)
x3(1)
x3(2)
x2(2)
x1(2)
Where ?i is attention span of DM i
Precedence Graph
18
Efficiency of Organizations

• Deng, Papadimitriou 99 Solve LP within a
  hierarchical multi agent organization
• k Decision variables partitioned among N agents
• Common constraints
• Each agent has different and possibly
  conflicting objective
• Each agent knows all other agents objectives
• Authority gt precedent DM make decisions
  prior to subordinates.
• How efficient is this organization? Efficiency
  is ability to meet individual and precendent
  objectives.
• For an arbitrary partition, the necessary and
  sufficient condition for efficiency is that
  precendents agree on the value (cost) of
  variables assigned to each subordinate.
• Defining the partition a priori s.t. the
  organization is efficient is NP-complete (3SAT
  reduction).

X. Deng, C. Papadimitriou, Decision making by
hierarchies of discordant agents, Mathematical
Programming, Vol. 86, No. 2, pp 417-431 (1999).
19
Communication Efficiency of Organizations
Marschak, Reichelstein 98studied cost of
communication and hierarchies that minimize cost
for performing shared tasks. Agent index set
N 1,2,,n Decision index set A
1,2,a,l 1. Value function. Decision
vector z z1, z2,
za,, zl?Z Z1 x Z2 x x Zl Agent value
function ei(.) ?Ei gt all concave
functions Agent i decision concerns Ai
?A Decisions concerning agent i
zAi Organization value function ?(e) arg
maxz?Z ? ei(zAi) 2. Organization Message
Space. Space of messages from i to j Mij I/O
message space for i Pi(M) Mi x Mi
T. Marschak, S. Reichelstein, Network mechanisms,
informational efficiency, and hierarchies, J. of
Economic Theory, 79, pp 106-141 (1998).
20
Communication Efficiency of Organizations
Communication burden for agent i dim
Pi(M) Organization communication burden dim M
1/2 ? dim Pi(M) 3. Outcome Functions. Goal is
to translate information into decisions. First
partition l decisions among n agents Partition
index for agent i Ji ? Ji A set of agents
concerned with decision za t(a) i?N a ? Ai
Lower bound on organization communication
burden for any rule set, any hierarchy dim
M ? ? 2(t(a) - 1)
i?N
i?N
a?A
21
Communication Transmission or Direct
Observation?
At least 2 ways to communicate active and
passive Agent A single dynamical entity in an
M entity system. Marginal State a state that is
exclusively a function of the dynamics of a
single agent. Joint State a state that is a
function of the marginal states of more than one
agent. For 2 agents For k agents
where I is a set of indices corresponding
to the k agents with a joint state. Marginal
Measurement Joint measurement by agent j
22
Cooperative System of 2 Agents Direct
Measurement of Joint State (Passive)
Controller 1
Plant 1
Measurement process 1
Measurement, not communication
Measurement process 2
Controller 2
Plant 2

• Assume
• Joint state is measureable,
• in general nonlinear and noisy
  gt

23
Cooperative System of 2 Agents Derived
Measurement of Joint State (Active)
Measurement process 1
Controller 1
Plant 1
Fusion process 1
Communication
Fusion process 1
Measurement process 2
Controller 2
Plant 2

• Assume
• Noise free (digital) communication
• Derivation of joint state from marginal states
  exists

24
Implication informational decentralization,
even on a local basis, may not be such a good
idea.
25
Stability of Cooperative Systems

• Must ensure systems are immune to
• Cascading failures
• Lack of global convergence
• Swaroop, Hedrick 96 derived sufficient
  conditions for asymptotic stability of a string
  of interconnected subsystems.
• Assume a string of r subsystems with a leader.
  Object is to maintain some inter linked dynamic
  measure even under subsystem perturbations.
• Example is the vehicle following system
• System dynamics xi f(xi, xi-1, , xi-r1)
  where xi-j 0 ?i ? j

leader
L2
L3
.
D. Swaroop, J.K. Hedrick, String stability of
interconnected systems, IEEE transactions on
automatic control, vol 41, No. 3, pp 349-357
(1996).
26
Conclusions
There is a lot of research that may be applied to
the study of cooperative systems. Team
theory Decentralized stochastic
control Resource allocation with stochastic
demand However, a fundamental theory does not
yet exist.