Multiagent Coordination and Cooperation:

1

• Multiagent Coordination and Cooperation
• challenges and techniques

• Sarit Kraus
• Bar-Ilan, Israel
• UMD,USA

2
No Agent is an Island

• Monitoring electricity networks (Jennings)
• Distributed design and engineering (Petrie et
  al.)
• Distributed meeting scheduling (Sen Durfee)
• Teams of robotic systems acting in hostile
  environments (Balch Arkin, Tambe)
• Collaborative Internet-agents (Etzioni Weld,
  Weiss)
• Collaborative interfaces (Grosz Ortiz, Andre)
• Information agent on the Internet (Klusch)
• Cooperative transportation scheduling (Fischer)
• Supporting hospital patient scheduling (Decker
  Jin)

3
Design of automated agents to interact effectively

• Coordinate to act upon one another in harmony
  (necessary)
• Cooperate to work together (beneficial)
• Example driving in Tel-Aviv v.s. Driving in a
  convoy.

4
Teams and Individuals

• Teams of agents that need to coordinate joint
  activities problems distributed information,
  distributed decision solving, local conflicts.
• Self-motivated agents acting in the same
  environment problems need motivation to
  cooperate , conflict resolution, trust,
  distributed and hidden information.

5
Cooperation and Coordination by Others

• Other entities coordinate their actions and
  cooperate in multi-entities environments humans,
  animals, computers, particles.
• Formal theories game-theory, decision theory,
  physics, logic.
• Non-formal theories organizational theories,
  political science theories, advisory
  negotiation.

6
Using other disciplines results

• No need to start from scratch!
• Required modification and adjustment AI gives
  insights and complimentary methods.
• Is it worth it to use formal methods for
  multiagent Systems?

7
Negotiations in the Pollution Sharing Problem

• Collaborator Esti Freitsis
• (forthcoming book Strategic Negotiation in
  Multiagent Environments, MIT Press)

8
Environment Description

• There are some closely grouped plants in an
  industrial region.
• Each plant can produce several types of products
  and. has a utility function (profit).
• There are several types of pollutants.
• Each plant has norms, restricting maximal
  emission of each pollutant that it emits. We
  refer to the situation when only these norms have
  to be carried out as usual circumstances.

9
Special circumstances

• Sometimes there is a need to reduce pollution for
  some period because of external factors such as
  weather (high humidity, wind towards residential
  area). In this case plants receive new norms. We
  refer to this situation as special circumstances.

10
Current solution

• Current solution each plant reduce pollution
  according to the new norms.
• Disadvantage for one plant it is less costly to
  reduce one substance while for another it is less
  costly to reduce another substance.

11
Negotiations

• Our solution plants negotiate to reach
  beneficial agreements about the emission of what
  substances and by which percent each of them must
  be reduced.
• The conflict solution following the new norms.
• First, we consider complete information
  situations.

12
Strategic Negotiation Model

• Model of alternative offers (Rubinstein) which
  takes negotiation time into consideration
  reduces negotiation time.
• During the strategic-negotiations agents
  communicate their respective desires to reach
  mutually beneficial agreement.
• The model provides a unified to many problems.

13
Structure of the Negotiation

• There are N self motivated agent, randomly
  designated 1,2,...
• All the agents negotiate to reach an agreement
• The negotiation process may include several
  equidistant iterations 0,1,2 ?Time and can
  continue forever. In each time period t, agent
  j(t) t mod N makes an offer.

14
Structure of the Negotiation - cont.

• The other agents respond simultaneously YES4
  or NO8 or OPTM.
• If the offer was accepted4 by all the agentsthe
  last offer is implemented.
• If at least one agent opts outM a conflict
  occurs.
• Otherwise (the offer was rejected8 by at least
  one agent), the negotiation proceeds to period
  t1.

15
Negotiations Protocols

• Simultaneous responsesan agent responding to an
  offer is not informed of the other responses.
• Sequential responses an agent responding to an
  offer is informed of the responses of the
  preceding agents (assuming that the agents are
  ordered).

16
Equilibrium

• Nash equilibriumA strategy profile p is a Nash
  Equilibriumif no player has a different strategy
  yielding an outcome that he prefers to that
  generated when it chooses pi.
• Subgame Perfect EquilibriumIf the strategy
  profile induced in every subgame is a Nash
  Equilibrium of this subgame.

17
Negotiations strategies for simultaneous responses

• For each possible agreement x that is better to
  all the plants than the conflict solution there
  is a subgame-perfect equilibrium of the
  bargaining game, with the outcome x offered and
  unanimously accepted in period 0.

18
Choosing the Allocation

• The owners of the plants can agree in advance on
  a joint technique for choosing x
• giving each server its conflict utility.
• maximizing a social welfare criterion
• the sum of the servers utilities.
• or the generalized Nash product of the servers
  utilities P (Us(x)-Us(conflict)).

19
Negotiations strategies for sequential responses

• Assumption there is a time period, T where
  negotiation cannot continue anymore. In T the
  conflict allocation is implemented.
• Perfect equilibrium by backward induction
• At T-1 if negotiations hasnt ended, AT-1
  suggests the best agreement to itself which is
  better to all agents than the conflict solution
  (denoted by OT-1 ) the other agents accept.
• At T-2, AT-2 suggests the best agreement to
  itself which is better to all agents than the
  conflict solution and OT-1 (denoted by OT-2).
  The other agents accept.
• By induction, at the first time period A0 O0 the
  others accept.

20
Assumptions about the environment

• Profit is a linear function of the number of
  items of each product produced by the plant
• Pollution is a linear function of the number of
  items of each product produced.

21
Techniques that were checked

• Sequential response backtracking
• Simultaneous response
• Maximization of the sum with guaranties of
  default profit (MaxSum)
• Maximization of the sum and Nash Products with
  side payments (MaxSumNash)
• Simplex - method for linear optimization
• Maximization of the Nash Product
• Praxis - method for multi-variable nonlinear
  function minimization.
• Hill Climbing

22
Simulation Parameters

• Number of plants is varied from 5 to 20.
• Number of pollution types is varied from 5 to 20.
  For each product pollution of some type is
  produced with probability 1/2.
• Each plant produces Max_prod different types of
  products. Max_prod is varied from 5 to 20.
  Pollution and profit per item of product and
  pollution constraints are set randomly.
• Results Average of 25 simulation runs.

23
Plants utility as the function of the number of
plants
24
Plants utility as a function of the number of
products
25
Plants utility as the function of the number of
pollutants
26
Conclusions (Complete Information)

• Simultaneous response
• If side payments are permitted the MaxSumNash
  method is the best.
• If side payments are not permitted either
  BackTracking or MaxSum should be used.
• Sequential response BackTracking should be
  used.
• Techniques game theory, heuristic search,
  optimization methods

27
Incomplete Information

• In real world situations the plants do not have
  complete information about each others utility
  function.
• Solution using economic theories for distributed
  mechanisms for reallocation of resource in
  markets with many agents and many divisible
  resources (Wellman 93).

28
General Equilibrium theory

• The general-equilibrium theory studies how the
  market prices are determined by the actions of
  the individuals.
• General equilibrium is obtained when a set of
  prices is found such that supply meets demand for
  each good and where the agents optimize their use
  of the goods at the current price levels.

29
General Equilibrium theory (Cont)

• Assumption each agent behaves competitively - it
  takes prices as given, independently of its
  actions.
• Used for distributed mechanisms for resources
  allocation in environments with many agents and
  many divisible resources (Welman).

30
Tatonnement

• It is a price-adjustment process (Wallras1954).
• The tatonnement process starts with some
  arbitrary price vector p0.
• The agents determine their demand at those prices
  and report the quantities demanded from an
  auctioneer.
• The auctioneer repeatedly adjusts the prices,
  pt1pt?(quantity_demanded-quantity_available )

31
Tatonnement (Cont)

• If the sequence p0,p1,... converges then the
  result is competitive equilibrium.
• However, the tatonnement process does not
  converge to equilibrium in general.
• Gross substitutability if the price for one good
  rises, the demand for other goods does not
  decrease.
• In the pollution allocation environment this
  condition does not hold.

32
Tatonnement (Cont)

• Moreover, in our case the utility functions are
  the result of constrained optimization and
  therefore the aggregate demand function is not
  continuous
• Thus, general equilibrium does not always exists!

33
Market Mechanisms

• We propose three algorithms for finding
  suboptimal solution of the pollution allocation
  problem.
• Tatonnement based mechanism Competitive
  Equilibrium Market (CEM) the allocation of the
  pollutants is performed only after the process is
  terminated very similar to WALRAS algorithm
  Wellman.

34
Greedy market mechanisms

• Market-Clearing with Intermediate Transactions
  (MCIT)
• Market-Clearing Intermediate Exchange (MCIE)
• A redistribution of the pollutants is done in
  each cycle of the mechanism. In MCIT a monetary
  transaction is performed after each cycle and in
  the MCIE exchange of two pollutants is done after
  each cycle.

35
The Three Market Mechanisms

• In all the mechanisms, at the beginning of the
  process the plants are allowed to emit their
  default allocation.
• In each cycle of the three mechanisms the
  auctioneer chooses one (or two in MCIE) of the
  pollutants randomly, and tries to determine its
  clearing price - the price at which demand is
  equal to supply, while keeping the prices of the
  other pollutants fixed. It uses binary search to
  find the clearing price.

36
Market Mechanisms (Cont)

• The process is terminated when the prices do not
  change for a predefined number of iterations, or
  when it reaches the predefined maximal number of
  iterations.
• The differences from the Tatonnement
• the procedure used to find the clearing prices
• the division of the pollutants given the clearing
  prices
• the maximization problem is solved by the plants
  when computing their demands.

37
The Influence of the Number of Plants on Plants
Utilities
38
The Influence of the Number of Products per Plant
on the Plants Utilities
39
The Influence of the number of pollutants on the
Plants utility
40
Conclusions (Incomplete Information)

• If side payments are permitted and the number of
  pollutants is small then MCIT method is the
  best.
• If side payments are not permitted or the number
  of pollutants is large then the MCIE method is
  the best.
• Techniques economics, heuristic search,
  optimization methods, binary search.
• Problem will each plant behave competitively??

41
Motivating Example
b upgrade software on a network of
workstations as part of a sys-admin
group tomorrow from 6-8 p.m.
g go to theatre with friends tomorrow from 7-9
p.m.
???

• Agent must reconcile intentions
• its intention to do the group task b
• a potential intention to do g

42
Problem Description

• Self-interested agents
• committed to a collaborative activity
• receive outside offers
• They need to reconcile intentions, deciding
  between
• defaulting on their group-related commitment
• rejecting the outside offer
• Agents assess outcomes using utility functions.
• How can agents be encouraged to consider the
  groups good?
• What utility functions should agents use?

43
SPIRE Simulation System(SharedPlans Intention
Reconciliation Experiments)

• Study the impact of
• group norms and policies
• agent utility functions
• environmental factors
• Goal provide insights that agent developers can
  use to develop collaboration-capable agents
  (Grosz, Sullivan, Das, Kraus)

44
Decision-theory Based Frameworks

• Multi-attributed decision making application
• Intentions reconciliation in SharedPlans
• Benefits using results of MADM, e.g., Specific
  method is not so important, standardization
  techniques.
• Problems choosing attributes assigning values,
  choosing weights.

45
Game-theory Based Frameworks(Non-cooperative
Models)

• Strategic-negotiation model based on
  alternating offers model of Rubinstein.
  Applications Forthcoming book Kraus, 2001
  MIT Press)
• pollution allocation
• Data allocation (Schwartz kraus AAAI97),
• Resource allocation , task distribution
• hostage crisis (Kraus Wilkenfeld).

46
Advantages and DifficultiesNegotiation on Data
Allocation

• Beneficial results proved to be better than
  current methods simple strategies.
• Problems
• Need to develop utility functions
• Finding possible action identifying optimal
  allocations is NP complete
• Incomplete information game-theory
  provides limited solutions.

47
Game-theory Based Frameworks(Non-cooperative
Models)

• Auctions applications
• Data allocation (Schwartz Kraus ATAL97,
  ICMAS00),
• Electronic commerce.
• Subcontracting based on principle agent
  models. Applications
• Task allocation (kraus, AIJ96).

48
Advantages and DifficultiesAuctions for Data
Allocation

• Beneficial results proved to be better than
  current methods.
• Problems
• Utility functions,
• Difficult to find bidding when there is
  incomplete information and the evaluations are
  dependant on each other no procedures Need to
  combine with learning.

49
Game-theory Based Frameworks(Cooperative Models)

• Coalition theories applications
• Group and teams formation (shehory kraus CI99).
• Benefits well-defined concepts of stability
  mechanisms to divide benefits.
• Difficulties utility functions, no procedures
  for coalition formation exponential problems.
• DPS model combinatory theories operations
  research (shehory kraus AIJ98).

50
Logical Models

• Building agents on top of any software packages.
• Logic is a basis for an agent programming
  language (Subrahmanian et al. Heterogeneous Agent
  Systems Theory and Implementation, MIT Press,
  2,000.)

service layer
message layer
code P
decision layer
authorization layer
per Wwrap
51
Logical Models

• Modal logic BDI modelsapplications
• Automated argumentation's (kraus, sycara
  eventchick AIJ99).
• Specification of sharedplans (Grosz Kraus
  AIJ96).
• Bounded agents (Nirkhe, Kraus,Perlis JLC97).
• Agents reasoning about other agents (Kraus
  Lehmann TCT88 Kraus Subrahmanian IJIS95).

52
Advantages and DifficultiesLogical Models

• Formal models with well studied
  propertiesexcellent for specification.
• Problems
• Some assumptions are not valid (e.g.,
  omnicience).
• Complexity problems.
• There are no procedures for actions required a
  lot of programming decision making developing
  preferences.

53
Physics Based Models

• Physical models of particle-dynamics
  Applications Cooperation in large-scale
  multi-agent systems freight deliveries within a
  metropolitan area. (Shehory
  Kraus ECAI96 Shehory, Kraus Yadgar ATAL98
  AIJ99).
• Benefits efficient inherits the physics
  properties.
• Problems adjustments potential functions

54
Summary

• Benefits formal models which have already been
  studied lead to efficient results. No need to
  invent the wheel.
• Problems
• Restrictions and assumptions made by other
  disciplines are not valid in real world MAS
  situations extensions are needed.
• It is difficult to develop utility functions.
• Complexity problems.