Tag mechanisms evaluated for coordination in open MAS

1
Tag mechanisms evaluated for coordination in open
MAS
ESAW07 October 23, 2007

• Isaac Chao (ichao_at_lsi.upc.es)
• Supervisors Oscar Ardaiz
• Ramon Sanguesa
• Distributed Systems group LSI (AI dept)
• UPC Barcelona

2
Outline

• Tag mechanisms and Group Selection
• Motivating applications
• PD and pure coordination games, simulator setup
• Tags evolving cooperation
• Tags evolving coordination
• Conclusions

3
Tag mechanisms

• Tags Social marks used as coordination mechanism
  to self-organize agents interactions in MAS
  (HOLLAND93) (RIOLO 00) (HALES 00 06) .
• Two steps mechanism
• Tag biased Interaction agents interact and
  derive utilities, interaction preferred with
  agents of same Tag
• Tag Evolution fitness comparison random agent,
  the loser copies the winner's strategy
  (cooperate or defeat) and Tag (i.e. Joins the
  winner's group) -gt LEARNING BY IMITATION
• eventually includes Mutation (variability)
• Game PD, Iterated (Riolo), Single round (Hales)
• System quickly emerge highly (90-99) cooperative
  groups (defined each by a tag).

4
Tag mechanisms (graphical view)
Interaction Mutual cooperation Payoff R
A3
Evolution Compare utility Copy Strategy Copy Tag
TAG A
A4
A3
A3
A6
A1
A5
A2
TAG B
A6
TAG C
Interaction Mutual defeat Payoff P
A6
Tag mutation
5
Group Selection

• The process behind Tag mechanism evolutionary
  learning Group Selection of fittest
  characteristics in agents
• Groups are formed around similar tags
• agents inter-group migration and imitation of
  fittest in destination groups
• enables the growing of more cooperative/coordinate
  d groups
• From Biology (natural selection levels issue) to
• explaining altruism between non kin individuals
  in human societies BOYD03, BOWLES04,
  HENRICH04
• firms co-evolution trough inter-firm competition
  at the group level CORDES06, HODGSON04.
• free-riding in P2P networks HALES05 (bitorrent
  success?) leader elections in groups in A-life
  KNOESTER07.

6
Outline

• Tag mechanisms and Group Selection
• Motivating applications
• PD and pure coordination games, simulator setup
• Tags evolving cooperation
• Tags evolving coordination
• Conclusions

7
The Grid (definitions)

• The Grid consists in coordinated resource sharing
  and problem solving in dynamic,
  multi-institutional Virtual Organizations VOs
  FOSTER01.
• VO Virtual entity englobing many physical
  organizations sharig a common goal.

The group structure already exists in Grids
8
Example of Grid scenario
VOP2
VO2
VO1
P3
P2
VOP1
O2
P1
O1
O3
9
Tags applied to the Grid

• Groups in Tag models ??VOs in a Grid
• Individual Agents ?? Physical Organizations
  (single resources or pools)
• Emergence of cooperation /coordination ??
  Emergence of profit maximizing cooperation-based
  VOs in the Grid

10
Outline

• Tag mechanisms and Group Selection
• Motivating applications
• PD and pure coordination games, simulator setup
• Tags evolving cooperation
• Tags evolving coordination
• Conclusions

11
Cooperation and competition in MAS

• Two fundamental games of game theory model the
  two basic coordination scenarios in MAS
• Pure Coordination Game Models cooperation, where
  roughly individual and social welfare match. This
  is a common scenario in fully cooperative MAS
  systems when many agents goals are to be
  aligned, leading to very suboptimal outcomes when
  this is not the case.
• PD Models competition of conflicting interests,
  (e.g. social dilemmas). The challenge is to
  provide incentives for the evolution of
  cooperation.

12
Prisionners Dilemma
T gt R gt P gt S, and the constraint 2R gt T S
makes a PD. E.g. T3, R 2 and PS1 Captures
social dilema of competitive exchange game (Nash
equilibrium Defect)
13
Pure Cooperation Game
The following relations must hold AgtC and DgtB.
Rational players will cooperate on either of the
two strategies to receive a high payoff.
14
Proposed Tag Algorithm
Interaction constrained to current group
Learns from agents in other groups
15
MAS learning mechanisms implemented in the
simulator

• Generalized TFT, with strategy is applied to the
  whole population.
• WSLS (Win stay, loose shift) , agents responding
  partner agent moves following its experienced
  utilities, rather than partners previous moves
• RL Basic Reinforcement Learning algorithm.
• Q-Learning
• Applies evolutionary learning based on imitation
  of the fittest (the same used by Tags, but
  applied to the whole population).

16
Simulator setup
Games matrix payoff instantiation
Simulator parameters
17
Outline

• Tag mechanisms and Group Selection
• Motivating applications
• PD and pure coordination games, simulator setup
• Tags evolving cooperation
• Tags evolving coordination
• Conclusions

18
PD results
19
PD evaluation

• Similar performance of TFT and Tags but several
  rounds before in Tag-based models.
• Pavlov strategy (WSLSjust able to maintain the
  initial random distribution of strategies
• The simple RL mechanism performs badly
  Q-Learning not able to promote further
  cooperation.
• Evolutionary mechanism is not able to sustain
  cooperation

20
PD Conclusion

• Niches formed by dividing the population in
  groups sharing Tag are essential in promoting
  cooperation
• Meets the context preservation referred in
  literature COHEN99). Tags perform between
  space-based and no context preservation (the
  plain population evolutionary algorithm)

21
Outline

• Tag mechanisms and Group Selection
• Motivating applications
• PD and pure coordination games, simulator setup
• Tags evolving cooperation
• Tags evolving coordination
• Conclusions

22
Pure coordination game results
23
Pure coordination game evaluation

• TFT and WSLS are not helpful (as expected)
• The RL does not achieve any important
  improvement. The more elaborate Q-Learning
  algorithm is able to evolve a small level of
  coordination
• Evolutionary algorithm also performs very well

24
Pure coordination game Conclusion

• It is is the evolutionary aspect from Tag
  mechanisms which is mostly provoking the
  convergence of actions in fully cooperative
  domains.
• but other coordination scenarios such as
  segmenting adaptive RL-based schedulers can
  increase coordination (cf. later)

25
Outline

• Tag mechanisms and Group Selection
• Motivating applications
• PD and pure coordination games, simulator setup
• Tags evolving cooperation
• Tags evolving coordination
• Conclusions

26
Conclusions

• Tag mechanism can be applied as a convenient MAS
  coordination mechanism in open MAS, without any
  costly assumption on agent rational or
  computational capabilities.
• Tags are the simple, requiring for the agent just
  maintaining a marker visible to the rest of
  agents and show equal or better performance in
  the two games than any other of the mechanisms
  tested
• Open questions
• Is the mechanism truly relevant to realistic
  Grids applications?
• If yes, is it possible an implementation in
  realistic Grid settings ?

27
In the mean time
1) Adaptive Job Scheduling (RL agents)
2) Economic-based resource allocation
Formalization into a Group Selection pattern for
MAS Group selection evolves small and dynamic
VOs into optimized outcomes in several Grid
Computing domain applications
3) Policy alignment in VOs
28
Future work

• Multiple Tags per agent and variations on the
  learning
• Extending the comparison to more general
  coordination and organizational mechanisms
  markets, tokens and scalable coalition-formation
  mechanisms.
• Deployment in a Grid prototype
• App domains Optimization of any artificial
  system structured in groups
• E.g. Automatic management of online communities

29
References

• HOLLAND93 J. Holland. The effects of labels
  (Tags) on social interactions. Working Paper
  Santa Fe Institute 93-10-064, 1993.
• HALES00 Hales, D. (2000) Cooperation without
  Space or Memory Tags, Groups and the Prisoner's
  Dilemma. In Moss, S., Davidsson, P. (Eds.)
  Multi-Agent-Based
• RIOLO00 R. Riolo. The efects of Tag-mediated
  selection of partners in evolving populations
  playing the iterated prisoners dilemma. Nature
  414, pages 441443, 2000.
• COHEN99 Cohen, M., R. Riolo, and R.
  Axelrod(1999) "The emergence of social
  organization in the Prisoner's Dilemma how
  context-preservation and other factors promote
  cooperation," Santa Fe Institute Working Paper
  99-01-002FOSTER01 I. Foster, C. Kesselman, S.
  Tuecke. The Anatomy of the Grid Enabling
  Scalable Virtual Organizations. International J.
  Supercomputer Applications, 15(3), 2001
• HENRICH04 Henrich, Joseph Cultural Group
  Selection, Coevolutionary Processes and
  Large-scale Cooperation. At target article in
  Journal of Economic Behavior and Organization,
  53 3-35 and 127-143. Complete with Commentaries
  and Reply, (2004)
• HODGSON04 Geoffrey M. Hodgson and Thorbjorn
  Knudsen, "The firm as an interactor firms as
  vehicles for habits and routines", Journal of
  Evolutionary Economics 14 (2004) 281307
• GOWDY03 Jonh Gowdy and Irmi Seidl. Economic Man
  and Selfish Genes The Relevance of Group
  Selection to Economic Policy, Journal of
  Socio-Economics 33(3), 2004, 343-358.
• BOWLES04Bowles S., Gintis H. (2004). The
  Evolution of Strong Reciprocity, Theoretical
  Population Biology 65, 2004, 17-28.
• BOYD03 R. Boyd, H. Gintis, S. Bowles, and P. J.
  Richerson. The Evolution of Altruistic
  Punishment. Proceedings of the National Academy
  of Sciences (USA) 100 35313535, 2003
• KNOESTER07 David B. Knoester, Philip K.
  McKinley, Charles Ofria Using group selection to
  evolve leadership in populations of
  self-replicating digital organisms. GECCO 2007
  293-300
• HALES05 Hales, D. Patarin, S. (2005)
  Feature Computational Sociology for Systems "In
  the Wild" The Case of BitTorrent. IEEE
  Distributed Systems Online, vol. 6, no. 7, 2005

30
End
Questions?