A Methodology for Improving the Cooperative Behavior of Hedonistic Multiagents

1
A Methodology for Improving the Cooperative
Behavior of Hedonistic Multi-agents

• Michael Helm
• Computer Science Dept TTU
• May 3, 2006
• Committee Dr. Cooke (chair)
• Dr. Becker, Dr. Pyeatt, Dr. Rushton

2
Overview of Objectives
R3
R7
R6
R12
R9
LARGE OBJECT
R2
R1
R11
R8
R10
R4
R5
TASK AN OBJECT TO BE MOVED REQUIRES COORDINATED
EFFORTS OF AT LEAST 6 OF THE 12 ROBOTS
3
How to achieve coordination?
R7
R3
R6
R9
LARGE OBJECT
R2
R1
R11
R8
R10
R4
R5
R12
AT LEAST 6 ROBOTS MUST COODINATE ACTIONS AND
DIRECTION, THE REST MUST NOT INTERFERE
4
Objectives and Definitions

• OBJECTIVE achieving cooperative behavior from a
  group of agents/robots
• MULTI-AGENTS a group of robots or other
  intelligent subsystems organized to achieve
  either a common goal or a higher level objective
• HEDONISTIC acting to satisfy internal goals
  rather than directed by an outside agent

5
Points of Discussion

• Why tackle this issue?
• Related work in this very large field
• Control system structures
• Coordination / Communications issues
• Hedonistic multi-agents as a solution
• Domains of application
• Questions of interest
• Hypothesis
• Early results
• The specific focus of this effort
• Why this is important

6
Related Work

• Bonabeau, Kube (Santa Fe Institute)
• shortest path solutions using ant-like agents
  and pheromones
• emergent cooperation in an object movement task
  with ant-like agents
• Wolpert (NASA Ames)
• collective intelligence with world based reward

7
Related Work

• Mataric (USC) robot formations
• No awareness of other robots
• Limited awareness of robots, avoidance follows
  fixed action pattern
• Actions mimic actions of the majority of other
  robots
• Kennedy (Purdue) swarm intelligence
• Balch (CMU) ,Arkin (GT) robot teams
• Goal/state communication
• Task performance improvement with some
  communications
• Korf (UCLA) cooperation from hedonism
• Cooperative behavior emerges from hedonistic
  actions

8
The Problem

• Control systems are pervasive.
• Increasingly complex
• Critical applications
• Increasingly difficult to design
• Scalability
• Reliability/robustness

9
Structures of Control Systems

• Monolithic systems vs distributed systems
• Monolithic omnipotent, complex, brittle
• Distributed with central control layered,
  delegated
• Distributed with distributed control more robust
  possibly higher comms cost
• System level coordination in distributed control

10
A
A
A
S
S
OMNIPOTENT CONTROLLER
S
S
S
S
A
A
11
A
A
S
S
SUB
SUB
A
SUP
SUP
S
SUB
SUB
S
A
CENTRAL CONTROLLER
S
SUB
S
SUP
SUP
SUB
A
HIERARCHICAL LAYERED DELEGATED
A
SUB
SUB
A
S
A
S
12
A
A
S
S
SUB
SUB
Distributed control with full comms O(n(n-1))
SUB
S
A
SUB
SUB
A
S
A
S
13
A
A
S
S
SUB
SUB
SUB
FULLY HEDONISTIC NO EXPLICIT COMMUNICATIONS COMMUN
ICATIONS IS VIA STIGMERGY, i.e. THE LOCAL STATE
OF THE WORLD AS DETECTED BY THE SENSORS
S
A
SUB
SUB
A
S
A
S
14
A
A
S
S
SUB
SUB
Broadcast - O(n) if reading all msgs, O(1) if
nearest neighbor only
SUB
S
A
SUB
SUB
A
S
A
S
15
Increasing communications complexity
Every Agent to Every Agent
Central Control
Nearest Neighbor
None
Is this region more robust, efficient, and
simpler from a design standpoint?
CONTINUUM OF COMMUNICATIONS COMPLEXITY IN A
MULTI-AGENT SYSTEM
16
Hedonistic Multi-agent Systems

• Hedonistic agents have internal action/reward
  system
• Eliminates need for centralized control
• Communications of limited scope/range reduces
  overhead costs
• Economic Game Theory market approach
• Tolerant of single/multiple agent failure

17
Cooperative Behavior

• Organized via extensive communications?
• O(n2) for every agent to every agent
• Multiple messages for consensus
• Hedonistic agents - alternative
• Hedonistic goals fixed action patterns based on
  perception of local world state
• Limited communications
• Emergent cooperative behavior from resonance of
  actions over time
• DEMO ANT AGENT SIMULATION

18
Flexibility

• Homogeneous agents for tasks that partition into
  similar sub-problems
• Heterogeneous agents for tasks requiring
  multi-faceted approach
• Sub-tasks can be addressed multi-spatially and in
  parallel temporally
• Adaptable to dynamically sized task

19
Scalability

• Communications of limited scope/range
• Individual agents communications needs do not
  expand with larger task size
• Larger/smaller task addressed by modifying number
  of agents.
• Individual agents are less complex

20
Domains of Application

• Tasks involving exploration
• Tasks covering a large region of space
• Tasks that dynamically change scale
• Tasks that benefit from redundancy
• Where communication is difficult
• Tasks too complex or physically difficult for a
  single agent
• Tasks that benefit from lower cost, simpler
  agents (disposable?)
• Where results can emerge over time but do not
  require initial synchronization of all agents

21
Questions of Interest

• What are useful domains of application?
• To what extent are communications costs an
  efficiency factor in current systems?
• Is this approach efficient (duplicate effort)?
• Does this approach scale easily?
• Is it more robust?
• Does this make for simpler system design?
• Can this approach provide improved solutions?
• How does it fit into S-A, S-P-A models?

22
Hypothesis

• Cooperative behavior can emerge in multi-agent
  robotic systems with simple agents and highly
  constrained communications. Such behavior
  results from the resonance of reinforced
  actions from pursuing hedonistic goals. Reduced
  agent complexity and communications will result
  in a robust solution that is scalable and
  adaptable under dynamic circumstances, and it
  will be simpler from the design standpoint

23
Preliminary Results

• Investigations to date indicate emergent
  cooperation is possible in multi-agent systems
  with simple fixed action patterns and only
  stigmergy communications.
• Cooperative behavior can be learned in
  competitive tasks by RL agents where agents only
  consider their own hedonistic rewards

24
The Work Going Forward

• Extend the ideas of Bonobeau, Mataric, Korf, et
  al by
• Utilizing ideas from the behavior of social
  insects, particularly ants in nature
• Possibly applying simple Reinforcement Learning
  capability to the agents
• Using ideas from Economic Game Theory where
  agents perceive the local world state but do not
  have extensive agent to agent communications
• Allowing for the possibility of simple one-way
  pheromone nearest neighbor communications

25
What I Plan to Do

• 1. Determine communications efficiencies of this
  approach via analysis and experimentation
• 2. Define a minimalist set of pheromone-like
  communications for efficient performance with
  this approach
• 3. Determine system level efficiencies with this
  approach via analysis and experimentation

26
Specific Investigations

• Hedonistic multi-agents finding prime numbers
• Hedonistic multi-agents in coordinated object
  movement task
• Hedonistic multi-agents with competing interests

27
Why This is Important

• Such a system is potentially more robust
• Such a system appears to easily scale
• Such a system appears to have potential across
  dispersed spatial applications
• Reducing complexity of individual agents and
  communications should lead to simpler system
  level designs

28
References

• Arkin, R. Behavior-Based Robotics Cambridge,
  Massachusetts The MIT Press, 1998.
• Balch,T. and Parker, L., Eds. Robot Teams From
  Diversity to Polymorphism, A. K. Peters, 2002.
• Beckers, R., Holland, O.E., and Deneubourg, J.
  From Local Actions to Global Tasks Stigmery and
  Collective Robotics. In Artificial Life IV
  Proceedings of the Fourth International Workshop
  on the Synthesis and Simulation of Living
  Systems, R. Brooks and P. Maes, Eds., pp.
  181-189, Cambridge, MA MIT Press, 1994.
• Bekey, G. Autonomous Robots From Biological
  Inspiration to Implementation and Control
  Cambridge, Massachusetts The MIT Press, 2005.
• Bonabeau, E., Dorigo, M., Theraulaz, G. Swarm
  Intelligence From Natural to Artificial Systems,
  New York, NY Oxford University Press, 1999
• Brooks, R. A Robust Layered Control System for
  a Mobile Robot, IEEE Journal of Robotics and
  Automation, RA-2, April 1986, pp. 14-23.
• Brooks, R. Flesh and Machines
• Coulouris, G., Dolimore, J., and Kindberg, T.
  Distributed Systems Concepts and Design
  Addison-Wesley, 2001.
• Dutta, P. Strategies and Games Theory and
  Practice Cambridge, MA MIT Press, 1999, 3rd
  printing 2001.
• Detrain, C., Deneubourg, J., Pasteels, J.
  Information Processing in Social Insects Berlin,
  Germany Birkhauser Verlag, 1999.
• Feddema, J.T., Lewis, C., and Schoenwald, D.A.
  Decentralized Control of Cooperative Robotic
  Vehicles Theory and Application, IEEE
  Transactions on Robotics and Automation, Vol. 18,
  No. 5, Oct. 2002.

29
References

• Gat, E. On-Three Layered Architectures,
  Artificial Intelligence and Mobile Robots, David
  Kortenkamp, R. Peter Bonnasso, and Robin Murphy,
  Eds. MIT Press, 1998.
• Gordon, D.. Ants at Work How an Insect Society
  is Organized NY, NY The Free Press, 1999.
• Holldobler, B., Wilson, E.. Journey to the Ants
  Cambridge, Massachusetts The Belknap Press,
  1994.
• Kennedy, J., Russell, E. Swarm Intelligence, San
  Francisco, CA Morgan Kaufmann Publishers, 2001.
• Kim, J.H. and Vadakkepat, P. Multi-Agent
  System A Survey from the Robot-soccer
  Perspective, International Journal Intelligent
  Automation and Soft Computing, 6 (1), 3-17,
  2000.
• Klavins, E. Communication Complexity of
  Multi-Robot Systems In WAFR 02, Nice, France,
  December 2002.
• Kreps, D. Game Theory and Economic Modeling New
  York, NY Oxford University Press, 1990.
• Lewis, T. Insect Communication Orlando, FL
  Academic Press, 1984.
• Lui, J. and Wu, J. Multi-Agent Robotic Systems,
  CRC Press, 2001.
• Mataric, M. Behavior-Based Systems Main
  Properties and Implications, IEEE International
  Conference on Robotics and Automation, Workshop
  on Architectures for Intelligent Control Systems,
  1992.
• Mataric, M. Behavior-Based Control Examples
  from Navigation, Learning, and Group Behavior,
  Journal of Experimental and Theoretical
  Artificial Intelligence, special issue on
  Software Architectures for Physical Agents
  9(2-3), H. Hexmoor, I. Horswill, and D.
  Kortenkamp, eds., 1997, 323-336.
• McKelvey, R., Palfrey, T. An Experimental Study
  of the Centipede Game, Econometrica , Vol. 60,
  No. 4, July 1992, 803-836.

30
References

• Moravec, H. Robot Mere Machine to Transcendent
  Mind New York, NY Oxford University Press,
  1999.
• Murphy, R. Introduction to AI Robotics MIT
  Press, 1998.
• Ray, I. Game Theory and the Environment Old
  Models, New Solution Concepts, Department of
  Economics, University of York, Helsington, York,
  UK, January 2000.
• Sahin, E., Spears, W. (Eds), Swarm Robotics SAB
  2004 International Workshop Santa Monica, CS,
  July 2004 Revised Selected Papers Berlin,
  Germany Springer, 2005.
• Simon, H. The Sciences of the Artificial 2nd
  edl Cambridge, MA MIT Press, 1969, 1981, 8th
  printing 1994.
• Sutton, R., Barto, A. Reinforcement Learning An
  Introduction, Cambridge, MA MIT Press, 1998, 4th
  printing 2002.
• Szuba, T. Computational Collective Intelligence,
  New York, NY John Wiley Sons, 2001.
• Thornton, C. Truth From Trash How Learning Makes
  Sense, Cambridge, MA MIT Press, 2000.
• Wagman, M. The Ultimate Objectives of Artificial
  Intelligence Theoretical and Research
  Foundations, Philosophical and Psychological
  Implications, Westport, CT Praeger Publishers,
  1998.
• Ward, M. Virtual Organisms, New York, NY Thomas
  Dunne Books St. Martins Press, 1999, first US
  edition 2000.
• Williams, S. The Arguing A.I. The Battle for
  Twenty-first-Century Science, New York, NY
  Random House, 2002.

31
Questions/Comments?