MarketDriven MultiAgent Collaboration in Robot Soccer Domain

1
Market-Driven Multi-Agent Collaboration in Robot
Soccer Domain
2
Todays Presentation

• Multi-Agent Systems
• Robot Soccer
• The Market Methodology
• Market-Driven Approach
• Reinforcement-Based Market-Driven Approach
• A New Approach

3
Multi-Agent Systems
4
Multi-Agent Systems

• Why use multi-agent systems?
• Multi-agent systems are becoming more popular
  than complex single agent systems because they
  eliminate the problem of single point of failure.

5
Multi-Agent Systems

• How do they work?
• Multi-agent systems work by decomposing a
  complex task into several low-level actions which
  can then be assigned to the individual team
  members.

6
Multi-Agent Systems

• How to assign tasks?
• This is a key problem, the system must break up
  the tasks and coordinate the team such that the
  team collectively completes the overall task.

7
Multi-Agent Systems

• How to assign tasks?
• The system must keep track of each robots
  capabilities (trivial in a homogeneous team, but
  more complicated in a heterogeneous team)

8
Robot Soccer
9
Robot Soccer

• Problem Domain?
• We will look at robot soccer as the problem
  domain as it provides a very good real world
  domain for developing multi-agent systems.

10
Robot Soccer Domain

• Robot soccer is a well-defined environment which
  provides a good test-bed for developing
  multi-agent strategies. Each robot has simple,
  clearly defined actions available and the overall
  task easy to understand Beat the other team.

11
Robot Soccer Domain

• Robot soccer provides a good way of comparing two
  systems/strategies. The two systems can simply be
  played against each other and see which team wins
  the most matches.

12
The Problem

• We need a way of coordinating the robots to each
  perform a task/fulfil a role (ie attack, support,
  defend, goalie etc).
• The Market-Driven Approach for coordinating the
  multi-agent system is based on the way
  free-markets maximize profits.

13
The Market Methodology
14
The Market Methodology

• The main goal in free-markets is the maximization
  of the overall profit. The theory is that if each
  participant in the market tries to maximize its
  profit, the overall profit should increase.

15
Market-Driven Approach
16
The Market-Driven Approach

• The Market-Driven Approach splits up the main
  task into simple tasks and an auction is then
  held for each task. The robots work out the cost
  for them to perform a task and then put in their
  best bid to the auctioneer. The robot which puts
  in the lowest bid gets the assignment.

17
The Market-Driven Approach
18
The Market-Driven Approach

• In Robot Soccer an auction is held for each of
  the different roles. The robots calculate the
  cost of fulfilling those roles (based on distance
  to ball etc) and bid on them. The robots with the
  best bid on each role will be assigned the role.

19
The Market-Driven Approach

• Two (or more) robots may get the same assignment
  where they must cooperate to perform the task (ie
  a robot with the ball attacks the goal and
  another robot supports it by driving close behind)

20
The Market-Driven Approach

• An advantage of the Market-Driven Approach is
  that each robot calculates the cost of performing
  each role and communicates that cost to the other
  robots. This cost value is much easier and
  quicker to communicate rather than sending all of
  the metrics to the other robots.

21
The Market-Driven Approach

• What about how the auction is run?
• Centralized
• Distributed
• Hybrid

22
Centralized

• There exists a master agent (auctioneer) that
  controls the auctions and assigns the roles.
• The master agent receives offers from all other
  agents for each task and sends the auction
  results back.
• Computationally efficient.
• Prone to single point failures.

23
Distributed

• No master agent.
• Every agent broadcasts its offer for every task.
• Every agent runs the same auction mechanism and
  parallely computes the auction results.
• Robust against single point failures
• Requires more computation in total.

24
Hybrid

• There exists a master agent
• There is also an auction for the task of being
  the master
• Robust against single point failures
• Computationly efficient
• Still not implemented, no test results.

25
The Market-Driven Approach

• Problem How to calculate the costs?
• Each robot must be able to calculate the cost of
  filling a particular role. The settings for the
  cost calculations must be calibrated, the
  performance of the system depends on the
  calibrations being correct.
• Eg. - Cattacker M2distBall M2distOppGoal

26
Reinforcement-Based Market-Driven Approach
27
Reinforcement Learning

• Reinforcement-Based Market-Driven Approach makes
  use of Reinforcement Learning (RL) to learn the
  role assignment process. RL is used when the
  agent is informed about the consequences of its
  actions. RL replaces the role assignment as
  described above.

28
Reinforcement Learning

• With the RL system, the robot closest to the ball
  assigns itself as the attacker and the remaining
  agents (excluding the static goalie) assign
  themselves according to a state vector. (see next
  slide)

29
Reinforcement Learning

• The Rules
• Goalie is statically assigned.
• The Robot closest to the ball is assigned the
  role of attacker.
• The other robots are assigned roles by a state
  vector.
• State vector metrics distances to the ball,
  goals, robots, the cost values and the closest
  player to the ball.

30
Reinforcement Learning
Broadcast Position and Cost Data
Calculate Attack Cost Array
Calculate Defence Cost Array
Closest to Ball
Cheapest
Yes
Yes
Shoot
No
No
Role Assigned According to Cost Value
Pass To Cheapest
31
New Approach
32
New Approach

• The New Approach is effectively a simplified
  version of the Reinforcement Learning system.
  However instead of using the exact positions of
  the robots, the field is divided into a grid.

33
New Approach
34
New Approach

• The system can now use this grid to make a
  decision on what role the robot should be
  performing. To assign roles, the system uses a
  state vector with the following metrics Ball
  Position (grid number), Ball Possession, Current
  Role assigned by the Market-Driven strategy,
  Teammate positions and Opponent positions.

35
New Approach

• This approach combines the Market-Driven Approach
  and the Reinforcement Learning based team with
  the grid separation of the board to keep the
  number of variables in the state vector to a
  minimum.

36
Results
37
Results

• The New Approach which combines the
  Market-Driven, RL and grid system out performs
  all of the other teams consistently over 90
  matches.

38
Questions?
39
References

• Kose, H., Kaplan, K., Mericli, C., Tatlidede, U.
  Akin, L. (2005). Market-Driven Multi-Agent
  Collaboration in Robot Soccer Domain. Cutting
  Edge Robotics, 407-416.
• Kurt, B. (2007). Bogazici University Robotics
  Server. Retrieved September 09, 2007, from
  http//robot.cmpe.boun.edu.tr/robsem/ailab_market.
  ppt