MAS and MASRL Framework A OO Framework for Building Software Agents that Learn through Reinforcement

1
MAS and MAS-RL Framework A OO Framework for
Building Software Agents that Learn through
Reinforcement Learning

• José A. R. P. Sardinha, Ruy L. Milidiú, Carlos J.
  P. Lucena, Patrick Paranhos
• TecComm Group Software Engineering Laboratory
• PUC Rio, Brazil

2
Main Goals

• Build Applications using a Muti-Agent System in a
  Distributed Environment
• Diminish Development time
• Code Reuse
• Extend the MAS Framework
• Learning through Reinforcement Learning
• Competition Environments
• Games
• Auctions
• E-commerce

3
Motivations

• The OO Framework was built to reuse code for the
  following Applications
• VGroups Sardinha - 2001
• Agents that build Consumer Groups
• Babilônia Clark 2002, Sardinha 2003
• Agents that perform Automatic Negotiation
• TIC-TAC-TOE Sardinha 2003
• Agents that learn to play
• Trading Agent Competition Melcop - 2003
• Agents to participate in the TAC Competition
• Gaia Methodology
• Models for the analysis and design phase

4
Trading Agent Competition
5
Introduction

• Agent Definition
• An autonomous entity driven by
• beliefs, goals, capabilities, plans
• and a number of behavioral properties, such as
• autonomy, adaptation, interaction, learning,
  mobility, etc.
• MAS Definition
• An Environment with
• Agents
• Objects

6
Introduction

• Agent Type
• Cognitive Agents and Reactive Agents
• Relationship between Agents and Objects
• Some flaws related to design and implementation
  of Agents
• The most practical programming language to
  implement the agent technology
• Large Scale Multi-Agent System
• System with many Agents
• Complex Properties
• Solve Complex Problems
• Distributed Problem Solving

7
MAS and MAS-RL Framework
8
MAS and MAS-RL Framework

• MAS Framework
• An easy mapping from the models built in the Gaia
  Methodology to OO Code
• Interaction Protocols and Services are divided in
  the OO Code
• MAS-RL Framework
• Extension of the MAS Framework 3 classes were
  added to introduce Reinforcement Learning in
  Competition Environment

9
Gaia Methodology
10
MAS Framework
11
MAS Framework

• Hot Spots
• Agent, AgentMessage, AgentBlackboardInfo,
  InteractionProtocols, AgentInterface
• Frozen Spots
• AgentCommunicationLayer, ProcessMessageThread

12
MAS-RL extensions to MAS
13
How to Instantiate the MAS Framework
14
How to Instantiate the MAS-RL Framework
15
Design Phase Decisions

• How to Model the Decision Tree ?
• Information of the Decision Points
• Decisions
• Size of the Tree
• Which Evaluation Function to Use ?
• Machine Learning Technique
• Percpetron, BackPropagation, PLS, etc.
• Which Reinforcement Learning Mechanism to Use?
• TD Learning, Adaptive Dynamic Programming

16
Design Phase Decisions

• Reinforcement Learning Management
• Critical Agents
• Learning Goals
• Training Competitors
• Training Rounds Schedule
• Current Ad hoc solution
• Future To be conceptually added to the
  Framework

17
Reinforcement Learning Adjustments and Testing
Phase

• Reinforcement Learning
• Reinforcement Learning Parameter adjustments
• Testing
• Benchmarks

18
Trading Agent Competition

• A Case Study

19
Trading Agent Competition
20
The MAS
Demand Segmentation
Captures Auction Information
Classifies Customers
Sensors
Allocation
Market Knowledge Base
Supervisor
Market
Results of the Solver
Obtains Segmentation Results
Classifies the Solvers Results
Negotiates the Goods
Sends Buying Orders
Solver
Offering Segmentation
Buying Orders
Negotiator
21
Decision Tree Modeling

• D1,D2,D3 -gt
• Sale(t?t) max(Sale(t), askPrice(t)?.
  ?askPrice),
• where
• ? 1n
• ?askPrice askPrice(t) askPrice(t-?t)
• R1,R2,R3 -gt
• askPrice askPrice(t) ?. ?askPrice
• where
• ? 1n
• ?askPrice askPrice(t) askPrice(t-?t)

Negotiator Decision
D1
D2
D3
Market Response
R1
R2
R3
22
Evaluation Function and Reinforcement Learning

• Evaluation Function Perceptron
• 8 inputs for the ?askPrice
• 8 inputs for the Sale(t)askPrice(t)

• Reinforcement Rule (Temporal Difference Learning)
• S(t- ?t) S(t) ß.(reward(t-?t)
  (S(t)-S(t?t)))
• S(t) Expected Output
• ß Reinforcement Learning rate
• reward(t) 0 or 1

23
Reinforcement Learning Management

• Critical Agents
• Negotiator Agent
• Learning Goals
• Buy the Goods in the Buying Order
• Training Competitors
• Dummies, LivingAgentLike
• Training Rounds Schedule
• MAS and 7 Dummies
• MAS, LivingAgentLike, and 6 Dummies
• MAS, 2 LivingAgentLike and Dummies
• MAS, and 6 LivingAgentLike

24
The Negotiator Agent Class Diagram
25
Testing Phase

• Evolution of the MAS versions

26
Final Comments

• A framework conceived from applications
• Re-use code and reduce development time
• A framework in evolution
• Complex Phase
• Reinforcement Learning Management
• Testing phase

27
Future Work

• Add the Reinforcement Learning Management to
  MAS-RL
• Add classes and a Tool
• Instantiate the framework with different
  Evaluation Functions
• PLS Regression Technique
• Add TAC MAS Agents to the RL Management
• Offering Segmentation Agent

28
Questions?