TIELT: A Transfer Learning Evaluation Testbed for MultiAgent Adversarial Games

1
TIELT A Transfer Learning Evaluation Testbed for
Multi-Agent Adversarial Games
David W. Aha Intelligent Decision Aids Group Navy
Center for Applied Research in AI Naval Research
Laboratory (Code 5515) Washington,
DC david.aha_at_nrl.navy.mil
Testbed for Integrating and Evaluating Learning
Techniques (FY04-FY05 IPTO Seedling)
The information presented in these slides reflect
the views of the author, not those of DARPA.
30 March 2005 TL Bidders Conference Arlington,
VA
2
Context
Area A Transfer Learning Technology Development

• Area B Transfer Learning Evaluation Framework
• Testbed
• Experiment design and administration

3
Outline

• Overview
• Objective
• Description
• Status Features, Collaborations, Application
• Transfer Learning
• Example challenge problem
• Future Plans

4
TIELT Objective
Reduce the effort required to evaluate learning
techniques on performance tasks from complex
simulators.
Learning
5
Learning in Cognitive Systems
Status

• Few deployed cognitive systems integrate
  techniques that exhibit rapid enduring learning
  behavior on complex tasks
• Its costly to integrate evaluate embedded
  learning techniques

6
A Vision for Supporting Cognitive Learning
1986
7
Goal Reduce Integration Costs
Problem Simulator/system integrations are
expensive! (time, )
1 integration
Decision Systemn
8
Why Multi-agent Adversarial Games?
Attractive characteristics

• Several AI research challenges
• e.g., huge search spaces, adversarial,
  collaborative, real-time (some), on-line,
  uncertainty, multiple reasoning levels,
  relational (social, temporal, spatial) data
• Feasible Breakthroughs are expected to occur (w/
  funding)
• Inexpensive
• Military analog (i.e., training simulators)
• Popular (with public, industry, academia,
  military)

9
Gaming Genres of Interest(modified from (Laird
van Lent, 2001) (Fairclough et al., 2001))
10
Real-Time Strategy Games (informal intro)
Goal Defeat opponents, build Wonder,
etc. Measure Score f(time, achievements) Map
Size (e.g., 128x128), location, terrain, climate
Time Stamp Real-time advancement

• State
• Map (partially observed)
• Resources
• Units
• Attributes Location, Cost, Speed, Weapon,
  Health,
• Actions Move, attack, gather, Build generator,
• Generators (e.g., buildings, farms)
• Attributes Location, cost, maintenance,
• Actions Build units, Conduct research, Achieve
  next era
• Consumables (used to build) Gold, wood, iron
• Research achievements Determines what can be
  built
• Actions selected
• Generators
• Units
• Diplomacy
• Adversaries State for each (partially observed)

11
Wargus A Simple Decision Space Analysis
Variables

• W Number of workers
• A Number of possible actions
• P Average number of workplaces
• T Number of troops
• D Average number of directions a unit can move
• S Choice of troops stance
• B Number of buildings
• R Average number of research objectives
• C Average number of buildable units

Decision complexity per turn (for a simple game
state)

• O( 2W(AP) 2T(DS) B(RC) ) this is a
  simplification
• This example W4 A5 P3 T8 (offscreen)
  D2 S3 B4 R1 C1
• Decision Space 1.4x103
• This becomes large with the number of units
• Motivates the use of domain knowledge (to
  constrain this space)
• Comparison Average decision space per chess move
  is 30

12
Multiplayer Adversarial Games for Transfer
Learning
Relevance

• Transfer learning allows humans to quickly
  perform well
• Significant background knowledge (e.g., score,
  states, processes, objects, tasks, actions)
  provides opportunities for rapid learning

13
Target Audience Some Specifications
14
TIELT Description

• A free, supported tool for integrating decision
  systems with simulators
• Users define their own APIs (message format and
  content)
• As-is license
• Initial focus Evaluating learning techniques in
  complex games
• Learning foci Task, Player/Opponent, or Game
  Model
• Targeted technique types Broad coverage
• Supervised/unsupervised, immediate/delayed
  feedback, analytic, active/passive,
  online/offline, direct/indirect,
  automated/interactive

15
Functional Architecture
TIELTs User Interface
Evaluation Interface
Prediction Interface
Coordination Interface
Advice Interface
TIELT User
TIELTs Internal Communication Modules
Selected Game Engine
Selected Decision System
Learned Knowledge (inspectable)
Game Player(s)
TIELTs KB Editors
Selected/Developed Knowledge Bases
Game Model
Agent Description
Game Interface Model
Decision System Interface Model
Experiment Methodology
TIELT User
Knowledge Base Libraries
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
16
Knowledge Bases
Game Interface Model
Defines communication processes with the game
engine
Decision System Interface Model
Defines communication processes with the decision
system
Game Model

• Defines interpretation of the game
• e.g., initial state, classes, operators,
  behaviors (rules)
• Behaviors could be used to provide constraints on
  learning

Agent Description
Defines what decision tasks (if any) TIELT must
support
Experiment Methodology
Defines selected performance tasks (taken from
Game Model Description) and the experiment to
conduct
17
UT Example Game Interface Model
Communication
Medium TCP/IP, Port 3000 Message Format ltnamegt
ltattr1gt ltvalue1gt ltattr2gt ltvalue2gt
Action Templates
TURN(Pitch real, Yaw real, Roll
real) SETWALK(Walk boolean) //Start walking or
running RUNTO(Target integer) //ID of object in
world
Sensor Templates
CWP(Weapon integer) //Change Weapon to Weapon
with this Id FLG(Id integer, Reachable boolean,
State Symbol ltheld, dropped, homegt

• Examples interface messages from the GameBots API
• http//www.planetunreal.com/gamebots/docapi.html

18
UT Example Game Model
State Description
Operators
Players Array of Player Self Player Score
Integer
Shoot(Player) Preconditions Player.isVisible
Effects Player.Health - rand(10) MoveTo(Location
) Preconditions Location.isReachable()
Effects Self.position Location
Classes
Player Team String Number Integer
Position Location
Rules
GetShotBy(Player) Preconditions
Player.hasLineOfSight(Self) Effects
Self.Health - rand(10) EnemyMovements(Enemy,
Location1, Location2) Preconditions
Location2.isReachableFrom(Location1)
Enemy.position Location1 Effects
Enemy.position Location2
Location x Integer y Integer z
Integer
19
UT Example Decision System Interface Model
Communication
Medium Standard I/O Message Format (ltnamegt
ltvalue1gt ltvalue2gt ltvalue3gt )
Example Template Messages Sent By TIELT
InitializeGameRules(ruleSet Array of
Rule) SendStateUpdates(CurrentState Array of
Object) LoadScenario(SavedGameFilename String)
Template Messages Received By TIELT
GiveAdvice(AdviceMessage String) PerformAction(Op
eratorName String, Parameters Array of
String) AskForValue(AttributeName String)
20
UT Example Agent Description
Think-Act Cycle
Shoot Something
Pick up a Healthpack
Go Somewhere Else
Call Shoot Operator
Ask Decision System Where Do I Go?
Call Pickup Operator
Ask Decision System Where Do I Go?
21
UT Example Experiment Methodology
Initialization
Game Model Unreal Tournament.xml Game Interface
GameBots.xml Decision System MyUTBot.xml Runs
100 Call slowdown(0.5)
Metrics
FragCount Self.kills FragsPerSecond
Self.kills/LengthOfGame AverageHealth
Self.health x
Plot
FragCount vs. Runs AverageHealth vs. of
players FragsPerSecond vs. Outdegree of net nodes
22
Use Controlling a Game Character
TIELTs User Interface
Evaluation Interface
Prediction Interface
Coordination Interface
Advice Interface
TIELT User
TIELTs Internal Communication Modules
Selected Game Engine
Selected Decision System
Learned Knowledge (inspectable)
TIELTs KB Editors
Selected/Developed Knowledge Bases
Game Model
Agent Description
Game Interface Model
Decision System Interface Model
Experiment Methodology
TIELT User
Knowledge Base Libraries
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
23
Use Prediction
TIELTs User Interface
Evaluation Interface
Prediction Interface
Coordination Interface
Advice Interface
TIELT User
TIELTs Internal Communication Modules
Selected Game Engine
Selected Decision System
Learned Knowledge (inspectable)
TIELTs KB Editors
Selected/Developed Knowledge Bases
Game Model
Agent Description
Game Interface Model
Decision System Interface Model
Experiment Methodology
TIELT User
Knowledge Base Libraries
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
24
Use Game Model Revision
TIELTs User Interface
Evaluation Interface
Prediction Interface
Coordination Interface
Advice Interface
TIELT User
TIELTs Internal Communication Modules
Selected Game Engine
Selected Decision System
Learned Knowledge (inspectable)
TIELTs KB Editors
Selected/Developed Knowledge Bases
Game Model
Agent Description
Game Interface Model
Decision System Interface Model
Experiment Methodology
TIELT User
Knowledge Base Libraries
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
25
A Researcher Use Case

• Define/store decision system interface model
• Select game simulator interface
• Select game model
• Select/define performance task(s)
• Define/select expt. methodology
• Run experiments
• Analyze displayed results

Selected/Developed Knowledge Bases
Knowledge Base Libraries
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
26
Performance Tasks
TIELT Decision systems can be applied to
performance tasks that users select from the
simulator
Problem Solving Tasks
Analysis
Synthesis
Decision Support
Classification
Planning
Design
Scheduling
Diagnosis
Structural
Parametric
27
An Example Performance Task
Task description
This involves several challenging learning tasks
Win a real-time strategy game
Subtasks and supporting operations

• Diagnosis Identify (computer and/or human)
  opponent strategies goals
• Classification Opponent recognition
• Recording Actions of opponents and their effects
• This repeatedly involves classification
• Diagnosis Identify goal(s) being solved by these
  effects
• Classification Identify goal(s), if solved, that
  prevents opponent goals
• Planning Select/adapt or create plan to achieve
  goals and win the game
• Classification Select top-level actions to
  achieve goals
• Iteratively identify necessary sub-goals and,
  finally, primitive actions
• Design (parametric) Identify good initial layout
  of controllable assets
• Execute plan
• Recording Collect measures of effectiveness, to
  provide feedback
• Planning If needed, re-plan, based on feedback,
  at Step 2

28
TIELT Status
Features (v0.6a)

• Pure Java implementation
• Message protocols
• Java Objects, Console I/O, TCP/IP, UDP
• Note Now supports external shared memory
• Message content User-configurable
• Instantiated templates tell it how to communicate
  with other modules
• Messages
• Start, Stop, Load Scenario, Set Speed,
• Agent description representations (w/ Lehigh
  University)
• Task-Method-Knowledge (TMK) process models
• Hierarchical Task Networks (HTNs)
• Experimental results can be stored in databases
  supporting JDBC or ODBC standards (e.g., Oracle,
  MySQL, MS Excel)

29
TIELT http//nrlsat.ittid.com
30
Current Development Priorities

• Speed Compile TIELT Script XML to Java
• GUI
• Parse interface code from free text (e.g., emacs
  plugin)
• Simplify deletion of variables
• Graphical depiction of system information flow
• Language
• TSL Higher level data structures (e.g., Vector,
  Hash)
• Game Model Parameterized game events
• State Separate agent state from game state
• Experiment Methodology
• More analytical tools
• Link independent variables with the other models
• Debugging
• Real-time view of the Game Model
• More powerful logging scheme (e.g., access to log
  levels)

31
FY05 Collaboration Projects
User Interfaces
TIELT User
Prediction Interface
Evaluation Interface
Coordination Interface
Advice Interface
U.Minn-D.
USC/ICT
U.Mich.
Game Library
TIELT Test Bed
Decision System Library
ISLE ICARUS Lehigh Case-based
planner U.Michigan SOAR UTA DCA UT Austin
Neuroevolution
ToEE
EE2
Mad Doc
Troika
FreeCiv
NWU
ISLE
KB Editors
Knowledge Base Library
LU, USC
Mich/ISLE
U. Mich.
Many
Game Model
Agent Descriptions
Game Interface Model
Decision System Interface Model
Experiment Methodology
Many
TIELT User
32
TIELT APIs Status
33
Collaborator Lehigh University
PI Héctor Muñoz-Avila

• Integrate TIELT with Stratagus/Wargus (i.e., to
  control an AI player)

Actions (TCP/IP)
STRATAGUS Modified Client
TIELT
Sensors (TCP/IP)
34
First Research Application (w/ Lehigh U.)
Previous work context
Ponsen, M.J.V., Muñoz-Avila, H., Spronck, P.,
Aha, D.W. (IAAI05). Automatically acquiring
adaptive real-time strategy game opponents using
evolutionary learning.
Tactics1
Tactics1
Game Opponent Scripts
Evolved Counter- Strategies
Dynamic Scripting
Genetic alg.
. . .
. . .
Tactics20
Tactics20
Initial Policy (i.e., equal weights)
Opponent-Specific Policy f(Statei)Tacticij
Action Language

• The tactics (action sequences) you can execute
  depends on which units and technology is
  available, which depend on which buildings youve
  constructed.
• Thus, the 20 building states in Wargus are
  manually predefined and correspond to the set of
  buildings a player possesses.

Wargus
35
First Research Application (w/ Lehigh U.)
Our contribution CaT
Aha, D.W., Molineaux, M., Ponsen, M.
(ICCBR05). Learning to win Case-based plan
selection in a real-time strategy game.
Learns to Select Tactics fi(Casesi, featuresq)
tacticij
Acquired Cases (1 set per Building
State) caseijltBuilding statei, featuresj,
tacticij, performancegt
Hypothesis CaT can significantly outperform
(i.e., higher winning percentage and higher
average relative scores) the evolved
counter-strategies
Evolved Counter- Strategies
Evaluation Interface
Matt Marc
Game Opponent Scripts
TIELTs Internal Communication Modules
Editors
Tactics1
Wargus
CaT
. . .
Matt
Tactics20
Knowledge Bases
(Case-based Tactician)
36
First Research Application (w/ Lehigh U.)
Our results (LOOCV)
We relaxed the assumption that the
opponent/adversary is static
37
Collaborator Mad Doc Software (MDS)
PI Ian Lane Davis

• Integrating state-of-the-art real-time strategy
  game engines with TIELT

Option 1 Commercially available Empire Earth II
engine (w/ patch)
Option 2 Custom TIELT-enabled Mad 3d engine
(MOUT focus). (Note EE II was built using the
Mad 3d engine.)

• Challenge problems (for TIELT competitions at
  workshops)

38
Collaborator University of Michigan
PI John Laird

• Simple game integration (Towers of Hanoi) to
  familiarize with TIELT
• SOAR integration

SWIG Wrapper
SML
Game-specific
Game/Sim
Game Interface Model
Kernel SML/ Soar
Agent Description
Actions
Library calls and returns
ClientSML
Slots
Decision System Interface Model
Library calls and returns
Sensors
SML
General

• SML (Soar Markup Language) is an in-house XML
  wrapper around UMs gSKI-Soar interface
• SWIG is a wrapper that provides access to their
  C SML functions (from any prog. language)
• Slots are places to store data that is shared
  between a decision system and TIELT

• SOAR application (e.g., Mad Docs game engine)

39
Collaborator USC/ICT
PI Michael van Lent

• Simple game integration (Tic-Tac-Toe) to
  familiarize with TIELT
• Integration of Full Spectrum Command/Research
  (FSC/R) with TIELT
• Note FSC 1.5 is a PC-based training aid that
  models the command and control of a U.S. Army
  Light Infantry Company in a MOUT environment.
• Design completed
• Initial API has been delivered includes a subset
  of sensors and actions
• General knowledge bases (KBs)
• A general game-play KB for knowledge common to a
  game genre
• 2 game-specific KBs that, with (a), can be used
  to play 2 games in the genre

40
2. TIELT Support for Transfer Learning
Goal
Show that comprehensive knowledge learned from
experience with a complex game can be transferred
to increase initial performance, performance
rate, or asymptotic achievable performance
Cognitive System
Learning Module
Transferring learned knowledge
Learned Knowledge
What can be transferred?

• Experiences
• Plans for achieving goals
• e.g., for situation assessment, adversary intent
  recognition, course-of-action generation and
  analysis, and ultimately to win a game
• User preferences

41
City Control Challenge Problem (MDS)
Real-time strategy game, where the goal is to
gain control over city blocks that vary in
utility, and require different levels of control.
Design due 4/15, implementation due 5/15, in
time for summer workshops.
42
Initial Challenge Problem Components
City Control A Planning Challenge Problem

• State S
• Map M B11,,Bmn A map
  consists of a set of city blocks
• City Block Bij
• features(Bij) f1(),,fl() A block
  has a set of features (e.g., location, utility)
• objects(Bij)?O
• Objects O objectType(), location(), status(),
• Units U player(), type(), actions(), weapon(),
  armor(),
• Resources R type(), amount(),
• Categories Controllable (e.g., petrol,
  electricity), natural (e.g., uranium, water)
• Actions A
• apply(Ui,Aj,S) S?
  Units can apply actions to modify the state
• Tasks T type(), preconditions(), subtasks(),
  subtaskOrdering(), ...
• Categories Strategic, tactical (where primitive
  tasks invoke actions)

• CP M,S0,SG,F,R,E
• M Map
• SG Goal is to achieve a state (i.e., control
  of specified city blocks)
• e.g., (Bij, Gij) Bij?M Gij?G (1im, 1jn)
• Goals can be achieved by tasks or actions
• F Friendly units
  Controllable units
• R Resources Objects
  that can be used in actions
• E Enemy units
  Non-controllable adversary units

43
Example Challenge Problem (Sketch)
City Control A Planning Challenge Problem

• Goal Control of a set of city blocks BG
  currently under control of E
• Starting State
• F all located in on block Bij (of varying
  significance)
• 3 concentrations of Ei
• Notes
• Success may involve controlling some
  strategic/tactical blocks BST ? BG
• Securing blocks nearby BG may assist with the
  goal
• Controlling a block Bij involves leaving F??F on
  Bij
• Several issues involved with force distribution
  decisions

Block structure of the challenge problem map
44
Example CPs for the Transfer Learning Levels
Differing
Train on a turn-based game, test on a real-time
game
Reformulating
Train on one real-time game, test on another
Train when using deception only for unit
locations, and test when deception is also used
for weapon identities
Generalizing
Abstracting
Vary weapons and armor available
Train with only dismounted or only mounted
soldiers, test with them both available
Composing
Restyling
Vary map M
Vary number of units in F and/or E
Extending
Vary non-combatants on M
Restructuring
Change composition of F and/or E
Extrapolating
Change initial locations for units in F and/or E
Parameterization
Memorization
Same S0 and SG
45
Example Transfer Learning Problems
Extending
Wargus 3x1 battle
Wargus 1x1 battle
Reformulating
46
Some Experimental Variables
Dependent variables

• Goal(s) achievement frequency
• Solution quality (where applicable)
• Differences from SG
• Time/moves required
• Losses
• Units
• Resources expended
• Control

Independent variables

• Transfer learning level
• Game(s)
• Initial state
• Amount of applicable domain knowledge

47
Some Issues for Defining CPs

• Judicious domain knowledge can decrease problem
  complexity
• e.g., win Wargus given complete plans, state
  lattice (Ponsen Spronck, GAME-ON04)
• Thus, domain knowledge must be controlled, or
  measured, to compare TL approaches

• Some problem characteristics can apply at all
  transfer learning levels
• e.g., Percent observability of opponent weaponry
  and/or model
• Thus, problem definitions should hold their
  values constant (if feasible) to isolate the
  impact from the independent variables of interest
  (i.e., those that define the TL level)

• Goals can appear at different transfer levels
• e.g., Select building location, with a partial
  (Restyling) vs. the complete map (Memorization)
• Thus, higher transfer levels dont imply that the
  goals must be more ambitious

48
AAAI05 General Game Playing Competition(M.
Genesereth, Stanford U.)
3. Future Plans
Goal Win any game in a pre-defined category

• Initial category Chess-like games
• Games are produced by a game generator
• Input Rules on how to play the game
• Move grammar is used to communicate actions
• Output (desired) A winning playing strategy

Annual AAAI Competition

• WWW games.stanford.edu
• AAAI05 Prize 10K

49
Games
Dimensions One player or n players
Simultaneous or alternating play x Complete or
incomplete information x Competition and
Cooperation Current Examples
Blocks Minichess Buttons and
Lights Endgame Maze Corridor
Tictactoe Roshambo Ticticoe Diplomacy
50
Game Description Language
Game Description Language (GDL) Conceptual
Basis - Relational Nets Logical Encoding
using Fluent Representation Specification
of GDL published Issue Currently complete
information games only Future partial
information games as well
51
GGP-TIELT Integration
Use Case

• Researcher develops a General Game Player
  competitor
• Connects to GGP Game Manager through TIELT
• Using TIELT, designs experiment for playing n
  matches of m different General Games, against
  reference opponents and unknown GGP competitors
• Runs experiment online through GGP Game Manager
• Evaluates experimental results using TIELT
  evaluation methodologies and GGP performance data
  available online

52
TIELT as Intermediary
GGP Reference Player
GGP Game Manager
TIELT

• Parse GGP
• messages
• Store in KBs
• Reform messages
• from KB and
• pass to player
• Relay moves
• back to GGP GM

Runs as normal TIELT interface of player is
transparent to Game Manager.
Currently, a GGP player running at Stanford
in general, any TIELT-capable or GGP player.
53
Upcoming TIELT-Related Events
General Game Playing Competition

• AAAI05 (24-28 July 2005 Pittsburgh)
• Chair Michael Genesereth (Stanford U.)
• We our integrating TIELT with GGP, attracting
  additional competitors

TIELT-Related Workshops

• At IJCAI05 (31 July 2005 Edinburgh)
• Reasoning, Representation, and Learning in Gaming
  
• Co-chairs Héctor Muñoz-Avila, Michael van Lent
• At ICML05 (7 August 2005 Bonn)
• Knowledge-Intensive Learning in Simulated Task
  Environments
• Co-chair Dan Shapiro
• At ICCBR05 (24 August 2005 Chicago)
• Computer Gaming and Simulation Environments
• Co-chair David C. Wilson

54
TIELT Support for Transfer Learning
Project Description

• Key Ideas TIELT enables the evaluation of
  transfer learning techniques on complex simulated
  gaming tasks.
• Facilitating Breakthroughs
• Configurable Interfaces For both simulators
  cognitive systems, these simplify their
  integration with TIELT.
• Interface libraries Interfaces for
  previously-integrated simulators simplify access
  to challenge problem tasks, while interfaces for
  previously-integrated cognitive systems permit
  immediate comparison with competing baseline
  approaches.
• Transfer opportunities Complex games provide
  many opportunities for transfer learning at all
  levels.

TIELT facilitates evaluations of methods that
transfer declarative and procedural knowledge
among gaming tasks.
Payoff
Technical Details
Goal Encourage researchers to evaluate their
transfer learning techniques on challenge
problems concerning tasks defined in complex
simulators (e.g., games).
Challenge Problems TIELT will supply a highly
flexible environment for defining challenge
problem tasks on transfer learning, linking to
simulators that can execute those tasks, and
testing approaches on those tasks.

• TIELT Features
• Communication Supports several key message
  protocols (TCP/IP, Java objects, console I/O,
  UDP), and message content is user-configurable.
• Game representation Multiple formats (e.g., TMK,
  HTN) are supported for declarative procedural
  game knowledge.
• Task selection Permits cognitive system to focus
  on user-selected subtasks in a task network, and
  creation of agents to solve the remaining tasks.
• Testing User-defined evaluation methodology.
• Java implementation Works on many software
  platforms.

Competition Vision TIELT, challenge problem
definitions, and interfaces with required
simulators and competing cognitive systems will
be made available through the TIELT www site.
Documentation (e.g., tutorials) and on-line
support for using TIELT will address all user
questions. Releases of training and testing
problems will be done according to the PMs
schedule. TIELT will be used to evaluate each
approach.
55
Backups

56
TIELTs Internal Communication Modules
Database
Evaluation Interface
Advice Interface
Database Engine
State
Evaluator
Controller
Stored State
Current State
Translated Model (Subset)
Decision Communicator
Model Updater
Selected Decision System
Learning Task
Selected Game Engine
Percepts
Action / Control Translator (Mapper)
Decision System Outputs
Actions
Perf. Task
Game Model
Game Interface Model
Agent Description
Decision System Interface Model
Experiment Methodology
User
Game Interface Model Editor
Decision System Interface Model Editor
Game Model Editor
Agent Descr. Editor
Expt. Method. Editor
57
Sensing the Game State (City placement example,
inspired by Alpha Centauri, etc.)
1
In Game Engine, the game begins a colony pod is
created and placed.
TIELT
Current State
2
The Game Engine sends a See sensor message
identifying the pods location.
4
Action Translator
Actions
3
Updates
Game Engine
1
The Model Updater receives the sensor message and
finds the corresponding message template in the
Game Interface Model.
Sensors
3
5
2
Model Updater
Controller
3
4
Game Model
4
Game Interface Model
This message template provides updates
(instructions) to the Current State, telling it
that there is a pod at the location See describes.
User

Game Interface Model Editor
Game Model Editor
5
The Model Updater notifies the Controller that
the See action event has occurred.
58
Fetching Decisions from the Decision System (City
placement example)
1
The Controller notifies the Decision Communicator
that it has received a See message.
TIELT
Controller
2
The Decision Communicator finds a city location
task, which is triggered by the See message. It
queries the controller for the learning mode,
then creates a TestInput message to send to the
decision system with information on the pods
location and the map from the Current State.
Selected Decision System
1
Decision System Outputs
Action Translator
4
Translated Model (Subset)
Learning Module 1
Decision Communicator
Current State
3
. . .
2
2
Learning Module n
Agent Description
Decision System Interface Model
3
The Decision Communicator transmits the TestInput
message to the Decision System.
User
4

The Decision System transmits its output(s) to
the Action Translator.
Decision System Interface Model Editor
Agent Desc. Editor
59
Acting in the Game World (City placement example)
1
The Action Translator receives a TestOutput
message from the Decision System.
4.b, c
The Advice Interface receives Move and displays
advice to a human player on what to do next, or
makes a Prediction.
2
The Action Translator finds the TestOutput
message template, determines it is associated
with the city location task, and builds a MovePod
operator (defined by the Current State) with the
parameters of TestOutput.
TIELT

Advice Interface
Prediction Interface
4.b
4.c
Current State
1
2
Actions
Action Translator
4.a
Game Engine
3
3
The Action Translator determines that the Move
Action from the Game Interface Model is triggered
by the MovePod Operator and binds Move using
information from MovePod.
2
3
Game Interface Model
Decision System Interface Model
User
Game Interface Model Editor
Decision System Interface Model Editor
4.a
The Game Engine receives Move and updates the
game to move the pod toward its destination, or
60
Promising Learning Techniques
61
We solicit and fix bugs via this site