MSc Thesis: Building Reactive Agents for Dynamic Environments

1
MSc ThesisBuilding Reactive Agents for Dynamic
Environments Heuristic Navigation in Maze-Based
Games

• Student Seminars
• Date 2nd March 2005
• Peter Blackburn
• Barry O Sullivan
• Funded by Enterprise Ireland under their Basic
  ResearchGrant Scheme (Grant Number SC/02/289)

2
Publications

• Building Reactive Characters for Dynamic Gaming
  Environments, IEEE Symposium on Computational
  Intelligence and Games (CIG2005), April 4th-6th
  2005. Peter Blackburn and Barry O Sullivan

3
Motivation

• Why use computer video games for A.I.
  investigation?
• Killer A.I. Application (Laird 2001)
• Complex
• Contains elements of uncertainty
• Dynamic Environments

4
Aim

• To make NPCs more interesting and competitive.
  (Nareyek 2004)
• To develop a tool which aids the development of
  complex NPC
• To shorten development times of these NPC with a
  more structured and maintainable approach

5
Aim

• To enhance an NPC agents reasoning
• DT provide reactive behaviour
• Do not allow for learning new information
• To demonstrate that cheating is no longer
  required in games to make them enjoyable
• Cheating in relation to pathfinding (complete
  knowledge of terrain)
• Cheating in relation to player sensitive
  information (access to player stats)

6
3 Phase Approach

• Acquire
• Capture a users behaviour/strategy
• Analyse
• Analyse the user provided data and build a
  structure which is compact but represents the
  choices which the user made during training
• Embed
• Embed this information in a NPC

7
Phase 1 Capturing the User's Behaviour\Strategy

• How do we tackle this problem?
• 2 Possible Approaches
• Online (Training\Showing)
• (Geisler 2004, Sammut 1992)
• Offline (Telling)

8
Phase 1 Capturing the User's Behaviour\Strategy

• Scenario 1 of 10.
• In This Scenario what would you do, do you
  think?
• These are your stats and your possible choice of
  behaviour
• Health Low
• Armour Low
• Weapon CloseLong
• Ammo Medium
• Enemy Present True
• Enemy Sees you
• Your Weapon is Stronger False
• Your options are as follows
• Please enter the value of your choice
• 0 Head on Attack
• 1 Flank Attack
• 2 Retreat
• 3 Seek Health

9
Phase 2 Analysing the User

• How do we analyse this information?
• What is a Classifier?
• What is a Decision Tree classifier?
• Benefits of using this type data structure
• Easily understood by non- technical personnel
  (i.e. some game designers)
• Quick and easy to convert to code (if-else-then
  like code)
• Easily adapted\extended after the code has been
  built and deployed

10
Phase 3 Plug and Play!

• Now we have a human strategy how do we use
  their knowledge in the NPC?
• What are FSMs (Finite State Machines)?
• How do FSMs and our technique work together?

11
Evaluation

• Optimal agent is created using a 50 questionnaire
• Notice that the agent finishes on average in the
  top 3

12
Performance using 50 Questions

• DT based NPC vs Novice skilled Unreal Tournament
  NPC

• DT based NPC vs Average skilled Unreal Tournament
  NPC

13
Maze Navigation

• How do current A.I. programmer do it?
• Way points
• Variations of A Algorithm for path finding and
  other tricks
• Why this is incorrect reasoning?
• Complete knowledge
• Cheating

14
Our Approach The Maze
15
Agents are Goal Oriented

• What is Goal oriented or goal aware?
• Why do we want it?
• DT classified actions but didnt understand
  consequence of these actions
• What sort of goals do we set?
• Time
• Hostages
• Ammo
• Health

16
Agent Remembers

• What do our agents remember?
• Nodes which they have seen
• Items which they have seen
• How to reach these nodes in the shortest length
  of time
• Why is this important?
• If a player plays a level do they not learn the
  level layout and how to navigate it? Why shouldnt
  a NPC?

17
Navigational Algorithm

• Algorithm
• Determine Initial state and Goal state
• Determine goals which must be satisfied
• Check if enough time remains for further
  investigation
• Scan area for neighbouring nodes and place in
  memory
• Processes nodes based on some strategy
• Repeat from step 3 until time becomes a constraint

18
Search Strategies

• Optimal
• Always choses the best node (based on complete
  knowledge)
• Min
• Try to chose the node which will improve on some
  goal and if none exist, chose the node which does
  least worst
• Max
• Try to chose the node which improves on the most
  number of goals
• Random
• Choose any node randomly
• Pareto Optimal
• An outcome of a node move is P.O. if there is no
  other outcome that makes every goal at least as
  well off and at least one goal strictly better off

19
Evaluation

• Currently experimenting with randomly generated
  graphs
• consisting of 100 nodes
• Densities 0.25, 0.50 and 0.75
• Randomly scattered entities
• Varying goals
• Varying Strategies

20
How can we use this to enhance our DT Approach

• Hierarchical Planning
• DT determines action i.e. find health
• HS determines how to achieve this goal.
• Method allows agent to learn
• Memory allows learning in a non-cheating manner
• Reason more about its environment based on
  observation

21
QA