Learning

1
Learning User Modeling

• Presentation By Brigette Swan

2
What Is Good AI?

• Display of realistic behavior when reacting to
  the player.

• Able to challenge

the player both tactically and strategically.
3
But What About

• Adaptation to the quirks and habits of a
  particular player over time.
• At the moment, many games only implement
  difficulty sliders.

4
Standards?

• This is a coarse level of control at best. It
  forces the player to estimate his or her own
  abilities, which can often be incorrect.
• What happens if one mode is too easy, but the
  next is too hard?

5
(No Transcript)
6
Why?

• Learning algorithms are computationally
  expensive, enough to make game programmers run
  screaming.
• Is there a method that is not so daunting, and
  yet still gives a realistic feel to
  player-computer interaction?
• Enter.

7
The Player Model

• Something like a profile, the player model is a
  set of demographics on a particular individuals
  skills, preferences, weaknesses, and other
  traits.
• The model can be easily updated as the game
  progresses, whenever the AI interacts with the
  player.

8
The Player Model

• The game AI can then query the user model,
  allowing it to select reactions and tactics that
  would best challenge the players personal style.
• The profile could be for a single play, or even
  be updated over multiple sessions.

9
Model Design

• The model is a collection of numerical
  attributes, representing the traits of the
  player.
• Each attribute is an aspect of player behavior,
  which can be associated with strategies,
  maneuvers, or skills.

10
Model Example
11
Model Example

• This particular trait does not take into account
  the players behavior relative to the game
  context.
• Models can hold detail like this, but the more
  detailed the information, the longer the profile
  takes to design, and the more computationally
  expensive the updates become.

12
Model Implementation

• At the most basic level, the player model is a
  statistical record of the frequency of some
  subset of player actions.

• Class PlayerModel
• public
• enum ETrait
• kUsesSmokeGrenades,
• kAlwaysRuns,
• kCanDoTrickyJumps
• void Initialize()
• void UpdateTrait(ETrait trait, float observe
  value)
• float GetTrait(ETrait trait)
• private
• vectorltfloatgt _traitValues

13
Model Implementation

• Each trait is represented by a floating-point
  value between 0 and 1, where 0 roughly means the
  player never does this and 1 means the player
  always does this.
• Each trait is initialized to 0.5 to reflect the
  lack of knowledge.

14
Model Implementation

• The update method is based on the least mean
  squares (LMS) training rule, often used in
  machine learning.

• float PlayerModelUpdateTrait(ETrait trait,
  float observeValue)
• float currValue _traitValuestrait
• float delta observeValue currValue
• float weightedDelta kLEARNINGRATE delta
• traitValuestrait weightedDelta

traitValue a observedValue (1- a)
traitValue
15

• UsesSmokeGrenades
• FleesFromBattle
• UsesTrickyJumps
• UsesRangedAttacks
• UsesCloseAttacks

0.5 0.5 0.5 0.5 0.5
0.5 0.5 0.5 0.55 0.45
0.5 0.5 0.5 0.62 0.38
0.54 0.56 0.5 0.62 0.38

• New Game
• User Model Set To Default

Ninja Vanish!
16

• UsesSmokeGrenades
• FleesFromBattle
• UsesTrickyJumps
• UsesRangedAttacks
• UsesCloseAttacks

1.0 0.56 0.3 1.0 0.25


17
Model Updates

• In order for the user model to be effective, the
  game must be able to update the profile. This
  requires two steps.
• The game must detect when an update should be
  made.
• The game must then tell the model to update
  itself.

18
Model Updates

• While the latter is easy, with the function just
  displayed, the former can prove a more daunting
  task.
• The game must be able to recognize that the
  player has taken, or failed to take, an action,
  or sequence of actions, which correspond to a
  certain trait.

19
Model Updates

• For example, the CanDoTrickyJumps trait, might
  be triggered if the player can navigate across
  rooftops. Thus, the game must have a mechanism
  to determine that the player had made, or failed,
  such a jump.
• This detection can be hardcoded into the jump
  routines.

20
Model Updates

• What about a trait like DetectsHidingEnemies?
• In a game where enemies hide in shadows and
  around corners, the game must compute the
  visibility of every nearby enemy. This also must
  include the processing of scene geometry.

21
Model Updates

• The game might be able to make use of the AI as
  it does all these calculations for interaction,
  or the keep a cache of past computations to help
  reduce the cost of the algorithm.
• A queue for player actions allows background
  processing as a solution.

22
Model Updates

• Once the game has detected a need for an update,
  a value is assigned to the action and placed into
  the profile.
• For example, a player that successfully makes the
  difficult jump, as previously described, would
  receive an update like this.
• model.UpdateTrait(PlayerModelkCanDoTrickyJumps,1
  .0)

23
Alternatives?

• While hard-coded trait detection routines are
  straightforward, it leaves the modeling code
  scattered throughout the source, which would make
  maintenance a headache and a half.
• Also, modifying the structure of the player model
  in any way requires changing the code, and
  therefore, rebuilding the game.

24
FSM Updates

• Trait detection routines can also be implemented
  as a collection of finite state machines. Each
  concrete trait is associated with a single FSM
  which contains the entire detection algorithm.
• The FSMs monitor the stream of AI and player
  actions, and update the model whenever necessary.

25
FSM Updates
26
FSM Updates

• The FSM system is designed to be data-driven, and
  thus, can be separated from the source code and
  loaded into the game dynamically.
• Removing the player model from the source code
  removes it from the build process, allowing it to
  be altered without a time-consuming recompilation.

27
Hierarchy

• User models can also be designed in a hierarchy,
  using a tree. Each leaf node represents one of
  the concrete traits that we have been discussing.
• These nodes are clustered under particular
  abstract traits, which allows the AI to
  generalize when selecting a strategy against a
  player.

28
(No Transcript)
29
Player OnePrimary Abstract Trait MeleeAttacker
Enemy AI
?
Player TwoPrimary Abstract Trait
RangedAttacker Secondary Abstract Trait Healer
30
Hierarchy

• The hierarchical model is slightly more complex
  to implement, but it not difficult, resembling a
  binary tree in execution.
• This design also makes it simple to maintain a
  hashtable to the various leaf traits, speeding up
  the search for the GetTrait() process.

31
Using The Model

• The player model exists to make the AI less
  episodic and more aware of past interactions.
• From the players perspective, it appears as
  though his enemies have gotten smarter.

32
Using The Model

• The profile can also be used by the AI to exploit
  weaknesses in a player, to give them more of a
  challenge.
• In a friendly game, such demographics could be
  used to provide helpful advice to a novice player.

33
In Games Today

• Rare are the games that implement user models
  well, when they do at all. At the moment, it is
  difficult to say when player modeling is being
  used, and when it is not.
• But we have a few guesses.

34
In Games Today

• Sports Recording play styles of users is
  important in sports style games. The NFL 2k
  series actually experimented with player models,
  but according to most players, they were
  implemented rather poorly.

35
In Games Today

• FPS Adaptive enemies would be an asset to games
  like Splinter Cell Pandora Tomorrow. Developers
  and designers always seem to have grand plans,
  but reviews indicate mediocre execution at best.

36
In Games Today

• RTS While Homeworld does not implement the exact
  type of player model described here, it does
  compensate based on the users skills.

Since the player keeps his units from the
previous encounter, the enemy force is based on
the number and type of units that the player has
at the beginning of the next mission.
37
In Games Today

• Fighters This game style does some minor
  profiling of its players. In tournaments, the AI
  creates a single session model, allowing the
  opponents to learn and counter the players
  favored movement patterns. The difficulty slider
  is used to determine the learning rate of the AI.

38
(No Transcript)