Tracking DragonHunters with Language Models

1
Tracking Dragon-Hunters with Language Models
Anton Leuski and Victor Lavrenko
2
Outline

• Problem
• Establishing dependencies between words and
  actions
• Domain
• Virtual Worlds or MMORPGs
• Experiments
• 3 questions what, who, and where

3
Example

• Paris, France. October 27, 2005
• 20 days, 274 towns, 8,973 cars destroyed, 200 Mil
  Euro in damages
• SMS messages and Blogs as tools to organize

Nice work people. The cops are petrified of us,
everything must burn, starting Monday, the
operation Midnight Sun starts, tell everyone
else, rendezvous for Momo and Abdul in Zone 4 ...
jihad Islamia Allah Akhbar.
4
Example

• Given the text of all SMS and blogs, can we...
• detect the riots are starting?
• where are the riots?
• how many people are involved?

5
The Big Picture
Link relevant text with relevant activities
6
Approach

• Linking an activity and relevant text
• Bi-directional
• given activity find relevant text
• given text find the relevant activity
• Too much information
• Long term goal mathematical model that describes
  the inter-dependencies

7
Other Domains

• Education student/tutor chat vs. test scores
• Financial news vs. market response
• Military com net chat vs. battlefield actions
• UI text viewed/typed vs. application activity
• Public opinions articles vs. approval ratings

8
Whats been done?
9
Data?

• Problem Difficult to get large amounts of
  informative data from real world scenarios
• Possible solution Virtual Worlds

10
Outline

• Problem
• Establishing dependencies between words and
  actions
• Domain
• Virtual Worlds or MMORPGs
• Experiments
• 3 questions what, who, and where

11
Virtual Worlds

• MMORPG Massive Multiplayer Online Role-Playing
  Game
• 3D graphics, avatars, persistent world
• Activities chatting, teaming, completing tasks
  and quests, trading, killings, etc...

12
Virtual Worlds

• 35-40 million subscribers world-wide
• 40 spend gt 20 hours/week
• average age 30
• Virtual Gross National Product in thousands
  /player (2,266 in EverQuest, 2001)

13
Virtual Worlds

• It is a serious human activity
• making sense of the things in VWs, monitoring,
  making predictions is becoming very important
• It is a good model of real-world activities
• techniques developed in VWs can be transferred
  back to real world

14
Blade Mistress

• Small, low-bandwidth, fantasy-oriented
• 3D world
• We have logs (thanks to Mike van Lent)
  09/2002-09/2003
• .. and no, I have not played it. Ever.

15
(No Transcript)
16
Blade Mistress
17
Chat Log

• 01/12/03, 222202, 64, 56, rradeon says, magic
  helps us out
• 01/12/03, 222206, 64, 56, rradeon says, plz
• 01/12/03, 222207, 78, 30, Burto tells
  everyone, WE need to kill it before he reaches
  the sacred stones !!!!! or there will be Hell to
  pay!!
• 01/12/03, 222208, 64, 56, Agnes says, hey
  magic fair... help us get to spirit?
• 01/12/03, 222208, 78, 29, Gavron says, My
  brothers, Help me!!
• 01/12/03, 222213, 78, 30, Burto tells
  everyone, HE MUST DIE!!!!
• 01/12/03, 222215, 64, 56, MagicFair says, sure
• 01/12/03, 222219, 77, 29, Bloodwarlord
  says, this dude is just like lord of thuindra
  from fingle
• 01/12/03, 222220, 64, 56, Agnes says, great,
  just 1 more then
• 01/12/03, 222225, 78, 29, Gavron says, Please,
  Borthers!!! .....
• 01/12/03, 222226, 79, 29, Shard tells
  everyone, SLAY GAVRON
• 01/12/03, 222247, 64, 56, rradeon says, hmm

18
Chat Logs Spellings

• k
• devilgod u on yet?
• i thgouhgt u guys were lieing
• it qqqqes me off
• just wait, its comming LOL
• wat would have happens if the server restarted
  while ppl were on?
• lol smart azz

19
Game Log

• ----------------------------- Green Dragon
  killed! 1/12, 2320
• ----- attending cool_jaws, -----
• closing connection Prince Zyidus, Prince
  Zyidus 1/12, 2321
• closing connection teela, newfy666 1/12,
  2321
• ----------------------------- Green Dragon
  killed! 1/12, 2321
• ----- attending warphoenix2, -----
• new connection MedievalSlayer, Medieval
  1/12, 2322
• ----------------------------- Green Dragon
  killed! 1/12, 2322
• ----- attending warphoenix2, -----
• ----------------------------- Green Dragon
  killed! 1/12, 2322
• ----- attending cool_jaws, -----
• ----------------------------- Gavron killed!
  1/12, 2322
• ----- attending ThunderBoobs, Bloodwarlord,
  lord_samot, Warrior Arcana, Infused, Nova, -----
• ----------------------------- Green Dragon
  killed! 1/12, 2323

20
Data
21
03/02/03, 163531, 64, 69, xxxdaisyxxx says,
what color were they? 03/02/03, 163535, 25,
122, RedHeadSmith tells the guild, we gonna ding
the mighty ding 03/02/03, 163539, 64, 70, Angry
Amazon! says, the dragons? 03/02/03, 163540,
64, 71, xxxdaisyxxx says, there is one 03/02/03,
163549, 2, 2, Katunka tells the guild, come to
the creepy stalker and youcan have them
all 03/02/03, 163554, 66, 71, Angry Amazon!
says, i saw a yellow one 03/02/03, 163559, 67,
72, xxxdaisyxxx says, eeww 03/02/03, 163605,
67, 72, xxxdaisyxxx says, yellow ones are
hard 03/02/03, 163610, 67, 72, xxxdaisyxxx
says, get ready to attack 03/02/03, 163611, 2,
3, Katunka tells the guild, rod 3.3 03/02/03,
163613, 25, 122, RedHeadSmith tells the guild,
lol red cant even kill the bone warrior 03/02/03,
163615, 65, 57, DATA tells you, lol, Smurf
warrior 03/02/03, 163616, 67, 72, xxxdaisyxxx
says, attack after me 03/02/03, 163617, 67, 72,
Angry Amazon! says, k 03/02/03, 163619, 25,
122, RedHeadSmith tells the guild, hehe 03/02/03,
163620, 67, 72, xxxdaisyxxx says, goi 03/02/03,
163623, 67, 72, xxxdaisyxxx says,
attack 03/02/03, 163627, 65, 57, Joey tells
everyone, selling some eggs and viz
ingots 03/02/03, 163629, 2, 3, Katunka tells
the guild, no i will give you all i
have 03/02/03, 163629, 66, 56, Jane Slayer
tells everyone, buying 1 gbd 66,56 03/02/03,
163633, 65, 57, DATA tells you, so i can hit
him and hit ), Smurf warrior 03/02/03, 163634,
67, 72, xxxdaisyxxx says, im dead 03/02/03,
163646, 25, 122, RedHeadSmith tells the guild,
no i mean to cme down..i need a
jinwiese 03/02/03, 163655, 40, 54, Smurf
warrior tells you, i tried before i had this
sword couldnt hit him at all, DATA 03/02/03,
163655, 68, 72, Angry Amazon! says,
arg 03/02/03, 163700, 68, 72, Angry Amazon!
tells everyone, arg 03/02/03, 163704, 65, 56,
tagar68 says, \can any one spare some
food 03/02/03, 163707, 65, 57, xxxdaisyxxx
says, soni 03/02/03, 163708, 69, 61, Deathspine
tells you, where r u?, Dragoon67 03/02/03,
163715, 65, 56, Joey tells everyone, SELLING
STUFF 03/02/03, 163720, 25, 122, RedHeadSmith
tells the guild, the boney kills red lol..no
totems no sword strong enough 03/02/03, 163720,
2, 3, Katunka tells the guild, ok kar and i will
come out and then you can send us back 03/02/03,
163725, 65, 56, Kar tells everyone,
lol 03/02/03, 163726, 25, 122, RedHeadSmith
tells the guild, k thanks 03/02/03, 163728, 65,
56, tagar68 tells everyone, can any one spare
some food 03/02/03, 163729, 65, 56, Kar says,
jinwiese 03/02/03, 163730, 65, 56, Joey tells
everyone, who is interested in viz
ingots 03/02/03, 163739, 25, 122, RedHeadSmith
says, lmao 03/02/03, 163749, 65, 56, Kar says,
me 03/02/03, 163751, 67, 72, Angry Amazon!
tells everyone, its still alive 03/02/03,
163752, 70, 52, Deathspine tells you, how
much?, Joey 03/02/03, 163800, 65, 68,
xxxdaisyxxx says, lol 03/02/03, 163800, 65, 56,
Joey says, syre 03/02/03, 163801, 67, 72, Angry
Amazon! tells everyone, should i attack 03/02/03,
163801, 65, 56, Kar tells you, how many do you
have and how much, Joey 03/02/03, 163807, 65,
56, Joey says, uh how do u make tachi
sword 03/02/03, 163817, 67, 71, xxxdaisyxxx
says, attack 03/02/03, 163818, 25, 122,
RedHeadSmith says, ok..this is gonna
rule 03/02/03, 163818, 65, 56, Joey says, i
have 54 viz 03/02/03, 163821, 65, 56, tagar68
says, i dont konw i got it given to me 03/02/03,
163823, 25, 122, RedHeadSmith says,
hhehe 03/02/03, 163825, 65, 56, Dragoon67 says,
tagar weren't u a sister of war 03/02/03,
163830, 67, 72, xxxdaisyxxx says, its dead
22
03/02/03, 163531, 64, 69, xxxdaisyxxx says,
what color were they? 03/02/03, 163535, 25,
122, RedHeadSmith tells the guild, we gonna ding
the mighty ding 03/02/03, 163539, 64, 70, Angry
Amazon! says, the dragons? 03/02/03, 163540,
64, 71, xxxdaisyxxx says, there is one 03/02/03,
163549, 2, 2, Katunka tells the guild, come to
the creepy stalker and youcan have them
all 03/02/03, 163554, 66, 71, Angry Amazon!
says, i saw a yellow one 03/02/03, 163559, 67,
72, xxxdaisyxxx says, eeww 03/02/03, 163605,
67, 72, xxxdaisyxxx says, yellow ones are
hard 03/02/03, 163610, 67, 72, xxxdaisyxxx
says, get ready to attack 03/02/03, 163611, 2,
3, Katunka tells the guild, rod 3.3 03/02/03,
163613, 25, 122, RedHeadSmith tells the guild,
lol red cant even kill the bone warrior 03/02/03,
163615, 65, 57, DATA tells you, lol, Smurf
warrior 03/02/03, 163616, 67, 72, xxxdaisyxxx
says, attack after me 03/02/03, 163617, 67, 72,
Angry Amazon! says, k 03/02/03, 163619, 25,
122, RedHeadSmith tells the guild, hehe 03/02/03,
163620, 67, 72, xxxdaisyxxx says, goi 03/02/03,
163623, 67, 72, xxxdaisyxxx says,
attack 03/02/03, 163627, 65, 57, Joey tells
everyone, selling some eggs and viz
ingots 03/02/03, 163629, 2, 3, Katunka tells
the guild, no i will give you all i
have 03/02/03, 163629, 66, 56, Jane Slayer
tells everyone, buying 1 gbd 66,56 03/02/03,
163633, 65, 57, DATA tells you, so i can hit
him and hit ), Smurf warrior 03/02/03, 163634,
67, 72, xxxdaisyxxx says, im dead 03/02/03,
163646, 25, 122, RedHeadSmith tells the guild,
no i mean to cme down..i need a
jinwiese 03/02/03, 163655, 40, 54, Smurf
warrior tells you, i tried before i had this
sword couldnt hit him at all, DATA 03/02/03,
163655, 68, 72, Angry Amazon! says,
arg 03/02/03, 163700, 68, 72, Angry Amazon!
tells everyone, arg 03/02/03, 163704, 65, 56,
tagar68 says, \can any one spare some
food 03/02/03, 163707, 65, 57, xxxdaisyxxx
says, soni 03/02/03, 163708, 69, 61, Deathspine
tells you, where r u?, Dragoon67 03/02/03,
163715, 65, 56, Joey tells everyone, SELLING
STUFF 03/02/03, 163720, 25, 122, RedHeadSmith
tells the guild, the boney kills red lol..no
totems no sword strong enough 03/02/03, 163720,
2, 3, Katunka tells the guild, ok kar and i will
come out and then you can send us back 03/02/03,
163725, 65, 56, Kar tells everyone,
lol 03/02/03, 163726, 25, 122, RedHeadSmith
tells the guild, k thanks 03/02/03, 163728, 65,
56, tagar68 tells everyone, can any one spare
some food 03/02/03, 163729, 65, 56, Kar says,
jinwiese 03/02/03, 163730, 65, 56, Joey tells
everyone, who is interested in viz
ingots 03/02/03, 163739, 25, 122, RedHeadSmith
says, lmao 03/02/03, 163749, 65, 56, Kar says,
me 03/02/03, 163751, 67, 72, Angry Amazon!
tells everyone, its still alive 03/02/03,
163752, 70, 52, Deathspine tells you, how
much?, Joey 03/02/03, 163800, 65, 68,
xxxdaisyxxx says, lol 03/02/03, 163800, 65, 56,
Joey says, syre 03/02/03, 163801, 67, 72, Angry
Amazon! tells everyone, should i attack 03/02/03,
163801, 65, 56, Kar tells you, how many do you
have and how much, Joey 03/02/03, 163807, 65,
56, Joey says, uh how do u make tachi
sword 03/02/03, 163817, 67, 71, xxxdaisyxxx
says, attack 03/02/03, 163818, 25, 122,
RedHeadSmith says, ok..this is gonna
rule 03/02/03, 163818, 65, 56, Joey says, i
have 54 viz 03/02/03, 163821, 65, 56, tagar68
says, i dont konw i got it given to me 03/02/03,
163823, 25, 122, RedHeadSmith says,
hhehe 03/02/03, 163825, 65, 56, Dragoon67 says,
tagar weren't u a sister of war 03/02/03,
163830, 67, 72, xxxdaisyxxx says, its dead
new connection Fierdon, Evelyn 3/02,
1636 ----------------------------- Dragon
Prelate killed! 3/02, 1636 ----- attending
Smurf warrior, ----- -----------------------------
Green Dragon killed! 3/02, 1636 -----
attending ELRICK, ----- new connection
dragoon67, Dragoon67 3/02, 1636 new
connection Arcanis, Ungarmax 3/02,
1637 ----------------------------- Dragon
Prelate killed! 3/02, 1637 ----- attending
Smurf warrior, ----- closing connection
Fierdon, Evelyn 3/02, 1637 ----------------------
------- Dragon Prelate killed! 3/02, 1637 -----
attending Smurf warrior, ----- ------------------
----------- Dragon Prelate killed! 3/02,
1638 ----- attending Smurf warrior,
----- ----------------------------- Green Dragon
killed! 3/02, 1638 ----- attending Soni,
xxxdaisyxxx, Angry Amazon!, ----- new
connection Lostone, Lostone 3/02,
1638 ----------------------------- Dragon
Prelate killed! 3/02, 1638 ----- attending
Smurf warrior, ----- -----------------------------
Dragon Prelate killed! 3/02, 1639 -----
attending Smurf warrior, ----- closing
connection ShadowBlades, ShadowBlades 3/02,
1639 closing connection frznh3ll, tagar68
3/02, 1639 ----------------------------- Dragon
Prelate killed! 3/02, 1639 ----- attending
Smurf warrior, ----- new connection
maxkill, Kinga 3/02, 1639 -----------------------
------ Green Dragon killed! 3/02, 1640 -----
attending ELRICK, ----- -------------------------
---- Dokk's Centurion killed! 3/02, 1640 -----
attending ripper, ----- new connection
eyut2, Polrbear 3/02, 1640 ----------------------
------- Dragon Prelate killed! 3/02, 1640 -----
attending Smurf warrior, ----- ------------------
----------- Dragon Prelate killed! 3/02, 1640
23
Handout

• January 12, 2003, 222300
• 6 players killed Gavron
• Log of 25 minutes surrounding the event
• Colors indicate lines spoken by the killers

24
Outline

• Problem
• Establishing dependencies between words and
  actions
• Domain
• Virtual Worlds or MMORPGs
• Experiments
• 3 questions what, who, and where

25
Questions

• Activity detection is there a kill taking place?
• Player forensics who is involved?
• World mapping where they talk about this?

26
Language Models
Y
Segmenting space and time
Y
Time
X
27
Questions

• Activity detection is there a kill taking place?
• Player forensics who is involved?
• World mapping where they talk about this?

28
Questions

• Activity detection is there a kill taking place?
• Segment messages by time and space
• Classify segments into segments w/ or w/o kill
• Player forensics who is involved?
• Segment messages by time and player
• Classify segments into segments w/ or w/o kill
• World mapping where they talk about this?
• Segment messages by space
• Group topically similar segments together
• Plot them against the location

29
Language Models
Counting words and computing probabilities

• Probability estimation
• probability of a word occurring in a t-s segment
  with kill
• Count of a word W in box G
• Decision criteria

Jelinek-Mercer smoothing
30
Evaluation

• True positive
• False positive
• ROC curve

31
Activity Detection
Whats happened?

• Segment messages by speaker location and time
• 6x6 squares
• 10 minutes

32
Activity Detection
Time
Space
Accuracy of activity detection for different
parameters. (area under ROC curve)
33
Player Forensics
Who did it?

• Segment messages by speaker and time
• 10 minutes

34
Player Forensics
Accuracy of participants detection for different
time spans.
35
World Mapping

• Combine all messages from a 2x2 region
• Distance (KL-div)
• Cluster these texts
• Draw the clusters on the map

36
The World Map
37
(No Transcript)
38
(No Transcript)
39
Future Work

• Other activities
• Long-distance transporting
• Trades
• Other tasks
• Story detection and extraction (annotations?)
• Player qualities (skill, friendliness, type, etc)
• Other formal models
• Get rid of sharp segments
• Message frequency
• Movement patterns

40
FW Activities

• Long-distance transporting
• Trades
• Need to do activity extraction/detection

41
FW Tasks

• Story detection/extraction
• needs annotations
• Player qualities
• Estimating from initial conversation
• Player experience
• Player type, e.g.,achievers, explorers,
  socializers, and griefers

42
FW Formal Models

• Problem sharp, rectangular segments
• Other features
• player movement patterns
• message stream patterns, e.g., frequency and
  periodicity

43
Thank you!