Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution

1
Discovery, Analysis and Monitoring of Hidden
Social Networks and their Evolution

• Malik Magdon-Ismail
• Rensselaer Polytechnic Institute

2
Our Group

• Students
• Mykola Hayvanovich
• Apirak Hoonlor
• Stephen Kelley
• Konstantin Mertsalov

• M. Goldberg
• M-I
• B. Szymanski
• A. Wallace

3
Motivation
Analysis of the patterns can reveal the groups as
well as their internal group structure.
Communications supporting IED planning have
patterns and are correlated.
4
Communications
Time January 12, 2005, 0935 From
joe_at_xyz.com To sue_at_abc.com Subject
Hello Message Where have you been?
160631 ltFreeTradegt Republicans were the worst
pacifists before ww1 and ww2 160643
ltSweetLeafgt France Fries 160650 ltFreeTradegt
As a generality, of course their were Republican
Hawks. 160713 ltFreeTradegt Sweet, good pun but
bad story! 160718 ltSweetLeafgt yup 160723
ltLupinegt anyways, he's perpetually tormented by
presidential actions 160725 ltSweetLeafgt it
aint good for no one 160747 ltSweetLeafgt I
think they knew it was commiing 160751
ltFreeTradegt Rossevelt met monthly in New York
with mostly trusted Republicans to talk about how
to get america into the war. 160810
ltFreeTradegt and he spent 2 year with Churchill
meeting him sometimes secretly in the ocean to
discuss the same topic. 160822 ltFreeTradegt
Exchanging a lot of letters. 160825
ltFreeTradegt telegrams 160828 ltLupinegt There
really is nothing like a shorn scrotum. It's
breathtaking, I suggest you try it. 160855
ltFreeTradegt Well they didnt literally meet in the
ocean, they were on ships.
5
Streaming Example

• Time From To Message
• 1000 Alice Charlie Golf tomorrow? Tell
  everyone.
• 1005 Charlie Felix Alice mentioned
  golf tomorrow.
• 1006 Alice Bob Hey, golf
  tomorrow. Spread the word.
• 1012 Alice Bob Tee off 8am at
  Pinehurst.
• 1013 Felix Grace Hey guys, golf
  tomorrow.
• 1013 Felix Harry Hey guys, golf
  tomorrow.
• 1015 Alice Charlie Pinehurst Tee
  time 8am.
• 1020 Bob Elizabeth Were playing golf
  tomorrow.
• 1020 Bob Dave Were playing
  golf tomorrow.
• 1022 Charlie Felix Tee time 8am at
  Pinehurst
• 1025 Bob Elizabeth We tee off 8am at
  Pinehurst.
• 1025 Bob Dave We tee off 8am at
  Pinehurst.
• 1031 Felix Grace Tee time 8am,
  Pinehurst.
• 1031 Felix Harry Tee time 8am,
  Pinehurst.

6
Streaming Example

• Time From To Message
• 1000 Alice Charlie Golf tomorrow? Tell
  everyone.
• 1005 Charlie Felix Alice mentioned
  golf tomorrow.
• 1006 Alice Bob Hey, golf
  tomorrow. Spread the word.
• 1012 Alice Bob Tee off 8am at
  Pinehurst.
• 1013 Felix Grace Hey guys, golf
  tomorrow.
• 1013 Felix Harry Hey guys, golf
  tomorrow.
• 1015 Alice Charlie Pinehurst Tee
  time 8am.
• 1020 Bob Elizabeth Were playing golf
  tomorrow.
• 1020 Bob Dave Were playing
  golf tomorrow.
• 1022 Charlie Felix Tee time 8am at
  Pinehurst
• 1025 Bob Elizabeth We tee off 8am at
  Pinehurst.
• 1025 Bob Dave We tee off 8am at
  Pinehurst.
• 1031 Felix Grace Tee time 8am,
  Pinehurst.
• 1031 Felix Harry Tee time 8am,
  Pinehurst.

7
Streaming Example

• Time From To
• 1000 Alice Charlie
• 1005 Charlie Felix
• 1006 Alice Bob
• 1012 Alice Bob
• 1013 Felix Grace
• 1013 Felix Harry
• 1015 Alice Charlie
• 1020 Bob Elizabeth
• 1020 Bob Dave
• 1022 Charlie Felix
• 1025 Bob Elizabeth
• 1025 Bob Dave
• 1031 Felix Grace
• 1031 Felix Harry

8
Overview SIGHTS RDM
Level 2
3
,
s
e
l
l
.
.
.
3
,
h
e
l
l
Pattern id 3Pattern 2tradebb
Level 1
2
t
r
a
d
e
.
.
.
2
t
r
a
d
e
Pattern id 2Pattern buy,
Level 0
b
u
y
,
t
r
a
d
e
.
.
.
b
u
y
Higher ranked leaders
Group leader
Subgroup leaders
Members
9
Communications

• Email, Telephone, Newsgroup, Weblog, Chatrooms,

Time January 12, 2005, 0935 From
joe_at_xyz.com To sue_at_abc.com Subject
Hello Message Where have you been lately?
Time January 12, 2005, 0935 From
joe_at_xyz.com To sue_at_abc.com Subject
Hello Message Where have you been lately?
10
Communication Graph
January 12, 2005, 0935
sue_at_abc.com
joe_at_xyz.com
11
Communication Graph
What are the social groups/coalitions?
12
Social Groups are Clusters

• Clusters may overlap.

13
Social Groups are Clusters

• Clusters may overlap.
• A cluster is a locally defined object.

14
Social Groups are Clusters

• Clusters may overlap.
• A cluster is a locally defined object.
• Group members are more introverted than
  extroverted.

YES
NO
15
Social Groups are Clusters

• Clusters may overlap.
• A cluster is a locally defined object.
• Group members are more introverted than
  extroverted.
• Social groups (clusters) persist

16
SIGHTS

• Statistical Identification of Groups Hidden in
  Time and Space
• - System for statistical analysis of social
  coalitions in communication networks

• Coalition Discovery
• Overlapping Clustering
• Streaming groups
• Persistent groups.

• Coalition Analysis
• Leaders
• Opposing groups
• Topic matching

• Visualizations
• Size-Density plots
• Static coalitions
• Dynamic coalitions

• Data Sources
• Blogs
• Emails (Enron)
• Chatroom
• Synthetic data

Different analyses on dataset
Visualization options
Size vs. Density Plot
Group members
Leader index
Groups matching analyst topic in red
Choose time window
17
Examples
ENRON
Ali Baba Data Set (DoD)
Two clusters Electric circuit design
Optimization of Neural Networks Intersection
Sensitivity analysis in degenerate quadratic
programming

• GROUND TRUTH
• Group A
• Dog
• Vulture
• Camel
• Yassir Hussein
• Bird
• (6 others)
• Group B
• Ahmet
• Saleh Sarwuk
• Shaid
• Pavlammed Pavlah
• Osan Domenik

• SIGHTS
• Group A
• Dog
• Vulture
• Camel
• Gopher
• Group B
• Ahmet
• Saleh Sarwuk
• Shaid
• Dajik

Citeseer
18
Recursive Data Mining (RDM)

• Build a classifier to identify the relationship
  between sender and receiver of a message
• EXAMPLE
• Do you have time to meet some time this week?
• Lets meet 2pm today, ok?
• Which is advisor, which is student?

19
Pattern Definition

• Hierarchical Pattern Construction (recursive
  definition)
• Captures patterns patterns of patterns patterns
  of patterns of patterns (can even capture
  long-range patterns)

Pattern id 4Pattern 3,_ell
Level 2
Pattern id 3Pattern 2trade
Larger patterns
Level 1
Pattern id 2Pattern buy,
Level 0
20
A Classifier Joining the Pieces

• Ensemble of classifiers
• Classifier for each level in the hierarchical
  approach
• Features gathered from the training messages
• Global features include average length and number
  of sentences
• Approximate matching allows treatment of noise

21
Results on Enron
Binary classification for a given message m, is
m sent by a person with role r? r ? CEO,
Manager, Trader, Vice-President
Multi-classification for a given message
m, which role r is the most likely for the
sender? r ? CEO, Manager, Trader,
Vice-President
The bars show the error of classification.
Universally RDM_SVM outperforms other classifiers
22
Summing Up

• SIGHTS
• Structural non-semantic language independent
• Finds groups, their dynamics and structure
  visual analytic capabilities.
• RDM
• Uses statistical semantics language independent
• Identifies roles within the group

23
Thank You

• http//www.cs.rpi.edu/magdon