Automating%20Frame%20Analysis%20Antonio%20Sanfilippo,%20Lyndsey%20Franklin,%20Stephen%20Tratz,%20Gary%20Danielson,%20Nick%20Mileson,%20Rick%20Riensche,%20Liam%20McGrath%20Pacific%20Northwest%20National%20Laboratory

1
Automating Frame AnalysisAntonio Sanfilippo,
Lyndsey Franklin, Stephen Tratz, Gary Danielson,
Nick Mileson, Rick Riensche, Liam McGrath
Pacific Northwest National Laboratory
2
Overview

• Review of Frame Analysis
• Automating frame annotation
• An application of automated Frame Analysis
• Ongoing and future developments

3
Frame Analysis What is it? What is it for?

• Frame Analysis focuses on how people understand
  situations through the analysis of communicative
  and mental processes to explain

• How communication sources construct issues to
  influence target audiences, e.g. framing suicide
  bombing as martyrdom
• How the target audiences respond to framing, e.g.
  degree of resonance
• Recognizing framing intent leads to an
  understanding of the goals of the communication
  source

4
A brief history of Frame Analysis

• Approach pioneered by Goffman in 1974 has become
  an important analytical components across the
  social sciences
• Strongest impact on the study of social movements

• Renewed interest in the social psychology of
  collective action in the early 1980s led to
  further work which strengthened Goffmans initial
  insights
• Frame Analysis is now a main component in
  theories of social movement

5
Our objectives in automating Frame Analysis

• Despite great recent theoretical advances, there
  still is no systematic method to identify and
  marshal frame evidence in a time/cost effective
  manner
• Address current limitations in the
  representation, acquisition and analysis of frame
  evidence
• Leverage complementary approaches to Frame
  Analysis
• Combine theoretical insights from Frame Analysis
  and Linguistics with Information Extraction
  capabilities and Content Analysis methods

6
Frame Analysis components

• Collective action frames
• Social movement entrepreneurs offer a strategic
  interpretation of issues, e.g.
• Islam is the solution proclaimed Akef
• Create inter-subjective meaning to recruit and
  mobilize people for the promotion of movement
  goals, e.g. establish Shariah law
• Frame resonance (not the focus of this talk)
• Describes the relationship between a collective
  action frame, the target audience, and the
  broader cultural context
• E.g. credibility of the frame and its promoter,
  relevance of the frame to the target audience,
  frame consistency

7
Theories of collective action frames
Gamson Snow and Benford
Injustice identify individuals or institution to blame for grievances Identity specify aggrieved group with reference to shared interests and values Agency recognize that grieving conditions can be changed through activism Diagnostic frame tell new recruits what is wrong and why Prognostic frame present a solution to the diagnosed problem Motivational frame give people a reason to join collective action
Entman Entman
Substantive frame functions Substantive frame foci
Defining effects or conditions as problematic Identifying causes Conveying moral judgment Endorsing remedies or improvements Political events Issues Actors
8
Frame representation intelligent union approach

• PROMOTER
• used by Snow and Benford
• corresponds to the result of Gamsons identity
  frame function
• overlaps with Entmans notion of actors
• COMMUNICATIVE INTENT
• implicit in the frame classification of Gamson
  (injustice, identity, agency) and Snow and
  Benford (diagnostic, prognostic, motivational)
• TARGET
• corresponds to the result of Gamsons injustice
  frame function
• ISSUES
• as in Entman

GROUP-X denounces the insistence of the security apparatus on terrorizing innocent people and on using the emergency law against honest COUNTRY-X citizens, through its campaign of raids and detentions against GROUP-X in the governorates of LOCATIONS A, B, C, and D. PROMOTER
GROUP-X denounces the insistence of the security apparatus on terrorizing innocent people and on using the emergency law against honest COUNTRY-X citizens, through its campaign of raids and detentions against GROUP-X in the governorates of LOCATIONS A, B, C, and D. INTENT
GROUP-X denounces the insistence of the security apparatus on terrorizing innocent people and on using the emergency law against honest COUNTRY-X citizens, through its campaign of raids and detentions against GROUP-X in the governorates of LOCATIONS A, B, C, and D. TARGET
GROUP-X denounces the insistence of the security apparatus on terrorizing innocent people and on using the emergency law against honest COUNTRY-X citizens, through its campaign of raids and detentions against GROUP-X in the governorates of LOCATIONS A, B, C, and D. ISSUES POLITICS SOCIAL LAW SECURITY
9
A methodology that promotes objectivity and
automation

• INTENT is further broken down into 15 speech act
  classes
• ASSERT, BELIEVE, CRITICIZE, EXPLAIN, REQUEST,
• Each INTENT class has various lexical
  realizations (from WordNet)
• We distinguish 9 types of ISSUES
• SECURITY, RELIGION, POLITICS, SOCIAL, LAW,
  MILITARY,
• Each ISSUE has a list of lexical realizations
  (from WordNet Domains)

INTENT CRITICIZE
Lexical realizations accuse, blame, calumniate, charge, condemn, criticize, denigrate, deplore, impeach, incriminate, lambast, malign,
10
A methodology which can be effectively evaluated

• Four human subjects edited frame annotations
  automatically assigned to 30 documents
• The annotation judgments of the four annotators
  were compared and assessed for agreement using
  the kappa test

Cohen kappa test six pairs or annotators Cohen kappa test six pairs or annotators Cohen kappa test six pairs or annotators
Average Ratings Average Kappa Average z-score
1700 0.70 28.68
Fleiss kappa test group of four annotators Fleiss kappa test group of four annotators Fleiss kappa test group of four annotators
Ratings Kappa z-score
1660 0.499 46.2
11
Frame extraction

• Designed and implemented fully automatic
  extraction algorithm to find frames in naturally
  occurring text
• See Sanfilippo et al. (2007) for details

12
Evaluating automatic frame extraction

• Used kappa and precision/recall tests to evaluate
  of manually and automatically assigned
  annotations to 30 documents

Cohen kappa test human vs. computer (four pairs) Cohen kappa test human vs. computer (four pairs) Cohen kappa test human vs. computer (four pairs)
Average Ratings Average Kappa Average z-score
1674 0.52 z 21
Fleiss kappa test four human annotators plus computer Fleiss kappa test four human annotators plus computer Fleiss kappa test four human annotators plus computer
Ratings Kappa z-score
1433 0.422 50.5
(frame detection)
13
Analyzing frame evidence

• Developed a semantically-driven and visually
  interactive search environment to query and
  quantify frame evidence

14
(No Transcript)
15
Using Frame Analysis to support predictive
intelligence

• At the year X elections Group-X (a radical
  religious group) tripled the number of seats
  previously occupied to become the largest
  opposition bloc in the parliament
• Assess whether
• Group-X will adopt more secular views, or
• the recent electoral success will lead to
  increased radicalization

16
Process

• Harvested 619 documents from Group Y official
  website for years X and X1
• Processed documents with frame extraction
  pipeline
• Loaded the results into the frame search
  environment
• Issued semantic queries to identify
• Negotiation frames accept, explain, support,
  etc.
• Contentious frames accuse, criticize, correct,
  reject, etc.

17
Frame query results
Contentious Frames Negotiation Frames z-score
Year-X 24 14 2.28
Year-X1 35 10 6.16
z-score 2.40 1.22
18
Conclusions and further work

• Our approach enables the analysis of messaging
  strategies from document collections in a time
  and cost effective fashion
• Current and future work
• Frame Analysis with direct unreported speech
  (ongoing)
• Frame Resonance (planned)

19
Thanks!
Antonio Sanfilippo Pacific Northwest National
Laboratory Tel. [phone] antonio.sanfilippo_at_p
nl.gov