A Pragmatic View on the Integration of Text Analysis and Statistical Analysis Techniques

1

• A Pragmatic View on the Integration of Text
  Analysis and Statistical Analysis Techniques
• By
• Normand Péladeau
• President
• Provalis Research Corp.

2
ANALYSIS OFNUMERICAL DATA
Statistical Analysis
ANALYSIS OFTEXTUAL DATA
3
THREE TYPES OF TEXT ANALYSIS
4
(No Transcript)
5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
THREE TYPES OF TEXT ANALYSIS
9
THREE TYPES OF TEXT ANALYSIS
10
THREE TYPES OF TEXT ANALYSIS
11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
Reactions to the launching of CRS-1
16
THREE TYPES OF TEXT ANALYSIS
17
THREE TYPES OF TEXT ANALYSIS
18
THREE TYPES OF TEXT ANALYSIS
19
(No Transcript)
20
Clustering of Cases
21
more staff in hospitals ,police ,social
workers Police and social workers More community
midwifery and social worker input. more
frontline staff eg social workers, police youth
aid etc.
parenting skills and support Courses on parenting
skills for parents Parenting skills programes for
all. Helping Young parents in parenting
skills Parenting skills for young, as well as
new, parents.
drug and alcohol abuse Alcohol and drug
prohibition Drug and alcohol abuse. Reintroduce
six o'clock closing. alcohol and other drug
agencies to work with families and the
addicted Education, with an emphasis on drug and
alcohol use and abuse
Fund families to look after each other Fund
healthy parenting courses Funding in schools
Funding in hospitals Funding in poor
neighborhood education and funding for help
centers Funding of organizations like Parent Inc
to help them help more people
22
Clustering of Words small clusters
23
Clustering of Words larger clusters
24
Clustering of Words even larger clusters
25
Correspondence Analysis on Words
26

• Other Examples of Text Mining Techniques
• Automatic Document Classification
• Sentiment Analysis (positive vs negative)
• Machine Learning to support human coders

27
THREE TYPES OF TEXT ANALYSIS
28
THREE TYPES OF TEXT ANALYSIS
29
THREE TYPES OF TEXT ANALYSIS
Qualitative Analysis
ContentAnalysis
TextMining
Validity
Minutes Hours Days
Days Weeks Months
Time Requirements
30
TEXT MINING SOFTWARE CLAIMS Our algorithm
reproduce human understanding, work all the time,
and may thus replace human coders.

• ANSWERS From a pragmatic point of view,
• Give me a break!
• Text-mining techniques sometimes work amazingly
  well, sometimes fail poorly.

31

• EXAMPLE OF FAILURE 1
• TASK For a given news article, predict the
  following vertical markets
• Education
• Health
• Finance
• Government
• RESULTS
• Education, Health and Finance gt 80-90
• Government 55

32

• EXAMPLE OF FAILURE 2
• TASK Sentiment analysis of product reviews
• RESULTS
• Works well on some consumer products (MP3
  players, cameras, etc.) and services (hotels,
  restaurants).
• Works not as well on some others (internet
  security, insurance policy, legal services,
  movies)

33

• Text Analysis Software Should
• Provide feedback to allow you to assess success
  or failure.
• Have the ability to manually override decisions
  made by the computer and ideally learn from those
  mistakes.
• Give you the opportunity of using an alternative
  approach, or provide a way to export data to
  another text analysis tool.

34

• Because
• No single method is appropriate for all text
  analysis tasks.
• A single text analysis task often profit from
  combining several methods.
• Text analysis software should facilitate the
  combination of several methods rather than hinder
  it.

35
Thank you for your attention!