Concept Mapping for

1
Concept Mapping for Planning and
Evaluation Pre-Conference Workshop European
Evaluation Society 4 October, 2006 Mary
Kane Concept Systems, Incorporated, USA William
Trochim Cornell University, USA
2
AGENDA

• 930 a.m.
• 940 a.m.
• 1010 a.m.
• 1020 a.m.
• 1030 a.m.
• 1050 a.m.
• 1100 a.m.
• 1120 a.m.
• 1200 Noon
• 1220 p.m.
• 1230 p.m.

• Introductions and Expectations
• Defining Concept Mapping
• Your Research Interests
• Break
• Planning the Research Initiative
• Reports
• Data Collection in Concept Mapping
• Analysis and Data Representation
• Concept Maps and Evaluation Some examples
• Closing Using concept mapping in your
  evaluation work
• End of Session

3
We assume.

• Research interest in group- or community-based
  inquiry
• Familiarity with multi-method evaluation research
  
• Interest in integrating qualitative data with
  quantitative analyses

4
Workshop Goals

• At the end of this workshop, you will
• Be able to describe how concept mapping works
  with groups to conceptualize and plan evaluation
• Know how concept maps are built, and how they can
  be used
• Understand the linkage between multivariate
  statistical analyses and community-driven
  evaluation plans.

5
Workshop Logistics

• Activities Lecture and practica
• Agenda Time is of the essence
• Brief group activities on central topics
• Descriptions based on standard Concept Mapping
  process

6
Introductions

• Lightning Round
• Name, Institution
• A specific thing I want to know about concept
  mapping is.

7
Concept Mapping An Integrated Mixed Methods
Approach

• Concept mapping is a structured process, focused
  on a topic or construct of interest, involving
  input from multiple participants, that produces
  an interpretable pictorial view of their ideas
  and concepts and how these are interrelated. The
  process is participatory in that it is inherently
  a mixed methodology that integrates high-quality
  qualitative and quantitative techniques.
• Trochim, 1989

940
8
Concept Mappings two keys

• focused on a topic or construct of interest
• To the field
• To the client
• To the participant
• involving input from multiple participants
• Who can contribute
• Who have a stake in the issue
• Who will be heard

9
Concept Mapping

• Seeks the variability in stakeholder knowledge,
  opinion
• Presents a process to manage the variability
• Develops results that are a synthesis of group
  articulation
• Connects qualitative and quantitative approaches
  in a mixed methods application.

10
Three major steps in concept mapping

• Planning
• Data Collection
• Analysis

11
Concept Mapping Planning
Develop the focus prompt to ensure inclusive and
relevant content development,
Identify participants whose knowledge, expression
of opinion or position will contribute to the
resulting framework,
Consider and define respondent questions for use
in analysing input,
Devise a timeframe and project plan that planners
agree to.
12
Data Collection via Participatory Approaches
Participants contribute brainstormed ideas,
knowledge and opinions
Participants individually organize the results of
the idea development, authoring the structure
domain of the issue
Participants individually organize and rate the
results of the idea development, authoring the
value dimensions of the issue
13
Analysis and Application
The conceptual framework, including visual
results of maps, pattern matches and go zones,
enables clear and agreed-upon interpretation by
communities of interest
Strategies and tactics for action follow directly
from the interpretation of the results, providing
the stakeholders with group-authored direction,
measures and evaluation strategies.
14
An example

• Driven by Goals
• Develop a current state identification of the
  issues that affect mental health
• To guide program development, gap identification,
  and evaluation planning
• Focus A specific issue that is relevant to the
  mental health of women and girls is
• Participants Federal, state and local agency
  representatives, consumer advocates, public
  health, mental health researchers

15
The initial map shows all the elements in
relation to one another
16
Conceptually similar ideas are in close proximity
Point Map Location has Meaning
18
18
4
4
16
16
33
33
44
44
8
8
12
90
12
90
27
27
82
96
82
96
64
64
11
94
11
94
92
106
92
50
50
77
77
48
48
68
68
21
24
21
24
38
51
38
51
78
78
69
69
89
89
15
15
107
59
59
19
19
55
55
57
57
63
63
10
10
85
85
84
84
5
5
100
100
26
26
72
20
20
81
81
54
54
13
88
13
88
99
99
76
76
40
40
79
97
79
97
37
73
37
73
56
56
80
102
80
23
62
23
62
105
67
67
1
29
1
29
66
66
52
52
60
60
39
39
6
6
53
53
28
36
91
28
36
91
83
42
83
2
2
74
74
22
22
45
75
45
75
42
31
31
104
25
25
49
49
101
32
32
58
58
30
30
17
17
14
14
98
98
7
87
7
87
103
41
41
46
46
86
86
71
95
71
95
93
93
9
34
47
9
34
47
3
43
3
43
35
35
70
70
61
61
65
65

• Recognition of enduring effects of depression.
  (72)
• The behavioral role of depression and stress in
  contributing to obesity in women. (76)
• Co morbidity of mental disorders (depression,
  mood disorders, substance abuse including
  smoking, eating disorders, harming oneself and
  suicide). (79)

17
The detailed ideas are grouped
so that many concepts can be considered in a
shared structure
18
Sample Concept Map
19
Sample Cluster Social Stress
Increased risk of victimization for women with
severe mental disorders receiving inadequate
treatment. (3) Body image issues- (breast size,
hair color/texture, nose, other physical features
vs external valuation of "beauty"). (9) The
extent to which lower socioeconomic status and/or
immigrant status relates to mental health.
(41) Negative images of girls and women,
particularly among minority women, in television,
magazines, and film-related media. (43) Lack of
encouragement and opportunity at the elementary,
middle and high school levels for career
opportunities that girls can aspire to. (61) The
discrimination and lack of social acceptance that
those with mental disorders face. (65) .Etc.
101
101
98
98
41
41
86
86
71
95
71
95
9
9
3
43
3
43
70
70
61
65
61
65
20
The Rating ResultsPoint and Cluster Rating Maps

• Describe as the value dimension of the maps
• Refer to rating(s) the group supplied
• Present point rating map
• Overlay cluster rating map and surface comments
  regarding patterns, regions

21
Interpreting Ratings Information
Importance
High Importance
4.16
Access Insurance (4.16)
Vertical number line of selected ratings
Violence Abuse (3.80)
Protective Factors Resilience (3.70)
Identification Treatment Issues (3.67)
Medical System Issues (3.58)
Mood Anxiety Disorders (3.55)
Sex/Gender Differences (3.42)
Social Stress Factors (3.39)
3.39
Low Importance
22
Interpreting Ratings Information
Already in Action
3.42
Action Potential
Access Insurance (3.42)
Medical System Issues (3.41)
Protective Factors Resilience (3.34)
Violence Abuse (3.31)
Identification Treatment Issues (3.25)
Mood Anxiety Disorders (3.19)
Sex/Gender Differences (3.13)
Vertical number line of selected ratings
Social Stress Factors (2.89)
2.89
No Potential Action
23
Pattern Match ComparingImportance and Action
Potential (Absolute Scale)
24
Looking at Go-Zones

• Bivariate plot for each cluster, showing each
  statement in the cluster
• Compares ratings on 2 scales OR 2 subgroups on
  one rating
• Places each item in a field according to how high
  or low each of the 2 ratings on it were
• High/high rated items are the go items
• Low/low items are the No go items
• Others are gap items
• For discussion and action decision making

25
Go-Zone Analysis Example
Increased risk of victimization for women with
severe mental disorders receiving inadequate
treatment. (3) The extent to which lower
socioeconomic status and/or immigrant status
relates to mental health. (41) The
discrimination and lack of social acceptance that
those with mental disorders face. (65) Internal
barriers to mental health care such as shame and
guilt. (101)
Adjustment to physical illness and declining
ability. (98)
Negative images of girls and women, particularly
among minority women, in television, magazines,
and film-related media. (43) Lack of
encouragement and opportunity at the elementary,
middle and high school levels for career
opportunities that girls can aspire to. (61) Care
giving and nurturing for spouse and family
including aging parents or those with special
needs. (95)
Body image issues- (breast size, hair
color/texture, nose, other physical features vs
external valuation of "beauty"). (9) Social
devaluation of the nurturer/ maternal role.
(70) The support to pursue personal development
and engagement in fulfilling societal roles (to
include major leadership roles). (71) Media
pressures on adolescent sexuality. (86)
26
Planning

• Planning in concept mapping
• Selection of participants
• Finalizing focus
• Identifying demographic or respondent questions
• Determining time frame for participation
• Anticipating participation issues
• Anticipating audiences for, and applications of,
  results
• Developing results for dissemination and use

27
Your research interest

• On the sheet provided, note a specific research
  question or evaluation planning initiative that
  you are considering or have interest in. Note
• Type of organization
• Purpose of the project, desired outcome
• Potential participants, communities of interest

1010
28
Break

• 10 minutes
• Prompt attendees are rewarded!

1020
29
Our four sample research questions

• Red Group
• Blue Group
• Yellow Group
• Green Group

1030
30
Instructions, and Timing

• In your group,
• Review the projects goals and desired outcomes
• Acting as the project planners, discuss and
  suggest answers to the questions on your Exercise
  2 Worksheets
• Be prepared to report back at 1050

31
Reporting Observations
1050
32
Participatory Data Collection

• Simple data collection methods

Participants contribute brainstormed ideas,
knowledge and opinions
Participants individually organize and rate the
results of the idea development, authoring the
structure and value domain of the issue
1100
33
Idea generation approaches

• Traditional brainstorming
• Statement sets abstracted from reports,
  literature reviews or other documents
• Prescribed statement sets
• Statement sets abstracted from interviews or
  focus groups

34
Sorting and Rating Simple Rules

• Individual organization of the brainstormed
  content
• According to relative similarity of meaning
• Individual judgment
• Used to organize the conceptual domain according
  to all participants input
• Rating(s)
• According to relative value to the question
• Examples importance, current capacity,
  feasibility
• Used to compare subgroups on value

35
Sorting and Rating Exercise

• Use the cards provided.
• Organize the 10 ideas into piles of ideas that
  you believe represent their relative meaning,
  following the instructions on the worksheet.
• Record your sort data on the worksheet.
• Discuss with your group members the similarities
  and differences among your sorts, following the
  prompt questions.
• Use the sample rating sheet and complete the
  importance ratings for the 10 items. Discuss
  with your group the common or different values
  you assigned.
• Consider the questions on the worksheet. Be
  prepared to report one observation from the group.

1110
36
Inputs, Analyses and Outputs
Output Edited randomized list of statements on
the issue to be used by stakeholders
Analyses Sort data aggregation,
Multidimensional scaling, Hierarchical Cluster
Analysis, Labeling Analysis, Anchoring/Bridging
Analysis Outputs Point Maps, Cluster Maps,
Labels, Reports
Analyses Ratings averages per statement and
per cluster Outputs Point Rating Maps, Cluster
Rating Maps, Pattern Matches, Go Zones
1120
37
Analysis Guided Exercise

• Review the exercise 4 and 5 worksheets, and
  use as lecture guides.
• Note discussion observations at the end of the
  worksheet.

38
Aggregate Sort Data
39
Multidimensional Scaling
40
Multidimensional Scaling
Similarity Matrix
1 2 3 1 5 1 2 2 1 5 0 3 2 0 5
41
Multidimensional Scaling
Similarity Matrix
1 2 3 1 5 1 2 2 1 5 0 3 2 0 5
1
42
Multidimensional Scaling
Similarity Matrix
1 2 3 1 5 1 2 2 1 5 0 3 2 0 5
2
1
43
Multidimensional Scaling
Similarity Matrix

• 1 2 3 4
• 1 5 1 2 4
• 2 1 5 0 0
• 3 2 0 5 2
• 4 4 0 2 5

2
3
1
44
Multidimensional Scaling
Similarity Matrix
1 2 3 4 1 5 1 2 4 2 1 5 0 0 3 2 0 5 3 4 4 0 3 5
A maps similarity data can be depicted as a
coordinate matrix
45
Multidimensional Scaling
Similarity Matrix
1 2 3 4 1 5 1 2 4 2 1 5 0 0 3 2 0 5 3 4 4 0 3 5
y
2
3
A map can be depicted as a coordinate matrix
1
x
And from the coordinates we can compute the
distances between all pairs of points
a2 b2 c2
a difference between x values b difference
between y values c distance
46
Multidimensional Scaling
Similarity Matrix
1 2 3 4 1 5 1 2 4 2 1 5 0 0 3 2 0 5 3 4 4 0 3 5
y
2
3
And can show these as a matrix of distances
between points
A map can be depicted as a coordinate matrix
1
x
And from the coordinates we can compute the
distances between all pairs of points
a2 b2 c2
a difference between x values b difference
between y values c distance
47
Multidimensional Scaling
Similarity Matrix
1 2 3 4 1 5 1 2 4 2 1 5 0 0 3 2 0 5 3 4 4 0 3 5
y
2
3
And can show these as a matrix of distances
between points
A map can be depicted as a coordinate matrix
1
x
And from the coordinates we can compute the
distances between all pairs of points
a2 b2 c2
a difference between x values b difference
between y values c distance
48
Multidimensional Scaling
Similarity Matrix
1 2 3 4 1 5 1 2 4 2 1 5 0 0 3 2 0 5 3 4 4 0 3 5
Low stress values means there is a greater
correspondence between the similarities and the
map
49
Multidimensional Scaling

• Directionality
• Does MDS know North from South?
• Dimensionality
• Why only two dimensions?
• Stress
• How much does it really matter?

50
Cluster Analysis

• Hierarchical
• clusters get built in a tree-like method
• Agglomerative
• builds toward all items in one pile (divisive -
  all start in one and divide)
• Clustering criterion
• the rule used to decide the next cluster merge
• Wards algorithm
• Number of Clusters

51
Cluster Analysis
7
5
8
9
10
1
6
2
3
4
Merge
Points Merged
1
52
Cluster Analysis
7
5
8
9
10
1
6
2
3
4
Merge
Points Merged

1 6
1
53
Cluster Analysis
7
5
8
9
10
1
6
2
3
4
Merge
Points Merged
1 6 5 7
1 2
54
Cluster Analysis
7
5
8
9
10
1
6
2
3
4
Merge
Points Merged
1 6 5 7 9 10
1 2 3
55
Cluster Analysis
7
5
8
9
10
1
6
2
3
4
Merge
Points Merged
1 6 5 7 9 10 (1 6) 8
1 2 3 4
56
Cluster Analysis
7
5
8
9
10
1
6
2
3
4
Merge
Points Merged
1 6 5 7 9 10 (1 6) 8 3 4
1 2 3 4 5
57
Cluster Analysis
7
5
8
9
10
1
6
2
3
4
Merge
Points Merged
1 6 5 7 9 10 (1 6) 8 3 4 2 (9 10)
1 2 3 4 5 6
58
Cluster Analysis
7
5
8
9
10
1
6
2
3
4
Merge
Points Merged
1 6 5 7 9 10 (1 6) 8 3 4 2 (9
10) ((1 6) 8)) (3 4)
1 2 3 4 5 6 7
59
Cluster Analysis
7
5
8
9
10
1
6
2
3
4
Merge
Points Merged
1 6 5 7 9 10 (1 6) 8 3 4 2 (9
10) ((1 6) 8)) (3 4) (5 7) ((2 (9
10))
1 2 3 4 5 6 7 8
60
Cluster Analysis
7
5
8
9
10
1
6
2
3
4
Merge
Points Merged
1 6 5 7 9 10 (1 6) 8 3 4 2 (9
10) ((1 6) 8)) (3 4) (5 7) ((2 (9
10)) (((1 6) 8)) (3 4)) (5 7)
((2 (9 10))
1 2 3 4 5 6 7 8 9
61
Summary Map production is based on

• Similarity matrix
• MDS
• Hierarchical cluster analysis

62
Ratings Analysis

• Performed after the maps are finalized, and map
  data is saved.
• Input
• Individual stakeholder ratings on each statement
  on each scale relevant to the issue (usually two)
• Respondent questionnaire of demographic or
  organizational characteristics
• Used to produce ratings maps, pattern matches, go
  zones.

63
Running the Analysis

• SPSS
• Enter sort data
• Compute similarity matrix
• Run MDS
• Run Cluster Analysis (on MDS coordinates)
• Select number of clusters
• Plot maps
• Produce rating statistics
• Produce rating maps
• Produce pattern matches (in Excel)
• Produce bivariate plots
• Post-process plots in graphics program
• Produce reports

• Concept System
• Select sorts
• Run analysis
• Select number of clusters
• Produce maps, matches and go-zones
• Produce reports

64
Concept Mapping Applications
20
71
82
56
1200
65
Concept Map Process Model
66
Logic Model
External Recognition
External Recognition
And Support
And Support
Internal Recognition
Internal Recognition
And Support
And Support
Communication
Communication
Training
Training
Publications
Publications
Methods
Methods
Health
Health
Outcomes
Outcomes
Collaboration
Collaboration
Interventions
Interventions
Science
Science
Models
Models
Transdisciplinary
Transdisciplinary
Integration
Integration
Immediate Markers
Immediate Markers
Intermediate Markers
Intermediate Markers
Long
-
Term Markers
Long
Term Markers
67
Your Packet

• Exercise Worksheets
• Take Home Worksheets
• CM in Evaluation Trochim
• Bibliography

68
Thank you!
William Trochim Cornell University, Policy
Analysis and Management 607 255-0887 wmt1_at_cornell.
edu Mary Kane Concept Systems Inc. 607-272-1206 w
ww.conceptsystems.com
1230