Qualitative Assessment

1
Qualitative Assessment

• Qualitative inquiry cultivates the most useful
  of all human capacitiesthe capacity to learn
  from others.
• Halcoms Evaluation Laws. Cited in Patton
  (1990). Qualitative Evaluation and Research
  Methods. pp. 7.
• Moving Forward with Assessment
• Difficult Dialogues Initiative
• Assessment Conference
• Ford Foundation, NY, NY
• October 16-17, 2008
• (H. Hernández-Gravelle)

In addition to works cited, the author is
indebted to the work of M. Patton A. Driscoll
B.A. Miller D.T. Spaulding L.F. Gardiner, C.
Anderson, and B.L. Cambridge T.A. Angelo and
K.P. Cross and C. McTighe Musil, M. Garcia, Y.T.
Moses, and D.G. Smith.
2
Qualitative Assessment Two Definitions

• An inquiry process of understanding a social or
  human problem, based on building a complex,
  holistic picture, formed with words, reporting
  detailed views of informants, and conducted in a
  natural setting.
• Creswell (1994). Research design Qualitative
  quantitative approaches. Cited in Detlor (2004).
  Towards Knowledge Portals.
• Explores and tries to understand people's
  beliefs, experiences, attitudes, behaviors and
  interactions. It generates non-numerical data
• http//new.wales.gov.uk/about/aboutresearch/socia
  l/glossary/?langen.

3
Qualitative Assessment Tidbits

• 1899 W.E. Dubois, undertook first social survey
  in the United States.
• 1900s Early phase of qualitative research.
  Rooted in Social Investigation.
• 1950s Government provided grants in support of
  educational research. Preferred modality is
  qualitative research.
• Late 1960s Qualitative research came into
  being. Development of socially responsive
  programs.
• Late 1900s Accountability movement in education
  results in increasing use of qualitative research
  and assessment.

4
Why DDI Qualitative Assessment?

• Well suited in areas that are complex,
  multidimensional sensitive and nuanced.
• Illustrative DDI Project Titles-
• Engaging Controversy Religion, freedom and the
  topics of identity
• Imagining the Future Dissent, dialogue and the
  freedom to inquire
• Practicing Pluralism Interactive theater, campus
  climate and academic freedom
• The open nature of qualitative approach allows
  the subjects to respond according to their own
  framework.
• In assessing the impact of a faculty development
  workshop in adoption of new pedagogy, the
  teachers experience, reasons for participating
  in the workshop or the students that she teaches
  may impact her view of the value of the workshop
  and her sense of its applicability.
• Quote from grantee.
• Helps to understand a program / situation as a
  whole.
• ..the holistic approach assumes that the whole
  is understood as complex system that is greater
  than the sum of its parts.
• Patton (1990). Qualitative Evaluation and
  Research Methods.

5
Why DDI Qualitative Assessment?Continued

• To provide a richer, more descriptive picture of
  the information being gathered or responses to
  the questions being asked. For example, assessing
  and describing in a rich manner the quality of
  increased capacity for exchange between Arab and
  Israeli students.
• On college campuses addressing pluralism issues,
  qualitative assessment can in itself serve as a
  process of inclusion and appreciation for
  multiple values and views.

6
SOME CHARACTERISTICS OF THE QUALITATIVE PROCESS

• Process extends through data reporting, data
  analysis, and utilization of findings.
• Researchers lens can impact data gathering and
  interpretation. Expertise in both assessment and
  subject area are significant.
• It is complex and requires multiple iterations to
  better uncover information (experiences,
  perspectives, views).
• Large volumes of data can accumulate quickly.
  Allocation of time and resources for data
  collection and analysis is imperative.
• Value You get out what you put into it!

7
Qualitative Approaches Used by DDI

• FOCUS GROUPS In qualitative assessment you go
  to the source (naturalistic group)
• INTERVIEWS direct source
• PARTICIPANT OBSERVATIONS To fully describe the
  process, you want to observe it
• PORTFOLIO ASSESSMENT inductive
• FILM DOCUMENTATION profiles and pictures

8
Assessment at its Coreis about Student Learning
Outcomes

• Question How do we get from here to there?
• Pluralism is ongoing work
• Assessment is an ongoing process
• Tools to assess student learning outcomes
• Portfolio
• Rubrics
• In classroom exercise (presentations, debate)
• Journal

9
ENHANCING DDI INDICATORS OF SUCCESS IN
QUALITATIVE ASSESSMENT

• Ensuring indicators of success are sound
  Important particularly for subjects and areas
  that are considered outside of the core
  curriculum or where the topics are controversial
• Unearth narratives about the complex history of
  race
• Quote from grantee.
• Members check-in This process can also serve to
  inform the constituencies and to begin process of
  change.
• Triangulation Using multiple sources to confirm
  findings.
• Enlisting assessment advice and support from
  experienced and knowledgeable colleagues Serves
  as a form of triangulation and check-in. Can
  address gaps in assessment capacity.
• Detailed reporting Necessary to illustrate how
  the assessment team arrived at findings and to
  project the richness of the data.

10
DISTINGUISH ANECDOTAL EVIDENCE FROM QUALITATIVE
DATA

• Unsolicited responses to programs in the form of
  criticism or praise is considered anecdotal
  evidence. Such evidence is important and worthy
  of consideration, but it does not constitute
  formal assessment. Qualitative approaches are not
  the same as random responses and are designed to
  collect data that can be verified and validated.
• The Ford Foundation is interested in vigorous
  assessment that
• allows you to shape programs as they are
  implemented
• demonstrates what has been achieved
• enables you to examine what might be done
  differently as you continue to develop programs

11
ANALYSISMIRRORS PROCESS

• INDUCTIVE
• ON GOING
• MULTILAYERED
• COMPLEX
• ENRICHED THROUGH MEMBERS CHECK-IN

12
Qualitative Reporting

• Reporting formats are shaped by assessment
  purpose and by the meaning that is being
  conveyed.
• Typical qualitative reporting includes narrative
  accompanied by matrix or charts that demonstrate
  key issues or causal relationships.
• Ford Foundation assessment reporting seeks to
  communicate outcomes, indicators of program
  success and lessons learned through DDI programs.
  Use of case studies and rich description is of
  particular interest.