Application of Case Study Evaluations

1
Evaluative Qualitative Research
How are researchers using QR to do evaluation?
June Lee Choon Muay Susan Heng Tan Ting
Ting Wong Ting Ting 24 July 2000
2
Evaluative Qualitative Research

• What? A QR used for evaluation purposes
• Who? Manager, education or government official or
  school teaching team
• Where? Business organisation, ministry or
  government sector or school
• Why? Describe, document or assess planned change
  Provide considered information to enable a
  decision

3
Evaluative Qualitative Research

• A framework for evaluating the effectiveness of
  technology-assisted learning
• James A. Athanasou
• How? Steps in an evaluation
• Is the program or service ethical?
• To what extent does it cover those who are most
  in need?
• What are the costs, benefits and utilities of it?
• Did the program or service achieve its key
  objectives?
• What is the net effect of the program or service?
• To what extent have the perspectives and
  interests of all stakeholders been considered and
  met?

4
Application of Case Study EvaluationsLynn Davey

• An effective method of learning about a complex
  instance through extensive description and
  contextual analysis.
• Product
• -an articulation of why the instance occurred as
  it did
• -What may be important to explore in similar
  situations.
• Why use it?1. In-depth, longitudinal examination
  of a single instance or event
• 2. Systematic way of looking at what is
  happening, collecting data, analyzing information
  and reporting the results.
• 3.Sharpens the understanding of why an instance
  happened as it did. Point out the important
  points needed to be refined in future research.
• 4. Effective in generating hypotheses rather than
  testing them

5
Types of Case studies
provided by Datta. (1990) previously, we may not
be aware that there are different types of case
studies available. in each type of case study,
some of the items listed will be discussed. A)
features B) functions C) Strengths and
weaknesses Examples 1. Illustrative Case
studies 2. Exploratory Case studies 3. Critical
instance Case studies 4. Program Implementation
Case Studies 5. Program effect Case Studies 6.
Cumulative Case studies.

• .

6
Illustrative Case studies

• Features 1. Descriptive
• 2. uses 1-2 instances to show
  the situation.
• 3. Contains a small number of
  cases.
• Function 1. Helps to make readers who are
  unfamiliar in the field familiar.
• Limitation
• 1. Requires in-depth information on each
  illustration and the truth is that for a
  researcher, you may not have the time on site for
  such examinations.
• 2. Difficulty in the selection of a typical site.
• 3. Problems with the selection of instances.
• Presents a very macro view.

7
Exploratory case studies

• Features
• 1.condensed case studies
• 2. Undertaken before implementing large scale
  investigations.
• 3. Usually great uncertainty exists about the
  program operations, goals and results.
• Strengths
• help identify key questions
• select measurement constructs/help to develop
  measures
• safeguard investments.
• Weaknesses
• pre-maturity conclusions
• the findings may seem convincing enough to be
  released inappropriately as conclusions.
• Tendency to extend the exploratory stage.
• Inadequate representation of diversity.

8
Critical Instance Case studies

• Features
• examines a situation of unique interest, with
  little or no interest in generalizability.
• it can be applied in situations whereby a highly
  generalized or universal assertion is called into
  question.
• Strength
• suited for answering cause and effect questions.
• Weakness
• Inadequate specification of the evaluation
  question
• It is rare. That is why its so difficult to
  predict on problems.

9
Program Implementation Case Studies

• Features
• helps to discern if implementation is in
  compliance with its intent.
• Extensive and thus investment of sufficient time
  is necessary to obtain data and breadth of
  information
• .Multiple sites are usually needed.
• Strength
• Useful for analysis of implementation problems.
• Weakness
• demands on training and supervision of
  researchers as multiple sites are typically
  required to answer program implementation
  questions.
• Demands of data management, quality control and
  validation procedures may lead to the cutting of
  too many corners to maintain quality.
• May lead to generalisation

10
Program Effect Case Studies

• Function
• To determine the impact of programs and provide
  inference about reasons for success or failure.
• Similar to program implementation program case
  study
• main problemevaluation questions usually require
  generalizability and thus for highly diverse
  programs, difficult to answer the question
  adequately or retain a manageable number of
  sites.
• Solution
• to first conduct case studies in site chosen for
  their representativeness. And then verify the
  findings through examination of administrative
  data or survey.
• Use other methods to narrow the scope and
  identify findings of specific interest first
  before implementing them in selected sites to
  maximize the usefulness of the information.

11
Cumulative Case studies.

• Features
• retrospective
• collecting data across studies done in the past.
• Case survey method, used as a means of
  aggregating findings, and backfill techniques.
• Weakness
• Opinions vary as to the credibility of cumulative
  case studies for answering program implementation
  and effects questions.
• Biases that favour program that seem to work,
  thus misleading positive view.
• Problems in verifying the quality of the original
  data and analyses.

12
Conclusion

• Evaluators considering the case study as a design
  for evaluation must
• a) decide what type of evaluation question they
  have and then examine the ability of each type of
  case study to answer it
• b)determine whether the methodological
  requirement of the chosen case study method can
  be met in the situation at hand.
• C) too much data can be generated and these data
  may be difficult to analyse.

13
An example of how researchers are doing
Evaluative QR

• Multimedia, magic and the way students respond to
  a situated learning environment
• Program Effects Case Study
• WHO?
• Jan Herrington Ron Oliver
• Edith Cowan University

14

• WHY?
• Interactive multimedia is a relatively new
  educational innovation
• few studies on how students learn from multimedia
• WHAT PURPOSE?
• To investigate patterns of behaviour of students
  using an interactive multimedia program designed
  according to a situated learning model.

• Situated learning model emphasize on the
  importance of learning within the context of the
  real-world application.

15

• HOW?
• Videotape
• Teachers were videotaped using the program over
  two semester weeks as they completed the
  authentic task.
• Interviews
• Students gave their presentation to the class,
  and shortly after, were interviewed.
• Analysis
• (data collection, data display, and drawing
  conclusion and verification)
• data collected from the interview
• transcripts of observation
• other documentary evidence and notes

16

• CONCLUSION
• The findings suggest that the use of the situated
  learning model was successful in providing
  guidelines for the development of an interactive
  multimedia program which
• - students enjoyed using,
• - enabled them to reflect on their learning and
• - they found easy to navigate
• can be a successful design strategy for
  interactive multimedia programs.

17
The explanation of practice why Chinese students
copy assignmentsViviane M. J, Robinson, Lai Mei
Kuin

• Previously few descriptive accounts, others are
  quantitative survey approach
• Quantitative data decontextualize the practice of
  copying - 2 ways

18

• Needs a method which
• address prob. of rationalisation
  self-deception
• requires gt resource to collect and analyse
  data
• external validity
• Problem-based Approach
• Qn What is the nature of copying? What does the
  ans tell us abt how it shld be inquired into?
• Discovery of the reasons which sustain copying

19

• Setting of the 111 referrals on cheating, 59
  were Asian nationalities (comprises only 18 of
  pop)
• wants to learn more abt how these students
  understood themselves
• researcher mixes with students, find out how
  they understood process of assign completion, why
  they make up a large of those referred.
• 6 students HK, Msia, Spore. Year 1-3,
  Science, Engineering, Commerce or Science

20

• Subjects obtained thru networking. Key indiv and
  nominations
• semistructured interview (probing)
• subj describe how they complete assignmt,
  constraints on practices, consequences.
• Reflection on mtd limited sample size, is
  descriptive, detailed understding, lack of
  properly designed comparison
• implications Uni policy, emphasis on wrongness,
  raise risk of being caught