Program Evaluation : Quantitative Vs. Qualitative Approaches

1
Program Evaluation Quantitative Vs.
Qualitative Approaches IndiaCLEN Program
Evaluation Network
Narendra K. AroraDepartment of PediatricsAll
India Institute of Medical SciencesNew
Delhi narendrakumararora_at_hotmail.com
Program Evaluation- Chandigarh 29th February 2004
2

• Research is not being translated into policy

Evidence
The dominant finding of our review is that
there are large gaps between the care people
should receive and the care they do receive.
This is true for all types of care - preventive,
acute, and chronic ... for different types of
health insurance for all age groups, from
children to the elderly Schuster et
al, Milbank Quarterly, 1998
3
Research vs. Policy Practice

• Research FOR policy
• Priority setting for health research
• Program design
• Program monitoring and evaluation
• Practice guidelines Rational use of drugs
• Needs assessment introduction of new technology
  programs
• Research ON policy
• Privatization of health services
• Clinical Audit

4
Research and Policy Practice

• Policy for Research
• Promotion of Research
• Practice (based) on Research
• Evidence based practice

5
Research and Policy Practice

• Research
• Publications
• Changes in Policy Practice

Natural Passive Diffusion of Information
6
Research and Policy Practice

• Information alone is insufficient for changes in
  policy and practice
• Researchers are not systematically involved in
  implementation of their own findings
• Lack of policy relevant evidence
• Scientific evidence vs. need of policy makers
• Guidelines
• Not commensurate with results
• Not understandable by Policy makers

7
(No Transcript)
8
Components of a Program

• Policy Needs Rationale
• Policy Aims Goal
• Program Strategy
• Planning Implementation

9
Program Evaluation

• A systematic effort to describe the status of a
  program
• Extent to which program objectives achieved

10
Consumers of Evaluation Research

• Government
• Program Managers
• Policy Makers
• Funding Agencies
• Health Care Research Institutions
• Clinicians
• Community
• Identifying the stakeholders

11
Use of Program Evaluation Data

• Policy Makers / Program managers
• Needs Assessment
• Redefining aims / objectives
• Modifying or fine tuning strategies (process)
• Sustainability (including fatigue factor)
• Judge the worth (impact)
• Expense / cost
• Interaction with other activities / health
  systems
• Needs of Policy Makers

12
Use of Program Evaluation Data

• Academia
• - Wider application of program strategies / Cross
  country / culture
• Unique features (success/failure)
• Determinants of provider and client behavior

13
Types of Evaluation

• Needs Assessment
• To identify
• Goals
• Products
• Problems
• Conditions

14
Types of Evaluation Contd

• Formative (Process) Evaluation
• To improve developing or ongoing program
• Role as helper/advisor/planner
• Progress in achievements
• Potential problems/needs for improvements in
• Program Management
• Inter-sectoral coordination
• Social mobilization
• ? Implementation
• ? Outcomes

15
Types of Evaluation Contd

• Summative (Coverage) Evaluation
• (To help decide ultimate fate)
• Summary statement about
• Programs achievements
• Unanticipated Out comes
• Comparison with other programs

16
Qualitative Vs Quantitative Research - the
dichotomy
Qualitative Quantitative
Social theory Action Structure
Methods Observation, Interview Experiment, Survey
Question What is X ? (classification) How many Xs? (enumeration)
Reasoning Inductive Deductive
Sampling method Theoretical Statistical
Strength Validity Reliability
17
Qualitative Research Key Strengths

• Qualitative methods aim to make sense of, or
  interpret, phenomena in terms of the meanings
  people bring to them
• Qualitative research may define preliminary
  questions which can then be addressed in
  quantitative studies
• A good qualitative study will address a clinical
  problem through a clearly formulated
  question using more than one research method
  (triangulation)
• Analysis of qualitative data can should be done
  
• using explicit, systematic, reproducible
  methods

18
Potential limitation of Qualitative Methods

• Accuracy of the information
• representative nature
• Cultural inappropriateness
• Subjectivity of the Investigator

19
Qualitative Research Methods

• Findings
• - Presented alone / in combination with
  quantitative data
• - Validity and reliability depends on
  methodological skills, sensitivity,
  integrity of the researchers
• - Skillful interviewing - more than just asking
  questions
• - Content analysis - more than just reading to
  see whats there
• - Generate useful and credible findings through
  observation, interviewing and content
  analysis
• How?
• - Discipline, knowledge, training, practice,
  creativity, hard work

20
(No Transcript)
21
Multidisciplinary Central Coordinating Team
Program Evaluation Expert
Health Social Scientist
Epidemiologist
Program Evaluation
Clinicians
Economist
Biostatistician
Anthropologist
22
Health Program Evaluation - Quantitative Research
Methods

• Approach
• - Measures the reaction of a great many
  people to a limited set of questions
• - Comparison and statistical aggregation of
  the data
• - Broad, generalizable set of findings
  presented succinctly and parsimoniously.

23
Multi-Centric Evaluation Studies

• Cohesive network of partners
• Common understanding of the program
• Common understanding of aims
  objectives of evaluation exercise
• Standardization of research instruments
• Standardization of protocol implementation
  at various sites
• Regional variation in program implementation
• Multiple layers of quality assurance measures
• Generalizable and hence more confidence

24
Challenges in Delivery of Routine Public Health
Programs in India Vitamin-A and Iron folic
acid Supplementation Program(s) - a case study
Supported by Ministry of Health Family Welfare
Government of India, The Micronutrient
Initiative, Canada, INCLEN, USAID AIIMS
25
Characteristics of A Routine Program?
26

• We are not bothered about those we could not
  reach because they are not deficient in
  Vitamin-A. Otherwise they would have come with
  night blindness in OPD.
• District level provider (8001) Kolkata
•  
• Those who come to us we give them. Clients not
  coming must be healthy.
• Government health worker (1006) Ballabgarh
• We are already over burdened. What can we do
  there?
• Government health worker (11013) Nagpur

27
Passiveness in Implementation
28
Resistance to Polio Drops among Clients
29
Who are Resistant/Reluctant Clients

• Rumours
• Rationale and safety of repeated dosing not
  clear
• Rumors regarding vaccine safety
  (sterility/HIV-AIDS)
• AFP cases occurring in children who had
  received OPV drops in the past

"We heard that the only purpose (of giving these
drops to our children) is that they cannot
produce children in future." Client
(Non-utilizer Men)
"Cold (OPV) drops will make them (children) cold
(no sexual arousal). People believe in rumors
that children will become impotent."
Client (Non-utilizer Women)
30
Resistant/Reluctant Clients

• Social Mobilization Strategy
• Use of force counter productive/fatal

Police came and they forcibly gave us the
drops. Client (Non-utilizer Men)
I have seen it myself that force was not used on
Hindus but when they (health workers) approached
Muslim houses, they used force to give drops to
their children. Client (Non-utilizer Men)
31
There is no need to use force (for good things
like polio drops). If you try to explain by
compassion, people will understand (accept it).
Suppose you give me something to eat and I refuse
to oblige, if you pick up a stick and force
(threaten) me to do it, it invites suspicion that
something is fishy about it and that is why I am
being forced to eat. If you offer with love, I
will take it. This problem exists
everywhere. Client (Male Non-utilizer)
"If a Mohammedan doctor or a priest (Maulvi)
comes and explains, people will understand and
give the drops. Otherwise people will think that
government is asking us to drink poison (drink
drink....... )." Client (Non-utilizer Men)
32

• This is an unprecedented event where all people
  irrespective of caste, creed and religion take
  part in PPI program on the same day (NID)
  throughout the country
• Health worker (150) Burdwan
• He (my husband) told me that everybody is going
  for polio drops. Then why should we be left out ?
  After all, everybody is not a fool
• Utilizer (1422) Delhi

33
Quotable Quotes

• Give a vivid, meaningful flavor which is far
  more convincing than pages of summarized
  numbers
• - These should not be distracters
• - Should not take the reader away from the
  real issues in hand

34
Reasons for Non-utilization of FHAC Program
Services Process Evaluation
Reasons Provider perspective NGO/ leader Perspective Client perspective
Lack of awareness about FHAC 1 1 4
Low motivation 2 2 1
Shyness / Embarrassment 2 to 3 2 1
Adversities (facilitatory factors) 1 lt1 lt1
Client Conveniences Season / month Distance Timing of camp lt1 1 2 1 1 2 to 3 1 to 2 1 3
Camp facilities 1 1 1
Low credibility 1 lt1 lt1
Semi Quantitative Qualifiers 1 Some/a few, 2
About half, 3 A majority, 4 Most, 5 Almost all
IndiaCLEN FHAC Program,2000 Process Evaluation
35
Reasons for Non-participation in FHAC Camps by
the Clients (RURAL)
Reasons for Non-participation Men (95 CI) Women (95 CI) Total
Not aware about FHAC program 82.3 (79.9-84.8) 79.7 (76.9-82.5) 81.0 (78.7-83.4)
Indifferent 2.4 (1.6-3.4) 2.6 (1.9-3.4) 2.5 (1.9-3.1)
Embarrassed 0.3 (0.0-0.6) 1.3 (0.8-1.8) 0.8 (0.5-1.2)
Not aware about camp 0.8 (0.4-1.3) 0.7 (0.3-1.0) 0.7 (0.4-1.0)
Need not felt 2.3 (1.6-3.0) 5.6 (4.4-6.8) 3.9 (3.2-4.7)
Inconvenient timings 2.0 (1.1-3.0) 2.4 (1.2-3.5) 2.2 (0.5-2.9)
Loss of wages (affordability) 8.0 (6.5-9.4) 5.8 (4.5-7.3) 7.0 (5.8-8.1)
Not in village (availability) 1.7 (1.0-2.3) 1.6 (1.0-2.1) 1.6 (1.2-2.1)
IndiaCLEN FHAC Program,2000 Coverage Evaluation
36
What is a Safe Injection?

• All injections are safe
• Right drug/ good medicine/ expiry of drug has
  been checked/ right dose/specific drugs are
  safe/skin sensitivity test should be done before
  giving
• injection/ drug dissolve in muscles/drug should
  be tested/injection should be given with
  glucose/ single dose vial to be used
• Use new plastic syringes/ syringe needle from
  a sealed pack / expensive syringe/needle/boiled
  syringe and needle/sterilized glass syringes
  needle/one needle and syringe for one patient/
  use of disposable syringes / needles / ISI
  mark/avoid reuse chemical stylizations of
  syringes of syringes needle/ new needle from
  the sealed pack
• Needle and syringe clean / proper/ flushing
  of syringes and needle using spirit /
  water/steel needles.
• Technique of giving injection gives
  slowly/properly/painless injection/proper site of
  giving injection/giving intramuscularly is safe

37
What is a Safe Injection?
Conti .

• Before injection ask patient to roll up shirt/
  clean the injection site with spirit swab
• Injection giver wash hands/ use gloves/observe
  sterile precaution
• Injection taken from recognized hospital
• Injection given when necessary/ rational use
• Disease should get completely cured/ maximum
  curative effect
• Prevention of disease by immunization
• Injection that prevent blood borne diseases
• Does not cause reaction
• Needle and syringe flushed cleaned before
  disposal/ syringe thrown away
• Doctors know about safe injection
• Safe to reuse syringes/ needles to the same
  person
• No injection is safe
• Clients should be made aware about the safe
  injection

38
When do you (Prescriber) give an injection?

• Initial stages of diseases, for quick relief
• Patient not improving with oral drug, For
  serious diseases
• Side effects of oral drugs
• Doctor decides, Prescriber prefer injection
• Doctor wants to earn money
• Type of disease
• In case of emergency
• Better compliance to injection
• To restore energy
• Patient demand Injection
• Certain categories of patients (poor,
  uneducated, rural patient)
• Those who can afford injection
• Vaccines and immunization
• No preference- both oral injections are
  equally effective
• Supply is more/ date of expiry is near

39
Results
40
Qualitative Vs. Quantitative

• It is not qualitative vs. quantitative but
  qualitative and quantitative
• Mechanism and dynamics of events captured by
  qualitative methods
• Range of possible answers expanded through
  qualitative methods and then relative
  significance determined by quantitative methods
• Combination of methods make the results as well
  as recommendations more meaningful and
  operational

41
Concluding Remarks
Evaluators express their opinions explicitly -
Based on evidence gathered - Consistent quality
assurance measures - Limitations of methods
accepted up front - Politics of evaluation -
Remains a scientific endeavor - Efforts to
overcome barriers (keep in mind the perspectives
perceptions of the stakeholders)
42
An IndiaCLEN Program Evaluation Network 2003-04
Network Partners(n84)
Medical Colleges 69 NGOs 9 Pub Health Inst
6
43
IndiaCLEN Program Evaluation Network IPEN
VISION Facilitate development and implementation
of people friendly, effective Public Health
Programs that are in harmony with
their socio-cultural beliefs, thus creating a
milieu where service providers get motivated
and communities consider Public Health
Services beneficial and willingly participate