Formative Research,

1

• Formative Research,
• Monitoring Evaluation for
• AI/HPI Communication Programs

2
Useful Resource Documents for this Section

• UNICEF Essentials for Excellence Research,
  Monitoring Evaluating Strategic Communication
  in the Prevention Control of AI/PI
• http//www.jhuccp.org/asap/file.php/1/UNICEF/UNICE
  F_Essentials_for_Excellence_Researching_Monitoring
  _and_Evaluating_Strategic_Communication_in_the_Pre
  vention_and_Control_of_AI_PI.pdf
• WHO Planning Communication for Behavioral Impact
  (COMBI) Programs for Health Manual
• http//www.jhuccp.org/asap/file.php/1/WHO/WHO_Plan
  ning_Communication_for_Behavioural_Impact_COMBI_Pr
  ogrammes_for_Health_Manual.pdf

3
Research answers three basic questions

• Are we doing the right things?
• Are we doing them right?
• Are we making a difference?

Formative research Monitoring research Evaluatio
n research
4
Types sequence of research
Focus of Formative Research
5
(No Transcript)
6
(1) Role of Formative Research

• What should we do and how?
• Assess current situation, information needs, key
  issues
• Identify relevant behaviors for behavioral change
  communications (e.g., risk-taking, preventive and
  health seeking behaviors)
• Identify priority audiences for AI/PI behavioral
  change messages
• Identify existing preferred communication
  channels for AI/PI info
• Assist in message development (identify key
  issues, messages, concepts, terminology, cultural
  appropriateness, acceptability) pretesting
• Assists in developing indicators and baseline
  measurement

7
(2) Role of Monitoring

• How are we doing?
• Track inputs
• Money, staff, other resources
• Track processes
• Timing, reach, mobilization, dissemination
• Assess quality
• Comprehension, liking, initial reactions

8
(3) Role of Evaluation

• Are we making a difference?
• Measure change
• Behavioral, organizational and social
• Measure impact of communication
• Knowledge, attitudes and behavior, health impact,
  social interaction, community involvement, media
  content
• Assess effectiveness and efficiency
• Reach, cost

9
Semi-structured interview When to use At
baseline as well as during monitoring and
evaluation. With whom Conducted with key
informants such as rural poultry farmers, wet
market butchers and poultry sellers, child
care-givers, food store owners, government
officials, local health service personnel,
traditional healers, community leaders (elected
or self-appointed), religious leaders, and
members of nongovernmental organizations. Always
collect background information of respondents so
that you can characterize the people interviewed
and describe these details in the final
report. How sampled Non-randomly. Around 10-25
respondents per participant group rural poultry
farmers, wet market butchers and poultry sellers,
child care-givers, food store owners, government
officials, local health service personnel,
traditional healers, community leaders, religious
leaders, and members of nongovernmental
organizations. Maximum of 50-
(1) Methods for Formative Research

• Qualitative Methods
• Review of existing data and documents
• Observations (e.g., households farms clinics
  laboratories etc)
• In-depth interviews (IDI) (could include
  freelisting and pile sorts)
• Focus-group discussions (FGD)
• Trials of improved practices (TIPS)
• 24-hour recall DILO / MILO
• Quantitative Methods
• Survey (Probability random samples Structured
  questionnaire)

For more detail on these and other methods, see
UNICEF Essentials for Excellence and WHO Planning
Communication for Behavioral Impact (COMBI)
Programs for Health Manual
10
(2) Methods for Monitoring

• Implementation monitoring
• Bounceback cards
• Client satisfaction
• Tracking surveys (small sample)
• Media content analysis
• Monitoring policy changes
• Behavioral surveillance, observation

For more detail on these and other methods, see
UNICEF Essentials for Excellence and WHO Planning
Communication for Behavioral Impact (COMBI)
Programs for Health Manual
11
(3) Methods for Evaluation

• Single group, post-intervention only
• Single group, pre- and post-intervention
• Single group, time series
• Cross-section, pre- and post-intervention
• Intervention versus non-intervention, before and
  after
• Intervention versus non-intervention, time series
• Randomized control trial (rare)

For more detail on these and other methods, see
UNICEF Essentials for Excellence and WHO Planning
Communication for Behavioral Impact (COMBI)
Programs for Health Manual
12
Eight Criteria for Causal Attribution

• 1. Change over time in the expected outcome is
  observed
• An association between that change and program
  exposure is
• observed (correlation)
• Exposure occurs before the observed change is
  measured. (time-order)
• No evidence of confounding variables that may
  have accounted for the change
• The observed change is abrupt and large
  (immediacy and magnitude)
• Impact increases in proportion to level or
  duration of exposure (Dose Response)
• A causal connection is justified (causal
  pathways and theoretical coherence)
• Consistency with previous program research
  (replication with variation)

13
Special casesMonitoring and evaluating advocacy

• Documentary review
• Changed policy
• Change in legislation
• Policy/legislation change implemented (and in the
  very long term)
• Changed funding, other resources
• Longitudinal population tracking
• Positive change in people's lives as a result of
  the policy/legislation change

14
Special cases Social Change

• Expanded public and private dialogue and debate
  
• Increases in Family peer discussion? Community
  gatherings? Coverage and discussion in news
  media? Debate and dialogue in the political
  process?
• Increased leadership role by people disadvantaged
  by the issues of concern
  
• Change in who makes the major decisions about
  communication interventions? Number and
  commitment of community leaders?
• Increased involvement of and coordination among
  groups
• Which groups are involved? Have they been linked
  together? How does that linking take place? Is
  there an alliance? How does the alliance work?

Figueroa, M.E. Kincaid, D.L. Rani, M. Lewis,
G. (1999) Communication for Social Change An
Integrated Model for Measuring the Process and
Its Outcomes. NY Rockefeller Foundation.
15
Qualitative and quantitative approaches
16
Are they compatible?
Qualitative phase
Quantitative phase
Qualitative phase
DATA

• Community engagement
• Insights into why
• Non-probability samples
• Sample definition
• Question testing

• Insights into how much, how often
• Probability samples
• Hypothesis testing
• Variable analysis

• Explanation of results and follow-up of unusual
  cases
• Community feedback and verification

Graphs Statistics Tables Quotes Pictures
Stories Diagrams Interpretations Conclusions
17
Qualities of Good Indicators

• Measurable (quantitative and qualitative)
• Reliable (consistent results)
• Valid (measure what intend to measure relevant
  sensitive acceptable)
• Simple (and affordable)
• Specific
• Achievable/realistic
• Accessible (e.g. in MIS), reported, used

18
Key Indicators of Outputs and Outcomes for AI/PI
Communication

• Number of AI materials produced, by type, during
  reference period
• Number of AI materials disseminated, by type,
  during reference period
• Percentage of target audience exposed to AI
  program messages, based on respondent recall
• Percentage of target audience who correctly
  comprehend a given AI message
• Percentage of target audience expressing
  knowledge, attitudes, and beliefs consistent with
  AI messages
• Percentage of target audience who acquire the
  skills recommended by the AI messages
• Percentage of target audience who discuss AI
  messages with others, by type of person
• Percentage of target audience who engage in
  recommended AI protective practices

19
Steps in development of a research plan

• Define research objectives
• Research helps you answer program decision-making
  questions. What do you need to know?
• Develop research questions
• Determine key information that needs to be
  gathered to plan interventions and develop
  messages, identify problems, information needs,
  formulate hypotheses, and identify key variables.
• Questions should be linked to theory and project
  objectives.
• Select the appropriate research methods, set
  timeframe
• (develop budget develop research
  instruments)
• Choice of methods based on research questions and
  purpose of study specific qualitative methods
  quantitative methods or mix?
• Select the Sample
• Choose study population groups choose categories
  of participants select sites and determine
  sample type and size
• Select and train the Field Research Team
• Plan Logistics
• Implement the Study

20
Sample topics for research

• General awareness of AI
• Specific Knowledge about H5N1 AI virus, modes of
  transmission, AI symptoms in poultry and in
  humans AI preventive measures for poultry and
  for humans
• Background characteristics of priority groups
• Attitudes about AI causes, potential health and
  economic consequences of AI/PI, government
  compensation for lost/culled flocks
• Perceptions of personal AI risk, AI risk of
  poultry stock pandemic influenza risks
• Barriers to adopting safer practices to prevent
  AI in poultry and in humans
• Misperceptions and rumors about AI causes,
  symptoms, modes of transmission and prevention
  measures

21
Sample topics for research (cont)

• Risky behavior and practices for AI transmission
  in poultry in humans among rural residents
  backyard poultry raisers
• Bio-security practices at commercial/industrial
  poultry farms, medium and small poultry farms
  unsafe practices existing bio-security
  regulations and actual practices
• Main Channels of information about AI preferred
  sources and information needs media exposure
  social mobilization
• Role of journalists in communication about AI/PI
  issues
• Private Sector involvement role in AI/PI
  prevention and preparedness
• AI risks to children and protective measures
• Health seeking behavior in rural urban areas
  (e.g., seasonal flu care opportunities/barriers
  to AI/PI prevention, treatment).
• Issues and opportunities for social mobilizations
  in communities, associations, social networks
  for AI/PI prevention response
• Organizational perspectives on AI/PI (health
  agriculture ministries, NGOs, associations,
  donors coordination support communication
  preparedness response).

22
Some Priority Groups for Formative Research

• Backyard poultry raisers, rural household heads
• Policy makers, community leaders
• Government Ministries of Health and Agriculture,
  Veterinary inspectors, health/epidemiological
  surveillance workers
• Emergency/civil defense workers border/customs
  officials
• Poultry Farmers (small, medium, large commercial)
• School teachers students Children-home
  setting General population (urban and rural
  areas)
• Journalists
• Poultry transporters, poultry breeders, poultry
  slaughter houses, Poultry food processing factory
  managers/workers, meat market vendors
• Health/Agricultural NGOs and Associations,
• Other groups (Hunters, zoo animal keepers, food
  preparers, tourists/tourist bureaus,
  international donors, etc.)

23
Group Exercise AI Formative Research Proposal

• Develop 1-2 page Formative Research proposal to
  support AI Communication Strategies of countries
  in your group
• (Focus on questions around your key action)
  (45 minutes)
• Follow Steps for designing of Formative Research
  Study and Identify
• Research objectives of the FR
• Main research questions/issues to be studied
• Research methods to be used to collect data
• Identify priority population groups and possible
  study sites
• Identify key variables (knowledge, attitudes,
  behaviors, risks, key socioeconomic, demographic
  variables)
• Specify how the research links to and supports
  the National AI Plan, the Communication Strategy
  and Communication Objectives in the countries of
  your group
• Report your proposal in Plenary

24
Selected References
(some materials for presentation extracted or
adapted from several sources listed
below) Cabanero-Verzosa, Ceceilla. 2003.
Strategic Communication for Development Projects
A Toolkit For Task Team Leaders. World Bank.
Figueroa, M. Bertrand, J., Kincaid, D. 2001.
Evaluating the Impact of Communication Programs.
Measure Evaluation Project and PCS Expert
Meeting. Figueroa, M.E. Kincaid, D.L. Rani,
M. Lewis, G. (1999) Communication for Social
Change An Integrated Model for Measuring the
Process and Its Outcomes. NY Rockefeller
Foundation. Flanagan, D, Mahler H. and Cohen, S.
1994. How to Conduct Effective Pretests Ensuring
Meaningful BCC Messages and Materials.
FHI/AIDSCAP. JHU/CCP. 2007?. Monitoring and
Evaluation of Health Communication Campaigns. Pp.
1-4. Kincaid, D. 2004. Measuring Impact of
Entertainment-Education Programs.
JHU/CCP Matveev, Alexi V. 2002. The Advantages
Of Employing Quantitative And Qualitative Methods
In Intercultural ResearchBulletin of Russian
Communication Association Parks, Will. 2006.
Essentials of Excellence Researching, Monitoring
and Evaluating, Strategic Communication in the
Prevention and Control of Avian/Pandemic
Influenza. ASAP. UNICEF. Rossi, P., Freeman,
H. 1993. Evaluation A Systematic Approach. Sage
Publications.
25
Definitions
Monitoring is continuous and aims to provide
management and other stakeholders of an ongoing
program with early indications of progress or
lack thereof in the achievement of objectives.
Continual and careful monitoring of relevant
indicators and processes generates information
for evaluation and, more importantly, for
corrections that may be needed as a strategic
communication initiative unfolds.
26
Definitions
Evaluation is a periodic exercise that attempts
to comprehensively and objectively assess
progress towards and the achievement of a
programs objectives or goals. Unlike monitoring,
evaluation is usually resource intensive because
it should be comprehensive in its scope, using
many different forms of data collection and
analysis. Ideally, evaluations should measure
progress made against an initial baseline
situation. There are several research designs
that can help you determine the contribution that
strategic communication has made if changes are
detected in an endline evaluation compared to a
baseline. External evaluators who bring a degree
of objectivity should join the evaluation
exercise.
27
Key Indicators to Track Outputs and Outcomes of
Communications Programs

• Number of AI Communications Produced, by type,
  during reference period
• Number of AI Communications Disseminated, by
  type, during reference period
• Percentage of Target Audience Exposed to AI
  Program Messages, based on respondent recall
• Percentage of Target Audience who correctly
  comprehend a given AI message
• Percentage of Target Audience Expressing
  Knowledge, Attitudes, and Beliefs consistent with
  AI message
• Percentage of Audience who Acquire the Skills
  recommended by the AI message
• Percentage of Target Audience Who Discuss AI
  message with others, by type of person
• Percentage of Target Audience who Engage in
  Recommended AI Prevention Practices

28
UNICEF (2006) Essentials of Excellence
29
(No Transcript)
30
Asking and formulating questions It is important
to ask appropriate questions at the right time
and really listen to the responses. The way
questions are asked and the types of questions
asked can influence the answers given. It is
important to formulate questions that stimulate
rather than discourage discussion. Questions
should be tested and reviewed prior to being used
in formative research to ensure they are as open
as possible. Open-ended encourage answers that
go beyond one word. Disclose feelings, actions,
reactions, and knowledge. Examples include
Questions that begin with what, how, where
What do you think about? How do you feel about?
Can you tell me more about? Closed-questions
encourage short responses or yes/no answers. Do
not encourage respondents to expand on the
subject. Examples include questions that begin
with Do you, did you, how many? Two-in-one
questions create confusion because they force
respondents to react to two things at once, and
responses are usually difficult to analyze.
Example How many times do you feed your
chickens a day and what do you give? Biased or
leading questions lead respondents to say
something they may not have thought about and can
limit the responses. Probing Probing can be
achieved through Silence that gives the
respondent time to expand their thoughts A
question to follow up (Why?) A statement (I am
a little unclear about that) or restatement (You
said ) Repetition of key words in response to
what the talker has said (You said that is good,
but what is good about it?) Reference to
another person/group (What might other people
think about that?) Probing is important because
it allows for a deeper understanding of the
reasons behind a response. The key to successful
qualitative research is deep probing of issues
raised as respondents respond to the questions.
Try not to cover too many topics, or the
interview will become a survey with little
probing.