Evaluating Media Assistance Programs: What We Have Done and What We Have Learned

1
Evaluating Media Assistance ProgramsWhat We
Have Done and What We Have Learned

• Lee B. Becker Tudor Vlad

2
Background

• University of Georgia
• Grady College of Journalism Mass Communication
• James M. Cox Jr. Center for International Mass
  Communication Training and Research
• Conducts training programs
• Focuses on evaluation

Athens and the university in 1840 as depicted in
the painting by George Cooke.
3
My Research Interests

• Media effects
• Opinion and learning
• Organizational behavior and message construction
• Characteristics of workers, including educational
  credentials
• Sociology of education and labor markets
• Needs assessment
• Evaluation of impact

4
Evaluation of the Knight International Press
Fellowship Program

• Interviewed 531 persons in 8 European and 3 Latin
  American Countries
• At 31 people in each of the 11 countries
• Field work completed in 1999.

5
Basic Distinction

• Assessing and monitoring program process.
• Measuring and monitoring program outcomes.

6
Locus of Impact

• On journalists
• Attitudes
• Behaviors
• On organizations
• On media system
• On society overall

7
Problems with Self-Reports

• Timing is important
• Immediately after training is too soon
• Duration of effect is variable
• Likely to overestimate effect
• To satisfy the program provider
• To justify own investment of time and effort
• Lacks reference
• Most participants in training on projectile to
  change
• Participant may not know the answer

8
Journalists in Philippines

• Overestimated the amount of coverage they gave to
  the issues behind the conflict.
• Underestimated the amount of coverage of the
  Manila media to issues other than the conflict.

9
Measurement Solutions

• Go from the general to the specific
• Now I want to start with a very general
  question. Please tell me some of the things that
  you learned from your participation in the XXX
  Fellowship that you consider at this point to
  have been most important to your work as a
  journalist. Of course, if you didnt learn
  anything just tell me that.
• OK. Now Im going to ask about some specifics.
  You may already have touched on some of these,
  but I want to make sure I cover everything. Did
  your participation in the XXX Fellowship help you
  develop your skills in findings sources for
  stories you have reported since you returned?
• Use what we can a jab and probe questioning
  technique.
• Are there specific sources that you came into
  contact with through the XX Fellowship that you
  still use today?
• IF YES Who are some of those?

10
Validation Measures

• Interviews others who also can attest to
  attitudes, behaviors and changes in both
• Colleagues
• Supervisors
• Examples
• Since you completed the XXX Fellowship, have you
  held any brown bag or other information
  discussions about your experiences in the program
  with other journalists?
• As far as you know, has (NAME OF FELLOW) made any
  efforts to share her/his experiences at the
  Fellowship with others at your organization since
  she/he returned?

11
Design Solutions

• Treat timing as a variable
• We have compared short-term programs conducted
  across time
• Timing matterseffect several years
• But topic matters too and can offset lag
• Develop a control group
• True experimental design isnt often possible
• Imperfect control better than none
• Develop before and after design

12
Some Control Examples

• Control group made up of matched respondents
• Successful applicants for subsequent ongoing
  programs
• Those who work in similar settings
• Can match based on lists
• Can get respondent to help create a match
• Control groups made up of matched organizations
• Best control is the group itself
• Before and after has limitation of systemic change

13
Post-Employment Training Common Part of Media
Assistance

• When evaluating journalism training component of
  media assistance, keep in mind
• Measurement problems
• Design problems
• Use tactics to overcome them

14
Central Concerns of Media Assistance

• Concern with Media Freedom or media independence
• Concern with media performance
• Concern with the connection between Media Freedom
  and media performance
• Concern with the relationship between media
  performance and democracy

15
Mapping Media Assistance

• Effort by Monroe Price and colleagues
• Our purpose was to expand that
• Shifted to narrower short-term goal
• Rough estimate 1 billion year spent on media
  assistance each year
• Ultimately, valuable to understand media
  assistance this way

16
Two Actor Groups

• Media assistance community
• Donors
• Media assistance providers
• Targets of media assistance
• Media monitors
• Academic community
• Political scientists
• Communication scientists

17
Academic Community

• Political scientists
• Extensive literature on democratization
• Extensive debate on meaning of democracy
• Communication scientists
• Extensive literature on media freedom
• Extensive debate on meaning of media freedom
• Relatively little empirical testing of
  contribution of media freedom to democracy or
  democratization

18
Different Perspectives of Advocates of Media
Assistance and Political and Communication
Scientists

• Former believe media assistance leads to
  democratization
• Role of the research is to document this
• Scientists think all of these questions are open
  to examination

19
Model of Impact of Media Assistance
Media Assistance
Trained, Skilled, Motivated Journalists
Media Organizations that Facilitate and
Distribute the Work of Journalists
Information Needed for Functioning of Democracy,
Open Economy
Independent Media System
Functioning Democracy, Economy
?
Institutions of Civil Society
Civil Society Assistance
20
Model of Impact of Media Assistance
Media Assistance
Trained, Skilled, Motivated Journalists
Media Organizations that Facilitate and
Distribute the Work of Journalists
Information Needed for Functioning of Democracy,
Open Economy
Independent Media System
Functioning Democracy, Economy
?
Institutions of Civil Society
Civil Society Assistance
21
Does Media Assistance Lead to Trained, Skilled,
Motivated Journalists?

• Yes, probably
• Evaluation is spotty
• Process evaluation rather than impact evaluation
• Often rely on self-reports of impact
• Control groups are rarely used
• Evaluations often not independent of funder,
  media assistance provider

22
Model of Impact of Media Assistance
Media Assistance
Trained, Skilled, Motivated Journalists
Media Organizations that Facilitate and
Distribute the Work of Journalists
Information Needed for Functioning of Democracy,
Open Economy
Independent Media System
Functioning Democracy, Economy
?
Institutions of Civil Society
Civil Society Assistance
23
Does Media Assistance Lead to Facilitative Media
Organizations?

• Maybe
• Most evidence is indirect, based on observations
  of participants
• Designs are inadequate
• Many examples of failed investments
• Problems of sustainability paramount

24
Model of Impact of Media Assistance
Media Assistance
Trained, Skilled, Motivated Journalists
Media Organizations that Facilitate and
Distribute the Work of Journalists
Information Needed for Functioning of Democracy,
Open Economy
Independent Media System
Functioning Democracy, Economy
?
Institutions of Civil Society
Civil Society Assistance
25
Does Media Assistance Produce Independent Media
Systems?

• Some limited concrete evidence it does
• Comparative study by Steven Finkel and colleagues
  for USAID
• Found that USAID Media investments across 165
  countries led to gains in media freedom
• Used Freedom House indicators
• Considerable controversy over what is meant by
  media freedom

26
Model of Impact of Media Assistance
Media Assistance
Trained, Skilled, Motivated Journalists
Media Organizations that Facilitate and
Distribute the Work of Journalists
Information Needed for Functioning of Democracy,
Open Economy
Independent Media System
Functioning Democracy, Economy
?
Institutions of Civil Society
Civil Society Assistance
27
Do Independent Media Produce Information Needed
for Democracy?

• Almost no evidence either way
• No real explication of what information is needed
  for democracy

28
Model of Impact of Media Assistance
Media Assistance
Trained, Skilled, Motivated Journalists
Media Organizations that Facilitate and
Distribute the Work of Journalists
Information Needed for Functioning of Democracy,
Open Economy
Independent Media System
Functioning Democracy, Economy
?
Institutions of Civil Society
Civil Society Assistance
29
Do Independent Media Produce Civil Society
Institutions?

• No one knows
• Mixed evidence of the effectiveness of civil
  society assistance as well

30
Model of Impact of Media Assistance
Media Assistance
Trained, Skilled, Motivated Journalists
Media Organizations that Facilitate and
Distribute the Work of Journalists
Information Needed for Functioning of Democracy,
Open Economy
Independent Media System
Functioning Democracy, Economy
?
Institutions of Civil Society
Civil Society Assistance
31
Do Independent Media Foster Democratization?

• Surprisingly little empirical evidence has been
  gathered
• No understanding of what conditions might be
  placed on this relationship

32
Six Hypotheses

• The media-supremacist position, which holds that
  media freedom and independence produce democracy.
• The democracy-supremacist position, which holds
  that democratic reform determines and produces
  media freedom and independence.
• The media-freedoms-are-an-element-of-democracy
  position, which argues that media freedoms are a
  part of democracy and, as such, have no causal
  force leading to democracy.
• The null-effect position, which holds that there
  is no relationship between media freedom and
  democracy.
• A media-freedom-hinders-democracy argument.
• A democratization-hinders-media-freedoms stance.

33
Rozumilowicz Stages of Media Reform Linked to
Stage Theory of Democracy
34
Media Democracy Link
35
What We Know About Media Freedom Measures?

• Strong evidence of reliability across time
• Strong evidence of reliability across measures
• Some evidence of validity
• Criterion
• FH measures reflected changes in Warsaw Pact
  countries
• Construct
• IREX MSI related in predicted way in analysis of
  impact of hypercompetition on press performance

36
(No Transcript)
37
Relationship between Market Competition and Press
Performance
38
Limitations of Media Freedom Measures

• Focus on structure of media system
• Limited evidence of performance
• Limited notion of media independence
• Focus on independence from government
• Do not recognize pressures of commercial
  dependence
• Largely ignore audience
• Largely ignore needs of democracy

39
Additional Monitoring

• Develop checklist of needs of democracy
• A democratic state is a state in which all
  citizens have access to information about how the
  state operates.
• A democratic state is one in which citizens have
  the ability to communicate to each other.
• A democratic state is one in which conflicts are
  managed without resort to violence.
• A democratic state is one in which
  representations of members of the state are
  presented in a way that foster appreciation and
  understanding.

40
Relationship between Press Freedom and Confidence
in Media
41
Media Assistance
Trained, Skilled, Motivated Journalists
Media Organizations that Facilitate and
Distribute the Work of Journalists
Information Needed for Functioning of Democracy
Independent Media System
Functioning Democracy
?
Institutions of Civil Society
Civil Society Assistance
Model of Impact of Media Assistance
42
(No Transcript)