Evaluation for the People

1
Evaluation for the People

• Research Methods for Community Change

2
Reassessing the Pyramidal Structure of Evaluation
Research

• The typical structure of evaluation is quite
  pyramidal, with interlocking directorates of
  funding organizations, government institutions,
  and academic evaluators imposed on lower levels
  of service agencies, bureaucrats, and community
  members
• The top imposes programs on the bottom, which
  implements projects to fulfill the program
  mandates

3
Programs and Projects

• Programs are comprehensive social change systems
  that are generally broad in scope with long term
  goals
• LBJs Great Society Program
• Projects are delimited implementation of
  specified program goals usually bounded in time
  and space by statute
• The Texas Workforce Commission or more
  specifically, the Upper Rio Grande Workforce
  Development Board

4
Evaluation on the Pyramid Scheme

• Evaluation is rarely done of entire programs, it
  is commonly found a the level of projects
• Elites dont mind evaluating others so long as
  they are not evaluated in turn
• As a practical matter, even socially aware
  researchers must comply with this unstated basic
  structure of evaluation research

5
Navigating the Political Landscape Anticipate
or Be Tackled

• Keep the following basic questions about
  political opportunity structures in mind when
  proposing to engage in policy evaluation and
  policy formation as an SMO member
• How is formal power distributed?
• Power brokers
• Linkage figures
• How is informal power distributed?
• Power brokers
• Linkage figures
• What are the relations between formal and
  informal power structures?
• Which is more powerful?
• Which is your SMO better connected to?
• What are the controversial sore issues?
• Degree of polarization
• Elites
• Public
• Where does the project fit into the communitys
  political cleavage structure?
• Will it build bridges or dig ditches?

6
Taking Research From the Activists Perspective

• To many activists, the groundwork that they do to
  prepare for a successful action campaign is not
  academic research
• This view is slightly erroneous

7
Old Research Paradigm

• Classical academic research is to be value
  neutral
• The researcher is to have no horse that he/she
  is backing
• This is to avoid slanting questions, design, and
  methods to get desired answers

8
Post Modern/Realistic Research Paradigm

• No one is value free. The best we can hope for
  an open accounting of the researchers biases
• This means that there is no reason that a frank
  and open activist cannot do quality research
• There is really no such thing as in too deep,
  unless the researcher decides to not reveal how
  deep they are in

9
Intensive and Extensive Approaches

• Intensive case study/qualitative approaches
  concentrate in detail on one or a few examples of
  a phenomena
• Extensive survey/quantitative approaches try to
  discern general patterns of behaviors or events

10
Intensive

• The goal is really to get deep enough into the
  situation to derive causal predictions
• By getting into intense scrutiny, we try to
  eliminate alternate explanations for a an event
• With enough scrutiny, we also can become quite
  familiar with possible history and maturation
  processes that extensive analysis tends to gloss
  over
• The problem with this approach is that findings
  tend to be idiosyncratic (unique to the case)

11
Extensive

• The goal here is generalizability based on causal
  hypotheses gained from initial intensive studies
  
• By dealing with repeated observations over space
  and time, we can make a case for the observed
  connection being a causal relationship or law of
  human behavior
• The problem with such studies is that findings
  rarely investigate extensively multiple lines of
  causality
• One line is preferred and any others are by
  default either used as controls or are not even
  investigated

12
Project-Based Research Cycle

• The project-based research cycle that Stoecker
  (2003) uses is akin to the logic described in
  Levins research cycle (see CRIJ 3300 powerpoints)

13
The Goals are Deceptively Simple

• Engage community members in evaluation
• Diagnose community needs and strengths
• Define potential solutions that a community would
  find acceptable and consonant with community
  norms
• Use the community evaluators to help gain
  acceptance of the implementation of the selected
  solution(s)
• Evaluate achievement of objectives according to
  both sponsors and community clients

14
Finding Ones Place in the Cycle

• Diagnosis
• New services are in demand
• New problem exists, cause unknown
• General need to be in touch with community
  clients
• Strategic planning
• Prescription
• Finding best practices (common solutions) for our
  problem
• Seeking to decide if best practice(s) will apply
  to our community
• Seeking to efficiently mobilize SMO and service
  provider extant resources
• Implementation
• Moving aspect of SMO to the fore in aiding
  community
• Using POS (political opportunity structure, see
  Tarrow 1994) to get policy enacted
• Public
• Private (corporate)
• Evaluation
• Seeking to find if our project had an impact
• Seeking to determine if a change in strategy or
  tactics is needed to meet SMO goals and community
  needs
• Adapted from Stoecker 2005, p.76.

15
Who are These Community Persons?

• Staff and volunteers of local social movement
  organizations and movements
• The academic institutions and government
  institutions that house evaluation units
• Interested college students that either research
  the community out of a personal connection or are
  committed to evaluation projects as part of their
  degree
• Academic researchers who are engaged in community
  service or contract-research with sponsors of
  community affecting projects
• Funding organizations
• Service providers that do not house evaluation
  units
• General community members who come in for a wide
  variety of personal reasons

16
Who are These Community Persons? (contd)
17
Staff and Volunteers of SMOs1

• These folks are the foot soldiers of project
  based evaluation the civil society version of
  street level bureaucrats
• They have the history and maturation knowledge
  essential for good research design and
  implementation

1 SMO Social Movement Organization, the general
class of organizations that community
organizations belong to.
18
College Students and Faculty

• These are the specialists or guns for hire
• They have the methodological expertise to merge
  with the case expertise of SMO members to
  complete the designs and implementation plans
• Keep in mind, the hero of Have Gun Will Travel
  was not a callous fellow. He was actually an
  idealist. Just because you are a methodological
  expert does not mean that you are aloof

19
General Community Persons

• These folks add perspective
• SMO members tend to be a bit overcommitted at
  times
• Academics may not know enough facts on the ground
• Thus incorporating additional views can be a real
  bonus, if done with a view to improving the study
  design

20
Steps for Doing Project-Based Research

• Choosing questions
• Designing the methods
• Data collection
• Data analysis
• Reporting findings

21
Choosing Questions

• The selection of what to ask is always a
  negotiated process between stakeholders
• The difference with project-based research is the
  relative emphasis one puts on the stakeholders
• Here community stakeholders take moral (if not
  actual) priority over financial and institutional
  stakeholders
• Theoretically the people matter most, only after
  their needs are addressed do the considerations
  of finances and legal mandates come into play

22
Designing the Methods

• Design can be top-down or bottom-up
• Top-down prefabricated designs are placed in
  many contexts to insure comparability and
  reliability
• Bottom-up using community input contextually
  relevant designs are employed to insure validity
  and consensual implementation of studies

23
Data Collection

• A personal favorite
• I like to use students to do phone surveys and
  intercept surveys. The students generally belong
  to the El Paso community
• This increases their ability to converse in a
  linguistically appropriate manner with
  respondents
• I would be abysmal at doing such interviews
  myself and hiring folks from even a prestigious
  firm like Princeton Research Associates would not
  make situation much better

24
Data Analysis

• Community contact brings nuance to the analysis
• Non-SW US surveyors rarely understand the ire
  they raise when not considering Latino/Hispanic a
  race
• History and maturation processes for time series
  analysis are fundamental
• The totally bungled testimony by ASARCOs expert
  witness on the health impacts of reopening the
  plant foundered on the out of towners ignoring
  the impacts that the plant would have on Cd.
  Juárez and that Juárenses would make it an issue
  at an El Paso hearing

25
Reporting Findings

• While the word of an impartial judge is the
  epitome of legal jurisprudence, it does tend to
  lack for the persuasive impact one might desire
• Thus the classical academic discourse of findings
  lacks as Israelis say, Zazooah or Ooomph
• Having a trusted community member express
  findings that are counter to their known
  positions on an issue is both highly credible and
  attention-getting
• The challenge is getting a reputable community
  member to express such counter-ideologicals
• The best way to do this is to have credible
  community members from all sides involved in the
  entire evaluation process so that they buy into
  the end products

26
So How Do We Bring the Community in?

• Try to be useful
• Use a multimethod evaluation style
• Listen as well as talk, learn as well as teach

27
Useful

• Not every evaluation need be stripped down to
  only the quintessential nuts and bolts of
  measurement and data analysis. In fact, to do so
  is generally counterproductive
• Like in all genuine forms of politics, logrolling
  is an essential practice
• Trade of services as part of the evaluation,
  provide some community needed functions
• Help organize community archives
• Help generate donor lists

28
Multimethod Evaluation

• Despite my desires to the contrary, not all of us
  are statisticians at heart nertz!
• Thankfully, nor are all of us folklore
  specialists like my sister-in-law
• We need both extensive/quantitative and
  intensive/qualitative methods

29
Multimethod Evaluation (contd)

• As my favorite Aesops fable goes, the ability of
  three blind men to describe a elephant/pachyderm
  depended on two things
• Touching multiple sites of the big tusks
  (elephant)
• Coordinating findings about the wrinkly skinned
  (pachy-derm) critter
• So do our evaluations

30
Dont Be Your Professor or a College Freshman

• Dr. Levin is a notorious talker and a lousy
  listener
• College freshmen are notorious for not actively
  engaging in classroom discussions
• Genuine collaboration needs a maximization of all
  parties to both listen and talk

31
What to Think About When Collaborating with
Community SMOs

• SMO skills
• SMO financial and personnel capacity
• Compatibility of evaluation with SMO goals

32
SMO Skills

• By default, SMOs ought to be able to contribute
• Community member contacting expertise
• Street credibility
• In addition, SMOs may be able to contribute
• Archives
• Funder/donor databases

33
SMO Financial and Personnel Capacity

• Clerical staff
• Surveyors
• Data entry specialists
• Historians
• Morale boosters never underestimate the impact
  of a warm relationship on research productivity

34
Compatibility of Evaluation With SMO Goals

• Try to limit the input on evaluation goals to
  consensus goals that SMOs from several sides can
  agree on
• This requires you to be slightly vague
• This does not mean you lie!
• This means keeping the goals to what is to be
  evaluated. Refrain from goals about what is to be
  done with the findings

35
Flipping the Coin to the Other Side SMO
Questions

• Can the academic/institutional evaluator listen
  to the SMO when needed?
• Can the academic/institutional evaluator keep
  deadlines?
• Can the academic/institutional evaluator speak to
  the SMO in its own language?
• What credentials does the academic/institutional
  evaluator have?

36
Can the Evaluator Listen?

• Despite Dr. Levins notoriety, the Planned
  Parenthood Center of El Paso decided to take the
  plunge.
• Since that decision, Dr. Levin has learned a new
  vocabulary of three letter words
• If the formal evaluator cant listen, the SMO
  should drop the relationship
• Otherwise they risk having words put in their
  mouth

37
Can the Evaluator Keep Deadlines?

• In the period between retiring from the federal
  service and prior to starting at IPED, I have had
  nothing but deep schooling in the inability of
  academics to keep schedules
• It is a norm for academics to
• Come late
• Turn in reports late
• Turn in budgets late
• Never write contracts out fully
• Solution 1 work only with academic research
  units
• They are built to work with federal and state
  budgeting and grant application schedules
• Solution 2 work with private research units
  that share interests with your SMO
• They too generally work on business schedules

38
What Credentials Does the Evaluator have?

• Credentials are a useful way to separate out the
  wheat from the chaff, but have one serious
  deficiency young talents/institutions have
  limited credentials
• Get as much input from the evaluators colleagues
  on the evaluators abilities as possible

39
Tackling the Four Aspects of Project-Based
Research

• Diagnosis, Prescription, Implementation, and
  Evaluation are all worthy of discussion on their
  own. Thus the remainder of this lecture, like
  the Stoecker text, will address them separately

40
Diagnosis

• Forming a core group
• Needs assessment
• Asset mapping
• Melding needs and assets

41
Forming a Core Group

• All social policy projects, like most complex
  games, require teams. Project success hinges on
  building a group that can
• Survive mild degrees of internal conflict
• Has enough breadth of external contacts to build
  allies and exploit POS
• Either intentionally links in with the existing
  issue arena institutional structures or
  intentionally decides to work around the
  existing issue arena institutions

42
Needs Assessment

• As discussed last week, needs assessment is a
  complex form of evaluation
• In this respect, I differ with Stoecker, who
  views only outcome evaluation/impact assessments
  and process evalutations as evaluation

43
Extensive Needs Assessment

• Extensive needs assessment
• This is essentially what we reviewed last week
• Surveys
• Census data
• SWOT analysis

44
SWOT Analysis

• Survey of Strengths, Weaknesses, Opportunities
  and Threats
• This is a focus group approach
• Bring in core stakeholders for a few 1-2 hour
  discussion sessions to
• Discussion 1 Generate a list of SMO or
  community successes and failures within a
  relevant time frame
• Discussion 2 Generate a list of SMO or
  community strengths and weaknesses within a
  relevant time frame
• Discussion 3 Maximize strengths and avoid
  weaknesses by utilizing POS of issue arena

45
Intensive Needs Assessment

• Beyond the quantitative assessment of community
  needs from surveys or census data exist the views
  of informed stakeholders
• The basic SWOT method does not address the
  perceived depth of needs, polarization on goals,
  etc
• Intensive assessments will add greater emphasis
  in SWOT to building lists of ranked priorities
  for needs
• Weaknesses and threats will be prioritized on
  which need to be addressed most

46
Asset Mapping

• Some of us find a glass of water half empty (your
  Prof.), others find the same glass to be half
  full (his pal Marika). It takes both
  perspectives to realize what real potential the
  glass holds
• Asset mapping is all about seeking to know what
  resources the general community, relevant SMOs,
  and service providers have as opposed to what
  they need
• The goal behind asset mapping is to sensitize
  communities to their own potential to solve
  community issues without government largesse
• Appropriate tools include surveys and census data

47
Melding Needs and Assets

• In truth, the best approach to a diagnosis of a
  body politic is the same as of a body you need
  to know both what is healthy and what hurts
• In the course of plotting out what hurts, you
  know what might be healthy
• In the course of plotting out what is healthy,
  you know what you can rely on to deal with what
  hurts
• It is much more sensible to approach a community
  diagnosis with a mind to seek to find the gap
  between internal needs and assets. This will
  avoid the overstatement of community and SMO
  helplessness that needs assessment often encouages

48
Prescription

• Service versus policy
• Finding alternatives
• Evaluating alternatives
• Choosing an alternative C-B/C-E Analysis

49
Service Versus Policy

• Service
• Inward-focused on community
• Concrete plans
• Narrow application to a specific goal and context
• Can have policy changes in it
• Think of this as an experiment in Kuhns (1970)
  sense

• Policy
• Outward-focused on setting agenda for best
  practices solution to problem
• Abstract rules that service prescriptions will
  fill in
• Wide application
• Think of this as a paradigm in Kuhns (1970) sense

50
Finding Alternatives

• To come up with a vision that is broad,
  long-range, and has substantive meaning
• Dig in deep in the existing literature
• Use academic books, online academic journals, and
  research websites
• Work with trade publications
• In many service professions the federal
  government sponsors trade publications. So do
  unions. Find the union websites and you will
  find the publications
• Ask friends in the field for their experiences
• We all know someone who knows someone. Screw up
  your courage and ask around
• Brainstorm with stakeholders
• This is another focus group approach
• Beware of groupthink!

51
Evaluating Alternatives

• Seek the field recognized best practices
• Often the trade publications are better sources
  for overall assessments of what works and what
  are duds
• Pull evaluation criteria from prior formal
  evaluation studies
• While specific criteria are not always available
  for a new issue, the basic methods (C-B analysis)
  and measures (, QLI, etc) are very adaptable
• If you have access, convene a panel of experts
• Add specific standards from the theoretical
  literature
• Bring in the community context
• Different communities have different baseline
  conditions, what most folks define as poor in El
  Paso is really poorer than what most folks
  define as poor in Chicago
• This will also help to establish the practical
  limits of the expected impacts
• Bring in the projects stated goals
• Seek the stakeholders input for measurement
  criteria
• After all, it is their lives that are impacted
  the most
• Expect this input to be political in nature
• This can help to integrate dissenting groups
  early in the process though

52
Choosing an Alternative

• Rank criteria
• When resources are scarce, the easiest method is
  bottom-line priority
• Other methods are paired comparisons and Q-sorts
• Calculate benefits and costs
• We will go into this in detail with Levin and
  McEwan (2000) later this semester
• Do forecasts
• We will look at forecast modeling next week as
  part of the REMI and REDYN models
• Do decision tree analysis
• We will cover this in Ch. 12 of Stokey and
  Zeckhauser (1978) during November

53
Implementation

• Research as action
• Community research
• Target research

54
Research as Action

• Organizing the community on its own behalf
• The goal here is to use the project to bring the
  community to support SMOs and service providers
  Help us to help you
• Getting Hudspeth residents to support their
  public libraries
• Knowledge for its own sake
• To get a community more aware of its own problems
  so as to address them more effectively
• Educate an AIDS susceptible community about safe
  sex the PPCEP-IPED project for the CDC

55
Community Research

• History
• Most communities have persons and local archives
  the Rogers Park Historical Society has a great
  photo of some 1900s farm horses fertilizing what
  would become Davids childhood backyard
• Living documentaries
• The communitys knowledge embodied via plays,
  home movies, exhibitions, local cable TV shows
• Community media
• Local newspapers, the internet, and local cable
• GIS, ARCView, and Mapping the Neighborhood
• The National Institute of Justice pushes crime
  mapping heavily on us poor criminologists
• NIJ loves to fund GIS based research in to the
  epidemiology of crime

56
Target Research

• This is where you research an organization or
  policy in order to target it as needing to be
  changed by public policies
• This is an essentially confrontational method

57
Target Research (contd)

• Types of Targets
• Government (agencies)
• Texas Women, Infants, and Children Program help
  service website speeds
• Corporations includes financial corporations
• Wells Fargo lending practices, ASARCO
• Nonprofits
• Boys and Girls Club in NE El Paso embezzlement
  dispute
• Foundations
• Kaiser Foundations pattern of choices in
  spending money on health care issues in the US
  tends to avoid questions about reassessing the
  valuation of medicines, but tends to push
  questions about supplementing purchaser ability
  to pay in the scale of things, this is not a
  great sin, but it is a needless limitation on
  research

58
Government

• Governments are the typical targets of research
  since they are the primary source of service
  provision and evaluation funds
• These evaluations can range from purely pro-forma
  validation to highly investigative
• A key tool for such research is the federal
  Freedom of Information Act (FOIA) and its state
  and local equivalents
• You can open most government records upon
  request, if the Homeland Security bills have not
  closed them off

59
Corporations

• Corporations are another typical target
• Their actions are often traceable through
  stockholder reports, Equal Employment Opportunity
  Commission complaints, National Labor Relations
  Board complaints, local Better Business Bureau
  complaints, and financial area specific databases

60
Evaluation

• External versus participatory
• Outcome versus process
• The participatory approach

61
Evaluation

• As we discussed last week, evaluation can come in
  many forms, from early stage needs assessments
  and formative evaluations to the more end stage
  process evaluations and impact assessments
• Stoecker shows a preference here for outcome
  evaluations/impact assessments, but I think this
  has more to do with the organization of the text
  than that he rejects needs assessment as being a
  forma of evaluation

62
External Versus Participatory

• External
• Controlled by someone not directly involved in
  the project to avoid the evaluator going native
• Use
• When project stakeholders are riven by factions
• When sponsors or agencies demand it
• When time is an issue
• When highly specialized skills are required

• Participatory
• Partly controlled by persons directly involved in
  the project
• Full control would really have a bias problem
  most of us really cannot self critique
• Use
• When there is a manageable consensus on goals and
  methods
• When SMOs and service providers truly want to get
  critiqued
• When SMOs and service providers have time and
  skills to commit to project evaluation

63
Outcome Versus Process

• Outcome
• Focus is on the net impact of a project on the
  welfare of a target community
• Tends to be holistic in evaluation measures
• Many measures will be combined to present a
  complete picture at a single point in the
  project the end
• No social project has a real end unless all
  subjects exposed and those they contact are dead
  e.g. a whole community is exterminated
• Will use standard array of social, economic, and
  biomedical indicators of well being income,
  education, leisure time, toxic content of air or
  soil, incidence of cases of disease etc

• Process
• Focus is on the steps towards reaching an outcome
• Can be incremental in evaluation methods
• Often multiple methods and points in the process
  will be examined and adjustments in both process
  and evaluation will be done on the fly
• Will use array of efficiency indicators
  service time, number of cases processed,
  complaints registered, etc

64
The Participatory Approach

• SMO members, service providers, and general
  community members can be fruitfully involved in
• Determining the question
• Choosing the methods
• Gathering the data
• Analyzing the data
• Presenting the data

65
Determining the Question

• While a specialist has a prefabricated set of
  questions to ask about a project, those most
  engaged in the context will be able to add
  specific questions or eliminate ones that are
  unlikely to show sufficient variance due to
  context
• The naysayers among the stakeholders may be
  co-opted at this stage by adding a question or
  two

66
Choosing the Methods

• Again, the specialist will have a prefab set of
  common indicators of project success
• Stakeholders will know which are practical to
  obtain, which are expensive but manageable, and
  which are simply boondoggles

67
Gathering the Data

• Like Dorothy once said Theres no data like free
  data or was that Theres no data like home
  cooked2 data? Anyway, the best data is that
  which is
• High fidelity
• Low cost
• Stakeholders who have an interest in getting
  useful evaluations will be the persons best in
  position to
• Access original source data
• Be/recruit volunteer labor

2This is not an instruction to cook (fake your
data). If you do this, I and every other
evaluator worthy of the title will hound you to
your grave for dishonoring our profession.
68
Analyzing the Data

• Herein lies the evaluators best chance to be an
  educator
• While participation in other stages may involve
  education, it tends to be more about alerting
  people to skills they already possess, not giving
  them new ones
• As a teacher of evaluation methodology for the
  MPA program and the Criminal Justice program, I
  can attest to you that the skills needed for
  being a skilled evaluator are in short supply in
  El Paso. You have a valuable commodity, but like
  most forms of education, it gains value by
  teaching your consumers how to use it and do
  rudimentary stages of it on their own
• Have the SMO members, service providers, general
  community members, and other interested
  stakeholders come in for
• Training on SPSS, SAS, STATA, etc
• A working knowledge of what the term
  statistically significant means
• Basic cost-benefit/effectiveness analysis
• Decision analysis
• Remember, the idea is to give them enough to
• Make them aware of the potential benefits to them
  from using evaluations
• Make them aware of the need to bring you or a
  colleague in for specialized analysis services
• Yes, I guess this is saying that we want you to
  get people addicted to evaluation

69
Presenting the Data

• Picture this a city budget analyst stands in
  front of a city council meeting and is roundly
  booed by all and sundry for giving them news they
  dont want to hear about programs that must be
  cut to meet the councils demand for a balanced
  budget without raising taxes. Sound familiar?
• Avoid! How? Simple! Bring the SMO members,
  service providers, and other stakeholders into
  the presentation process
• That way they are unlikely to boo, since they
  would be booing themselves
• They would be more understanding of the actual
  implications of the findings
• It allows findings to leak early yes, leaks can
  be good sometimes
• Policymaking subsists on strategic leaks to
  co-opt and demobilize opposition

70
Bringing it All Around Again

• Having touched on the high points of the
  project-based research cycle, lets be aware of
  some of its limitations
• It tends to be too darn inward focused if the
  specialist is edged out to the margins of the
  process
• It is the evaluation specialists job to look
  beyond the project to the program and to social
  welfare as a whole
• Participatory process tend to marginalize experts
  in the name of an ersatz average Jane/Joe
• It cannot and should not entirely insulate the
  evaluators or evaluation targets from criticism
• Not all projects are created equal. Lets face
  it, some projects stink and should be terminated.
  Someone has to take the heat for the flawed
  design, implementation, or less than kindly
  breaking of the bad news
• It tends to be antagonistic towards traditional
  service providers (e.g., government agencies)
• Sorry Neos of the world, but the government is
  rarely a Matrix enmeshed in extracting a
  parasitic existence from you
• Nevertheless, the project-based research cycles
  participatory emphasis is well-worth the risks
  and limitations enumerated above