Community Level Instruments

1
Community Level Instruments

• Liz Robertson, NIDA
• Kristi Pettibone, MayaTech
• Shelly Kowalczy, MayaTech

2
NIDA SAMHSA Collaborations

• State Research Capacity Building RFA 2005
• Community-based Organizations Capacity Building
  RFA- 2006
• SPF SIG Cross-site Evaluation 2004-2009
• And others .

3
Why the SPF-SIG Cross-site Evaluation?

• Excellent match between the missions of NIDA and
  SAMHSA
• Largest effort to evaluate a Federally funded
  prevention effort ever!
• Draws on the science base by recognizing the
  importance of identifying key problems and key
  problem areas in allocation of resources

4
Important Considerations for Data Collection

• Analysis plans should be compatible with the
  state's strategic plan state can use CLI to
  guide state analysis
• Measures must be consistent across sites to
  ensure the ability to make comparisons (both for
  the National evaluation and between and within
  states)

5
Important Considerations for Data Collection

• Comparisons data can, over time, help states to
  better focus prevention efforts and assess what
  works, under what conditions, and for whom
• Important policy implication of a rigorous
  evaluation Demonstrating the Importance of the
  SPF-SIG Program as a Proven Long-term Prevention
  Strategy

6
Value of Community Level Instruments

• Sharing of data between cross-site team and state
• Potential use as planning tool
• Facilitates communication between state and
  community partners

7
Community Level Instrument Review Process

• Draft sent to state grantees in early July
• Ten states provided feedback
• Instruments revised based on state feedback and
  sent back to states in September
• Pilot testing of instrument with community level
  organizations in October
• Revisions to instrument to incorporate feedback
  from pilot test
• Submit to OMB on October 31, 2005

8
Overview of Comments, Concerns and Suggestions

• Improved applicability for community partners
  that are operating as coalitions
• Revised questions about target populations and
  risk and protective factors to include targeting
  consequences and outcomes
• Revised questions on community readiness to
  remove make the approach less prescribed
• Re-worked the types of interventions to parallel
  CSAPs five intervention types

9
Concerns Still to be Addressed

• Instruments may make assumptions about how
  community level work is being implemented, which
  may not fit for all states/communities
• Evaluation of environmental strategies
• Structuring instrument to enable respondent to
  account for multiple project components, possibly
  being completed by different organizations within
  a community
• State level quality control procedures/burden

10
Web-based Data Collection

• Community instruments will eventually be
  completed via the Web
• Skip patterns will be used to decrease respondent
  burden
• Previously entered data will be available to
  respondents to reduce duplication of effort

11
NOMs and the Community Level Instrument

• NOMs provide a small consistent body of national
  data
• Access
• Capacity
• Use of Evidence-Based Interventions
• NOMs are a small part of the overall evaluation
  design

12
Pilot Test of Community Level Instrument

• Description of pilot test process
• What we hope to obtain from pilot test
• Estimate of response burden
• Assessment of clarity of questions/responses
• Feedback on questions community testers had
  problems with
• General comments on content, flow, ease of use
• Community level pilot testers still needed see
  Shelly for more info

13
Technical Assistance Plans

• Initial training for state grantees and community
  partners, possibly via video-conferencing or
  internet conferencing
• Formal administration guide provided
• Embedded definitions on web-based version
• One-on-one telephone TA to help resolve issues
  available M-F, 8 to 5