And the Survey Says. Putting YOUR Healthy Youth Survey Results to Work

1
And the Survey Says.Putting YOUR Healthy Youth
Survey Results to Work

• Spring 2003

2
Purpose Help people understand use their HYS
reports
3
Welcome!

• Introduction of presenters
• How many agencies are here today?
• Review Agenda
• What we will have accomplished by the end of the
  day?

4
Ice breaker
5
Ice Breaker

• In small groups, each person shares
• Name
• Where you work
• Share one interesting or obscure fact about
  yourself

6
Background of HYS 2002

• History of WA survey efforts
• Content of HYS 2002
• Sources of HYS 2002 items
• Data processing and quality control steps

7
Past Washington Surveys Primary Content
02
01
99
97
96
95
94
93
89
88
00
92
90
91
98
Combination
Health-risk Focused
Risk Protective Factor Focused
8
Joint Survey Planning Committee

• Office of the Superintendent of Public
  Instruction
• Division of Alcohol Substance Abuse, DSHS
• Department of Health
• Office of Community Development
• Governors Family Policy Council
• RMC Research (survey contractor)

9
HYS 2002 Collaborative Effort of OSPI, DOH,
DSHS, and CTED

• Simple random sample of schools recruited at
  state level
• County samples drawn as appropriate
• Survey consistently administered in the Fall of
  even years
• Survey given to 6th, 8th, 10th, and 12th graders
• Survey booklets will have one-page tear-off
  answer sheet
• Survey for 8th, 10th, and 12th graders will use 2
  form interleaved administration

10
HYS 2002 Participation

• The Healthy Youth Survey 2002 was completed

By 137,335 students
In 752 schools
In 203 school districts
In 39 counties.
11
Risk Protective Factor DataConsistent with
CTC/MTF SurveyDevelopment led by DASA/DSHS OSPI

• WSSAHB Questionnaire
• Demographics
• Alcohol, Tobacco, other drugs
• School risk/protective factors
• Community risk/protective factors
• Peer-Individual-Family risk/ protective factors

12
Youth Health Behavior DataConsistent with
National YRBS YTSDevelopment led by DOH

• YRBS Questionnaire
• Demographics
• Alcohol, Tobacco, Other drugs
• Nutrition
• Physical Activity
• Safety Behaviors (Helmets, Seatbelts)
• Mental Health/Depression/Suicide
• Additional Tobacco Indicators

13
Getting the Best of Both
Form A WSSAHB-like
Form B YRBS-like
Survey Core
14
6th Grade Survey
One Form
Survey Core Mutually Agreed Items
15
Core Survey Items

• Demographics
• Alcohol, Tobacco and Other Drug Use Items
• Key Violence-Related Items
• School-Specific Asset Items
• Depression

16
Sources of HYS 2002 Items

• Monitoring the Future (MTF)
• Youth Risk Behavior Survey (YRBS)
• Communities that Care (CTC)
• Youth Tobacco Survey (YTS)
• PRIDE Survey

17
Data Processing and Quality Control

• Administration procedures
• Data cleaning procedures

18
Statistical Issues

• Validity and reliability
• Confidence intervals
• Comparing state and local results
• Generalizability

19
Validity and Reliability

• Definitions
• How we assure validity Items from established
  instruments, validity checks
• How we assure reliability Standardized
  administration procedures

20
Confidence Intervals

• What are they?
• How to interpret the numbers
• How to interpret the graphs
• Practice activity (Ex. 12)

21
What is a Confidence Interval?

• The reported value is probably a little different
  than the true value for all your students
• We are 95 confident that the true value is
  within the /- range, called the confidence
  interval

22
Interpreting CIs Numerically

• Smoked cigarettes 9.2 ( 1.1)
• 9.2 1.1 8.1
• 9.2 1.1 10.3
• Between 8.1 and 10.3 smoked cigarettes

23
Interpreting CIs Graphically
24
Practice Activity

• Exercises 1 and 2

25
Why Are Confidence Intervals Different Sizes?

• Number of students
• Inherent variability
• Level of confidence (All 95 for HYS)
• Sampling design

26
If the data are valid, why do we need confidence
intervals?

• Confidence intervals account for variability
  among students, NOT validity of the data
• Variability is inherent in any population worth
  studying
• Variability causes uncertainty in the results
• Doubt is not a pleasant condition, but certainty
  is absurd Voltaire

27
Comparing State and Local Results

• Looking at differences numerically
• Looking at differences graphically
• Practice activity (Ex. 34)

28
Significant Differences Numerically (Ex. 1)

• Smoked cigarettes in the state 9.2 ( 1.1)
• Between 8.1 and 10.3
• Smoked cigarettes at my school 14.1 ( 2.2)
• Between 11.9 to 16.3
• Conclusion Difference IS statistically
  significant

29
Significant Differences Numerically (Ex. 2)

• Smoked cigarettes in the state 9.2 ( 1.1)
• Between 8.1 and 10.3
• Smoked cigarettes at my school 14.1 ( 9.6)
• Between 4.5 to 23.7
• Conclusion Difference is NOT statistically
  significant

30
Significant Differences Graphically (Ex. 1)
31
Significant Differences Graphically (Ex. 2)
32
Practice Activity

• Exercises 3 and 4

33
Generalizability

• What is generalizability?
• To whom can we generalize results?
• State County sample
• Non-sampled counties, districts, and ESDs
• Schools
• Why 70 participation is required

34
Quotations From Teachers Who Administered the
Survey

• About 25 students missed part of the session
  because of club photos
• Absent students are taking a PSAT test
• Many students had too much homework. Chose to do
  that rather than take the survey.
• Seven students came too late to take the survey
• The perfectionists, poor direction followers and
  slow readers could not finish

35
Challenges to Generalizability

• Brainstorm challenges to generalizability
• School level
• ESD, County, or District level
• How do these challenges affect data
  interpretation?

36
What if I dont want to generalize?

• Confidence intervals matter if you want to
  generalize to a larger population
• You can describe the students surveyed, in that
  moment, without confidence intervals
• Elmo Elementary School students who took the
  survey said

37
Final Note About Significance

• Even if a difference is statistically
  significant, it might not be practically
  significant
• Example
• State alcohol use 12.8 (0.2)
• Local alcohol use 14.4 (1.2)
• Statistically significant difference, but should
  it influence program planning?

38
Comparing data over time
39
Things to think about

• Did the questions change?
• Were the challenges to generalizability similar?
• Is there a reason to think that things would have
  changed? (a prevention program)

40
Surveys over time

• WSSAHB 2000 reports did not report confidence
  intervals but the variability was still there!
• Future HYS reports will have confidence
  intervals, so 2002 vs. 2004 comparisons will be
  easier

41
What should I know about comparing 2000 to 2002
data?

• If you think you have comparable data, then
• If the 2000 percentage is INSIDE the 2002
  confidence interval, then there is not a
  statistically significant difference
• If the 2000 percentage is OUTSIDE the 2002
  confidence interval, then you cant be sure if
  the difference is statistically significant but
  the further away it is the more likely that a
  difference is significant
• If you really, really need to know whether its
  significant, you can contact DOH for a
  specialized data run

42
What if I am in a small school, and we have huge
confidence intervals?

• Having a confidence interval protects you (and
  your program) from appearing to be ineffective
  when just a few students can make big changes
• Consider the input of teachers staff from
  small-school environments when interpreting data
  the data should be used to complement what they
  already know about their kids

43
Break
44
Local Report Overview

• Guide to the Local Report
• FAQ
• How to use your local report

45
Local Report Contents

• Introduction and Overview
• Key to the Notes
• Highlights of the Local Results
• Graphical summary of selected results
• Selected results disaggregated by gender
• Responses to all items

46
Guide to the Local Report, cont.
Healthy Youth Survey 2002Survey Results
Sample Middle School, Grade 8
The number of students who submitted survey forms
Number of students surveyed
64
Number of valid responses
59
The impact of adolescent health risk behaviors
remains a primary concern of citizens throughout
the country. Many health problems experienced by
adolescents are caused by a very few preventable
behaviors.
47
Guide to the Local Report, cont.
Healthy Youth Survey 2002Survey Results
Sample Middle School, Grade 8
Number of students surveyed
64
Number of valid responses
59
The number of forms that were considered usable.
The impact of adolescent health risk behaviors
remains a primary concern of citizens throughout
the country. Many health problems experienced by
adolescents are caused by a very few preventable
behaviors.
48
Guide to the Local Report, cont.
To protect anonymity when data are broken down
into identifiable groups.
49
Guide to the Local Report, cont.
Number of students who responded to this item
50
Where Do Find Information About . . . ?

• General Information (e.g. demographics)
• Alcohol, Tobacco, and Other Drug Use
• Other Health Concerns (e.g. health/safety)
• School climate (e.g., bullying)
• Quality of Life (e.g., student outlook)
• Risk and Protective Factors
• Scale Results
• Item Results

51
Frequently Asked Questions

• Review handout
• Note about optional items

52
How to use your local report

• Example
• Practice Activity

53
Local Report Example
54
State Agencies Using the HYS Data
55
Federal Initiatives

• Drug-Free Schools and Communities Act
• Department of Education Safe and Drug-Free
  Schools Program
• Principles of Effectiveness
• Healthy People 2010 National Health Promotion
  and Disease Prevention Objectives
• National Drug Control Strategy
• No Child Left Behind

56
Washington State Initiatives

• Washington Education Reform Act
• Omnibus Alcohol and Controlled Substances Act
• Violence Reduction Programs Act
• Department of Health Priority Health Goals
• Governor's Council on Substance Abuse

57
Why Do You Keep Changing Your Terminology?

• OrWhen is an Impact an Outcome?

58
(No Transcript)
59
Putting Your Data to Work

• Program Planning and Evaluation

60
Whats the Problem?

• What is a problem?
• Getting worse
• Worse than average
• Just plain bad

61
Whats the Problem?

• Combine HYS data with other information,
  including knowledge of community stakeholders
• For example, HYS data may show that the highest
  rates of tobacco use are in a particular area of
  the county, where police know there is a problem
  with retailers who provide tobacco to minors

62
Exercise 1 Whats the Problem?
63
Lunch
64
Whats the Priority?

• HYS data helped to identify a set of problems
• Need to prioritize among them
• Identify available resources
• Who is already working on the issue, or who has a
  stake in the issue
• Gaps in services or programs
• Modify existing programs, instead of creating all
  new ones
• Figure out what the community is ready for

65
Whats the Priority?

• Target Efforts
• Describe a target audience (grade level, gender,
  universal or selective approaches)
• Write SMART goals and objectives to get FOCUSED
  and CLEAR with partners, and evaluation
• Specific
• Measurable
• Achievable
• Relevant
• Time-bound

66
Setting a Good Objective

• You have 1,000 students
• Current smoking prevalence is 25 (250 kids)
• SMART Objective By 2004, we will implement a
  comprehensive tobacco prevention program and
  reduce current smoking among our schools
  students by 10
• This translates into about 25 fewer smokers
• This is a change from 25 to 22.5

67
Keeping it Real

• Will your program reach enough kids to affect the
  overall prevalence?
• Remember the difference between a percentage
  point goal and a percent change goal!

68
Population Group Changes - Gender
Source WSSAHB 2000, HYS 2002, state sample data
69
Exercise 2 Whats the Priority?
70
What Should I Do About It?

• Find Best Practice programs
• Search for proven activities
• The Community Guide to Preventive Services
• CSAP Science-based Prevention Programs
• Agency recommendations
• for example, DOH recommends conducting a retailer
  compliance program public awareness about
  sources of tobacco

71
How Do I Know If It Worked?

• Evaluate
• Identify and collect process indicators,
  outcomes, and impacts relative to your program
• Process measures may include compliance checks
  compliance rates
• Outcomes may include perceived availability of
  tobacco usual sources of tobacco
• Impacts may include lifetime and current use of
  tobacco

72
Evaluation made easy

• What do I do when I go to work every day?
• .and then what happens?
• .and then what happens?
• .so that the WHY of going to work every day
  happens.

73
(No Transcript)
74
Exercise 3 How Do I Know If It Worked?
75
Just in case you need more money

• Healthy Youth Survey data can help
• Provide justification of need for funding
• Provide means for evaluating funded activities
• Find out about grant scoring
• You will need to justify that HYS data are
• Valid
• Reliable

76
Exercise 4 Grant Writing
77
Next Steps

• Communicating With Your Data

78
Communications

• Nuts and bolts
• Communicating about your program
• Communicating to the media
• Using data to make change

79
The Nuts and Bolts

• Round percents to whole numbers
• 17.0 to 17.4 becomes 17
• 17.5 to 17.9 becomes 18
• Use language to acknowledge that these percents
  are estimates
• About 17
• 14 to 20 percent
• plus or minus three percent

80
Different Ways to Say It

• Instead of saying
• 17.2 /- 3.2 youth said that they mostly or
  definitely felt unsafe at school
• You could say
• About 17 of our youth feel unsafe in school
• About one in five youth feel unsafe in school
• Remember you can present it positively
• About 83 of our youth feel SAFE in school
• About four out of five youth feel SAFE in
  school

81
Additional Tips

• If you had a low response rate for your survey,
  it would be best to say
• Elmo Elementary School students who took the
  survey said
• Instead of
• Elmo Elementary School students said

82
Communicating About Your Program

• What data are appropriate for each type of
  communication?
• Your co-workers
• Your neighbor at a cocktail party
• Your grandmother

83
Communicating to the Media

• Whats your SOCO? (Single Overriding
  Communication Objective)
• For broadcast media 10-12 word sound bite
• For print media 1-3 line quotation
• Message map

84
For Example Tobacco Program Results

• SOCO Our program has been successful in reducing
  youth smoking
• Detail 1 Fewer youth are currently smoking than
  prior to the program
• Fact 1 53 fewer 6th graders
• Detail 2 Fewer youth have ever tried smoking
• Fact 29 fewer 10th graders have ever smoked a
  whole cigarette
• Detail 3 Washington is doing better than the
  nation
• Fact there was a 22 decrease in the nation for
  current smoking, so Washingtons decline is
  greater

85
Potential Pitfalls

• When communicating with data, its easy to make
  these mistakes
• Include information thats not totally relevant
  to your SOCO
• Include information that is relevant, but goes
  beyond what you NEED (too much detail)
• Speculate beyond what you have in the data
• Treat non-significant or unimportant differences
  as significant
• Forget to double-check your results and/or the
  math

86
Keep on message

• The answer to every question is your SOCO.
• Thats an interesting point, but what Id really
  like people to know is,
• my SOCO

87
Message Mapping Exercise

• In small groups
• Choose 1 topic of interest from HYS
• Develop SOCO and 3 supporting details
• Use HYS local report to find 3 supporting facts
  for each detail
• Message map template in handouts

88
Using Data to Make Change

• The message should tell your audience what you
  want them to do
• Do you want them to be outraged? And then what
• Do you want them to call a policymaker?
• Do you want them to change their beliefs or
  behaviors?
• Do you want them to support your program?

89
Choose and Describe Your Audience

• Potential Audiences
• Co-workers
• Area residents
• Elected officials
• Civic organizations
• Health care providers
• Media
• Regulatory agencies
• Activists

• Demographics
• Characteristics
• Concerns
• Attitudes
• Levels of interests
• Levels of involvement
• Histories
• Levels of knowledge
• Opinions
• Reasons for interest
• Types of involvement

90
How Will You Reach Your Audience?

• Area residents
• Community meetings
• Newspaper articles and ads
• Radio and TV talk shows
• Fliers
• Direct Mailings

• Elected officials
• Frequent phone calls
• Fact sheets
• Personal visits
• Invitations to community meetings
• News releases

91
Cardinal Rules of Communication with Data

• Accept and involve the public as a partner
• Theyre going to comment on your stuff anyway
• Know ahead of time what public opinion is around
  your topic
• Anticipate (and prepare for) the tough questions
  smart answers including criticism of the
  survey itself
• Plan carefully and evaluate your efforts
• Was your SOCO heard?
• Did you reach your target audience?
• Did they know what you wanted them to do?
• Be honest, frank, and open
• Be prepared to talk about the limitations of your
  data
• Work with other credible sources
• Have someone critique your findings or
  interpretation
• and double-check your math

92
Cardinal Rules of Communicating with Data (cont.)

• Meet the needs of the media
• Meet their deadlines
• Help them understand what the data mean
• Help them understand how the survey works, if
  needed
• Speak clearly and with compassion
• data are just people with the tears wiped away
• Practice saying it out loud (numbers can trip
  your tongue)
• Use AV or graphics with caution
• Sometimes a graph hurts more than helps you
• Visual aids might distract from your message
• Dont try to exaggerate with graphics

93
Thank You!!
94
More questions about the training?

• Department of Health
• Lauren Jenks
• 360-236-3567
• lauren.jenks_at_doh.wa.gov
• OSPI
• Bob McArdle
• 360-725-6046
• BMcArdle_at_ospi.wednet.edu
• Mona Johnson
• 360-725-6044
• MonaJ_at_ospi.wednet.edu

• DASA, DSHS
• Linda Becker
• 360-407-0640
• BeckeLG_at_dshs.wa.gov
• RMC Research Corporation
• Eric Einspruch
• Gwen Hyatt
• 1-800-788-1887
• Healthyyouthsurvey2002_at_rmccorp.com
• http//www.rmccorp.com/project/hys02.html

95
The Joint Survey Planning Committee

• DASA
• Linda Becker
• Steve Smothers
• DOH
• Lillian Bensley
• Julia Dilley
• Lauren Jenks
• Susan Richardson
• Judy Schoder
• Juliet VanEenwyk

• OSPI
• Mona Johnson
• Bob McArdle
• Martin Mueller
• Pam Tollefsen
• OCD
• Susie Roberts
• Family Policy Council
• Bill Hall