Healthy Youth Survey Using Your Data Workshop

1
Healthy Youth SurveyUsing Your Data Workshop

• Spring 2007

2
Welcome!

• Who is here today?
• Review agenda and objectives for the day
• Is there anything else you are hoping to get out
  of this training?
• Logistics
• Breaks
• Workbook
• Local Reports???

3
Ice Breaker

• In small groups, each person shares
• Name
• Where you work
• One interesting or obscure fact about yourself

4
Background of HYS 2006
1

• History of WA survey efforts
• Content and sources of HYS
• Data processing and quality control

5
Past Washington youth surveys
W
H
W
W
Y
H
H
U
U
W
05
06
02
01
99
97
96
95
94
93
89
88
00
92
90
91
98
03
04
U Student Alcohol and Drug Use Survey W
Washington State Adolescent Health Behaviors Y
Youth Risk Behavior Survey H Healthy Youth
Survey
6
HYS 2006 administration

• A collaborative effort among multiple state
  agencies
• Joint Survey Planning Committee
• State-level simple random sample of schools
• County-level samples drawn as appropriate
• Non-sampled schools also invited to participate
• Survey consistently administered in the Fall of
  even years
• Survey given to 6th, 8th, 10th, and 12th graders

7
2006 HYS participation

• The 2006 Healthy Youth Survey was completed

By 198,312 students
In 1,028 schools
In 234 school districts
In all 39 counties.
8
Content and Sources of HYS Items

• Monitoring the Future (MTF)
• Youth Risk Behavior Survey (YRBS)
• Communities that Care (CTC)
• Youth Tobacco Survey (YTS)
• Questions added in 2006 were validated with field
  testing

9
Survey form variations
8th, 10th, 12th grade - Forms A and B
(interleaved)
6th grade - Form C
Form B
Form A
Form C
Core
Core
Each form has a 1 page scannable answer sheet
10
Form ADevelopment led by DSHS/DASA, OSPI and
CTEDGrades 8,10,12 In 2006 133 questions on
main survey 15 questions on the tear-off

• School risk/protective factors
• Community risk/ protective factors
• Peer-Individual-Family risk/ protective factors

11
Form BDevelopment led by DOHGrades 8,10,12 In
2006 115 questions on the main survey 20
questions on the tear-off

• Nutrition
• Physical Activity
• Safety Behaviors (Helmets, Seatbelts, Violence)
• Mental Health/ Depression/ Suicide
• Tobacco Indicators

12
Form COne survey formGrade 6 In 2006 87
questions on the main survey 8 questions on
the tear-off
Mutually agreed on and simplified items
13
Core survey items33 questions on A and B, 18 on
A,B,C

• Demographics
• Alcohol, Tobacco and Other Drug Use Items
• Key Violence-Related Items
• School-Specific Risk and Protective Factor Items
• Depression

14
Tear-off items

• Optional section at the end of the survey that is
  perforated so districts or schools can tear off
  the questions.
• Examples of tear-off
• Family risk and protective factors
• Physical abuse
• Asthma
• Additional tobacco questions

15
Data Processing and Quality Control

• Standard administration procedures
• Data cleaning procedures

16
Administration procedures

• Student and parent notification
• Safe and confidential environment
• Consistent administration procedures
• Students informed of importance of survey
• No student name or other identifying information

17
Data cleaning procedures

• Inconsistent answers
• Evidence of faking high levels of substance use
• Dishonesty
• Wrong grade

18
Local Report Overview
2

• Introduction and overview
• Summary of selected results
• Selected results by gender
• Item groupings
• Risk and protective factors
• List of core items item index

19
Introduction and Overview

• Types of reports
• Report of Results
• Report of Participating Schools

20
Your report
Healthy Youth Survey 2006Survey Results
Sample Middle School, Grade 8
Number of students surveyed
173 165 206 80
Number of valid responses
Estimate of enrolled students
Survey participation rate
The impact of adolescent health risk behaviors
remains a primary concern of citizens throughout
the country.
21
Your Report, cont.

• Key to the Notes

A  wording on Form AB  wording on
Form BC  wording on Form C  optional item
22
Summary of Selected Results

• Reports include same questions and graphs as in
  2002 and 2004 report

23
Selected Results by Gender

• Same as in 2002 and 2004 report
• Results suppressed to protect anonymity

If p value lt 0.05 the difference between local
males and local females is statistically
significant
24
Why asterisks instead of numbers for gender?

• To produce results by gender a minimum of at
  least 15 respondents must give each response
  available
• For example, to produce results for cigarette
  smoking
• at least 15 girls have to report smoking and at
  least 15 dont, AND
• at least 15 boys have to report smoking and at
  least 15 boys dont

25
Item Groupings

• General Information
• ATOD Use
• Other Health Concerns
• School Climate
• Quality of Life
• Risk and Protective Factor Items

26
Item example (Grade 8)
27
Why is item n lt overall n?

• Not a core item (Grades 8, 10, and 12)
• Optional look for symbol
• Students did not reach item near end of survey
• Students skipped or did not mark clearly

28
Why asterisks instead of numbers for frequencies?

• No students responded (n0)
• Suppressed to protect anonymity
• Overweight (Item 68)
• Results by Gender (cell nlt15)

29
Risk and Protective Factors

• Framework and Reporting Schedule
• Scale Results
• Different order from 2002 and 2004
• Standardized to be consistent in future
• Graphs
• Individual Item Results

30
Last Page

• List of Core Items
• Secondary only
• Helps interpret ns
• Item Index

31
Statistical Issues
3

• Validity and reliability
• Generalizability
• Confidence intervals
• Comparing state and local results
• Comparing data over time

32
Validity and Reliability

• Definitions
• How we assure validity
• Items from established instruments, validity
  checks
• How we assure reliability
• Standardized administration procedures

33
Generalizability

• What is generalizability?
• Why 70 participation is important
• Challenges to generalizability
• School level
• ESD, County, or District level
• How do these challenges affect data
  interpretation?

34
Do I have to generalize?

• Yes, if you want to apply the results to a larger
  population.
• 8th graders in our district said.
• Yes, if you want to compare to others or results
  over time
• No, if you want to just describe the students
  surveyed, in that moment, without confidence
  intervals
• Students at our school who took the survey said.

35
Confidence Intervals

• What are they?
• How to interpret the numbers
• How to interpret the graphs
• Practice Activity 1

36
What is a confidence interval?

• The reported value is probably a little different
  than the true value for all your students
• We are 95 confident that the true value is
  within the /- range, called the confidence
  interval

37
What do confidence intervals look like?
Smoked cigarettes (Grade 10, 2006) 14.9 (
1.4) 14.9 1.4 13.5, 14.9 1.4
16.3 Between 13.5 and 16.3 smoked cigarettes
38
Exercise 1 Confidence Intervals
39
Why are confidence intervals different sizes?

• Number of students
• Inherent variability
• Level of confidence (All 95 for HYS)
• Sampling design

40
Why do we need confidence intervals?

• Confidence intervals account for variability
  among students, NOT validity of the data
• Variability is inherent in any population worth
  studying
• Variability causes uncertainty in the results
• Doubt is not a pleasant condition, but certainty
  is absurd Voltaire
• They help us compare our results to others and to
  ourselves over time

41
Comparing State and Local Results

• What is statistical significance?
• Looking at differences with confidence intervals
• Numerically
• Graphically

42
Significant differences numerically

• Smoked cigarettes in the state 14.9 ( 1.4)
• Between 13.5 and 16.3
• Smoked cigarettes at my school 20.0 ( 2.2)
• Between 17.8 to 22.2
• Conclusion Difference IS statistically
  significant (confidence intervals do not overlap)

43
Significant differences numerically

• Smoked cigarettes in the state 14.9 ( 1.4)
• Between 13.5 and 16.3
• Smoked cigarettes at my school 20.0 ( 9.6)
• Between 10.4 to 29.6
• Conclusion Difference is NOT statistically
  significant (confidence intervals overlap)

44
Significant differences graphically
45
Significant differences graphically
46
Combining item responses
To determine the confidence interval for Any
days. Use the same confidence interval as the
0 days" response. ( 2.0)
To determine the result for Any days Subtract
the 0 days result from 100 100.0 - 97.0
3.0
47
Combining item responses cont.
To determine the result for YES Add the
Probably yes and Definitely yes responses
together 29.5 59.0 88.5
When you combine 4 response options down to 2
groups like this, you cannot determine the
confidence interval
48
Comparing Data Over Time

• Things to think about first
• Did the questions change?
• Were the challenges to generalizability similar?
• Is there a reason to think that things would have
  changed? (a prevention program)

49
Comparing 2006 to older data

• If you think you have comparable data, then
• If the confidence intervals do NOT cross, then
  the difference IS statistically significant
• If the confidence intervals DO cross, you cant
  be sure if there are differences
• Results prior to 2002, did not have confidence
  intervals. If older data point estimate falls
  within the range of the 2006 confidence interval,
  then the difference is not statistically
  significant
• Other statistical tests provide more precision in
  determining differences.

50
Exercise 2Comparing State and Local Results

• Directions
• Page 23 in your workbook
• Questions 1 2
• Figure out whether any differences between your
  local results and state results are statistically
  significant
• What does that mean?

51
What if I am in a small school, and we have huge
confidence intervals?

• Having a confidence interval protects you (and
  your program) from appearing to be ineffective
  when just a few students can make big changes
• Consider the input of teachers staff from
  small-school environments when interpreting data
  the data should be used to complement what they
  already know about their kids

52
Final notes about significance

• Even if a difference is statistically
  significant, it might not be practically
  significant
• For example
• State use of some drug 12.8 (0.2)
• Local use of that drug 14.4 (1.2)
• The difference statistically significant, but is
  it different enough to influence program
  planning?
• CI are quick and available, but there are more
  precise ways to determine significance.

53

• Dont worry, help is available from many sources
• Local Health Department
• Local ESD
• JSPC agencies
• Web materials

54
Break
55
Putting Your Data to Work
4

• Where to find HYS Data
• Program planning with HYS data
• Identifying problems and setting priorities
• Setting objectives
• Determining effectiveness

56
Remember.Data are only ONE piece of the puzzle
57
Where to Find HYS Data

• Local reports
• Powerpoint slides
• New website under development
• Pre-formatted reports
• Data query website

58
New preformatted reports
59
Program Planning with HYS Data

• Different agencies use different program planning
  models
• Your HYS data can help you answer key, consistent
  program planning questions
• What is the problem?
• What is the priority?
• What do we do about it?
• How do we know it worked?

60
Using HYS Data to Identify Problems and Set
Priorities

• Results are worse compared to others
• Trend increasing or decreasing
• Results are concerning
• There is community interest or opportunity
• But remember Data can help!
• They have helped determine best practices, and
  can help lead to the implementation of good
  programs, not just programs that feel good.

61
How do data tell you what to do about it?
Lifetime Alcohol Use Percent of students who have
ever had more than a sip of alcohol
62
Setting an Objective with HYS Data

• Are there already defined objectives for you to
  meet?
• How do you make your own objective?
• Write a statement about what you will achieve if
  program successful
• Describe your target audience (grade level,
  gender, universal or selective approaches)
• One option SMART objectives
• Specific, Measurable, Achievable, Relevant, and
  Time-bound

63
Setting a SMART objective

• By 2008, we will implement a comprehensive
  tobacco prevention program and reduce current
  smoking among our schools 10th grade students by
  10, to 22.5 or less (2006 baseline 25)

64
Calculating and talking about percent change

• By 2008 there will be 10 fewer smokers
  (currently 25 are smokers)
• Do you mean to say
• Percentage point change (absolute)?
• or
• Relative percentage change (rate of change)?

65
Percentage point change (absolute)

• By 2008 there will be 10 fewer smokers
  (currently 25 are smokers)
• You mean that there will be 15 smokers in 2008.
• Calculated by subtracting the percent fewer
  (10) from your original rate (25)
• 25 - 10 15

66
Relative Percentage Change (Rate of Change)

• By 2008 there will be 10 fewer smokers
  (currently 25 are smokers)
• You mean that there will be 22.5 smokers in
  2008.
• Calculated by multiplying the percent fewer
  (10) times your original rate (25).
• Then subtract that number (2.5) from your
  original rate (25).
• 25 X 10 2.5
• 25 - 2.5 22.5

67
Exercise 3 Calculating Percent Change

• Answer the questions on page 32 of the workbook
  individually or in your group

68
Evaluation Determining Effectiveness

• One way to help identify how to determine
  effectiveness is to use a Logic model
• Logic model to describe the relationship between
  your program activities and the outcomes you are
  trying to achieve

69
Sample logic model
70
How can data tell you if it worked?

• Did you meet your objective?
• What actually happened, and was it different than
  if youd done nothing?
• Compare community/school to itself over time
• Compare community/school to state over time
• Compare graduating classes
• for example how things changed between the 8th
  to 10th grade years for the class of 2006 vs. 2008

71
Exercise 4Using Data to Inform your Program

• Answer the questions on page 35 of the workbook
  in your group
• Use the data in the program evaluation grid on
  page 36

72
Exercise 4 DiscussionProgram Evaluation Grid
73
Exercise 4 discussion

• What were some of the changes or differences that
  you found?
• Did the data help you get the answers to these
  questions?
• Did you decide to make any changes to your policy
  efforts?

74
Final thoughts on evaluation

• Changes take time- especially for long- term
  outcomes
• Be realistic about what can be achieved
• HYS may not be the best source of data for your
  evaluation
• Consider additional ways to evaluate as well

75
Using Your Data to Communicate
5

• Talking about numbers
• Presenting numbers
• Communication planning

76
Simplify when Talking about Numbers

• Dont say 17.4 plus or minus 4.1 of students
  said
• Round whole numbers
• 17.0 to 17.4 rounds down to 17
• 17.5 to 17.9 rounds up to 18
• Use language to convey that these percents are
  estimates
• About 17
• About 17 plus or minus three percent
• Between 14 to 20 percent

77
Use the correct terminology

• Use language to convey that these percents only
  represent a specific group of students (if you
  have low response rates, or do not want to
  generalize to a larger population)
• About 17 of students in our community who took
  the survey said
• When quoting the data
• data are plural
• The data were not was collected
• Our data suggest not suggests that
• datum is singular

78
Consider different ways to say it

• Instead of saying
• 17.2 3.2 youth said that they mostly or
  definitely felt unsafe at school
• You could say
• About 17 of our youth feel unsafe in school
• About one in six youth feel unsafe in school
• Remember you can present it positively
• About 83 of our youth feel SAFE in school
• About five out of six youth feel SAFE in school

79
Presenting Your Numbers

• What is already available?
• Frequency Reports from RMC
• State, County and special population reports on
  the web
• Powerpoint slides from RMC
• New online pre-formatted reports
• You can create your own materials such as fact
  sheets, brochures etc.

80
Communication Planning

• Whats your SOCO? (Single Overriding
  Communication Objective)
• The so what of your message
• Meaningful to your target audience
• Tells them what you want them to learn or do
• What does a SOCO look like?
• For broadcast media 10-12 word sound bite
• For print media 1-3 line quotation
• For real people people just wont remember more
  than 3-4 ideas

81
Using a message map

• A SOCO needs to be supported by Details
• Provide logical justification
• These are reasons why
• The details need to be proven by Facts
• These are the evidence for your details
• Use your data here

82
Message map
83
For example Tobacco program results
84
Exercise 5Using your Data to Communicate

• Pick one scenario (A,B,C) on page 41
• Use the data in the grid on page 42
• Create a SOCO -page 43
• Share

85
Exercise 5 discussion
86
Exercise 5 discussion

• Some SOCOs that we came up with
• The school board needs to take action to
  prioritize alcohol prevention programs for our
  middle school children
• Every parent in our community should talk to
  their young children about saying no to drinking
  and driving with people who are drinking
• Funding this program will reduce the excess
  danger that children in our community face from
  alcohol
• Details (supported by facts in the data)
• Alcohol use in 8th graders is higher for our
  children compared to the state
• Both driving while drinking and riding with a
  driver who had been drinking are significantly
  higher for 10th and 12th graders compared to the
  state
• Our 10th and 12th graders are less likely than
  children statewide to believe that regular
  alcohol use is dangerous

87
Potential traps.

• Including non-relevant information or too much
  detail
• Speculating beyond the data or treating
  differences as significant when they are not
• Forgetting to double-check your numbers
• Using too many graphics, or graphics that distort
  the data
• Talking about numbers instead of the people they
  represent

88
Helpful tips.

• Have someone you trust critique your message for
  accuracy and understandability
• Practice saying everything, including the
  numbers, out loud
• Practice saying I dont know, but I can get back
  to you about that
• Be prepared to discuss the limitations of your
  data
• Speak with clarity and compassion

89
Keep on message

• The answer to every question is your SOCO.
• Thats an interesting point, but what Id really
  like people to know is,
• my SOCO

90
Final thoughts

• Dont flip through your HYS report to find the
  story, find your story and use data to support it
  (most of the time)
• If your story is we have important data to
  describe our kids (overview) then use the local
  slide sets provided to you

91
Obtaining More Information

• Survey participation information and training
  materials
• www.hys.wa.gov/
• State and county level results, state level
  online data query, and survey background
  information
• http//www3.doh.wa.gov/hys
• Preformatted fact sheets with HYS results
• http//www.AskHYS.net (under development)

92
More questions about this training?

• Department of Health, Diane Pilkey 360-236-3526,
  Diane.Pilkey_at_doh.wa.gov
• OSPI, Jeff Soder 360-725-6044,
  Jeff.Soder_at_k12.wa.us
• OSPI Dixie Grunenfelder 360-725-6045
  dixie.grunenfelder_at_k12.wa.us
• DASA, DSHS, Linda Becker 360-725-3705,
  BeckeLG_at_dshs.wa.gov
• CTED, Ramona Leber 360-725-3033
  ramonal_at_cted.wa.gov
• RMC Research Corporation, Eric Einspruch
  1-800-788-1887, www.rmccorp.com

93
Thanks! From the Joint Survey Planning Committee

• OSPI Martin Mueller, Mona Johnson, Jeff Soder,
  Dixie Grunenfelder
• CTED Ramona Leber, Amy Tsai
• Family Policy Council Bill Hall
• DSHS, DASA Steve Smothers, Linda Becker
• DOH Judy Schoder, Diane Pilkey, Susan
  Richardson, Lillian Bensley, Julia Dilley, Juliet
  VanEenwyk, Julie Simon
• RMC Research Corporation Eric Einspruch

Dont forget to fill out an evaluation!