State RBS Program Measures of Effectiveness Workshop

1
State RBS Program Measures of EffectivenessWorks
hop

• Navigating the process of developing meaningful
  performance measures for recreational boating
  programs
• Thursday, September 6, 2007
• Facilitated by Dr. Deborah Gona
• NASBLA Annual Conference, Burlington, Vermont

2
INTRODUCE YOURSELF . How long have you been a
BLA? Had formal training in developing measures?
What kind? /what source?
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
3
PICK ONE OF THESE . Whats the one thing wed
measure if we could actually figure out how to
get the data. Whats the best measure were
using right now to help tell us how effective
we are. What measure just doesnt cut it, but we
use it because there isnt a better
alternative. What one (or two) things do I wish
I understood better about setting goals or
creating measurements or collecting data.
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
4
INTRODUCE YOURSELF . How long have you been a
BLA? Had formal training in developing measures?
What kind? /what source? PICK ONE OF THESE
. Whats the one thing wed measure if we could
actually figure out how to get the data. Whats
the best measure were using right now to help
tell us how effective we are. What measure
just doesnt cut it, but we use it because there
isnt a better alternative. What one (or two)
things do I wish I understood better about
setting goals or creating measurements or
collecting data.
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
5
BACKGROUND Project builds on NASBLA research into
goal-setting, program assessment, and measures of
effectiveness for one state RBS program component
numbering titling Produced and conducted
under a grant from the U.S. Coast
Guard Introduced at NASBLAs Spring BLA workshop
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
6
LEARNED AT BLA WORKSHOP Almost 70 reported
having some measures training familiar with
basic terms, theories, processes Agency-based,
structural differences in ability/ capacity to
collect/analyze data for internal or external
use Concern about over-emphasis on apparent
changes in numbers, especially when a reporting
or methodology change likely was basis for
perceived change in program effectiveness
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
7
LEARNED AT BLA WORKSHOP (continued) Interest in
developing better grant applications and more
complete documentation for Performance Reporting
to the USCG Interest in real examples of
measures And in testament to the need for
continuous information sharing on measures
development and processes? We learned just
under half of the BLA participants had been on
the job three years or less
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
8
THE RUNDOWN FOR TODAY

• Straight talk about measures
• Why bother?
• The theory versus the practice
• Measurement challenges for RBS programs
• Commonly-used concepts and definitions
• Program assessments, constructing goals,
  constructing new measures, evaluating old ones
• Interactive exercise

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
9
THE RUNDOWN FOR TODAY (continued)

• Linking to the National RBS Strategic Performance
  Goals and Objectives
• Interactive exercise
• Data and evaluative/research tough to do
  reports and measures without them
• National Recreational Boating Survey
• What it could mean for effectiveness measurement
  at the state-level
• Vessel Identification System (VIS) and MOA

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
10
WHY MEASURE FOR RBS PROGRAM PERFORMANCE AND
EFFECTIVENESS?

• To get some indication of what your program
  components are accomplishing
• To give some evidence as to whether results are
  being achieved
• To get information for management and
  programmatic decision-making
• To keep focused on larger goals and outcomes

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
11
BECAUSE YOU HAVE TO!

• To fulfill state planning and budgeting
  requirements
• To fulfill federal requirements
• To get and keep your resources
• To head off resource raiding with evidence of
  what your program components are accomplishing
• To get support from the public, the legislature,
  other stakeholders and partners
• To answer inquiries from all of the above and
  the media

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
12
Why measure?To raise questions about your
programs components because measures by
themselves do NOT give all the answers.
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
13
IN THEORY Performance measures are thought to
be answers to questions because they are
presented quantitatively, they have the
appearance of fact and convey impressions of
objectivity and neutrality.
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
14
IN PRACTICE Program performance measures raise
questions they are quantitative
interpretations of reality they can be used as
arguments and weapons in policy and budget
debates they are objective if the program
seems to be doing well, and if I support it!
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
15
IN PRACTICE Performance measures built on
reliable data and kept up-to-date tell the
funding source, the state, the agency, the
program personnel, in a general way, how the
program is doing they do NOT tell the agency
or program personnel what to do next
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
16
Performance measures are best understood as
information that can help to sharpen questions,
rather than all of the answers to questions
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
17
IN THEORY Performance measures measure whats
most important actual results actual outcomes
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
18
IN PRACTICE, we tend to Measure the
measurable a lot of the time, rather than the
most important because collecting data is
expensive, creating measures is time consuming,
and the measures rapidly get out-of-date
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
19
IN PRACTICE, we often have to Create
surrogates and indices for actual results
because some concepts are next to impossible to
measure otherwise
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
20
IN PRACTICE, we often have to Measure
processes
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
21
Thoughtfully aggregated, measures can give us
bigger picture clues, sharpen the questions we
ask about programs, and guide changes in policy
and programs
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
22
IN THEORY Performance measures demonstrate
causality
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
23
IN PRACTICE It is very tough to demonstrate
precise causality especially in the absence of
underlying research or when so many factors can
play a role
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
24
SOME OF THE BIGGEST MEASUREMENT CHALLENGES

• The program or a specific activity is just
  one of many contributing factors to the hoped-for
  outcome
• The program outcomes are just plain tough to
  measure
• Results wont be evident for quite a while
• The program or a specific activity involves
  deterring certain behaviors or preventing
  something from happening

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
25
Information generated from program performance
and effectiveness measurement is just one part of
the decision-making equationCombined with other
program evaluation data and underlying research
assumptions, we can better understand WHY results
occur and what VALUE a program really adds
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
26
Goal-setting and measurement definitions and
concepts
27
GOALS or strategic goals
What you hope to achieve / accomplish, usually
stated in a big picture way Might be statements
of purpose or mission in strategic planning
Gives insight into what the agency, program,
organization values
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
28
GOALS (continued)
To ensure the public has a safe, secure,
enjoyable recreational boating experience by
implementing programs that minimize the loss of
life, personal injury, and property damage while
cooperating with environmental and national
security interests . (USCG Strategic Plan,
2007-2011)
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
29
GOALS (continued)
to strengthen the ability of the state and
territorial boating authorities to reduce death,
injury and property damage associated with
recreational boating and ensure a safe, secure
and enjoyable boating environment . (NASBLA
Strategic Plan and Association Mission)
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
30
GOALS (continued)
to provide safe and enjoyable fishing, hunting
and boating opportunities to the citizens of
Delaware and its visitors . (excerpt, Delaware
Department of Natural Resources and Environmental
Control Division of Fish Wildlife Enforcement
mission)
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
31
GOALS or strategic goals (continued)
At this level, the goals are NOT easily
measured. But they do contain elements that can
be broken out and used to develop more measurable
objectives or performance goals
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
32
OBJECTIVES or performance goals
Statements of something measurable that would let
you know youre progressing toward the goal Can
be statements of levels of performance or
effectiveness
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
33
OBJECTIVES or performance goals
To ensure the public has a safe, secure,
enjoyable recreational boating experience by
implementing programs that minimize the loss of
life, personal injury, and property damage while
cooperating with environmental and national
security interests . To reduce recreational
boating casualties . To reduce five-year average
annual deaths To reduce five-year average
annual injuries
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
34
OBJECTIVES or performance goals
To ensure the public has a safe, secure,
enjoyable recreational boating experience by
implementing programs that minimize the loss of
life, personal injury, and property damage while
cooperating with environmental and national
security interests . To reduce recreational
boating casualties . To reduce five-year average
annual deaths To reduce five-year average
annual injuries Safety education
certificates Awareness of safe boating
practices Advanced boating education Life jacket
wear And so on
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
35
OBJECTIVES or performance goals
to strengthen the ability of the state and
territorial boating authorities to reduce death,
injury and property damage associated with
recreational boating and ensure a safe, secure
and enjoyable boating environment Provide
national leadership and advocacy to focus
policies and resources on reducing risks in
recreational boating and in waterway security
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
36
OBJECTIVES or performance goals
to provide safe and enjoyable fishing, hunting
and boating opportunities to the citizens of
Delaware and its visitors . Provide
consistent, coordinated and increased marine law
enforcement/public safety services while
promoting and enhancing boating education
opportunities
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
37
STRATEGIES
are the broader approaches taken to accomplish
the goals and objectives to get there
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
38
TACTICS
are the specific tools youre going to use
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
39
MEASURES OF EFFECTIVENESS OR PERFORMANCE
MEASURES
are any of the quantitative measures that can
offer some insights into how well or how
effectively a program is performing can be
individual outcome measures or composites of
output measures.
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
40
OUTPUTS
are the products, services, things produced by
a program activity
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
41
OUTPUT MEASURES (counts)
measure those direct products or the volume
resulting from program activities describe what
the program did/generated or how much was
delivered/produced
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
42
OUTPUT MEASURES

• Numbers of recreational boat registrations
  processed
• Number of boating safety education certificates
  issued
• Number of patrol hours
• Number of vessel safety inspections
• Number of arrests
• OTHERS?

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
43
OUTCOMES
are the intended results or consequences from
carrying out the program activities
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
44
OUTCOME MEASURES
try to assess progress toward the bigger goal,
mission, vision help us understand the ultimate
benefit of the program activities the extent to
which an activity has affected the intended
target did something change or improve as a
result of what we did?
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
45
OUTCOME MEASURES

• Numbers of recreational boat registrations
  processed (output measure)
• Percent of customers satisfied with divisions
  service
• Number of boating safety education certificates
  issued (output measure)
• Percent of boaters with boating safety education
  certificates involved in recreational boating
  accidents within two years of certificate
  issuance
• Number of arrests, of patrol hours, of vessel
  safety inspections (output measure)
• Recreational boating accident rate number of
  boating accidents per what??

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
46
TARGETS OR BENCHMARKS
are the specific, measurable characteristics of
performance the program is trying to achieve
set the bar for program or task accomplishment
within a time period or serve as comparison
points what are we aiming for? how fast do we
want to achieve something?
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
47
TARGETS or BENCHMARKS

• Starting in FY 2007, increase observed adult
  life jacket wear-rate in open motorboats by 3
  percent from the previous years
• 30-minute wait time for customers receiving
  service from division .

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
48
IN SUMMARY . GOALS state what you hope to
accomplish in a big picture sense shorter-term,
longer-term OBJECTIVES break down key elements
in goals and state something measurable that
would let you know youre making
progress MEASURES offer up values that help
determine magnitude or degree and indicate
whether progress is being made TARGETS describe
the numeric performance to achieve within a given
timeframe
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
49
PROGRAM EVALUATIONS

• Assessing Program Status (in general)
• Checklists for Program Evaluation in Selected
  Functional Areas
• Boater Education Program
• RBS Law Enforcement Program
• Approved State Numbering System Compliance

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
50
Ask

• Whats your goal? What do you want? What outcome
  (ultimate benefit) do you want to achieve?
• How would you know if you were achieving it?
• What would it take to get there?

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
51
After you identify your goal(s)

• Break it down into its basic elements
• Create a list of measures candidates
• Test them what would each one tell you?
• Pick only a few of the most important measures
  initially

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
52
For each measure, wheres the data?

• Are the data you need available or attainable?
  Will you need to start counting, gathering or
  calculating something new?
• If a data source is available
• Will you need to collect the data with the same
  or greater frequency?
• Will you need to do anything differently?
• If a data source is not already available
• Will you be able to design a new method /
  calculation?

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
53
Start collecting and tabulating

• Collect the data during a trial period
• Tabulate and develop any accompanying explanatory
  information
• Present the data findings in a predetermined
  format that is clear and understandable
• Review the trial run and decide if adjustments
  need to be made before full-scale implementation
  DID YOU MEASURE WHAT YOU THOUGHT YOU WERE
  MEASURING???

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
54
EXERCISE

• LINKING TO THE
• NATIONAL RBS STRATEGIC PLAN

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
55
DATA and RESEARCH EVERYWHERE or not?The impact
on measurement
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
56
NATIONAL RECREATIONAL BOATING SURVEY

• The Survey Redesign Project
• The Information Needs
• The Current Status
• Whats In It?

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
57
NATIONAL RECREATIONAL BOATING SURVEY

• The Survey Redesign Project
• Technical experts and recreational boating
  stakeholder representatives
• Collaboratory of Partners (COP) primary
  goals, current and future data needs and
  priorities (Aug. 2006 May 2007)

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
58
NATIONAL RECREATIONAL BOATING SURVEY

• The Information Needs
• To collect reliable, consistent data for valid
  safety performance measures, especially exposure
  and risk estimates

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
59
Number of reported injuries per number of
registered boatsNo idea of the number of actual
participants on those boats ... how frequently
theyre out on the boats or the duration of
their boating activities.
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
60
NATIONAL RECREATIONAL BOATING SURVEY

• The Information Needs
• To collect reliable, consistent data for valid
  safety performance measures, especially exposure
  and risk estimates
• To collect more information about the boater
  demographics numbers and types of boats and
  boating activities other indicators for the
  national RBS program and goals

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
61
NATIONAL RECREATIONAL BOATING SURVEY

• The Information Needs
• To collect reliable, consistent data for valid
  safety performance measures, especially exposure
  and risk estimates
• To collect more information about the boater
  demographics numbers and types of boats and
  boating activities other indicators for the
  national RBS program and goals
• To better define and measure the effectiveness
  of state RBS program activities in reducing the
  number of boating fatalities in support of USCG
  performance goals

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
62
NATIONAL RECREATIONAL BOATING SURVEY

• The Current Status
• Federal Register notice USCG requesting public
  comment deadline 9/14/07 Docket
  USCG-2007-28578
• Following comment period, USCG anticipates
  seeking OMB approval end of September
• Implement survey early 2008 seek 2007 data as
  baseline

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
63
NATIONAL RECREATIONAL BOATING SURVEY What
broad research questions?

• About the individual boating participants
• Who are they? Where do they boat? How long is
  their typical boating trip? What types of boats
  do they use? What types of activities are they
  involved in?
• About the boats in use
• Who uses specific types of boats? Where are the
  boats being used? Who owns them? How often and
  how are they used and for what activities?

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
64
NATIONAL RECREATIONAL BOATING SURVEY What
topics?

• Boating participation 2007
• Life participation
• Boat ownership (current)
• Life boat ownership
• Boats they went out on type, days, operate?
• Boat rentals type, days

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
65
NATIONAL RECREATIONAL BOATING SURVEY What
topics?

• Boating activities
• Household characteristics number ages
  boating participation
• Child boating participation life jacket wear
  boating safety course?
• Boat ownership type length taken out
  average hours average persons

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
66
NATIONAL RECREATIONAL BOATING SURVEY What
topics?

• Unregistered boats who took out? where
  operated? months operated states/days
  operated? activities lifetime operating hours
  transported
• Registered boats (on the specific boat sampled)
  length taken out average number of persons
  states operated primary operator lifetime
  operating experience hours under power
  purchased new/pre-owned model year number of
  times launched

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
67
NATIONAL RECREATIONAL BOATING SURVEY What
topics?

• Boater safety life jackets safety equipment
  incidents education alcohol use
• Agreement with boating safety statements
• Non-current boat owners boating participation
• Never owned a boat

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
68
NATIONAL RECREATIONAL BOATING SURVEY What
topics?

• Non-participants who own boats
• Never boated
• Internet usage
• Socioeconomic information
• Annual boat-related spending

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
69
NATIONAL RECREATIONAL BOATING SURVEY Methods
Sample

• Postal mail and telephone
• Mail to sample where registration data are
  shared
• Phone (2 versions) where mail survey not used to
  sample non-boaters boaters who dont own boats
  households with unregistered craft
• Target 20K completed surveys
• Stratifies by region/state coastal-non-coastal
  splits into 12 regions based on geography.
• Target 400 boats/households each state

NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007
70
Vessel Identification System and Memorandum of
AgreementUSCG and the States
NASBLA -- State RBS Program MOE Workshop Sept. 6,
2007