Establishing S'M'A'R'T Goals and Measuring Outcomes

1
Establishing S.M.A.R.T Goals and Measuring
Outcomes

• Presented by
• Jim Spitler, Researcher Evaluation
• Shelba Waldron, Training
• JWB Childrens Services Council of Pinellas
  County
• Revised September 2007

2
Purpose

• Provide an overview the SMART approach to goal
  setting
• Provide an introduction to Goal Attainment
  Scaling and how it is applied
• Review of the MO and SAMIS data entry requirements

3
Why a New Direction?

• Provide a systematic and SMART approach to goal
  setting
• Integrate SMART goals with appropriate measures
  of success
• Integrate goal outcomes with more concrete
  measures of impact

4
Goal Attributes

• Based on evaluation and reflection
• Operationalized as needs, objectives, aims
• Characterized as SMART
• Established with accountability

5
Characteristics of SMART Goals

• Specific
• Measurable
• Attainable
• Realistic
• Time bound

6
Specific

• Specifics help focus and define the overall
  effort and tasks involved
• Goals are no place to be vague ambiguous goals
  produce ambiguous results
• Specifics establish
• Who is involved
• What is to be accomplished
• Where it will be accomplished
• When it should be accomplished
• Which conditions are to be met (requirements and
  constraints)
• Why it is important (purpose or benefit)

7
Measurable

• If you cant measure it, you cant manage it
• Establishing concrete criteria for measuring
  progress helps to
• Stay on track
• Reach target dates
• Observe change
• Know when the goal has been accomplished

8
Attainable

• Goals need to stretch and challenge current
  capabilities
• Goals that are unreachable at the onset, lead to
  negative experiences and frustration
• Attainable does not mean EASY

9
Realistic

• Set the bar high enough for satisfying outcomes
  that participants are both willing and able to
  work toward
• The learning curve should not be a vertical slope
• Ask what conditions would have to exist to
  accomplish the goal

10
Time bound

• Without a timeframe, there is little urgency to
  begin or continue to take action
• Time must be measurable, attainable, and realistic

11
Goal Attainment Scaling

• Developed by Kiresuk and Sherman in 1968
• Measures achievement of treatment or program
  intervention goals
• Can address different kinds of issues
• Can accommodate different numbers of goals
• Produces a Goal Attainment Score to track progress

12
How are goals scaled?

• Identify issues
• Translate issues into at least 2 goals
• Choose a brief label that conveys intent
• Select the TYPE of goal that corresponds to
  pull-down menu categories
• Select a measurable indicator

13
How are Goals Scored?

• -2 Most unfavorable outcome
• -1 Less than expected success
• 0 Expected level of success
• 1 More than expected success
• 2 Best anticipated outcome

14
Goal Attainment Scaling Form
15
Identifying Outcomes Levels

• First, specify the expected (most likely) level
  of outcome (progress anchor) by the end of
  treatment or program intervention
• Consider
• Usual outcomes of the treatment or program
  intervention
• Resources of the client
• Time period allotted for treatment/program

16
Identifying Outcomes Levels (cont)

• Specification should include the nature and
  parameters of what the client will most likely
  achieve
• Quantitative frequency, percentage, score,
  intensity
• Qualitative degree of control, action taken,
  feeling, behavior
• Employment example 1 to 3 applications
  submitted
• Self-esteem example feels negative and positive
  features
  are about equal

17
Identifying Outcomes Levels (cont)

• Second, specify the somewhat more and somewhat
  less than expected levels of outcome (progress)
• Employment example (more than expected) 1
  interview is set
• (less
  than expected) 1 to 3 applications obtained
• Self-esteem example (more than expected) more
  positive features than negative
• (less than expected) more negative
  features than positive
• Finally, specify the minimal expected level of
  progress, and what constitutes best anticipated
  outcome.
• Employment example (best anticipated) Employment
  obtained
• (most unfavorable) No applications
  obtained
• Self-esteem example (best anticipated) General
  positive regard for self
• (most unfavorable) Overall negative
  regard for self - worthlessness

18
Identifying Outcomes Levels (cont)

• Recommend the establishment of a committee to
  review goals with a focus toward outcome (anchor)
  levels
• Is the goal consistent with diagnosed need?
• Are outcomes consistent with goal label?
• Are outcomes clear and unambiguous?
• What information will be use to measure outcomes?
• Just as with goals, outcome levels are SMART

19
Goal Examples
20
Goal Examples (cont)
21
Goal Examples (cont)
22
Common Problems

• Use of jargon or technical language
• Vagueness
• Scale continuity
• Overlapping outcome levels
• Gaps between outcome levels
• Omission of outcome levels
• Inclusion of more than one dimension

23
Exercise and Discussion
24
MOoooooos
25
MOs and SAMIS Data Entry

• Seventy percent (70) of primary (youth/adult)
  participants will achieve significant improvement
  in all identified goal areas as defined in the
  (document). A minimum of two (2) goals must be
  established for all participants. Significant
  improvement is defined as attaining a score of
  zero (0) - the expected level of success - or
  higher on the participant-specific Goal
  Attainment Scaling Form. The Goal Attainment
  Scaling Form will be completed by (who), in
  collaboration with the participant, within one
  (1) month of program intake. Outcome scores will
  be identified and entered by program staff every
  (time period as determined by methodology) after
  program intake or at case closure. Program staff
  will enter outcome scores into SAMIS within two
  (2) weeks of data collection. (CDG Summary)

26
MOs and SAMIS Data Entry (cont)

• Five goals may be entered into SAMIS for each
  participant. Two (2) data elements are entered
  for each goal
• The goal type, from pull-down menus and
• The outcome score (-2 to 2). Goal scores are
  entered at the specified times given in the MO.

27
SAMIS Measurable Objective Screen
28
SAMIS Data Entry Screen
29
SAMIS Pull-Down Menu Screen
30
Pull-Down Menu Categories for Goal Types

• Abuse/Neglect
• Academic achievement
• Adjudication
• Behavioral Issues
• Child Care
• Child Placement/Custody
• Clothing
• Criminal Involvement

• Daily Functioning
• Developmental Milestones
• Domestic Violence
• Educational Improvement
• Emotional Issues
• Employment
• Family Involvement
• Financial Aid

31
Pull-Down Menu Categories for Goal Types (cont)

• Financial Skills
• Food
• Health/Dental Care
• Housing
• Legal Issues
• Literacy
• Parenting Skills
• Recidivism

• Resiliency
• Run-Away
• School Attendance
• Sexual Responsibility
• Strategic Planning
• Substance Abuse
• OTHER

32
Implementing New MOs

• After 9/30/2007, new participants entering a
  program element subject to the new MO will use
  the G.A.S. measurement tool and procedures
• Participants continuing from previous years in
  program elements subject to the new MO may follow
  current measurement procedures during fiscal year
  2007-08
• By the beginning of fiscal year 2008-09
  (10/1/2008) all participants in program elements
  subject to the new MO will use the G.A.S.
  measurement tool and procedures.

33
Continuing Support

• Team support provided by JWB Contract Manager,
  Researcher, Community Planner, Trainer
• Individual agency site visits about 2-4 weeks
  after initial training
• Informal bi-monthly discussion meetings for
  agency program staff
• Quarterly refresher training

34
Additional Information

• Kiresuk, T.J., and Sherman, R.E. (1968). Goal
  attainment scaling A general method for
  evaluating comprehensive community mental health
  programs. Community Mental Health Journal,
  4(6)443-453.
• Kiresuk, T.J. and Lund, S.H. (1979). Goal
  attainment scaling Research, evaluation and
  utilization. In Scholberg, C. and Baker, F.
  (eds.), Program evaluation in the health fields
  (Vol. 2, pp. 214-239), New York Human Sciences
  Press.
• Kiresuk, T.J. Lund, S.H. and Larsen, N.E.
  (1982). Measurement of goal attainment in
  clinical and health care programs. Drug
  Intelligence and Clinical Pharmacy, 16145-153.
• Kiresuk, T.J. Smith, A. and Cardillo, J.E.
  (eds). (1994). Goal Attainment Scaling
  Applications, Theory, and Measurement. Hillsdale,
  New Jersey Lawrence Erlbaum Associates,
  Publishers.
• MM Research and Evaluation Associates. GAS
  Databank. www.gasbankscales.com