Building and Sustaining Quality in NRS Data: Strategies for Program Improvement

1
Building and Sustaining Quality in NRS
Data Strategies for Program Improvement

• American Institutes for Research
• and the U.S. Department of Education
• Office of Vocational and Adult Education
• July 2008

2
Welcome! Were Pleased to See You!

• Introductions by State
• Take one minute to state the following
• Your state
• Names and positions of each member of your team
  attending this training
• Total number of staff in your state adult
  education office
• Number of staff who are new to your state office
  within the past three years

3
Objectives

• By the end of this session, participants will be
  able to
• Identify NRS requirements and systems needed for
  producing quality data
• Describe a four-step process for building and
  sustaining change
• Work with Data Planning Tool to plan a change for
  improving data or programs
• Refine approach and plan for involving staff and
  stakeholders in the change process.
• (Refer to H-1)

4
Agenda Day 1

• Welcome, Introductions, Objectives, Agenda
• A Quick Guide to the NRS
• The NRS Jeopardy Game!
• Building Data Quality Activity for Self-
• Reflection
• Building and Sustaining Change for
• Program Improvement Activities for
• Self-Reflection
• (Refer to H-2)

5
Agenda Day 2

• Demonstration of the Tool to Build and Sustain
  Change
• Work in State Teams, Using Tool
• (Refer to H-2)

6
Agenda Day 3

• State Teams Report on Their Change Strategy and
  Implementation Plans
• Getting Staff Buy-in and Ownership
• Data Stories
• Resources Needed?
• Evaluation and Wrap-Up
• (Refer to H-2)

7
This is NRS Jeopardy!
8
Rules of the Game

• Five Table Teams (15)
• Select a team member as spokesperson to pick
  category and cell and to respond for the team
• Signal that your team wants to answer by holding
  up the placard on your table
• Judges decision is final concerning the accuracy
  of a response
• When a team gives a correct response, the dollar
  amount on the cell is added to the teams score
  when the response is incorrect, the dollar amount
  is subtracted from the teams score
• When a team gives an incorrect response, other
  teams may signal that they want to respond by
  holding up placard
• Scoreboard is posted on flipchart
• Game is over when all cells have been selected
• Winning team is the team with the highest score
  

9
Making Data Dummies Obsolete

• It all boils down to the following actions
• Determine the data you want to collect.
• Decide how to organize it.
• Define the data system to
• manage it.
• Identify and prioritize questions
• to ask of your data.
• Learn how to access it and
• extract reports.
• Make sense?

10
Building Data Quality Three Essential,
Interrelated Systems
1
2
3
11
Data Collection System

• Review data collection times, data elements, and
  quality control procedures
• (Refer to H-3ab,
• Quality Data Collection Self-Reflection)

12
The Importance of Quality Data
One of the greatest limits to data-based change
is lack of access to quality data.

• No confidence in data?
• No data quality?
• -No change

13
Can I Use This Data to Make Decisions about
Program Improvement?
14
The Care and Feeding of Quality

• 
  

Your first important consideration
Time Commitment Resources
Quality
15
I forgot to collect thisoh well, whats a little
missing data?
Attitudes
I dont have time to do this. Let someone else
handle it!
We have a good data collection processI dont
need to check on anything.
I dont know what goes in this area of the
reportIll just guess.
What does it matter? Who pays attention to this
data anyway?
16
2. Database System for Storing and Retrieving
Data

• What Exactly Is a Database System?
• A collection of data organized in tables, which
  can be accessed and manipulated, without having
  to restructure the tables
• Elements of a Database System
• A storage system
• Data structures
• Manipulation tools

17
Advantages of a Database

• Allows you to
• Analyze sophisticated correlations more easily
  because relationships are established between
  data sets
• Make decisions based on information derived from
  data
• Streamline operations
• Organize data and eliminate
• Inconsistent data
• Missing data
• Redundant data

18
Requirements of the Database System

• Electronic, relational database with data at the
  individual student level
• Ability to disaggregate data at the site or
  classroom level
• Ability to produce NRS reports

19
Data-Driven Decision-Making for Quality Control
The Power of a Relational Database

• Data-driven decision-making requires
• an integrated system of collecting
• data from many different sources.

20
Systemic

• All parts of the system (state and local
  programs) need to be vested in the collection of
  data.
• Data collection systems must be in place at all
  levels.
• Data needs to be collected on programs, classes,
  students.

21
Integrated

• All data sets need to be connected so
  relationships can be established
• Queries made
• Reports generated
• Correlations and relationships analyzed.

22
Adult Education Goal

• Ultimate goal is to improve teacher quality and
  impact achievement for all students.
• Data provides the means to do this.
• Relational database is the
• engine that makes this
• possible.

23
Six Factors Critical to the Success of a Data
System

• Reporting Requirements
• Range of Data
• Quality of Data
• Granularity of Data
• Privacy, Security, and Data Integrity
• Interoperability

24
Common Oversights and Errors
in Data System Planning
Ooops!
Insufficient Attention to Reporting Needs
Lack of Ongoing Training and Support Needs
Insufficient Attention to Usability
Lack of a Test Plan with Realistic Operational
Scenarios
Failure to Leverage States Existing Technology
Infrastructure
25
3. Monitoring System

• Measures of
• Outcomes
• Program Processes
• Data Processes
• Three Ways to Collect These Data
• Desk Reviews
• Onsite Monitoring
• Desk Monitoring
• Monitoring Reports
• Descriptive Reports
• Management Reports
• Reports for evaluation and program improvement

26
Measures for Program Monitoring
27
Methods for Collecting Data for Monitoring Desk
Review

• What Review of existing documents, e.g.,
  proposals, budgets, progress reports, program
  improvement plans, NRS data reports
• Advantages Simple, inexpensive, states already
  have these documents
• Disadvantages Data may be outdated, no way to
  verify validity or accuracy of data reported,
  limited data collected

28
Methods for Collecting Data for Monitoring
Onsite Monitoring

• What Onsite visits to local programsincludes
  interviews with staff and students, class
  observations, review of student records.
• Advantages Interaction provides insights about
  program process measures
• Disadvantages High cost in terms of travel and
  staff time out of the office, large state often
  cannot visit all programs annually

29
Methods for Collecting Data for Monitoring Desk
Monitoring

• What Desk monitoring using a set of reports from
  State NRS data system.
• States decide on measures, standards that define
  good performance, and a rubric to provide an
  overall rating.
• Advantages Inexpensive and effective
• Disadvantages Time to set up the system

30
Types of Monitoring Reports

1. Basic Descriptive Reports
2. Management Reports
3. Reports for Evaluation and Program Improvement

31
Monitoring Reports
Descriptive Reports Management Reports Reports for Evaluation Program Improvement
Feature Simple lists or tables Data broken out by individual program Tables charts graphs Data broken out by individual program Tables charts graphs
Informa-tion Display Attendance Pre-/posttest scores Learner goals Comparisons across sites, classes, teachers Comparisons against state standards Comparisons of performance across programs, site, classes or against state standards Trends over time
Use For Monitoring Review basic, minimum info about the program Determine if programs provide appropriate services, meet student needs, collect data according to state policies Check on compliance to policy or data quality Identify areas needing better data quality Identify areas for program improvement Evaluate change efforts Obtain on as-needed basis for specific purposenot part of regular monitoring.
32
Descriptive Report Purpose
Class lists Provides basic contact information for use by teachers
Student profile report Enables program staff to review individual student needs, goals, and achievements
Program profile Enables states to review demographic snapshots of each program useful for planning and understanding data trends
Attendance report (by class) Enables teachers to monitor student attendance for their classes
Student goals and achievements Provides detailed student information for conducting follow-up surveys
Student posttest planning report Provides list of students who are nearing posttesting time, based on contact hours
33
Management Report
Completion Rates for Beginning and Low-Intermediate ABE by Program,Performance Standard, and State Average Completion Rates for Beginning and Low-Intermediate ABE by Program,Performance Standard, and State Average Completion Rates for Beginning and Low-Intermediate ABE by Program,Performance Standard, and State Average Completion Rates for Beginning and Low-Intermediate ABE by Program,Performance Standard, and State Average Completion Rates for Beginning and Low-Intermediate ABE by Program,Performance Standard, and State Average Completion Rates for Beginning and Low-Intermediate ABE by Program,Performance Standard, and State Average Completion Rates for Beginning and Low-Intermediate ABE by Program,Performance Standard, and State Average
Beginning ABE Beginning ABE Beginning ABE Low-Intermediate ABE Low-Intermediate ABE Low-Intermediate ABE
Perform-ance Standard Percent Completed Enrolled Perform-ance Standard Percent Completed Enrolled
State Average 20 28 5,442 25 32 6,055
Program 1 20 21 591 25 27 450
Program 2 20 49 61 25 50 70
Program 3 20 14 365 25 18 225
34
Management Report Comparison of Program to
State Average for Goal Setting
35
Report for Program Improvement Goal Setting
over Time
36
The POWER of the Pyramid
Data Use
Data Quality
37
Steps to Build and Sustain Change A Continuous
Improvement Process
38
So, lets get started!

• Begin by examining
• your data.

39
On Data Overload??
Youve got the datanow what?
40
Dont be a DRIP

• Data
• Rich
• Information
• Poor

Commit to Analyzing and Using Your Data
41
Step 1. Examine Your Data to Assess Where You Are

• Refer to H-4ab,
• Questions for Program Self-Review

42
Finding Your Way Through The Data Smog
                                          
"Where is the knowledge that is lost in
information?" - T.S. Eliot
43
Clearing the Smog Getting Under the Numbers
44
Analyze DataWhat does this tell us?
Review the NRS Data Detective Training for
suggestions for examining and analyzing your data.
45

• Helpful Hint

Provide support to make data more accessible to
the people who need to use it! Give them a piece
of the pie. Teach them how to read and use the
data.
46
Step 2. Identify Your Goals and Expected Outcomes

• Goals Must be SMART
• Specific
• Measurable
• Achievable
• Relevant
• Timely

Ask yourself Where are we going?
47
Examples of SMART Goals

• Example 1 We will have a ten-percent increase in
  enrollment of non-Hispanic ESL students by the
  end of 2009, as a result of our targeted
  promotional efforts in (specific) neighborhoods.
• Example 2 Adult education instructors in our
  program will use data reports at least weekly to
  assess student educational progress and plan for
  instruction, as a result of a series of regional
  trainings that the state adult education office
  will conduct throughout the summer of 2008 on the
  topic of data-driven decision-making.
• Example 3  Our adult education program will
  convert to a Web-based data collection and
  reporting system that will be operational in five
  counties by July 2009, ten counties in 2010, and
  fully operational statewide by July 2012.

48
Be Sure To

• Prioritize your goals
• Establish program outcomes and results indicators
  
• When its all over, how
• will you know if the effort
• was successful?

49
Step 3. Use Data to Plan a Strategy

• Having data does not necessarily mean that they
  will be used to drive decisions or lead to
  improvements. RAND
  Study
• If you dont use data, youre making decisions in
  a fog.
• The consequences of not using data We keep doing
  the same things over and over and expect
  different results.
• The evil is half-cured whose cause we know.
• - William Shakespeare

50
Determine StrategiesHow are we going to get
there?

• Three critical actions
• Examine all relevant data related to your goals.
• Consider the resources and limitations to
  determine the parameters of what you can do.
• Select and plan out how you will implement the
  strategy.

51
Targeting the Change Strategy

• Consider
• Wherestatewide or focused on specific programs?
• Whoother stakeholders to support effort?
• Howprocesses you will use roles for staff and
  partners?
• Timelineshort- and long-range outcomes? Persons
  responsible for various actions? Evaluation
  strategy?
• Staff Buy-in and OwnershipInclude staff in
  development of change strategy

52
The results came first the buy-in came next.
Data-driven instruction does not require buy-in,
it creates it. - Paul Bambrick-Santoyo, Data
in the Drivers Seat, Educational Leadership,
December 2007/January 2008, P.46
53
Planning for Change
Consult-ant
54
Step 4. Implement andEvaluate Your Strategy

• Questions for consideration
• Did the change occur as planned?
• How did the change affect the program improvement
  effort?
• What were the outcomes?
• Was the strategy successful? In what way?
• Were there unintended consequences? What were
  they?
• What were the limitations? Did we anticipate them
  and plan well to address them?
• Did we find additional resources along the way?
• What were the successes as well as the
  challenges?
• How can we refine/revise our plan so that it runs
  more smoothly in the future?
• How can we revise the plan to account for staff
  feedback and/or data analysis?
• What are the issues that might challenge the
  sustainability of this change?

55
Steps 2, 3, and 4

• Refer to
• H-5ab
• Step 2. Identify Your Goals and Expected
  Outcomes
• H-6ab
• Step 3 Use Data to Plan a Strategy
• H-7
• Step 4 Implement and Evaluate Your Strategy

56
Using the Data Pyramid
57
Level 1
58
Level 2
59
Level 3
60
Level 4
61
Introducing.

• The Tool for Building
• and
• Sustaining Change

62
Content of Report-out

• Step 1 Show table or graph of data you used to
  assess where you are
• Step 2 State your identified goal
• Step 3 Describe your strategy
• Step 4 Share how you will implement and evaluate
  your strategy

63
Getting Staff Buy-in and Ownership Refer to H-8
Data StoriesRefer to H-9a Jigsaw reading and
sharingH-9b Telling Your Own Data Story
64
Comments/Questions/Reactions?

• Next steps?
• Resources needed?
• Challenges you anticipate?
• Plans to get staff to own the change process?

65

• Like pushing on a giant, heavy flywheel,
• it takes a lot of effort to get the thing moving
  at all,
• but with persistent pushing in a consistent
  direction over a long period of time,
• the flywheel builds momentum, eventually hitting
  a point of breakthrough.
• - Collins, J. Good to Great. New York,
  NY
• Harper Collins Publishers Inc., 2001.

66

• When you let the flywheel do the talking,
  you dont need to fervently communicate your
  goals.
• People can just extrapolate from the momentum of
  the flywheel for themselves Hey, if we just
  keep doing this, look at where we can go!
• As people decide among themselves to turn the
  fact of potential into the fact of results, the
  goal almost sets itself.
• Collins, J. Good to Great. New York, NY
• Harper Collins Publishers Inc., 2001.

67
Thank You and Best Wishes as You Continue Your
Journey to Quality Data and Continuous Program
Improvement!

• Let us know how we can help.
• NRS_at_air.org
• Web site www.NRSweb.org