Title: Improving Performance: The Five Step Process
1Improving Performance The Five Step Process
- Mimi Lufkin Chief Executive Officer
- Rick Larkey State Facilitator
- National Alliance for Partnerships in Equity
- Oklahoma November 13-14, 2008
2The Five Step Process
3Perkins Act Accountability
- Core Indicators on Nontraditional CTE
- Participation in CTE programs preparing students
for nontraditional fields - Completion of CTE programs preparing students for
nontraditional fields
4Nontraditional Fields
- Occupations or fields of work, including careers
in computer science, technology, and other
current and emerging high skill occupations, for
which individuals from one gender comprise less
than 25 percent of the individuals employed in
each such occupation or field of work.
5Document Performance Results
- GOAL Understand the problem completely before
you seek solutions
How do you analyze performance data? What
questions should be addressed? What tools and
methods can be used to present and analyze
data? How should data quality problems be
considered in analyzing data?
6Perkins Accountability Measure
- Participation Rate
- underrepresented students participating in NT
CTE - all students participating in NT CTE
- OR
- of females enrolled in pre-engineering
- All students (males and females)
- enrolled in pre-engineering
7Perkins Accountability Measure
- Completion Rate
- underrepresented students completing NT CTE
- all students completing NT CTE
- OR
- of females completing pre-engineering
- All students (males and females)
- completing pre-engineering
8Data Quality Concerns
Alignment, Scope, Reliability, Time Coverage
- Alignment Extent to which instruments accurately
measure what theyre supposed to measure - How are participation s generated? By teachers?
Districts? - Scope Breadth of measurement with respect to
state-identified performance outcomes - Do districts measure single parents in programs?
9Data Quality Concerns
Alignment, Scope, Reliability, Time Coverage
- Reliability Consistency in conducting
measurements - Do all districts define concentrator similarly?
- Timing When measurement is made
- Do some districts measure at diff. times of year?
- Coverage Base of students included in
measurement - Include all secondary students? Only
concentrators?
10Data CollectionDisaggregation required in
Perkins IV
- Special Population
- Underrepresented gender students in a
nontraditional CTE program - Single Parent
- Displaced Homemaker
- Limited English Proficiency
- Individuals with a Disability
- Economically Disadvantaged
- Race/Ethnicity
- American Indian or Alaskan Native
- Asian or Pacific Islander
- Black, non-Hispanic
- Hispanic
- White- non-Hispanic
11Student Populations for Data Comparisons
12Recommended Analyses
- Trends
- At least 2 years
- Prefer 3-5 years
- Site specific
- Statewide
- District
- School/College
- Programs
13Benchmarking
- State performance level
- Best performer in state
- National program benchmark
- Selected peer benchmark
- Set your own benchmark
14Perkins Accountability Resources
- Peer Collaborative Resource Network
- www.edcountability.net
- National Alliance for Partnerships in Equity
- Growing Pains
- Nontraditional SOC/CIP Crosswalk for Males and
Females - www.napequity.org
- Your State Agency Website
15National Program Evaluations
- Project Lead the Way
- Annual Assessment Report for 2006-07
- Report 17 female enrollment nationally
- Cisco Networking Academy Gender Initiative
- Set a benchmark for programs at 30 female
enrollment - National Center for Women in Information
Technology - Report on the Status of Women in IT
16Other Data Sources
- National Center for Educational Achievement
Just 4 the Kids - National Center for Educational Statistics
- Southern Regional Education Board Data Library
- NCLB State and Local Report Cards
- State and Local Participation and Achievement
Data
17Perkins Consolidated Annual Report Data
- State, School, and Program
- Level Analysis
18State Level Perkins Nontraditional Data Analysis
19Schools Included in Statewide Analysis
- Caddo Kiowa Tech Center Fort Cobb
- Canadian Valley Tech Center El Reno
- Gordon Cooper Tech Shawnee
- Great Plains Tech Center Lawton
- Meridian Tech Center Stillwater
- Metro Tech Oklahoma City
- Moore Norman Tech Center Norman
- Red River Tech Center Duncan
- So. OK Tech Center Ardmore
- Tulsa Tech - Tulsa
20Statewide Secondary Participation Data
Secondary Participation All Nontrad Programs
04-05 13.50
05-06 13.58
06-07 13.96
21Statewide Secondary Participation Data
Sec Participation All Nontrad Programs Nontrad Programs for Females Nontrad Programs for Males
04-05 13.50 13.64 13.29
05-06 13.58 13.74 13.33
06-07 13.96 13.52 14.61
22Statewide Secondary Participation
23Statewide Adult Participation Data
Adult Participation All Nontrad Programs
04-05 13.23
05-06 14.53
06-07 12.44
24Statewide Adult Participation Data
Sec Participation All Nontrad Programs Nontrad Programs for Females Nontrad Programs for Males
04-05 13.23 15.38 10.50
05-06 14.53 15.56 13.20
06-07 12.44 13.46 11.13
25Statewide Adult Participation
26Statewide Secondary Nontraditional Program
Participation
Pre-Eng /PLTW Electrical Trades CMP Rep/Netw Tech Auto Service Tech
04-05 26.81 2.36 11.43 5.75
05-06 19.15 0.00 14.46 6.94
06-07 19.50 0.85 10.71 5.20
27Statewide Secondary Nontraditional Program
Participation
28Statewide Adult Nontraditional Program
Participation
Pre-Eng /PLTW Electrical Trades CMP Rep/Netw Tech Auto Service Tech
04-05 50.00 6.15 17.98 8.12
05-06 25.00 15.19 18.75 7.78
06-07 N/A 7.79 16.46 9.80
29Statewide Adult Nontraditional Program
Participation
30Statewide Secondary and Adult Data
- What does the data say or suggest?
- How has state performed on the measure?
- What happened to the results over time?
- What questions arise from this data?
31School Level Analysis
32School Secondary Participation
Year All Nontrad Programs Programs Nontrad for Females Programs Nontrad for Males
04-05 12.17 11.40 13.42
05-06 10.83 9.13 13.10
06-07 12.52 9.88 15.89
33School Secondary Participation
34School and State Secondary Participation
35School Adult Participation
Year All Nontrad Programs Programs Nontrad for Females Programs Nontrad for Males
04-05 11.40 9.96 12.70
05-06 15.10 11.73 18.01
06-07 10.97 8.77 12.80
36School Adult Participation
37School and State Adult Participation
38School Level Data
- How has the school performed on the participation
measure? - How has the college performed as compared to the
state average on the measure? - What happened to the results over time?
- What questions arise from this data?
39Case Study
40School Level Perkins Worksheet
41Nontraditional Participation 04-05 04-05 04-05 05-06 05-06 05-06 06-07 06-07 06-07
Num1 Denom2 Num Denom Num Denom
STATE LEVEL STATE LEVEL STATE LEVEL STATE LEVEL STATE LEVEL STATE LEVEL STATE LEVEL STATE LEVEL STATE LEVEL STATE LEVEL
All Nontraditional Programs 1076 7968 13.50 1069 7871 13.58 1122 8039 13.96
Nontraditional Programs for Females 670 4913 13.64 668 4862 13.74 652 4821 13.52
Nontraditional Programs for Males 406 3055 13.29 401 3009 13.33 470 3218 14.61
LOCAL LEVEL LOCAL LEVEL LOCAL LEVEL LOCAL LEVEL LOCAL LEVEL LOCAL LEVEL LOCAL LEVEL LOCAL LEVEL LOCAL LEVEL LOCAL LEVEL
All Nontraditional Programs
Nontraditional Programs for Females
Nontraditional Programs for Males
PROGRAM LEVEL PROGRAM LEVEL PROGRAM LEVEL PROGRAM LEVEL PROGRAM LEVEL PROGRAM LEVEL PROGRAM LEVEL PROGRAM LEVEL PROGRAM LEVEL PROGRAM LEVEL
Program
Program
Program
42Program Level Perkins III Analysis
43School Secondary Nontraditional Program
Participation
Pre-Eng /PLTW Electrical Trades CMP Rep/Netw Tech Auto Service Tech
04-05 3.85 2.94 0.00 8.52
05-06 2.78 0.00 0.00 3.39
06-07 14.92 0.00 0.00 2.81
44School Secondary Nontraditional Program
Participation
45State and School Secondary Nontraditional Program
Participation
46School Adult Nontraditional Program Participation
Pre-Eng /PLTW Electrical Trades CMP Rep/Netw Tech Auto Service Tech
04-05 N/A 6.67 9.09 14.44
05-06 0.00 7.69 18.18 5.00
06-07 N/A 0.00 10.00 10.17
47School Adult Nontraditional Program Participation
48State and School Adult Nontraditional Program
Participation
49Program Level Data
- What does the data say or suggest?
- How has the state performed in each measure?
- What happened to the results over time?
- What questions arise from this data?
50Program Level Perkins Worksheet
51Nontraditional Participation 04-05 04-05 04-05 05-06 05-06 05-06 06-07 06-07 06-07
Num1 Denom2 Num Denom Num Denom
STATE LEVEL STATE LEVEL STATE LEVEL STATE LEVEL STATE LEVEL STATE LEVEL STATE LEVEL STATE LEVEL STATE LEVEL STATE LEVEL
All Nontraditional Programs 1076 7968 13.50 1069 7871 13.58 1122 8039 13.96
Nontraditional Programs for Females 670 4913 13.64 668 4862 13.74 652 4821 13.52
Nontraditional Programs for Males 406 3055 13.29 401 3009 13.33 470 3218 14.61
LOCAL LEVEL LOCAL LEVEL LOCAL LEVEL LOCAL LEVEL LOCAL LEVEL LOCAL LEVEL LOCAL LEVEL LOCAL LEVEL LOCAL LEVEL LOCAL LEVEL
All Nontraditional Programs
Nontraditional Programs for Females
Nontraditional Programs for Males
PROGRAM LEVEL PROGRAM LEVEL PROGRAM LEVEL PROGRAM LEVEL PROGRAM LEVEL PROGRAM LEVEL PROGRAM LEVEL PROGRAM LEVEL PROGRAM LEVEL PROGRAM LEVEL
Program
Program
Program
52The Five Step Process
53STEP TWO
54Why Search for Root Causes?
- Keep from fixating on the silver bullet
strategy - Identify the conditions or factors that cause or
permit a performance gap to occur - Two types of causes
- Direct cause (i.e. instructional practice)
- Indirect cause (i.e. teacher training)
55How to Identify Root Causes
- Search for most direct highest impact causes
- Employ a systematic evidence-based process
- Draw on current research and evaluation
- Formulate and test theories or hypotheses
- Use multiple methods and data sources
- Note, youre likely to find multiple causes
56Phase 1 Identify Potential Causes
- Review Research Literature
- Review Program/Institutional Evaluations and
Effectiveness Reviews - Conduct Focus Groups
- Peer Benchmarking
- Interviews Surveys
- Brainstorm
57Resources on CD
- Survey Instruments
- Destination Success
- How to Conduct Interviews
- How to Conduct Focus Groups
58Other Resources
- The New Look
- Self-Study
- Illinois Center for Specialized Professional
Support
59Phase 2 Analyze and Evaluate Potential Causes
- Group Causes Into Two Categories
- Group 1 Causes Within Your Control
- School scheduling
- Classroom climate
- Faculty awareness and capacity
- Group 2 Causes Outside Your Control
- Media representation
- Family demographics
60Phase 3 Test and Evaluate Potential Causes
Within Your Control
- Select root causes that
- Have the strongest theory and evidence to support
them - Focus on direct causes of performance gaps
- Address the most critical needs
- Provide the best opportunity to have high impact
on performance - Are supported by stakeholders who will help
develop and implement solutions - (See page 17 of the OVAE Guidebook)
61Review Research Summary
- Root Causes Tables
- Chapter 6 Tables 16, 17, 19, and 20 from,
Improving Performance on Perkins III Core
Indicators Summary of Research on Causes and
Improvement Strategies - Authors Robert Sheets, Mimi Lufkin, and David
Stevens for the National Centers for Career
Technical Education - Available at www.napequity.org
62Root Causes
- Career Guidance Materials and Practices
- Access to and Participation in STEM
- Instructional Strategies
- Classroom Climate
- Role Models
63Root Causes
- Early Exposure
- Curriculum Materials
- Occupational Choice/Segregation
- School Climate
- Student Attitudes
64Root Causes
- Support Services
- Student Isolation Based on Gender
- Self-efficacy
- Attribution Theory
- Locus of Control
65Review Research Summary
- Bias Literacy
- A review of concepts in research on
discrimination - By Ruta Sevo, Ph.D. and
- Daryl E. Chubin, Ph.D.
66Root Causes
- Bias and Discrimination
- Gender schema
- Accumulative Advantage
- Stereotype threat
- Implicit bias
67Group Root Causes Activity
- In groups of 5
- Review the root causes cards
- Arrange the root causes by your groups sense of
their impact and relationship to students in
programs nontraditional by gender - Post the cards on the wall in whatever
arrangement best fits your groups thinking
68Individual Root Causes Activity
- Place stickers on the posters for the four root
causes that most significantly impact students
who are in, or considering entering,
nontraditional programs for their gender - On the other root causes poster
- Write any additional root causes that have not
been identified
69Case Study
70Implementation Planning
71The Five Step Process
72STEP THREE
73Choose Best Solutions
- Dont be too quick to adopt best practices before
getting the facts straight - How do you identify possible strategies and model
practices? - How do you evaluate strategies and models?
- How do you compare and assess alternative
solutions and make a decision?
74Find and Evaluate Solutions
- Failure is expensive
- Build consensus among staff and stakeholders
- Get full support and commitment
- Select full range of choices be creative
- Implement systematic analysis
75Identify Potential Strategies and Models
- Review What Others Propose
- NSF- New Formulas for Americas Workforce
- Benchmark Peers and Leading Performers
- Programs and Practices That Work
- Best Practices in STEM Education
- EEES Best Practices
- Develop Your Own Solutions
76Review Research Summary
- Strategies
- Chapter 6 Tables 18 and 21
- Improving Performance on Perkins III Core
Indicators Summary of Research on Causes and
Improvement Strategies - Authors Robert Sheets, Mimi Lufkin, and David
Stevens for the National Centers for Career
Technical Education - Available at www.napequity.org
77Assessing and Comparing Alternative Strategies
- Sound Theory
- Strong Evidence
- Costs/Time of Further Testing
- Resources
- Stakeholder Support
78Strategies for Recruitment
79Strategies for Recruitment
- Review career guidance materials and practices
for gender bias and nontraditional exposure and
support - Guidelines for Identifying Bias in Curriculum and
Materials - Safe Schools Coalition
- EEES-NAPE-PLTW Guidance Counselor Presentation
- Am I a Fair Counselor
- Destination Success, MAVCC
- Could This Be Your Life?
- Illinois Center for Specialized Professional
Support
80Strategies for Recruitment
- Invite, involve and educate parents
- Talented Girls Bright Futures
- Publication by Project Lead the Way
- American Careers Magazine
- Expanding Career Options Issue
- Career Communications, Inc.
81Strategies for Recruitment
- Conduct middle school programs
- Minot Public Schools
- Minot, North Dakota
- Programs and Practices That Work
- 2005 Award Winner
82Strategies for Recruitment
- Provide role models and mentors
- Girls E-Mentoring in Science, Engineering and
Technology GEM-SET - USDOL, Womens Bureau
- Engineer Girl
- National Academy of Engineering
83Strategies for Recruitment
- Conduct targeted recruitment activities
- Summer Camps
- Cisco Gender Initiative Strategies
- I am an Engineer
- Cisco Systems, Inc.
84Strategies for Recruitment
- Conduct pre-technical training programs
- Rosies Girls
- Northern New England Tradeswomen
- TechBridge
- Chabot Space and Science Center
85Strategies for Recruitment
- Collaborate with community-based organizations
- Operation SMART, Girls, Inc.
- Expanding Your Horizons
- Girl Scouts
- National Girls Collaborative Project
86Strategies for Recruitment
- Conduct professional development with teachers at
all levels - Generating Expectations for Student Achievement,
Graymill - STEM Equity Pipeline
- Career Technical Education Equity Council
87Strategies for Recruitment
- Implement and model gender-fair institutional
strategies - Checking Your School for Sexism
- Destination Success, MAVCC
- Gender Equity Item Bank
- Midwest Equity Assistance Center
88Strategies for Retention
89Strategies for Retention
- Evaluate all school and curriculum materials for
gender bias and positive nontraditional images - Gender Equity Tip Sheets
- Bias Evaluation Instrument
- Nova Scotia Department of Education
- Curricular Bias, David Sadker
90Strategies for Retention
- Increase teacher and administrator quality and
equity-capacity through professional development - The Equity Principal
- Graymill
- NAPE Professional Development Institute
- Washington, DC
91Strategies for Retention
- Increase competence in diversity and sexual
harassment prevention - Gender Equity Tip Sheets
- Tolerance.org
- Southern Poverty Law Center
- Project Implicit, Harvard University
92Strategies for Retention
- Conduct nontraditional student support groups and
peer counseling - Computer Clubhouse
- Bostons Museum of Science
- NASA Harriett G. Jenkins Pre-Doctoral Fellowship
Program - (Facebook Group)
93Strategies for Retention
- Provide nontraditional role models, mentors, and
job shadowing - IGNITE, Seattle Public School System
- MentorNet
- How to Plan and Facilitate a Job Shadowing
Experience - Destination Success, MAVCC
94Strategies for Retention
- Invite, involve and educate parents
- Trailblazers
- VA Department of Education
- Ways for Parents to Support Expanded Occupational
Options - Destination Success, MAVCC
95Strategies for Retention
- Provide a continuum of support services
- Tutoring
- Child care
- Transportation
- Financial Aid
- Books, Equipment, Tools, Clothing
- Tuition
- Modification of Curriculum, Equipment
- Student/Teacher Aides
- More
96Strategies for Retention
- Invite, involve, and educate business
- Cisco Systems, Inc. Gender Initiative
- Cisco Systems, Inc.
- Society of Women Engineers
97Resources on CD
Dont Forget These Additional Teacher Focused
Resources!
- Gender Equity Tip Sheets
- Teacher Resources Folder
98Case Study
99The Five Step Process
100Why Evaluate?
- Make sure your improvement strategy works
somewhere before you attempt to apply it
everywhere
101Pilot Test
- Choose a Study Design
- Random Assignment with a Control Group
- Comparisons with Similar Populations
- Comparing Individuals Against Themselves
102Pilot Test
- Select Pilot Sites
- Student Characteristics
- Site Characteristics
- Select Outcome Measures
- Short-term
- Long-term
103Pilot Test
- Identify Data Sources
- Train Pilot Site Staff
- Evaluate Initial Results
104Assessment Resources
- STEM Equity Pipeline Webinars
- May
- June
- Archived at www.stemequitypipeline.org
- Assessing Women and Men in Engineering
105The Five Step Process
106Questions?
National Alliance for Partnerships in Equity
- Mimi Lufkin
- Executive Director
- P.O. Box 369
- Cochranville, PA 19330
- 610-593-8038 phone
- 610-593-7283 fax
- mimilufkin_at_napequity.org
Rick Larkey STEM Equity Pipeline
Consultant rlarkey_at_napequity.org