Evaluation

1
Evaluation

• EnAct Campuses
• Sonoma, Humboldt, Chico, San Francisco, San Jose,
  Fresno, Bakersfield, Pomona
• Presentation
• July 11, 2007
• Public Works, Inc.
• Mikala L. Rahn, PhD
• Patty ODriscoll, MPA

2
Who Are We?

• Public Works, Inc. a nonprofit in Pasadena
  dedicated to working with communities,
  government, schools and parents by providing
  services and resources to educate and inform
  children, youth and families. Our work is in
  three areas
• Education Reform
• Workforce Development
• Intervention/Prevention

3
Evaluation Goals

• Measure the success of participating faculty and
  their students with disabilities overtime as well
  as compared to students without disabilities on a
  number of measures
• Monitor the project based on federal requirements
  and offer program improvement strategies
• ? Improve the consistency of measurement across
  the institutions

4
Research Questions

• 1) Which of the 14 elements of the Universal
  Design for Learning (UDL) are most important or
  are they all equal?
• 2) How does faculty training in UDL impact
  student resilience (persistence/graduation)? Are
  there more appropriate courses or faculty to
  target to affect resilience?

5
Research Questions

• 3)Why do faculty participate?
• 4)How is the use of online and other technologies
  cost-effective, beneficial, and scalable for
  rural and remote regions serving the target
  populations?
• 5)Did the project reach the target segments of
  the CSU system with information about research
  and developments in effective teaching of
  students with disabilities?

6
Balanced Evaluation Approach

• Process measures focus on how a program is
  implemented and the outcome measures focus on the
  results of the program or intervention.
• Public Works, Inc. is using both quantitative and
  qualitative data collection methods

7
Methodology

• ?Baseline data from participating institutions
  annual data collection
• ?Attendance data at workshops and institutes
• ?Preliminary Assessment of Faculty Implementation
  of UDL
• ?UDL Training Evaluation
• Faculty Activity Survey/Assessment
• ?Course Artifacts
• Web user statistics

8
Federal Grant Performance Report (GPR)

• ObjectivesMeasures...Targets...Performance Data
• The difference between the rate at which students
  with documented disabilities complete courses
  taught by faculty trained in project activities,
  and the rate at which other students complete
  those courses.
• The percentage of faculty trained in project
  activities that incorporate elements of training
  into their classroom teaching.

9
Federal Grant Performance Report (GPR)

• ObjectivesMeasures...Targets...Performance Data
• Collaboration to Ensure Access to a Quality
  Postsecondary Education
• Technology to Ensure Access to a Quality
  Postsecondary Education
• Dissemination of EnACT Content and Processes

10
Program Improvement

• How do we better serve students with
  disabilities?
• How do we better support faculty?
• What do we need to know and be able to do?

11
Overall Evaluation Goals

• Keep the grant by meeting the requirements
• Learn lessons from each other
• Support faculty and students

12
Summary of ResultsAnnual Performance Report

• Goal 1 Collaboration
• EnACT Leadership Team, Advisory Committee, and
  campus-based Communities of Practice established
• CSU Fresno, Bakersfield and Pomona to join
  current collaborative
• CSU ATI Initiative partnership at 2007 Institute
• UDL workshops attended by 194 attendees-target
  was 100

13
Summary of ResultsAnnual Performance Report

• Goal 2 Technology to ensure access
• Enhancement of AIM resource specifications for
  authoring
• 13 AIM resources proposed or published to date

14
Summary of ResultsAnnual Performance Report

• Goal 3 Dissemination of EnACT
• Enhancements of Web site
• Dissemination at 11 professional conferences or
  Webcasts

15
Summary of ResultsAnnual Performance Report
Highlights

• Project Specific Performance Measures Course
  Completion Rate
• Students with disabilities completed at 100
  (increase of 7 from baseline) n9
• Students without disabilities completed at 96
  n328
• GPA for students with disabilities was 3.22
  (increase of .22 from baseline)
• GPA for students without disabilities was 3.26

16
Summary of ResultsAnnual Performance Report
Highlights

• Project Specific Performance Measures Faculty
  participation
• Faculty participation across campuses totaled 34,
  above target of 25
• 73 of faculty agreed they would more likely
  make changes to their courses after exposure to
  UDL workshops (target was 60)
• All participating faculty reported making changes
  to their courses as a result of participation

17
Summary of ResultsFaculty Interview Highlights

• Changes to Courses
• Changes the result of the combination of training
  they received, not individual components
• Changes ranged from revisions to teaching
  strategies to more technologically complex
  (multi-media, Web CT)
• Prompted by desire to make course accessible to
  all students rather than for accommodations to
  individual students

18
Summary of ResultsFaculty Interview Highlights

• Sample of course changes
• Students acting out lessons or concepts
• Improvements to posting on the Internet
• Revisions to Syllabi using EnACT info
• Revising PowerPoint to be more accessible
• Study guides, guided lecture notes, assigning all
  students to be note takers
• Using rubrics and adding new assessment strategies

19
Summary of ResultsFaculty Interview Highlights

• Faculty Feedback about EnACT
• Each component received positive feedback
  overview from UDL I/II, specific information from
  Institute, and collaboration in Faculty Learning
  Community
• Changes to courses had positive impact on
  teaching and learning support from EnACT
  prepared them to continue to make changes

20
Summary of ResultsFaculty Interview Highlights

• Faculty Feedback about EnACT
• Would like support in implementation of
  technology
• Would like continued opportunities for
  collaboration that occurred during FLC

21
Evaluation Recommendations

• Continue to examine ways to implement EnACT to
  impact larger numbers of students with
  disabilities
• Use lessons learned from EnACT to implement ATI
  initiative and expand impact on campus
• Examine the impact, use and usefulness of AIMs

22
Evaluation Next Steps

• Orient new campuses to evaluation requirements at
  Leadership meeting
• Work with campuses to continue to streamline data
  collection processes
• Survey students in Fall 2007
• Collect Cohort II implementation data in Spring
  2008 (Fall 2007 grades)
• Collect Cohort III baseline data in Spring 2008
  (Fall 2007 grades)

23
Contact Us

• Mikala L. Rahn, PhD
• President
• Public Works, Inc.
• 90 N. Daisy Avenue
• Pasadena, CA 91017
• [phone]
• [phone]
• mrahn_at_publicworksinc.org

24
Contact Us

• Patty ODriscoll
• Consultant
• Public Works, Inc.
• 1191 Loma Ct.
• Sonoma, CA 95476
• [phone]
• [phone] fax
• patty_at_publicworksinc.org