Assessment Theme Plenary Session

1
Assessment Theme Plenary Session Peter Gray
Doris Brodeur Svante Gunnarsson
Program Evaluation
Inputs.
Improvement
Implementation
.
Impact

2
STANDARD 12 - CDIO PROGRAM EVALUATION

• A system that evaluates programs against these
  twelve standards, and provides feedback to
  students, faculty, and other stakeholders for the
  purposes of continuous improvement
• Judgment of the overall value of a program based
  on evidence of a programs progress toward
  attaining its goals
• Feedback of the results form the basis of
  decisions about plans for continuous improvement

3
STANDARD 12 - CDIO PROGRAM EVALUATION

• Rationale
• Evidence collected during the program evaluation
  process serves as the basis of continuous program
  improvement.
• Many external evaluators and accreditation bodies
  require regular and consistent program evaluation
  for this purpose.

4
PROGRAM EVALUATION MODEL
Program Evaluation
Inputs.
Improvement
Implementation
.
Impact

5
PROGRAM EVALUATION MODEL

• Inputs
• Program Purposes, Resources, and Planned
  Activities
• Implementation
• Actual Program Activities Participation and
  Perceptions
• Impact
• Program Outcomes
• Improvement
• Using Results

6
PROGRAM EVALUATION MODEL
Program Evaluation

• Inputs
• Program Purposes, Resources, and Planned
  Activities
• .

Improvement
Implementation
.
Impact

7
THE CDIO STANDARDS

• Seven standards are considered essential to
  distinguish a CDIO program because they deal
  explicitly with personal, interpersonal and
  product/system building skills (marked with )
• Supplementary standards are highly desirable and
  reflect best practice, but are not necessarily
  distinguishing features of a CDIO program
• Standards focus on
• Principles and goals (1, 2)
• Curriculum (3, 4)
• Workspaces and laboratories (5, 6)
• Teaching and learning (7, 8)
• Competence of the teaching faculty (9, 10)
• Assessment and evaluation (11, 12)

8
1. CDIO as Context
Existing Workspaces
Existing Curriculum
2. CDIO Syllabus Outcomes
Existing Teaching and Learning
Existing Assessment and Evaluation
Survey of Assessment and Evaluation Methods
Survey of Teaching and Learning Methods
Curriculum Benchmarking
Survey of Labs and Workspaces
Best Practices and Possible Innovations
Input
Plans to Improve Teaching and Learning
Assignment of CDIO Outcomes to Curriculum
Workspace Usage Planning
Assessment and Evaluation Plans
10. Enhancement of Faculty Teaching Skills
9. Enhancement of Faculty CDIO Skills
Implementation
6. CDIO Workspaces
12. CDIO Program Evaluation
3. Integrated Curriculum
7. Integrated Learning Experiences
4. Intro to Engineering
8. Active Learning
11. CDIO Skills Assessment
5. Design-Build Experiences
Program Operation and Student Learning
Impact
9
PROGRAM EVALUATION MODEL
Program Evaluation

• Inputs
• Program Purposes, Resources, and Planned
  Activities
• .

Improvement

• Implementation
• Actual Program Activities Participation and
  Perceptions
• .

• IMPACT
• Program outcomes
• .

10
1. CDIO as Context
Existing Workspaces
Existing Curriculum
2. CDIO Syllabus Outcomes
Existing Teaching and Learning
Existing Assessment and Evaluation
Survey of Assessment and Evaluation Methods
Survey of Teaching and Learning Methods
Curriculum Benchmarking
Survey of Labs and Workspaces
Best Practices and Possible Innovations
Input
Plans to Improve Teaching and Learning
Assignment of CDIO Outcomes to Curriculum
Workspace Usage Planning
Assessment and Evaluation Plans
10. Enhancement of Faculty Teaching Skills
9. Enhancement of Faculty CDIO Skills
Implementation
6. CDIO Workspaces
12. CDIO Program Evaluation
3. Integrated Curriculum
7. Integrated Learning Experiences
4. Intro to Engineering
8. Active Learning
11. CDIO Skills Assessment
5. Design-Build Experiences
Program Operation and Student Learning
Impact
11
PROGRAM EVALUATION MODEL
Program Evaluation

• Inputs
• Program Purposes, Resources, and Planned
  Activities
• .

Improvement

• Implementation
• Actual Program Activities Participation and
  Perceptions
• .

• IMPACT
• Program outcomes
• .

12
1. CDIO as Context
Existing Workspaces
Existing Curriculum
2. CDIO Syllabus Outcomes
Existing Teaching and Learning
Existing Assessment and Evaluation
Survey of Assessment and Evaluation Methods
Survey of Teaching and Learning Methods
Curriculum Benchmarking
Survey of Labs and Workspaces
Best Practices and Possible Innovations
Input
Plans to Improve Teaching and Learning
Assignment of CDIO Outcomes to Curriculum
Workspace Design and Usage Planning
Assessment and Evaluation Plans
10. Enhancement of Faculty Teaching Skills
9. Enhancement of Faculty CDIO Skills
Implementation
6. CDIO Workspaces
12. CDIO Program Evaluation
3. Integrated Curriculum
7. Integrated Learning Experiences
4. Intro to Engineering
8. Active Learning
11. CDIO Skills Assessment
5. Design-Build Experiences
Program Operation and Student Learning
Impact
13
EVALUATION FRAMEWORK
Student Learning Assessment
Program Evaluation
Oral Presentations
Course Evaluations
InternshipsPerformance Reviews
Diagnostic Tests
Senior Exit Interviews
Concept Questions

Written Reports
Oral Exams
Project Lab Reviews
Reflective Memos
Baseline Interviews
Portfolios
Peer Assessment
Written Exams
Senior Surveys
Alumni Surveys
Employer Surveys

Follow-up Studies
Data-Based Program Improvement
14
EVIDENCE IN PROGRAM DOCUMENTS

• Student surveys
• Follow-up studies of graduates
• Enrollment, retention, graduation, and
  post-graduate placement statistics
• Other studies conducted by the university

15
PROGRAM EVALUATION MODEL
Program Evaluation

• Inputs
• Program Purposes, Resources, and Planned
  Activities
• .

• Implementation
• Actual Program Activities Participation and
  Perceptions
• .

• IMPROVEMENT
• Using Results
• .

• IMPACT
• Program outcomes
• .

16
Standard 12

• CDIO Program Evaluation
• Documented continuous improvement process
• Evidence of data-driven changes

17
EXAMPLES

• Exit interviews and surveys at MIT Doris
• The Balanced Scorecard at LiU Svante