ASSESSING STUDENT LEARNING AND OVERALL PROGRAM EFFECTIVENESS

1
ASSESSING STUDENT LEARNING AND OVERALL PROGRAM
EFFECTIVENESS Introduction Doris R. Brodeur CDIO
External Review 19 June 2003
2
ASSESSMENT THEME MEMBERS

3
OUTLINE OF PRESENTATION

• Assessment Vision and Goals
• Presentation Goals
• Overview of Activities and Results
• Highlights
• Assessing Student Learning in Mathematics and
  Mechanics
• Assessing Students and Programs With Reflective
  Portfolios
• Expectancies and Realities A Longitudinal Study
• Summary
• Future Plans
• Questions and Answers

4
ASSESSMENT THEME VISION

• The vision for assessment in the Wallenberg CDIO
  project is a system of methods, tools, and
  resources that determine the quality of teaching,
  learning, and program effectiveness and that
  include processes for continuous improvement.
  This vision is realized when faculty and
  administrators take an active part in
  investigating instructional needs, writing
  learning objectives, designing teaching and
  learning activities, and applying appropriate
  assessment methods. The vision can be achieved by
  adopting and modifying best practices in
  assessment to each institutions mission and
  goals. The resulting common wisdom and shared
  practice would have impact on the four partner
  institutions and extend to other engineering
  institutions in Sweden, the United States, and
  other countries.
• - adopted December 2000

5
ASSESSMENT GOALS
Educational Goals
Curriculum
Workshops
Assessment
Teaching Learning
Authentic experiences that support disciplinary
and CDIO skills
Models for curriculum structure and design
Tools and processes for program evaluation
Models for the design and utilization of
workshop/labs
Program
Tools and processes for assessing student
learning
Active, experiential learning with enhanced
feedback
Curricular materials and introductory courses for
CDIO education
Design-build educational experiences
Student Experience
6
REVIEWERS COMMENTS FROM YEAR 1

• The project team needs to continue to
  challenge whether the set of assessment tools is
  sufficient to measure the success of their
  progress. (External Review Committee Report, p.
  3)
• An opportunity exists to create new assessment
  methods which help foment a continuous
  improvement process which leads to better
  teaching, improved learning, and an overall
  progressive and increasingly effective
  engineering education program. (External Review
  Committee Report, p. 3)
• We further suggest including a continuous
  Improvement element that can allow for
  adjustments and improvements over time.
  (External Review Committee Report, p. 4)

7
PRESENTATION GOALS

• To highlight the development and use of new tools
  and procedures for assessing student learning,
  expectations, satisfaction, and overall program
  effectiveness
• To show evidence of a wide variety of CDIO
  learning outcomes that can be measured with these
  tools
• To provide samples of data collected to date that
  give information about student learning and
  program effectiveness
• To demonstrate ways in which assessment and
  evaluation data are used in continuous program
  improvement processes

8
COURSELEVEL ASSESSMENT
Student Learning
Expectations and Satisfaction
CDIO 1.0
CDIO 2.0 - 4.0

Peer and Self Assessment

Course Evaluations
Oral Exams

Oral Presentations
Concept Questions
Reflective Memos
Lab and Product Assessment
Reflective Portfolios
Course Portfolios

Highlighted presentation
9
EXAMPLE IN A TECHNICAL COURSE

• Concept Questions
• Pose a concept question (in multiple-choice
  format) and ask students to indicate their
  answers.
• If most have the correct answer, give a brief
  explanation, then move on.
• Else, clarify the concept by having students
  discuss it with a partner, or by giving a short
  lecture on the concept. Poll students again.
• A typical class period includes about 3 or 4
  concept questions.
• Oral Exams
• Give students the questions 30 minutes prior to
  the oral exam.
• Conduct the exam for 30 minutes.
• Develop a grading sheet listing each concept and
  the level achieved.

Aerodynamics Dave Darmofal, Instructor
10
ORAL EXAM SCORES 2000 AND 2001

11
STUDENT EVALUATION
Very Effective
Effective
Not Effective
Pre-class assignments, concept questions, and
oral exams are effective approaches for enhancing
conceptual understanding in advanced engineering
courses
12
EXAMPLE IN A DESIGN COURSE

• Peer and Self-Assessment
• Create a rubric to assess technical contributions
  and collaboration.
• Ask students to rate themselves, each team
  member, and at least one person from another
  team.
• Collect the rating sheets and summarize the
  results for each student.
• Provide this feedback to students at midterm and
  end of term.
• Oral Presentations
• Create a rubric to assess students technical
  knowledge and presentation style.
• Ask technical staff, education staff, and project
  research representatives to rate each students
  presentation.
• Collect the rating sheets and summarize the
  results for each student.
• Provide this feedback to students for all major
  technical briefings.

Space Systems Product Development Dave Miller and
John Keesee, Instructors
13
PROGRAM-LEVEL EVALUATION
Student Learning
Expectations and Satisfaction
Program Entry
Program Exit
Program Entry
Program Exit
Senior Exit Interviews
Baseline Interviews
Baseline Interviews
Senior Exit Interviews

Senior Exit Surveys
Senior Exit Surveys
Program Portfolios
Program Portfolios
Longitudinal Studies of Cohort Groups
The Balanced Scorecard
Highlighted Presentations
14
EXAMPLES OF CDIO IN THE CURRICULUM

• CDIO 2.0 Personal
• Engineering Reasoning and Problem Solving (CDIO
  2.1)
• Modeling (CDIO 2.1.2)
• CDIO 3.0 Interpersonal
• Teamwork (CDIO 3.1)
• Oral Communication (CDIO 3.2.6)
• CDIO 4.0 System-Building
• CDIO (CDIO 4.2 - 4.6)

1.0 Technical Knowledge and Reasoning
2.0 Personal and Professional Skills and
Attributes
CDIO ACROSS THE CURRICULUM
4.0 CDIO in Enterprise and Societal Context
3.0 Interpersonal Skills
15
ENGINEERING REASONING (CDIO 2.1)

• Engineering reasoning and problem solving were
  explicitly addressed in 25 of undergraduate
  courses taught in Fall 2002. (MIT Reflective
  Memos, Fall 02)
• In the MIT Portfolio Reflection Activity, close
  to 100 of respondents reported that they were
  able to demonstrate skill in engineering
  reasoning and problem solving as a result of one
  or more systems problems. (MIT Portfolio Activity
  03)
• In the end-of-term course evaluations, students
  commented on their progress in engineering
  reasoning.
• I really enjoyed following one project all the
  way through from planning to testing. (MIT SEF
  00-01)
• The best part of the course was this project
  that gave me a more grounded sense of what I was
  learning in the classroom environment. (MIT SEF
  01-02)

16
MODELING (CDIO 2.1.2)

• More than 25 of undergraduate instructors
  reported that they included modeling in their
  courses in Fall 2002. (MIT Reflective Memos, Fall
  02)
• In the MIT Portfolio Reflection Activity, about
  40 of respondents reported that they were able
  to demonstrate skill in modeling as a result of
  one or more systems problems (MIT Portfolio
  Activity 03)
• learned to model a real engineering system
• gained experience with using models to predict
  the performance of a system
• was able to see the computer-generated model
  come to life with the actual balloon flight
  experiment
• learned how important it is to consider all the
  important factors of a model

17
TEAMWORK (CDIO 3.1)

• Entering students believe they can work
  effectively on teams (MIT Baseline Interviews,
  2001)
• Teamwork was rated as Effective or Very Effective
  by most students (in courses that use teamwork)
  for three consecutive years (MIT SEF 00-01,
  01-02, Fall 02)
• Instructors reported that they included teamwork
  in three courses (MIT Reflective Memos, Fall 02)
• Seniors from MIT Classes of 2000 and 2001
  reported the ability to function on
  multidisciplinary teams as one of their most
  developed skills. (EBI, 2000, 2001)
• In exit interviews, seniors from the Classes of
  2002 and 2003 reported that they learned a great
  deal about working in teams, but that training in
  teamwork skills, while theoretically sound, was
  not effective in lecture format. (MIT Senior Exit
  Interviews, 2002, 2003)

18
ORAL COMMUNICATION (CDIO 3.2.6)

• In their Reflective Memos, instructors in four
  courses reported using oral presentations to
  assess students. (MIT Reflective Memos, Fall 02)
• Three formal presentations were accompanied by
  annotated viewgraphs.
• Group debriefing of both practice and oral
  presentations resulted in improved presentations
  through the term.
• In end-of-term course evaluations, more than 80
  of students reported that oral presentations were
  Effective or Very Effective in courses that
  include them. (MIT SEF 00-01, 01-02, Fall
  02)
• Presentations are useful and engaging.
• The oral presentations were the best part of the
  course.
• In senior exit surveys, seniors reported the
  ability to communicate using oral progress
  reports as an area of most developed skills. (EBI
  2000 and 2001).
• In senior exit interviews, seniors described the
  ability to give technical briefings with
  competence, poise, and confidence as an area of
  major growth over the course of their programs.
  (MIT Senior Exit Interviews, 2002 and 2003)

19
C-D-I-O (CDIO 4.2 - 4.6)

• In end-of-term course evaluations, nearly 100 of
  students reported that system/product lifecycle
  projects (CDIO) were Effective or Very Effective
  in courses that include them. (MIT SEF 00-01,
  01-02, Fall 02)
• The best part was actually building the
  vehicles in the lab, being given discretion as to
  the design of the vehicles.
• I think CDIO is an extremely valuable way for us
  to learn about lab experiments, teamwork, and
  being a part of a large complex systems
  engineering project.
• In the Reflective Portfolio Activity, nearly 100
  of 2nd-year students reported progress on C-D-I-O
  in their systems problems. (MIT Reflective
  Portfolio Activity, 2003)
• In senior exit interviews, seniors (to a person)
  described CDIO activities as the major highlight
  of their programs and overall MIT experience.
  They reported that they had developed the
  requisite skills for working with a team on a
  project from conception through operation. (MIT
  Senior Exit Interviews, 2002 and 2003)

20
HIGHLIGHTED PRESENTATIONS

• Assessing Student Learning in Mathematics and
  Mechanics

Assessing Students and Programs With Reflective
Portfolios
Expectancies and Realities A Longitudinal Study