Assessing Team Based Learning

1
Assessing Team Based Learning
Jeffrey F. McCauley, Engineering Instructor and
Technology Division ChairGreen River Community
CollegeKenneth L. Gentili, Physics and
Engineering InstructorEngineering Transfer
CoordinatorTacoma Community College

• Workshop at PNW Higher Education Assessment
  Conference
• Spokane-May 3, 2001

2
Agenda for Workshop

• Background
• 830 Introduction
•  Developing Teams
• 845 Forming Teams (Two-truths and a Lie)
  
• 900 Refining Sounds of Effective Teams
• 920 Strategies for Developing Effective Teams
• Assessing Team Performance
• 935 Engaging in TIDEE Mid-Program Team
  Performance Assessment

• 1000 Break (30 minutes)
• 1030 Scoring Reflective Essay to Assess Team
  Performance
• 1055 Adapting the Scoring Matrix for other
  Disciplines
• 1120 Assessing Team Performance Using Exit
  Interview
•  
• Assessing Critical Thinking
• 1130 Introduction to Definitions and Performance
  Criteria
• 1140 Assessing Team Performance
• 1155 Wrap-up
• 1200 Workshop Ends

3
Transferable Integrated Design Engineering
Education (TIDEE)
-- Regional Implementation --
Partner Organizations

• Washington State University (WSU)
• Denny C. Davis, Michael S. Trevisan, David I.
  McLean
• University of Washington (UW)
• Cynthia J. Atman, Craig Lewis
• Seattle University (SU)
• Patricia D. Daniels, Robert G. Heeren

• Tacoma Community College (TCC)
• Kenneth L. Gentili
• Green River Community College (GRCC)
• Jeffrey F. McCauley
• University of Idaho (UI)
• Steven L. Beyerlein

• Funding from
• National Science Foundation (NSF) EEC-9973034

4
TACOMA COMMUNITY COLLEGE INSTRUCTIONAL OUTCOMES
CRITICAL THINKING PROBLEM SOLVING
LIVING WORKING COOPERATIVELY

• Identification of Issues
• Decision Making
• Making Judgments
• Understanding Thought Processes

• Diversity
• Cultural Competence
• (Multicultural Course Designation)
• Values
• Team Work

LIFELONG LEARNING Workforce Education Preparatio
n for Transfer Continuing Education
RESPONSIBILITY

• For Own Learning
• Accept Consequences
• Modify Behaviors

COMMUNICATION

• Effective Appropriate
• Inter/Intra-personal
• Verbal
• Written (Writing Intensive Course Designation)

INFORMATION

• Accessing
• Evaluating
• Ethical Use
• Technology Competency

5
Educational Processes and Outcomes
End-of-ProgramEducationalOutcomes
Mid-ProgramEducationalOutcomes
Educational Objectives
Freshman Sophomore Education Process
Junior Senior Education Process
Freshman Sophomore Education Process
Educational Processes
6
Assessment for Improvement of Engineering
Education
End-of-ProgramOutcomes
Mid-ProgramOutcomes
Educational Objectives
Freshman Sophomore Education Process
Junior Senior Education Process
Mid-Program OutcomesAssessment
End-of-Program OutcomesAssessment
Assessment Processes
7
Transferable Integrated Design Engineering
Education (TIDEE) Project
Teaching Materials
Outcomes Definition
Freshman Sophomore Education Process
Junior Senior Education Process
Mid-Program OutcomesAssessment
End-of-Program OutcomesAssessment
Alumni/Employer Feedback
Assessment Testing
Assessment Development
8
Elements of a Successful Assessment Process
Educational Objectives
Define Educational Outcomes
Define Performance Criteria
Define Plan to Obtain Evidence
Gather Data as Evidence
Take Action as Planned
Interpret Assessment Results
Plan Actions to Improve
Score Student Performance
9
General Levels of Achievement for Outcomes

• Level 1 Basic Knowledge
• recite and recall facts, concepts
• Level 2 Application of Knowledge
• use knowledge appropriately and skillfully
• Level 3 Critical Analysis
• judge relevance or quality based on knowledge
• Level 4 Extension of Knowledge
• define improved concepts or processestransfer
  to new uses

Levels affect verbs used to define outcomes
10
Example Educational Outcome
Engineering Design Students are able to apply a
systems engineering process to derive physical,
functional, and financial requirements and create
engineering systems and subsystems that satisfy
needs of clients

• Verbs Denote Levels
• Application of knowledge
• Critical analysis
• Extension of knowledge

11
Design Education Outcomes Category-Level Matrix
Increasing Level
Basic Knowledge
Application of Knowledge
Critical Analysis
Extension of Knowledge
Design Process
Subcategories
Teamwork
Subcategories
Communication
Subcategories
Product
Subcategories
Mid-Program Target
End-of-Program Target
12
Examples Outcomes at Different Levels of
Achievement

• Category Design Process
• Basic Knowledge
• Students are able to define steps used in a
  simple engineering design process.
• Application of Knowledge
• Students, when given an open-ended design
  problem, are able to plan and execute steps that
  comprise a relevant engineering design process.
• Critical Analysis
• Students are able to explain the relevance and
  effectiveness of a process used in an engineering
  design project.
• Extension of Knowledge
• Students are able to define effective ways to
  improve a process used in an engineering design
  assignment.

13
Types of Educational Outcomes
End-of-program
Mid-program
100
Product
Process
Percent ofLearning
Knowledge
0
Yr 1
Yr 2
Yr 3
Yr 4
Stage in Curriculum
Types affect the selection of assessment method
14
QUESTION What strategies are needed to create
quality teams that use the strengths of their
team members to create quality results?
15
Two Truths and a Lie
Activity
Activity
Activity
Activity
Activity

• Purpose Break down the barriers and begin to
  form synergy between team members.
• Tasks
• Each team member tells two-truths and a lie
  about themselves.
• Team members discuss and come to a consensus as
  to which are truthful.
• The team is to find two truths and a lie about
  their team.
• Present this to the group for the group to try
  and find out which is the lie.
• Reporting
• Present your highest rated data source for each
  outcome
• Time Available 12 minutes

16
Refining Sounds of Effective Teams
Activity
Activity
Activity
Activity
Activity

• Purpose Team members learn how to use positive
  statements to make the team become more
  effective. Your teams assigned role for this
  activity is______________.
• Tasks
• Review statements for teams assigned role on
  handout and decide whether they would be
  attributed primarily for the role.
• Focus on two situations where person in role
  would need to communicate ideas or information to
  the team.
• For each situation, generate positive and
  negative statements that could be used to
  communicate this information.
• Identify how the person in this role can improve
  team performance through effective use of
  statements.
• Reporting
• Describe one situation and positive and negative
  statements that could be used to communicate
  information to other team members.
• Time Available 12 minutes

17
Effective Statements for Team Members
Each of the roles has a performance criteria acts
like a job description. Listed are some helpful
sayings that can assist you to help make the team
a positive and rewarding experience.  

• A leader or team manager should reduce stress. A
  leader might say
• What do you think about...........?
• Good job!
• How can we apply or use this concept?
• Would this be a good approach to........?
• As I understand it, our conclusion is.......
• Lets listen to one idea at a time.
• We need to refocus and deal with..
• What is your idea?
• Should we do............?
• Could this be summarized by saying............?
• Have we reached this conclusion...............?

• A reporter is responsible for representing the
  teams consensus. A reporter might say
• How could we present that?
• How can I relay that effectively?
• Can I say that our team decided...
• Does our list consist of...
• Is it correct to say that our graph shows...
• How can I put that simply?
• Through a process of elimination did we...?

18
Effective Statements for Team Members

• An explorer creates new ideas and takes risks.
  An explorer might say
•          Lets consider this another way.
•          What about this idea?
•          Lets find new methods for presenting
  this material.
•          What would happen if we combined these
  ideas?
•          What is this analogous to?
•          How should we present this?
•  
• A reflector looks at things as a big picture. A
  reflector might say
•          This is what it sounds like, but it
  could also.........
•          So what you are saying is...........
•          But lets focus on...........
•          How can we improve on.......?
•          Do we have all of the data?
•          Is that going to achieve our goal?

• A recorder should be able to summarize items
  without editorializing. A recorder might say
•          Can we say this is........?
•          Up to this point we gathered.........
•          Where are we going with.........?
•          What data do we have to........?
•          Can we say that more succinctly?
•         Do we agree on this?
•         Should it be recorded in our team
  journal?

19
Common Team Experiences

• Everyone has had a team experience.
• Almost no-one has been on a quality team.
• Most common experience We were put onto a team
  to do a team project. After about two weeks,
  someone rescued us to preserve our grade. We have
  experienced many social loafers.
• Almost no-one was given teaming skills. It was
  assumed that the team would know the process.

20
Stages of Team Development

• Forming, Storming, Norming, Performing
• V
• a
• l
• u
• e
• Time

21
Strategies for Developing Effective Teams

• Uncover useful information about team members.
• Do activities like Two Truths and a Lie.
• Have team members discuss their previous
  experience in performing different roles and the
  strengths they bring to the team.
• Do a learning style inventory and describe how
  that effects team dynamics. (Herrmann four
  quadrant brain model.)

22
Strategies for Developing Effective Teams

• Assign individuals to 4 to 6 person teams
• -Create teams with diversity so the team has
  whole brain thinking. There is a dearth of
  students with quadrant B thinking, or
  organizational skills.
• -Check individual schedules for available meeting
  times outside class.
• -Dont isolate genders or ethic minorities.

• Create a team identity or symbol
• Identify functional team roles with specific job
  responsibilities like manager, recorder,
  reporter, reflector and explorer.
• Develop the process, not just the expectations
  for the product, like reflectors reports, peer
  review, goal setting, evaluation of process
  improvement.

23
Strategies for Developing Effective Teams

• Develop positive communication skills.
• Do activities like Refining Sounds of Effective
  Teams, Killer and Ignitor Phrases, Integrate
  New Members into the team.
• Provide resources on how to cope with
  dysfunctional teams or team members.
• Provide evaluation material which clearly states
  expectations of positive oral communication.
• Provide an organized structure with time lines
  and classroom activities to support the process.

24
Problem/Need
Information Gathering
Gather information about the issue of concern
clarify problem
Define requirements to be met by a good solution
Problem Definition
improve information
Process Development
Manage process steps to improve solution
Identify many creative approaches to the solution
Idea Generation
revise requirements
Evaluation and Decision Making
Select ideas that best satisfy all requirements
improve ideas
Carry solution to completion and delivery to user
Implementation
improve solution
Creative Solution
Elements of the Engineering Design Process
25
Mid-Program Performance Criteria Category
Design Process
INFORMATION GATHERING Appropriate information is
accessed and used to inform the design
process. PROBLEM DEFINITION Project requirements
are defined to include relevant technical and
non-technical criteria. IDEA GENERATION Creative
ideas are generated to address needs throughout
the design process. EVALUATION and DECISION
MAKING Analysis of ideas and design decisions
are based on appropriate methods and
criteria. IMPLEMENTATION Design products stem
from design decisions and meet client
expectations. PROCESS DEVELOPMENT Activities in
the design process are managed to enhance
productivity and product quality.
26
TIDEE Seven-Point Scoring Scale for Student
Achievement

• Basis for developing individual scoring criteria
  for individual assessment tools
• Scoring scales for design process, communication
  skills and teamwork skills
• Multiple subcategories for each category
• Benchmarked for entering, mid-program, graduating
  seniors and practicing professionals.

27
Mid-Program Design Assessment(3-Components)

• GROUP ACTIVITY
• (45 minutes)
• Timed team project
• Organize team
• Record process
• Produce required deliverables

• SHORT ANSWER
• (15 minutes)
• Define essentials of
• Design process
• Teamwork
• Communication

• TWO-PAGE ESSAY
• (take-home)
• Reflect on project
• What was done
• What produced desired results

Basic Knowledge
Basic Knowledge and Application of Knowledge
Application of Knowledge and Critical Analysis
28
QUESTION What is the TIDEE Mid-Program
Performance Assessment and how can it be adapted
for assessment in other disciplines?
29
Taking the Performance Segment of the Mid-Program
Assessment
Activity
Activity
Activity
Activity
Activity

• Purpose To engage in a team activity and gain
  experience for assessing written work.
• Tasks
• Complete the mid-program assessment
• Take a break and be ready to come back at 1030
  AM.
• Time Available 25 minutes 15 minutes of break
  time if desired.

30
Scoring Reflective Essay to Assess Team
Performance
Activity
Activity
Activity
Activity
Activity

• Purpose To learn how to score team and team
  communication skills using a limited set of
  performance criteria.
• Tasks
• Assess two of the papers assigned to you using
  the performance criteria, information, and tips.
• Come to a consensus for a single value.
• Reporting
• Present your assessment value for each category
  and difficulties coming to a consensus.
• Time Available 15 minutes.

31
Adapting the Scoring Matrix for other Disciplines
Activity
Activity
Activity
Activity
Activity

• Purpose Create opportunities to use this
  assessment tool in other disciplines
• Tasks
• Discuss how this assessment could be used as an
  assessment in your discipline.
• Select one of your three categories of outcomes
  and review the proposed scoring scale provided.
• Suggest revisions for the criteria and/or
  definitions that would allow you to adapt this
  scale for use in your discipline.
• Prepare your teams worksheet for reporting
• Reporting
• Present your assessment value for each category
  and difficulties coming to a consensus.
• Time Available 15 minutes.

32
Assessing your Team Performance using Exit
Interview
Activity
Activity
Activity
Activity
Activity

• Purpose To assess your teams performance at
  this workshop using the exit interview
• Tasks
• Assess your teams performance in process and
  communication skills.
• Prepare a reflectors report on how your team has
  performed.
• A reflectors report includes
• Our teams strengths are_______.
• Our team can improve by________.
• An insight that I personally had was______
• Reporting
• Present your assessment value for each category
  and difficulties coming to a consensus.
• Time Available 7 minutes.

33
QUESTION What is Critical Thinking and how can
we assess it using this assessment tool?
34
Resource Material
35
Summary of Critical Thinking Definitions

• A critical thinker should be able to
• Define tasks, solve problems and implement plans
  by gathering, evaluating, and synthesizing
  information and ideas.
• Recognize patterns and ways to use those patterns
  to solve a problem.
• Recognize what is irrelevant or extraneous
  information including errors in logic, reasoning
  or the thought process.

36
Ctilacs Operative Definitions of Critical
Thinking

• Establishes a useful set of definitions
• Critical thinking recognizes
• Critical thinking implies
• Critical thinkers
• Fosters common understanding of critical thinking
  and information literacy
• Developed by BCC through an NSF Division of
  Undergraduate Education grant

37
Establishing Performance Criteria in Critical
Thinking

• Create categories
• Define expected outcomes for students, faculty
  and institutions
• Use clear language to be able to measure
  incremental change in knowledge and skills
• Use criteria to establish baselines and
  benchmarks for learning/disciplines/institution
• Check for transferability across disciplines

38
Categories of Performance Criteria in Critical
Thinking

• Identification of Issues
• Information Processing
• Selection of Solution
• Implementation
• Analyze Results
• Benefit and Value of Critical Thinking
• ? Tacoma Community College

39
Levels of Critical Thinking for Each Category

• Levels of higher order thinking skills
• Basic
• Progressing
• Accomplished
• Each discipline/department/institution uses
  desired performance level for each category
• Basis for developing and creating learning
  activities and assessment materials

40
Assessment RubricsTables 1 and 2
41
Assessing Teams Critical Thinking
Activity
Activity
Activity
Activity
Activity
Activity
Activity
Activity
Activity
Activity

• Purpose To analyze the level of critical
  thinking of each team.
• Tasks
• Review the critical thinking performance
  criteria.
• Select three of the five categories of outcomes
  and assess your teams performance.
• Prepare your teams worksheet for reporting your
  results.
• Reporting
• Present your assessment value for each category
  and difficulties coming to a consensus.
• Time Available 12 minutes.

42
Summary / Conclusions

• Teamwork and team communication skills may be
  measured at various critical thinking levels.
• Outcomes and desired achievement must be defined.
  
• Collaboration across disciplines and across
  institutions offers potential benefits for
  improved assessments

43
QUESTION What questions do you have? Have a
great conference!
44
Transferable Integrated Design Engineering
Education (TIDEE)

• Kenneth Gentili,
• Leader of Curriculum Design Team
• kgentili_at_tcc.ctc.edu
• www.cea.wsu.edu/TIDEE/