Goal for this presentation

1
The AERO CRCD Smart Materials Project
1 Preparation for Evaluation Adaptation,
Evolution and Application of The TAMU-TIDEE
Design Knowledge Test
Rita Caso, Ripla Kohli, Jong-Hwan Lee, Texas
AM University Developed from Work In Progress
presentation made at FIE Conference, Session
1199, 6-9 November , Boston, MA.
2
Acknowledgements

• This material is based upon work supported by
  the National Science Foundation under the
  following grants
• EEC-9802942
• EEC-0088118

3
Adaptation of the Transferable Integrated Design
Engineering Education (TIDEE), Mid Program
Assessment A Three Part Assessment of Team Based
Design for Entering Juniors 2, Part 1, and its
evolution as the TAMU-TIDEE Design Knowledge
Test
Part I
4
Legend for.. Diagrams of Instrument Rubric
Evolution, and Data Collection

• Instrument Version
• Informed Judgment to Change Instrument or Rubric
• Instrument Scored with Indicated Program Rubric
• Baseline Data Collected for Program Evaluation
• Data Collected for Design Assessment Instrument
  Development

5
Instrument Rubric Evolution
TIDEE Instrument Prototype
TIDEE Rubric Prototype
TAMU-TIDEE Test V.1
V.1 TAMU-TIDEE Rubric
TAMU-TIDEE Test V.2
V.2 TAMU-TIDEE Rubric
TAMU-TIDEE Test V.3
V.3 TAMU-TIDEE Rubric
Future TAMU-TIDEE Test V.4
6
Data Collection
TIDEE Instrument Prototype
TIDEE Rubric Prototype
Fall 2001 V.1 TAMU-TIDEE Test Freshmen Aero
Seniors
V.1 TAMU-TIDEE Rubric
Spring 2002 V.2 TAMU-TIDEE Test Freshmen Aero
Seniors Professors
V.2 TAMU-TIDEE Rubric
Fall 2002 V.3 TAMU-TIDEE Test EE Seniors
V.3 TAMU-TIDEE Rubric
V.4 TAMU-TIDEE Test
7
TIDEE Definitions for Levels of Achievement in
Design Education Outcomes

• Level Description
• Basic Knowledge Recognition and understanding of
  facts, terms, definitions, descriptions,
  relationships, processes.
• Application of Use of knowledge in ways that
• Knowledge demonstrate understanding of concepts
  or terms, their proper use, and limitations
  of their applicability
• Critical Analysis Examination and evaluation of
  informative required to judge its value to a
  design solution and to make decision.
• Extension of Extensive knowledge beyond what
  was received, creating new knowledge, making
  inferences, transferring knowledge to
  usefulness in new area of application, making
  improvements.

8
Key to Changes Made in Next Version of Instrument

• Text in red was identified as needing change in
  NEXT version
• Example Respond to the following
  questions/statements. You have 15 mins.
• Text in red, set apart by indicates that
  something additional is needed in NEXT version
• Example ( Lacks specific instructions for
  response)
• Text in italics indicate new wording to that
  version
• Example In the space provided, describe your
  understanding of the engineering design process,
  in diagram or flowchart form, in complete
  sentences, or in list form.
• Text which is both italicised and in red
  indicates that the wording is new to that
  version, but that it was identified as needing
  change in NEXT version
• Example List the qualities that describe
  effective teamwork

9
Prototype The TIDEE Design Knowledge Assessment
Instrument 3(TIDEE Website, June 2001)
TAMU AERO CRCD/TIDEE

• Objective Demonstrate your knowledge of key
  elements in the engineering design process,
  teamwork, and communication associated with
  team-based engineering design.
• Assignment Respond to the following
  questions/statements. You have 15 mins.
• In a general sense, a process is an ordered set
  of activities to accomplish a goal. In the space
  provided, describe and/or diagram your
  understanding of the engineering design process.
  ( Lacks specific instructions for response)
• List the elements required for effective teamwork
  in a design project assignment, and explain why
  each element is important.
• In team-based design, documentation and exchange
  of design information are important. List
  features of effective communication. Consider a
  wide range of oral and written communication
  situations.

 

 
Davis, D. C. (2001). Transferable Integrated
Design Engineering Education (TIDEE), Mid Program
Assessment A Three Part Assessment of Team Based
Design for Entering Juniors.
http//www/cea/wsu.edu/TIDEE
10
V.1--TAMU TIDEE Test 1st Adaptation Given, as
Pre-test, to Aero Senior Design Class and Smart
Material Freshman Class in Fall 2001

• Objective Demonstrate your knowledge of key
  elements in the engineering design process,
  teamwork, and communication associated with
  team-based engineering design.
• Assignment Respond to the following
  questions/statements as instructed (Time allowed
  20-30 mins.)
• In a general sense, a process is an ordered set
  of activities to accomplish a goal. In the space
  provided, describe and/or diagram your
  understanding of the engineering process.
  (Express yourself in complete sentences or in
  acombination of diagrams and clear phases.)
• What kinds of things should teams do to prepare
  themselves beforehand for effective teamwork in a
  design project assignment? Respond in the form
  of a list. (Separate your list into different
  categories of activities)
• In team-based design, documentation and exchange
  of design information are important. List the
  characteristics of written and oral communication
  that will be essential in a team design project.

11
V.2--TAMU TIDEE Test 2nd Adaptation Given, as
Post-test, to Aero Senior Design Class and a
Different Freshman Class in Spring 2002

• Objective Demonstrate your knowledge of key
  elements in the engineering design process,
  teamwork, and communication associated with
  team-based engineering design.
• Assignment Respond to the following
  questions/statements as instructed (Time allowed
  20-25 mins.)
• In a general sense, a process is an ordered set
  of activities to accomplish a goal. In the space
  provided, describe your understanding of the
  engineering design process, in diagram or
  flowchart form, in complete sentences, or in list
  form.
• What kinds of things should teams do to prepare
  themselves for effective teamwork in a design
  project assignment? Respond in the form of a
  list or in sentences.
• In team-based design, documentation and exchange
  of design information are important. Describe
  communication qualities and how communication
  should occur among team members in a design
  project.

12
V.3--TAMU TIDEE Test 3rd Adaptation Given, as
Pre-test, to Electrical Engineering Design Class
in Fall 2002

• Objective Demonstrate your knowledge of key
  elements in the engineering design process,
  teamwork, and communication associated with
  team-based engineering design.
• Assignment Respond to the following
  questions/statements as instructed (Time allowed
  20-25 mins.)
• In a general sense, a process is an ordered set
  of activities to accomplish a goal. In the space
  provided, describe your understanding of the
  engineering design process, in diagram of
  flowchart form, in complete sentences, or in list
  form.
• Part A For a Team-Based Engineering Design
  Project, what are the processes that contribute
  to good teamwork? List as many as you can.
• Part B List the qualities that describe
  effective teamwork. List as many as you can.
• C. What are the component characteristics of
  good communication with regards to the quality of
  information communicated and with regard to
  receiving and presenting information? List as
  many qualities as possible.

13
V.4--TAMU TIDEE Design Knowledge Test 4th
Adaptation Based on WMU Faculty Feedback For
future use

• Objective Demonstrate your knowledge of key
  elements in the engineering design process,
  teamwork, and communication associated with
  team-based engineering design.
• Assignment Respond to the following
  questions/statements as instructed (Time allowed
  20-25 mins.)
• In a general sense, a process is an ordered set
  of activities to accomplish a goal. In the space
  provided, describe your understanding of the
  engineering design process, in diagram of
  flowchart form, in complete sentences, or in list
  form.
• For a Team-Based Engineering Design Project, what
  are the processes that contribute to good
  teamwork? What are the qualities that describe
  effective teamwork. List and describe as many as
  you can.
• C. What are the characteristics of good
  communication? (Include information about quality
  of communication with people external to the
  design team, as well as internal to tam.)
  Include as many characteristics as possible.

14
Our Prototype The TIDEE Design Knowledge
Assessment Scoring Rubric(TIDEE Website, June
2001)
Category Design Process Subcategory Problem
Definition 1 5 No design requirements
stated, or few but Goal and requirements
defined fully, revised ambiguous at best. over
time, address technical and non-technical
issues such as performance, cost,
reliability, manufacturability, and
safety. Subcategory Evaluation and Decision
Making 1 5 Only cursory analysis of ideas
decision made arbitrarily Quantitative and
qualitative issues analyzed appropriate
analytical and experimental methods,
tools, and information used
decisions based on established
criteria Subcategory Process
Development 1 5 Design steps performed
linearly without steps towards Process elements
repeated, used iteratively to improvement. impr
ove results design process planned,
recorded, and revised for improvement.
15
Our Prototype The TIDEE Design Knowledge
Assessment Scoring Rubric(TIDEE Website, June
2001)
Category Teamwork SubcategoryPurpose and Goals
1

5 Team goals non
existent or poorly defined members Team driven
by clear goals members know goals
clueless about team goals. and how will be
assessed members committed to common
purpose. Subcategory Roles and
Responsibilities 1 5 Team members have no
clear role assigned or role Members understand
responsibility team defines, ownership
important roles not performed. enforces
expectations members own and fulfill their
roles to support team all important roles
covered well. Subcategory Resource
Management 1 5 Team without adequate
skills and resources to Team has proper mix,
levels of skills needed for Complete
project no effort to obtain needed
resources. Project team uses talents, knowledge,
skills of all members other resources
accessed and used effectively.
16
Our Prototype The TIDEE Design Knowledge
Assessment Scoring Rubric(TIDEE Website, June
2001)
Category Communication SubcategoryContent 1
5 Information incomplete or has obvious
errors Information stated clearly, completely,
and overall Reliability questioned. accurately
measures of reliability stated and
justified. SubcategoryRelevance to
Audience 1 5 Information not relevant to
audience as presented. Information in form,
appearance, language, and level attractive
and understandable to intended audience
understanding communicated
well. Subcategory Value 1 5 Information
recorded or communicated has little Information
is timely and of great value to team
value to design team or others. and to others
support quality design decisions and
value-added actions by others.
17
V.1--TAMU TIDEE Rubric 1st Adaptation (Fall
2001) -- Used on Fall 2001 Pre-tests , Spring
2002 Post-tests, and Tests completed by TAMU
Freshman and Design Profs.
5 PointsStudents response shows breadth and
depth of knowledge, and indicate
interrelationships among most (5 or 6 of 6) of
the listed elements in the engineering design
process necessary to produce design products. 4
PointsStudents response shows more than
moderate knowledge, if it includes 4 of the 6
elements (listed here) of the engineering design
process. 3 PointsStudents response shows
moderate knowledge, if it include 3 elements of
the engineering design process necessary to
produce design products. 2 PointsStudents
response shows little knowledge, if it includes 2
elements of the engineering design process. 1
Point---Students response indicates an effort
made to describe the engineering design process
and one element is indicated but the response
lacks coherent organization. 0 Point---Student
did not make any effort to define the engineering
design process and shows no knowledge of any of
the elements listed.
Part A Knowledge of the Engineering Design
Process Six specific elements articulated by
students. 1. Information gathering (i.e. looking
for resources in the library, Web-sites or from
the other sources) 2. Problem definition (i.e.
understanding problem, goals or requirements)
defined 3. Idea generation (i.e. brain
storming, creative ideas to improve the design
products) 4. Evaluation and decision making
(i.e. analyzing ideas, selecting, utilizing,
judging and making design decisions based on
established requirements) 5. Implementation (i.e.
product delivered on time, product satisfy
requirements) 6. Process development (i.e.
reviewed for improvement)
18
V.1--TAMU TIDEE Rubric 1st Adaptation (Fall
2001) -- Used on Fall 2001 Pre-tests , Spring
2002 Post-tests, and Tests completed by TAMU
Freshman and Design Profs.
Part B Knowledge of Effective Teamwork Six
specific elements articulated by students. 1.
Purpose and Goals (i.e. goals, members know
goals) 2. Roles and Responsibilities (i.e.
members understand roles and fulfill to support
teamwork) 3. Team attitude (i.e. polite
acceptance and respect for each other, show
pride in team and motivated toward team success,
commitment) 4. Planning (i.e. time
management) 5. Resource management (team uses
talents, knowledge, skills of all members, other
resources accessed) 6. Operating procedures
(i.e. support effective interaction, discipline,
listening, decision making etc.)
5 Points---Students response (if include 5-6 of
6) shows detailed knowledge of the listed
elements of effective teamwork 4
Points---Students response shows above moderate
knowledge, if include 4 elements (of listed) of
the effective teamwork. 3 Points---Students
response if include 3 elements of the effective
teamwork, shows moderate knowledge of the
subject. 2 Points---Students response including
2 elements of the effective teamwork shows little
knowledge of the subject. 1 Points---Students
response shows little effort in giving the
elements of effective teamwork (one element
indicated). 0 Points---Students response shows
no knowledge about the effective teamwork.
19
V.1--TAMU TIDEE Rubric 1st Adaptation (Fall
2001) -- Used on Fall 2001 Pre-tests , Spring
2002 Post-tests, and Tests completed by TAMU
Freshman and Design Profs.
Part C Knowledge of Effective Communication  Five
specific elements articulated by students. 1.
Structure (i.e. organization, highly
understandable, flow of thoughts) 2. Content
(i.e. details, key points, clarity of ideas,
complete and accurate information) 3. Relevance
to audience (i.e. communicated well and
understandable to audience) 4. Team attitude
(i.e. co-operation, listening) 5. Involvement
(i.e. planning meetings, effective interaction
between members)
5 Points--- Students response (if include 5of 5)
shows detailed knowledge of the listed elements
of effective communication. 4Points ---
Students response shows above moderate
knowledge, if include 4 elements (of listed) of
the effective communication. 3 Points---Students
response if include 3 elements of the effective
communication, shows moderate knowledge of the
subject. 2 Points---Students response including
2 elements of the effective communication shows
little knowledge of the subject. 1
Points---Students response show little effort in
giving the elements of effective communication
(one element indicated). 0 Points---Students
response shows no knowledge about the effective
communication.
20
V.2 TAMU-TIDEE Rubric 2nd Adaptation (Spring
2002) Piloted in FC Assessment Workshop with
Faculty, at Western Michigan University, Oct 2002

• Knowledge of Engineering Design Process
• Six specific elements articulated by students.
• Information gathering
• Problem definition (i.e. understanding problem
  goals or requirements defined)
• Idea generation (i.e. brain storming, Creative
  ideas to improve the design products)
• Evaluation and decision making (i.e. analyzing
  ideas)
• Implementation (i.e. product delivered on time,
  product satisfy requirements)
• Process development (i.e. reviewed for
  improvement)

• 5 points total 1 point given for each topic
  named.
• 5 points total 1 point given for each
  definition of the main topics
• 5 points total 2 points given for listing the
  topics in the correct order
• 2 points given for showing an iteration in the
  process
• 1 point given for showing more that one iteration
  in the process
•  Grand Total of 15 Points for Section

21
V.2 TAMU-TIDEE Rubric 2nd Adaptation (Spring
2002) Piloted in FC Assessment Workshop with
Faculty, at Western Michigan University, Oct 2002

• Knowledge of Effective Teamwork
• Three specific elements articulated by students.
• Purpose and planning (goals, members know goals,
  time management, meetings)
• Rules, roles and responsibilities (members
  understand and fulfill roles to support
  teamwork team uses talents, knowledge, skills of
  all members, other resources accessed, and
  communication)
• Team attitude (i.e. polite acceptance and respect
  for each other, show pride in team and motivated
  toward team success, commitment, constructive
  criticism, and communication)

• 5 points total for each main category
• 1 point given for each underlying characteristic
  within the three elements. (Up to 3 points
  awarded for this part)
• 2 points given for examples or more explanation
  of why it is important
• Grand Total of 15 Points for Section

 
22
V.2 TAMU-TIDEE Rubric 2nd Adaptation (Spring
2002) Piloted in FC Assessment Workshop with
Faculty, at Western Michigan University, Oct 2002

• Knowledge of Effective Communication
• Five specific elements articulated by students.
• Structure (i.e. organization, highly
  understandable, flow of thoughts)
• Content (i.e. details, key points, clarity of
  ideas, complete and accurate information)
• Relevance to audience (i.e. communicated well and
  understandable to audience)
• Team attitude (i.e. co-operation, listening)
• Involvement (i.e. planning meetings, interaction)

• 5 points total 1 point given for each
  characteristic that relates to the five elements
  for communication within the team.
• 5 points total 1 point given for each
  characteristic that relates to the five elements
  for communication outside of the team.
• 5 points total
• 2.5 points for more explanations of communication
  within the team
• 2.5 points for more explanations of communication
  outside the team
• Grand Total of 15 Points for Section

23
V.3 TAMU-TIDEE Rubric 3rd Adaptation (Fall
2002) Result of WMU Faculty Feedback For Use
on Fall 2002 Electrical Engineering Design Class
Pre Post tests

• Knowledge of Engineering Design Process
• Six specific elements articulated by students.
• Information gathering
• Problem definition (i.e. understanding problem
  goals or requirements defined)
• Idea generation (i.e. brain storming, Creative
  ideas to improve the design products)
• Evaluation and decision making (i.e. analyzing
  ideas)
• Implementation (i.e. product delivered on time,
  product satisfy requirements)
• Process development (i.e. reviewed for
  improvement)

• 5 points total 1 point given for each of these
  or similar process actions named.
• 5 points total 1 point given for each addtl
  point of information about a process action.
  No more than 2 for any one action
• 5 points total 2 points given for listing
  process actions in logical, developmental order
  similar to that given here.
• 2 points given for showing an iteration in
  the process
• 1 point given for showing more that one
  iteration in the process
•  Grand Total of 15 Points for Section

24
V.3 TAMU-TIDEE Rubric 3rd Adaptation (Fall
2002) Result of WMU Faculty Feedback For Use
on Fall 2002 Electrical Engineering Design Class
Pre Post tests

• Knowledge of effective Teamwork
• Three specific elements articulated by students.
• Purpose and planning (goals, members know goals,
  time management, meetings)
• Rules, roles and responsibilities (members
  understand and fulfill roles to support
  teamwork team uses talents, knowledge, skills of
  all members, other resources accessed, and
  communication)
• Team attitude (i.e. polite acceptance and respect
  for each other, show pride in team and motivated
  toward team success, commitment, constructive
  criticism, and communication)

• 5 points total for each main category
• 1 point given for mention of each underlying
  contibutor to any of the3 main categories given.
  (Up to 3 points awarded for this part)
• 2 points given for examples or more explanation
  about how or why a category or any of its
  contributing elements is important
• Grand Total of 15 Points for Section

 
25
V.3 TAMU-TIDEE Rubric 3rd Adaptation (Fall
2002) Result of WMU Faculty Feedback For Use
on Fall 2002 Electrical Engineering Design Class
Pre Post tests

• Knowledge of Effective Communication
• Five specific elements articulated by students.
• Structure (i.e. organization, highly
  understandable, flow of thoughts)
• Content (i.e. details, key points, clarity of
  ideas, complete and accurate information)
• Relevance to audience (i.e. communicated well and
  understandable to audience)
• Team attitude (i.e. co-operation, listening)
• Involvement (i.e. planning meetings, interaction)

• 5 points total 1 point given for each
  characteristic or property that relates to the
  five elements given with regard to communication
  within the team.
• 5 points total 1 point given for each
  characteristic that relates to the five elements
  given with regard to communication with persons
  outside of the team.
• 5 points total
• 2.5 points for more information or explanations
  about communication within the team
• 2.5 points for more information or explanations
  about communication outside the team
• Grand Total of 15 Points for Section

26
Application of the TAMU-TIDEE Test for baseline
and benchmark measurement in Preparation for AERO
CRCD Smart Materials Project Evaluation.
Part II
27
Application

• The TAMU-TIDEE Design Knowledge Test was used in
  assessments which prepared for the evaluation of
  The CRCD Smart Materials Project, and also
  provided data for use in instrument revisions
• Baseline and Benchmark Measurements
• 2001-2002 AERO Senior Design Class
• 2001-2002 Freshman Engineering Class

28
Comparative Results 2001-2002 AERO Senior
Design Students
29
Content Analysis Comparison with Rubric Senior
AERO Design Students -- Knowledge of Design
Process
30
Content Analysis Comparison with Rubric Senior
AERO Design Students -- Knowledge of Teamwork
31
Content Analysis Comparison with Rubric Senior
AERO Design Students -- Knowledge of Teamwork
32
Freshman Benchmark Comparisons Beginning vs.
End of 2001-2002 Yr.
Different students in Pre and Post test groups
33
Content Analysis Comparison with Rubric Profs.
Freshman (Post-test )Knowledge of Design
Process
34
Content Analysis Comparison with Rubric Profs.
Freshman (Post-test )Knowledge of Teamwork
35
Content Analysis Comparison with Rubric Profs.
Freshman (Post-test )Knowledge of Teamwork
36
Concluding ObservationsAbout Student Performance

• Both student groups (freshmen and seniors)
• performed well in the pre-test, and improved the
  most, from pre to post testing, in question 1
  (knowledge about the design process).
• performed as well or better, in the pre-test, on
  question 2 (knowledge about teamwork) than on
  question 1, their performance declined
  considerably from pre to post testing.
• Performed worst in the pre-test and declined the
  most, from pre to post testing, in question 3-
  knowledge about communication in team design.

37
Concluding Observations

• Overall, students tended to answer questions as
  their professors did more than by the Scoring
  Rubric criteria
• The better the students performed from pre to
  post testing on a question, the more likely they
  were to respond to questions as their professors
  did
• Many of the professors answers were not in line
  with the Scoring Rubric s criteria
• Professors and students both diverged most from
  the Scoring Rubric s criteria on question 3
  (knowledge about communication) and then on
  question 2 (knowledge about teamwork)

38
Concluding Observations

• Regardless of revisions aimed at improving
  alignment of student answers with Scoring Rubric
  s criteria, performance on question 3 (
  communication in team design) did not improve
• Given the discrepancies, on question 3, between
  the Scoring Rubric criteria and the answers of
  professors, we believe that conceptions about
  what is important and what should be taught about
  communication in team design, may be different at
  Texas AM Univeristy than among the engineering
  professors who developed the TIDEE design
  assessment instrument used as a prototype for the
  TAMU-TIDEE adaptations.

39
REFERENCES END NOTES
1 CRCD Development of an Integrated
Multidisciplinary Curriculum for Intelligent
Systems. NSF Proposal Number 0088118 P I D
Lagoudas. 2 Davis, D. C. (2001). Transferable
Integrated Design Engineering Education (TIDEE),
Mid Program Assessment A Three Part Assessment
of Team Based Design for Entering Juniors.
http//www/cea/wsu.edu/TIDEE 3 The adaptations
and data collections in this presentation
involved only Part I of the TIDEE Three Part
Assessment of Team Based Design -- Design
Knowledge