Assessment Toolbox for International Educators

1
Assessment Toolbox for International Educators

• Presenters
• David Comp, Darla K. Deardorff,
• Elaine Meyer-Lee,
• Lee Sternberger, Victor Savicki

2
Session Overview

• Assessment Overview and Context
• Highlights of Specific Assessment Tools/Methods
• Tools/Methods Exploration through Roundtable
  discussions

3
Assessing International Education An Overview

• Dr. Darla K. Deardorff
• Duke University
• d.deardorff_at_duke.edu

4
Why assess international education?

• Inform continuous quality improvement
• Advocate for international education
• Satisfy regional accrediting bodies
• Add to basic understanding of student growth in
  this area
• Provide feedback to students on their personal
  growth

5
Ways of approaching assessment

• Some key points

6
ASSESSING OUTCOMES

• Starting point
• Mission Statement
• into
• Goals
• into
• Measurable Objectives

7
Assessment Cycle

• Define outcomes (based on mission/goals) and
  establish measurable criteria
• Identify appropriate assessment methods
• Collect data
• Analyze data
• Reflect on needed changes
• Design and apply changes

8
Who to involve?

• Set up
• Assessment Team
• (include students, faculty, stakeholders.)

9
Assessment Methods

• Tools determined by clearly articulated
  assessment questions
• What do we want to measure?
• What is the evidence of student success?
• DIRECT METHODS INDIRECT METHODS
• COMBINATION OF QUANTITATIVE AND QUALITATIVE

10
DIRECT METHODS

• Embedded course assessment
• Portfolios
• Performance
• Testing
• Papers/projects
• Capstone

11
INDIRECT METHODS

• Surveys (inc. self-assessments)
• Interviews
• Focus groups
• Curriculum/transcript analysis
• Documented data (student numbers)

12
Assessment Key Points

• Multi-method, multi-perspective
• Ongoing
• Integrated
• Aligned

13
Questions to consider

• What are the specific goals and objectives to be
  measured and why?
• What does the tool/method specifically measure?
• Does the tool/method match goals and objectives?

14
More questions to consider

• How will assessment data be utilized?
• (How will data be used to provide feedback to
  students? To improve the program/curriculum,
  etc?)
• Is assessment multi-method, multi-perspective,
  ongoing, intentional, integrated, part of
  strategic plan?

15
Assessment Lessons Learned

• Collaborate - Put together Assessment Team
• Adapt - Build on what you have
• Measure what is valued (align!)
• Plan - Develop assessment plan
• Use use what you assess
• Support from leadership, stakeholders
• Deardorff, 2008

16
Global Perspective Inventory (GPI)

• Presented by
• David Comp
• The University of Chicago
• dcomp_at_uchicago.edu

17
Global Perspectives Inventory (GPI)

• Developed by Larry Braskamp, David Braskamp
  Kelly Carter Merrill.
• -Larry Braskamp is a senior fellow at the
  Association of American Colleges and Universities
  (AACU). He is professor emeritus at the School
  of Education at Loyola University Chicago where
  he also served as senior vice president for
  academic affairs for four years. Braskamp also
  served as the first executive director of the
  Council for Higher Education Accreditation (CHEA).

18
Global Perspectives Inventory (GPI)

• The GPI is a survey of 46 items specifically
  designed to provide self-reports of students
  perspectives in three domains of holistic student
  development--cognitive, intrapersonal, and
  interpersonal.
• The GPI can provide evidence of students global
  perspectives
• At entry, during and at graduation from college
• At entry and conclusion of an education/study
  abroad experience
• Braskamp, Braskamp Carter Merrill, 2009

19
Global Perspectives Inventory (GPI)

• Theoretical Background of GPI
• The construction of GPI is based on the
  developmental perspective that students are on a
  journey during college. In this journey,
  students are given opportunities to reflect on
  three big questions
• How do I know? reflects the Cognitive
  dimension.
• Who am I? reflects the Intrapersonal dimension.
  
• How do I relate to others? reflects the
  Interpersonal dimension.
• Braskamp, Braskamp Carter Merrill, 2009

20
Global Perspectives Inventory (GPI)

• Description of the six GPI scales
• The Cognitive domain consists of two scales
  Knowing and Knowledge.
• The Intrapersonal domain consists of two scales
  Identity and Affect.
• The Interpersonal domain consists of two scales
  Social interactions and Social responsibility.

21
Global Perspectives Inventory (GPI)

• Validity and Reliability
• Reliability. Four GPI scales each loaded on their
  own independent factors (some with the exception
  of one or two items) considering a combination of
  both the highest and second highest loading
  scores Cognitive/Knowing, Cognitive/Knowledge,
  Intrapersonal/Affect, and Interpersonal/Social
  Responsibility. The Intrapersonal/Identity and
  Interpersonal/Social Interaction scales did not
  seem to load onto any independent factor. The
  overlapping of scales on each of the factors is
  not troubling, but rather encouraging.
  Theoretically, scholars agree these factors would
  all contribute to one concept global
  perspective.
• Validity. As students progress through college
  their development is likely to increase (Baxter
  Magolda, 1992 Pascarella Terenzini, 2005).
  Presumably, therefore, as people get older their
  global perspective may increase as well.
• Braskamp, Braskamp Carter Merrill, 2009

22
Advantages and Disadvantages

• Advantages
• GPI can be administered at
• entry, during and at graduation from college
• entry and conclusion of an education/study
  abroad experience
• GPI has only 46 survey questions and brief
  demographic questions
• GPI was developed by a leading expert in higher
  education assessment
• Disadvantages
• GPI is a relatively new assessment tool
• GPI has not been used in any research studies

23
Global Perspectives Inventory (GPI)
https//gpi.central.edu/
24
New Tools for Intercultural Learning Outcomes
Assessment

• Elaine Meyer-Lee
• Director
• Joy Evans
• Assistant Director for Research and Scholarship
• Saint Marys College
• Center for Womens InterCultural Leadership
• Notre Dame, Indiana

25
Evaluation Assessment Goals

• Evaluate programs effectiveness
• Measure Students
• Changes in sensitivity to cultural differences
• American identity development
• Own sense of growth toward goals
• Explore correlations of change with
• Intercultural engagement while abroad
  (interaction and reflection)
• Program or demographic characteristics

26
Measure of Intercultural Engagement

• Asks students to quantify the frequency of their
  engagement through
• Interaction with cultural resources
• Interaction with local people
• Explicit reflection on these interactions

27
American Identity Measure

• Adaptation of Jean Phinneys Multigroup Ethnic
  Identity Measure (good track record), and her
  much newer American Identity Questionnaire
  developed for underrepresented groups
• Based on Eriksonian identity development theory.
  Measures 2 Factors
• identity search (e.g. I have often talked to
  other people about what it means to be an
  American.)
• affirmation, belonging, and commitment (e.g.
  Being American plays an important part in my
  life.)

28
American Identity Measure

• Adaptation of Jean Phinneys Multigroup Ethnic
  Identity Measure (good track record), and her
  much newer American Identity Questionnaire
  developed for underrepresented groups
• Based on Eriksonian identity development theory.
  Measures 2 Factors
• identity search (e.g. I have often talked to
  other people about what it means to be an
  American.)
• affirmation, belonging, and commitment (e.g.
  Being American plays an important part in my
  life.)

29
American Identity Measure

• Adapted it (with Phinneys blessing) for study
  abroad students because one goal is more
  complexity in understanding ones own culture
• Students in returnee courses felt this national
  identity version captured an important dimension
• 10-item Likert-style Self Assessment (5 for each)
• Currently refining adaptations psychometric
  properties, but alpha coefficients good so far
  (.79)

30
Sources

• Phinney, J. (1992). The Multigroup Ethnic
  Identity Measure A new scale for use with
  adolescents and young adults from diverse groups.
  Journal of Adolescent Research, 7, 156-176.
• Phinney, J., Devich-Navarro, M. (1997).
  Variations in bicultural identification among
  African American and Mexican American
  adolescents. Journal of Research on Adolescence,
  7, 3-32.

31
Measure of Intercultural Engagement

• Asks students to quantify the frequency of their
  engagement through
• Interaction with cultural resources
• Interaction with local people
• Explicit reflection on these interactions

32
Benefits to Study Abroad Assessment
Challenges to Institutional Research

• MEI
• Comparable program specific data on students
  intercultural interactions
• AIQ
• Examine identity dimension, connect to literature
  on this

• MEI
• Labor-intensive development and administration
• AIQ
• Reliability and Validity of adaptation still
  being established
• Needs refining

33
Intercultural Leadership Certificate Program

• Capture Saint Marys strengths and CWILs unique
  location at the intersection of intercultural
  (local and global) and leadership education in
  the context of a womens college
• Guide students through a developmental process
  into a deeper capacity for intercultural
  leadership
• Bring coherence and provide an organizing
  framework for every effort (curricular and
  co-curricular) being undertaken at Saint Marys
  College to encourage intercultural leadership
• Portfolio assessment of students proficiency as
  an intercultural leader in
• 6 proficiency areas
• Recognize the Leader Within
• Articulate Your Ethical/Spiritual Center
• Engage With Value Diversity
• Dialogue on Power Privilege
• Create Inclusive Equitable Community
• Make Your Difference in the World

34

• For more handouts or bibliography on assessing
  intercultural outcomes of study abroad programs,
  go to
• http//www.saintmarys.edu/cwil/php/intercultural
  .learning/IILOutcomes.php
• or email me meyerlee_at_saintmarys.edu

35
Sociocultural Adaptation Scale (SCAS)

• Presented by
• Vic Savicki, PhD

36
Sociocultural Adaptation Scale (SCAS)

• Developed by Colleen Ward and Antony Kennedy
• Ward, C. Kennedy, A. (1999). The Measurement of
  Sociocultural Adaptation. International Journal
  of Intercultural Relations, 22, 659-677.
• Several subsequent supporting studies

37
ABC Theory of Acculturation

• Ward, C. (2001). The A, B, Cs of acculturation.
  In D. Matsumoto (Ed.), Handbook of Culture and
  Psychology. (pp. 411-446). NY Oxford University
  Press.
• A Affect based on stress and coping theory
• B Behavior based on learning cultural knowledge
  and skills
• C Cognition based on social identification
  theory..

38
Sociocultural Adaptation Definition

• SCAS measures the Behavior aspect of the ABCs
• The ability to fit in, to acquire culturally
  appropriate skills and to negotiate interactive
  aspects of the host environment.
• Applied to both foreign nationals coming to the
  U.S and U.S. nationals going abroad..

39
SCAS Scales

• SCAS Total Sum/average of 29 items
• Two sub-scales based on a factor analytic study
• Cultural Empathy and Relatedness (13 items)
  (32.1 of variance)
• Understanding local perspectives, values, world
  views, and communication
• Impersonal Endeavors and Perils (7 items) (8.7
  of variance)
• Management of impersonal interactions and/or
  awkward situations..

40
Advantages and Disadvantages

• Based on a theory of acculturation
• Tracks key intercultural adaptation issues
• Gives several scores to interpret
• Predicts an overall progression of adaptation
• Can be used formatively and summatively
• List may be incomplete
• List may have cultural bias..

41

• The Beliefs, Events, and Values Inventory
• (BEVI)
• Developed by Craig N. Shealy, Ph.D.
• International Beliefs and Values Institute
• James Madison University
• shealycn_at_jmu.edu
• Presented by Lee Sternberger, James Madison
  University

42

• The Beliefs, Events, and Values Inventory
• (BEVI)
• What does the BEVI measure?
• The BEVIis designed to assess a number of
  processes relevant to international and
  multicultural learning including (but not limited
  to) basic openness receptivity to different
  cultures, religions, and social practices the
  tendency (or not) to stereotype in particular
  ways self and emotional awareness and preferred
  but implicit strategies for making sense of why
  other people and cultures do what they do
  (Shealy, 2005, p. 99).

43

• The BEVI
• Has been in development since the early 1990s
• Empirically and theoretically grounded (e.g., in
  Equilintegration or EI Theory and the EI Self
  Shealy, 2004, 2006, in press)
• Strong evidence of good reliability and validity
• Is derived from belief-value and life event
  statements as well as background information
  items

44
The BEVI

• Is comprised of two validity scales, eighteen
  process scales, and three qualitative
  experiential reflection items
• Does not appraise correctness of individual
  responses (e.g., whether responses are right or
  wrong) but facilitates 1) comparisons between
  individuals and groups to the normative sample as
  well as 2) the association of response patterns
  to a wide range of processes and outcomes (e.g.,
  who learns what and why, and under what
  circumstances)

45
The BEVI

• Includes 65 background/demographic items, 334
  belief, value, and life event items rated on a
  four-point Likert scale (strongly agree, agree,
  disagree, strongly disagree)
• Takes 45-55 minutes to complete
• Is available online and accessible globally
• Allows for a wide range of analyses, reports, and
  feedback at the individual, group, and
  organizational level

46
The BEVI

• Sample Background Variables (65 on
  Current BEVI)
• Age
• Gender
• Ethnicity
• Citizenship
• Countries Visited
• Sample BEVI Scale Items (334 on Current
  BEVI)
• There are too many foreigners in our country.
• Everyone could benefit from learning
  more than one language.
• Hinduism and Buddhism have much to
  teach the modern world. If it comes down to it,
  the man must act as head of the house.
• I worry about the health of our
  planet.

47
BEVI Scales

• I. Validity Scales
  VI. Self Access
• Consistency 10. Emotional
  Attunement
• Congruency
  11.
  Cognitive Control
• II. Formative Variables
  12. Self Awareness
• Demographic and Background Items
  VII. Openness to Others
• 1. Negative Life Events
  13.
  Socioemotional Openness
• III. Fulfillment of Core Needs
  14. Sociocultural Openness
• 2. Needs Closure
  15.
  Socioreligious Openness
• 3. Identity Closure VIII.
  Global Worldview
• IV. Tolerance of Disequilibrium
  16. Gender Awareness
• 4. Basic Openness 17.
  Ecological Resonance
• 5. Basic Closedness
  18. Global
  Engagement
• V. Critical Thinking
  IX. Experiential Reflection Items
• 6. Causal Closure
• 7. Naïve Determinism
• 8. Refined Determinism
• 9. Socioreligious Determinism

48
The Forum BEVI Project
49
The Forum BEVI Project

• SAMPLE BEVI STUDY
• Mean Differences on Significant Intercultural
  Openness BEVI Items Between Participants Who
  Have and Have Not Participated in an
  International Learning Program1
• No Participation
  Some Participation
• 
  (N 460) (N 564)
• We should try to understand cultures that are
  different from our own. 3.642
  3.74 3
• Knowledge of other cultures helps me understand
  my own culture. 3.28 3.45
• In this country, everyone should be required
  to speak the same language. 2.18
  2.09 (p .086)
• Everyone could benefit from learning more than
  one language. 3.34 3.50
• I enjoy learning about other cultures.
  3.41 3.57
• Too many foreigners live in our country.
  2.16 1.93
• It is important to be informed about world
  events. 3.49 3.57

50
The Forum BEVI Project

• Sample BEVI Study
• Experiential Reflection Item Responses
•  
• Question 1. Which event or aspect of your
  international learning experience had the
  greatest impact upon you and why?
• Interacting with the people of different cultures
  in their everyday lives helped me to understand
  the motivations behind world affairs.
• Everyday interaction with people from another
  culture and using a language other than my own
  it forced me to look at myself as an other and
  to see what someone else might see, and to
  critique my own culture and myself.

51
The Forum BEVI Project

• Sample BEVI Study
• Experiential Reflection Item Responses
•  
• Question 2. Was there some aspect of your own
  self or identity that became especially
  clear or relevant to you or others as a
  result of this experience?
• My feelings about my home country and certain
  cultural practices became more clear. Because I
  had to really think about the origins and motives
  behind many cultural practices, I was able to
  have clearer opinions about them.
• The experienced actually produced more ambiguity
  about my own identity.
• Yes, I learned I want to further my education by
  helping other people.

52
The Forum BEVI Project

• Sample BEVI Study
• Experiential Reflection Item Responses
•  
• Question 3. What have you learned and how are
  you different as a result of this experience?
• I learned about what it means for me to be an
  American in this world of increasing
  globalization.
• I have learned that there is a global community.
  We are all people just living our lives the best
  we can in different environment. We are required
  to help each other enjoy all the benefits of
  life.

53
Your Turn.

• Join us for
• Roundtable Discussions on the specific tools
  youve just heard about ask more questions,
  learn more in detail about how and when to use
  these instruments

54
Resources

• ACEs Internationalization website
  http//www.acenet.edu/Content/NavigationMenu/Progr
  amsServices/cii/res/assess/index.htm
• NCSUs Assessment Website http//www2.acs.ncsu.edu
  /UPA/assmt/resource.htm
• SIETAR Europas website http//www.sietar.de/SIETA
  Rproject/Assessmentsinstruments.htmlTopic26
• Fantinis List www.sit.edu/SITOccasionalPapers/fe
  il_appendix_f.pdf
• Forums Guide to Outcomes Assessment in Education
  Abroad (Forum, 2007) ed. by Mell Bolen
• The SAGE Handbook of Intercultural Competence
  (Sage, 2009) ed. by D.K. Deardorff
• Deardorff, D.K. Principles of International
  Education Assessment. In IIENetworker (Spring
  2007).
• Maki, P. Assessing for Learning (2004, Stylus)