The USG GLOSSARI Project on Study Abroad Learning Outcomes

1
The USG GLOSSARI Project on Study Abroad Learning
Outcomes

• Presented by Emily Gung (Graduate Research
  Assistant)
• PI
• Richard C. Sutton
• University System of Georgia Board of Regents
• Director of Research
• Don Rubin
• Speech Communication
• Language Literacy Education
• Linguistics
• GAIE Conference
• 18 July 2008

2
Agenda

• Information and overview about the GLOSSARI
  initiative, outlines of the project
• GLOSSARI project update and recent findings
• Related research projects

3
Growth of USG Study Abroad Participation

• 15 average annual growth rate

4
Significance of Study Abroad in Higher Education

• Nearly 200,000 U.S. students currently go abroad
  each year
• About 8 of all UG degree recipients have part of
  their education abroad
• Students can earn up to one-fourth of their
  academic degree from overseas study

• Study abroad participation among all U.S.
  students increased 20 since 2001.
• Georgia participation will triple within 10 years
  (1997-2007)
• New federal funding initiatives to increase study
  abroad

5
Accountability in Higher Education

• Evaluating resource allocation (e.g., cost/credit
  hr)
• Evaluating instructional processes (e.g.,
  technology utilization, 6-year graduation rate,
  studying abroad)
• Evaluating learning outcomes
• Most appropriate from the teaching-and-learning
  paradigm
• Emphasis on value-added
• Treats study abroad as integral to academic
  affairs, not enrichment

6
The Teaching and Learning Paradigm
7
Assessing Student Learning Outcomes

• Focus Discerning what students
• Know (cognitive),
• Do (skill)
• as a result of the educational program
• Purpose To help faculty to answer the following
  questions
• Are our students learning what we think is
  important?
• Are they learning what they need to succeed in
  this field or profession?
• Are we improving in our ability to help students
  learn?
• Should our curriculum or teaching strategies be
  modified?
• Goal To close the feedback loop between
  assessment and program improvement

8
Student Learning Outcome Assessment Techniques
Direct
Indirect

• Standardized tests
• Performance on licensure or professional exams
• Essays
• Exhibits
• Performances
• Course assignments
• Portfolios of work samples
• Authentic assessments
• Job placement rates

• Student surveys
• Graduate follow-ups
• Focus groups
• Exit interviews

9
Assessment Efforts to Date in Study Abroad

• Strong research efforts to assess second language
  acquisition learning outcomes from study abroad
• Some good studies on documenting gains in
  intercultural sensitivity and personal growth
• Limited attention paid to knowledge and skills
  acquired abroad

10
Prior SA foreign language research indicates

• SASD for FL oral proficiency
• SDSA for grammatical accuracy
• SASD for conversational competence
• SASD for pronunciation
• Reciprocal causation of prior FL proficiency on
  SA gains
• B. Freed,
  et al. R. Brecht et al ACTR

11
Prior SA psychosocial research indicates

• SA SD for self-efficacy, autonomy
• SA or SD for ethnorelativism (reported decline
  in IDI scores)
• SASD for mature life plans, career orientation
• SASD for cultural pluralism
• SA or SD for world/global mindedness
• SASD for cultural interest

12
GLOSSARI

• G EORGIA
• L EARNING
• O UTCOMES OF
• S TUDENTS
• S TUDYING
• A BROAD
• R ESEARCH
• I NITIATIVE

13
Primary Objectives of GLOSSARI

• Identify cognitive learning outcomes attributable
  to
• diverse study abroad experiences
• for students at a wide variety of public
  institutions
• Identify impact on academic performance
  indicators
• Identify impact of study abroad on core liberal
  arts aspirations (critical thinking, leadership,
  adaptability, etc.)
• Identify program characteristics that optimize
  learning outcomes, so as to guide program
  development
• Identify student characteristics that predict
• likely participants
• successful participants
• Refine, replicate, and disseminate methods for
  assessing the impact of study abroad on student
  learning outcomes.

14
Federal Grant to Accelerate the GLOSSARI Research
Project

• GLOSSARI project began in 2001 with modest
  internal fundingfocused on self-reported
  learning in domains that cut across disciplines
• U.S. Department of Education International
  Research Studies Program Grant for 2006-2009
• Six components examine discipline-specific and
  cross-disciplinary learning outcomes

15
6 Phases

• Phase I Learning Outcomes of SA Participants
  Non-Participants
• Phase II Pre- and Post-participation Learning
  Outcomes with multiple measures
• Phase III Teaching the same course content
  abroad at home
• Phase IV Academic performance measures among SA
  participants and non-participants
• Phase V Program design features that make a
  difference
• Phase VI Impact 2- to 5-year post-graduation

16
Inside 6 Phases of GLOSSARI

• Study I Comparing learning outcomes of SA
  participants domestic students
• self-reported
• Pre-test/post-test
• matching within disciplines and year in college
• skills and knowledge testing
• Study II Intercultural learning outcomes
  validation
• Compared with open-ended knowledge questions
  (name 6 African countries, describe a cultural
  contrast
• correlated with other testing instruments (e.g.,
  Intercultural Development Inventory)
• Study III Teaching the same course content
  abroad at home
• evaluating comparable learning artifacts across
  settings

17
Inside 6 Phases of GLOSSARI (cont.)

• Study IV Comparing academic performance
  indicators of SA participants and
  non-participants
• GPA
• Time to graduation
• Professional certification results
• Study V Program design features that make a
  difference in learning outcomes
• Duration
• Immersion factors
• Curriculum transformation
• Stage VI Impact 2- to 5-year post-graduation
• Career path
• Intercultural friendships and contact
• Residence patterns

18
What weve learned so far.

• Better navigational skills
• Improved academic performance upon return
• Much higher persistence and graduation rates

19
Phase I Spr 07 GLOSSARI Factor Analysis
A snapshot of our findings.
20
Design

• N 298
• Study Abroad Domestic Participants
• Pre-term
• Spring 2007 Semester
• 58 study abroad programs exchanges
• 11 USG institutions

21
ILO Intercultural Learning Outcomes
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
Learning Outcomes (Rubin, et al)

• Factor 1 Knowledge of How to Function in a
• Foreign Country
• Factor 2 Knowledge of Global Policy
• Factor 3 Understanding Ethno-Relativism
• Factor 4 Knowledge of Verbal Resources
• Factor 5 Knowledge of World Geography
• Factor 6 Knowledge of How to Interact with
• People from Other Cultures
• Factor 7 Knowledge of Self as a Cultural
• Being

34
Hypothesized 7-Factor Model (Rubin, et al)

• Rectangles are measured variables
• Large circles are latent constructs
• Small circles are residual variances.
• Factor loadings are standardized and all are
  significant (p

35
Confirmatory Factor Analysis (Rubin, et al)
36
Factor 1 How to Function

• I know how to find different ways to express an
  idea that I am having trouble saying to a
  foreigner.
• I know how to locate basic information in a
  foreign newspaper.
• I know what kinds of jokes would be funny to a
  person from a foreign country.
• I know enough about the essential norms and
  taboos (greetings, dress, behavior) of a foreign
  culture to adjust my behavior appropriately.

• I know how to take a train or bus between cities
  in a foreign country.
• I know how to use a public telephone in a foreign
  country.
• I know how to locate a safe and comfortable
  nightclub or bar in a foreign country.
• I know how to buy toothpaste or a can opener in a
  foreign country.
• I know how to pacify an angry person (e.g.,
  shopkeeper) in a foreign culture.
• I know how to give coherent, logical directions
  in a foreign country.

37
Comparison between study-abroad participants and
nonparticipants on functional knowledge of
cultural practices (p 38
Factor 2 Global Policy

• I understand how world markets overseas can
  affect my intended career.
• I understand how foreign manufacturing affects
  the prices of consumer goods (e.g., clothing) in
  the US.
• I know how to explain US foreign policy to my
  peers.
• I understand why US troops are concerned about
  the military situation in North Korea.
• I know how the political and personal freedom
  experienced by US citizens compares and contrasts
  with the degree of freedom experienced by
  citizens in another country.

39
Comparison between study-abroad participants and
nonparticipants on knowledge of global
interdependence (p 40
Factor 3 Ethno-Relativism

• I understand the significance of differences
  among languages and cultures.
• I know the importance of not judging others.
• I understand my own reactions to people from
  different language and cultural backgrounds.
• I know how to interact with people of different
  cultures.
• I am sensitive to differences among languages and
  cultures.

41
Comparison between study-abroad participants and
nonparticipants on sensitivity to cultural
context (p 42
Factor 4 Verbal Resources

• I know when it is to my advantage to take risks.
• I know how to talk my way out of difficult
  situations.
• I know how to lead discussions and conversations.
• I know how to find different ways to express an
  idea that I am having trouble saying to a
  foreigner.

43
Comparison between study-abroad participants and
nonparticipants on knowledge of verbal resources
(not statistically significant p.05)
44
Factor 5 World Geography

• I know the names of at least 6 countries in
  Africa.
• I know the capitol cities of at least four
  countries in South America.
• I know the names of at least four rivers in
  Europe.
• I know the names of at least three rivers in Asia
• I know some contributions of foreign researchers
  or scholars to my academic major.

45
Comparison between study-abroad participants and
nonparticipants on knowledge on world
geography(p 46
Factor 6 Interact w/ Other Cultures

• I know how to be patient when interacting with
  people.
• I know how to be flexible when interacting with
  people.
• When interacting in a foreign country, I know how
  to talk my way out of difficult situations.

47
Factor 7 Self as a Cultural Being

• I know the basic definition of culture and I
  can identify its components.
• I am sensitive to how specific settings affect my
  style of interacting with others.

48
Comparison between study-abroad participants and
nonparticipants on knowledge of the self as a
cultural being (p 49
(No Transcript)
50
(No Transcript)
51
Adequate Sampling
52
(No Transcript)
53
(No Transcript)
54
(No Transcript)
55
(No Transcript)
56
Reliability

• Factor 1 Cronbach a .905
• Factor 2 Cronbach a .739
• Factor 3 Cronbach a .674
• Factor 4 Cronbach a .801
• Factor 5 Cronbach a .665
• Factor 6 Cronbach a .806
• Factor 7 Cronbach a .354

57
T-test
58
(No Transcript)
59
ANOVA
60
Correlations Among FactorsFactor 1 Verbal
adeptnessFactor 2 Sensitivity to cultural
contextFactor 3Knowledge of self as cultural
beingFactor 4 Knowledge of cultural
practiceFactor 5 Knowledge of world
geographyFactor 6 Knowledge of global
interdependence
61
Persistence Toward Graduation at Test Sample of
Three USG State Universities
Blue column includes 2000-2001 study abroad
students from three universities who had
graduated or were still enrolled in Spring
2003. Red column represents average graduation
rate at the three institutions.
62
Related Research

• Michael (Mick) Vande Berg
• Quality, Accountability, and Research in U.S.
  Study AbroadMick Vande Berg is a comparative
  literature professor by training, and currently
  serves as Vice President for Academic Affairs and
  Chief Academic Officer for the Council on
  International Educational Exchange (CIEE).

63
Improved Graduation Rates
University of Minnesota study of 1999 entering
freshman class.
64
especially among lower-aptitude students
(ACT/SAT HSGPA index)
University of Minnesota study of 1999 entering
freshman class (cohorts with low admissions
index AAR 100-119)
65
and among students of color
University of Minnesota study of 1999 entering
freshman class.
University of Minnesota study of 1999 entering
freshman class.
66
Next Steps, Questions, Challenges

• Analyze assessment results (Ph 1, 2, 3)
• Correlations b/t Critical Thinking Test and Other
  Knowledge / Skills Assessments (Ph 2)
• Rate Content Learning Abroad vs. At-Home (Ph 1,
  pre/post)
• Cont collecting CATs (Ph 3)
• Analyze Academic Performance Measures (Ph 4)
• Correlate Engle Engle taxonomy of program
  design elements with Ph1 SLO (Ph 5)
• Follow-up with students lives 2-5 yrs. down the
  road.. (Ph 6)

67
Really BIG Questions of Intrigue

• What is academic learning anyway? What should
  count as a learning outcome and what should count
  as an outcome of some other sort, say, personal
  development?
• What academic value is added by deliberate
  instruction, as opposed to ambient learning? By
  learning in the classroom as opposed to learning
  on the streets?

68
Really BIG Questions of Intrigue (cont.)

• When are two classes rightfully instances of the
  same course? Can/Should a class taught abroad
  constitute the same course as one taught
  domestically?
• What counts as studying abroad? How are degrees
  of cultural immersion to be gauged? Are all
  students in the same SA program equally immersed?
• To what degree are SA learning outcomes
  predetermined by student predilections (and self
  selection) rather than by program interventions?

69
For Further Information

• On-going reports on this project will be posted
  periodically on our website
• www.usg.edu/oie
• Or contact us at
• Office of International Education
• University System of Georgia Board of Regents
• 270 Washington Street SW
• Atlanta, GA 30334
• Tel [phone]
• Richard.sutton_at_usg.edu or Reliang.Tsang_at_usg.edu