Assessing Chinese Language Development: Matching Appropriate Assessments to Individual, Class, and P

1
Assessing Chinese Language Development Matching
Appropriate Assessments to Individual, Class, and
Program Needs
2
Introductions Marty Abbott - ACTFLT. Richard
Chi - University of UtahKyle Ennis - Avant
Assessment
3
Why do we teach Chinese in Secondary Schools?
4
Why Do We Assess Students?How Do We Assess
Students?When Do We Assess Students?Formative
SummativeAchievement Performance
ProficiencySelf-Evaluation External
Evaluation
5
Assessment Foundations
Assessment is a process of professional
judgment. Assessment is based on separate but
related principles of measurement evidence and
evaluation. Assessment decision-making is
influenced by a series of tensions. Assessment
influences student motivation and
learning. Assessment contains error.
6
(No Transcript)
7
Assessment Modalities
Assessment FOR Learning Assessment AS
Learning Assessment OF Learning
8
Common Assessments.....
STAMP / ClassPak Linguafolio
9

• STAMP / ClassPak
• Administered on-line
• Assess reading, writing and speaking
  proficiencies
• Graded by computer (reading) and computer-aided
  humans (writing and speaking)
• Analysis reports provided for teachers
• Items verified by rigorous statistical analysis
  (piloting)
• Utilize Computer Adaptive Testing (CAT)
  technology

10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
Uses of STAMP
Large Scale Test Administration Program
Evaluation Progress Tracking/Placement Test
Preparation
16
(No Transcript)
17
Linguafolio.....
Language Passport - Summary of Skills / Can Do
Statements Language Biography - Learner
Background Dossier - Work Samples / Collection
of Evidence
18
(No Transcript)
19
ACTFL Proficiency Testing

• Marty Abbott, Director of Education
• ACTFL
• mabbott_at_actfl.org
• April 2008

20
Proficiency is.

• the ability to use a language to communicate
  meaningful information in and spontaneous
  interaction, and in a manner acceptable and
  appropriate to native speakers of the language.

21
Proficiency is not an all or nothing

• A speaker is proficient at something.
• A speaker is proficient to a degree.

22
ACTFL Proficiency Tests

• Assess Functional Language Ability
• What a person can do with language in real life
• Not Curriculum or Program Dependent
• Recognize Competence Gained in Non-Traditional
  Settings
• Fair and Non-threatening
• Adaptive and Interactive
• Open ended - Free response
• ACTFL Proficiency Guidelines provide basis for
  rating

23
What is Language Proficiency?

• Unrehearsed general ability to accomplish
  real-world communication tasks across a wide
  range of topics and settings
• The ability to use language to accomplish real
  world linguistic tasks
• Order a meal
• Tell a story
• Defend a point of view
• Negotiate a contract

24
ACTFL Proficiency Guidelines

• Based on the ILR (Interagency Language
  Roundtable) language descriptors
• Published in 1986 Revised in 1999
• Four major levels
• Novice
• Intermediate
• Advanced
• Superior

25
Novice

• Memorized language
• Lists words/phrases
• Telegraphic
• Attempts at conversation
• Reactive
• Limited topic areas
• Social courtesies
• Dates, numbers, colors
• Family, home, common objects

26
Intermediate

• Survival Proficiency
• Has sufficient language to express own meaning
• Engages in simple conversation
• Can deal with a simple social transaction
• Asks and answers questions

27
Advanced

• Limited Work Proficiency
• Speaks with confidence
• Can narrate and describe in all major time frames
• Can elaborate, clarify, illustrate
• Can handle a situation with a complication
• The Story Teller

28
Superior

• Professional Proficiency
• Can support opinions and hypothesize
• Converse both formally and informally
• Handle abstract treatment of topic

29
The ACTFL OPI

• Face-to-face or telephonic assessment
• Interactive and adaptive
• Measure of consistent functional ability
• Standardized structure and protocol
• Topics from the interests experiences of
  interviewee
• Criterion-referenced evaluation
• Function, Content/Context, Text type, Accuracy
• Rating based on the ACTFL Proficiency Guidelines
  Speaking (Revised 1999)

30
The ACTFL OPI

• The ACTFL Oral Proficiency Interview provides
  evidence of
• What a person can DO in a language
• Order a meal
• Tell a story
• Defend a point of view
• Negotiate a contract

31
The ACTFL OPI Protocol

• Warm Up
• Establish the level at which speaker can sustain
  criteria
• Prove the level at which the speaker can no
  longer sustain criteria
• Elicit a sample that demonstrates patterns of
  strength and patterns of weakness (ratable
  sample)
• Role play situation to prove function(s) that
  cannot be elicited through conversational mode
• Wind Down

32
The ACTFL OPIcAdapting the ACTFL OPIto a
Computer-delivered Format
33
What is the ACTFL OPIc?

• A semi-direct English oral proficiency test
  administered by computer
• Accessed via the Internet
• Based on the structure of the ACTFL OPI
• Adaptive and interactive
• Test takers complete background survey and self
  assessment
• Valid and reliable according to industry
  standards
• Rated against same criteria as the ACTFL OPI
• Scores correspond to OPI scores
• Rated by certified ACTFL raters

34
What makes the ACTFL OPIc unique?

• ACTFL Certified Rating System
• ACTFL trained and certified raters
• Ongoing Quality Assurance

• Adaptive Features
• Each test is unique
• Background survey answers determine topic
  choices
• Self-assessment determines initial level

• Interview Format
• Uses the interview format of the OPI
• Is like a conversation with a live tester
• Produces a sample of speech that can be rated

Secure Valid Reliable
Each test is individualized
Conversational format
35
Background Questionnaire

• Work
• School
• Home
• Activities
• Interests
• Sports
• Travel

36
Self Assessment

• I can say only a few words (less than 10) in
  English.
• I can name basic objects, colors, days of the
  week, foods, clothing items, numbers etc. I
  cannot always make a complete sentence or ask
  simple questions.
• I can give some basic information about myself,
  work, familiar people and places, and daily
  routines speaking in simple sentences. I can ask
  some simple questions.
• I can participate in simple conversations about
  myself, daily routines, work/school and hobbies.
  I can easily produce a series of simple sentences
  on these familiar topics and routines. I can
  also ask questions to get what I need.
• I can participate in conversations about familiar
  topics and activities related to home,
  work/school, personal and community interests. I
  can speak in connected discourse about things
  that have happened, are happening and will
  happen. I can explain when asked. I can handle
  routine situations, even when there may be an
  unexpected complication.
• I can participate confidently in any conversation
  and/or discussion about work/school, personal
  interests and current events. I am able to
  elaborate at length and in detail on most topics
  with a high level of accuracy and a wide range of
  vocabulary.

37
Question Types by Level
38
Novice

• Lists
• Name all the people in your family Mother,
  sister name all the people in your family.
• What do you do on the weekend? Sleep, read name
  all the things you do on the weekend.
• What colors do you see in the restaurant? Brown,
  black what other colors?

39
Intermediate

• Create, ask answer simple questions
• Tell me about your pet. What kind of animal is
  it? What does your pet look like? Is there a
  special place your pet stays?
• What does your manager do on a typical day? What
  are some of his or her responsibilities?
• I live in a house in the U.S. Now ask me 3 or 4
  questions to learn more about my house.

40
Advanced

• Narrate and describe in major time frames
• Tell me the story of one very memorable
  experience you had while visiting a park. Maybe
  something funny, unexpected or wonderful
  happened. Tell all about that experience. Start
  with some background about when and where this
  took place, who you were with and what you were
  doing in the park and then tell me the story of
  what happened that day to make that trip so
  unforgettable or special.
• Most people say that travel has become more
  difficult in the past five years. Tell me what
  types of changes you have observed when traveling
  and how these changes affect the traveler or
  travel experience.

41
Test Construct

• Designed to replicate real OPI
• Mimics a one-on-one conversation with a native
  speaker of the target language
• Addresses required functions at each of the
  levels
• Testlet structure provides follow up on each
  topic to elicit maximum discourse

42
Certified Raters

• Native/Superior level speakers of language
• Language professionals
• Face-to-face and online training
• Must meet criteria for certification
• Must maintain high rating reliability

43
Scores
Reported holistically and according to ACTFL
Proficiency Guidelines for Speaking
OPI LevelsNovice - Superior

• Superior
• Advanced High
• Advanced Mid
• Advanced Low
• Intermediate High
• Intermediate Mid
• Intermediate Low
• Novice High
• Novice Mid
• Novice Low

OPIc LevelsNovice - baseline Advanced
44
Rate Major Levels

• The major levels in the OPIc are
• Advanced
• Intermediate
• Novice

45
Rate Sublevels

• The sublevels in the OPIc are
• Intermediate High
• Intermediate Mid
• Intermediate Low
• Novice High
• Novice Mid
• Novice Low
• There is only a baseline Advanced rating

46
Status of OPIc Testing

• Operational
• English and Spanish
• Over 20,000 OPIcs conducted to date
• OPIcs in Development
• ACTFL OPIc to Superior
• ILR OPIcs - Arabic, Chinese Mandarin, Korean,
  Persian Farsi, Bengali, Cambodian, Hausa, Yoruba
• Near-Future Development Plans
• French, German
• Other Languages

47
Contact Information

• For information on ACTFL, OPI, OPIc
• testing_at_actfl.org
• To schedule an ACTFL Proficiency Test (OPI, WPT,
  OPIc)
• testing_at_languagetesting.com

48
SAT Test in Chinese with Listeningand AP
Chinese Language and Culture Exam National
Chinese Language ConferenceT. Richard Chi
49
SAT Test in Chinese with Listening

• Purpose
• Measure understanding of Mandarin Chinese in the
  context of contemporary Chinese culture
• 85 multiple-choice questions 20 minutes of
  listening and 40 minutes of usage and reading
• Skills measured Listening Comprehension
  (approximately 33 of test) Usage (33) Reading
  Comprehension (33)
• Total Score 200-to-800 scale listening, usage,
  and reading subscores 20-to-80 scale
• Target Group
• study of Chinese as a second language for two to
  four years in high school, or the equivalent
• gradual development of competence in Chinese over
  a period of years

50
SAT Chinese Listening Comprehension

• Purpose test the ability to understand spoken
  Chinese primarily about everyday topics.
• A spoken statement, question, or exchange
  followed by 3 possible spoken response
• A spoken dialogue or monologue with a printed
  question or questions in English about what was
  said.

51
SAT Chinese Usage

• Select the answer that best completes a Chinese
  sentence that is structurally and logically
  correct.
• Questions are written to reflect instructional
  practices of the curriculum.
• Presented in four columns across two pages of the
  test book to allow each question and its answer
  choices to be shown in four different ways of
  representing Chinese traditional and simplified
  Chinese characters on the left page, and phonetic
  transcriptions in Pinyin romanization and the
  Chinese phonetic alphabet (Bopomofo) on the right
  page.

52
SAT Chinese Reading Comprehension

• Test understanding of main and supporting ideas,
  themes, and the setting of passages.
• Some of the passages are based on real-life
  materials such as timetables, forms,
  advertisements, notes, letters, diaries, and
  newspaper articles.
• All passages are written in both traditional and
  simplified Chinese characters.
• Most questions deal with understanding of literal
  meaning although some inference questions may be
  included.
• All reading comprehension questions are in
  English.

53

• Chinese with Listening
• http//www.collegeboard.com/student/testing/sat/lc
  _two/chinese/chinese.html?chinese
• Offered only in November at designated test
  center
• Fees 40

54
Why Take SAT Chinese Test

• Measure/demonstrate knowledge and skills in
  Chinese, as well as ability to apply that
  knowledge in primarily everyday life situations
• Many colleges use the Subject Tests for
  admissions, for course placement, and to advise
  students about course selection.
• Used in combination with other background
  information (high school record, scores from
  other tests like the SAT Reasoning Test, teacher
  recommendations, etc.), they provide a dependable
  measure of a students academic achievement and
  are a good predictor of future performance.
• Some colleges specify the Subject Tests they
  require for admissions or placement others allow
  applicants to choose which tests to take.

55
SAT Chinese Test

• Test mainly skills of interpretive communication
• Authentic stimulus for listening and reading
  comprehension questions
• Focus on context and content areas of everyday
  life
• Usage part of test focuses on grammar knowledge
  and is presented in a non-contextualized manner

56
AP Chinese Language and Culture Exam

• Test knowledge of Chinese language and culture
  (contemporary and historical) and the ability to
  communicate in Chinese in linguistically,
  culturally and socially appropriate ways.
• Focus on interpersonal, interpretive and
  presentational communication using the four
  skills
• Test scaffolding ability to interact with
  authentic materials
• Rubrics-based holistic assessment of students
  responses
• Test delivered on CD

57
AP Chinese Exam Format

• Section I Multiple Choice
• 70 Questions 50 of final score 80 minutes
• Part A Listening
• Part B Reading
• Section II Free Response
• 12 Questions 50 of final score 85 minutes
• Part A Writing
• Part B Speaking

58
Section I Listening

• Rejoinders 10-15 questions 10 minutes
• Test interpersonal communication
• Using set phrases and social formulae
  communicating opinion, attitude, intent
• Sample Stimulus Types 10-20 questions 10
  minutes
• Announcement
• Conversation
• Instructions
• Message
• Report

59
Section I Reading

• 35-40 questions 60 minutes
• Test interpretive communication
• Comprehension inference application of
  introductory cultural knowledge
• Sample Stimulus Types 10-20 questions 10
  minutes
• Advertisement
• Article
• E-mail
• Note
• Poster
• Sign
• Story

60
Section II Writing

• STORY NARRATION
• 1 question 15 minutes
• Test presentational communication
• Narrating a story as depicted by series of
  pictures
• PERSONAL LETTER
• 1 question 30 minutes
• Test presentational communication
• Informing describing expressing preference
  justifying opinion
• E-MAIL RESPONSE
• 1 question 15 minutes
• Test interpersonal communication
• Reading responding to request
• RELAY TELEPHONE MESSAGE
• 1 question 6 minutes
• Test presentational communication
• Listening summarizing message conveying
  important details

61
Section II Speaking

• CONVERSATION
• 6 question 5 minutes
• Test interpersonal communication
• Participating in conversation by responding
  appropriately
• CULTURAL PRESENTATION
• 1 question 7 minutes
• Test presentational communication
• Describing and explaining significance of a
  Chinese practice or product
• EVENT PLAN
• 1 question 7 minutes
• Test presentational communication
• Describing comparing contrasting explaining
  justifying applying introductory cultural
  knowledge

62
Scoring Criteria

• TASK COMPLETION
• response to the prompt
• text type and
• organization
• DELIVERY
• pace
• pronunciation
• register
• LANGUAGE USE
• vocabulary
• grammatical structures

63
Scoring Ratings

• 6 Excellent Demonstrates excellence
• 5 Very Good Suggests excellence
• 4 Good Demonstrates competence
• 3 Adequate Suggests competence
• 2 Weak Suggests lack of competence
• 1 Very Weak Demonstrates lack of competence
• 0 Unacceptable Contain nothing that earns credit

64
Speaking (Conversation) Performance
Characteristics Task Completion

• 6 Thorough and appropriate response with
  elaboration and detail smoothly connected
  sentences
• 5 Thorough and appropriate response may include
  elaboration and detail connected sentences
• 4 Appropriate response loosely connected
  sentences
• 3 Basic but appropriate response disconnected
  sentences
• 2 Appropriate but incomplete response
  fragmented sentences
• 1 Marginal or minimal response disjointed
  sentences or isolated words
• 0 Clearly does not respond to the prompt not in
  Chinese blank

65
(No Transcript)
66
(No Transcript)
67
Nearly three-quarters of all AP Students take the
AP Exam, although test taking is lower among AP
German and AP Japanese students
Most recently, what percent of your AP language
students took the corresponding AP Exam?
Average of students who took the AP Exam
AP Course Taught
Q510 Base All AP Teachers (n1,624)
67
68
On average, AP Teachers report that 62 of AP
Language students who took the AP Exam received a
score of 3 or higher
Of the students who took the corresponding AP
Exam, what percent received a score of 3 or
higher?
Average of students scoring 3 or higher on AP
Exam
Higher among Urban/City districts (68) Lower
among Smaller/Rural districts (53) Higher among
AP Teachers with 11 or more years of teaching
experience (70)
AP Course Taught
Q520 Base AP Teachers with students who took the
AP Exam (n1,547)
68
69
Standard Group of Test Takers

• Standard Group refers to students who
  generally receive most of their foreign language
  training in U.S. schools. They did not indicate
  on their answer sheet that they regularly speak
  or hear the foreign language of the examination,
  or that they have lived for one month or more in
  a country where the language is spoken.

70
AP Scores Total Group
71
AP Scores Standard Group
72
Student Performance Data Total Group
73
Student Performance Data Standard Group
74
AP Spanish and AP Chinese are most likely to have
native speakers enrolled
What percentage of the students in your AP
Foreign Language course are heritage learners
(native speakers of that language)?
Average of native speaking students
AP Course Taught
Q205 Base All AP Teachers (n1,624)
74
75
(No Transcript)
76
(No Transcript)
77
(No Transcript)
78
College Credit-Granting and Placement Policies
(College Board 2007 Study)

• Two-thirds of colleges interviewed give students
  credit if they score a 3 on the AP Exam this
  jumps to 90 for scores of 4 or 5.
• Three-quarters of College Faculty report that
  students who score a 5 or higher on the AP
  language exam will be placed in a third semester
  or higher course.
• 18 of respondents would place a student scoring
  a 4 on the AP Exam into the fifth semester of
  language study or higher.
• 36 of respondents would place a student scoring
  a 5 on the AP Exam into the fifth semester of
  language study or higher.
• Colleges appear to be more generous with
  placement policies than with credit policies for
  scores of 4 or 5.
• Colleges appear to be more generous with
  placement policies than with credit policies for
  scores of 4 or 5.

79
Dates and Fees

• Wednesday, May 7, 2009 at 12 noon
• Late testing Friday, May 23, 2009 at 12 noon
• Fees 84 (College Board will provide a fee
  reduction of 22 for students with accute
  financial need)

80
Questions?