Integrating ICALL into synchronous CMC

1
Integrating ICALL into synchronous CMC

• Markus Dickinson, Rebecca Sachs, Yunkyoung Kang,
  Soojeong Eom, Chong Min Lee
• CALICO/IALLT Conference
• March 20, 2008
• San Francisco, CA

2
Intelligent CALL (ICALL)

• Intelligent CALL, using natural language
  processing technology, provides many promising
  means of facilitating L2 development
• Detailed information about learners L2
  production errors can foster awareness of
  language and encourage cognitive comparisons
• Feedback can be precisely tailored to learners
  proficiency levels, common mistakes, activity
  goals, learning styles, cognitive abilities, etc.
• Specific improvements can be tracked across
  exercises through the use of learner models
• (Amaral Meurers, 2006 Heift Schulze, 2007
  Nagata, 2002)

3
Limitations of ICALL

• However, there is a tension between
• Contextualized language use is increasingly
  emphasized in ICALL, and with increasing success
• (e.g., Amaral et al., 2006 Amaral Meurers,
  2006 Nagata, 2002)
• But actual communicative interaction remains
  relatively unexplored in systems capable of
  providing feedback
• (though see Petersen, 2006)
• In order to manage computational complexity
• Exercises are often restricted to the sentence
  level
• Activities often do not simulate true
  communication

The ability of an ICALL system to provide
meaningful, accurate feedback
The flexibility an ICALL system allows for in
terms of meaningful, communicative interaction
4
Synchronous CMC

• Synchronous CMC (computer-mediated communication)
  between L2 learners can also provide beneficial
  contexts for language development
• Activities can approximate target tasks
  relevant to real-life communicative situations
• Learners can function as language users as
  opposed to simply displaying language or
  manipulating L2 forms systematically without
  attending to meaning (Ellis, 2003)
• Possibly optimal psycholinguistic environment
  for several interactional features linked to L2
  development (Doughty Long, 2003)

5
Pros of CMC

• Potentially beneficial features
• Negotiation of lexical meaning (Blake, 2000)
• Attention to form, monitoring of linguistic
  output, self-correction (Salaberry, 2000)
• Incorporation of others feedback (Pellettieri,
  2000)
• Use of more complex language (Warschauer, 1996)
• Drawing of form-meaning connections (Doughty
  Long, 2003)
• Less pressing time constraints and less ephemeral
  language in the written modality, reducing
  processing demands (Payne Whitney, 2002)
• Comfortable environment for expression of ideas
  (Warschauer, 1996)

6
Limitations of CMC

• Concerns regarding the quality of
  learner-to-learner interactions The blind
  leading the blind?
• Without feedback from a trusted authority,
  learners might
• reinforce each others errors
• not have the resources necessary for correcting
  each other
• naturally tend to focus on vocab without
  attending to grammar
• (Blake, 2000 Kern, 1995)
• ? Teachers may need to set explicit expectations
  for grammatically correct language, while
  simultaneously trying to balance this with a
  primary focus on meaningful communication (Lee,
  2001 Pellettieri, 2000)

7
Combining the benefits

• Can ICALL and synchronous CMC be integrated in a
  way that exploits the unique benefits of each
  while avoiding their limitations?

detailed, informative, individualized feedback
communicative purpose
negotiation and interactional modifications
attention to form and meaning
approximation of real-world target tasks with
less pressing time constraints
development of functional L2 proficiency with
grammatical accuracy
Intelligent computer-generated feedback in
synchronous, task-based, computer-mediated
learner-learner interactions
8
A balancing act

• We think so, but ICALL has certain limitations
• We want learners interactions to be as free as
  possible, promoting authentic and meaningful
  communication
• But must constrain the communicative situation
  somehow in order to make processing manageable
• (cf. discussion in Amaral et al., 2006)
• We want to allow learners to make errors and then
  provide them with detailed, informative
  computer-generated feedback on nontargetlike
  language use
• But must reduce the amount of variability in
  learner input so that accurate feedback can be
  given

9
Proposed study

• Questions
• What design features will allow an ICALL system
  to provide detailed, accurate, individualized
  metalinguistic feedback on L2 errors in
  meaningful learner-to-learner CMC?
• How can we make the system as user-friendly as
  possible?
• How effective will this sort of ICALL system be
  in terms of promoting L2 development?

10
We would argue

• Feedback on grammar can be provided in
  synchronous CMC without sacrificing an undue
  amount of communicative authenticity, as long as
  we can design natural ways of
• Controlling the activity specification
• Controlling the range of learner input
• Participants English-speaking university
  students in first-year Korean classes
• Target of feedback Korean postpositional
  particles

11
Activity specification

• Dyadic picture-based spot-the-differences task
• Each participant will see one version of a house
  and have to exchange information in the L2 in
  order to find similarities and differences
  between the two pictures
• A game record will provide additional guidance
  and potentially increase motivation through
  including an element of competition
• Each participant must
• Record the activities and locations of all
  characters in his/her partners house using a
  provided chart
• Indicate whether each of these represents a
  similarity or difference
• Following the activity, each dyad will be able to
  compare their score to the average scores of
  other dyads

12
Picture 1
13
Picture 2
14
Activity specification (cont.)

• Guided and goal-oriented
• Constrains the vocabulary and domain, thereby
  reducing many of the complexities involved in
  generating feedback (computationally speaking)
• Interactionally authentic
• Perhaps not so authentic in terms of real-world
  relevance, but the sort of task often used in
  interaction research to target specific areas of
  language and promote negotiation and L2 learning

15
Target of feedback Korean particles

• Korean has relatively free word order, so
  postpositional particles are used to indicate
  grammatical functions, thematic roles (e.g., who
  is doing what to whom), and the locations of
  people and objects.
• Subject Verb Object
  Location Time

I
ate
pizza
last night
at a restaurant
?
?
??
?
?
?? ?
???
??
???
Subject
Time
Location
Object
Verb
16
Target of feedback Korean particles

• Particles must be used even in simple sentences
  thus, they are taught from the beginning of L2
  Korean study
• However, the system is quite complex and
  difficult to master for adult learners of Korean
• Korean particles make distinctions not made in
  English
• Some verbs have different argument structures
  across the two languages
• Several particles are ambiguous
• ? Particle errors account for a substantial
  proportion of the mistakes made by beginning
  learners (Ko et al., 2004), and errors persist
  even at advanced levels

17
Sources of difficulty for native speakers of
English

• No one-to-one correspondence between Korean
  locative particles and English prepositions

18
Sources of difficulty for native speakers of
English

• Korean has a topic marker, best translated as as
  for or speaking of X, which English does not
  have.
• The distinction between subject and object
  markers can be confusing because some English
  transitive verbs are used as intransitive verbs
  in Korean.

I (subject)
need
a book (object)
?? I (topic)

• ????
• need

?? book (subject)
iss-ta (have) ? exist pilyoha-ta (need)
? be needed choh-ta (like) ? be
liked shil-ta (do not like) ? not be liked
19
Examples of targetlike and non-targetlike
particle use

• Intransitive verbs need a subject marker
• ??-? ?-? ???? ??-? ?-? ????
• kitchen-LOC what-SUBJ is kitchen-LOC what-OBJ
  is
• What is in the kitchen?
• Transitive verbs need an object marker
• ???-? ??-? ???. ???-? ??-? ?? ??.
• father-SUBJ meat-OBJ grill father-SUBJ
  meat-SUBJ is grilling
• Father is grilling meat.
• A static location must be marked with a static
  locative marker
• ???-? ??-? ???. ???-? ??-?? ??.
• cat-SUBJ living room-LOC is cat-SUBJ living
  room-LOC is
• A cat is in the living room.

(TL examples are on the left with correct
particles non-TL are on the right with
asterisks)
20
Other expected error types

• Missing particles
• ? ø ?? ? ø ??? ø ?? ø ???.
• I last night restaurant meat ate
• Incorrect particles (morphology)
• subject - i/ka sister (dongsang) i/ ka
• object - eul/rul rice (bab) eul/rul
• topic - un/nun elephant (kokkiri) un/nun
• comitative - wa/kwa rice (bab) wa/kwa meat
  (gogi)
• Incorrect particle order
• ????? (??) ?? ????
• restaurant-Loc-Top Who-Sub work-Ques
• As for a restaurant, who works (there)?

21
Can beginning learners use CMC?

• We can expect problems with particles regardless
  of the communicative situation
• Unconstrained tasks might be stressful or
  frustrating for beginning learners, making it
  important to sequence, guide, and scaffold tasks
  appropriately (cf. Doughty Long, 2003)
• AND Beginning learners of Korean do not yet know
  how to type in Korean
• Since we wish to provide communicative practice
  with particles, we need to ensure that the focus
  of the task does not become that of simply
  inputting Korean

22
A possible solution

• Word and particle banks
• Learners can select the tokens they wish to use
  simply by clicking on words and particles
• For some morphophonological alternations, the
  system will transform some adjacent characters
  where necessary
• Misspellings will be less of an issue
• Learners will be given positive evidence of
  particle attachment in Korean

23
The interface

• Spot-the-differences picture
• Word and particle banks
• Sentence drafting area
• Check and Send buttons
• Feedback-providing avatar
• Chat window
• Game record

24
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ???? ???? ParticipantB

S
S
D
D
??? ??? ?? ???.
S
S
D
D
S
D
S
D
S
D
S
D
CHECK
SEND
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
?
??
???
?
?
??
?
??
??
?

????
?
????
??
25
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ??? ???? ????
ParticipantB
TASK PICTURE Partners have slightly different
versions and must communicate to find
differences. They can scroll over the picture to
enlarge it.
S
D
S
D
??? ??? ?? ???.
S
S
D
D
S
D
S
D
CHECK
SEND
S
D
S
D
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
?
??
???
?
?
??
?
??
??
?

????
?
????
??
26
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ???? ???? ParticipantB

S
S
D
D
??? ??? ?? ???.
S
S
D
D
S
D
S
D
CHECK
SEND
S
D
S
D
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
WORD PARTICLE BANKS To create a sentence,
participants click on words and particles
?
??
???
?
?
??
?
??
??
?

????
?
????
??
27
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ???? ???? ParticipantB

which then appear in the sentence drafting area.
S
S
D
D
S
D
??? ??? ?? ???.
S
S
D
D
S
D
S
D
S
D
S
D
S
D
S
D
CHECK
SEND
S
D
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
?
??
???
?
?
??
?
??
??
?

????
?
????
??
28
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ???? ???? ParticipantB

If they want help with Korean particle usage,
they can request feedback on their sentences
before entering them into the conversation.
S
D
S
D
??? ??? ?? ???.
S
S
D
D
S
D
S
D
CHECK
SEND
S
D
S
D
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
?
??
???
?
?
??
?
??
??
?

????
?
????
??
29
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ???? ???? ParticipantB

S
D
S
D
??? ??? ?? ???.
S
S
D
D
S
D
S
D
CHECK
SEND
S
D
S
D
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
FEEDBACK AREA Here, participants receive
metalinguistic feedback with advice on particle
usage.
?
??
???
?
?
??
?
??
??
?

????
?
????
??
30
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ???? ???? ParticipantB

S
D
S
D
When they are ready, they click SEND to enter
their utterance into the conversation.
??? ??? ?? ???.
S
S
D
D
S
D
S
D
CHECK
SEND
S
D
S
D
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
?
??
???
?
?
??
?
??
??
?

????
?
????
??
31
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ???? ???? ParticipantB

S
D
S
D
CHAT WINDOW They can scroll up and down to
review the conversation so far.
??? ??? ?? ???.
S
S
D
D
S
D
S
D
CHECK
SEND
S
D
S
D
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
?
??
???
?
?
??
?
??
??
?

????
?
????
??
32
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ???? ???? ParticipantB

GAME RECORD When participants find similarities
or differences, they drag the relevant words for
locations and activities here to record
information about their partners pictures, then
click on S or D to indicate whether the
pictures match in those respects or not.
S
D
S
D
??? ??? ?? ???.
S
S
D
D
S
D
S
D
CHECK
SEND
S
D
S
D
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
?
??
???
?
?
??
?
??
??
?

????
?
????
??
33
Is processing feasible?

• Learners sentence construction is guided by
• The nature of the picture-based task
• Instructions and the game record
• Word and particle banks, which
• Limit the types of argument structure by limiting
  the verbs used
• May be necessary for beginning learners who cant
  type in Korean
• May serve as a scaffold for using receptive vocab
  in conversation
• Intensive feedback is provided on one particular
  error type

34
Upshot

• Synergy between pedagogical and computational
  constraints
• Beginning learners will feel comfortable
  communicating meaningfully in the L2 (with
  familiar content and sufficient guidance)
• The learners can still make mistakes in the L2
  while attempting to express themselves
• ICALL processing can focus just on detecting
  particle errors in a known domain

35
How can we detect ill-formed sentences?

• A combination of techniques will ultimately be
  used to feed into an error diagnosis module
• Linguistic processing will be kept separate from
  error detection/diagnosis and feedback generation
• Since general relations between elements of the
  task pictures are fixed, fairly traditional
  anticipation-based pattern matching (i.e.,
  regular expressions) could be used
• This will need to be augmented with basic
  linguistic abstraction (part-of-speech tags and
  syntactic chunks)
• Partial parsing methods are extremely robust
  provide information even when a full syntactic
  parse is not possible
• Linguistic abstraction ensures applicability to
  new exercises

36
Opportunity to experiment with different
techniques

• Particle errors will often result from a mismatch
  between the argument relations of the sentence
  and the morphological forms used by the learners
• Could use multiple parsing models to check for
  mismatches (cf. Metcalf Boyd, 2006)
• One parser captures particle usage patterns from
  real language
• Another parser captures general argument
  structure patterns between words, irrespective of
  particles
• Currently exploring this other techniques
  (Dickinson Lee, 2008)
• Regardless of the techniques, generating learner
  data will provide evaluation material to help
  advance the state-of-the-art in processing Korean
  learner input

37
Is focused feedback beneficial?

• Some have argued that intensive feedback on one
  pre-selected error type may be more effective in
  certain contexts than wide-ranging incidental
  feedback on a variety of errors
• (e.g., Lyster, 1998 Nicholas, Lightbown,
  Spada, 2001)
• In our study, we will inform learners that they
  will be receiving feedback only on particles
• Important for meaning (i.e., communicating who is
  doing what to whom) in Korean
• Will hopefully prevent them from mistaking
  non-feedback for correctness
• Leaves open the possibility of providing other
  feedback, if needed

38
Is meaningful communication promoted?

• Does this set-up truly represent synchronous
  CMC as it is commonly conceptualized?
• ?How much will the learners focus on meaningful
  communication if it is clear that the feedback is
  focusing exclusively on Korean particles?
• Particles are crucial to expressing and
  understanding meaning in Korean sentences thus,
  the ostensibly grammar-oriented feedback should
  facilitate communication
• ?How can the word and particle banks be made
  sufficiently rich for the participants
  communicative purposes?

39
The importance of piloting

• Picture-based tasks have been used successfully
  in other experimental CMC research as a means of
  guiding content and controlling amount/type of
  feedback (e.g., Sachs Suh, 2007), but learners
  were already proficient typists in the L2.
• Will beginning learners be capable of interacting
  smoothly in the current context?
• What sorts of scaffolding will they actually
  need?
• What can we do to make the banks as easy to use
  (and as facilitative of L2 development) as
  possible?

40
Future directions

• Pilot the tasks and competitive game component
  with L2 learners
• Get a clearer sense of what to expect in learner
  input
• Test how the word and particle banks are actually
  used
• Develop the system in modular fashion, ensuring
  it will be extendible to other Korean language
  activities
• Activity model (indicating expected constructions
  and words for the word bank)
• Expert model (for linguistic analysis)
• Error diagnosis module
• Feedback module
• (cf. TAGARELA Amaral Meurers, 2006)

41
Future directions (cont.)

• Develop activities to target more areas of
  language
• Make the tasks more complex, meaningful, and
  relevant to real-life communicative situations
• Use this set-up to test questions of SLA theory
  and language pedagogy
• Assess L2 development under different feedback
  conditions (e.g., metalinguistic info vs.
  recasts) with pre-test/post-test experimental
  designs
• Investigate optimization of feedback for
  different areas of language, proficiency levels,
  aptitude profiles, etc.
• Integrate this system with the Korean language
  curriculum at Georgetown

42
Questions?Comments?

• Please email

43
References

• Amaral, L., Meurers, D. (2006). Where does
  ICALL fit into foreign language teaching. CALICO
  2006. University of Hawaii.
• Amaral, L., Metcalf, V., Meurers, D. (2006).
  Language awareness through re-use of NLP
  technology. Pre-conference Workshop on NLP in
  CALL Computational and Linguistic Challenges.
  CALICO 2006. University of Hawaii.
• Blake, R. (2000). Computer-mediated
  communication A window on L2 Spanish
  interlanguage. Language Learning Technology,
  4(1), 120-136.
• Chapelle, C. (2003). English language learning
  and technology. Philadelphia John Benjamins.
• Dickinson, M. Lee, C.M. (2008). Korean Particle
  Error Detection via Probabilistic Parsing.
  Workshop on Automatic Analysis of Learner
  Language at CALICO-08. San Francisso.
• Doughty, C.J., Long, M.H. (2003). Optimal
  psycholinguistic environments for distance
  foreign language learning. Language Learning
  Technology, 7(3), 50-80.
• Ellis, R. (2003). Task-based language learning
  and teaching. Oxford Oxford University Press.
• Heift, T., Schulze, M. (2007). Errors and
  intelligence in computer-assisted language
  learning Parsers and pedagogues. Routledge.

44
References

• Kern, R.G. (1995). Restructuring classroom
  interaction with networked computers Effects on
  quantity and characteristics of language
  production. The Modern Language Journal, 79,
  457-473.
• Ko, S., Kim, M., Kim, J., Seo, S., Chung, H.,
  Han, S. (2004). An analysis of Korean learner
  corpora and errors. Hanguk Publishing Co.
• Lee, L. (2001). Online interaction Negotiation
  of meaning and strategies used among learners of
  Spanish. ReCALL Journal, 13(2), 232-244.
• Lyster, R. (1998). Form in immersion classroom
  discourse In or out of focus? Canadian Journal
  of Applied Linguistics, 1, 53-82.
• Mackey, A., Goo, J. (2007). Interaction
  research in SLA A meta-analysis and research
  synthesis. In A. Mackey (Ed.), Conversational
  interaction in second language acquisition A
  series of empirical studies (pp. 407-452).
  Oxford Oxford University Press.
• Metcalf, V., Boyd, A. (2006, December).
  Head-lexicalized PCFGs for verb subcategorization
  error diagnosis in ICALL. Workshop on Interfaces
  of Intelligent Computer-Assisted Language
  Learning. The Ohio State University.
• Nagata, N. (1995). An effective application of
  natural language processing in second language
  instruction. CALICO Journal, 13(1), 47-67.

45
References

• Nicholas, H., Lightbown, P.M., Spada, N.
  (2001). Recasts as feedback to language learners.
  Language Learning, 51, 719-758.
• Payne, J.S., Whitney, P.J. (2002). Developing
  L2 oral proficiency through synchronous CMC
  Output, working memory, and interlanguage
  development. CALICO Journal, 20(1), 7-32.
• Pellettieri, J. (2000). Negotiation in
  cyberspace The role of chatting in the
  development of grammatical competence in the
  virtual foreign language classroom. In M.
  Warschauer R. Kern (eds.), Network-based
  language teaching Concepts and practice (pp.
  59-86). Cambridge CUP.
• Petersen, K. (2006, December). Measuring L2
  development in an ICALL context. Workshop on
  Interfaces of Intelligent Computer-Assisted
  Language Learning. The Ohio State University.
• Russell, J., Spada, N. (2006). The
  effectiveness of corrective feedback for the
  acquisition of L2 grammar. In J.M. Norris L.
  Ortega (Eds.), Synthesizing research on language
  learning and teaching. Amsterdam John Benjamins.
• Sachs, R., Suh, B-R. (2007). Textually enhanced
  recasts, learner awareness, and L2 outcomes in
  synchronous computer-mediated interaction. In A.
  Mackey (Ed.), Conversational interaction in
  second language acquisition A series of
  empirical studies (pp. 197-227). Oxford Oxford
  University Press.
• Warschauer, M. (1996). Comparing face-to-face and
  electronic discussion in the second language
  classroom. CALICO Journal, 13(2), 7-26.