Assessing Confidence in the Chinese Learner

1
Assessing Confidence in the Chinese Learner

• Stephen Bruce
• Napier University, Edinburgh

2
Assessing Confidence in the Chinese Learner

• Chinese students at Napier University
• Rationale for confidence-based assessment
• Student responses to the test
• Academic self-concept, confidence and achievement
  research

3
English Foundation Programme

• Set up in 2001 to prepare Chinese students for
  entry to Napier University, both in terms of
  linguistic and study skills.
• The needs of these students were complex, the
  learning curve for them was extremely steep,
  and that the demands they faced on a
  linguistically complex and culturally unfamiliar
  one-year Masters would be considerable, even
  given the requisite IELTS pass.
• Proportion of Chinese students on Napier Business
  Masters programmes has increased in recent years,
  which has resulted in staff and the institution
  facing a very different cohort with different
  strengths, and also needs, to their predecessors.
  
• Presents a major challenge to lecturers as they
  struggle to adapt their teaching approaches to
  suit both the increased number of overseas
  students and also the home students on their
  modules.
• Dr Lesley Gourlay, Napier University (2004).
  Crossing Boundaries A Case Study Masters Level
  Chinese Students, LTSN (In Press).

4
Linguistic and Cultural Issues

• Challenges for Students
• LANGUAGE
• Extreme difficulties in understanding / taking
  notes in lectures
• Difficulties with tutorial participation due to
  lack of confidence in English
• Some difficulties with tutorial tasks due to slow
  reading speeds
• Fear of failing exams due to difficulties in
  writing English in a time limit
• EDUCATIONAL CULTURE
• Feeling lost in semester 1
• Some students unclear about expectations in
  UK-style coursework reading
• Experience extremely stressful for some
• Lesley Gourlay, Napier University
  (l.gourlay_at_napier.ac.uk)

5
Linguistic and Cultural Issues

• Challenges for Staff
• LANGUAGE
• Unsure if students are understanding lectures
• Required to spend extra time explaining lecture
  content at the end
• Some difficulties with tutorial participation and
  integration
• EDUCATIONAL CULTURE
• Some staff faced with numerous examples of
  plagiarism in coursework
• Some difficulties with tasks involving critical
  appraisal
• Some students struggled with application of
  theory to practice
• Lesley Gourlay, Napier University
  (l.gourlay_at_napier.ac.uk)

6
Potential responses by the university
Lesley Gourlay, Napier University
(l.gourlay_at_napier.ac.uk)
7
Confidence and achievement

• Chinese learners show a preference for
  rote-learning and memorisation. However their
  responses to learning strategy questionnaires
  indicate preference for deep learning.
• Learner self-esteem has been found to be
  positively associated with academic achievement
  (Brookover et al., 1964 Prendergast Binder,
  1975 Song Hattie, 1984)
• Comparative studies support a view that the
  self-esteem of young Chinese students is lower
  than UK and American (Chan, 2000).
• self-effacing and modest values in Chinese
  culture, strongly influenced by the Confucian
  tradition of a humble character
• traditional authoritarian style of education or
  the highly competitive pressures created by
  schools, families and society
• gap in living standards in the UK and Chinese
  cultures
• Recent advances in academic self-concept and
  achievement research

8
Assessing Confidence in the Chinese Learner

• This short pilot study is interested in two
  questions
• Would the Chinese students responses to a
  confidence-based test differ from UK students ?
• A good cultural test of this format
• Would their use of confidence levels in the test
  relate to their academic self-concept, or
  attitudes to academic life ?
• Improving academic self-concept is often posited
  as mediating other desirable attributes

9
Rationale for confidence-based scoring

• A students ability in answering may fall into a
  number of categories
• I know it
• Im not quite sure, but I think I know it
• Perhaps I can identify the answer by a deductive
  process on the distracters
• If I guess Ive a 25 chance of being correct
  (for 4 answer choices)
• I really havent a clue
• and perhaps worst of all
• I really know it what do you mean Ive got it
  wrong !! Davies (2002)

• The standard MCQ cannot distinguish between the
  above
• possible to pass exams with knowledge only half
  learned or poorly understood
• the student certain of their knowledge should be
  rewarded
• the student should not be rewarded for guesswork
• encourage students to reflect on the reliability
  of their answers

• Would the student use this knowledge to make a
  decision or perform an action (usable knowledge)
  ? Hassmen and Hunt (1994)

10
Confidence-based MCQ

• Select an answer in the usual way for a multiple
  choice question

• Indicate your confidence that you are correct.

C1 (low) C2 (mid) C3 (high) Gardner-Medwin
and Gahan (2003)
11
Scoring confidence-based MCQ
What is my score if my answer is or ?
C 2 (Mid)
C 3 (High)
C 1 (Low)
Confidence level
2
1
3
- 4
0
-1

• Confidence level decision is governed by 2
  judgements
• estimated probability that the chosen answer will
  be correct
• the impact of the reward / benefit for a right /
  wrong answer
• Students rarely discuss their decisions in terms
  of explicit probabilities

Gardner-Medwin and Gahan (2003)
12
Encouraging good confidence judgement
Students are encouraged to reflect on the
reliability of their answers. Good confidence
judgement means more marks !

• If you are sure that your answer is correct .

• select C2 (mid) or C3 (high) confidence level

get the marks your confidence deserves!

• If your are unsure (or guessing !) .

• select C1 (low) confidence level

dont lose marks through misplaced confidence!
13
Confidence-based MCQ feedback
A 100 confidence-based score is 40 answers at
C2 (mid) confidence . so that 40 correct at
C3 would be 150 Gardner-Medwin and Gahan
(2003)
A summary of confidence levels.
14
Confidence-based MCQ feedback
Is the confidence-based score higher than the
correct score?
No

• Check the instances of INCORRECT answers at C3
  (high) and C2 (mid). These can indicate areas
  of knowledge where you are misinformed. A large
  number of these perhaps indicates general
  overconfidence.
• Check your number of CORRECT answers at C1
  (low). A large number indicates that you know
  more than you are willing to admit and are
  perhaps underconfident.

15
Confidence-based MCQ feedback
Is the confidence-based score higher than the
correct score?
Yes
You have shown good confidence judgement!
You are willing to express an appropriate
level of confidence when considering your
knowledge. This is important when making
decisions and performing actions based on that
knowledge.
16
The student group

• 31 Chinese students (54 sample)
• 18 male, 13 female
• 24 postgraduate and 7 undergraduate
• Ages 18-25 (19), 26-32 (8), 33-40 (4)
• Current on English as a Foreign Language (EFL)
  Foundation Programme
• 35 weeks of 20 hours of EFL study
• Gateway and Starter routes dependant upon English
  proficiency
• Students recently completed their IELTS exam
  (International English Language Testing System).
• Entry on to a Napier Programme conditional on a
  students IELTS score
• At the time the students did not know their
  results
• Students complete a confidence-based academic
  vocabulary test (40 Qs)

17
Chinese student responses to confidence-based MCQ
test

• No indications that Chinese students (red) behave
  differently from home based students (blue)
• Questions appear quite difficult, since low marks
  are more represented.
• 930 Blue data from UCL including 40 first time
  responses.

18
Chinese student responses to confidence-based MCQ
test

• No indications that Chinese students are
  reluctant to select high confidence levels (C3)
• on average, high confidence was selected most
  often
• Higher test scorers showed better confidence
  judgement
• Lowest test-scorers tended to be over-confident
• No significant difference between genders
  observed.

19
Chinese student responses to confidence-based MCQ
test

• Instances of C3 (high) confidence increased as
  the question difficulty decreased.
• Difficult language comprehension test as some
  small grammatical differences separated answers
  and distracters.

Harder Easier
20
Correlating confidence-based score and final exam
(IELTS)

• only 7 students (of 31) improved on their
  correct score with good confidence judgement.
• 6 of these students were the highest test scorers
  in the class.
• no apparent correlation between IELTS exam and
  confidence-based test
• IELTS exam measuring larger scope of competencies

21
Student opinions of the test

• Strongly agree
• Agree
• Neutral
• Disagree
• Strongly disagree

22
Student comments
It is a real good interesting test and I can
receive different aspects of my knowledge.
However, it will cost lots of time to finish this
exam. As a result, I suggest we can do the exam
at a regular time such an once a week. It is
an interesting test and I would like to do it at
my university. I think that is obviously fair for
students. It is interesting and a little
difficult to understand the result. I think I
should get a higher confidence-based score
because I chose (c3) nine times, and the number
answered correct is 7. Maybe I have not
understood this well. I think it is quite
interesting and helpful.it is also a good way to
show me the link between confidence and academic
study.
23
Academic self-concept and confidence judgement

• Improving academic self-concept is often posited
  as mediating other desirable attributes such as
  persistence on academic tasks, motivation, and
  self-efficacy.
• Is good confidence judgement a measure and a
  positive mediator
• an improved measure than the standard correct
  score
• informed interventions for enhancing academic
  self-concept can make use of recent advances in
  theory Craven (1996)

• The locus of control refers to how people explain
  events that happen to themselves and others.
• internal locus - guided by his/her personal
  decisions and efforts.
• external locus - guided by fate, luck, or other
  external circumstances
• a more 'surface' approach to learning is
  associated with an external locus of control,
  Watkins Biggs (1996)
• Scoring scheme feedback helps to equate
  internal expectation (probability answered
  correctly) with external performance (score)
• desirable attribute that is associated with an
  internal locus of control

24
A model for self-concept

• Posited to be multifaceted and hierarchical in
  nature

• Shavelson, Hubner and Stanton (1976)
• Numerous studies now based or extended from this
  model Hattie (1992)

25
Self-concept and academic achievement

• From the literature
• Support for the multifaceted nature.
• academic self-concept and academic achievement
  strongly correlated
• Marsh, Byrne and Shavelson (1988), Hattie
  (1992)
• little correlation between social self-concepts
  and academic achievement Song Hattie (1984),
  Waugh (1999)

• Support for the hierarchical nature less clear
  cut.
• more support for a hierarchical model for
  adolescents and
• a unitary structure for younger
  children Hattie (1992)
• achievement and academic self-concept deserve
  special interpretations at the level of specific
  subjects Marsh (1990)
• standard self-description questionnaires based on
  the Shavelson model for preadolescents,
  adolescents and late adolescents Marsh (1992a,
  b, c)

26
Measuring academic self-concept

• Your perceptions of your ability and achievement
• adapted from Marsh (1992), Song Hattie (1984)
  and Waugh (2001)
• indicate your relative agreement with each of
  the 20 statements
• All the time, or nearly all the time
• Most of the time
• Some of the time
• None of the time, or almost none of the time

1st 10 general university experience
eg. I am capable of getting good marks at
university I am proud of my achievements at
university 2nd 10 experience of English Language
classes only eg. I am sure of myself in
English Language classes I am achieving at a
high level in English Language classes
Questionnaire feedback For your general
university experience, your responses total 18
points (30 maximum). For your English Language
classes experience, your responses total 19
points (30 maximum).
27
Measuring academic self-concept

• Your attitudes to academic life
• locus of control inventory adapted from Trice
  (1985)
• select True or False to indicate your agreement
  with each statement
• eg. My academic marks most often reflect the
  effort I put into classes.
• I came to university because it was expected of
  me

Questionnaire feedback
28
Correlating academic self-concept, confidence and
achievement

• perception of ability and achievement scores
  (general and EFL scales), increase in tandem.
• as locus scores become increasingly internal
  (towards 0), perception of ability and
  achievement scores increase accordingly

29
Gender observations in the questionnaire responses
Scores from questionnaire Females tended to
respond with lower scores than males for the
perception of ability and achievement (general
and EFL) scales. The responses indicating the
most internal of locus of control scores, were
male students.
30
Correlating academic self-concept, confidence and
achievement

• Students perception of ability and achievement
  in EFL shows a moderate correlation with their
  IELTS exam result.
• in line with academic self-concept and
  achievement research strongly subject based
  support for the hierarchical structure.
• No significant gender differences in the
  confidence-based test scores.
• Females attained higher scores in the IELTS exam.

31
Correlating academic self-concept, confidence and
achievement

• Perception of EFL ability achievement scores
  are moderately correlated with the number of
  correct answers at high confidence.
• These figures may indicate a trend, but may be
  noise
• one confidence-based test is insufficient to
  identify genuine correlations

32
Conclusions

• Indications that Chinese students perform in a
  similar manner to confidence-based assessment as
  their UK counterparts.
• Provide a scheduled series of confidence-based
  tests for Chinese students on the English
  Foundation programme
• Examine academic self-concept and
  confidence-based assessment
• potentially valuable role for the enhancement of
  academic self-concept, and the development of
  other positive academic behaviours
• Effective with younger learners who may enjoy a
  game perspective
• The scoring scheme and test feedback seems
  applicable to recent research that posits a
  reciprocal relationship between academic
  self-concept and academic achievement (Marsh,
  2003).

Acknowledgments

• Lecturing staff in the Centre for Business
  Languages, Napier University
• Nicola Beasley (Napier University) for initial
  software development
• Tony Gardner-Medwin (UCL) for valued discussions
  and analysis

33
Bibliography

• Brookover, W.B., Thomas, S. and Paterson A.
  (1964) Self-concept of ability and school
  achievement, Sociology of Education, 37, pp.
  271-279.
• Craven, R. (1996) Enhancing Academic
  Self-Concept A Large-Scale Longitudinal Study in
  an Educational Setting, PhD thesis, University of
  Sydney
• Davies, P. (2002) Theres no confidence in
  multiple-choice testing, Proceedings of the 6th
  International CAA conference, Loughborough, pp.
  119-130.
• Echternacht, G.J. (1972) The use of confidence
  testing in objective tests, Review of Educational
  Research, 422, pp. 217-237.
• Gardner-Medwin, A.R. and Gahan M. (2003)
  Formative and summative confidence-based
  assessment, Proceedings of the 7th International
  CAA conference, Loughborough, pp. 147-155.
• Gourlay, L. (2004) Crossing Boundaries A Case
  Study Masters Level Chinese Students, LTSN (In
  Press)
• Hassmen, P. and Hunt, D.P. (1994) Human
  self-assessment in multiple-choice testing,
  Journal of Educational Measurement, 31, pp.
  149-160.

34
Bibliography

• Hattie, J. (1992) Self-Concept, Lawrence Erlbaum
  Associates, New Jersey
• Khan, K.S., Davies, D.A. and Gupta, J.K. (2001)
  Formative self assessment using multiple
  true-false questions on the Internet feedback
  according to confidence about correct knowledge.
  Medical Teacher, 23, pp. 158-163.
• Marsh, H.W (1992a) Self-Description
  Questionnaire (SDQ)III A theoretical and
  empirical basis for the measurement of multiple
  dimensions of preadolescent self-concept A test
  manual and research monograph, MacArthur, New
  South Wales, Australia, University of Western
  Sydney, Faculty of Education.
• Marsh H.W., Byrne, B.M. and Shavelson, R.J.
  (1988) A multifaceted academic self-concept Its
  hierarchical structure and its relation to
  academic achievement. Journal of Educational
  Psychology, 80, 366-380.
• Marsh H.W. (1990b) The structure of academic
  self-concept The Marsh/Shavelson model, Journal
  of Educational Psychology, 82, 623-636.
• Marsh, H.W. (2003) A reciprocal effects model of
  the causal ordering of academic self-concept and
  achievement, Paper presented at NZARE AARE,
  Auckland, New Zealand.

35
Bibliography

• Prendergast, M.A. and Binder D.M. (1975)
  Relationships of selected self-concept and
  academic achievement measures, Measurement and
  Evaluation Guidance, 8, pp. 92-95.
• Song, I.S. and Hattie, J. (1984) Home
  environment, self-concept and academic
  achievement measures, Journal of Educational
  Psychology, 76, pp. 1269-1281.
• Watkins, D.A. and Biggs, J.B. (Eds, 1996) The
  Chinese Learner Cultural, Psychological and
  Contextual Influences, CERC ACER, (ISBN 0 86431
  182 6).
• Waugh, R.F. (2001) Measuring ideal and real
  self-concept on the same scale, based on a
  multifaceted, hierarchical model of self-concept,
  Educational and Psychological Measurement, 611,
  pp. 85-101.