Assessing the academic writing skills of

1
Assessing the academic writing skills of home
students the next big thing in EAP?Jonathan
Rees and Dorota Pacek English for International
Students UnitUniversity of Birmingham
2
Evidence of a deficit.It would be funny if it
wasnt serious

• This essay will look at whether development can
  be predicted or unpredicted.
• (student essay register, style, incoherence)
• The menopause is similar to puberty only
  backwards.
• (student report register, incoherence)
• Mr X became upset and slightly agitated and asked
  how long he would be at the clinic as he had a
  dog. The staff nurse, student doctor and
  associate psychiatrist left the meeting the
  social worker decided it was appropriate to
  discuss the situation with Mr Xs dog.
• (student observation report lack of logical
  sequencing)

3
Evidence of a deficit.Not even funny..

• This structure still plays many of important
  roles in biological proteins
• (student essay grammatical errors, confusing
  ellipsis)
• The most important interaction within proteins
  hydrophobic interactions do a lot in causing the
  protein to fold into the correct conformation
• (student descriptive assignment lexical issues,
  punctuation problems and academic style.

4
Overview of presentation

• Investigations to date of deficits
• Three exacerbating factors
• University of Birmingham Response
• Background to the test development
• (The Biosciences Project)
• V. Test construction
• VI. Findings
• VII. Conclusion

5
Acknowledged but poorly researched problem

• There has been a longstanding recognition of
  deficits in academic literacy amongst students
  in HEIs in the UK (Jones, 2004 and Lillis and
  Scott, 2007)
• Most of the evidence is anecdotal
• Only one meaningful study the MASUS study at
  the Open University (Erling and Richardson 2010)

6
MASUS STUDY(Measuring the Academic Skills of
University Students)

• MASUS procedure developed at the University of
  Sydney to identify students in need of academic
  writing development. Procedure employs 4 point
  rating scales for 5 assessment criteria
• 1. Use of source materials
• 2. Structure and development of texts
• 3. Control of academic writing style
• 4. Grammatical correctness
• 5. Qualities of presentation

7
MASUS STUDY(Measuring the Academic Skills of
University Students)

• The procedure was employed by Erling and
  Richardson (2010) with sample of 78 students from
  the Open University.
• Agreed low ratings on each of the five criteria
  ranged between 17 and 47 of the students in the
  sample.
• According to the study, this indicated that
  a
• considerable number of students in the
  sample were
• at risk of underperformance due to deficits
  in
• academic writing skills.

8
Worsening situation?

• Outdated student categorisation discourse
• Diversity of home student intake and
  consequences of increasing student mobility.
• Diversity of teaching staff background

9
Home student?
A. The archetypal home student (sheltered UK
background, knowledge of only UK education
system, very limited travel experience,
monolingual) B. The mixed exposure home
student (UK and other nationality parentage,
lived and studied in two countries, bilingual,
early years education in the UK, returned to the
UK to finish secondary education and to enter
HE) C. The home student without borders
(mixed non UK parentage, born outside the UK,
lived and studied in different countries,
multilingual, came to the UK to complete
secondary education before university study
10
Diversity of teaching staff background

• At the UoB, 31 of academic teaching staff are
  from outside the UK (average across Colleges).
• Certain Schools/Departments have a higher
  concentration of international teaching staff
  e.g. Business School - 43 (68/159)
• Many of these staff studied up to 1st degree
  level in their home country

11
The University of Birmingham Response

• Piecemeal support from academic tutors
• Recent attempt to draw together all those
  involved in AW support (Graduate School)
• Development of online resources (e.g.
  Studyskills4campus, Guide for effective learning)
• Centralised Academic Skills Centre
• Experimental projects (College of Arts and Law)
  for tutorial support

12
Opportunity knocks the Biosciences Project

• Small survey (2010) of key staff involved with
  undergraduate teaching in 3 Schools at the
  University of Birmingham
• Grammar and sentence structure
• Text organisation lack of logical order
• Style too colloquial, or inconsistent
• Lack of proofreading correcting and refining
• Not understanding the purpose of referencing,
  literature
• reviews, introductions
• Not following instructions, guidance and feedback

13
The Biosciences Project Aims

• Develop an additional mechanism to identify home
  students with the most significant problems in
  academic writing (particularly essay writing)
• Provide targeted support in a cost/time effective
  way at an early stage of the students
  university career.

14
Mission Impossible for EISU

• to develop a version of the test within 1 month
• to pre-test (particularly to assess face
  validity)
• to deliver a test with the following
  specifications
• no more than 40 minutes in length
• assessing skills related to essay writing
• easy to score/allowing quick turnaround of
  results

15
Content considerations

• Five ability areas to be assessed
• ability to employ academic grammar (e.g. verb
  complementation)
• ability to employ academic lexis (e.g. academic
  collocations)
• ability to employ an academic register (e.g
  avoidance of contractions,
• ability to sequence logically and structure
  discourse
• (e.g. cohesive devices, discourse markers)
• v. ability to comprehend and respond to
  instructions

16
Speed Development Process

• recycle
• use authentic materials
• go with intuition
• pre-test with small sample of Biosciences
  postgraduates (face validity issue)

17

• RESULTS/FINDINGS

18
PRE-TEST QUESTIONNAIRE ITEMS (Likert Scale)YEAR
1 AND YEAR 2 OF PROJECT
MEAN 2011 2012 MEDIAN 2011 2012
ITEM 1 PG 2.45 UG 2.50 2.52 PG 2.0 UG 2.0 2.0
ITEM 2 PG 2.27 UG 2.61 2.77 PG 2.0 UG 2.0 3.0
ITEM 3 PG 3.64 UG 3.53 3.58 PG 4.0 UG 4.0 4.0
ITEM4 PG 2.91 UG 2.89 3.01 PG 3.0 UG 3.0 3.0
19
MAIN MESSAGES FROM THE QUESTIONNAIRE DATA

• Both cohorts are over-optimistic about their
  academic writing skills (item 1 and 3)
• Both cohorts are neutral about whether they have
  developed their AW skills over the last year
  (item 2)
• Both cohorts are neutral about the need for
  further AW training (item 4)

20
TEST SCORES YEAR 1 AND YEAR 2 OF PROJECT
2O11 (n 238) 2012 (n 226)
MEAN SCORE 30.87 33.54
LOW 10 13
HIGH 41 45
21
Score distribution for 2011 cohort(Biosciences
undergraduates, n 231)
22
MAIN MESSAGES FROM THE TEST SCORE DATA

• Apparently consistent measurement of skills
• However, highly significant difference in
  mean
• scores (p lt .001)
• Negatively skewed distribution long tail in
  performance
• (2011 cohort, bottom 20 score lt 28)
• (2012 cohort, bottom 20 score lt 30)
• Mean overall performance for both cohorts is much
  better than that of a small sample (n 37) of
  MBA international students (M 23.4)

23
Conclusion

• Further evidence (quasi-experimental) of
  deficit in AW skills for a substantial number of
  home undergraduates
• Any assessment only useful if part of a change in
  HEI culture (resources/timetabling)