The OSTE: Objective Structured Technical Examination for Engineering students

1
The OSTE Objective Structured Technical
Examination for Engineering students

• Nandini Alinier, Guillaume Alinier
• University of Hertfordshire, U.K.
• ICEE 2005

2
Overview

• Background origins
• Application in an Engineering context
• Stages of the OSTE
• Development of stations
• Preparation for sessions
• Pilot
• First series
• Second series
• Analysis

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
3
Background

• Objective Structured Clinical Examination (OSCE)
  (Harden 1975)
• Originally developed in the mid-seventies in
  Dundee, Scotland
• Used in healthcare education
• Method to objectively assess the skills of
  healthcare trainees
• Used in a project funded by the British Heart
  Foundation at UH during 2001-03 (Alinier et al
  2004)
• Perceived positively by students lecturers
  (Alinier 2003)

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
4
OSCE

• Comprises 15 to 20 stations
• Station short task performed in 5 mins
• Practical exercise
• Requires an assessor
• Theoretical exercise
• Pen paper
• Gap period between stations (1 min)
• Students rotate individually through the stations
• Allows time for assessors to complete marking
  reset any equipment
• Timing of session by an electronic clock

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
5
OSCE (continued)

• Students given explicit written instructions
• Objective marking sheet for each station

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
6
Example of a station from an OSCE
N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
7
Applied to Engineering

• Project undertaken in the School of Electronic,
  Communication Electrical Engineering at the
  University of Hertfordshire, UK
• Funded by the Higher Education Academy
  Engineering Subject Centre (Learning Teaching
  Support Network (LTSN) Engineering)
• Objective Structured Technical Examination (OSTE)

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
8
Study about the OSTE

• Used for formative assessment
• Piloted with level 2 BEng students
• Conducted with level 1 BEng students (1st 2nd
  semesters)
• Each student did the OSTE twice
• Circulated feedback questionnaires for students
  assessors

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
9
Stages of the OSTE
Stage No. Stage
1 Development of OSTE stations
2 Preparation for OSTE sessions
3 Pilot of OSTE
4 First series of OSTE (in Dec 04)
5 Second series of OSTE (in Mar 05)
N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
10
Stage 1 Development of OSTE stations

• Gathering a variety of ideas about essential
  electrical engineering skills
• Seminar within School
• Call for ideas via Email
• Identified appropriate range of exercises doable
  in 5 mins
• 16 stations were short-listed
• Developed instruction sheets marking criteria
  for each station

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
11
Stage 1 Development of OSTE stations (continued)

• Types of stations
• Theoretical pen paper
• Practical assessor required
• Computer-based assessor required

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
12
(No Transcript)
13
Stage 2 Preparation for OSTE sessions

• Development of PC-based clock with an integrated
  voice prompt
• Briefing of students
• Training of staff to act as assessors
• Designing of student assessor feedback
  questionnaires

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
14
Stage 3 Pilot of OSTE

• Group of 15 volunteer 2nd year BEng students
• Resulted in changes in
• Tasks to be carried out
• Marking sheet
• Instruction sheet

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
15
(No Transcript)
16
(No Transcript)
17
Example of a computer-based station
N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
18
Stage 4 First series of OSTE

• 1st year BEng students
• During 1st semester (Dec 04)
• Formative OSTE without feedback
• To determine the level of students performance
  at the start of the course without preparing them
  too much for their 2nd OSTE
• Feedback collected from students assessors

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
19
Stage 5 Second series of OSTE

• Same group of 1st year students
• During 2nd semester (Mar 05)
• Run as mixed-mode OSTE
• (Alinier 2003)
• Collection of data about students performance
• Feedback to students by assessor
• Feedback collected from students assessors

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
20
Results students perspective

• Out of 46 student feedback questionnaires
• The OSTE was beneficial 93.5
• The OSTE helped them developed confidence in
  using equipment 73.9
• 52.2 rated the OSTE as very good or good
• (5-point Likert Scale with 1very good, mean2.4
  with 95 C.I. 1.43.4)
• The OSTE should be used as a summative assessment
  tool 28.3
• Number of times OSTE should be repeated per year
  2.9 (95 C.I. 1.4 - 4.4)

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
21
Results assessors perspective

• Out of 33 assessor feedback questionnaires
• The OSTE should be incorporated into the 1st year
  BEng curriculum 93.1
• The OSTE helped students develop their confidence
  in using equipment they were exposed to 75.8
• 90.6 rated the OSTE as very good or good
• (5-point Likert Scale with 1very good, mean1.5
  with 95 C.I. 0.8 2.2)
• The OSTE should be used as a summative assessment
  tool 75.9
• Number of times OSTE should be repeated per year
  2.2 (95 C.I. 1.4 - 3.0)

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
22
Further analysis

• Identify key weaknesses among students
• Any correlation between performance in OSTE and
  performance as per in-class assessment

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
23
OSTE results
N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
24
Conclusions

• OSTE perceived positively by students and
  assessors
• Formative assessment ?
• Summative assessment ?

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
25
Thanks for your attention

• Any questions?
• N.D.Alinier_at_herts.ac.uk
• G.Alinier_at_herts.ac.uk

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland
26
References

• Harden R M, Stevenson M, Downie W W, Wilson G M,
  "Assessment of clinical competence using an
  objective structured clinical examination",
  Medical Education, Vol 1, 1975, 447-451.
• Alinier G, "Nursing students and lecturers
  perspectives of OSCE incorporating simulation",
  Nurse Education Today, Vol 23, No 6, 2003,
  419-426.
• Alinier G, Hunt W B, Gordon R, "Determining the
  effect of realistic simulation in nurse
  education Study design and initial results",
  Nurse Education in Practice, Vol 4, No 3, 2004,
  200-207.
• Ross M, Carroll G, Knight J, Chamberlain M,
  Fothergill-Boubonnais F, Linton J, "Using the
  OSCE to measure clinical skills performance in
  nursing", Journal of Advanced Nursing, Vol 13,
  1988, 45-56.
• Hulett S, Gilder N, "The application and
  development of the Objective Structured Practical
  Examination in evaluating physiotherapy students
  performance at the University of Cape Town",
  South African Journal of Physiotherapy, Vol 42,
  No 2, 1986, 40-43.
• Marshall G, Harris P, "A study of the role of an
  objective structured clinical examination (OSCE)
  in assessing clinical competence in third year
  student radiographers", Radiography, Vol 6, 2000,
  117-122.
• Mossey P A, Newton J P, Stirrups D R, "Scope of
  the OSCE in the assessment of clinical skills in
  dentistry", British Dental Journal, Vol 190, No
  6, 2001, 323-326.
• Townsend A H, Mcllvenny S, Miller C J, Dunn E V,
  "The use of an objective structured clinical
  examination (OSCE) for formative and summative
  assessment in a general practice clinical
  attachment and its relationship to final medical
  school examination performance", Medical
  Education, Vol 35, 2001, 841-846.
• Harden R M, "Twelve tips for organising an OSCE",
  Medical Teacher, Vol 12, No 3/4, 1990, 259-264.
• Sloan D A, Donnelly M B, Schwarts R W, Strodel W
  E, The Objective Structured Clinical
  Examination The new gold standard for evaluating
  postgraduate clinical performance, Annals of
  Surgery, Vol 222, No 6, 1995, 735-742.

N Alinier G Alinier, ICEE 2005, July 25-29,
Gliwice, Poland