Variations in higher education portfolio assessment Discussion of quality issues based on a survey i

1
Variations in higher education portfolio
assessmentDiscussion of quality issues based on
a survey in Norway across institutions and
disciplines

• Olga Dysthe, University of Bergen
• Knut Steinar Engelsen Anne Kristin Sjo,
  Stord/Haugesund U. College
• Marit Allern,University of Tromsø
• EARLI - 2007

2
Aim of this presentation

• Present findings from a Norwegian nationwide
  survey of portfolio use in higher education
• Focus on
• Teacher attitudes
• Variations in understanding and use of pf
• Disciplinary differences
• Quality issues

3
The context

• Dramatic increase in use of portfolios in
• Norwegian higher education from 2004-06
• Quality Reform of Higher Education (related to
  Bologna process)
• Structural changes
• All study programs modularized (10-15 ECT)
• Bachelor reduced from 4 y to 3 y
• Pedagogical changes (Evaluation Report 2007)
• closer follow-up of each student
• changes in assessment continuous ass
  portfolios
• more undergraduate writing

4
Methods

• Electronic survey spring 2006 based on a
  randomized selection from all public universities
  (5) and university colleges (22) in Norway
• Identification and selection of respondents
• Professors responsible for topics within a
  subject-field
• Standard statistical methods
• Interpretation of findings supplemented by survey
  interview study at Tromsø University

5
Types of disciplines institutions

• Universities (125)
• Humanities (44)
• Social science (17)
• Math and Science (54)
• Medicine (10)
• University colleges (178)
• Teacher education (58)
• Health education (37)
• Engineer (25)
• Economics (21)
• Other disciplines (37)

• Response rate
• Universities 57
• University colleges 51

6
Research questions

• How is the portfolio conceptualized and
  practiced?
• Working portfolio assessment portfolio
• Types of student work in pf (genres)
• Feedback practices
• Grading practices and use of criteria
• Are there disciplinary differences?
• Hard and soft disciplines
• Professional non professional
• What quality issues are involved?
• Reflection
• Feedback
• Teacher attitudes

7
Results from the survey (1)

• Teachers attitude to
• student learning
• teacher workload
• student workload
• cheating

8
Teachers attitude to pf as tool for learning
Students overview over subject
Students effort in general
9
Teachers attitude to pf as tool for learning
Students writing abilities
The students learn more in the subject than
before
10
Teachers attitude to pf as tool for learning
Pf assessment demands too much work of me in
relation to students learning benefit
Pf assessment demands too much work of the
students in relation to their learning benefit
11
Teachers attitude to pf as tool for learning
Plagiarism has been a problem in relation to
pf-assm
12
Results from the survey (2)

• Discipline related differences
• Working pf and assessment pf?
• Types of student work in pf
• Feedback
• Reflective texts

13
Working portfolio and assessment portfolio

• Do you differenciate between working portfolio
  and assessment portfolio?
• Findings
• Health- and teacher education have substantial
  higher scores than the other disciplines

14

• Types of student work in pf (genres)
• Big difference between hard and soft
  disciplines

15
Use of working and assessment portfolio

• Findings
• Close relationship between the use of reflective
  texts and the use of both working and assessment
  pf
• Close relationship between diversity in genres
  and the use of both working and assessment pf

16

The use of peer feedback

• Findings
• Soft disciplines use peer-feedback to a greater
  extent than hard disciplinesSoft disciplines
  use feedback on public arenas to a greater extent
  than hard disciplines

17
Conclusions based on our studies (1)

• A common understanding of portfolios seems
  lacking
• Pf-practices are diverse
• Differences are related to type of education
  (professional/non-professional
• Indications that hard disciplines have more
  focus on control, - soft discipline on
  reflexive learning
• Portfolios seem to have positive impact on
  students effort, writing abilities and students
  learning

18
Conclusions (2)

• Problem areas
• Teacher work load
• Fear of plagiarism

19
Discussion

• How much variation in pf concept and practice is
  acceptable in order to still call it a portfolio?
• Collection Reflection Selection
• How does variation affect quality issues?
• Feedback
• Teachers attitude towards pf
• Challenges for further development of
  pf-practices in HE

20
Variation and portfolio conceptualization

• Collection Reflection Selection
• A portfolio is a purposeful collection of
  student work that exhibits the student's efforts,
  progress, or achievements in one or more areas.
  The collection must include student participation
  in selecting contents, the criteria for judging
  merit, and evidence of the student's self
  reflection (Poulson, Poulson Meyer 1991).

21
Variation and portfolio practice

• Questions raised by our findings
• Are the variations between disciplines due to
  lack of understanding of what portfolios are?
• or
• Do variations reflect that portfolios are
  different because diciplines have different
  demands to pf as useful pedagogical tools?
• Our answer
• Both

22
Discussion point

• Findings from University of Tromsø illustrate the
  problems that occur when teachers are supposed to
  implement portfolios with no theoretical
  background and little or no training. In many
  cases old practices are continued but given a new
  name
• Is a normative definition of pf desirable to
  avoid this?

23
Is variation in pf (concept and practice) a
problem for assessment quality?

• Different stakeholders views
• Students
• Big variations problematic. Reason important for
  students with predictable assessment formats?
• Teachers
• Flexible pf increase learning quality?
• Flexible pf give higher validity
• Administrators and governing bodies
• Standardization necessary to ensure reliability
  of assessment
• post Bologna
• Standardization for mobility?

24
How does variation affect quality issues?

• Feedback crucial to learning quality (Black
  Wiliam, 1998)
• Peer feedback
• More used in soft than in hard disciplines
• 41 of students got no instruction or training in
  giving feedback
• Teacher feedback (accessible on VLE)
• Other studies indicate higher quality feedback if
  public

25
Quality issues related to feedback

• Different stakeholders views
• Students
• Good feedback is a major quality issue in
  portfolios
• Teachers
• Ambivalence Increased quality but also workload
• Administrative/governing bodies
• Feedback contaminates assessment results Whose
  work is it anyway?

How do we deal with these quality dilemmas?
26
Research challenges

• Plagiarism
• Our survey indicates less concern about
  plagiarism among the teachers who give feedback
  on public (digital) arenas
• Workload
• Time consuming to combine formative assessment
  (feedback) and summative assessment
• Teachers take too much responsibility for the
  feedback processes (Allern, 2005 Bratseth, 2007)
• Reseach needed
• Does peer feedback reduce teacher workload? How?

27
Research challenges

• How can electronic portfolio systems and the
  devolpment of new tools (e.g. Web 2.0)
• address the problems of plagiarism and teacher
  workload?
• be used to meet the needs of different
  disciplines?