Assessing

1
HLST Annual Conference Assessment Feedback
Whats the Answer? 12th November 2009 Online
Discussions To Assess or not to Assess? That is
the Question
John Erskine Sport Exercise Faculty of
Health Staffordshire University j.w.erskine_at_staffs
.ac.ukAward Leader Online MSc Applied Sport
Exercise Science
2
To Assess or not to Assess? That is the
Question

• Presentation Objectives the shortened version!
• Develop a better understanding of why the
  assessment of online discussions should be a
  learning strategy adopted in most online modules
• (2) Become familiar with Discussion Rubrics as
  a means of assessing online discussions
• (3) Undertake a task using discussion rubrics to
  assess (a) an individual discussion and (b) a
  selection of discussion contributions
• in 15 minutes!!!!!!

3
To Assess or not to Assess? That is the
Question

• So. you are about to deliver a fully online 15
  credit masters module to students from all round
  the globe..
• would you summatively assess online discussions
  in your module?
• Yes No It Depends.

4
To Assess or not to Assess? That is the
Question

• What you assess on a module is what you value
• Many theories related to online learning
  emphasise the importance of active participation
  and collaboration and interaction between
  learners..
• it is the relationships and interactions among
  people through which knowledge is primarily
  generated (Palloff Pratt, 1999, pg. 15)
• Implication.. quality of interaction directly
  relates to the quality of learning obtained.
• yet this interaction is rarely assessed

5
To Assess or not to Assess? That is the
Question
Learning Theory Social Learning Theory
(Bandura 1987) Social Constructivism (Vygotsky
1978) Community of Practice (Wenger
1998) Conversational Learning (Laurillard
1993) Swan (2004) adapted from Rourke et al (
2001) Community of Enquiry
Model Cognitive Presence Interaction with
content Teaching Presence Interaction with
instructors Social Presence Interaction with
peers
6
To Assess or not to Assess? That is the
Question
Why Assess Online Discussions? The arguments
for (1) Because if you dont, students wont
participate. (2) Because if you do,
thoughtfully, you encourage active involvement
and in doing so support the learning outcomes of
the module (3) To recognise student workload
and time commitment associated with
engagement The arguments against. (1) Forced
participation is participation of the worst kind
(2) Online discussions are not able to take into
account the learning style/preferences of the
students (3) Participation for credit is an
online form of question spotting leading to
dubious measures of learning (4) Valid and
reliable assessment of discussions is
difficult
7
To Assess or not to Assess? That is the
Question

• Successful online discussion is directly
  related to it being assessed.
• (e.g. Swan, Schenker, Arnold Kuo 2007)
• Q If I wish to formally assess online
  discussions
• Where do I start and.
• (b) How do I grade the quality of discussion
  contributions?

8
To Assess or not to Assess? That is the
Question

• Where do I start?
• Edelstein Edwards (2002), suggest the following
  as an initial check off list to consider
• How much time is required to effectively
  participate in discussions? if other
  assignments required, then time management is a
  sig. issue.
• How critical are discussions to the achievement
  of learning outcomes?
• What discussion formats should be used?
• How often do you plan to interact to direct the
  discussion?
• How do you plan to grade the quality of
  participation in the discussions?
• Will students be made aware of the assessment
  criteria at the outset of the module?
• Will you intervene if students contribution is
  below standard?
• Will original postings be weighted the same as
  responses to other postings
• Will a minimum number of postings be required?
• Will you set time frames for discussion
  contributions?

9
To Assess or not to Assess? That is the
Question
(b) How do I grade the quality of discussion
contributions? Most attempts at assessing online
contributions make use of Discussion Rubrics and
many of them are available online (see the Web
Available Discussion Rubrics as part of
handout) Rubrics have been devised as a means of
assessing the content of online contributions of
students. A rubric consists of (1) a set of
categories or aspects of student work that are of
interest and (2) levels of performance within
each category often with accompanying score
10
To Assess or not to Assess? That is the
Question
Rubric Template
performance level 1 performance level 2 performance level 3
Characteristic 1
Characteristic 2 Explicit performance
Characteristic 3 behaviours specified in each block
11
To Assess or not to Assess? That is the question
Summary of Discussion Rubrics Appendices A to E Summary of Discussion Rubrics Appendices A to E
Discussion Rubric Key Elements
Appendix A Simple Example A quick and easy example to score individual discussion contributions, with weightings applied (via points) to criteria
Appendix B Edelstein Edwards (2002) Good use of identified categories. Appears to focus on overall score for a number of contributions. Minimum available score 5, maximum score 20, below 10 unacceptable
Appendix C Pelz (2004) Both points and grades included. Peltz incorporates his 2 cardinal rules for online discussions (1) comment must introduce new, relevant information and (2) insertion of a subject field that highlights the essence of your main point note the use of penalties for not doing this.
Appendix D Erskine (2008) Rubric used in M level module Skill Acquisition in Sport. Rubric used to evaluate 6 student selected contributions (3 major, 3 minor)
Appendix E Vogt Porter (2007) Well selected criteria with excellent use of language to describe the performance indicators. Weightings and accompanying scoring also included.

12
Appendix A
The following points are what is looked for in your original postings to the Discussion Board and your replied to other postings (total of 10 points for each Discussion Board Assignment) Discussion at a critical level means discussing your opinion of the point mentioned, why you hold that opinion, what you see wrong with the point mentioned, how you see the point being consistent/inconsistent with what you have learnt so far, implications for the future, consistencies/inconsistencies within the article reading itself and so forth. In other words critiquing an article means analysing the good/bad aspects of the article and justifying your analysis. The following points are what is looked for in your original postings to the Discussion Board and your replied to other postings (total of 10 points for each Discussion Board Assignment) Discussion at a critical level means discussing your opinion of the point mentioned, why you hold that opinion, what you see wrong with the point mentioned, how you see the point being consistent/inconsistent with what you have learnt so far, implications for the future, consistencies/inconsistencies within the article reading itself and so forth. In other words critiquing an article means analysing the good/bad aspects of the article and justifying your analysis. The following points are what is looked for in your original postings to the Discussion Board and your replied to other postings (total of 10 points for each Discussion Board Assignment) Discussion at a critical level means discussing your opinion of the point mentioned, why you hold that opinion, what you see wrong with the point mentioned, how you see the point being consistent/inconsistent with what you have learnt so far, implications for the future, consistencies/inconsistencies within the article reading itself and so forth. In other words critiquing an article means analysing the good/bad aspects of the article and justifying your analysis.
Original Posting Mentions at least 2 specific points form the article of reading 1 point
Relation of new information to old information learned on the course to date 1point
Relation of information in the article to personal experience 1 point
Discussion at a critical level, not just recitation of facts form the article 3 points
Length of posting approximately 1 word processing page 1 point
Reply to Others Postings Discuss one point you like/agree with and one point you dislike/disagree with and why 2 points
Length half page (approx. 100 words) 1 point
Total/10

13
Appendix B Edelstein
Edwards EDU6707 Discussion Rubric

California State University at Hayward EDUI 6707 Discussion Board Grading Rubric California State University at Hayward EDUI 6707 Discussion Board Grading Rubric California State University at Hayward EDUI 6707 Discussion Board Grading Rubric California State University at Hayward EDUI 6707 Discussion Board Grading Rubric California State University at Hayward EDUI 6707 Discussion Board Grading Rubric California State University at Hayward EDUI 6707 Discussion Board Grading Rubric
Category 1 2 3 4 Score
Promptness Initiative Does not respond to most postings rarely participates freely Responds to most postings several days after initial discussion, limited initiative Responds to most postings within a 24 hour period requires occasional prompting to post Consistently responds to postings in less than 24 hours demonstrates good self initiative
Delivery of Post Utilises poor spelling and grammar in most posts, posts appear hasty Errors in spelling and grammar evidenced in several posts Few grammatical or spelling errors Consistently uses grammatically correct posts with rare misspellings
Relevance of Post Posts topics which do not relate to the discussion content, makes short or irrelevant remarks Occasionally posts off topic, most posts are short in length and offer no further insight into topic Frequently posts topics that are related to discussion content, prompts further discussion of topic Consistently posts topics related to discussion topic, cites additional references related to topic
Expression within Post Does not express opinions or ideas clearly, no connection to topic Unclear connection to topic evidenced in minimal expression of ideas Opinions and ideas are stated clearly with occasional lack of connection to topic Expresses opinions and ideas in a clear and concise manner with obvious connection to topic
Contribution to the Learning Community Does not make effort to participate in learning community as it develops, seems indifferent Occasionally makes meaningful reflection on groups efforts, marginal effort to become involved with group Frequently attempts to direct the discussion and to present relevant viewpoints for consideration by the group, interacts frequently Aware of the community, frequently attempts to motivate the group discussion, presents creative approaches to the topic
Total Score
14
Appendix C Peltz 2004

Points Interpretation
4 Excellent (A) The comment is accurate, original, relevant, teaches us something new and well written. 4 point comments add substantial teaching presence to the module and stimulate additional thought about the issue under discussion.
3 Above Average (B) The comment lacks at least one of the above qualities, but is above average in quality. A 3 point comment makes a significant contribution to our understanding of the issue being discussed
2 Average (C The comment lacks 2 or 3 of the required qualities. Comments which are based on personal opinion or personal experience often fall within this category
1 Minimal (D) The comment presents little of no new information. However 1 point comments may provide important social presence and contribute to a collegiate atmosphere
0 Unacceptable (D) The comment adds no value to the discussion
No Penalty Excellent Subject The subject field contains the main point of the comment. The reader clearly understands the main point of the comment before reading it
-1 Minimal Subject The subject field provides key word only. The reader knows the general area the comment deals with.
-2 Subject field is unacceptable The subject field provides little or no information about the comment
15
Appendix D Erskine 2009

Specific Grading Criteria for Identified Discussion Contributions The table below represents a guide to assessing the quality of discussion contributions that you have selected for your assignment. Note it is not expected that each contribution will necessarily cover each and every criteria outlined below. However in terms of your six identified and selected contributions, the expectation is that these selected examples will allow all criteria to be examined and evaluated Specific Grading Criteria for Identified Discussion Contributions The table below represents a guide to assessing the quality of discussion contributions that you have selected for your assignment. Note it is not expected that each contribution will necessarily cover each and every criteria outlined below. However in terms of your six identified and selected contributions, the expectation is that these selected examples will allow all criteria to be examined and evaluated Specific Grading Criteria for Identified Discussion Contributions The table below represents a guide to assessing the quality of discussion contributions that you have selected for your assignment. Note it is not expected that each contribution will necessarily cover each and every criteria outlined below. However in terms of your six identified and selected contributions, the expectation is that these selected examples will allow all criteria to be examined and evaluated Specific Grading Criteria for Identified Discussion Contributions The table below represents a guide to assessing the quality of discussion contributions that you have selected for your assignment. Note it is not expected that each contribution will necessarily cover each and every criteria outlined below. However in terms of your six identified and selected contributions, the expectation is that these selected examples will allow all criteria to be examined and evaluated Specific Grading Criteria for Identified Discussion Contributions The table below represents a guide to assessing the quality of discussion contributions that you have selected for your assignment. Note it is not expected that each contribution will necessarily cover each and every criteria outlined below. However in terms of your six identified and selected contributions, the expectation is that these selected examples will allow all criteria to be examined and evaluated
Criteria Fail (- 50) Pass (50-60) Merit (60-70) Distinction (70)
Selected Contributions provide a comprehensive overview of the identified discussion topic that have currency and application Very limited evidence within the Discussions of having obtained used and understood primary/secondary source materials related to the identified topic area Some source material has been obtained, used and understood related to the topic but discussion limited to a degree in range of depth of recency A useful range of both primary and secondary resources have been obtained and incorporated into the discussion topic. Successful attempts made to link material into applied settings There is a depth and breadth to the resources utilised in support of the specific discussion topic identified, A high level of understanding of the material evident via the discussion content
Selected contributions refer to and critically appraise empirical literature related to the discussion topic, when at all possible Discussion does not engage with related literature to a significant degree. Literature is described rather than evaluated. Key literature is utilised within the discussion but mainly at a descriptive level with pockets of analysis evident Good literature base is evident within the discussion. Good levels of analysis of content offered High level of engagement with evidence related to the discussion topic being examined, with this engagement material at the forefront of current thinking
Selected contributions demonstrate the ability to apply research findings into practical sports related applications Practical examples and comments offered demonstrate a lack of understanding and appreciation of research findings related to the discussion topic area Attempts to provide practical application of the research findings are mixed Discussion provides examples of being able to apply findings into appropriate practical situations Discussion demonstrates innovative, interesting and appropriate practical examples which are used to enlighten and clarify points being made
Selected contributions provide evidence of having examined, understood and responded to comments offered by others No real evidence of having read and/or learnt form other postings Limited evidence offered of having paid attention to the discussion offerings of fellow students Useful and timely comments offered in response to discussion comments made by fellow students Comments offered consistently demonstrate attention to and consideration of discussion comments offered by other students
16
Appendix E Vogt Porter 2007

ETEC 522 D. Vogt D. Porter MET Program 2007 ETEC 522 Participation in Discussion Forums Relatively continuous online Discussion Forums are woven into the fabric of ETEC 522, and your insightful, active participation in these will be evaluated as 25 of your overall grade in the course. You will be expected to demonstrate excellent knowledge of the course material, as well as the ability to think critically and profoundly about the issues arising in discussion, course readings, and in the contributions of your peers. Discussion Forums are essentially conversations active and successful participation requires due consideration of the topic, of the contributions of others, and of the timing, quantity and quality of your own contributions. ETEC 522 Participation in Discussion Forums Relatively continuous online Discussion Forums are woven into the fabric of ETEC 522, and your insightful, active participation in these will be evaluated as 25 of your overall grade in the course. You will be expected to demonstrate excellent knowledge of the course material, as well as the ability to think critically and profoundly about the issues arising in discussion, course readings, and in the contributions of your peers. Discussion Forums are essentially conversations active and successful participation requires due consideration of the topic, of the contributions of others, and of the timing, quantity and quality of your own contributions. ETEC 522 Participation in Discussion Forums Relatively continuous online Discussion Forums are woven into the fabric of ETEC 522, and your insightful, active participation in these will be evaluated as 25 of your overall grade in the course. You will be expected to demonstrate excellent knowledge of the course material, as well as the ability to think critically and profoundly about the issues arising in discussion, course readings, and in the contributions of your peers. Discussion Forums are essentially conversations active and successful participation requires due consideration of the topic, of the contributions of others, and of the timing, quantity and quality of your own contributions.
OBJECTIVE/ CRITERIA Excellent PERFORMANCE INDICATORS Acceptable Re-focus
Presence Value 7 Establishes and maintains a consistent and valuable presence in the conversational flow of the materials covered on the course (6-7) Contributes well and conscientiously in general, with leading contributions in areas of individual interest and expertise (4-5) Sporadic and/or scattered contributions (0-3)
Original Voice Value 6 Consistently raises the discussion to new levels with creative and original interventions and/or starts new discussions that carry the discourse (6) Provides original thoughts and inspirations relative to topics of individual interest or expertise (4) Follows or echoes original discussions without original contributions (2)
Constructive Response Value 6 Actively follows discussion threads to provide constructive responses that celebrate, elaborate and encourages the contributions of participants (6) Responds appropriately to the flow of discussions in ways that demonstrate god timing, lively consideration and consideration of thought (4) Responds in ways that are off topic, poorly paced or discouraging to wider participation (2)
Demonstrated Knowledge Value 6 Always on topic, well researched and well reflected with respect to the immediate questions under consideration and the broader objectives of the course (6) Demonstrates an active connection with the resources, ideas and questions relevant to the current discussion (4) Participates without a sound understanding or review of the basic principles under discussion (2)
17
To Assess or not to Assess? That is the
Question

• Task 1
• Once Basic Skills are acquired coaches and
  teachers should always practice using a Random
  Varied Practice format True or False?
• Handout (permissions obtained)
• 7 replies. Using rubric from Peltz (2004) to
  score each contribution

18
To Assess or not to Assess? That is the
Question

• So. you are about to deliver a fully online 15
  credit masters module to students from all round
  the globe..
• would you summatively assess online discussions
  in your module?
• Yes No It Depends.

19
To Assess or not to Assess? That is the
Question

• Thanks for your time and attention!

20
References

• Rourke, L., Anderson, T., Garrison, R. Archer,
  W., (2001). Assessing social presence in computer
  conferencing transcripts. Journal of Distance
  Education, 14,2.
• Swan, K., Schenker, J., Arnold, S. Kuo, C-L.
  (2007). Shaping online discussion assessment
  matters. e-Mentor, 1 (18), 78-82 at
  http//www.e-mentor.edu.pl/_xml/wydania/18/390.pdf
  .
• Vygotsky, L.S. (1978). Mind in society The
  development of higher psychological processes.
  (In M. Cole, V. John-Steiner, S. Scribner, E.
  Souberman, Eds.). Cambridge Mass Harvard
  University Press.
• Wenger, E. (1998). Communities of Practice.
  Learning, Meaning, and Identity. Cambridge,
  England. Cambridge University Press.

Bandura A. (1977). Self-efficacy Toward a
unifying theory of behavioral change.
Psychological Review, 84, 191-215. Edelstein, S.
Edwards, J. (2002). If You Build It, They Will
Come Building Learning Communities Through
Threaded Discussion. Online Journal of Distance
Learning Administration, Volume V, Number I.
Retrieved June 3rd 2009, from http//www.westga.ed
u/distance/ojdla/spring51/edelstein51.html Lauri
llard, D. (1993). Rethinking University Teaching
A Framework for the Effective Use of Educational
Technology. London Routledge. Palloff R.M.
Pratt, K. (1999). Building Learning Communities
in Cyberspace Effective Strategies for the
Online Classroom. San Francisco, CA Jossey-Bass
Publishers. Pelz, B. (2004). Three principles
of effective online pedagogy. Journal of
Asynchronous Learning Networks 8(3) 3346