Online Course Assessment Methods: How Effective Are They?

1
Online Course Assessment Methods How Effective
Are They?

• Dr. Kim Marie McGinnis

2
My Background

• Doctorate in Educational Leadership and Policy
  Analysis, ETSU
• Masters in Education, WCU
• BS in Landscape Horticulture, NCSU
• Director of Occupational Extension Training,
  Carteret Community College
• Dean of Technical and Vocational Programs,
  Mayland Community College

3
My Interest in this Study

• I develop Internet courses
• I want to continually improve the quality and
  effectiveness of the courses that I teach
• I am also an Internet course student and have
  questions about the choices of assessments that
  have been used in those courses
• Assessment is a very current accountability issue
  and distance education is becoming a common
  method of instruction

4
Need for this Study

• Assessment has become an integral part of
  accountability in the American educational system
• Distance education is the fastest growing form of
  education
• With increased emphasis on accountability in
  general and increased scrutiny of online teaching
  and learning, issues of assessment have taken on
  more importance than ever before (Comeaux, 2005)

5
Research Questions

• Academic discipline Are there differences in
  assessment methods being used among faculty who
  teach in different academic disciplines in the
  online environment?
• Learning Objectives being met Are there
  differences in perceived effectiveness of the
  assessment methods being used among individual
  instructors in determining if the course learning
  objectives have been met?

6
Research Questions

• Internet course development training Are there
  differences in assessment methods used between
  those online instructors who received training in
  Internet course development as compared to those
  who did not?
• Number of assessments per course Are there
  differences in the number of different types of
  assessments being used per course by each
  instructor?

7
Research Questions

• Years teaching Internet courses Are there
  differences in the types of assessments being
  used by online instructors who have been teaching
  in the online environment for more than three
  years, as compared with instructors who have been
  teaching in the online environment for three or
  fewer years?

8
Research Questions

• Number of Internet courses per year Are there
  differences in the types of assessments being
  used by online instructors who teach more than
  one Internet course per year and instructors who
  teach only one Internet course per year?

9
Research Design

• Primary data analysis was conducted based on
  responses to an original survey document
  developed by the researcher
• The survey was conducted online and sent to all
  online instructors who had taught an Internet
  course or a web-enhanced course during the
  2004-2005 academic year at the 15 Western North
  Carolina community colleges that serve the
  Appalachian region (the population - 371)
• 174 responses were received (47)

10
The Survey

• The survey consisted of 16 questions
• Questions 10 and 12 were open-ended questions
• Question 1 identified the academic department for
  which the instructor taught
• Questions 2,3,4, and 5 addressed the number of
  unduplicated Internet courses that the instructor
  taught during different academic years
• Question 6 identified the assessment methods that
  individual instructors were using in the Internet
  courses they taught

11
The Survey

• Question 8 required a yes or no response to
  receiving training in Internet course development
• Question 9 focused on the area in which the
  training was received
• Question 11 provided the number of different
  types of assessment being used by individual
  instructors per course
• Questions 13-16 retrieved demographic information

12
Demographics

• Gender Of 167 responses, 71 respondents were
  male (40.8) and 96 were female (55.2)
• Age Of 164 responses, the mean age was 44.65
  with a range from 25-76 years old
• Academic degree Of 166 responses, 8 had an
  associates degree (4.6), 19 had a bachelors
  degree (10.9), 125 had a masters degree
  (including EdS) (71.8), and 14 had a doctoral
  degree (8.0)

13
Demographics

• Years experience in education Of 169 responses,
  the mean number of years of experience in
  education was 13.95 with a range from 1-45 years

14
Research Question 1

• Are there differences in assessment methods being
  used among faculty who teach in different
  academic disciplines in the online environment?
• Chi-square with a two-way contingency table
  analysis was conducted for each assessment method

15
Research Question 1

• Academic discipline and portfolio use were not
  found to be significantly related
• Uses portfolio Business (21.3), Vocational
  (23.1), Health Occupations (0), Arts (20),
  Public Safety (0), Continuing education (0),
  Social Sciences (18.2), Hard Sciences (0),
  Other (0)

16
Research Question 1

• Academic discipline and true/false test use were
  found to be significantly related
• Uses true/false tests Business (68.0),
  Vocational (76.9), Health Occupations (57.1),
  Arts (30), Public Safety (60), Continuing
  Education (100), Social Sciences (27.3), Hard
  Sciences (56.3), and Other (100)

17
Research Question 1

• Academic discipline and multiple-choice test use
  were found to be significantly related
• Uses multiple-choice tests Business (92),
  Vocational (92.3), Health Occupations (100),
  Arts (63.3), Public Safety (100), Continuing
  Education (100), Social Sciences (81.8), Hard
  Sciences (87.5), and Other (100)

18
Research Question 1

• Academic discipline and short-answer test use
  were found to be significantly related
• Uses short-answer tests Business (44),
  Vocational (84.6), Health Occupations (57.1),
  Arts (36.7), Public Safety (60), Continuing
  Education (100), Social Sciences (31.8), Hard
  Sciences (62.5), Other (100)

19
Research Question 1

• Academic discipline and essay use were not found
  to be significantly related
• Uses essay tests Business (28), Vocational
  (53.8), Health Occupations (42.9), Arts
  (56.7), Public Safety (80), Continuing
  Education (50), Social Sciences (50), Hard
  Sciences (37.5), Other (100)

20
Research Question 1

• Academic discipline and discussion use were found
  to be significantly related
• Uses discussion Business (48), Vocational
  (84.6), Health Occupations (57.1), Arts (80),
  Public Safety (100), Continuing Education
  (100), Social Sciences (90.9), Hard Sciences
  (37.5), Other (100)

21
Research Question 1

• Academic discipline and individual project use
  were not found to be significantly related
• Uses individual projects Business (74.7),
  Vocational (69.2), Health Occupations (57.1),
  Arts (73.3), Public Safety (80), Continuing
  Education (50), Social Sciences (40.9), Hard
  Sciences (6.3), Other (0)

22
Research Question 1

• Academic discipline and group project use were
  found to be significantly related
• Uses group projects Business (24), Vocational
  (30.8), Health Occupations (71.4), Arts
  (53.3), Public Safety (80), Continuing
  Education (0), Social Sciences (40.9), Hard
  Sciences (6.3), Other (0)

23
Research Question 1

• Academic discipline and problem-solving activity
  use were not found to be significantly related
• Uses problem-solving activities Business (44),
  Vocational (23.1), Health Occupations (28.6),
  Arts (40), Public Safety (20), Continuing
  Education (50), Social Sciences (27.3), Hard
  Sciences (43.8), Other (0)

24
Research Question 1

• Academic discipline and self-assessment use were
  not found to be significantly related
• Uses self-assessment Business (14.7),
  Vocational (15.4), Health Occupations (28.6),
  Arts (20), Public Safety (20), Continuing
  Education (0), Social Sciences (4.5), Hard
  Sciences (0), Other (0)

25
Research Question 1

• Academic discipline and other assessment use were
  not found to be significantly related
• Uses other assessments Business (6.7),
  Vocational (7.7), Health Occupations (0), Arts
  (23.3), Public Safety (0), Continuing Education
  (0), Social Sciences (18.2), Hard Sciences
  (12.5), Other (0)

26
Research Question 2

• Are there differences in perceived effectiveness
  of the assessment methods being used among
  individual instructors in determining if the
  course learning objectives have been met?
• A chi-square test of frequencies was used
• There was a statistically significant difference
  in the perceptions of effectiveness of all of the
  assessment methods with the exception of
  self-assessment and the category of other

27
Research Question 2

• Figure 1 Bar Graph of the Perceived
  Effectiveness of Portfolio Assessment (least
  effective N 7, somewhat effective N 5,
  effective N 12, more effective N 13, most
  effective N 22)

28
Research Question 2

• Figure 2 Bar Graph of the Perceived
  Effectiveness of True/False Tests (least
  effective N 12, somewhat effective N 19,
  effective N 58, more effective N 22, most
  effective N 7)

29
Research Question 2

• Figure 3 Bar Graph of the Perceived
  Effectiveness of Multiple-Choice Tests (least
  effective N 6, somewhat effective N 16,
  effective N 61, more effective N 49, most
  effective N 23)

30
Research Question 2

• Figure 4 Bar Graph of the Perceived
  Effectiveness of Short Answer Tests (least
  effective N 2, somewhat effective N 2,
  effective N 28, more effective N 49, most
  effective N 23)

31
Research Question 2

• Figure 5 Bar Graph of the Perceived
  Effectiveness of Essay Tests (least effective N
  0, somewhat effective N 2, effective N 10,
  more effective N 41, most effective N 41)

32
Research Question 2

• Figure 6 Bar Graph of the Perceived
  Effectiveness of Discussion Questions (least
  effective N 3, somewhat effective N 5,
  effective N 27, more effective N 54, most
  effective N 39)

33
Research Question 2

• Figure 7 Bar Graph of the Perceived
  Effectiveness of Individual Projects (least
  effective N 6, somewhat effective N 0,
  effective N 12, more effective N 37, most
  effective N 72)

34
Research Question 2

• Figure 8 Bar Graph of the Perceived
  Effectiveness of Group Projects (least effective
  N 4, somewhat effective N 13, effective N
  15, more effective N 29, most effective N 15)

35
Research Question 2

• Figure 9 Bar Graph of the Perceived
  Effectiveness of Problem- Solving Activities
  (least effective N 4, somewhat effective N 3,
  effective N 8, more effective N 30, most
  effective N 43)

36
Research Question 2

• Figure 10 Bar Graph of the Perceived
  Effectiveness of Self Assessment (least effective
  N 11, somewhat effective N 13, effective N
  16, more effective N 10, most effective N 3)

37
Research Question 2

• Figure 11 Bar Graph of the Perceived
  Effectiveness of Other Assessment Methods (least
  effective N 5, somewhat effective N 4,
  effective N 3, more effective N 12, most
  effective N 9)

38
Research Question 3

• Are there differences in assessment methods used
  between those online instructors who received
  training in Internet course development and those
  that did not?
• A chi-square two-way contingency table analysis
  was used to compare each assessment method
  individually with training

39
Research Question 3

• Training and uses portfolio were not
  significantly related
• 16.1 of instructors who received training used
  portfolio
• 22 of instructors who did not receive training
  used portfolio

40
Research Question 3

• Training and true/false test use were not found
  to be significantly related
• 55.6 of instructors who received training used
  true/false tests
• 52 of instructors who did not receive training
  used true/false tests

41
Research Question 3

• Training and multiple-choice test use were not
  found to be significantly related
• 87.9 of instructors who received training use
  multiple-choice tests
• 78 of instructors who did not receive training
  use multiple-choice tests

42
Research Question 3

• Training and short-answer test use were not found
  to be significantly related
• 47.6 of instructors who received training use
  short-answer tests
• 50 of instructors who did not receive training
  use short-answer tests

43
Research Question 3

• Training and essay test use were not found to be
  significantly related
• 41.9 of instructors who received training use
  essay tests
• 40 of instructors who did not receive training
  use essay tests

44
Research Question 3

• Training and discussion use were found to be
  significantly related
• 70.2 of instructors who received training use
  discussion
• 48 of instructors who did not receive training
  use discussion

45
Research Question 3

• Training and individual project use were not
  found to be significantly related
• 71.8 of instructors who received training use
  individual projects
• 70 of instructors who did not receive training
  use individual projects

46
Research Question 3

• Training and group project use were not found to
  be significantly related
• 36.3 of instructors who received training use
  group projects
• 24 of instructors who did not receive training
  use group projects

47
Research Question 3

• Training and problem-solving activity use were
  not found to be significantly related
• 41.9 of instructors who received training use
  problem-solving activities
• 30 of instructors who did not receive training
  use problem-solving activities

48
Research Question 3

• Training and self-assessment use were not found
  to be significantly related
• 16.1 of instructors who received training use
  self-assessment
• 6 of instructors who did not receive training
  use self-assessment

49
Research Question 3

• Training and other methods of assessment use were
  not found to be significantly related
• 12.1 of instructors who received training use
  other methods of assessment
• 8 of instructors who did not receive training
  use other methods of assessment

50
Research Question 4

• Are there differences in the number of different
  types of assessments being used per course by
  each instructor?
• A chi-square frequencies test was conducted

51
Research Question 4

• There was a significant difference in the number
  of different types of assessments being used per
  course by each instructor
• The number of different types of assessments
  being used per course per instructor ranged from
  1-20 with a mean of 3.6

52
Research Question 5

• Are there differences in the types of assessments
  being used by online instructors who have been
  teaching in the online environment for more than
  three years as compared with instructors who have
  been teaching in the online environment for three
  or fewer years?

53
Research Question 5

• A chi-square two-way contingency table analysis
  was conducted comparing each individual
  assessment method with the length of time
  teaching online courses

54
Research Question 5

• Length of time teaching online courses and
  portfolio use were not found to be significantly
  related
• 17.8 of online instructors who have taught more
  than 3 years use portfolio
• 18.2 of online instructors who have taught 3 or
  fewer years use portfolio

55
Research Question 5

• Length of time teaching online courses and
  true/false test use were not found to be
  significantly related
• 54.6 of online instructors who have taught for
  more than 3 years use true/false tests
• 55.9 of online instructors who have taught for 3
  or fewer years use true/false tests

56
Research Question 5

• Length of time teaching and multiple-choice test
  use were found to be significantly related
• 85.1 of online instructors who have taught for
  more than 3 years use multiple-choice tests
• 85.3 of online instructors who have taught for 3
  or fewer years use multiple-choice tests

57
Research Question 5

• Length of time teaching and short-answer test use
  were not found to be significantly related
• 48.3 of online instructors who have taught for
  more than 3 years use short-answer tests
• 48.8 of online instructors who have taught for 3
  or fewer years use short-answer tests

58
Research Question 5

• Length of time teaching and essay test use were
  not found to be significantly related
• 41.4 of online instructors who have taught for
  more than 3 years uses essay tests
• 41.2 of online instructors who have taught for 3
  or fewer years uses essay tests

59
Research Question 5

• Length of time teaching and discussion use were
  not found to be significantly related
• 63.8 of online instructors who have taught for
  more than 3 years use discussion
• 64.1 of online instructors who have taught for 3
  or fewer years use discussion

60
Research Question 5

• Length of time teaching and individual project
  use were not found to be significantly related
• 71.3 of online instructors who have taught for
  more than 3 years use individual projects
• 71.2 of online instructors who have taught for 3
  or fewer years use individual projects

61
Research Question 5

• Length of time teaching and group project use
  were not found to be significantly related
• 32.8 of online instructors who have taught for
  more than 3 years use group projects
• 32.9 of online instructors who have taught for 3
  or fewer years use group projects

62
Research Question 5

• Length of time teaching and problem-solving
  activity use were not found to be significantly
  related
• 38.5 of online instructors who have taught for
  more than 3 years use problem-solving activities
• 38.8 of online instructors who have taught for 3
  or fewer years use problem-solving activities

63
Research Question 5

• Length of time teaching and self-assessment use
  were not found to be significantly related
• 13.2 of online instructors who have taught more
  than 3 years use self-assessment
• 13.5 of online instructors who have taught 3 or
  fewer years use self-assessment

64
Research Question 5

• Length of time teaching and other assessment
  method use were not found to be significantly
  related
• 10.9 of online instructors who have taught more
  than 3 years use other assessment methods
• 10.6 of online instructors who have taught 3 or
  fewer years use other assessment methods

65
Research Question 6

• Are there differences in the types of assessments
  being used by online instructors who teach more
  than one Internet course per year as compared
  with instructors who teach only one Internet
  course per year?
• A chi-square two-way contingency table analysis
  was used

66
Research Question 6

• The number of online courses taught per year was
  not significantly related to portfolio,
  true/false tests, short answer tests, essay,
  discussion, individual projects, problem-solving
  activities, self-assessment or other assessment
  methods

67
Research Question 6

• The number of online courses being taught per
  year was significantly related to multiple-choice
  tests and group projects
• 73.2 of online instructors who teach only 1
  Internet course use multiple-choice tests
• 85.1 of online instructors who teach more than 1
  Internet course per year use multiple-choice tests

68
Research Question 6

• 23.2 of online instructors who teach only 1
  Internet course per year use group projects
• 32.8 of online instructors who teach more than 1
  Internet course per year use group projects

69
Qualitative Data

• Please describe and indicate how effective the
  training you received was?
• 67 stated that the training was effective
• 18 stated that the training was ineffective

70
Qualitative Data

• Themes
• Those trained would have liked to have training
  in assessment and pedagogy in addition to
  BlackBoard/technology
• Training was technology based only
• Training was minimal
• A distance education coordinator that was
  regularly available for follow up training was
  extremely helpful

71
Qualitative Data

• If you have taught Internet courses for more than
  one year, have your methods of assessment changed
  over time? Why?
• 61 responded that their methods have changed
  over time
• 38 responded that their methods have not changed
  over time

72
Qualitative Data

• Themes
• A variety of assessment methods need to be used
  in each course to ensure that the course
  objectives are being met
• Alternative assessment methods are more reliable
  indicators of course effectiveness
• Lack of time some instructors are using less
  effective objective testing because of a lack of
  time

73
Qualitative Data

• Previously, I avoided objective
  (multiple-choice, true/false) assessments and
  used only short answer/discussion-type questions
  however, they became too time consuming to grade
  considering the number of students in my sections
  (often 30).

74
Conclusions

• There is a difference between some academic
  disciplines in relation to the type of assessment
  methods being used in online courses
• There is a difference in perceived effectiveness
  of assessment methods among the individual
  instructors surveyed

75
Conclusions

• The survey results confirmed that objective
  testing is not considered the most effective
  method of assessment but several instructors
  still use this method because of time constraints
• The survey results confirm that a variety of
  assessment methods need to be used within each
  Internet course to determine the effectiveness of
  the course

76
Conclusions

• There was no difference in the assessment methods
  being used by those instructors who received
  training and those who did not
• There is a difference in the number of types of
  assessment being used by online instructors

77
Conclusions

• The survey data indicated that there was not a
  difference in assessment methods being used by
  instructors who have taught for more than 3 years
  as compared to instructors who have taught for 3
  or fewer years
• There is a difference in some of the types of
  assessments being used by instructors who teach
  more than 1 Internet course per year

78
Implications for Practice

• The findings of this study indicated that online
  instructors should be trained in assessment
  methods and pedagogy in addition to the course
  management system and technology
• A policy needs to be developed within the NCCC
  system that sets guidelines for workload for
  Internet course instructors so that the most
  effective methods of assessment will be used in
  the online environment

79
Implications for Practice

• A policy needs to be developed within the NCCC
  system that places caps on the number of students
  who are allowed to be in one course section of an
  online course so that the instructor has time to
  use the more effective means of assessment in
  online courses

80
Implications for Practice

• The findings of the study indicate that a
  distance education coordinator should be hired at
  all institutions and this individual should
  provide monthly training and individual
  consultation to online instructors to ensure that
  the instructors are adequately trained and
  comfortable in the online environment

81
Recommendations for Future Research

• Similar studies should be conducted in other
  parts of the state of North Carolina to confirm
  results from the western part of the state
• This study should be replicated after the groups
  of respondents have received training in
  assessment methods

82
Recommendations for Future Research

• This study should be conducted by surveying
  students instead of instructors to see if the
  students perspectives match the instructors
  perspectives
• A qualitative study investigating perceptions of
  effectiveness of assessment methods in the online
  environment should be undertaken

83
Recommendations for Future Research

• Similar studies should be conducted in other
  states