Part%20II:%20The%203%20T

1
Part II The 3 Ts of Online Assessment Tools,
Techniques, and (Saving) Time
Curtis J. Bonk, Professor, Indiana
University President, CourseShare http//php.india
na.edu/cjbonk cjbonk_at_indiana.edu Vanessa
Dennen, Assistant Professor Florida State
University vdennen_at_fsu.edu
2
Do you have a strategic plan for evaluation and
assessment?
3
Bonk et al.s (in press) Online Learning
Assessment and Evaluation Model
Bonk, C. J., Wisher, R. A. Champagne, M. V. (in
press). Toward a comprehensive model of
e-learning evaluation The Components.
4
What to Evaluate?

1. Learnerattitudes, learning, jobs.
2. Instructorpopularity, course enrollments.
3. Traininginternal and external.
4. Task--relevance, interactivity, collaborative.
5. Tool--usable, learner-centered, friendly,
   supportive.
6. Courseinteractivity, completion rates.
7. Programgrowth, long-range plans.
8. Organization or Universitycost-benefit,
   policies, vision.

5
Measures of Student Success(Focus groups,
interviews, observations, surveys, exams, records)

• Increased Comprehension Achievement
• High Student Attitudes
• High Retention, Completion Rates in Program
• Jobs Obtained, Internships
• Enrollment Trends for Next Semester
• Grades, Achievement, Certifications
• Computer Log Activity e.g., Number of Posts,
  Participation, Messages/day, Time in System

6
1. Student High-End Success

• Message complexity, depth, interactivity,
  questioning
• Collaboration skills
• Problem finding/solving and critical thinking
• Challenging and debating others
• Case-based reasoning, critical thinking measures
• Portfolios, performances, PBL activities

7
Assessments Possible

• Quizzes and Tests
• Peer Feedback, Mentoring, Responsiveness
• Tasks Attempted or Completed, Usage, etc.
• Discussion/Forum Participation
• Writing, Blogs, Weekly Reflections
• Cases and Problems
• Group Work
• Web Resource Explorations Evaluations
• Performances, Portfolios, etc.

8
Issues to Consider

1. Bonus pts for participation?
2. Pts for peer evaluation of work?
3. Assess improvement?
4. Are tests timed? Allow retakes if lose
   connection? How many retakes?
5. Give unlimited time to complete?
6. Cheating? Is it really that student?
7. How measure competency and demonstrate learning
   online?

9
2. Instructor Success

• High student evals, more signing up, student
  recommendations
• High student completion rates
• Utilize Web to share teaching
• Course recognized with awards

10
3. Training Outside Support

• Training (FacultyTraining.net)
• Courses Certificates (JIU, e-education)
• Reports, Newsletters, Pubs
• Aggregators of Info (CourseShare, Merlot)
• Global Forums (FacultyOnline.com GEN)
• Resources, Guides/Tips, Link Collections, Online
  Journals, Library Resources

11
(No Transcript)
12
3. TrainingInside Support

• Instructional Consulting
• Mentoring (strategic planning )
• Small Pots of Funding
• Laptops
• Summer and Year Round Workshops
• Colloquiums, Tech Showcases, Guest Speakers,
  Awards, Recognitions
• Newsletters, guides, active learning grants,
  annual reports, faculty development, brown bags

13
RIDIC5-ULO3US Model of Technology Use

• 4. Tasks (RIDIC)
• Relevance
• Individualization
• Depth of Discussion
• Interactivity
• Collaboration-Control-Choice-Constructivistic-Comm
  unity

14
RIDIC5-ULO3US Model of Technology Use

• 5. Tech Tools (ULOUS)
• Utility/Usable
• Learner-Centeredness
• Opportunities with Outsiders Online
• Ultra Friendly
• Supportive

15
6. Course Success

• Few technological glitches/bugs
• Adequate online support
• Increasing enrollment trends
• Course quality (interactivity rating)
• Monies paid
• Accepted by other programs

16
7. Online Program or Course Budget (i.e., how
pay, how large is course, tech fees charged, of
courses, tuition rate, etc.)

• Indirect Costs learner disk space, phone,
  accreditation, integration with existing
  technology, library resources, on site
  orientation tech training, faculty training,
  office space
• Direct Costs courseware, instructor, help desk,
  books, seat time, bandwidth and data
  communications, server, server back-up, course
  developers, postage

17
7. ProgramOnline Content Considerations

• Self-Paced or Live mentors?
• Interactive or content dumping?
• Individual or Collaborative?
• Lecture or problem-based learning?
• Factual or performance assessment?

18
8. Institutional Success

• E-Enrollments from
• new students, alumni, existing students
• Additional grants, new State monies
• Press, publication, partners, attention
• Cost-Benefit model, ROI
• Faculty attitudes, training, support
• Acceptable policies, ADL compliant

19
Lets Focus at the Individual LevelWhat Online
Testing Tools Do You Use?
20
What Can Online Tests Do?

• Assess student progress
• Allow for self-assessment
• Provide standards for success
• Timed testing and retesting
• Opportunity for instructor commenting

21
Test Selection Criteria (Hezel, 1999)

• Easy to Configure Items and Test
• Handle Symbols
• Scheduling of Feedback (immediate?)
• Provides Clear Input of Dates for Exam
• Easy to Pick Items for Randomizing
• Randomize Answers Within a Question
• Weighting of Answer Options

22
More Test Selection Criteria

• Recording of Multiple Submissions
• Timed Tests
• Comprehensive Statistics
• Summarize in Portfolio and/or Gradebook
• Confirmation of Test Submission

23
More Test Selection Criteria(Perry Colon, 2001)

• Supports multiple items typesmultiple choice,
  true-false, essay, keyword
• Can easily modify or delete items
• Incorporate graphic or audio elements?
• Control over number of times students can submit
  an activity or test
• Provides feedback for each response

24
More Test Selection Criteria(Perry Colon, 2001)

• Flexible scoringscore first, last, or average
  submission
• Flexible reportingby individual or by item and
  cross tabulations.
• Outputs data for further analysis
• Provides item analysis statistics (e.g., Test
  Item Frequency Distributions).

25
How Use Technology to Assess?
26
(No Transcript)
27
What Assess Online?

• Traditional Online Exams
• Self-Test Exams
• Learner-Content Interactions
• Guided Explorations Virtual Tours and Timelines
• Cases and Vignettes
• Blogs
• Chats with Visual Representations
• Soft Skill Simulations
• Virtual Reality and Role Play Simulations
• Cyber Fashion Shows Music Performance

28
1. Traditional Online Testing
29
Using WebCT Quizzes in a High-Demand Environment
(Brothen Wambach, Technology Source, May/June
2003)

• Several reviews and meta-analyseshave found
  superior student learning in PSI compared to
  traditional lecture/discussion methods. Here,
  students read a textbook and when they are ready,
  they take chapter quizzes after they master one
  chapter, they move on to the next.

30
2. Online Self-Testing
31
3. Learner-Content Interactions (Option 6
Designers)
32
4. Guided Explorations Virtual Tours and
Timelines
33
4. Guided Explorations Interactive Adventure
Content (Andrew Revkin, New York Times, May 25,
2003)
34
5. Case-Based Learning My Patient.com and
SimTeacher
35
6. Blogs (diaries, writing)
36
7. Visual with Chat Learningbydoing.net

• Participants a facilitator of online therapy,
  students at all levels, a doctoral candidate in
  DE, administrators, teachers, lecturers,
  researchers, a physicists, a professor of
  Psychology, a professor of Mathematics, a
  consultant in training, an HR trainer, and a
  psychotherapist. We were located in Herzelia, a
  beach town north of Tel Aviv, Stanford
  California, Baltimore, Montreal, and Ismir,
  Turkey.

37
Games and Simulations

• Theres something new on the horizon, though
  computer-based soft skills simulations, which let
  learners practice skills such as negotiation and
  team building.
• Clark Aldrich, The State of Simulations, Sept.
  2001, Online Learning

38
8. Simulations Virtual Univ Adminstrator
Virtual Leader
39
9. Virtual Worlds/Virtual Reality

• Avatars--representations of people
• Objects--representations of objects
• Maps--the landscape which can be explored
• Bots--artificial intelligence

40
10. Online Performances(e.g., Cyber Fashion
Shows)
41
Which might you use? How would you use?
42
Online Survey Tools for Assessment
43
Sample Survey Tools

• Zoomerang (http//www.zoomerang.com)
• SurveyMonkey (http//surveymonkey.com)
• QuestionMark (http//www.questionmark.com/home.htm
  l)
• Survey Solutions from Perseus (http//www.perseusd
  evelopment.com/fromsurv.htm)
• Infopoll (http//www.infopoll.com)

44
Sample Survey Tools

• Active Feedback
• (http//www.activefeedback.com/af)
• SurveyKey
• (http//www.surveykey.com)
• EZSurvey from Raosoft
• (http//www.raosoft.com/)
• SurveyShare (http//SurveyShare.com from
  Courseshare.com)

45
(No Transcript)
46
Survey Student Opinions (e.g., InfoPoll,
SurveySolutions, Zoomerang, SurveyShare.com)
47
(No Transcript)
48
(No Transcript)
49
Online Survey in Blackboard
50
Web-Based Survey Advantages

• Faster collection of data
• Standardized collection format
• Computer graphics may reduce fatigue
• Computer controlled branching and skip sections
• Easy to answer clicking
• Wider distribution of respondents

51
Why Conduct Online Surveys

• Formative assessment of class
• Increase student voice and ownership in class
• Involve students from other locations
• Quickly gather answers to questions

52
Web-Based Survey Problems Why Lower Response
Rates?

• Low response rate
• Lack of time
• Unclear instructions
• Too lengthy
• Too many steps
• Cant find URL

53
Survey Tool Features

• Support different types of items (Likert,
  multiple choice, forced ranking, paired
  comparisons, etc.)
• Maintain email lists and email invitations
• Conduct polls
• Adaptive branching and cross tabulations
• Modifiable templates library of past surveys
• Publish reports
• Different types of accountshosted, corporate,
  professional, etc.

54
Web-Based Survey Solutions Some Tips

• Send second request
• Make URL link prominent
• Offer incentives near top of request
• Shorten survey, make attractive, easy to read
• Credible sponsorshipe.g., university
• Disclose purpose, use, and privacy
• E-mail cover letters
• Prenotify of intent to survey

55
Plagiarism
56
Increasing Cheating Online(7-30/page,
http//www.syllabus.com/ January, 2002, Phillip
Long, Plagiarism IT-Enabled Tools for Deceit?)

• http//www.academictermpapers.com/
• http//www.termpapers-on-file.com/
• http//www.nocheaters.com/
• http//www.cheathouse.com
• http//www.realpapers.com/
• http//www.pinkmonkey.com/
• (youll never buy Cliffnotes again)

57
Sample Paper Assistance (i.e., Cheating) Sites
58
Sample Paper Assistance (i.e., Cheating) Sites
59

• In recent years there has been a sharp increase
  in students cutting and pasting material into
  papers from Web sites without attribution, or
  purchasing term papers from online term-paper
  mills.

60
(No Transcript)
61
(No Transcript)
62
(No Transcript)
63
Cell Phones to Cheat on Accounting Exam (1/30/03)
64
(No Transcript)
65
Plagiarizing Plagiarism(June 8, 2003 Dartmouth
Review)

• Michael Heberlingan author at the Chronicle and
  president of Baker College for Graduate
  Studiesread an article written in Syllabus
  magazine that looked remarkably like his own
  piece. The subject of the plagiarists piece
  plagiarism.
• the offending author, a graduate student,
  apologized for any passages accidentally
  cut-and-pasted into her own article. Her excuse
  deadline pressure.

66
When Values Collide, NY Times, p. 4, July 15, 2003

• Some of the tutors make questionable decisions.
  Writing papers for the players, if that is true,
  is very questionable.
• I guess it is difficult, if not impossible, to
  stop the faculty members that are devout fans and
  give any grade they want.

67
(No Transcript)
68
Ok, Million Dollar Question How can you reduce
cheating online?
69
Reducing Cheating Online http//www.syllabus.com/
January, 2002, Phillip Long, Plagiarism
IT-Enabled Tools for Deceit?) http//www.syllabus.
com/article.asp?id5916

• Ask yourself, why are they cheating?
• Do they value the assignment?
• Are tasks relevant and challenging?
• What happens to the task after submittedreused,
  woven in, posted?
• Due at end of term? Real audience?
• Look at pedagogy b4 calling plagiarism police!

70
Ways to Reduce Cheating Online (How to Proctor
from a distance, Dan Carnevale, Chronicle of
Higher Ed, Dec., 12, 1999)

1. Use proctored exams
2. Require cameras during discussions and tests
3. Get to know your students better through
   discussions and chats
4. Give random quizzes using chat tool
5. Write a long essay at start of semester to serve
   as an index of writing style
6. When in doubt, place document into a search
   engine (Eve2 or Turnitin.com)

71
Resources for Reducing Cheating
Online(7-30/page, http//www.syllabus.com/
January, 2002, Phillip Long, Plagiarism
IT-Enabled Tools for Deceit?)

• http//www.turnitin.com/ (software, 100, free 30
  day demo/trial)
• http//www.copycatch.freeserve.co.uk/ (free in
  UK)
• http//www.canexus.com/ (Eve2 software essay
  verification engine, 19.95)
• http//www.plagiarism.org/ (resource)
• http//www.academicintegrity.org/ (assoc.)
• http//sja.ucdavis.edu/avoid.htm (guide)

72
(No Transcript)
73
Turnitin.com
74
Turnitin Testimonials

• "Many of my students believe that if they do not
  submit their essays, I will not discover their
  plagiarism. I will often type a paragraph or two
  of their work in myself if I suspect plagiarism.
  Every time, there was a "hit." Many students were
  successful plagiarists in high school. A service
  like this is needed to teach them that such
  practices are no longer acceptable and certainly
  not ethical!

75
Check Papers with Eve2
76
Do Dennen Bonk Plagiarize?
77
More Ways to Reduce Cheating Online (How to
Proctor from a distance, Dan Carnevale, Chronicle
of Higher Ed, Dec., 12, 1999)

1. Vary items in exam
2. Have timed exams
3. Make course too hard to cheat
4. Random selection of items for item pool
5. Use mastery learning for some tasks
6. Assign collaborative tasks

78
Still More Ways to Reduce Cheating Online

1. Use test passwords
2. Rely on computer IP screening
3. Set expectations (e.g., scholarly integrity,
   syllabus procedures)
4. Emphasize consequences (e.g., stories of past
   offenders)
5. Have students make a vow of no cheating (e.g.,
   University of Virginia)

79
Academic Honor Codes!
80
More Ideas (Promoting Academic Integrity)(June
16, 2003, TLTL, IU Anti-Plagiarism Strats)

• Help students recognize citation probs
• Examples, cases, Web sites, 1 credit course
• Help Faculty and AIs recognize problems
• Examples, cases, practice grading , discuss
• Provide practice in citing properly
• Interactive workshops, discussions
• Design assignments to minimize dishonesty and
  poor citation
• Provide frameworks, well designed prompt
• Hold students accountable
• Plan for special situations
• Use available support

81
(No Transcript)
82
Plagiarism Test (Ted Frick, Indiana University)
83
Still More Ways to Reduce Cheating Online
(McMurty (2001) E-Cheating Combating a 21st
Century Challenge, Tech Horizons in Ed, 29(4),
36-41.

1. Require prewriting steps such as an outline and
   rough draft
2. Require electronic paper submission
3. Design writing assignments with high specificity,
   not open-ended (harder to find a match in a pool)
4. Use Google to search for phrases that do not
   appear to be from a particular student
5. Peruse paper help (i.e., cheat) sites

84
Tips on Authentification

• Check e-mail access against list
• Use password access
• Provide keycode, PIN, or ID
• Futuristic Other Palm Print, fingerprint, voice
  recognition, iris scanning, facial scanning,
  handwriting recognition, picture ID

85
The Ultimate in Plagiarism???
86
Fake Degrees (Oakley, 2003)
87
Fake Degrees (Burks Oakley, 2003)

• http//www.fakedegrees.com/
• http//www.myodegrees.com/
• http//www.boxfreeconcepts.com/magicmill/
• http//www.blackmarket-press.com/
• http//www.bogusphd.com/
• http//www.coastal-retreat.com/degrees/funny.html
• www.counterfeitlibrary.com/cl/qual.asp

88
Sample Fake Degree Sites
89
Imagine the Assessments!American
InterContinental University (AIU)
90
Final advicewhatever you do