Project 1 Pre-Final Presentation Quiz Preparation in an online management system

1
Project 1Pre-Final PresentationQuiz Preparation
in an online management system

• Kenneth Compass
• Sanjay Jauhar
• Camille Mercado
• Kristen Wakamoto

2
Methods Summary

• Phase 1
• Contact and set up either a face to face,
  telephone or Skype meetings with past instructors
  who have used Moodle v1.97
• Assess Moodle 1.97 using HCI Guidelines
• Primary
•  Gain insight of the quiz preparation system
  through a walkthrough with the users
• Think aloud procedure
• Done on Moodle 1.97
• Questionnaires are also given out Unstructured
  Interviews 
• Secondary
•  For users that were not able to have scheduled
  meetings with us
• Sent out questionnaires/surveys about their
  usage questions based on interviews with primary
  interviewees

3
Methods Summary

• Phase 2
• Formulate structured interviews
• Re-administer structured interviews to
  new respondents 
• Based on Moodle 2.0
• Primary
• Look into differences between 1.97 and 2.0
• Ask test subjects to try out 2.0 and ask them for
  feedback
• Structured Interviews and re-evaluation
• Secondary
•  Look into differences between 1.97 and 2.0
• Ask secondary users about differences found
  through questionnaires and see what they think
  about it
• Analyze data and make recommendations

4
Methods Summary

• Phase 3 
• Contacted users who did not respond to the first
  attempt
• Determined their availability.
• Sent out surveys to those who were under time
  constraints
• For others, we set up either face to face, Skype,
  or phone interviews
• Think out loud walk through's of Moodle 1.97
• Ask users to look at Moodle 2.0 and then send
  questionnaires regarding their thoughts

5
Preliminary Results

• Initial Interviews (done with Moodle
  1.97) Phone, Skype, and  In-Person Think Out
  Loud Walkthroughs
•     Major usability issues identified and
  confirmed with   pre-assessment of system by
  team.
•     Turn editing on/off  
•     Many options - confusing and daunting
•     Adding questions does not go directly to quiz
• Clunky interactions to correctly add a question
  to a quiz

6
Preliminary Results

• Demographic Interviewed
• Got in touch with 15 instructors
• Most were in their 40s and 50s
• Taught mostly Management, Engineering, or
  Computer Science classes at the University
  Extension
• Have used the Moodle system for at least 1.5
  years
• Create their own quizzes
• Have not really had to use any other learning
  management systems aside from Moodle

7
Preliminary Results

• In Person and Skype Interviews (1.97)
• Screen recording with walkthroughs of the Moodle
  System
• Found the initial set up of the quiz system to be
  easy
• e.g. Creating the title, description, etc.
• Did have issues concerning the help buttons
• Still had concerns about arbitrary processes
•  Main roadblock came with the question bank
  feature
• "Why is it that you have to set it up with the
  question bank?"
• How do you edit questions? In the question bank
  or the quiz itself?
• Interface is found to be too "crunched up"

8
Preliminary Results

• Phone Interviews (1.97)
• One concern brought up was how to prevent
  cheating
• Issues with the question bank feature as well
• Keeping the students updated (if he/she misses a
  quiz or if the instructor modifies the quiz)
• Surveys (1.97)
• In general, instructors say that the interface is
  good for conducting online classes
• Most brought up the point of functionality vs.
  usability
• "Yes there was a steep learning curve, but
  eventually, each time I use it I discover more"
• Compare it to Blackboard (has a better quiz/exam
  feature)

9
Compiled Results

• Common Usability Issues
• Turn editing on - Why? If needed - Move to a more
  noticeable area
• Drop down Menu's contain too much information
• Need more commonly used default values
• Question bank Interface - Completely confusing
• Needs directions or more intuitive flow
• All around - needs more on screen instructions,
  not just help actions
• Some users liked the system as is, but still felt
  it was hard to learn

10
Compiled Results

• Moodle 2.0 Addresses most usability issues in
  1.97
• Turn editing on - Moved to more "eye popping"
  position
• Drop down menu content re-vamped (slightly
  better)
• Adding questions to quizzes improved
• Quiz question bank improved
• Can add questions directly to quizzes and use in
  a bank

11
Adding questions in Moodle 1.97
12
Adding a question in Moodle 2.0
13
Moodle 2.0 Question bank
14
Compiled Results

• Many options - needed for high user customization
  - but they appear to complicate the system
• Still compiling results
• Almost all tests indicate an upgrade is needed
• Demo runs of Moodle 2.0 were positive and
  enthusiastic
• Preliminary Recommendations
• Leaning towards recommending quiz system be
  upgraded to newer version of Moodle
• Do not feel like customization is needed
•  

15
Updated Schedule

• Week Five (04/24 - 04/30)
• Meet up with users run usability tests (all
  members)
• First Phase of the usability tests with the
  primary group of users (all members)
• Create relevant features "list" Moodle 2.0 that
  might address usability issues. (Kenny)  
• Week Six (05/01 - 05/07)
• Analyze data received from usability tests
  (Camille and Kristen)
• Proceed with second phase of testing (all
  members)
• Publish web survey and notify all participants to
  participate(Sanjay).
• Week Seven (05/08 - 05/14)
• Analyze data received from usability tests
  (Sanjay and Kenny)
• Re-evaluate system(all members).

16
Updated Schedule

• Week Eight (05/15 - 05/21)
• Contact a user set for suggestions and opinions
  on redesign (Sanjay, Camille)
• Conduct Final remaining interviews (Sanjay,
  Kenny, Camille)
• 2. Weeks Nine and Ten (05/22 - 06/04)
• Conduct Final remaining interviews (Sanjay,
  Camille)
• Refine and render collected data (Sanjay,
  Kristen)
• Produce Final Report (Team)
• Final Presentation (Sanjay, Camille, Kenny)

17
Thank you

• Questions ?