Impact Assessment in Online Courses

1
Impact Assessment in Online Courses

• Dr. Joan D. McMahon
• John Sener

Presented at the ALN Conference, Nov 7-9, 2007,
Orlando, FL
2
Session Outcomes

• Review how traditional learning assessment
  reflects a lack of emphasis on assessing impact
• Re-configure online learning evaluations to
  reflect retention and impact for skill-based
  and/or professional courses.
• Determine ways to collect impact data on student
  learning.
• Project benefits of impact evaluation to
  professional and departmental portfolios

3
Traditional Assessments

• What are the traditional assessments you or your
  colleagues use?

4
Focus of Traditional Learning Assessment

• Most traditional learning assessment is focused
  on
• internally-oriented outcomes
• Grades, Course Completion Rates
• KSAs What did they learn (to do, think/feel)?
• Use of results to improve courses compare
  results
• Oriented toward institutional needs data
  collection process is primarily extractive
• This is not bad, but it is limited/limiting

5
What is Impact Assessment?

• Results vs. Impact What happened vs. The
  So-
• What Factor Influence
• Relevancy to the real world Force
• exerted by a new idea, concept, ideology
• Can use Quantifiable and Qualifiable Statistics
• Often Action Research Based
• Also focused on effects on learner,
  environment, world,
• not just on internal system effects

6
How is Impact Data Used?

• Course Evaluations
• Justification for online offerings
• Proof of Student Learning
• Proof of Skill/Competency attainment
• Proof of Compliance to Standards

7
Assessing Impact Example Framework

• Kirkpatrick Scale
• L1 Reaction (learner satisfaction, smiley
  tests)
• L2 Learning (learning effectiveness,
  perceived/real)
• L3 Behavior application of learning to
  environment
• L4 Results effects on environment resulting
  from learner performance
• Last two approaches are forms of assessing
  impact, but they tend to be employer-focused, not
  learner- or societal-focused

8
Re-Configuring Traditional Assessments

• Learner Focused Discussions What questions can
  you ask to get to impact?

9
Reframing the questions

• Formative Course Evaluations
• How am I helping you learn?
• What is hindering your learning?
• What course corrections would you suggest?

10
Examples of Impact Data Collection

• Justification for Online Learning
• (How does online learning help you become a
  lifelong learner?).
• Online learning allows for more of a
  multi-tasking environment to take place - for
  example, when participating in my virtual chat
  seminar, I was able to search through my files
  and the web to provide insight into the
  conversation.
• I also enjoy working at my own pace - sometimes
  that involves working at unconventional/non-workin
  g hours - and that's convenient for my lifestyle.
  
• Lastly, I appreciate the ability to share files
  with others and witness the progress that each
  member is making during the progression of class.
  

11
Falling Short on Impact Data Collection

• Proof of Student Learning cut short.
• We give examples but fail to ask so what?
• We go on field trips and then dismiss the class.
• We ask for reports on a topic but dont ask so
  what?
• We give team assignments without team training.
• We have discussions but dont develop communities.

12
Examples of Impact Data Collection

• Proof of Student Learning
• What was most useful to you in your learning
  process?
• The format of being able to answer the
  questionsabout the text material at any time
  and that the answers would be posted for all to
  see was motivating. I spent more time reflecting
  on my answers before writing than in a classroom
  discussion.
• Moreover, I was able to focus more clearly on my
  classmates answers to the discussion board
  questions.

13
Examples of Impact Data Collection

• Proof of Student Learning
• (How does online learning help you become a
  lifelong learner?).
• I learned that I much prefer to work on
  assignments in pieces...I found that keeping a
  log made the writing of the case study much
  easier, but, I also found that I have problems
  being precise in my writing.

14
Examples of Impact Data Collection

• Proof of Skill/Competency Attainment
• (How does online learning help you become a
  lifelong learner?).
• The online team effort at developing the team
  depot and the tools researched have been very
  useful to me professionally. The structure of
  looking for analysis, interventions and
  evaluation tools opened a new field of
  professional thought for me.
• As with self-directed learning, online learning
  requires that one must have superb
  time-management skills, if one does not (like me)
  the course becomes exceptionally challenging.

15
Reframing the questions

• Summative Course Evaluations
• Given the mandate to offer this course solely
  through distance learning (online and phone in
  this case), how could difficult issues be more
  clearly explained?
• After reading the syllabus, directions for the
  assignments and the corresponding grading rubric,
  how would you improve the clarity of the course
  requirements and evaluation procedures?
• Online teachers are usually available more often
  than on-ground ones. How could the instructor
  have been more available to answer questions?

16
Re-Configuring Traditional Assessments

• What is the question?
• To a large degree, I am an aural learner so that
  I missed that element of interaction. I also
  respond better to f2f situations as opposed to
  the solitude of the on-line set-up so that
  maintaining motivation was an ongoing issue for
  me.

17
Re-Configuring Traditional Assessments

• Learner Reflections/ Evaluations
• Regardless of how you felt about learning online,
  what has been the impact of this course in your
  work or personal life? What new skills have you
  used to affect change?

18
Societal Impact- Chart on Student Projects

• Student Projects
• Structured the disciplinary process for his
  Division of Coca-Cola by synthesizing/charting
  the actions supervisors must take when addressing
  the discipline of their employees. (Until this
  time, there had been no uniform system
  administered consistently.)
• Created resource guide for managers dealing with
  addictive behaviors in the workplace

• Impact on the Organization
• Has already been implemented as the new
  progression of discipline at one of the
  Coca-Cola facilities he represents.
• Student presented it to HR director of Holy Cross
  Hospital for incorporation in the hospitals
  management training program on substance abuse
  scheduled for late Fall.

19
Benefits to Professional and Departmental
Portfolios

• Documentation of tangible effectiveness evidence
  for promotion tenure process
• Learning effectiveness
• ROI/VOI
• Student Satisfaction Referrals to course
• Proof for accreditation purposes

20
Collecting Impact Data

• What additional ideas do you have in collecting
  impact data in your situation?

21
Impact of this presentation
22
Contact Us

• Joan D. McMahon, Professor Emeritus
• Towson University and
• CEO, Transforming Leaders, LLC
• mcmahon_at_towson.edu
• 410-733-0029
• John Sener, Founder and Chief Learner
• Sener Learning Services
• jsener_at_senerlearning.com
• 301-920-2263