Evaluating Californias Webbased Training for Medical Assistants - PowerPoint PPT Presentation

1 / 30
About This Presentation
Title:

Evaluating Californias Webbased Training for Medical Assistants

Description:

Web-based Training for. Medical Assistants. Leslie Heyden, MA ... primary target audience is medical assistants in California VFC provider practices. ... – PowerPoint PPT presentation

Number of Views:69
Avg rating:3.0/5.0
Slides: 31
Provided by: nnaka
Category:

less

Transcript and Presenter's Notes

Title: Evaluating Californias Webbased Training for Medical Assistants


1
Evaluating Californias Web-based Training for
Medical Assistants
  • Leslie Heyden, MA
  • California Department of Public Health
  • Immunization Branch
  • Information and Education Section
  • February 19, 2009

2
Overview
  • The purpose of the CDPH IZ Branch evaluation is
    to assess the use and effectiveness of the first
    two immunization skills elearning lessons
  • Preparing Vaccines
  • Administering Vaccines
  • The elearning is custom developed, interactive
    web-based training.
  • The primary target audience is medical assistants
    in California VFC provider practices.
  • The lessons are available via eziz.org.

2
3
eziz.org
  • The new California VFC program website
  • A one-stop shop for California VFC providers and
    their staffs
  • A repository of a wide variety of immunization
    skills training and VFC program resources,
    including
  • VFC forms (e.g., vaccine order and
    recertification forms, vaccine temperature and
    usage logs, patient eligibility screening forms)
  • Job aids, guidelines, posters, and
    up-to-the-minute news about vaccine availability

3
4
eziz.orghome page
4
5
Webinar learning objectives
  • By the end of this webinar you will be able to
  • List the objectives of the evaluation plan
  • Explain why CDPH IZ branch selected the EZ-IZ
    elearning lessons to evaluate
  • Describe the methodology CDPH developed to
    evaluate the training
  • Indicators
  • Targets
  • Methods, including data sources
  • Describe lessons learned

5
6
Why evaluate EZ-IZ elearning?
  • Is IZ Branchs first elearning project
  • Has potential for high impact re ability to
    reach unlimited number of learners with limited
    resources
  • Primary tool needed to conduct evaluation (LMS)
    was already being created
  • IZ Branch wanted to compare the reach of the
    elearning to existing classroom training

6
7
Goal of activity to be evaluated
  • To educate MAs in California VFC provider
    practices about best practices for vaccine
    preparation and administration

7
8
Evaluation objectives
  • By September 5, 2008 the first two elearning
    lessons of the EZ-IZ curriculum (Preparing
    Vaccines and Administering Vaccines) will launch,
    and the learning management system will begin
    tracking usage and learner performance.
  • By March 31, 2009, at least one person from 5
    (approximately 180) of Californias active VFC
    provider sites (approximately 3,600) will have
    completed one of the elearning lessons (either
    Preparing Vaccines or Administering Vaccines) and
    have passed its posttest.

8
9
Evaluation objectives (contd)
  • By September 5, 2009, at least one person from
    20 (approximately 720) of active California VFC
    provider sites will have completed the Preparing
    Vaccines and Administering Vaccines elearning
    lessons and passed the posttest for each.

9
10
Evaluation objectives (contd)
  • By November 30, 2008 all user feedback about the
    EZ-IZ website will have been gathered and
    analyzed, and recommendations made.
  • Recommendations concerning website usability will
    be forwarded to and resolved by the web developer
    and/or the web editor.
  • Recommendations concerning the EZ-IZ Curriculum
    and the lessons in it will be forwarded to and
    resolved by the instructional designer.
  • Recommendations about programming and usability
    issues within the elearning lessons will be
    forwarded to and resolved by the elearning
    vendor.
  • Recommendations concerning the learning
    management system (LMS) will be forwarded to and
    resolved by the LMS vendor.

10
11
Stakeholders
  • Management
  • John Talarico, Interim CDPH IZ Branch Chief
  • Tiffany Sutter, Section Chief Information
    Education
  • Natalie Nakahara, Project Manager and Creative
    Director
  • California VFC Program
  • Claudia Aguiluz, VFC Program Coordinator
  • VFC field staff and field services reps
  • VFC provider staff (particularly MAs who work in
    private provider practices)

11
12
Stakeholders (contd)
  • Immunization (federal, state, and local)
  • CDC (NCIRD)
  • Steve Nickell, Research Coordinator, IZ Branch
  • Sandra Jo Hammer, Nurse Consultant, IZ Branch
  • CDPH IZ Branch Evaluation team (Leslie Heyden,
    Natalie Nakahara, and Steve Nickell)
  • IZ Coordinators in LHDs
  • Others
  • California Medical Assistants Association (CMAA)
    and California medical assistant training
    programs

12
13
Table 3 Program Component/Activity Description
OutcomesShort term Long term
Inputs
ActivitiesInitial Subsequent
Outputs
Funding   Staff   Time
Review available materials choose
content Design training   Employ vendor to
program elearning and create learning management
system   Make training available via
website   Promote training
Review of content by content experts   Beta test
on target audience   Collect assemble
feedback   Decide on changes   Revise training
  Create a one-stop shop learning and resource
portal for CA VFC providers, immunizers and other
staff with clinical or administrative
immunization-related responsibilities
(Live) online elearning lessons, with learning
management system tracking
VFC providers will adapt to elearning as a
training delivery method   Increased skill and
knowledge (pre- posttest comparison) about
vaccination among medical assistants other
office staff   Steady increase in the number of
MAs who take the elearning lessons
Fewer real-time person hours required to train
MAs   Fewer vaccine administration
errors   Higher coverage rates   Fewer missed
opportunities   Fewer VPDs   Increased provider/
user satisfaction
14
Primary data sourceEZ-IZ LMS
  • An LMS (Learning Management System) is software
    that automates the administration of training.
  • The LMS registers users, tracks courses in a
    curriculum, and records data from learners
  • The LMS provides reports to management.

14
15
Evaluation question 1
  • Q Did the Preparing Vaccines and Administering
    Vaccines elearning lessons launch by September 5,
    2008 (target)?
  • Indicator and method
  • The Preparing Vaccines and Administering Vaccines
    elearning lessons launch request IZ branch staff
    to access lessons and report any problems
    accessing the lessons
  • Data source feedback (verbal and email) from IZ
    Branch staff (and confirmation from LMS vendor)

15
16
Evaluation question 2
  • Q Did the learning management system (LMS) begin
    tracking usage and learner performance data by
    September 5, 2008 (target)?
  • Indicators and methods
  • LMS is operational verify with vendor that LMS
    is operational
  • LMS is gathering usage data obtain registration
    and login data from LMS report
  • LMS is gathering learner performance data obtain
    pretest, lesson start, lesson completion, and
    posttest data from LMS report

16
17
Data sourceregistrationdata
17
18
Learning History Page
18
19
Data sourceLMS
Pre-test
Lesson
Post-test
19
20
Evaluation question 3
  • Q How many provider sites have at least one
    person who has completed one lesson and passed
    the posttest?
  • Target and indicator
  • 5 of provider sites have at least one person who
    has completed one lesson and passed its posttest
    by March 31, 2009
  • Method
  • Generate LMS report of provider sites that shows
    lessons completed and posttest scores by provider
    site and user
  • Data source LMS

20
21
Data sourcepre- and posttests
21
22
Evaluation question 4
  • Q How many provider sites have at least one
    person who has completed one lesson and passed
    the posttest?
  • Target and indicator
  • 20 of provider sites have at least one person
    who has completed one lesson and passed its
    posttest by September 5, 2009 (one year after
    launch)
  • Method
  • Generate LMS report of provider sites that shows
    lessons completed and posttest scores by provider
    site and user
  • Data source LMS

22
23
Evaluation question 5
  • Q What feedback has been received from EZ-IZ
    users that could indicate potential obstacles to
    accessing and completing the elearning lessons?
  • Targets and indicators
  • 100 of emails and 100 of all other feedback
    that could indicate technical problems with the
    website
  • 100 of emails and 100 of all other feedback
    that could indicate technical or any other
    problems with the lessons
  • Method
  • Emails about technical problems and overall
    feedback about the EZIZ website will be
    automatically sent to a specific (ezig.org) MS
    Outlook folder phone calls will be logged and
    categorized by November 30, 2008
  • Data source emails submitted by users via links
    on eziz.org

23
24
Data sourcefeedback from users
24
25
Development of evaluation planChallenges
  • Very few challenges. IZ Branch Research
    Coordinator collaborated on and provided very
    valuable input into plan.
  • Due to limited resources, not able to build as
    robust an LMS as originally planned.
  • Proposed revisions to methodology during plan
    development required consulting with LMS vendor
    who then had to modify LMS database or do
    additional programming.

25
26
Lessons learned
  • Did not add question about MAs experience could
    not separate primary target audience from
    secondary target audience.
  • Learned that more data was needed than provided
    by LMS. Added question to QAR for field reps to
    administer.
  • The project team learned a lot during this entire
    pilot effort (including the evaluation plan) and
    is applying lessons learned during the current
    development of the next two lessons.

26
27
Added question to QAR
27
28
Questions?
28
29
Leslie HeydenHealth Educator Instructional
DesignerLeslie.Heyden_at_cdph.ca.gov(510)
620-3754www.eziz.org
29
30
This document
  • This document can be found on the CDC website at
  • http//www.cdc.gov/vaccines/programs/progeval/down
    loads/cali-eval-02-09.ppt
Write a Comment
User Comments (0)
About PowerShow.com