Elizabeth L. Shoenfelt, Ph.D.

1
ASSESSMENT of the CONTENT VALIDITYof the WATER
TRAINING INSTITUTE (WTI) CURRICULUM

• Elizabeth L. Shoenfelt, Ph.D.
• Alicia Turner, M.A. Candidate
• Patricia Slack, M.A. Candidate
• David Normansell, M.A. Candidate
• Department of Psychology
• Western Kentucky University

2
OVERVIEW

• Content Validation Training Program Evaluation
  Methods
• Criterion Phase
• Participants
• Procedure
• Content Phase
• Participants
• Procedure
• Analysis

3
Background WTI

• 2 year associate program
• 2 certification programs (Water and Wastewater)
• Curriculum general requirements and 2
  specialized tracks (Water and Wastewater)
• 4 projected entry level jobs that WTI graduates
  will most likely enter o Water Treatment
  Operator o Wastewater Treatment Operator o
  Distribution Systems Operator o Collection
  Systems Operator

4
Background NSF Grant

• Grant evaluation includes an assessment of the
  content validity of the WTI Curriculum
• Will examine the 4 courses currently in place to
  ensure they are preparing students with the
  Knowledge Skills and Abilities (KSAs) they need
  to successfully enter the workforce in any of the
  4 identified entry level jobs

5
Content Validity

• CONTENT VALIDITY The extent to which the
  material taught in the training course reflects
  the actual KSAs required for effective job
  performance
• The more similar the WTI training program content
  is to the job, the more effective it should be in
  preparing WTI program graduates.

6
Content Validation Training Evaluation Methods

• Matching Technique (Ford Wroten, 1984)
  Analyzed the effectiveness of a Police
  department training program by identifying
  Knowledge, Skills, and Abilities (KSAs)
  considered important to job performance and the
  KSAs that currently taught in the training
  program (i.e., time spent)
• Used a matching matrix to graph the data to
  identify Hits and Misses
• Hit training emphasis reflects training needs
  (i.e., high importance KSA receives high emphasis
  in program)
• Miss training emphasis does not reflect training
  needs
• o Deficiency high importance KSA receives low
  emphasis
• o Excess low importance KSA receives high
  emphasis

7
Matching Matrix
8
Content Validation Training Evaluation Methods

• Linking Technique (Teachout, Sergo, Ford, 1997)
  Used the matching matrix but also linked training
  emphasis to difficulty of learning
• More effective picture High Importance KSA with
  Low Emphasis is it a Deficiency or does it have
  Low Difficulty of Learning?

9
Current Evaluation

• Will examine the content validity of the WTI
  curriculum
• Will use combination of the Matching and Linking
  Techniques
• 2 Phases Criterion (Job) Content (WTI
  Curriculum)

10
Criterion Phase Procedure

• Developed preliminary Job Knowledge Surveys (JKS)
  based on materials received from the Associate
  Director of the Center for Water Resource Studies
  at WKU that included the KSAs needed for the 4
  entry level jobs
• For each element of job information, JKS included
  ratings for a) Time Spent on job
• b) Importance to job
• c) Difficulty of Learning
• d) When learned (before hire/formal trn/on the
  job)
• e) Should it be taught in WTI
• f) Is it needed for certification

11
Job Knowledge Survey Sample
12
Criterion Phase Participants

• WTI Steering Committee Members
• Knowledgeable individuals about Water and
  Wastewater Industry for both Kentucky and
  Tennessee.
• From a variety of organizations from Kentucky and
  Tennessee such as Kentucky Rural Water
  Association and Tennessee Association of Utility
  Districts
• JKSs were distributed to steering committee
  members during their WTI workshop/conference in
  Louisville, KY. The members pilot tested the
  surveys and provided feedback to refine surveys

13
Criterion Phase Procedure

• The JKSs were refined based on Steering Committee
  feedback
• Steering Committee members collectively
  identified 40 water and wastewater incumbents to
  serve as job content Subject Matter Experts
  (SMEs) to complete the JKSs(Approximately 10 per
  entry level job)
• The JKS were mailed to the Steering Committee
  Members for distribution to the identified job
  incumbent SMEs with a return deadline of October
  30, 2009
• The JKS data have been entered.
• We are now in the process of analyzing the JKS
  data.

14
Content Phase Participants

• Second Phase is a content analysis of the WTI
  Curriculum
• We will develop Course Content Surveys to
  determine KSAs taught and training emphasis
  (time)
• WTI Curriculum Content SMEs o Bowling Green
  Community College WTI Instructor o Teaching
  Assistant for the Center for Water Resource
  Studies at WKU o The 4 current WTI students
  also will complete the Course Content Surveys

15
Content Phase Procedure

• The SMEs will develop a list of KSAs that are
  taught in each of the current 4 courses through a
  brainstorming panel discussion
• At the end of the discussion, Curriculum SMEs
  will be provided with the course syllabi and any
  relevant training materials to further identify
  KSAs taught
• The KSAs identified will be used to develop a
  Course Content Survey. Each KSA taught in the WTI
  course will be rated on the time spent teaching
  the KSA.
• The SMEs (including students) will complete the
  survey

16
Analysis

• Matching Matrix will be used to compare
• Importance ratings vs. Time Spent Training
  ratings
• Difficulty of Learning ratings vs. Time Spent
  Training ratings
• Identify hits and misses in the courses
• Hits appropriate emphasis in curriculum
• Deficiencies where more emphasis is needed
• Excesses where time on topic can be devoted
  elsewhere

17
Matching Matrix
18
Formative Evaluation

• Data from the Content Validation can be used
• To revise current WTI courses (if needed) to
  ensure appropriate emphasis of job-related KSAs
• In developing other WTI courses to ensure they
  contain appropriate emphasis of job-related KSAs

19
References

• Primary References
• Bownas, D., Bosshardt, M., Donnelly, L. (1985).
  A Quantitative Approach to Evaluating Training
  Curriculum Content Sampling Adequacy. Personnel
  Psychology, 38. 117-131.
• Ford, J., Wroten, S.(1984). Introducing New
  Methods for Conducting Training Evaluation and
  for Linking Training Evaluation to Program
  Redesign. Personnel Psychology, 37. 651-665.
• Sproule, C. Rationale and Research Evidence
  Supporting the Use of Content Validation in
  Personnel Assessment. Retrieved from
  International Personnel Assessment Council
  Website http//www.ipacweb.org
• Teachout, M., Sego, D., Ford, J. (1997). An
  Integrated Approach to Summative Evaluation for
  Facilitating Training Course Improvement.
  Training Research Journal, 3. 169-184.

20
Questions?
21
Content Validity Ratio ((N imp N not-imp) / N)