HANDS

1
HANDS
Helping and Nurturing Developmental
Stages Systematic preparation for credentialing
in child care and education

• Program evaluation
• Presentation to the Virginia Assessment Group
• November 2008 Lynchburg, VA
• Susan Barnes

2
Commonwealth of Virginia Department of Social
Services Division of Licensing Programs

• Healthy Module

3
Career Lattice for Early Childhood Education

• Doctorate (Ph D or Ed D)
• Masters (M Ed)
• Bachelors (BA or BS)
• Associates (CDA or AA)
• Certificate (1 year)
• HS Diploma or GED

4
Why Professional Development?
5
Purpose of Evaluation

• To determine if the training program provides
  support to candidates to meet the competency
  goal.
• Competency 1.
• Candidate promotes good health and nutrition
  and provides an environment that contributes to
  the prevention of illness.

6
Levels of Evaluation
7
Stakeholders

• Commonwealth of Virginia
• Council on Child Day Care and Early Childhood
  Programs
• Centers/Family Care Homes/Preschools
• Teachers/Directors
• Parents/Families
• Children

8
Audiences

• Trainer
• Success by Six Coalition of the Shenandoah Valley
• Valley Association for the Education of Young
  Children

9
Evaluation Design and Data Collection Methods

• Review curriculum materials and archival
  information
• Confirm alignment with current standards
• Observe instruction
• Pre- and post-test content
• Survey participants
• Interview trainer
• Conduct focus group of previous trainees

10
Evaluation Questions

• Do the training materials address the content of
  the objectives?
• Does the training process give candidates
  opportunities to practice new skills, thus
  facilitating transfer?
• Is the training a value added experience?
• Does training impact behavior or attitudes on the
  job?
• Does the training benefit stakeholders?

11
Data Collection Methods

• Curriculum Review
• Use a checklist of criteria to review materials
  for
• Accuracy of content
• Clarity of writing
• Depth of coverage of material
• Use of recent research
• Alignment with current standards

12
Data Collection Methods

• Formal Observation
• of Training
• Use time sample and anecdotal records to
  document
• Safe environment for learning
• Level of trainee engagement
• Various teaching strategies
• Opportunities for practice of new skills
• Checks for understanding

13
Data Collection Methods

• Pre- and Post-tests
• Use multiple choice test to
• Assess previous content knowledge (grows and
  glows)
• Check for comprehension

14
How much does someone from the general population
know without taking the course?

• Which of these common beverages has the most
  sugar?
• 1 cup of orange Sunny Delight
• 1 cup of whole chocolate milk
• 1 cup of regular Coke or Pepsi

15
Answer

• Which of these common beverages has the most
  sugar?
• 1 cup of Sunny Delight 45 grams
• 1 cup of chocolate milk 24 grams
• 1 cup of regular cola 38 grams

16
How much does someone from the general population
know without taking the course?

• Which of these common snacks has the most sugar?
• 3 Oreo cookies
• 1 cup Fruit Loops
• 1 pouch fruit gushers

17
Answer

• Which of these common snacks has the most sugar?
• 3 Oreo cookies 14 grams
• 1 cup Fruit Loops 15 grams
• 1 pouch fruit gushers 13 grams

18
How much does someone from the general population
know without taking the course?

• What does the rainbow have to do with nutrition?

19
Answer

• What does the rainbow have to do with nutrition?
• Selecting foods of many different colors helps to
  create a vitamin-rich diet.

20
Data Collection Methods

• Focus Groups
• Use a focus group to
• Compare currently credentialed providers to those
  seeking credential
• Assess application of content over time
• Get suggestions for training improvement

21
Data Collection Methods

• Survey
• Used a survey to look for
• Transfer of skills or knowledge from the
  classroom to the workplace
• Change in behavior or attitude
• Retention of content knowledge over time (about 2
  months)

22
Data Collection Methods

• Interview
• Interview trainer to assess
• Accuracy of content representation
• Level of preparation for teaching
• Comparability to other groups

23
Measures of Success

• Curriculum review and observation reveal close
  alignment to objectives of training
• Post tests show improved scores
• Surveys show high retention and recognition of
  benefits
• Interview and focus group feedback confirms and
  replicates other results

24
Sampling Procedures

• All participants (17) offered opportunity for
  pre- and post-test and asked to take survey
• Former participants (3) interviewed in a focus
  group were from different sessions
• Trainer interviewed (1)
• Curriculum development project director (1 of 2)
  interviewed

25
Issues Surrounding this Sample

• All trainees are female
• Little diversity one ESL, one African-American
• Various levels of background experience prior to
  class

26
Threats to Validity

• Small sample size
• Bias due to self selection
• Lack of confidence in anonymity
• Pre- and Post-test items do not measure the same
  content exactly. Some items are not
  well-written.

27
Timetable

• Week 1 Contact trainer
• Weeks 2-4 Curriculum and archive review
• Week 3 Observe training
• Week 3 Initial interview with trainer
• Week 11 Follow up survey
• Week 11 Interview curriculum development project
  director
• Week 12 Conduct focus group
• Week 13 Present initial findings to trainer
• Week 18 Present findings to other stakeholders

28
Data Analysis

• Curriculum Review of Materials
• Web page of association granting credential
• Hard copies of texts used
• Additional handouts
• Interview curriculum developers

29
Results

• Good alignment with goals of credentialing
  program for which participants are preparing.
• Instructor brought many additional materials to
  supplement the course text.

30
Data Analysis

• Formal Observation of Training
• Safe environment for learning
• Content
• Schedule
• Methods of instruction
• Trainee engagement
• Opportunity for practice
• Assessment of understanding through various means

31
Results

• The instructor followed a schedule that allowed
  for frequent opportunities for participants to
  move around, interact with each other, think
  critically about the content, and apply new
  skills.
• Many assessment strategies used.

32
Data AnalysisPre- Post-tests
33
Results

• Every participant who completed the pre- and
  post-test either retained the same score or
  improve their score.
• Those who remained the same only missed one or
  two questions.

34
Data Analysis

• Trainer Interview Questions
• Do you deliver the curriculum as written?
• Is the method of delivery similar to past
  training sessions?
• Is this a typical class size?
• Are these participants representative of most
  trainees as far as their level of participation?
• Do these trainees have experience with the
  content before your class?

35
Results

• Trainers are highly qualified and experienced
  professionals who have received train-the-trainer
  instruction.
• These participants are typical of most groups.
• Participants come with a wide variety of previous
  experiences.
• Participants have high expectations.

36
Data AnalysisSurvey
37
Results

• Respondents felt that they, their programs, and
  the children benefited from the training.
• Most perceived benefits are related to the
  effectiveness of their teaching.

38
Data Analysis

• Focus Groups Questions
• How long ago did you complete the HANDS course?
• Do you use any of the skills or information from
  that course?
• What was the benefit of the HANDS course?
• Would you recommend this course to someone else?

39
Results

• The HANDS course prepared them completely for the
  credentialing processes.
• Great information that they use all the time.
• The course and process has been updated several
  times.

40
Summary

• Does the training address the objectives? YES
• Does the training facilitate transfer? YES
• Is the training value added? YES
• Does training impact behavior or attitudes on the
  job? YES
• Does the training benefit stakeholders? YES

41
Questions
42
Thank you