Donna Rickert, M'A', Dr'P'H'

1
Counting the Shots
A Model for Immunization Assessment and Referral
in Non-Medical Settings
Donna Rickert, M.A., Dr.P.H. Abigail Shefer,
M.D. Lance Rodewald, M.D. National Immunization
Program Immunization Services Division August,
2002
2
The task of the public health agency has been
not only to define objectives for the health care
system but also to find means to implement
health care goals within a social structure.
Institute of Medicine, The Future of Public Health
3
Special Challenges

• Over the past decade, major changes in our
  national health and welfare delivery systems have
  presented special challenges for the
  WIC-Immunization initiative
• State Medicaid agencies have increasingly
  relied on private managed care organizations to
  provide services to indigent children
• The service delivery role of local public
  health agencies has been weakened
• WICs direct link to the health care system has
  been weakened

4
Special Challenges

• As fewer public health departments provide
  services to women and children
• public health officials are turning to WIC to
  help address the needs of low income children
• WIC has become the virtual gateway to health
  services for low-income preschool children such
  as those related to overweight and obesity,
  anemia, HIV, and blood lead level screening in
  addition to immunization

5
Special Challenges

• In addition, WIC Programs
• Are seeing significant health and demographic
  changes in the low-income populations that it
  serves
• Are facing difficulties in recruiting and
  keeping skilled staff
• Are having difficulties in funding information
  technology to manage program operations while
  simultaneously enhancing service delivery

6
The White House Memorandum, December, 2000

• Addressed socioeconomic disparities in
  pediatric immunization coverage levels
• Directed Secretaries of DHHS and Agriculture
  to work together to
• make IZ screening and referral a standard part
  of WIC certification
• develop a national strategic plan to improve IZ
  coverage rates in WIC children
• report back on progress

7
The Final WIC Policy Memorandum, August, 2001

• Sets minimum guidelines for IZ screening and
  referral
• Requires documented immunization history
• Applies to infants and children under age 2
  years
• Screening and referral to be done at
  certification visits
• WIC clinic staff will count DTaP
  immunizations only
• ? 1 dose of DTaP by age 3 months
• ? 2 doses of DTaP by age 5 months
• ? 3 doses of DTaP by age 7 months
• ? 4 doses of DTaP by age 19 months

8
Diagram of the process described in the WIC
policy memorandum
9
Rationale
In studies using NIS data, the providers portion
of the record is considered the more accurate
index of the childs true immunization status.
The household portion of the record is assumed to
be a less accurate representation. In most WIC
clinic settings, only the household-based record
will be available. For this reason, our aim was
to see how closely the household-based record
approximates the provider-based record.
10
Purpose of this analysis

• Primary objective
• To estimate the percentage of WIC children who,
  by the new WIC minimum assessment criteria, will
  be correctly classified as either up-to-date
  (UTD) or not for the universal 4313 pediatric
  vaccination series
• To see how this compares with the percentage
  that would be correctly classified if all 4
  antigens were assessed

11
Method
Data Source and Statistics
We used approximately 6,000 household and
provider records from the 2000 National
Immunization Survey (NIS) to calculate the
sensitivity, specificity, and test efficiency of
using DtaP shots as a predictor of UTD status for
the universal 4313 pediatric immunization
series.
12
Method
Sample size and inclusion criteria
Total NIS 2000 Survey Records 34,087
Ever enrolled in WIC 17,451 (51 of NIS 2000)
Shot card used for household reporting 8,617
(25 of NIS 2000)
Adequate provider data for verification 6,277
(18 of NIS 2000)
13
Method Definitions
Up-to-Date by the 4313 ACIP schedule is
defined in terms of the age-specific vaccination
schedule for the following 4 vaccines
4 doses of DTaP or DTP at 2, 4, 6, and 15 - 18
months 3 doses of Polio at 2, 4, and 6 - 18
months 1 dose of MMR or MCV at 12 - 15 months 3
doses of Hib at 2, 4, 6, and 12 - 15 months
14
Method Variable Specification

• NIS Household Survey
• childs age at each vaccination
• NIS Provider survey
• childs UTD status for each vaccine at 3, 5,
  7, 13, 19, and 24 months
• We assigned bivariate UTD status variables
• Household DTaP
• Household 4313
• Provider 4313

15
Method Definitions

• Let test refer to any measure used to make a
  decision about the true status when information
  about the true status is incomplete.
• test 1 the household-reported DTaP count
• test 2 the household-reported 4313 count
• We compared test 1 and test 2 to see if they
  differed in their ability to predict the childs
  true 4313 UTD true status as shown in the
  provider record.

16
Method Definitions
Sensitivity of a test measures the percentage of
truly underimmunized children who are correctly
identified as such by the test. Specificity of a
test measures the percentage of truly UTD
children who are correctly identified as such by
the test. Test efficiency measures the total
percentage of times the test gives the correct
answer, relative to all times the test is given.
17
Method Statistical Analyses
SAS To develop the enhanced database SUDAAN To
conduct the weighted crosstabulation procedures
for the sensitivity, specificity and test
efficiency statistics.
18
Research Questions
Socioeconomic Disparities
What is the estimated nationwide difference in
immunization coverage rates between WIC vs
non-WIC children under age 2?
19
Differences in Immunization Completion Rates, WIC
vs Non-WIC Children, 2000
20
Research Question
Comparative Sensitivity
How good is the household-based record of UTD
status for DTaP at identifying children who, by
the provider-based record, are underimmunized?
How does this compare with the household-based
count of all 4 antigens?
21
Results Comparative Sensitivity
22
Research Question
Comparative Specificity
How good is the household-based record of UTD
status for DTaP at identifying children who, by
the provider-based record, are truly UTD for the
complete 4313 series? How does this compare
with the household-based count of all 4 antigens?
23
Results Comparative Specificity
24
Research Question
Comparative Test Efficiency
How often does the Household DTaP count
accurately reflect the provider-based overall
4313 UTD status? How often does the Household
count of all 4 antigens accurately reflect the
provider-based 4313 UTD status?
25
Comparative Test Efficiency
26
Summary
The NIS 2000 data confirm that children under
age 2 who have ever been enrolled in WIC are
significantly more likely to be underimmunized
than those who have never been enrolled. The
overall disparity is 7.
27
Summary

• The household DTaP count and the household
  4313 count are both imperfect predictors of
  true 4313 UTD status.
• If the DTaP count alone is used, 70 of
  underimmunized children, on average, will be
  identified.
• If all 4 antigens documented in the household
  record are counted, 77 of underimmunized
  children, on average, will be identified
• Therefore the count of all 4 antigens is a more
  sensitive predictor in that it is slightly better
  (7) at identifying truly underimmunized
  children. However

28
Summary

• If the DTaP count is used, 86 of truly UTD
  children, on average, will be identified as such.
• If all 4 antigens from the household record are
  counted, 82 of truly UTD children, on average
  will be identified as such.
• Therefore the DTaP count is a more specific
  predictor of true UTD status in that it is
  slightly better (4) at identifying truly UTD
  children. Moreover,
• Overall test efficiency is slightly higher (1
  on average) for the DTaP count, making it a more
  efficient predictor of true immunization status.

29
Conclusions
Use of the DTaP count as a predictor of true
4313 UTD Status has specific advantages

• Training non-medical staff to assess completion
  status for 1 vaccination is simpler than
  training them to assess it for 4 or 5
• The immunization assessment process should go
  more quickly and is therefore more efficient in
  terms the demands on staff time
• The likelihood that non-medical staff will make
  an error is smaller when only one vaccine is
  assessed.

30
Conclusions
Advantages (continued)

• The higher specificity of the DTaP count means
  that children are less likely to be
  inappropriately referred to an immunization
  provider
• If the gains in simplicity and efficiency of
  assessment result in more clinics being
  able/willing to evaluate childrens immunization
  status, this may offset the loss of sensitivity,
  and

31
Conclusions
Advantages (continued)

• Success in this initiative will likely strengthen
  the partnership, which will
• Set the stage for assisting WIC in developing
  achievable objectives toward improving
  immunization coverage in WIC children, as well as
• Serve as a model for developing immunization
  linking initiatives with other federal/state
  programs, such as Housing and Urban Development,
  Medicaid and TANF

32
Contact Information e-mail djr7_at_cdc.gov phone
[phone]