Performance and outcome measurement I: Epidemiological and statistical concepts

1
Performance and outcome measurement
IEpidemiological and statistical concepts

• 1999 AMCHP skill building session
• Michael A. Stoto, PhD
• The George Washington University
• School of Public Health Health Services

2
Outline of presentation

• Measurement theory and methods
• Epidemiological methods
• surveillance and public health assessment
• rates and proportions
• Statistical concepts
• population and measurement variability
• sampling theory and survey methods
• confidence intervals
• Binomial and Normal distributions
• simple regression analysis
• Practical examples
• validity and reliability
• promoting successful birth outcomes

3
Measurement theory methods

• Steps for developing measures
• 1. Clarify the purpose of measurement
• 2. Identify the concepts to be measured
• 3. Identify specific indicators of these concepts
• 4. Assess validity, reliability, responsiveness
  to change
• will discuss this in detail later

4
1. Clarify the purpose

• Establish accountability for Title V funds
• expenditures
• performance measures
• impact on outcomes
• Improve processes and health outcomes

5
2. Identify the concepts

• Identify responsibility and accountability for
  performance
• Evidence-based link between performance and
  health
• Balance between short- and long-range goals
• Balance among levels and types of service

6
Levels and types of service
7
3. Identify specific indicators

• Timely availability of data at a reasonable cost
• Explicit definitions
• Inclusion in other indicator sets

8
Concepts vs. indicators

• Concept
• mortality
• presence of disease
• health risks
• costs
• quality
• access

• Operational form
• disease specific mortality rate
• disease prevalence rate
• risk factor prevalence rate
• treatment costs per patient
• patient satisfaction ratings
• percent of population with health insurance

9
Concepts vs. indicators(capacity measures)

• SSI with CSHCN
• Percent of state SSI beneficiaries lt 16 years old
  receiving services from the state Children with
  Special Health Care Needs program
• Medical home
• Percent of CSHCN who have a medical/health home

10
Concepts vs. indicators(process measures)

• Medicaid children
• Percent of potentially Medicaid eligible children
  who have received a service paid by the Medicaid
  program
• Lead screening
• Proportion of children aged 6 mo. to 5 years
  screened for excess blood lead

11
Concepts vs. indicators(outputs, intermediate
outcomes)

• Prenatal care
• Percent of infants born to pregnant women
  receiving prenatal care beginning in the first
  trimester
• Newborn screening
• Percent of newborns with at least one screening
  for each of PKU, hypothyroidism, galactosemia,
  hemoglobinopathies
• Vaccine coverage
• Percent of children though age 2 who have
  completed MMR, DPT, polio, Hib, and hepatitis B
  immunizations

12
Concepts vs. indicators(health outcomes)

• Infant mortality
• Number of infant deaths (lt 1 year) divided by
  number of live births (per 1,000)
• Low birth weight
• Percent of live births lt 1500 grams at birth
• Teen pregnancy
• Birth rate (per 1,000) for teenagers aged 15
  through 17 years
• School success
• Average reading scale scores of 9 year olds

13
Sources of data for performance measurement

• Vital statistics
• maternal and infant mortality
• birth certificate information
• birth/death match
• Examples
• prenatal care
• low birthweight births
• low birthweight children born at level iii
  facilities
• teenage fertility rate
• motor vehicle crash deaths
• suicide deaths among youths

14
Birth certificate information

• Location information
• state, county, and place of residence
• place of birth and attendance
• Infant information
• race, sex, gestation, birthweight, Apgar
• Mother age, race/Hispanic, education, marital
  status
• Father age, race/Hispanic
• Prenatal care month began, number of visits
• Medical/health data
• tobacco alcohol use, weight gain during pregnancy
• medical risks, complications of labor

15
Sources of data for performance measurement

• Surveillance data
• Special purpose surveys
• Examples
• immunization coverage
• dental sealants
• breastfeeding at hospital discharge
• hearing screening before hospital discharge
• children with health insurance

16
Sources of data for performance measurement

• Program data
• Title V, other MCH, other programs
• Examples
• CSHCN with
• medical/health home
• insurance for primary and specialty care
• Medicaid services to Medicaid eligible
• Newborn screening

17
Epidemiological methods

• Surveillance and public health assessment
• data sources
• Mortality and morbidity analyses
• rates and proportions

18
Surveillance

• Purposes
• identification of emerging health problems
• identification of affected individuals
• continued watchfulness over disease in the
  population
• Types of surveillance
• active
• passive
• sentinel health events

19
Surveillance data sources

• Vital statistics (mortality and fertility)
• Disease control programs
• case reporting/surveillance systems
• case finding for selected populations
• Administrative data
• employers and schools
• absenteeism, periodic physical exams
• health plans, hospitals, pharmacies
• administrative data
• provider and patient surveys

20
Surveillance data sources

• Prevalence surveys
• Purpose to provide information on the frequency
  of disease in a population
• Types of cross-sectional surveys
• Interview National Health Interview Survey
  (NHIS)
• Examination National Health and Nutrition
  Examination Survey (NHANES)
• Health records National Hospital Discharge Survey

21
Public health assessment

• Definition
• The regular collection, analysis,
  interpretation, and communication of information
  about health conditions, risks, and assets in a
  community (IOM, 1988)
• Other assessment data
• census and general population surveys
• administrative data from health sector
• also social services, housing, education,
  transportation, etc.

22
Epidemiologic analyses

• Time e.g. seasonality
• Place spot and area maps
• Time and space clusters
• Person
• age, gender
• race and ethnicity
• genetic background
• social class and SES

23
Mortality analyses

• Mortality rates
• defined population group -- denominator
• time period
• number of deaths -- numerator
• in that population during that time period
• Crude death rate -- all ages and causes
• Age- and sex-specific death rates
• restrict numerator and denominator
• Cause-specific death rates
• restrict numerator

24
Mortality analyses

• Age adjustment
• direct
• indirect
• Survival/life table measures
• life expectancy
• Other mortality measures
• case fatality rate
• proportionate mortality rate/ratio (PMR)

25
Measures of disease frequency

• Incidence rate
• of new cases in a specified time period
• of persons at risk of developing the disease
• Prevalence rate
• of cases present in a specified time period
• of persons at risk of having the disease
• Point prevalence number of cases at a specified
  moment
• Period prevalence number of cases that occur
  during a specified period

26
Measures of disease frequency

• Proportion ratio with
• denominator of individuals in group
• numerator of these with specified
  characteristics
• Rates that are really proportions
• attack rate proportion of individuals in some
  group that get a specified disease
• case fatality rate proportion of individuals
  with a specified disease who die of that disease

27
Statistical concepts

• Unit of analysis
• Level of measurement
• Population and measurement variability
• Binomial and Normal distributions
• Confidence intervals
• Sampling theory and survey methods

28
Unit of analysis

• Individual
• Defined population
• nation, county, community, program
  participants, insured population, etc.
• Institution
• hospital, health plan, clinic, provider group,
  WIC program, etc.

29
Level of measurement

• Nominal
• categories without order
• Ordinal
• categories with order
• Interval
• continuous without natural zero
• Ratio
• continuous with natural zero

30
Population (subject) variability and measurement
error

• Subject variability Measurement error
• Outside temperature
• Daily variation Thermometer error
• Body temperature
• Interpersonal variation Thermometer error
• Body weight
• Interpersonal variation Scale error
• Self-reporting error

31
Normal/Gaussian distribution

• Many continuous variables are approximately
  Normally distributed
• but not all!
• subject variability and measurement error
• Normal distributions can be fully characterized
  by mean (?) and standard deviation (?)
• variance (?2) (standard deviation) 2

32
Normal/Gaussian distribution

• In a Normal distribution
• about 2/3 of subjects are within 1 ? of ?
• about 95 of subjects are within 2 ? of ?

33
Normal/Gaussian distribution

• Example Distribution of birthweight (X)
• ? 4000 g, ? 1000 g
• What is probability that X gt 5000g?
• Z (X - ?)/ ? (5000 - 4000)/1000 1
• Prob(Zgt1) 1/2 1/3 1/6
• What is probability that X lt 2000g?
• Z (X - ?)/ ? (2000 - 4000)/1000 -2
• Prob(Zgt-2) 1/2 0.05 0.025

34
Sampling theory

• Simple random sample
• each element in population has same, and
  independent, chance of being in a sample
• Sample average ?xi / n
• Central limit theorem
• has a normal distribution with
• mean ? and standard deviation ? / ?n

35
Sampling theory

• Sample size n 1 and 10

?
36
Sampling theory

• Sample size n 1, 10, and 100

?
37
Sampling theory

• Example Average birthweight ( )
• standard deviation of ? / ?n
• n ?n ? / ?n
• 1 1 1000/1 1000
• 10 3.16 1000/3.16 316
• 100 10 1000/10 100

38
Confidence intervals

• Basic idea
• is in the interval ? 1.96 ?/?n in 95 of
  samples
• 1.96 ?/?n covers ? 95 of the time, and
  is called a 95 confidence interval
• must use a t-value greater than 1.96 when ? is
  estimated from the data
• e.g. t 2.6 when n 5

39
Confidence intervals

• Example Average birthweight ( )
• Sample values 3000, 5500, 2500, 4600
• 15,600/4 3900
• ?/?n 1000 / ?4 500
• 95 Confidence interval (n 4)
• 1.96 ?/?n 3900 1.96(500)
• 3900 980 (2920, 4880)
• 95 Confidence interval (if n 25, 3900)
• 1.96 ?/?n 3900 1.96(1000)/ ?25
• 3900 392 (3508, 4292)

40
Statistics for proportions

• Many performance measures are in the form of
  proportions, i.e. a ratio with
• of individuals with specified characteristics
• p
• of individuals in group
• Counts have a Binomial distribution if
• fixed n of binary outcomes
• independent outcomes with same probability ?
• As n increases, distribution of p is
  approximately Normal with
• ? n? and ? ?(1-?)/n

41
Statistics for proportions

• Example standard deviation of p

?
n
42
Statistics for proportions

• n 10, ? .3

43
Statistics for proportions

• n 50, ? .3

44
Regression analysis

• What is the relationship between Y and X?
• If a straight line, can be expressed as Y a
  bX
• What is the slope, i.e. the increase in Y for a
  unit increase in X?
• Is the slope significantly different than 0?
• Since Y and Y are never exactly linear, use
• Y a bX e, where e has a Gaussian
  distribution with ? 0, and e are independent
• Choose a and b to minimize squared deviations
  ?(y - a - bx)2

45
Regression analysis
46
Regression analysis
47
Regression analysis

• Example IMR trend analysis

48
Regression analysis

• Example IMR trend analysis

49
Practical examples

• Assessing proposed measures
• Validity and reliability tradeoffs
• Differentials and disparities
• Successful birth outcomes

50
4. Assess the proposed measures

• Validity
• is the indicator measuring the right concept?
• Reliability
• is the indicator consistently measuring the
  concept?
• Is measurement error small compared to population
  variability?
• Robustness and responsiveness to change
• will the indicator change if and only if the
  concept being measured changes?

51
Reliability and validity
Reliable
Valid
Reliable and valid
52
Increasing reliability

• Reduce error variance
• increase number of observers
• increase number of observations
• observer training
• improve scales
• Enhance true variance
• improve scales

53
Reliability and validity

• Performance measures are not the same as practice
  recommendations
• example prenatal care in first trimester rather
  than by PHS recommendations
• trade validity for availability

54
Reliability and validity

• Use running averages and statistical smoothing
• example average infant mortality rate over 3 or
  5 years
• trade timeliness for reliability

55
Reliability and validity

• Use proxy measures that reflect trends and
  differences
• example low birth weight rather than infant
  mortality
• trade validity for reliability

56
Differentials and disparities

• Increasing national concern about racial, ethnic
  and other differentials in health outcomes
• Healthy People 2010 focus on eliminating
  disparities
• Use parallel indicators for different groups (not
  disparity measures)
• Use appropriate independent variables to
  understand problem and solutions

57
Promoting successful birth outcomes

• Example based on Access to Health Care in America
  (IOM, 1993)
• One of five objectives Promoting successful
  birth outcomes
• Combination of 4 utilization and outcome
  indicators

58
Promoting successful birth outcomes

• Indicator 1 (utilization measure)
• Percentage of pregnant women obtaining adequate
  prenatal care
• Strength direct measure of access
• Weaknesses
• Measured by initiation and frequency
• neither alone sufficiently measures adequacy
• nothing on distribution of visits, content,
  continuity
• large number could represent a problem
• more complex indices can be confounded by missing
  or incomplete data
• recall problems

59
Promoting successful birth outcomes

• Indicator 2 (outcome measure)
• Infant mortality rate
• Strengths
• commonly used measure of access
• available everywhere from vital statistics
• Weaknesses
• provides little information about access barriers
• rate includes causes of death that cannot be
  affected by the health care system
• high variability in infant deaths in some areas

60
Promoting successful birth outcomes

• Indicator 3 (outcome measure)
• Low birthweight rate
• proportion of infants weighing less than 2500 g
• Specific to adequate prenatal care and access to
  nutrition services
• Strengths
• important predictor of infant survival
• numerator not as rare, so more stable
• Weakness
• timeliness of published data

61
Promoting successful birth outcomes

• Indicator 4 (outcome measure)
• Congenital syphilis rate
• Strength
• reportable condition in most states
• very specific to lack of or inadequate prenatal
  care
• Weaknesses
• reporting may be incomplete
• syphilis is rare in most states

62
Conclusions

• Performance measurement can be an effective tool
  for accountability and learning organizations
• Depends on availability of valid and reliable
  indicators
• Need a limited yet comprehensive set of coherent
  and significant indicators
• which can be monitored over time and
• disaggregated to relevant social units

63
References

• IOM/NAS reports (www.nap.edu)
• Also www2.nas.edu/hpdp and www2.nas.edu/bocyf
• Improving Health in the Community (1997)
• Access to Health Care in America (1993)
• America's Children Health Insurance and Access
  to Care (1998)
• Systems of Accountability Implementing
  Children's Health Insurance Programs (1998)
• Paying Attention to Children in a Changing Health
  Care System (1996)

64
References

• Additional IOM/NAS reports
• Reducing the Odds Preventing Perinatal
  Transmission of HIV in the United States (1999)
• From Generation to Generation The Health and
  Well-Being of Children in Immigrant Families
  (1998)
• The Best Intentions Unintended Pregnancy and the
  Well-being of Children and Families (1995)
• Overcoming Barriers to Immunization (1994)
• Prenatal Care (1988)
• Preventing Low Birthweight (1985)