Monitoring and evaluation, patient surveillance, and loss to followup

1
Monitoring and evaluation, patient surveillance,
and loss to follow-up

• Constantin T. Yiannoutsos1,2, Ming-Wen An3,
  Beverly S. Musick1 and Constantine Frangakis3
• ______________________
• 1Indiana University School of Medicine,
  Indianapolis, IN, USA
• 2Moi University School of Public Health, Eldoret,
  Kenya
• 3Johns Hopkins Bloomberg School of Public Health,
  Baltimore, MD, USA

2
BackgroundThe PEPFAR program

• In 2006 there were almost forty million people
  around the world living with HIV/AIDS. Almost 25
  million of these live in sub-Saharan Africa.
• In May 2003, the U.S. President announced a
  global program, known as the United States
  President's Emergency Plan for AIDS Relief
  (PEPFAR), to address the epidemic, primarily in
  Africa.
• The U.S. Congress passed the United States
  Leadership Against HIV/AIDS, Tuberculosis, and
  Malaria Act of 2003. Under this Act, USD 15
  billion were allocated over five years (2004
  2008).
• The Act calls for an evaluation of the PEPFAR
  program, which screens, treats and follows HIV
  patients over time.

3
Monitoring and Evaluation of PEPFAR

• A key component of the evaluation of the impact
  of PEPFAR-supported programs on the HIV epidemic
  is the estimation of patient survival.
• A major obstacle for their appropriate monitoring
  and evaluation is patient loss to follow-up
  (Braitstein et al., 2006).
• There is evidence that standard analytic methods
  produce biased results since individuals lost to
  follow-up (dropouts) are generally sicker than
  those who stay in the study (Wu, 2007, Touloumi
  et al., 2002).
• Thus, estimates derived from passively monitored
  program may seriously underestimate patient
  mortality (Braitstein et al., 2006) even after
  adjusting for covariates measured prior to
  dropout.
• Under such circumstances, the problem cannot be
  addressed simply by analyses of the observed
  (non-dropout) patients, but rather, modifications
  are needed in the design.

4
Case in point

• Estimates of mortality produced by active and
  passive-surveillance programs are dramatically
  different (Braitstein, et al., Lancet, 2006).

5
Double sampling

• A design-based method known as double-sampling",
  first introduced in survey research (Neyman,
  JASA,1938), aims to address this issue by
  allocating resources to intensively pursue and
  find a sample of observed dropouts. Baker, Wax,
  and Patterson (Biometrics, 1993) and Frangakis
  and Rubin (Biometrics, 2001 FR01) both addressed
  analysis of double-sampling in the context of
  survival.
• In particular, FR01 showed that a bias arises
  when standard double-sampling methods are used
  with survival data and derived the empirical
  maximum likelihood estimator based on minimal
  data requirements.

6
Active versus passive follow-up
Passive follow-up
Active follow-up
7
Objectives of this study

• Goal
• Estimate survival in entire cohort of patients
• Possible Approaches
• Ignore LFU
• Double-sample a subset of the LFU patients
• Pool observed non-dropout and sampled dropout
  data
• Produce weighted combinations of survival
  estimates
• Statistically adjust (extended Frangakis Rubin
  approach)

8
Review of Frangakis Rubin (FR01)Definitions

• Observed versus potential data
• Observed data are those that we actually measure
  in our study design.
• Potential data (Neyman, 1923 Rubin, J Educ
  Psych, 1974) are those that could be potentially
  observed under different designs.
• ____________Translated in Stat Sci, 1990.

9
Potential data

• Ti survival time
• Ri potential dropout status if the study were
  to continue indefinitely under passive follow-up
  (Ri1 if non-dropout)
• Li potential time to dropout if the study were
  to continue indefinitely under passive follow-up
  (LiltTi for Ri0 LiTi for Ri1)

10
Observed data

• Ei entry date
• Emax study end date (common across individuals
  i)
• Ci time between study entry and end, i.e.
  administrative censoring time (Emax Ei )
• Di vector of covariates
• Riobs 1-(1-Ri)?I(LiltCi), observed dropout
  status (Riobs1 if observed non-dropout)
• Si indicator for being selected and found with
  double-sampling (if Riobs0)
• Xi min(Ti ,Ci)
• ?i indicator for whether survival time is
  shorter than the administrative censoring time

11
Observed versus potential data
Earliest
Study End Date
Entry date
(
E
)
max
individual
X
a
O
b
X
12
Main result from FR01

• The main result from Frangakis Rubin is that
  one can make valid inferences by working with the
  reduced data
• involving only data on observed dropout status
  and survival data.

13
The FR01 likelihood
14
The FR01 estimator

• From the likelihood above, Frangakis Rubin
  derived the following estimator for the hazard
  function ?(t) is equal to
• where wg(t) is
• and
  is the probability of being at risk at t,
  pgPr(Robsg) and g0,1.

15
Extending FR01 by including covariates

• Adding covariates to the FR01 methodology is
  feasible if two assumptions hold
• A1. Conditional on the set of observed covariates
  Zi , the time from entry to end of study Ci is
  independent of survival time Ti and potential
  dropout status Ri , i.e.,
• A2. Among observed dropouts, the observed
  covariates Zi include the variables used to
  decide whom to double-sample. We can express this
  by stating that after we condition on Zi ,
  selection for double-sampling is independent of
  survival and entry times, i.e.,

16
Likelihood in the presence of covariates

• The FR01 reduced data is still
• and the likelihood to be maximized becomes

17
The MLE of S(t )Net versus crude hazard functions

• Because of the independence of administrative
  censoring and survival times conditional on the
  covariates (assumption A1) , the net hazard
  within a covariate stratum z , defined as
• is equal to the crude hazard defined as

18
The MLE of S(t )Weighted estimate of the crude
hazard

• The crude hazard function in stratum
  z can be re-written as the weighted average
• where is the crude hazard within
  Robsg, Zz, for g0,1 and the weights
• wgZ(t) ? Pr(RobsgX? t, Zz)
• are the proportions of individuals with observed
  dropout status g within the risk set X? t and
  stratum Z.

19
The non-parametric MLE of S(t )

• These relations and arguments analogous to FR01
  imply that the non-parametric MLE of S( t ) is
• where are the observed proportions for
  strata z,
• 
  and
• is the Nelson estimator of
  and is the empirical estimator of
  based on .

20
ResultsThe cohort

• We have generated estimates of one-year mortality
  in a cohort of 8,977 patients enrolled in the
  study between January 1, 2005 and January 31,
  2007, of whom 3,624 (40) were lost over a period
  of two years.
• Out of the 3,624 dropouts, 1,143 (31.5) were
  pursued and 621 (54.3, 17 overall) were
  double-sampled.
• There were 230 total deaths, out of which, 124
  (53.9) were ascertained through routine
  (passive) follow-up and 106 (46.1) by outreach
  (active follow-up).

21
Stratification on factors predictive of first
discontinuation from follow-up

• We stratified on factors that are predictive of
  discontinuation from follow-up in an attempt to
  make independence of such discontinuation and
  survival times within strata more plausible.
• We used a Cox proportional hazards model to
  identify factors predictive of such
  discontinuation. Then we took the linear
  predictors from this model as a
  continuous score.
• We then categorized this linear score into
  quintiles with higher quintiles corresponding to
  increasing hazard of discontinuation.
• We included the following as predictive factors
  gender, baseline CD4 count, baseline WHO stage,
  indicator of urban versus rural clinic, and ART
  start status.

22
Baseline factors related to patient dropout
23
Comparison of alternative methods

• We compared four methods of estimation
• No double sampling (based on only observed
  deaths) Method 1
• Pooling all deaths without distinction of passive
  or active follow-up (double-sampling) Method 2
• Stratified Kaplan-Meier (SKM) Method 3
• Double-sampling method

24
One-year mortality estimates
25
Survival estimates
26
Discussion

• Estimates of one-year mortality were dramatically
  different when data from active follow-up are
  considered
• Naïve reporting of observed deaths (comprising
  most of the routine reporting of mortality among
  most programs) Method 1 vastly underestimates
  the truth.

27
Discussion (continued)

• Pooling death information without concern of its
  source (i.e., without differentiating between
  active and passive follow-up) Method 2 also
  underestimates reality.
• Stratifying by covariates measured prior to
  dropout (SKM Method 3) is better but may still
  be biased (Frangakis Rubin, 2001).

28
Discussion (continued)

• Double-sampling Method 4 seems to offer the
  only consistent methodology for estimation of
  survival.
• Moreover, increasing mortality within quintiles
  of risk for dropout means that there is no
  analysis, within the context of passive
  follow-up, that can adjust for covariates
  measured prior to dropout and eliminate
  estimation bias.

29
Conclusion

• Active follow-up alone cannot fully address the
  underestimation of mortality.
• Statistical adjustments, based on observed data
  obtained through passive follow-up, cannot
  address biases either.
• A combined approach of active follow-up and
  statistical modeling seems the only way to
  address biases and limitations in survival
  estimation.

30
Acknowledgements

• Academic Model for Prevention And Treatmetn of
  HIV/AIDS (AMPATH)
• Daniel Ochieng, Vincent Ochieng, Margaret
  Holdsworth
• Moi Teaching and Referral Hospital, Eldoret, and
  Mossoriot Rural Clinic, Mossoriot, Kenya.
• Moi University (Medical School and School of
  Public Health)
• PEPFAR
• International Epidemiologic Databases to Evaluate
  AIDS (IeDEA) East Africa Regional Collaboration.