Individual Patient Data IPD Reviews and Metaanalyses

1
Individual Patient Data (IPD) Reviews and
Meta-analyses

• Lesley Stewart, Jayne Tierney
• IPD Meta-analysis Methods Group

Stewart LA, Tierney JF. To IPD or Not to IPD?
Advantages and disadvantages of systematic
reviews using individual patient data. Evaluation
the Health Professions 200225(1)76-97.
2
IPD Systematic Review /Meta-analysis

• Described as yardstick gold standard of
  systematic review
• Central collection, validation re-analysis of
  source data
• Less common than other types of review but
  becoming increasingly used
• Overall philosophy is same as for other Cochrane
  reviews, process differs only in terms of data
  collection analysis

3
History

• Established history in cardiovascular disease
• Established history in cancer
• 50 IPD meta-analyses of treatments or screening
• bladder, brain, breast, cervix, colorectal, head
  neck, leukaemia, lymphoma, lung, myeloma,
  oesophagus, ovary, prostate, soft tissue sarcoma
• Becoming used in a wide range of fields
• hernia, epilepsy, Alzheimers, dyspepsia, AIDs,
  hepatitis B, leg ulcers, pain relief, pregancy
  childbirth

4
How IPD Meta-analyses are Organised

• Carried out by international collaborative group
• small local secretariat
• multi-disciplinary advisory group
• trialists who provide data
• Developing and maintaining group requires
  communication and careful management
• Publication in the name of collaborative group

5
(No Transcript)
6
Why IPD?

• Analyses based on published data can give
  different answers to an IPD meta-analysis
• Examples
• chemotherapy in advanced ovarian cancer
• radiotherapy in SCLC
• chemotherapy in NSCLC
• ovarian ablation in breast cancer
• immunisation for recurrent miscarriage
• chemotherapy for head and neck cancer

7
Example Ovarian Cancer
Platinum based combination vs non-platinum single
drugs, Lancet 1993 341 418-422
8
Ovarian Cancer Conclusions

• Differences due to
• excluded trials, excluded patients, analysis
  timepoint, extra follow up, analysis method
• Published summary data gives a more statistically
  convincing result
• Estimates of effect size are 7.5 and 2.5
  improvement in survival at 30 months
• When balanced against other factors, clinical
  interpretation of results of the two approaches
  may be different

9
Limits of Relying on Published Data

• Unpublished trials not represented
• Papers may present inadequate data, they may
• selectively report particular outcomes
• measure or define outcomes in differently
• measure or define patient characteristics
  differently
• may report at different timepoints and/or with
  different periods of follow up
• exclusion of patients
• Potentially restricted to fraction of trials
• Need to collect additional data
• summary data or IPD from trialists

10
Limits of Relying on Published Data

• Unpublished trials not represented
• Papers may present inadequate data, they may
• selectively report particular outcomes
• measure or define outcomes in differently
• measure or define patient characteristics
  differently
• report at different timepoints and/or with
  different periods of follow up
• exclude patients from reported analyses
• Potentially restricted to fraction of trials
• Need to collect additional data
• summary data or IPD from trialists

11
Types of review/meta-analysis
Extract data from published reports
Summary Data
Systematic Review Meta-analysis
Collect summary data
Contact Trialists
Collect individual patient data (IPD)
12
Comparing types of review/meta-analysis
Write protocol State objectives, searches,
inclusion criteria and planned analyses
prospectively
Identify all relevant trials
Establish Secretariat, Advisory and Trialist
Groups
Assemble the most complete dataset possible
Collect and validate data
Analyse individual studies and perform
meta-analysis
Collaborators Conference
Processes the same for summary data and IPD
Processes are similar for summary data and IPD,
but methodology and practical aspects differ
Prepare structured report
Only applies to IPD approach
13
Benefits of the IPD Approach

• Improve data quality
• Improve analysis quality
• Improve interpretation dissemination via
  collaborative approach

14
Data Quality Include all Trials

• Positive trials more likely to be published
  than negative trials
• Any review that uses only data from published
  reports at risk of publication bias
• Include all relevant trials
• published unpublished
• Unpublished trials not peer reviewed, BUT
• trial protocol IPD allows extensive peer
  review
• publication of quality manuscript does not
  guarantee quality of data

15
Data Quality Include all Trials

• Including unpublished trials ( those published
  with insufficient data) can be done for both IPD
  and summary data approaches
• Should be a priority for all systematic reviews
• Proportion of trials published will vary
• disease, intervention, over time
• Extent of unpublished data can be considerable

16
Identification of TrialsCervix Meta-analysis
(initiated 1999)
17
Data Quality Include all Trials

• Systematic reviews do not need to be IPD based to
  seek unpublished trials, but
• direct contact with trialists is useful
• based on raw rather than summary data so no
  difference in data from published and unpublished
  trials
• inclusion requires no special provision

18
Identification of TrialsCervix Meta-analysis
(initiated 1999)
19
Data Quality Relevant Trials

• Clarify proper randomisation, eligibility
• Avoid duplicate trial inclusion
• trials often published more than once
• authors differ, trial description differs,
  different groups of patients
• direct contact with trialists helps identify
  duplicates
• Combat poor reporting

20
Data Quality All Patients

• Aim to include maximum randomised evidence
• Reinstate excluded patients, because ad hoc
  exclusion of patients could introduce bias
• cannot do this with data extracted from
  publications
• can do this with summary data from trialists
• collect data summary data on all patients or
  supplementary for excluded patients
• can do this with IPD
• Whether information on excluded patients is
  retained may vary by disease and intervention

21
Data Quality All Patients
Tierney JF, Stewart LA. Investigating exclusion
bias in meta-analysis. Int J Epidemiol 3479-87
22
Data Quality All Patients

• Adjuvant chemotherapy for soft tissue sarcoma
• Obtained data for 14 trials, 1568 patients
• 341 (22) of these patients excluded from the
  investigators analyses

23
Data Quality All Patients

• Pre-specify in the protocol if any patients will
  be excluded from the analysis
• assess impact by sensitivity analyses
• probably only practical with IPD

24
Pre-specified Patient Exclusions
25
Data Quality Include All Patients

• IPD may not be available from all trials
• If data available is high proportion of all
  evidence (gt95?), confident of results
• But need to be aware that bias could be
  associated with trial availability
• sensitivity analyses
• Availability of data in each format may be major
  factor in determining approach

26
Data Quality Checking Data

• IPD enables detailed data checking,not easily
  achieved with any other approach
• Reasons for checking
• not to centrally police trials or to expose fraud
• improve accuracy of data
• improve follow-up
• ensure appropriate analysis
• ensure all randomised patients are included
• ensure no non-randomised patients are included

27
Data Quality Checking Data

• Standard
• missing data, excluded patients
• internal consistency
• range checks
• Randomisation
• balance across arms, baseline factors
• pattern of randomisation
• Follow up
• up-to-date
• equal across arms

28
Checking Pattern of Randomisation Radiotherapy
vs Chemotherapy in Multiple Myeloma
29
Checking Weekday Randomised
30
Checking Follow Up

• Reverse survival curve
• take patients event-free, use censoring as event

31
Analysis Quality

• Combining different scales of measurement
• outcome data
• patient characteristics
• Subgroup analyses (patient treatment
  interactions)
• Long-term outcomes
• Exploratory analyses (e.g. prognostic factors,
  baseline risk etc.)
• Time-to-event analysis

32
Analysis Quality

• Individual patient data used
• Most commonly, 2-stage analysis
• summary statistics generated for each trial
• these statistics combined in meta-analysis
• i.e. stratified by trial
• Less commonly, 1 stage analysis
• regression/modelling approach
• all patients are combined into a single mega
  trial (not appropriate)

Meta-Analysis of Individual Patient Data from
Randomised Trials A Review of Methods used in
Practice. Clinical Trials, in press.
33
Analysis Quality Time-to-event

• Major benefit of IPD is that it allows
  time-to-event analysis which accounts for
• whether an event happens
• the time at which it happens
• For some diseases just the ability to do such an
  analysis justifies the IPD approach
• cure is not likely, prolongation of survival
• time to onset of disease, time free of symptoms

34
Analysis Quality Time-to-event

• Published or Summary Data
• usually restricted to analysis at a fixed point
  in time
• or a series of fixed time points, different
  trials drop in/out of analyses over time
• usually estimates odds ratio (OR)
• can make better use of summary statistics to
  estimate HR
• may not always be practical

Parmar MKB, Torri V, Stewart L. Extracting
summary statistics to perform meta-analyses of
the published literature for survival endpoints.
Statistics in Medicine 1998172815-34.
35
Analysis Quality Time-to-event

• Individual patient data
• uses individual times at which each event takes
  place takes account of censoring
• uses log rank O-E V
• summarises entire survival experience
• estimates hazard ratio (HR)
• allows survival curves

36
Subgroup analysis

• By patient level characteristics
• e.g. age, sex, tumour stage
• group by type of patient (subgroup analysis)
• difficult to do from publications which usually
  report trial-level summaries of patient-level
  information e.g. mean age
• can be done with summary data from trialists
• more easily done with IPD especially for multiple
  analyses
• allows for consistent definition of subgroups
  across trials
• By trial level characteristics

37
NSCLC Subset Analysis
38
Analysis Quality Subgroup Analysis

• Does not compare the prognosis of subgroups
  (not comparing survival of young with old
  patients)
• Does compare the size of any treatment effect in
  different subgroups of patients
• do young patients benefit more than old patients?
• looking for treatment interactions
• Analyses stratified by trial
• for TTE, calculate log rank O-E and V for males
  only in each trial, then combine in overall HR
• Currently not easy to input results via RevMan

39
PORT Subgroup Analyses
40
PORT Subgroup Analyses
41
Analysis Quality Long-term Outcomes

• Adjuvant tamoxifen for early breast cancer
• 5 years of follow-up
• absolute survival benefit of 5
• 10 years of follow-up
• absolute survival benefit of 6
• benefit irrespective of dose
• reduced risk of contralateral breast cancer
• 15 years of follow-up
• benefit applicable to broader range of women
• greater benefit with longer duration of tamoxifen

42
Analysis Quality Different Scales

• Adjuvant chemotherapy for soft tissue sarcoma
• 3 grading systems used
• Translated into a common system

43
Benefits of Collaboration

• Trial identification
• Compliance in providing data
• Balance in interpretation
• multi-disciplinary group
• cross cultural group
• Integral part of research cycle
• further reviews
• new primary research

44
Barriers Negotiating Collaboration

• Practical (tracing people, language differences)
• e-mail, web-sites, directories, search engines
• Educational (unfamiliar with concepts)
• protocol, good communication
• Political (difficult people, powerful groups)
• protocol, good communication, intermediaries
• Financial (money for data)
• ???

45
Barriers Time to Assemble Data

• Time taken to transfer data
• Advanced Ovarian Cancer (initiated 1989)
• 44 on paper, 39 on disk, 17 by e-mail
• Soft tissue sarcoma (initiated 1994)
• 0 on paper, 29 on disk, 71 by e-mail
• Cervix cancer (initiated 1999)
• 0 on paper, 5 on disk, 95 by e-mail

46
Barriers Time to Assemble Data

• Combining different data formats (ASCII,
  spreadsheet, database, stats packages)
• used to be time-consuming
• software packages more compatible
• improved software automates more tasks
• Re-coding
• offer suggested coding
• Checking
• improved software automates more tasks
• queries by e-mail

47
Barriers Time to Assemble Data

• Neoadjuvant chemotherapy
• for locally advanced cervix
• cancer
• Protocol and searches May 98 - Jan 99
• Invite to collaborate Mar 1999
• Collaborators meeting Sep 2000

• Neoadjuvant chemotherapy
• for locally advanced
• bladder cancer
• Protocol and searches
• Dec 00 - May 01
• Invite to collaborate Jun 2001
• Collaborators meeting Feb 2002

48
Barriers Resources

• Time resource?
• often reported as MUCH more expensive
• but technology advances to cut costs/ time
• need empirical data
• Cost of Collaborators Conference not encountered
  in other types of review

49
Collaborators Meeting

• Integral part of IPD MA
• IPD MA a collaborative project
• Incentive to collaborate
• Trialists have opportunity
• to discuss results
• to challenge the analysis
• to discuss interpretation and implication of
  results
• Sets a deadline to which secretariat and
  trialists have to work

50
(No Transcript)
51
(No Transcript)
52
Barriers Software

• Primary analysis
• most IPD groups use own software
• carries out stratified analysis and produces
  graphical output
• Input into RevMan
• For time to event data use IPD or TTE outcome
• enter log rank summary statistics from primary
  analysis
• For TTE or other outcomes use generic inverse
  variance
• enter estimate and SE from primary analysis
• not easy to enter subgroup analyses

53
Summary Benefits of IPD

• Carry out detailed data checking
• Ensure quality of randomisation follow up
• Update follow up information
• Carry out time-to-event analyses
• Only practical way to do subgroup analyses
• More detailed exploration of patient
  characteristics
• Ensure appropriateness of analysis
• More flexible analysis of outcomes

Also possible with summary data obtained from
trialists
54
Summary Other Benefits

• More complete identification of trials
• Better compliance in providing missing data
• More balanced interpretation of results
• Wider endorsement dissemination of results
• Better clarification of further research
• Collaboration on further research
• Also possible with summary data obtained from
  trialists

55
Summary Barriers

• Likely to be more costly and time-consuming
• Range of expertise and skills required
• But differences between IPD and other types of
  systematic review may not be so great
• intense development phase
• longer data collection phase
• intense analysis and collaborators meeting phase
• IPD projects can be run concurrently
• Practical / political issues

56
To IPD or not to IPD?

• Some benefits may be potentially achieved by
  obtaining additional summary data
• Collecting summary data may entail
• too many tables,more work for trialists
• requires trialists to interpret and act on
  instruction eg. redo analyses, reclassify data
• So why not collect IPD ?

57
To IPD or not to IPD ?
When IPD may be beneficial
When IPD may not be beneficial
Poor reporting of trials. Information
Detailed and clear reporting of trials
inadequate, selective or ambiguous
(CONSORT quality)
Long-term outcomes
Short-term outcomes
Time-to-event outcome measures
Binary outcome measures
Multivariate or other complex analyses
Univariate or simple analyses
Differently defined outcome measures
Outcome measures defined uniformly
across trials
Subgroup analyses of patient-level
Patient subgroups not important
characteristics important
IPD available for high proportion of
IPD available for only a limited number of
trials/individuals
trials