Identifying and Selecting Measures for Health Disparities Research

1
Identifying and Selecting Measures for Health
Disparities Research

• Anita L. Stewart, Ph.D.
• University of California, San Francisco
• Clinical Research with Diverse Communities
• EPI 222, Spring
• April 9, 2009

2
Selecting Measures for Your Own Study The Problem

• You are beginning a study
• You know the concepts (variables) of interest
• Question Which measure of ________ should I
  use?
• A popular measure
• One that a colleague used successfully
• Create your own

3
Inappropriate Measures can Result in

• Conceptual inadequacy
• Measuring wrong concept
• Poor data quality (e.g. missing data)
• Poor variability
• Poor reliability and validity
• Inability to detect true associations
• e.g., no measured change in outcome when change
  occurred

4
Selecting Measures for Your Research

• Goal find a measure of your concept that has
  been developed using stringent measurement
  development methods
• Your task find measures and review them for all
  steps in measurement development process

5
Process of Selecting Good Measures for Your
Studies
Define concept (variable)
Specify study context
Identify potential measures
Review measures properties --conceptual and
psychometric adequacy
Pretest best 1-2 measures
Select final measure
6
Process of Selecting Good Measures for Your
Studies
Define concept (variable)
Specify study context
Identify potential measures
Review measures properties --conceptual and
psychometric adequacy
Pretest best 1-2 measures
Select final measure
7
Concept/Construct

• A variable that is relatively abstract as opposed
  to concrete
• e.g. health status, stress, acculturation vs.
  height, body temp
• An abstraction based on observations of certain
  behaviors or characteristics
• Cannot be assessed directly

8
Measures of Concepts

• Concepts are defined and operationalized in terms
  of observed indicators or measures
• Measures are proxies for the latent variables
  we cannot directly observe

9
Define Concept For Your Study

• Define concept from your perspective, taking into
  account your
• study questions
• target population
• For outcomes, describe
• how intervention or independent variables might
  affect it
• specific types of changes you expect

10
Example of Concept Interpersonal Processes of
Care

• Interpersonal processes between physicians and
  patients
• Communication, decision making, respectfulness,
  empathy
• Emphasis on processes relevant to vulnerable
  patients

11
Purpose of Measuring Concept

• Describe how your concept fits into your research
  question(s)
• Outcome measure?
• Determinant of health?
• Identify need for intervention?

12
Example Interpersonal Processes of Care

• Interpersonal processes of care may influence
  patient outcomes
• Proximal outcomes
• Knowledge of condition and recommendations
• Adherence to recommendations
• Patient satisfaction
• Ultimate outcomes
• Health status

13
Process of Selecting Good Measures for Your
Studies
Define concept (variable)
Specify study context
Identify potential measures
Review measures properties --conceptual and
psychometric adequacy
Pretest best 1-2 measures
Select final measure
14
Specify Context for Measures

• Study characteristics affecting choice of
  measures
• Nature of target population
• Practical constraints
• Nature of population (patients)
• Lower educational level? Limited literacy?
• Healthy or ill?

15
Practical Constraints

• Preferred mode of administration
• Acceptable respondent burden
• Budget to pay for measures
• Translations needed
• Method of data entry
• Time frame time to select and pretest measures

16
Process of Selecting Good Measures for Your
Studies
Define concept (variable)
Specify study context
Identify potential measures
Review measures properties --conceptual and
psychometric adequacy
Pretest best 1-2 measures
Select final measure
17
Locate Potential Measures

• Identify candidate measures for all concepts
• Redundancy OK for now
• DO NOT develop your own questions unless it is
  absolutely necessary

18
Locating Measures

• For major dependent or independent variables
• Multi-item measures with known psychometric
  properties
• For background variables and covariates
• Standardized survey measures

19
Locating Measures Possible Sources

• Compendia
• Organizations and research centers
• Government agencies
• National and state surveys
• Large relevant research studies

20
Helpful Handouts to Locate Measures

• CADC measurement core website
• Locating measures for health disparities research
• Measures compilations and reviews
• Summary of surveys National Center for Health
  Statistics
• Instructions for accessing questionnaires from
  several national and state health surveys

21
Handout Locating Measures for Health Disparities
Research

• To link to websites, need to log on to CADC
  website

http//dgim.ucsf.edu/cadc/cores/measurement/resour
cescode.html
22
Locating Measures Compendia

• Specific measures of various concepts are
  compiled, reviewed, listed, or otherwise provided
• Books SEE HANDOUT
• Many books review and critique various measures
• Web
• A few websites

23
Compendia by Web

• National Cancer Institute website
• Health behavior constructs theory, measurement,
  and research
• Reviews concepts and measures of constructs such
  as perceived control, social support, and
  perceived vulnerability

http//dccps.cancer.gov/brp/constructs/index.html
24
Locating Measures Organizations and Research
Centers

• Some organizations and research centers
  specialize in measurement and provide public
  access websites

25
RAND Health Program

• Measures, scoring manuals, and citations
• Specialty
• Quality of care, patient satisfaction
• Health-related quality of life
• Generic and disease specific
• Medical Outcomes Study (MOS) measures
• Adults and children
• www.rand.org/health/ (surveys and tools)

26
Ottawa Health Decision Centre

• Patient and physician decision aids
• http//www.ohri.ca/centres/DecisionAids/default.as
  p
• Patient measures, e.g., decisional conflict,
  decisional regret, stage of decision making,
  decision self-efficacy
• http//decisionaid.ohri.ca/eval.html

27
Commonwealth Fund Surveys

• Health insurance
• Medicare
• Health care quality, patient centered care,
  underserved populations
• Child health development, care of the elderly
• State health policy, international health policy

• www.commonwealthfund.org/surveys/

28
Commonwealth Survey of Physicians

• 2006 International Health Policy Survey of
  Primary Care Doctors
• Use of information technology
• Access to care
• Availability of financial incentives
• Chronic care management

29
MacArthur Research Network on Socioeconomic
Status and Health

• Reviews measures in several domains
• Psychosocial
• Social and physical environment
• Socioeconomic status (SES)
• SES across the lifecourse
• MacArthur Network on SES and Health

http//www.macses.ucsf.edu/Research/overview.htm
30
Duke University Center for Demographic Studies

• National Long Term Care Survey (with NIA)
• Prevalence and patterns of functional
  limitations, medical conditions, recent medical
  problems, use of health care, and housing and
  neighborhood characteristics
• www.nltcs.aas.duke.edu/index.htm

31
Roadmap K12 Data Resource Center

• Links to over 2 dozen national and state surveys
• Provides overview for each domains, time frame,
  population, scope, sample size, and contacts
• www.epibiostat.ucsf.edu/courses/RoadmapK12/PublicD
  ataSetResources/

32
Locating Measures Government Agencies

• Several federal and state government agencies
  provide measures for use in health and health
  disparities research
• Information about measures (e.g. clearinghouse)
• Actual measures

33
Agency for Healthcare Research and Quality (AHRQ)

• National Quality Measures Clearinghouse

34
Agency for Healthcare Research and Quality (AHRQ)

• Medical Expenditure Panel Survey (MEPS)
• www.ahrq.gov/data/
• Consumer Assessment of Health Plans Survey
  (CAHPS)
• www.ahrq.gov/qual/

35
Department of Veterans Affairs

• National Survey of Veterans, 2001
• www.va.gov/vetdata/surveyresults/index.htm
• Survey of Veteran Enrollees Health and Reliance
  Upon VA, 2003
• www.va.gov/vetdata/healthcare/index.htm
• Both include measures of demographics and
  socioeconomic status, military background,
  health, health insurance, and VA and non-VA
  benefits usage.

36
National Cancer Institute

• Special initiatives on measures
• Health Information National Trends Survey (HINTS)
• Compiled cancer screening questions, identified
  best ones, conducted extensive pretesting

http//hints.cancer.gov/
37
Centers for Disease Control and Prevention

• Behavioral Risk Factor Surveillance System
• Surveys of health behaviors
• www.cdc.gov/brfss/questionnaires/index.htm

38
National and State Surveys

• Good sources of descriptors, covariates
• Usually dont have multi-item measures

39
CDC National School-Based Youth Risk Behavior
Survey (YRBS)

• Survey conducted every other year
• random national sample of youth in grades 9-12
• Most states conduct survey
• Measures of substance use, risky sexual
  behaviors, diet, physical activity, overweight
• www.cdc.gov/HealthyYouth/yrbs/index.htm

40
California Health Interview Survey (CHIS)

• A behavioral risk surveillance survey modeled
  after the CDC BRFSS
• Numerous languages
• www.chis.ucla.edu/
• Go to Methodology review questionnaires

41
National Center for Health Statistics (NCHS)

• Surveys and data collection systems
• Can download
• Any survey or portion of survey
• Handouts
• Summary of surveys
• Instructions for accessing questionnaires

http//www.cdc.gov/nchs
42
NCHS National Health Care Surveys Surveys of
Physicians

• Family of provider-based surveys
• Provide objective, reliable information about
• organizations and providers
• services rendered
• patients they serve

http//www.cdc.gov/nchs/surveys.htm
43
Locating Measures Large Research Studies

• Many large-scale, multi-center and longitudinal
  studies have developed and used measures on
  health-related topics
• Increasingly, they are posting these on study
  websites

44
Study of Womens Health Across the Nation (SWAN)

• Physical, biological, psychological, and social
  changes of women during their middle years
• Questionnaire can be downloaded
• www.edc.gsph.pitt.edu/swan/public

45
Sacramento Area Latino Study on Aging (SALSA)
Study

• NIA funded longitudinal study of Latinos in the
  Sacramento region
• Started in 1996
• each person followed for up to 5 years
• http//sitemaker.umich.edu/salsa.study/home

46
Locating Measures Finding Authors of Measures

• Published research using measure you are
  interested in
• Unpublished measures often described in methods
• Authors may provide measures

47
Process of Selecting Good Measures for Your
Studies
Define concept (variable)
Specify study context
Identify potential measures
Review measures properties --conceptual and
psychometric adequacy
Pretest best 1-2 measures
Select final measure
48
Review Potential Measures for

• Conceptual adequacy for your study
• Psychometric adequacy in target group(s)
• Practicality, acceptability in your study
• Translations available if needed

49
Conceptual Adequacy for Your Study

• Concept being measured matches the concept you
  defined
• Sometimes can only be determined by reviewing
  items
• If not a perfect match
• How close is it to your concept?
• Can it be modified to get at missing components?

50
Conceptual Adequacy

• You are interested in reports of perceived
  discrimination in health care setting
• Measures of discrimination pertain to
• Discrimination over the lifecourse
• Discrimination in various settings (work, school)
• Not adequate for your purpose

51
Conceptual Adequacy interpersonal processes of
care

• You are interested in variety of processes
  relevant to vulnerable patients
• Measures available
• Designed for mainstream populations
• Focus on communication
• Missing decision making and interpersonal
  dimensions

52
Psychometric Adequacy for Your Study

• In samples similar to your target group
• good variability
• low percent of missing data
• good reliability
• good validity
• As an outcome for planned intervention
• responsive, sensitive to change in similar
  population
• able to detect expected magnitude of change

53
Reliability

• Extent to which an observed score is free of
  random error
• Population-specific reliability increases with
• sample size
• variability in scores (dispersion)
• a persons level on the scale

54
Internal Consistency Reliability Cronbachs
Alpha

• Requires multiple items measuring same construct
• Extent to which items measure same construct
  (same latent variable)
• It is a function of
• Number of items
• Average correlation among items
• Variability in your sample

55
Minimum Standardsfor Internal Consistency
Reliability

• For group comparisons (e.g., regression,
  correlational analyses)
• .70 or above is minimum
• .80 is optimal
• above .90 is unnecessary
• For individual assessment (e.g., treatment
  decisions)
• .90 or above (.95) is preferred

JC Nunnally, Psychometric Theory 3rd ed,
McGraw-Hill, 1994
56
Validity

• Does a measure (or instrument) measure what it is
  supposed to measure?
• AndDoes a measure NOT measure what it is NOT
  supposed to measure?

57
Validation of Measures is an Iterative, Lengthy
Process

• Validity is not a property of the measure
• Validity is a property of a measure for
  particular purpose and sample
• Validation studies for one purpose and sample may
  not serve another purpose or sample
• Accumulation of evidence
• Different samples
• Longitudinal designs

58
Construct Validity Basics

• A process of answering the following questions
• What is the hypothesis?
• What are the results?
• Do the results support (confirm) the hypothesis?

59
Practical Considerations Match to Your Context

• Study context
• Need permission to use? Any cost of using?
• Scoring rules available?
• Method of administration appropriate?
• Short forms if needed?
• Appropriate for your sample
• Reading level
• Translations
• Acceptability, respondent burden

60
Practical - Obtaining Permission

• Need permission to use or to adapt?
• Public domain
• If items are published or in the public domain,
  usually dont need permission
• Private or proprietary
• Need to write to author or distributor
• Allow 4-6 weeks to obtain measure and/or
  permission

61
Practical - Cost to Use or to Score Measures

• Cost of administering and scoring
• Fee for each instrument purchased
• Cost of any needed scoring software?
• Cost to have it scored by source?
• Cost per instrument?

62
Practical - Scoring

• Are scoring instructions clearly documented?
• Do you have a scoring codebook?
• Are computer scoring programs available?
• (Cost of scoring)

63
Practical Short Forms?

• Are there reliable and valid short-forms
  available if you need it?
• Many measures have short forms, but they
  typically have not been tested as thoroughly
• Shorter forms can have low variability,
  reliability, validity, and sensitivity to change

64
Reading Level

• Is reading level appropriate for your target
  population?
• Special concern in lower SES, limited English
  proficiency groups
• If reading level not known
• Make your own judgment
• Pretest with target population

65
Availability of Translations if Needed

• If you need measure in another language, are
  there translations available?
• Official (published and tested)
• Unofficial (by some other researcher)

66
Translation Availability and Quality

• Is the measure available in the language of your
  target populations?

No
Yes

• Know method of translation
• Assess adequacy or quality of translation

• Perform translation using state-of-the-art
  methods
• A resource issue

67
Acceptability

• Ease with which measure can be used in your
  setting and population
• Acceptability to target population
• respondent burden
• culturally sensitive
• Acceptability to interviewers
• amount of training needed

68
Respondent Burden

• Real burden
• Length, convenience, time needed to complete
• Perceived burden
• a function of item difficulty, distress due to
  content, perceived value of survey, expectations
  of length
• Some population subgroups may have more
  difficulty, take longer to complete

69
Process of Selecting Good Measures for Your
Studies
Define concept (variable)
Specify study context
Identify potential measures
Review measures properties --conceptual and
psychometric adequacy
Pretest best 1-2 measures
Select final measure
70
Pretest Potential Measures in Your Target
Population

• Select best measures for all concepts in your
  conceptual framework
• existing instrument in its entirety
• subscales of relevant domains (e.g., only those
  that meet your needs)

71
Pretest

• Pretesting essential for priority measures (e.g.,
  outcomes)
• Pretest is to identify
• problems with method of administration
• unacceptable respondent burden
• problems with questions or response choices
• words and phrases that do not mean what you
  intended to target population

72
Pretest Methods

• April 21 class by Anna Nápoles
• Using qualitative methods in developing and
  testing concepts, measures, and interventions

73
Summary

• Methods for selecting/reviewing measures
  described here are ideal
• Apply these methods to your most important
  measures
• e.g., outcomes, key independent variables

74
Homework

• See handout for class 2 homework
• Complete rows 1-13 in matrix
• Use form posted on the website
• Remaining rows are for the next measurement
  lecture
• Email responses to anita.stewart_at_ucsf.edu
• By Monday April 13