Nursing 503: Week 6

1
Nursing 503 Week 6

• Issues in data collection, data management, and
  data analysis

2
Overview

• Measurement error
• Relationships between reliability and validity
• Psychometric soundness

3
Measurement

• Trying to determine what really exists
• Error is the difference between what exists in
  reality and what is measured by a research tool
• Small measurement error when what exists in
  reality is concrete.
• Large measurement error what exists in reality is
  abstract.

4
True and observed scores
OTE O is observed score (research tool) T is
true score E is error
5
Which measure has less error?
Tallness (T)
Ruler (O)
Tallness (T)
Trouser length (O)
6

• If we measure A using A1, we get some information
  about things outside A.

A
A1
Interesting but irrelevant!!
7

• If we use measurement strategies that reduce the
  error term, we increase the accuracy, and hence
  the validity, of our measurement.

8
Small group exercise

• Identify 1 variable from your research question
  or capstone topic.
• How is it measured?
• How much measurement error do you think there is?

9
A case of mistaken identity

• Oral health vs. stomatitis severity
• Measurement tools
• WCCNR (Olson et al., 1998)
• Lesions, colour, bleeding
• Oral assessment guide (Eiler et al., 1988)
• Lips, tongue, mucous membranes, gingiva,
  teeth/dentures, voice, swallow, saliva

10
Why is measurement error such a big deal?

• Treatment decision are made based on the results
• Morbidity
• Mortality
• Quality of life

11
Types of measurement error

• Systematic or random (see Wood and Ross-Kerr, p.
  200)
• Systematic
• Directly affects validity
• Kinds
• Social desirability
• Acquiescent response set
• Causes
• Personal traits intelligence
• Design
• Tools lacking in validity
• Trials with a something vs. nothing independent
  variables

12

• Random
• Directly affect reliability
• Increases unexplained variation
• Causes
• Research situation
• Environment
• Lack of clarity in instruments
• Data collection

13

• What kind of error would be introduced in a study
  requiring stomatitis severity assessment if one
  used a tool that measured oral health?

14

• What kind of error would be introduced in a study
  of parenting if one collected data in the home?

15
Reliability

• Consistency
• Consistently inaccurate
• Consistently accurate
• Reliability testing evaluates the amount of
  random error in the measurement technique
• 0.8 is considered the lower bound of acceptability

16
Reliability testing

• Stability
• Test-retest reliability
• Homogeneity
• Internal consistency
• Equivalence
• Interrater reliability (people)
• Parallel forms (tests)

17
Validity

• Accuracy
• Validity testing evaluates the amount of
  systematic error in the measurement technique

18
Validity testing

• Content
• Method of measurement captures expected content
• Predictive
• Measure can predict future events
• Construct
• Tool measures the construct it was designed to
  measure
• Concept analysis
• Factorial validity
• Contrast groups approach
• Convergent validation
• Divergent validation
• Discriminant validation
• Nomological network validation
• Successive verification by others

19
Ethical Issues Scale

• Early work
• Berger et al., 1991
• Content validity only
• No psychometric evaluation
• Scanlon, 1994
• No psychometric evaluation

20

• Development of EIS
• Based on interviews and literature
• End-of-life treatments
• Patient care
• Human rights

21

• Testing the reliability and validity of EIS
• Sample
• Stratified, random (n8,536)
• Response rate of 28.8
• Evaluable cases 2,090
• Randomly split to facilitate cross-validation
  study

22
Reliability testing

• Stability
• Test-retest not done
• Homogeneity
• Cronbachs alpha
• All items .91 in both samples
• End of life treatment issues 0.86, 0.85
• Patient care issues 0.84, 0.82
• Human rights issues 0.74, 0.74
• Equivalence
• Inter-rater/parallel forms not done

23
Validity testing

• Content validity
• Focus groups
• Revision to items
• Added 3 items
• Content validity panel
• Factorial validity
• Factor analysis supported 3 factors in both
  samples
• End of life treatment issues (13 items, 18.9 of
  variance)
• Patient care issues (14 items, 14.8 of variance)
• Human rights issues (5 items, 8.7 of variance)

24
Statistical vs. clinical significance

• Statistical significance
• Inexact alternate hypothesis (range of values)
• Specific alternate hypothesis based on effect
  size (small, medium, large)
• Clinical significance
• Magnitude of effect (effect size, variance)
• Individual approaches
• Proportion improved
• Normative changes
• Return to normal functioning
• Social validation
• Quality of life from the standpoint of recipient
  of intervention
• Health care providers
• Society at large

25

• Norman, G., Sloan, J., Wyrwich, K. (2003).
  Interpretation of changes in health-related
  quality of life The remarkable universality of
  half a standard deviation. Medical Care 41 (5),
  582-592.

26

• Statistical significance and sample size
• Significance must be set before the study is done
  so that power can be calculated.
• The value of power
• Not too big
• Not too small
• Just right