Quantitative Research Methods Day 1 - PowerPoint PPT Presentation

1 / 37
About This Presentation
Title:

Quantitative Research Methods Day 1

Description:

Psychometric scales and outcome. Describing data and testing ideas ... argued that cognitive psychology has become 'restrictive as well as complacent' ... – PowerPoint PPT presentation

Number of Views:36
Avg rating:3.0/5.0
Slides: 38
Provided by: A108
Category:

less

Transcript and Presenter's Notes

Title: Quantitative Research Methods Day 1


1
Quantitative Research Methods Day 1 Research,
Quantification and Outcome
Vaughan Bell
2
Course Outline Day 1
  • Lecturer Dr Vaughan Bell
  • Why quantitative research?
  • Psychometric scales and outcome
  • Describing data and testing ideas
  • Research on assessing psychotherapy outcome

3
Course Outline Day 2
  • Lecturer Dr Katerina Fotopoulou
  • Quick revision of quantitative research
  • Evaluating the quantitative approach
  • Strengths and weaknesses vs qualitative research

4
Course Outline Day 3
  • Lecturer Dr Vaughan Bell
  • Randomised controlled trials
  • Publication issues

5
Two Important Types of Evidence
  • Evidence from clinical intuition
  • Evidence from research
  • In good clinical practice, each should inform the
    other.

6
What is Research?
  • Pretty much anything, but
  • Good research should have a number of features
  • Contribution to knowledge not wasting your and
    the participants time and resources
  • Methodical choosing the right tool for the job
  • Rigorous using the tool correctly

7
Quantitative Research
  • Quantitative research relies on several
    assumptions.
  • Perhaps the most important is that the thing your
    studying can be meaningfully quantified.
  • This is particularly important in psychology
  • where there is an ongoing debate about whether
    human experience or attributes can be
    meaningfully represented as numbers.

8
Cognitive Psychology
  • Quantification is now central to psychology
    because of the dominance of cognitive psychology.
  • Cognitive psychology has some core features
  • Information processing view of the mind.
  • Positivist / scientific approach to research
  • Experimental
  • Quantitative
  • Reductionist tendency
  • Aim for objectivity

9
Information Processing
  • To varying degrees, cognitive psychology seeks to
    explain the mind in terms of information
    processing.
  • Pylyshyn (1979, p435) argues that computation is
    not a metaphor but part of a literal description
    of cognitive activity.
  • Parkin (2000) argues that cognitive psychologists
    use information processing purely as an analogy
    for mental function.

10
Has this become a dogma ?
  • Still and Costall (1987) would agree and have
    argued that cognitive psychology has become
    restrictive as well as complacent.
  • Even Don Norman (1981) has criticized cognitive
    psychology for sterility and for descriptions
    which do not fit actual behaviour.
  • Criticisms of cognitive psychology come from two
    main sources Phenomenology and Algorithmic
    Information Theory

11
Meaningful Quantification
  • Whether you buy into these arguments or not,
    there is no doubt that quantification has proved
    useful.
  • The crucial question is is quantification useful
    for what I am doing / trying to understand?
  • Which brings us back to the question what can be
    meaningfully quantified?
  • Outcome is a major focus of quantitative methods.

12
Outline
  • Part1
  • What is outcome?
  • Types of measure
  • What attributes does a good measure need ?
  • Possible confounding factors and caveats.
  • Ethical issues
  • Part 2
  • Workshop on creating / assessing measures

13
What is Outcome ?
  • In the schizophrenia literature (Brekke et al,
    1993), outcome is classified in three ways
  • Clinical outcome signs, symptoms, service
    utilisation.
  • Functional outcome social, vocational,
    independent living.
  • Subjective outcome patients experiences of
    outcome.
  • It is important to be clear on what sort of
    outcome you want to assess and why.

14
Types of Measure
  • Clinical metrics length of hospital stay,
    readmissions, number of sessions with therapist.
  • Psychometric scales quantifies some aspects of
    psychological function or behaviour.
  • Self-report
  • Structured / semi-structured interview
  • Multi-rater measures multiple people are asked
    to rate the same material.
  • Tasks e.g. experimental / neuropsychogical tests
  • Physiological measures e.g. brain scans

15
Purpose
  • Screening tool Determines whether a symptom or
    psychological trait is present.
  • Aid to diagnosis typically increases the
    reliability of bedside diagnoses.
  • Quantification allows symptoms or traits to be
    quantified by intensity, duration etc.
  • Dimensional scale measures an attitude or trait
    through its range in the population.

16
Attributes of a Good Measure
  • The two essential features of a good measure,
    are
  • Reliability measures consistently.
  • Validity measures what it is designed for.
  • Each of these have various sub-categories that
    need to be fulfilled.

17
Reliability
  • Reliability must be established first, as
    validity relies on it.
  • i.e. A reliable scale could measure nonsense, but
    do so consistently
  • but it is impossible for a measure to be
    inconsistent and measure what it is designed for.

18
Reliability
  • Common forms
  • For psychometric scales
  • Internal reliability are similar items
    answered in similar ways ?
  • Test-retest reliability does the test produce
    similar results when used on the same people on
    different occasions ?
  • For multi-rater observational measures
  • Inter-rater reliability does the measure
    produce similar results when used by different
    observers

19
Internal Reliability
Are similar items answered in similar ways?
  • Typically tested with Cronbachs Alpha.
  • An alpha above 0.7 is usually considered
    satisfactory (Kline, 1993)
  • If a measure has multiple independent factors,
    they may need to be tested separately.

20
Test-Retest Reliability
Does the test produce similar results when used
with the same people on different occasions?
  • Typically tested with Pearson correlation.
  • Results from first occasion are correlated with
    results from second occasion.
  • Correlation should be above 0.8 (Kline, 1993)
  • Assumes that the object of measure is stable
    between occasions.
  • Therefore, this is usually tested on non-clinical
    groups.

21
Inter-rater Reliability
Does the measure produce similar results when
used by different observers?
  • Typically tested with Cohens Kappa.
  • Cohens Kappa controls for problems with directly
    comparing multiple raters scores.
  • e.g. one rater consistently scoring five-points
    more than the other will correlate despite not
    agreeing.
  • A two point rating (e.g. symptom present /
    absent) will have 25 agreement just by chance.

22
Cohens Kappa
  • Kappa values and level of agreement between
    raters (Landis and Koch, 1977)
  • Fair 0.21 - 0.40
  • Moderate 0.41 - 0.60
  • Substantial 0.61 - 0.80
  • Almost perfect 0.81 - 1.00

23
Validity
  • Face validity does the perception of the
    measure influence the outcome ?
  • Content validity does the measure cover
    everything it needs to cover ?
  • Construct validity do the results of the
    measure agree with what theory predicts ?
  • Criterion validity does the measure fulfil
    expected criteria ? usually the performance of
    a certain group
  • Incremental validity does it measure anything
    new ?

24
Face Validity
does the perception of the measure influence the
outcome?
  • If participants or testers misperceive the nature
    of the measure, it may affect the results.
  • e.g. if someone takes a verbal memory test but
    thinks it is a creative thinking test, it may
    look like they are confabulating.
  • Similarly, asking are you depressed ? may have
    good face validity for depression
  • but asking are you deluded ? has poor face
    validity for delusions.

25
Content Validity
does the measure cover everything it needs to
cover?
  • i.e. it is comprehensive ?
  • If a measure of anxiety asks only about social
    anxiety, it doesnt have good content validity.
  • This is usually assessed by comparing, or
    generating the measure, based on
  • Known phenomena
  • Literature reviews

26
Construct Validity
do the results of the measure agree with what
theory predicts?
  • There are two ways of assessing this
  • Convergence correlates with measures of things
    known to be associated
  • Divergence negatively correlates with things
    known to be mutually exclusive.
  • e.g. a good measure of depression should
    correlate with a measure of low mood
  • but negatively correlate with a measure of
    self-esteem.

27
Construct Validity
Or, a good measure or anxiety should predict
performance, as per the Yerkes-Dodson Law (1908)
28
Criterion Validity
does the measure fulfil certain criteria?
  • Often, this can be the same as construct validity
    (e.g. correlates with similar measures)
  • It is often tested by asking members of a certain
    group to take the test
  • who are known to have high levels of the measure
    being attributed.
  • e.g. people with psychosis should score higher
    than the general population on a good measure of
    anomalous perceptual experience.

29
Incremental Validity
does it measure anything new?
  • An assessment of what the measure adds to the
    toolkit of psychological assessment.
  • If it measures exactly the same as something
    else, in the same way
  • there may not be any point in developing it.

30
Wider Validity Issues
  • Rarely tackled in the textbooks, but it is
    important to assess how the validity tests were
    carried out.
  • Has validity been established for
  • different cultural groups ?
  • different ages ?
  • someone with a disability ?
  • etc.

31
Other Ongoing Changes
  • Particularly in the clinical environment, there
    may be a number of difficult-to-disentangle
    influences.
  • Children present a particular challenge as it
    might be difficult to separate the effects of
  • Cognitive development
  • Psychopathology
  • Treatment
  • Task engagement / motivation

32
Covariates
  • One way of controlling for this is to use
    covariates in statistical analysis, particularly
    ANOVA.
  • e.g. I want to see if my Special Brain Power
    Training boosts childrens intelligence.
  • I gave class A the training and see if they score
    more highly on an IQ test than class B.
  • However, class A are, on average, older, so they
    are likely to do better anyway.
  • I introduce age as a covariate into the analysis,
    to cancel-out its effects and make a fairer
    comparison.

33
Confounding Factors
  • Other factors can sometimes be more difficult to
    deal with
  • Floor / ceiling effects Where the test is so
    hard or easy that it is not possible to
    differentiate between participants.
  • Therefore, it is important to pilot the measure,
    and compare performance with norms.
  • Emotional impact Testing can be stressful for
    healthy individuals.
  • For patients, especially so, particularly if they
    see themselves as failing the tests.

34
Ethical Issues
  • There are distinct ethical (and potentially
    legal) implications when using such tests, e.g.
  • Are you using the measure for clinical decisions
    ?
  • If so, are you qualified and competent to develop
    / deploy / interpret the measure ?
  • If not, do you have adequate supervision from
    someone who is ?

35
Ethical Issues
  • Is the measure being used for research?
  • If so, has the research been given ethical
    approval?
  • Are you asking for informed consent ?
  • Does the patient know this is not part of their
    standard care ?
  • Are the results being kept private or anonymised ?

36
Conclusions
  • The purpose of measurement and the type of
    measure are crucial.
  • Measures need to be reliable and valid.
  • You need to be aware of possible confounding
    factors.
  • You need to be clear whether the context is
    research or clinical
  • and know and abide by the ethics of each.

37
Exercise
  • Completing the Magical Ideation Scale
  • Data analysis and discussion
  • Exercise in creating a measure of childhood
    psychosis-like experience
Write a Comment
User Comments (0)
About PowerShow.com