Complementary Research Methods

1
Complementary Research Methods

• Michele Jacobsen, PhD
• Postdoctoral Fellow, SERN
• University of Calgary
• dmjacobs_at_ucalgary.ca
• http//www.ucalgary.ca/dmjacobs/phd/methods/

2
Overview

• Outline relative merits of Triangulation - the
  integration of both qualitative and quantitative
  research methodologies
• Review a variety of research methods
• Experimental design, ethnography, case study,
  survey methods.
• Discuss Nowaczyk and Underwoods (1995) paper.

3
Fundamental Goals...

• Of Science
• To Understand, To Predict, To Control
• Of Scientists
• To communicate discoveries and findings to a
  community of peers

4
Designing ResearchDimensions of Analysis

• Research Purposes - theoretical or applied?
• Research Problems - what questions are asked?
• Research Settings - simulated or natural?
• Research Investigators - background and training
• Research Methods - a continuum
• Experimental, Ethnography, Case study, Survey

5
Evaluating Research

• Validity
• A concern for most social scientists is the
  complex nature of the phenomena under study
  human behavior.
• Multiple perspectives are required in order to
  adequately reflect the richness of these
  complexities.
• Reliability
• Consistency, Replicability
• Usefulness or Value of Investigation
• Contribution to knowledge
• Advance THEORY and PRACTICE in discipline

6
Research MethodologiesA continuum rather than
either/or

• Qualitative
• Goal To Understand, Predict
• Descriptive accounts
• Similarities and Contrasts
• Applied and Theoretical
• Research Questions
• Field study
• Natural conditions

• Quantitative
• Goal To Predict and Control
• Measure and Evaluate
• Generalize to population, reproduction
• Basic and Theoretical
• Hypothesis testing
• Lab study
• Controlled, contrived

7
Validity and Reliability

• Both Quantitative and Qualitative research
  designs seek reliable and valid results. For
  example
• Quantitative Reliability Data that are
  consistent or stable as indicated by the
  researcher's ability to replicate the findings.
• Qualitative Validity of findings are paramount
  so that data are representative of a true and
  full picture of constructs under investigation.

8
Part Versus Whole

• Whole is often greater than Parts
• It is a non-trivial matter to infer the behavior
  of the whole from the behavior of its parts
• Quantitative research designs strive to identify
  and isolate specific variables within the context
  (seeking correlation, relationships, causality)
  of the study.
• Qualitative design focuses on a holistic view of
  what is being studied (via documents, case
  histories, observations and interviews).

9
Data Collection

• Quantitative
• Emphasis on numerical data, measurable variables
• Data is collected under controlled conditions in
  order to rule out the possibility that variables
  other than the one under study can account for
  the relationships identified
• Qualitative
• Emphasis on observation and interpretation.
• Data are collected within the context of their
  natural occurrence.

10
Static and Dynamic

• Quantitative
• The accumulation of facts and causes of behavior
  through careful isolation, measurement and
  evaluation of variables.
• Predictability and Control over time.
• Qualitative
• Concerned with the changing and dynamic nature of
  reality.
• Understanding a Point in time

11
Triangulation

• Combines independent yet complementary research
  methods.
• Simultaneous triangulation
• Use of both qualitative and quantitative methods
  at the same time
• e.g., Survey methods and Case study
• Sequential triangulation
• Results of one method are essential for planning
  the next method
• e.g., Exploratory Pilot study precedes
  Experimental design
• Also known as mixed methods

12
Benefits of Triangulation

• Advantages of each complement the other
• resulting in a stronger research design, and
• more valid and reliable findings.
• Inadequacies of individual methods are minimized
• threats to Internal Validity are realized and
  addressed
• Example
• Quantitative design strives to control for bias
  so that facts, instances, phenomena can be
  understood in an objective way.
• Qualitative approach strives to understand the
  perspective of participants or a situation by
  looking at firsthand experience to provide
  meaningful data.

13
Additional Benefits

• Triangulation offers a balance between logic and
  stories.
• Qualititative research, which emphasizes
  exploration, understanding, contextualizing,
  introspection, and theory construction, provides
  a strong base for wider quantitative measures,
  scaling, and generalization.
• Quantitative research, which emphasizes large
  samples, can provide an overview of an area that
  can reveal patterns, inconsistencies, and so
  forth, that can be further investigated with
  qualitative methods.

14
Effectiveness of Triangulation

• In order for Triangulation to be used
  effectively, four principles must be adhered to
• 1. research question(s) must be clearly focused
• 2. strengths and weaknesses of each chosen method
  must complement each other
• 3. data collection methods should be selected
  according to their relevance to the nature of the
  phenomenon being studied
• 4. a continual evaluation of the approach should
  be under-taken during the study.
  
  Corner (1990)

15
Range of Research Methods

• Experimental design
• Ethnography
• Case study
• Survey

16
Experimental Design

• Hypothesis testing
• Independent and Dependent Variables
• For example - Predictor method of instruction,
  Resulting differences math performance
• Sampling of Population
• Experimental and Controlled Conditions
• Random assignment

17
Experimental Research

• The researcher does something to the subjects or
  objects or research, and then attempts to
  determine the effects of these actions
• Reporting
• Careful description of sampling procedure
• Inferential statistics, effect size, and so on.

18
Ethnography

• Defined a picture of the way of life of some
  identifiable group of people
• Anthropology - doing fieldwork, going native
• Preoccupied with culture, and how people interact
  with each other
• Qualitative Methodology - Both a research process
  and a product
• Outcome an ethnographic account

19
Ethnographic Process

• The ethnographer is the primary research
  instrument
• One year or more in the field setting
• long enough to see a full cycle of activity
• For example, a full school year
• Tension and balance between involvement and
  detachment
• Outsiders broad and analytical perspective on
  group studied
• Insider view, familiarity, empathy,
  identification with group

20
Field Research Techniques

• An Inquiry Process of multiple methods
• Participant observation
• privileged, active participant
• passive observer
• Interviewing
• key informants, structured, unstructured
• groups, surveys and questionnaires
• Making and using records
• historical documents, archives, written records

21
Validity and Reliability of the Ethnographic
Account

• The satisfactoriness of the explanation is what
  counts, not the power of the method for deriving
  it.
• Significance is derived socially, not
  statistically

22
Case Study

• Understanding the intricate complexity,
  idiosyncrasy of one particular case
• investigation of a bounded system
• Some entity deemed worthy of close watch
• a single child, a single classroom, a single
  school, a single national program
• Goals
• Understand and report the uniqueness of
  individual cases (both commonalities and
  differences)
• Usually no attempt to represent case by single or
  multiple scores

23
Case Study Methods

• Similar to ethnographic field methods
• ASKING - Interviews
• Gather narrative and testimony
• WATCHING - Observations
• SEARCHING - Written records and artifacts
• Reporting
• Develop a conceptual structure, look for
  patterns, consistencies, repetitions, and
  manifestations pertinent to your research
  question(s)

24
Validity and Reliability

• There are many different stories to be told
• Different researchers have different questions to
  answer, different conceptualizations of the
  situation, and set different boundaries for the
  case
• Generalizability What is true of one case is
  often true about other cases
• Consistencies can be found - predictability
• How many cases are needed before patterns emerge?
  It depends...

25
Survey Research Methods

• Purpose and Goal
• Describe specific characteristics of a large
  group of persons, objects, or institutions
• Understand present conditions, rather than the
  effects of particular intervention (as in
  experimental research)
• Sample of Population
• Groups of interest are well defined and chosen
  using well defined rules
• Representativeness

26
Survey Methods

• Mail
• postage and printing costs, participation rate
• Telephone
• sampling, wage and time costs, participation
  rates
• Face-to-Face
• wage and time costs, participation rates, like
  structured interview
• Web-based
• anytime, anywhere, cost effective

27
Issues in Survey Construction

• Item (question) and scale construction
• Pilot Testing and revision
• Sampling procedures
• Analysis and reporting of results
• Generalizability
• Drawing conclusions about the conditions,
  attitudes, opinions, or status of a population of
  persons, objects, institutions, or other entities.

28
Nowaczyk and Underwood (1995) http//epaa.asu.edu
/epaa/v3n20.html

• "Possible Indicators of Research Quality for
  Colleges and Universities"
• Used qualitative method, focus group, to
  investigate research quality in higher education

29
Results

• Academics promoted the use of both quantitative
  and qualitative measures to report on quality
• QUANTITY OF
• Journal publications, conference presentations,
  books and book chapters, awards, grants, budget,
  and so on
• QUALITY OF
• Reputation of publication, reputation of granting
  agency, quality of conference, peer reviews of
  research programs,
• Quality of institutions that hire graduate
  students
• Societal benefit of research

30
Resources

• Jaeger, R. M. (1997). Complementary Research
  Methods for Research in Education, (2nd ed).
  American Educational Research Association
  Washington, DC.
• Edyburn, D. L. (1998). The Electronic Scholar
  Enhancing Research Productivity with Technology.
  Prentice-Hall Columbus, OH.
• Nowaczyk, R. H., Underwood, D. G. (1995).
  Possible indicators of research quality for
  colleges and universities. Education Policy
  Analysis Archives, 3(20). On-line. Available
  http//epaa.asu.edu/epaa/v3n20.html
• Bowen, K. A. (1996). The Sin of Omission
  -Punishable by Death to Internal Validity An
  Argument for Integration of Qualitative and
  Quantitative Research Methods to Strengthen
  Internal Validity. On-line. Available
  http//trochim.human.cornell.edu/gallery/bowen/hss
  691.htm