Introduction to Research Methodologies and Internal Validity

1
Introduction to Research Methodologies and
Internal Validity

• Matthew A. Goto
• CSC426 Values in Computer Computer Technology
• 2009/04/21

2
Overview

• Research Process Overview
• Overview of Research Methodologies
• Overview of Validity Types
• Potential Challenges to Internal Validity
• Solutions to Internal Validity Challenges
• Discussion/Activity

3
Research Process Overview

• Find a Question and Formalize the Problem
• Make a Hypothesis/Research Question
• Research Design aka The Plan!
• Collect Data
• Analyze the Data
• Write up Results and Publish

4
Research Methodologies Overview

• What is a Research Methodology?
• It is a way to find meaning from/in data.
• Sometimes data forces the research methodology,
  e.g. historical data
• Qualitative
• Attempt to describe/understand a phenomena
• Quantitative
• Attempt to predict/explain a phenomena

5
Validity Types

• Reliability the consistency that an instrument
  gives the same result when the thing measured
  doesn't change
• Validity the extent that an instrument measures
  what it is attempting to measure
• Instrument Validity the extent that a research
  tool provides results which are true. E.g. face,
  content, criterion, construct validity
• External Validity the extent that results can be
  generalized to other phenomena
• Internal Validity the extent that a research
  design and its data provide accurate views of
  relationships within the data a cause-effect
  relationship

6
Showing Internal Validity

• Must have
• Temporal Precedence Cause comes before effect
  can be tricky.
• Covariation of Cause Effect if Cause then
  Effect if not Cause then not Effect
• No other explanations this is the real tricky
  part!

7
Challenges to Internal Validity

• Basically, anything that could cause the result
  that isn't the hypothesized property/cause.
• Outside Threat maybe something outside the
  research caused the result History Threat,
  Maturation Threat
• Design Threat maybe something in the design
  caused the result Testing Threat, Instrument
  Threat, Data Threat
• Tangential Threat maybe something related to
  the research, but (potentially) out of your
  control caused the result Mortality Threat,
  Regression Threat

8
Challenges to Internal Validity

• Multi-Group Studies same as single group
  studies but also have to worry about
  comparability of groups
• Make sure that the groups are comparable prior to
  the research program. Otherwise, you have a
  selection threat or bias.
• Best To pick out comparable groups, select an
  entire group (the sample) and then randomly
  assign them to a group.
• Not Great, but Ok Pick out select the sample,
  then non-randomly assign each individual to a
  group to make the groups as equal as possible.
• Social Threats Look at http//www.socialresearchm
  ethods.net/kb/intsoc.php

9
Regression to The Mean

• Only a problem if non-random samples used and two
  measures are imperfectly correlated
• Prm 100(1 r),
• where Prm the of regression to the mean,
• r the correlation between the two measures

10
Ways to Improve Internal Validity

• Try to restrict alternate causes.
• Use random sampling of representative population
• Use a control don't use the program on 1 group
• Use a controlled laboratory study restrict
  environmental contaminations
• Use a double-blind experiment wrt comparing gt2
  methods, don't let the methods administrators and
  the participants know what is being hypothesized
• Unobtrusive measures don't let the participants
  know that you are measuring them
• Triangulation use multiple data sources to
  reduce the likelihood of alternate causes

11
Discussion/Activity

• A researcher hypothesizes that people prefer to
  use Firefox over IE. What research methodology
  should the researcher use and why?
• A researcher wants to determine which of three
  pieces of software people prefer and why they
  prefer it. What methodology is best suited for
  this research?

12
Discussion/Activity

• A researcher studies a group of randomly sampled
  6th grade students in the US to determine whether
  a teaching technique will improve understanding
  of a concept. The students are divided into a
  control group who get taught the concept using
  the traditional teaching technique and a second
  group who do get taught the concept using the new
  teaching technique. Students who received the
  new teaching technique did 25 better on the
  post-test than those who didn't receive the new
  technique. The researcher concludes that all
  students will do better to be taught by the
  teaching technique.
• What, if any, are the validity problems for this
  research?

13
Questions

• Any Questions?
• Thanks!

14
Bibliography

• Leedy, Paul D. and Ormrod, Jeanne Ellis.
  Practical Research Planning and Design 8th ed.
  Pearson Prentice Hall 2005.
• Research Methods Knowledge Base
  http//www.socialresearchmethods.net/kb. Taken
  on 2009/04/21.
• Picture on Regression to The Mean slide taken
  from http//www.socialresearchmethods.net/kb/regr
  mean.php 2009/04/21.