Everything I wish I had known about research design and data analysis

1
Everything I wish I had known about research
design and data analysis

• Statlab Workshop
• Fall 2006
• Kyle Hood and Frank Farach

2
Outline of a paper

• Introduction
• Theory
• Data Description
• Analysis
• Conclusion

3
Identifying a Question

• Tradeoff between work in and results
• Easy to do, trivial results
• Result is interesting, but difficulty is high
• New tools open up new questions
• New statistical or computational tools make
  formerly difficult questions approachable
• New theory opens up new questions

4
Introduction

• Topic
• Most general level
• Question
• What is the question you want to answer?
• Be specific
• Ask only what you can answer
• Review the Literature
• Stay the course

5
Theory

• Categorize your theory
• Descriptive vs. causal
• Write down your theory
• In paragraph form
• Using a statistical model
• Identify testable hypotheses from theory
• Do you need statistics after all?
• Quantitative v. Qualitative research

6
Variables

• Dependent Variable (response, outcome, criterion)
• Independent Variables (explanatory or predictor
  variables)
• Treatment Variable
• Covariates / Confounding Variables
• Categorical and Continuous Variables
• Remember Types of variables we choose, determine
  the statistics we use

7
You need Data

• Think about analyses early!
• Collecting your own data
• Retrospective, prospective, experimental
  observational methods
• Can find most data youll need on-line!
• Statlab Webpage (http//statlab.stat.yale.edu)
• Advisors
• Yale StatCat (http//ssrs.yale.edu/statcat/)
• ICPSR (http//www.icpsr.umich.edu)
• Reference Librarian (Julie Linden)

8
Endogeneity

• Problem Independent variables are not really
  independent
• The dependent and independent variables are
  determined in equilibrium (example effect of
  education on wages)
• Treatment effects will be biased
• Modeling approaches to deal with this
• Assumption-based methods
• Instruments

9
So, you want to make a survey

• Extensive on-line resources and software
• Question types determine analyses
• Open vs. close ended questions, Likert scales,
  rank order data
• Assumptions of normality
• Validity
• Internal External validity
• Pilot testing
• You need variance to analyze!
• Sample size
• It depends power, effect size, cost (UCLA power
  calculator)

10
Once Youve Found or Collected your data

• Download the data and documentation
• StatTransfer (Statlab)
• Determine data file type
• Probably a text file (.txt, .dat, .raw)
• Converting text delimited files
• Choose a statistical software program
• SPSS, Stata, SAS, Matlab, Excel, R, C

11
Managing your data

• Back up all Master Data Files
• CDR/CDRW, USB Key
• Codebook
• All codes
• Adding variables, cases, computing new variables
• Keep a roadmap
• Keep a log of all analyses with what you have
  done
• Save syntax files

12
Syntax Files

• What are they?
• Text-files used to enter commands in bulk
• Is it worth learning?
• You will make mistakes, need to make changes
• SPSS and many other programs let you use pull
  down menus
• How do I know what to write?
• Programs manual provides the underlying command

13
So, how do I analyze my data?

• Correlation
• Correlation allows you to quantify relationships
  between variables (r, r-squared)
• Regression allows prediction of dependent
  variable based on one or more independent
  variables
• Group differences
• t-test ANOVA
• Chi-square for categorical and frequency data
• Significance v. effect size
• More Complex Models

14
Descriptive Statistics

• Variables
• Dependent Variable(s)
• Independent Variable(s)
• Important Covariates
• Graphs
• Summary Statistics on Key Variables
• Number, Mean, Minimum, Maximum, Standard
  Deviation
• Cross-Tabs

15
Putting Output into a Paper

• Cut and Paste
• Graphs
• Cut and Paste into Word Processing document
• Save as .jpeg or .tif file
• Tables
• Cut and Paste
• Format in Word Processing document
• Import into Excel, format, and then place in Word

16
More Advanced Analysis

• Multivariate techniques help to account for
  confounding factors, allow for testing change
  over time and more complex hypotheses
• (See Tabachnick Fidell, Using Multivariate
  Statistics)
• Be honest about your abilities.
• Ask for help
• Best off including techniques that you fully
  understand.

17
Take Away Messages

• Determine your question, methods and statistics
  before you start
• Keep a codebook of everything
• Keep a log of all commands issued
• Save data at every step
• Ask for help
• Dont get in over your head