Some Key Questions about you Data

1
Some Key Questions about you Data

• Damian Gordon

2
The Data

• If the data is the key consideration in your
  research (although not all projects will
  necessarily be concerned with large datasets) it
  is important to consider several questions for
  those projects that do.

3
Overview

• What is the type of the data?
• Where will you get it from?
• What format is it in?
• How much cleaning is required?
• What is the quality of the data?
• How do you deal with missing data?
• etc.

4
The Data

• What is the type of the data?
• Federated data
• High dimensional data
• Descriptive data
• Longitudinal data
• Streaming data
• Web (scraped) data
• Numeric vs. categorical vs. text data

5
The Data

• Where will you get it from?
• http//www.kdnuggets.com/datasets/
• http//www.google.com/publicdata/directory
• http//opendata.ie/
• http//lib.stat.cmu.edu/datasets/
• http//factfinder.census.gov/servlet/DatasetMainPa
  geServlet

6
The Data

• What format is it in?
• ACSII
• TXT
• MIME
• XML
• CSV

7
The Data

• How much cleaning is required?
• Parsing
• Correcting
• Standardizing
• Matching
• Consolidating

8
The Data

• What is the quality of the data?
• Frequency counts
• Descriptive statistics (mean, standard deviation,
  median)
• Normality (skewness, kurtosis, frequency
  histograms, normal probability plots)
• Associations (correlations, scatter plots).

9
The Data

• How do you deal with missing data?
• Imputation
• Partial imputation
• Partial deletion
• Full analysis

10
The Data

• How will you evaluate the data?
• Area under the Curve
• Misclassification Error
• Confusion Matrix
• N-fold Cross Validation
• Test predictions in real-world

11
The Data

• Other questions?