President's Breakfast on9/29/2000

1
Session 3 Framework for a Coding
Guide October 27, 2009 200-330 PM
(EDT) Instructor Chad Nye
2
Dr. Chads Wonder Drug
Lifes Elixir of Youth, Beauty, Wisdom
Painless Coding
3

• A wholesome blend of natural extracts of thirty
  different herbs and root vegetables that promotes
  and revitalizes the capacity for learning and
  memory, strengthens the nerves, and effectively
  wards off cataleptic neuroplexy AND coding
  anxiety.

4
Inclusion and Exclusion Criteria

• Reduce Reviewer Bias Criteria are determined
  before beginning Information Retrieval
• Standardize the process and resulting decisions

5
Inclusion and Exclusion Criteria Potential
Categories

• Study Design
• Population
• Intervention
• Outcomes
• Measures
• Publication Sources

6
Coding for Fun and Profit

• Dr. Chads Wonder Drug taken 2 times a day will
  result in the complete life experience of the
  life well lived!!!

7
Coding is a 2 Step Process

• Stage 1 Title and Abstract
• Eliminate studies that are clearly inappropriate
• Used a limited set of criteria

8
Inclusion CriteriaAbstract Title Stage

• Studies will meet at least one of the following
  criteria
• 1. Participant
• The participant sample of study must include only
  adults 18 years or older, with a diagnosis of
  Autism Spectrum Disorder.
• 2. Intervention
• The focus of the study intervention must center
  on the topic of employment.

9
Inclusion CriteriaAbstract Title Stage

• Studies can be advanced to
• the next stage if
• They meet the criteria OR
• If these criteria are not clear from the title or
  abstract

10
Coding is a 2 Step Process

• Stage 1 Title and Abstract
• Eliminate studies that are clearly inappropriate
• Used a limited set of criteria
• Stage 2 Full Text
• Identify the studies that will be included in the
  review itself
• Use full set of criteria

11
Inclusion CriteriaFull Text Stage

• Research Design
• Studies use an experimental (RCT),
  quasi-experimental design (QED), and single
  subject experimental design (SSED)
• Participants
• Individuals with an autism spectrum disorder who
  are 18 years of age or older and not enrolled in
  a secondary school program (e.g., high school)

12
Inclusion CriteriaFull Text Stage

• a. Research Design
• b. Participants
• c. Intervention
• The treatment must be directed towards the
  provision of vocationally oriented services
  intended to produce employment outcomes for
  individuals with autism spectrum disorders.

13
Inclusion CriteriaFull Text Stage

• a. Research Design
• b. Participants
• c. Intervention
• d. Outcome Measures
• The study must report subsequent attainment of an
  employment placement and provide specific data
  about the duration and/or retention of that
  placement. Employment will be considered to
  encompass full or part-time (25 time or more)
  placements. Interventions of any length duration
  will be included provided adequate description is
  provided.
•  

14
Inclusion CriteriaFull Text Stage

• a. Research Design
• b. Participants
• c. Intervention
• d. Outcome Measures
• e. Publication Status
• Eligible studies may be either published or
  unpublished, written in any language, and
  conducted in any country.

15
Inclusion/Exclusion Decision Tree
Potentially relevant reports identified (n583)
Reports excluded by title and abstract
review (n523)
Reports progressing to next level (n60)
Reports excluded at Phase II screening (n40)
Reports used to code studies for inclusion (n20)
16
Study Coding

• Purpose of Coding
• 1. Identify all those variables that you believe
  might make difference in the observed results.
• 2. Organize those individual variables into
  categories of characteristics in order to
  evaluate the quality of the study
• 3. Try to determine what is really important
  about the study
• 4. Quantify both text and data for analysis

17
Coding Form

• 5. Guide for locating and recording the who,
  what, where, how, when, and how much of a study
  (aka-variables)
• 6. A record of the interpretation of the
  variables that can be independently replicated
  (aka-a monitor on the accuracy of judgments)
• 7. Provides the basis for the data arrangement
  at the analysis stage

18
Study Coding

• Categories of Study Characteristics
• Design
• Participant
• Treatment
• Outcome

19
Design Characteristics

• Type of Design
• RCT
• QED
• Pre-Experimental
• Single Subject
• Blinding
• Attrition
• Sampling Method

20
Participant Characteristics

• Information is collected for BOTH Experimental
  AND Control groups.
• Number of Participants
• Age
• Gender
• Severity
• Language
• Setting

21
Treatment Characteristics

• Type of Tx Method
• Length of Tx Program
• Tx Mode Group-Individual
• Number of Tx Sessions
• Length of Tx Sessions
• Dosage

22
Outcome Characteristics

• Outcome Variable(s)
• Outcome Measure(s)
• Assessor of Outcome
• Assessor Qualifications
• Time Post Tx to Outcome Measure
• Immediate
• Follow-Up

23
CodeBook

• A dictionary of all the terms used in the Coding
  Form
• A guide for all coders to follow to reduce error
  and bias
• A reference source for accurate replication

24

• A. Publication Source
• 1. Journal 2. Dissertation 3.Thesis
• 4. Unpublished Paper 5. Book 6. Other
  ________

B. Subject Characteristics (page(s)
____________) Group Pretest Posttest
Attrition Age Grade(s) Males
(n) (n) (n) (n) (n)
(n) Tx _22___ _22__
______ ______ ______ _____ Cp
______ _____ ______ ______ ______
_____ Co _22___ __18_ __ 4___ ______
______ _____
25

• C. Socioeconomic Status (page(s)___________
• Low 2. Middle 3. Upper
• 4. Low-Middle 5. Middle Upper 6. Labeled Mixed
• Unclear 8. ___________ 9. NR
• Comments__________________________________
  
  

26

• D. Sample Source (page(s)_________)
• Public School
• 2. Secular Private School
• 3. Religious Private School
• 4. Other_________________
• 5. NR
• Comments

27

• E. Study Setting (page(s) ______)
• Urban 2. Suburban 3. Rural
• Geographic Setting ______________________
• Comments

28

• F. Authors Labeling of Participants Achievement
  Levels (page _____)
• Average Achieving 2. High Achieving 3.
  Underachieving
• 4. At-Risk 5. Unclear
• 6. Other___________________ 7. NR
• G. Race/Ethnicity (page ______)
• African American 2. American Indian 3. Asian
• 4. Hispanic/Latino 5. White
• 6. Other ____________________ 7. NR

29

• H. Intervention Characteristics
• Type of intervention (page __________________)
• 2. Total length of intervention program
  __________ or NR (page _____)
• 3. Length of time PER intervention
  session________ or NR (page _____)
• 4. Number of sessions ________ per ________
  or NR (page _____)
• 5. Describe the intervention(s) intended for
  delivery to the study groups including how and
  when it was administered._________________________
  _____________

30

• Fidelity of implementation data reported?
  (page(s)______)
• 0No 1Yes
• If yes, describe the intervention(s) intended
  for delivery to the study groups including how
  and when it was administered
• _________________________________________________
  __________________________________________________
  __________________________________________________
  ___________________________________________
• 7. Intervention implemented as described?
  (page(s) _____)
• 1No 2Yes 3 Somewhat 9NR

31

• Outcome Measure(s)

Type of Measure Instrument Variable(s) Who Administered the Measurement Instrument?
1. Norm Referenced Test (page____) ___________________________ Parent b. Teacher c. Researcher d. Other e. NR
2. Criterion Referenced Test (page____) __________________________ a. Parent b. Teacher c. Researcher d. Other e. NR
4. Rating/Survey (page____) _________ _________ a. Parent b. Teacher c. Researcher d. Other e. NR
5. Class Grade (page____) ___________________________ a. Parent b. Teacher c. Researcher d. Other e. NR
32

• J. Design Characteristics
• Recruitment Pool (page_____)
• Referral 2. Criterion 3. Norm Reference Test
• 4. Existing Group 5. Cohort 6. Volunteer
• Other ___________________ 99. NR
• Research Design Type (pg____)
• 1. Group (Cluster, Place-based) Randomized
  Design
• 2. Individual Randomized Design
• 3. Quasi-Experiment Interrupted Time Series
  Design
• 4. Quasi-Experiment Regression Discontinuity
• 5. Quasi-Experiment Nonequivalent Comparison
  Group Design
• 6. Single Subject Design
• 7. Other Designs ________________________________
  _____

33

• K. Assignment Procedures (If applicable,
  otherwise skip to section I)
• Subject Assignment (page(s) _____)
• Individual Random
• Group Random
• Individual Match Random
• Quasi-random

34
2. Method of Random Allocation (page(s) _____)
Method of Random Allocation Who Implemented the Random Allocation?
1. Random Numbers (page____) Parent b. Teacher c. Researcher d. Other e. NR
2. Coin Flip (page____) a. Parent b. Teacher c. Researcher d. Other e. NR
4.Other_______(page____) a. Parent b. Teacher c. Researcher d. Other e. NR
35

• Was concealment of allocation used as part of
  the
• randomization procedures?
• 1no 2 yes 3NR
• a. If yes, were the allocation procedures
  concealed from
• any of the following people?
• Researcher (pg _____)
• 2. Participant (pg _____)
• 3. Intervener (pg _____)
• 4. Assessor (pg _____)

36

• Quasi-Experimental Assignment Procedures (If
  applicable, otherwise skip to section J)
• 1. Was any of the following research design
  techniques used to equate groups prior to the
  intervention?
• a. Matching on pretest proxy (pg _____)
• b. Matching on a pretest (pg _____)
• c. Matching on other variables (pg _____)
• d. No Matching (pg
  _____)
• 2. Was statistical equating used prior to the
  intervention? 1no 2yes 3.NR
• Comments________________________________________

37

• J. Effect Size Characteristics
• Comparisons Group 1 _TX___vs. Group 2_Ctl___
• Outcomes _Reading___ __________
  ___________
• Groups __1__ _2_ __1__ __2__ __1_ __2__
• Mean _22__ __18_ _____ _____ ____
  _____
• SD _3.55 _4.22_ _____ _____ ____ _____
• N at Posttest _____ _____ _____ _____ ____
  _____
• F value _____ _____ _____ _____ _____
  ____
• Chi-square _____ _____ ___ _____ _____ _____
  ____
• p value _____ _____ _____ _____ _____ ___
  __
• t value _____ _____ _____ _____ _____ __
  ___

38
Method of Analysis (pg _____) a. Intention to
Treat b. Treated Participants Only
39
Excerpt from Master Coding Guidefor Study Coding
------------------------------------- Code the
Fantuzzo Study
40
Data Extraction Practice