Title: Experimental Research I
1Experimental Research I
2Business
- For Tomorrow
- One article from Day 2 reading list.
- Prepare to summarize comment
- Examine 1 Thesis proposal online
- List of 12 research articles in APA format
related you your study (bibliography) - Reference List APA example
- Kotora, E. J. (2005). Assessment practices in the
choral music classroom A - survey of Ohio high school choral music teachers
and college choral methods professors.
Contributions to Music Education 32(2), 65-80. - Spaces b/w reference elements 1 b/w sentences
in text 2.
3Experimental Research
- Only type of research with an intervention
- A direct attempt to influence a particular
variable - Only method that can truly begin to untangle
cause and effect hypotheses - Directional Hypothesis Theory statement
predicting the outcome directional (There will
be a significant difference). Reflects
researchers expectations. - Bilingual 3rd graders taught with the Kodaly
method will demonstrate significantly higher
musical achievement than bilingual 3rd graders
taught with a traditional eclectic method.
4Null Hypothesis
- Null Hypothesis Theory statement predicting the
outcome stated in the negative non-directional
(There will be no significant difference) The
statistical hypothesis that states that there are
no differences between observed and expected
data. Does not reflect researchers expectations
(value free) - There will be no significant difference in
musical achievement of bilingual 3rd graders
taught with the Kodaly methods and bilingual 3rd
graders taught with a traditional eclectic
method. - The goal is to REJECT the Null Hypothesis based
(on 95 Confidence level or above) - Cannot prove the null hypothesis (a negative)
- E.g. not guilty does not innocent
- Reject/not reject vs. accept
5Type I and Type II Error
- Type I Error is erroneously claiming statistical
significance or rejecting the null hypothesis
when in fact, its true (claiming success when
experiment failed to produce results) - Possible w. incorrect statistical test
- Type II Error is when a researcher fails to
reject the null hypothesis when it is in fact
false (claiming failure when successful) - The smaller the sample size, the more difficult
it is to detect statistical significance - In this case, a researcher could be missing an
important finding because of study design
6Group Comparisons
- Experimental Group
- Receives a particular treatment specified by the
researcher - Control/Comparison Group
- Does not receive that particular treatment
- Sometimes difficult in educational research to
have a strict no-treatment, control - Example Any instruction is likely to be more
effective than no instruction
7Randomization
- Random assignment to groups
- Every individual has an equal chance of being in
the experimental or control/comparison group - Supposed to help eliminate extraneous sources of
variance - For example if the groups are sufficiently
large, any differences between groups on
extraneous variables are likely to be due to
chance or randomly distributed among the groups - Quasi-Experimentalnon-randomized groups
- Most ed. research
- Intact classes convenience samples
- Impacts ability to generalize to whole population
8Variables
- Independent variable (IV)
- The experimental or treatment variable
- This variable is manipulated by the researcher
- Examples instructional approach, environmental
condition, the introduction of a particular
musical element - Participant attribute
- Dependent variable (DV) Compared b/w groups
- The criterion or outcome variable
- Examples student attitudes, student achievement,
teacher effectiveness as measure by ? - Experiments can be expressed as The effect of
the IV on the DV - Extraneous Variables
- Those that are not specifically included in the
study but never the less may effect the outcome - Object is to control for extraneous variables
- The researcher may not know them all
9Manipulating the IV
- Presence of the variable vs. absence of the
variable - Kodaly instruction (treatment group) vs. no
Kodaly instruction (control group) - One form of the variable vs. another
- modeling vs. verbal music instruction (vs.
control group?) - Varying degrees of the same variable
- 100 positive feedback, no negative feedback vs.
50 positive feedback, no negative feedback
10Controlling for Extraneous Variables
- Best case scenario all individuals are as
similar as possible on all variables other than
the independent variable - Methods to control
- Randomization large sample
- Holding variables constant (freeze private
lessons) - Build variable into the design (compare private
lessons w/ no private lessons) - Matching pairs one to control, other to exper.
- Statistical control analysis of covariance
(ANCOVA)
11Design and ExperimentEffect of Colored note
heads on Music Reading
- State Hypothesis and Null Hypothesis
- Select sample and assign to group (control and
treatment). How many in each? - Identify independent and dependent variables. Any
possible extraneous variables? - Describe experiment. What will you do w/ each
group and for how long? How will you know what
they already know?
12Discussion of Projects
- On task
- Practice explaining your project
- Background State the problem
- Purpose statement
- Research questions
- Methodology (research design)
13Experimental Research Designs
14Nomenclature/Abbreviations
- When looking at the symbols used to describe
various experimental design approaches - R random assignment
- O testing (pre- or post-)
- X treatment
- C control/comparison
- M matched
15Pre-Experimental DesignsPilot Studies
Generally Weak
- One Shot Case Study (X O)
- No random assignment, No control/comparison, no
pre-test - One-Group Pre-test, Post-test (O X O)
- No random assignment, No control/comparison group
- Static/Intact-group Comparison X O
- No random assignment O
- Static/Intact-group Pre-test, Post-test
- No random assignment, possible pre-test effects O
X O - O
16True Experimental DesignsStronger not always
possible in educ.
- Randomized Post-test Only, R X O
- Control Group R O
- Still not sure about pre-test levels
- Randomized Pre-Test, R O X O
- Post-test, Control Group R O O
- Good for checking if groups are actually similar
at the start of the study and possible effects of
pretest - If you do an experiment probably this one
except w/ non-randomized groups
17Randomized Solomon Four-Group Design
- Solomon 4Group controls for possible
sensitization effects due to testing or
maturation. - 1. R O X O
- 2. R O O (maturation or pretesting?)
- 3. R X O (effect of pretest?)
- 4. R O (control group)
- In a successful experiment, what would we expect
for each group? - What if the Post Test scores for group 2 were as
high as the Post Test for group 1? - What if the Post Test scores for group 3 were
lower than group 1? - What if the Post Test scores for group 4 were the
same as groups 1 2?
18Quasi-Experimental Design
- So called b/c there is no randomization
- Matching Only
- Participants matched in pairs to control for an
extraneous variable rather than randomly assigned - Equivalent Materials Design (next slide)
- Counterbalanced Design (2 slides down)
- Multiple groups receive all treatment types in
different order - Average post-test scores across groups are
compared to determine effectiveness/effect of the
treatment order - Vulnerable to multiple-treatment interference
- Time-series Design
- Outcome measured several times before and after
introduction of the treatment O O O O X O O O O
19Equiv. Mat. Design TuningTuba vs. Clarinet vs.
oboe vs. drone stimulus
- Band (N 1) OX(tu)O OX(clar)O OX(ob)O
OX(drone)O - Pre/Post tests indiv. tuning to different
stimuli. Success mes. w/ a tuner. - Research questions Which tuning process leads to
most growth? - Most interested w/ growth w/i group per treatment
- Other questions Could introducing drone second
be more effective? How do we know that using just
one would be just as or more effective? next
slide
20Counterbalanced Design(Latin Square) Order effect
- All groups take a pretest tuning to the piano
A440 no difference in groups after time 1,
post test serves as next pretest - Each treatment 2 weeks w/ post-test at the end
- 1 tuba 2 clarinet 3 oboe 4 drone
21Quasi-Experimental Design
- Factorial Design (Manipulate 2 or more factors
IVs at different levels) - Allows for examination of attribute (vs.
manipulated) variables (i.e. gender, age) and
interaction effects b/w combinations of IVs - Example Effect of teaching method on beat
competency of ELL and non-ELL students - Possible outcome showing interaction of two IVs
- Non-ELL students may do equally well w/ Kod. and
Gordon methods, while bilingual students may do
better w/ Kod. vs. Gordon. - What if you had not separated these groups out?
22Factorial Example(Two Way - 2x2)
- Beat competency improvement using Gordon or
Kodaly among biling. and non-biling students - IVs Language classification (biling. vs.
non-biling.) method (Kodaly vs. Gordon) - DV rhythm pre- post-tests
- Groups (Six 3rd gr. Sections-3 Kodaly 3 Gordon
Bilingual Non-Bilingual in all groups.) - Bilingual Kodaly
- Non-Bilingual Kodaly
- Bilingual Traditional
- Non-Bilingual Traditional
232 Way Factorial Designs (2 independent variables
often one manipulated, one attribute)
24Internal Validity - Effectiveness of Exp.
DesignControl of Extraneous Variables Time
Bound Factors
- What happens within the experiment
- History What happens b/w pretest and posttest
(private lessons, change in practice routine) - Maturation is change result of treatment
natural result of repetition and improvement over
time?) - Mortality Loss of participants may cause
imbalance b/w groups
25Internal Validity Control of Extraneous
Variables Sampling Measurement Factors
- Testing pretest affect posttest. Ceiling and
floor effects (eliminate outliers?) - Instrumentation changes in measurement or
observers (judges at contest from one site to the
next) - Statistical regression students who score
extremely high (ceiling) or low (floor) on
pretest may regress to the mean on posttest - Selection participants do not represent normal
population (also affects external validity) - Interactions influence of a combination of the
above factors
26External Validity Generalizability
- Population Validity
- Extent sample is representative of the population
to which the researcher wishes to generalize the
results. - Ecological
- Study conditions and setting are representative
of the setting in which the researcher would like
to apply the findings (e.g. university lab
school) - Replication
- Results cannot be reproduced (problem w/ Mozart
effect) - Detailed description of the sample needed in
study - Important regardless of sampling method
27Other Threats to External Validity
- Effect or interaction of testing (testing will
not occur in natural setting - Reactive effects of sample
- Hawthorne Effect
- Effects due simply to subjects knowledge of
being in a study - John Henry Effect
- Control group performs beyond usual level because
they perceive they are in competition with the
experimental group - Teacher or Researcher interactions different than
in population - Subconsciously encouraging or discouraging a
group
28Review Effect of Intensive Instruction on
Elementary Students Memory for Culturally
Unfamiliar Music (2013)
- Previous researchers have found that both adults
and children demonstrate better memory for novel
music from their own music culture than from an
unfamiliar music culture. It was the purpose of
this study to determine whether this
enculturation effect could be mediated through
an extended intensive instructional unit in
another cultures music. Fifth-grade students in
four intact general music classrooms (two each at
two elementary schools in a large U.S. city) took
part in an 8-week curriculum exclusively
concentrated on Turkish music. Two additional
fifth-grade classes at the same schools served as
controls and did not receive the Turkish
curriculum. Prior to and following the 8-week
unit, all classes completed a music memory test
that included Western and Turkish music examples.
Comparison of pretest and posttest scores
revealed that all participants (N 110) were
significantly more successful overall on the
second test administration. Consistent with
previous findings, participants were
significantly less successful remembering items
from the unfamiliar music culture, a result that
was consistent across test administrations and
between instruction and control groups. It
appears that the effect of enculturation on music
memory is well established early in life and
resistant to modification even through extended
instructional approaches.
29Identify or State
- Independent Variable
- Dependent Variable
- Treatment Group
- Control Group
- Diagram experimental design (O X)
- Write a hypothesis null hypothesis
- Paraphrase findings
- Implications for the classroom? Did the authors
reject or not reject the null hypothesis?
30Sampling
31Samples of individuals/entities
- Sample vs. population
- Some vs. All
- Examples where entire population could be
sampled? - Relationship between sample specificity and
generalizability - Representative sample
- Captures relevant and essential characteristics
of the population - What about a sample of teachers? What should the
sample look like?
32Sampling Methods
- Systematic
- Random start and sampling interval
- i.e., Randomly select pages from IHSA directory
- choose every ? Name (random number b/w 1-X)
- Convenience
- not as valuable but frequent in ed. research
why? - i.e., intact classes, pre-service teachers from
one institution, conference session attendees - Purposive
- Participants fit a particular profile (female
band directors in small towns) - Exclude those who do not fit profile
- Often consists of volunteers (problematic)
33Types of Samples
- Simple Random
- Everyone has equal chance of selection
- Reduce systematic bias error created by
sampling method - Phone book, MENC membership list (But??)
- Stratified Random
- Similar proportions between sample and population
- Gender, race, age, instrument, etc.
- Cluster Random
- Groups rather than individuals
- i.e., classes or ensembles in CPS
- Then groups can be assigned randomly
- Two-stage random - groups then individuals
- i.e., choose classes then assign individual
students or groups to control or treatment group
34Sample Size
- As large as possible given reasonable expenditure
of time and energy - Most likely to get significant results
- More statistically powerful (more likely to find
a significant difference b/w groups) - Sample size relative to
- the size of population (50 Cook Co. band
directors vs. 50 band students throughout US) - variability within population (years of teaching,
gender, etc.) - sampling method (need a large enough pool from
which to draw) - study design (qualitative vs. quantitative)