Experimental Research Designs have Two Purposes:

1
Experimental Research Designs have Two Purposes

• to provide answers to research questions
• ...to control variance (differences)

2
The main function of the experimental research
design is to control variance.

• Principle maximize systematic variance, control
  extraneous systematic variance, and minimize
  error variance.
• MAX-MIN-CON

In other words control variance.
3
Therefore the researcher attempts to

• maximize the variance of the variable(s) of the
  research hypothesis (i.e., maximize the
  difference in the dependent variable outcome
  caused by maximizing the differences in the
  independent variable treatment).
• control the variance of extraneous or "unwanted"
  variables that may have an effect on the
  experimental outcomes, but which he/she is not
  interested (limit factors other than the
  treatment (IV) that could be causing differences
  in the outcome (DV) .
• minimize the error or random variance (i.e.,
  avoid unreliable measurement instruments which
  have high errors of measurement ).

4
Maximization of Experimental Variance

• experimental variance
• the variance due to the manipulated (i.e.,
  treatment) or attribute (i.e., gender) variables
  (IV)
• research precept
• design, plan and conduct research so that
  experimental conditions are as different as
  possible on the independent variable.

5
Control of Extraneous Variables (EV)

• eliminate the variable (i.e., if you are worried
  about gender, only include one
  gender in the study).
• randomization (i.e., if you randomly assign
  subjects to groups, the extraneous variable
  should be equally distributed among the groups)
• build it into the design make it a moderator
  variable (i.e., if you are worried about gender,
  build it into the analysis 2-way ANOVAwell
  learn about this later)
• match subjects (i.e., match the characteristics
  of subjects and put one of each matched pair in
  each group)
• statistically equate groups (i.e., use ANCOVA
  Analysis of Covariance to analyze the data with
  the extraneous variable used as a covariatewell
  learn about this later)

6
Minimizing Error Variance has Two Principle
Aspects

• reduction of errors of measurement through
  controlled conditions (i.e., standardize testing
  procedures)
• increase in the reliability of measures (i.e.,
  revise test instruments or find more reliable
  ones)

7
Experimental Designs Should be Developed to
Ensure Internal and External Validity of the Study
8
Internal Validity

• Are the results of the study (DV) caused by the
  factors included in the study (IV) or are they
  caused by other factors (EV) which were not part
  of the study?

9
INTERNAL VALIDITY (KESAHAN DALAMan HASIL KAJIAN).

• Does the treatment affect the dependent varibale
  or so other things affect the dependent
  variable??
• Misalnya Pelajar Yang Diajar Dengan Kaedah
  Inquiri Mendapat Keputusan Yang Lebih Tinggi
  (Test On Critical Thinking) Berbanding Pelajar
  Yang Diajar Dengan Kaedah Lecture.
• Is the difference on achievement due to method of
  teaching (inquiry vs lecture methods?
• What if they are better critical thinkers to
  begin with?
• What if they also taking related course?
• What if the teacher is a better teacher than the
  other teacher?

10

• Hipotesis For educationally handicapped
  students, failure rate is related to amount of
  disruptive behavior. High correlation, so what?
  Can it be because of something else such as
  educational ability? Observed difference is
  directly related to the variables under
  investigation to be INTERNALLY VALID.
• kajian menunjukkan wujudnya korelasi yang tinggi
  (r.8) di antara ketinggian pelajar dan skor
  pencapaian dalam ujian matematik bagi pelajar
  sekolah rendah. Ini bermakna semakin tinggi
  pelajar semakin tinggi pula pencapaiannya. Apa
  ertinya disini? Bolehkah ia membawa perubahan
  kepada amalan di sekolah? Atau amalan pemakanan?

11
Threats to Internal Validity
There are 16 common threats to internal validity.
Subject Characteristics
(Selection Bias/Differential Selection) -- The
groups may have been different from the start. If
you were testing instructional strategies to
improve reading and one group enjoyed reading
more than the other group, they may improve more
in their reading because they enjoy it, rather
than the instructional strategy you used.
12

• SUBJEK KAJIAN SEMEMANGNYA BERBEZA DARI AWAL LAGI.
  PERBEZAAN ADALAH TIDAK RAWAK. Misalnya korelasi
  tinggi antara disruptive behaviour dan kadar
  kegagalan. Siapa subjek kajian? How about
  academik ability subject berkenaan? Lain-lain
  contoh ciri termasuklah umur, KEKUATAN,
  kematangan, gender, Ethnik, KECErDASAN, SIKAP,
  Kebolehan/Keupayaan dan banyak lagi.
• What to do? Pilih rekabentuk yang sesuai. Jika
  buat bandingan kumpulan pastikan kedua-dua
  kumpulan ada persamaan. Gunakan teknik statistiks
  yang sesuai bagi penyelidikan korelasi misalnya
  partial correlation

13
Threats to Internal Validity
Loss of Subjects
(Mortality) -- All of the high or low scoring
subject may have dropped out or were missing from
one of the groups. If we collected posttest data
on a day when the honor society was on field trip
at the treatment school, the mean for the
treatment group would probably be much lower than
it really should have been.
14

• Kesan yang serious jika kita buat perbandingan
  kumpulan (dua kaedah mengajar bagi dua kumpulan)
  apabila bilangan subjek hilang daripada satu
  kumpulan adakah lebih besar daripada kumpulan
  lain. (expect poor students to leave in every
  group) but if gt in one group than the other? One
  of the methods may appear more effective than the
  other. Paling sukar dikawal!!!!!!

15

• How to overcome?
• Berikan maklumat bahawa subjek yang terciri
  adalah serupa seperti subjek yang ada. How?
  Pretest score, ciri-ciri demografi atau lain-lain
  variable yang ada kaitan dengan kajian.
• Cuba halang keciciran atau minimize the loss, but
  how?
• Masa kajian sesuai atau tidak?
• (hampir perayaan, waktu tamat sekolahan)
• Subjek kajian sesuai atau tidak?
• (senior students selected too? May be too many of
  them 20-30--control group)Effect of new diet on
  endurance
• Masa untuk menjawab soal-selidik panjang atau
  tidak?
• Soal-selidik senang difahami atau tidak?

16

• Mula di sini mula di sini untuk bintulu

17
Threats to Internal Validity
Perhaps one group was at a disadvantage because
of their location.  The city may have been
demolishing a building next to one of the schools
in our study and there are constant distractions
which interfere with our treatment.
Location
18
Threats to Internal Validity
The testing instruments may not be scores
similarly. Perhaps the person grading the
posttest is fatigued and pays less attention to
the last set of papers reviewed. It may be that
those papers are from one of our groups and will
received different scores than the earlier
group's papers
Instrumentation Instrument Decay
19
Threats to Internal Validity
The subjects of one group may react differently
to the data collector than the other group. A
male interviewing males and females about their
attitudes toward a type of math instruction may
not receive the same responses from females as a
female interviewing females would.
Data Collector Characteristics
20
Threats to Internal Validity
The person collecting data my favors one group,
or some characteristic some subject possess, over
another. A principal who favors strict classroom
management may rate students' attention under
different teaching conditions with a bias toward
one of the teaching conditions.
Data Collector Bias
21

• Distort the data in such a way to make certain
  outcomes such as support for the hypothesis.
• Allow more time to complete the test for some
  classes.
• Ask leading questions.
• Favoring one over the other (instructional
  methods)

22

• Report on absentism (tidak tanpa sebab tidak
  dikira tidak hadir dan tidak hadir dengan sebab
  dikira hadir). New definition of absentism.
  Menghasilkan kedatangan yang lebih baik daripada
  sebelumnya. What if the other school define
  differently? Then comparison is not valid.
• Suggest a possible answer during the interview.
• Awareness of which subjects are being
  experimented.

23

• How to overcome?
• Standardized all the procedures.
• Give training to data collectors
• Example of instrumentation threat
• Grading an essay test for a few hours without
  stopping.

24
Threats to Internal Validity
Testing
The act of taking a pretest or posttest may
influence the results of the experiment. The
pretest may have actually increased both groups'
sensitivity and we find that our treatment groups
didn't score any higher on a posttest given later
than the control group did. If we hadn't given
the pretest, we might have seen differences in
the groups at the end of the study.
25

• Berikan Pretest Matematik (Standard Test),
  Pelajar Dapat Markah Rendah. Kemudian Beri
  Treatment. Uji Dapat Markah Tinggi. Bolehkan Kita
  Katakan Ianya Disebabkan Oleh Intervention
  Berkenaan? Mungkin Tidak Boleh!!!
• PRETEST---MAKE STUDENTS MORE ALERT

26
Threats to Internal Validity
History
Something may happen at the site during our study
that influences the results. Perhaps a classmate
dies in a car accident at the control site for a
study teaching children bike safety. The control
group may actually demonstrate more concern about
bike safety than the treatment group.
27

• Apa yang berlaku semasa kajian boleh mempengaruhi
  hasil kajian berkenaan.
• Exposure to certain programs while participating
  in the study.
• Endurance test. While taking special diet, jog
  everyday.
• Democratic leadership practices improve
  commitment at the same time guru attend courses.

28
Threats to Internal Validity
There may be natural changes in the subjects that
can account for the changes found in a study. A
critical thinking unit may appear more effective
if it taught during a time when children are
developing abstract reasoning.
Maturation
29

• Pengalaman dan natural Kepantasan berlari ada
  hubung kaitnya dengan kematangan fizikal
  (perubahan fizikal). maturation.
• Misalnya latihan lumba lari untuk pelajar darjah
  enam. Jadi kita tidak boleh sewenang-wenangnya
  katakan kalau berlatih boleh
• lari pantas.
• Mungkin yang tak belatih pun boleh lari
• Pantas. Keupayaan befikir pun ada kaitan
• dengan kematangan. Strength pun ada
• kaitan dengan kematangan.

30
Threats to Internal Validity
Hawthorne Effect
The subjects may respond differently just because
they are being studied. The name comes from a
classic study in which researchers were studying
the effect of lighting on worker productivity. As
the intensity of the factory lights increased, so
did the worker productivity. One researcher
suggested that they reverse the treatment and
lower the lights. The productivity of the workers
continued to increase. It appears that being
observed by the researchers was increasing
productivity, not the intensity of the lights.
31
Threats to Internal Validity
One group may view that it is in competition with
the other group and may work harder than they
would under normal circumstances. This generally
is applied to the control group "taking on" the
treatment group. The terms refers to the classic
story of John Henry laying railroad track.
John Henry Effect
32
Threats to Internal Validity
The control group may become discouraged because
it is not receiving the special attention that is
given to the treatment group. They may perform
lower than usual because of this.
Resentful Demoralization of the Control Group
33
Threats to Internal Validity
Regression
(Statistical Regression) -- A class that scores
particularly low can be expected to score
slightly higher just by chance. Likewise, a class
that scores particularly high, will have a
tendency to score slightly lower by chance. The
change in these scores may have nothing to do
with the treatment.
34

• Apa sebenarnya regrasi ke arah min?
• Berlaku apabila skor rendah semasa pre-test akan
  bergerak ke arah min skor post-test. Begitu juga
  skor tinggi semasa pre-test akan bergerak ke arah
  min semasa post-test. Tidak berlaku perbezaan
  skor yang di sebebkan oleh rawatan yang diberikan
  When you are at the bottom you have no where to
  go. When you are at the top you have no where to
  go.

35

• Cook and Campbell (1979, pp52-53) explained
  regression toward the as a phenomenon that
• 1) operates to increase obtained pretest-posttest
  gain scores among the low pretest scores since
  this group's pretest scores are more likely to
  have been depressed by error
• 2) operates to decrease the obtained change in
  scores among persons with high pretest scores
  since their pretest scores are likely to have
  been inflated by error and
• 3) does not affect obtained scores among scorers
  at the center of the pretest distribution since
  the group is likely to contain as many units
  whose pretest scores are inflated by error as
  units whose pretest scores are deflated by it.

36

• Bila berlaku?
• Apabila subjek dipilih berasaskan skor extreme
  dalam pra-uji.
• Examples of this are students with the worst math
  ability, workers with the lowest morale or
  patients with the most severe symptoms.
  Regression toward the mean is expected in these
  cases where there is non-random sampling or
  assignment in experiments.
• Penyelesaian

37
Threats to Internal Validity
The treatment may not be implemented as intended.
A study where teachers are asked to use student
modeling techniques may not show positive
results, not because modeling techniques don't
work, but because the teacher didn't implement
them or didn't implement them as they were
designed.
Implementation
38
Threats to Internal Validity
Compensatory Equalization of Treatment
Someone may feel sorry for the control group
because they are not receiving much attention and
give them special treatment. For example, a
researcher could be studying the effect of laptop
computers on students' attitudes toward math. The
teacher feels sorry for the class that doesn't
have computers and sponsors a popcorn party
during math class. The control group begins to
develop a more positive attitude about
mathematics.
39
Threats to Internal Validity
Experimental Treatment Diffusion
Sometimes the control group actually implements
the treatment. If two different techniques are
being tested in two different third grades in the
same building, the teachers may share what they
are doing. Unconsciously, the control may use of
the techniques she or he learned from the
treatment teacher.
40
Control of Extraneous Variables (EV)

• eliminate the variable (i.e., if you are worried
  about gender, only include one
  gender in the study).
• randomization (i.e., if you randomly assign
  subjects to groups, the extraneous variable
  should be equally distributed among the groups)
• build it into the design make it a moderator
  variable (i.e., if you are worried about gender,
  build it into the analysis 2-way ANOVAwell
  learn about this later)
• match subjects (i.e., match the characteristics
  of subjects and put one of each matched pair in
  each group)
• statistically equate groups (i.e., use ANCOVA
  Analysis of Covariance to analyze the data with
  the extraneous variable used as a covariatewell
  learn about this later)

41
Once the researchers are confident that the
outcome (dependent variable) of the experiment
they are designing is the result of their
treatment (independent variable) internal
validity, they determine for which people or
situations the results of their study apply
external validity.
42
External Validity

• Are the results of the study generalizable to
  other populations and settings?

External validity comes in two forms population
and ecological.
43

• KESAHAN LUARAN
• SEJAUHMANA HASIL KAJIAN BOLEH DI GENERALIZE
  KEPADA POPULASI DALAM SEMUA KEADAAN. ADAKAH HASIL
  KAJIAN BENAR-BENAR MENGGAMBARKAN POPULASI?
  Apabila kajian dijalankan hasilnya dijangka dapat
  di aplikasi kepada keseluruhan populasi kajian di
  tempat mana sekali pun populasi itu berada.
  Tetapi perlu diketahui bahawa hasil kajian kita
  tidak semestinya boleh digenralize kepada semua
  populasi dalam semua keadaan. Keupayaan kita
  generalize hasil kajian banyak bergantung kepada
  sampel dan pesampelan yang dilakukan.

44
Threats to External Validity (Population)
Population Validity is the extent to which the
results of a study can be generalized from the
specific sample that was studied to a larger
group of subjects. It involves...

• ...the extent to which one can generalize from
  the study sample to a defined population-- If
  the sample is drawn from an accessible
  population, rather than the target population,
  generalizing the research results from the
  accessible population to the target population is
  risky.
• ...the extent to which personological variables
  interact with treatment effects-- If the study
  is an experiment, it may be possible that
  different results might be found with students at
  different grades (a personological variable).

45

• CUBA MENGGUNAKAN HASIL KAJIAN UNTUK MENGATAKAN
  BAHAWA HASIL KAJIANNYA JUGA ADALAH BENAR UNTUK
  POPULASI/POPULASI MEMPUNYAI CIRI YANG SERUPA
  SEPERTI SAMPEL.

46

• POPULASI/POPULASI MEMPUNYAI CIRI YANG SERUPA
  SEPERTI SAMPEL.
• SO, HOW LARGE (SIZE) IS THE SAMPLE?
• HOW REPRESENTATIVE IS THE SAMPEL?
• WHAT IS MEANT BY REPRESENTATIVE?
• REP.
• PEMBOLEHUBAH YANG BERKAITAN
• YANG DIJANGKA BOLEH MENENTUKAN HASIL KAJIAN

47

• BAGAIMANA TREATMENT (KAEDAH MENGAJAR) DIBERIKAN?
  ADAKAH SECARA RAWAK?
• ADAKAH ORANG YANG MEMBERI TREATMENT JUGA DIPILIH
  SECARA RAWAK? Adakah ia juga represeantive?
  Dipilih secara rawak?
• ADAKAH DATA DIPEROLEH DARIPADA SEMUA UNIT
  PESAMPELAN?
• APAKAH LANGKAH YANG DIAMBIL UNTUK MEMPASTIKAN
  RESPONDEN ADALAH SERUPA SEPERTI NON-RESPONDENT?

48

• BAGAIMANA JIKA PERSAMPELAN RAWAK TIDAK DAPAT
  DILAKSANAKAN?
• Kita akan sentuh mengenai sampel dan persampelan
  sekejap lagi
• TERANGKAN SEPENUHNYA SIAPAKAH SAMPEL KAJIAN
• (JANTINA, BANGSA, KEUPAYAAN, DAN SEBAGAINYA)
• REPLICATE KAJIAN BEKENAAN- ULANG KAJIAN
  MENGGUNAKAN SAMPEL YANG LAIN DALAM SITUASI YANG
  LAIN (SES, GEOGRAPHY,ABILITY DAN SEBAGAINYA)

49
Threats to External Validity (Ecological)
Ecological Validity is the extent to which the
results of an experiment can be generalized from
the set of environmental conditions created by
the researcher to other environmental conditions
(settings and conditions).
There are 10 common threats to external validity.
50

• SEJAUHMANA HASIL KAJIAN ADALAH BENAR DALAM
  SETTING YANG LAIN. MAKSUDNYA JIKA DIULANG KAJIAN
  ITU DALAM SETTING YANG LAIN adakah HASIL YANG
  SERUPA AKAN DIPEROLEH.
• jelaskan dengan sejelas-jelasnya tentang SETTING
  KAJIAN berkenaan (SPACIOUS VS CROWDED CLASSROOM).
  
• HASIL KAJIAN YANG DIPEROLEH DARIPADA URBAN AREAS
  SUKAR UNTUK DIGENERALIZE KEPADA RURAL AREAS.
• MISALNYA KOMITMEN GURU TERHADAP PROFESION
  PERGURUAN ADALAH TINGGI TETAPI SUBJEK KAJIAN
  ADALAH DARIPADA BANDAR (KEMUDAHAN CUKUP MISALNYA)
  MAKA KITA TIDAK BOLEH MENGANDAIKAN IANYA JUGA
  SERUPA DENGAN GURU DILUAR BANDAR
• So How Close Is The Experimental Setting To The
  Natural Setting?

51
(No Transcript)
52
(No Transcript)
53
(No Transcript)
54
(No Transcript)
55
Threats to External Validity (Ecological)
Explicit description of the experimental
treatment
(not sufficiently described for others to
replicate) If the researcher fails to adequately
describe how he or she conducted a study, it is
difficult to determine whether the results are
applicable to other settings.
56
Threats to External Validity (Ecological)
Multiple-treatment interference
(catalyst effect)If a researcher were to apply
several treatments, it is difficult to determine
how well each of the treatments would work
individually. It might be that only the
combination of the treatments is effective.
57
Threats to External Validity (Ecological)
Hawthorne effect
(attention causes differences)Subjects perform
differently because they know they are being
studied. "...External validity of the experiment
is jeopardized because the findings might not
generalize to a situation in which researchers or
others who were involved in the research are not
present" (Gall, Borg, Gall, 1996, p. 475)
58
Threats to External Validity (Ecological)
Novelty and disruption effect
(anything different makes a difference)A
treatment may work because it is novel and the
subjects respond to the uniqueness, rather than
the actual treatment. The opposite may also
occur, the treatment may not work because it is
unique, but given time for the subjects to adjust
to it, it might have worked.
59

• new instructional method (COMPUTER ASSISTED
  LEARNING) mungkin didapati lebih berjaya
  berbanding kaedah lama (PERBINCANGAN). Pada hal
  mungkin kedua-dua kaedah ini memberi kesan yang
  sama sahaja. Why it happens that way? New
  methodenthusiastic, old methodleft out .
• mungkin juga kaedah baru tidak berkesan, then
  why?
• Subjeck kajian belum cukup mahir dengan kaedah
  berkenaan sehinggalah ukuran ke atas pembolehubah
  bersandar di ambil (ujian diberikan). new
  instructional method (COMPUTER ASSISTED LEARNING)
  mungkin didapati lebih berjaya berbanding kaedah
  lama (PERBINCANGAN). Pada hal mungkin kedua-dua
  kaedah ini memberi kesan yang sama sahaja. Why
  it happens that way? New methodenthusiastic, old
  methodleft out .
• mungkin juga kaedah baru tidak berkesan, then
  why?
• Subjeck kajian belum cukup mahir dengan kaedah
  berkenaan sehinggalah ukuran ke atas pembolehubah
  bersandar di ambil (ujian diberikan).

60
Threats to External Validity (Ecological)

• (it only works with this experimenter)The
  treatment might have worked because of the person
  implementing it. Given a different person, the
  treatment might not work at all.

Experimenter effect
61
Threats to External Validity (Ecological)
Pretest sensitization
(pretest sets the stage)A treatment might
only work if a pretest is given. Because they
have taken a pretest, the subjects may be more
sensitive to the treatment. Had they not taken a
pretest, the treatment would not have worked.
62
Threats to External Validity (Ecological)
Posttest sensitization
(posttest helps treatment "fall into place")The
posttest can become a learning experience. "For
example, the posttest might cause certain ideas
presented during the treatment to 'fall into
place' " (p. 477). If the subjects had not taken
a posttest, the treatment would not have worked.
63
Threats to External Validity (Ecological)
Interaction of history and treatment effect
(...to everything there is a time...)Not only
should researchers be cautious about generalizing
to other population, caution should be taken to
generalize to a different time period. As time
passes, the conditions under which treatments
work change.
64
Threats to External Validity (Ecological)
Measurement of the dependent variable
(maybe only works with M/C tests)A treatment
may only be evident with certain types of
measurements. A teaching method may produce
superior results when its effectiveness is tested
with an essay test, but show no differences when
the effectiveness is measured with a multiple
choice test.
65
Threats to External Validity (Ecological)
Interaction of time of measurement and treatment
effect
(it takes a while for the treatment to kick
in)It may be that the treatment effect does not
occur until several weeks after the end of the
treatment. In this situation, a posttest at the
end of the treatment would show no impact, but a
posttest a month later might show an impact.
66
(No Transcript)
67
(No Transcript)
68
(No Transcript)
69
(No Transcript)
70
(No Transcript)
71
(No Transcript)
72
Types research designs

• Kajian eksperimen.
• ? pre-EKSperiment
• ? quasi-eksperiment
• ? true-eksperiment

73
Mengapa kajian eksperimen?

• most powerful
• boleh kenalpasti kesan treatment ke atas
  dependent variable
• can test the hypotheses of cause-and effect
  relationship

74
Ciri-ciri kajian eksperimen.

• Manipulasi independent variable
• ? nature of treatment-bentuk treatment misalnya
  kaedah pengajaran, kaedah penyeliaan, bentuk
  kaunseling
• ? Siapa yang menerima treatment (kumpulan pelajar
  yang sebagaimana perlu menerima kaedah mengajar
  yang sebagaimana)
• ? Extent of treatment-selama mana pelajar akan
  didedahkan dengan kaedah mengajar yang kita kaji

75

• Randomization
• Randomly assign subjects to the groups (a or b)
• Randomly assign teachers to the group (a or b)
• Randomly assign treatment to the group (a or b)
• Should happen prior to the start of the
  experiment
• Kawalan
• pembolehubah pengganggu (extraneous variables)
• treatment
• Tujuan ialah mengawal keadaan supaya kesan ke
  atas DV adalah hasil daripada treatmen yang
  diberikan

76

• Misalnya Kajian dijalankan untuk mengetahui
  kesan kursus motivasi peperiksaan ke atas
  pencapaian pelajar dalam peperiksaan PMR.
  Treatment (IV) ? kursus motivasi peperiksaan.
• Observation (DV) ? pencapaian dalam PMR
• Mungkinkah kursus motivasi yang menyebabkan
  pencapaian cemerlang atau faktor lain?
• Hipotesis nul  tiada hubungan antara kursus
  motivasi peperiksaan dan pencapaian dalam PMR.
• Hipotesis nul  kursus motivasi peperiksaan tidak
  mempengaruhi pencapaian pelajar dalam PMR
• What if you get a correlation of .90?. Isnt
  kursus motivasi mempengaruhi pencapaian? Well,
  not it may be or may not. DONT JUMP TO
  CONCLUSION RIGHT AWAY !!!! Have we done our study
  right?

77
(No Transcript)