Guiding Principles of Scientific Research

1
Guiding Principles of Scientific Research

• G. Reid Lyon, PhD
• President and CEO
• Synergistic Education Solutions
• Dallas, TX

2
Scientific Research

• A process of rigorous reasoning based on
  interactions among theories methods, and
  findings
• Builds on understanding derived from the
  objective testing of models or theories
• Accumulation of scientific knowledge is
  laborious, plodding, circuitous, and indirect
• Scientific knowledge is developed and honed
  through critique contested findings, replication,
  and convergence
• Scientific knowledge is developed through
  sustained efforts
• Scientific inquiry must be guided by fundamental
  principles.

3
Fundamental Principles (NRC, 2002)

• Ask significant questions that can be answered
  empirically.
• The formulation of a problem is often more
  essential than its solution, which may be merely
  a matter of mathematical or experimental skill.
  To raise new questions, new possibilities, to
  regard old questions from a new angle, requires
  creative imagination and marks real advance in
  science (Einstein Infeld, 1938)
• The research questions must be asked in a way
  that allows for empirical investigation.

4
Fundamental Principles (contd)

• II. Link research to relevant theory.
• Scientific research can be guided by a conceptual
  framework model, or theory that generates
  questions to be asked or answers to the questions
  posed
• Theory drives the research question, the use of
  methods, and the interpretation of results.

5
Fundamental Principles (contd)

• III. Select and apply research designs and
  methods that permit direct investigation of the
  question.
• The trustworthiness of any research study is
  predicated initially on several major elements
• The suitability of the proposed research design
  or methodology to address the specific questions
  posed by the study
• The scientific rigor by which the methodology is
  applied

6
Fundamental Principles (contd)

• The trustworthiness of any research study is
  predicated initially on several major elements
  (contd)
• The link between question and methodology must
  
• be clear and justified
• Detailed description of the method, measures,
  data collection procedures, data analyses, and
  subjects must be available to permit
  replication.

7
Fundamental Principles (contd)

• IV. Provide a coherent and explicit chain of
  reasoning that can be replicated.
• What assumptions underlying the inferences were
  made? Were they clearly stated and justified?
• How was evidence judged to be relevant?
• How were alternative, competing hypotheses, and
  explanations identified, considered, and
  accounted for (accepted or discarded)?

8
Fundamental Principles (contd)

• IV. Provide a coherent and explicit chain of
  reasoning that can be replicated (contd).
• How were the links between data and the
  conceptual or theoretical framework made?
• The chain of reasoning depends upon the design
  which depends on the type of question
• Description what is happening?
• Cause is there a systematic effect?
• Process/mechanism- why or how does the effect
  occur?

9
Fundamental Principles (contd)

• V. Replicate and generalize across studies.
• Internal Validity The observations made are
  consistent an generalize from one observer to
  another, from one task to a parallel task from
  one measurement occasion to anther occasion.
• Statistical methods e.g. correlation
• Non-statistical methods e.g. triangulation,
  comparative analysis.
• External Validity The extent to which the
  treatment conditions and participant population
  reflect the world to which generalization is
  desired.

10
Fundamental Principles (contd)

• VI. Report research publicly to encourage
  
• professional scrutiny, critique and
  replication.
• Criticism is essential to scientific progress
• The extent to which new findings can be reviewed
  contested, and accepted or rejected by scientific
  peers depends upon accurate, comprehensive, and
  accessible records of
• Data
• Methods
• Inferential reasoning

11
Common Conceptions/Misconceptions About
Scientific Quality and Rigor

• Experimental research is more scientific than
  descriptive or qualitative research
• NOT TRUE The type of design/method does not
  render the study scientific.

12
Common Conceptions/Misconceptions About
Scientific Quality and Rigor (contd)

• A study is deemed to be scientific when
• There are a clear set of testable questions
  underlying the design
• The methods are appropriate to answer the
  questions and falsify competing hypotheses and
  answers
• The study is explicitly linked to theory and
  previous research
• The data are analyzed systematically and with the
  appropriate tools
• The data are made available for review and
  criticism.

13
Common Conceptions/Misconceptions

• Research in education is fundamentally different
  than in the hard sciences.
• NOT TRUE Scientific research in education,
  psychology, biochemistry, astrophysics, cultural
  anthropology, mathematics, etc., all
• Seek conceptual/theoretical understanding
• Pose empirical and testable and refutable
  hypotheses
• Design studies that test and rule out competing
  counter hypotheses
• Use observational methods that are linked to
  theory and can be publicly assessed for accuracy
• Recognize the importance of independent
  replication and generalization.

14
Features Common To Educational Research

• It is a challenge to design and implement
  randomized treatments and repeated measures under
  highly controlled conditions
• Double-blind controls are not feasible when
  using particular designs
• The level of certainty of research conclusions is
  lower than in the physical sciences
• Error limits associated with scientific
  inferences are larger in social, behavioral, and
  educational research than in the physical
  sciences
• The influential role of context makes
  interpretation of data messy
• Converging evidence is critical.

15
Some Reasons Why Educational Practices and
Policies Are Not Guided and Informed by Science

• Education Research is young 100 year history
• Battered by different epistomological
  perspectives
• Skepticism Concerning the value of a science of
  education
• NIE should conduct an evaluation on effectiveness
  of instructional programs emphasizing an
  ethnographic or descriptive case study approach
  because the audience for follow through
  evaluations is an audience of teachers that
  doesnt need statistical findings of experiments
  to decide how best to teach children. They decide
  such matters on the basis of complicated public
  and private understandings, beliefs, motives, and
  wishes. (Gene Glass, 1981)

16
Some Reasons Why Educational Practices and
Policies Are Not Guided and Informed by Science

• Lack of Federal financial support for educational
  research
• Lack of public support for education research
• Inadequate translation of trustworthy research
  findings into applied classroom practices
• Teacher reliance on practical experience rather
  than data
• Expertise based on subjective judgments of the
  individual professional rather than student
  learning and achievement

17
Some Reasons Why Educational Practices and
Policies Are Not Guided and Informed by Science
(contd)

• Tendency to embrace fads in instruction rather
  than identify and implement policies and
  instructional practices based on data
• DAP was never seen as needing to be exclusively
  or even primarily based on research
  literatureFolklore and personal accounts of best
  practices passed on from one generation of
  teachers to the next counted a great dealThe
  types of citations used to reference the NAEYC
  publications of DAP guidelines clearly indicate a
  reliance on sources other than articles reporting
  original empirical data (i.e. bona fide
  research)only 13 of 25 references cited in the
  DAP report were original reports of research.
  (Kontos, 1989)
• Limitations in research training.

18
Principals For Fostering Science In Education
What The Federal Government Can Do

• Provide highly experienced research leadership
  and research management
• Develop rigorous and transparent peer-review
  processes
• Insulate the program from political interference
• Develop and manage coherent research
  programs/portfolios that incorporate
  agency-initiated programs, investigator/field-init
  iated programs, and multiple funding mechanisms
• Adequately fund the agency
• Stress trans-agency initiatives to increase
  community of researchers.