Educational Research: Getting Started

1
Educational Research Getting Started
Steven B. Goldin Chief of Hepatobiliary and
Pancreatic SurgeryAssociate Professor of
SurgeryVice Chair of Surgical Education Chief of
Hepatobiliary and Pancreatic Surgery
Making Life Better
through Research, Education Healthcare
TM
2
NO DISCLOSURES EXCEPT Yes, they really are my
photos
3
(No Transcript)
4
Outline

• 1. Research Why do it?
• 2. What is educational research
• 3. Research How to get started
• 4. Features of rigorous studies
• 5. Pitfalls validity and reliability
• Surveys
• Ethics, funding, resources

5
Research why do it?
ASE No discipline can call itself a profession
that does not give top priority to research.
More specifically, the status of a given
profession depends on the adequacy of the
research activities, and the application of
research findings to routine professional
activities. (Mouly 1978)
6
Research- why do it?
Faculty advancement Teaching and research
separate academic surgeons from those in
community practice Develop a national
prominence/reputation May be a funding
source
7
Introduction
Importance of clinical productivity how do you
find the time to do a research project? Do
research with your daily activities Data
collection is common for all courses and
rotations. medical students residents facu
lty Innovation what makes an academic center
academic. What additional data would need to be
collected to begin a research project?
8
Introduction
Innovation and evaluation form the backbone of
research ideas and projects Many research
projects are done with students, residents, and
faculty Steady supply of potential
participants
9
Introduction
Traditionally empirical research (bench top) was
viewed as the most valid method. Other forms of
research (historical, phenomenological,
introspective, qualitative, etc.) contribute to
educational research Empirical research is
characterized by direct observation, objectivity,
falsification, replicability, parsimony and
quantification (scientific method) Parsimony
is defined as the adoption of the simplest
assumption in the formulation of a theory or in
the interpretation of data, especially in
accordance with the rule of Ockham's razor
10
Introduction
Occams razor (Ockhams razor) - entities should
not be multiplied unnecessarily. The
explanation of any phenomenon should make as few
assumptions as possible, eliminating those that
make no difference in the observable predictions
of the explanatory hypothesis or theory When
competing hypotheses are equal in other respects,
select the hypothesis that introduces the fewest
assumptions and postulates the fewest entities
while sufficiently answering the question
11
Evidence Based Medicine
Evidence based medicine interventions should be
guided by data from rigorous studies was
described at least 15 years ago Evidence-Based
Medicine (EBM) excellent framework for
evaluating the quality of clinical research EBM
is not focused on educational research, but
provides useful criteria for different categories
of research. (Treatment efficacy, accuracy of
diagnostic tests, clinical guidelines, prognosis
and risk, overviews and meta-analysis)
12
Evidence Based Medicine
Many EBM concepts are applicable to educational
research and should be familiar to all
educational researchers Medical educators,
likewise, should teach based on evidence
13
Introduction
Educational research is driven by the process of
documenting the steps of a project This process
is the same for all studies including bench
research, studying teaching methods, or student
attitudes Educational research focuses on
describing and documenting the transmission of
knowledge, competency, and understanding about
which teaching methods are best and can lead to
better practice
14
Making it Scholarly

• All research involves asking a question and
  obtaining an answer in a systematic, thoughtful
  way
• Scholarly teaching involves routinely asking
  questions about daily educational activities
• Research is answering these questions and
  scholarship is the dissemination of the research
  results

15
(No Transcript)
16
Types of Studies
Descriptive study describes what is going on or
what exists Relational study looks at the
relationships between two or more
variables Causal study is designed to
determine whether one or more variables causes or
affects one or more outcome variable
17
Research Steps

• Ask the question state clear goals (hypothesis
  and alternate hypothesis)
• Prepare (literature review, discussions,
  conferences)
• Specify the techniques or interventions
• Test methods and ways of measuring impact
• Pilot test the process and refine process
• Select the best method to answer the question

18
Research Steps

• Do the study
• Capture and analyze data
• Interpret findings and determine significance of
  the outcome
• Negative results can be meaningful depending on
  the question
• Reflective critique (thoughtful discussion, think
  about the implications of the findings and
  develop the next steps)
• Disseminate the findings (article, abstract,
  presentation, workshop)

19
Features of Rigorous Studies
Large numbers of student participants Multi-insti
tutional enrollment of participants Having a
control or appropriate comparison
group Measuring an objective outcome Measuring
some outcome at least 1 month after the
intervention Conduct the intervention more than
once Estimate statistical power
20
Features of Rigorous Studies
Measure objective outcomes including any
evaluation other than self-assessment by students
including Tests of knowledge Course grades
Objective structured clinical examinations or
observations of standardized patient Interaction
assessment of performance with a real patient
such as graded observation of patient
interactions or patient feedback Objective
clinical outcomes such as blood pressure
control Performance with mannequins, computer
simulations, or laboratory animals Psychological
inventories
21
Foundation
Language Types of questions (descriptive,
relational, causal) Time in research
(cross-sectional vs. longitudinal with repeated
measures or time series) Types of relationships
(correlational vs. causal) Patterns of
relationships (none, positive, negative) Variable
s and attributes (independent and dependent,
exhaustive, mutually exclusive) Hypothesis and
null hypothesis Types of data (quantitative vs.
qualitative)
22
The Hypothesis
Theoretically significant research questions are
different from other types of questions. A
research question is not a question of
fact. (How many residents passed the in-service
or their qualifying examinations?) about a
specific sample of subjects. (Do my students
perform better with open-ended rather than
multiple choice questions?) about things that
can not be measured. (Do lucky students have
more success than unlucky students?)
23
A good research question should
1. seek new knowledge 2. use the scientific
method 3. build upon existing scientific
knowledge 4. pick up where existing theories and
data end 5. lead to specific hypotheses which
can be tested 6. have an answer deducible from
the results and provide information applicable
not only to the subjects studied, but to a broad,
definable population of subjects and
settings
24
Reliability and Validity
Education research quality is linked to
reliability and validity, which are the two major
criteria for the quality of the
measurement Reliability involves the quality of
measurement and the consistency or repeatability
of the measures Theories of validity involve
each step which are interdependent
25
Reliability and Validity
Studies may be reliable but not valid valid
but not reliable neither reliable nor
valid both reliable and valid Consulting
with a research design and statistics expert to
avoid issues that may impact study reliability or
validity
26
General Surgeon Shortage
27
Surveys - References
Sage Publications, Inc 2nd edition (November 12,
2002), 10 volumes, 1434 pages Gall, Borg, and
Gall. Educational Research An Introduction (8th
edition). Allyn and Bacon. 2006
28
Surveys
One of the most important areas of measurement in
applied social research. Two types Questionna
ires Interviews Construction of the
Survey Types of questions Decisions about
question content Decisions about question
wording Decisions about response
format Question placement and sequence
29
Surveys
Often done because users think they are simple
and quick ways to publications Survey design has
fallen into disrepute among editors, reviewers
and consumers of medical research because they
have been poorly designed poorly
developed poorly administered poorly
analyzed Reinforces the reconceived notion that
surveys are not worth doing Like experimental
research, good survey research is very systematic
and rigorous
30
Surveys
The purpose of the study must be supported by a
review of the literature and understanding of the
problem. Then develop the research question,
which will guide the development of the content.
If an item does not help you answer a research
question, do not include it on the survey What
information do you want from respondents? attitu
des/beliefs? knowledge? behavior? attributes
(demographic characteristics)?
31
Surveys
Structure the questions carefully Unless the
sole purpose of the survey is to solicit
suggestions, probe memories, clarify positions
and or vent frustrations, limit the number of
open-ended questions Open ended questions make
it difficult to construct meaningful variables
for statistical analysis, are difficult to
collate, and difficult to decipher
32
Surveys
A questionnaire should 1. look
attractive/appealing to complete 2. have
questions laid out and be organized in such a
manner as to make them as easy to complete as
possible 3. be organized with items appearing
in some logical sequence 4. begin with those
questions that the respondent is likely to see as
useful to the purpose of the study, or with those
that are easy to answer and non-threatening 5.
place important items that are key to the study
near the beginning Items that are interesting
and clearly relevant to the study will increase
response rate. Length has a small effect on
response rate. Pilot the instrument
33
(No Transcript)
34
Ethics
Principle of voluntary participation Informed
consent Risk of harm physical or
psychological Confidentiality Principle of
anonymity Persons right to service (no
treatment control group) Institutional Review
Board (IRB) Students and residents are
considered a vulnerable population Social and
Behavioral Institutional IRB vs. Biomedical
IRB
35
Funding
Most surgical education research is conducted
without the benefit of research grants ASE
Foundation CESERT program (www.surgicaleducation
.com) National Board of Medical
Examiners Emergency Medicine Foundation Agency
for Health Care Research and Quality Sages America
n College of Surgeons (www.facs.org) lists
funding agencies that may be willing to fund
educational research
36
Resources
ASE Educational Research Committee Institutional
Department of Medical Education Departments of
psychology, education, sociology, and public
health Biostatistician ASE Surgical Education
Research Fellowship Program (SERF)
37
Conclusions
Clinical demands are taking on larger levels of
importance Protected time is a luxury Research
is required to advance academically Educational
research is a viable option even when faced with
high clinical demands
38
Conclusions
Many educational research projects have
significant limitations Insufficient sample
sizes Lack of generalizability when
interventions are assessed at only 1
institution or are conducted only once Lack of
appropriate control groups Use of subjective
and non-validated instruments to assess
outcomes Assessing only short-term
outcomes
39
Conclusions
Educational research challenges often relate to
measurement issues, bias and individual
differences An understanding of key educational
concepts in psychology, sociology, anthropology,
psychometrics, sociometrics, and statistics.
Collaboration is key! Assuming a significant
question has been clearly articulated, the
quality of the study will depend upon the
research design, data analysis, interpretation
and results reporting
40
References
Research Methods Knowledge Base. William M.K.
Trochim. http//www.socialresearchmethods.net.
Free on the internet Surgical Educators
Handbook. Curriculum Committee and Members of
the Association for Surgical Education. ASE
website. http//www.surgicaleducation.com/mc/page.
do
41
(No Transcript)
42
Educational Research Getting Started
Steven B. Goldin Chief of Hepatobiliary and
Pancreatic SurgeryAssociate Professor of
SurgeryVice Chair of Surgical Education Chief of
Hepatobiliary and Pancreatic Surgery
Making Life Better
through Research, Education Healthcare
TM