Using Mixed Method Approaches in Institutional Research

1
Using Mixed Method Approaches in Institutional
Research
Rick Kroc University of Arizona Richard
Howard University of Minnesota
Arizona Association for Institutional
Research Annual Meeting Tucson, Arizona
March 2007
2
Purpose of Decision Support is to reduce the
risk to the decision maker
CERTAINTY
3
Structure and Subjectivity
SUBJECTIVITY
QUESTION
CERTAINTY
STRUCTURE
4
Research Process
5
Ways of Knowing Two Classic Paradigms
Positivist
Constructivist
there is a single reality or truth across time
and contexts, and that this truth can be
understood through the objective study of
independent variables
there are multiple realities and that truth is
ever changing, dependent on context and the
individual (subject and researcher)
6
Conceptual Model of the Constructivist Research
Process
After a literature review and pilot study, define
the Study Problem Purpose Question
Data Collection Management
COMMUNICATE New Theory/Model
Data Restructuring/Coding Analyzing/Read-Theme
Data Verification with Respondents FACTUAL ERROR
Analysis Verification with Reviewers Interpretati
on Error
TEST Theory/Model
Iterative Process Delimited by PPQ
7
Conceptual Model of the Positivist Research
Process
After a literature review and pilot study define
the project Problem Purpose Que
stion Operationalize theory or model constructs

For internal decision support- test meaning of
the findings
Data Collection and Management
Communication of new knowledge
Data restructuring and analysis (Interpretation
of Results)
For external contexts- test/replicate for
generalization
8
RESEARCH PARADIGMS APPROACHES
Constructivist
Positivist
Quantitative Qualitative
Researcher Generalizes
Consumer Generalizes
9
Mixed Methods
Combines the collection, analysis, and
interpretation of qualitative and quantitative
data.
Often used in studies that require an
understanding of not only what is observed but
also why the observed behavior occurred. Such
as in assessment, evaluation, action research,
and other decision support studies.
10
Use of Mixed Method Approaches in Institutional
Research
Provides answers to two questions What -
What happened and what was the outcome?
Answers tend to be arrived at using quantitative
methods Why - Why did people behave the
way they did? Answers tend to be arrived at
using qualitative methods
11
So What?
What - goals are - how measured Supports
Accountability (External) Why goals were met
or not people issues Often the focus of
Assessment/Evaluation (Internal)
12
Ways of Knowing A Third Paradigm
Pragmatism
Use what ever approach and methodology that will
best answer the question.
Constructivist
Positivist
there are multiple realities and that truth is
ever changing, dependent on context and the
individual (subject and researcher)
there is a single reality or truth across time
and contexts, and that this truth can be
understood through the objective study of
independent variables
13
Charge from President Likins As Chair of the
NCAA Division I Presidential Task Force on the
Future of Intercollegiate Athletics, I have both
the opportunity and the responsibility to examine
the culture of higher education athletics
programs in America. My responsibilities begin at
home. My task force experience as well as events
at colleges and universities around the country
led me to conclude that it is appropriate to
examine the environment defined by the culture of
high-level athletic competition here at The
University of Arizona. At the national level a
serious movement to strengthen the academic
experience of student-athletes is well underway.
We must examine the presence of any cultural
influences here that have the potential of
distorting the shared values of academics and
athletics in higher education.
14
Focus of the Study

• The study was divided into two phases. Phase I,
  mostly quantitative in nature, was designed to
  look somewhat broadly at the two fundamental
  study areas
• Student athletes academic experience and
• Student athletes culture and environment.
• Using qualitative methods, Phase II was designed
  to dig more deeply into the critical issues
  identified from Phase I. Detailed, targeted
  recommendations would be developed from these two
  phases.

15
Domain of the Study

• Athletes academic experience
• Academic preparation
• Progress to degree
• Student engagement
• Academic support
• Post-eligibility academic experience
• Other academic issues

• Athletes culture and environment
• Alcohol and drugs
• Dishonesty and cheating
• Gambling
• Racism
• Sexism
• Sexual behavior
• Violence
• Weapons

16
Phase 1Quantitative

• Board of Regents and NCAA reports on athletes
  grades and graduation rates (ten year history).
• Institutional student records data.
• Health and Wellness Survey, administered annually
  to assess student alcohol, drug, and sexual
  behaviors (seven year history).
• Dean of Students Code of Conduct and Code of
  Academic Integrity violations (five year
  history).
• Coalition on Intercollegiate Athletics (COIA)
  report.
• Annual University Survey of Graduating Seniors,
  assessing student engagement, satisfaction, and
  other academic issues (five year history).
• NCAA GOALS survey (student-athletes and student
  non-athletes).
• Additional UA-specific survey questions appended
  to GOALS survey (student-athletes and student
  non-athletes).

17
Phase IIQualitative

• Focus groups
• Interviews
• Follow-up surveys
• Populations
• Students (athletes and non-athletes)
• Coaches
• Faculty
• Staff
• Advisors

18
The Value of a Mixed Methods Approach

• Athletic culture spans a spectrum of areas, some
  of which (alcohol use, for example) have become
  fairly well understood over the years from
  quantitative methods, others of which (physical
  violence, weapons use) need the deeper
  exploration and understanding provided by
  qualititative methods.

19
The Value of a Mixed Methods Approach

• The formative and summative aspects of this
  study, which is more like program evaluation in
  many ways than like a traditional research study,
  lend themselves to a mixed methods approach.

20
The Value of a Mixed Methods Approach

• Audiences and stakeholders consist of some people
  who respond best to systematic, inferentially
  rigorous, quantitative data, but also others who
  want to hear the complex, richly detailed stories
  derived from the individuals who are immersed in
  the culture. Different audiences resonate to
  different approaches.

21
The Value of a Mixed Methods Approach

• Mixed methods also means that multiple methods
  have been employed, which is essential to using
  triangulation as a means to establish greater
  validity of the findings. Any single approach
  used in the messy world of social science
  research will be flawed. If a wide array of
  methods leads to consistent conclusions, then
  analysts and stakeholders feel more confident
  about conclusions and recommendations.

22
Questions??