Relevance in academic research

1
Relevance in academic research

• Dan Remenyi PhD
• Visiting Professor
• School of Systems and Data Studies
• Trinity College Dublin
• dan.remenyi_at_tcd.ie

2
Establishing the context

• The issues of relevance and rigor are complex
  concepts and need careful understanding.
• Some words have a large semantic footprint and
  relevance, rigor, validity, reliability,
  generalisability and integrity are good examples
  of these.
• No one can proceed with any understanding of
  research without having a clear agreement with
  regards the meaning of terms.

3
The effectiveness and efficiency parallel

• A quick rule of thumb understanding of relevance
  is doing the right thing or looking at the right
  issues
• A quick rule of thumb understanding of rigor is
  doing the thing right
• These are the colloquialisms coined by Peter
  Drucker to explain effectiveness and efficiency
• Thus relevance equates to effectiveness and rigor
  relates to well not so much efficiency but
  something more like adequacy

4
Relevance and Rigor

• Relevance is a high level issue in that you can
  say that the research project is relevant or not
  relevant as a whole it could also be said
  that the use of case studies was not relevant to
  the objectives of the study
• Rigor is an issue which permeates the whole
  project in that you can say that the literature
  review was not rigorously approached or the
  sample was not rigorously chosen, or the
  conclusions were not rigorously written up.

5
Academic Research is a Challenge
6
Academic research

• What is the nature of academic research?
• It is an intellectual activity
• It is an attempt to answer a question
• It is a process of question, data and analysis
• Academic research has two primary attributes-
• It adds something of value to the body of
  theoretical knowledge.and practice!
• It demonstrate a high level of scholarship

7
Something of value to the body of theoretical
knowledge.and practice

• Something of value is reflected in the concept of
  relevance.
• Relevance needs to be perceived by both the
  non-academic stakeholders and also the academic
  community.
• Without some contribution to theory the research
  will not be much valued by academics.
• Mode 2 brings both together.

8
Scholarship and rigor

• Being well read and being able to demonstrate
  that in conversation
• Being able to craft an argument in writing
• Scholarship is connected to rigor
• An argument is a series of logical steps by which
  someone is persuaded. In this sense is not
  confrontational.
• Academic work i.e. either a dissertation or a
  peer reviewed paper needs to argue for relevance,
  rigor, validity, reliability, generalisability.

9
Quality academic research

• Dimensions of quality in academic research
• addresses a pertinent question
• uses academic language with care - remember the
  fog factor
• uses appropriate data
• is written up in intelligible English and correct
  format
• is presented in an attractive way - structure
• invoking authority
• finds an acceptable/useful answer
• Being parsimonious quality academic research
  needs to be relevant and rigorous

10
Double hurdle

• Academic research needs to be relevant i.e. of
  interest and of use to a wide range of management
  at different levels and rigorous i.e. display
  academic excellence.
• Being both relevant and rigorous has been
  described as a synthesis strategy.

11
Research question

• The research question has been reached through a
  careful examination of and understanding of
• The field of study
• The topic
• The specific questions and sub-questions

12
Some critical issues

• Something of value
• As judged by the academic community
• Body of theoretical knowledge
• Add something to established domains
• Parsimonious
• Expressed in a precise and economical way
• Authority
• Ideas/concepts/theories which have through the
  peer review process
• Relevance
• Has value to a group of stakeholders
• Rigor
• Complies with the rules of the research paradigm

13
Three more issues

• Validity ..Has the research addressed the
  intended issues i.e. addressed the issues which
  are required to answer the research question?
• Reliability..Has the research been conducted
  sufficiently well (rigorously) to provide results
  in which we can have confidence?
• Generalisabilty..To what extent can the
  research be applied else where?

14
Rationalism versus Empiricism

• Rationalism (theory)
• The mind is superior to the senses dating from
  the Greeksat least Plato..thought experiments
• Reason
• Logic
• Secondary data
• Empiricism
• If it cannot be held, felt or shaken etc it does
  not exist!
• In its pure form it fails due to particle physics
  etc
• Positivism versus interpretivism
• Primary data

15
Assumptions - preliminary concepts

• What are the assumptions under which academic
  research is conducted?
• The world is understandable.
• We are interested in understanding it.
• Researchers will be open minded.
• Researchers are capable of objective reasoning.
• We can find data/evidence to explore the
  phenomena and therefore understand them.
• The nature of the understanding can be both broad
  and also detailed.
• Occams razor points out that too much complexity
  can obscure what is happening.
• We always have to use ceteris paribus all other
  things being equal.
• That some degree of cause and effect can be
  identified/assumed.
• We all operate in terms of our cognitive capacity
  which changes, and hopefully develops/increases
  over time.

16
The Issue of Integrity Fleischmann and Pons
17
Integrity and Professor Woo Suk Hwang And
Professor Sir Cyril Burt
Sir Cyril Burt Reported nonexistent studies on
I.Q. tests in the 20s and 30s, and added
fictitious collaborators to his papers
18
Some ideas to examine

• Relevance ..doing the right thing
• Relevance ..to whom or to what?
  theory/academics/practitioners......Is it
  significant?
• Rigor .doing the thing right
• Is rigor a dimension of validity?
• If the research is not valid then do not worry
  about reliability?
• If the research is not reliable can it be valid?
• If the research is neither valid or reliable then
  is it worthless?
• What can go wrong question, data, analysis,
  finding conclusions etcthreats to the research

19
Evaluation of relevance and rigor

• Relevance is normally evaluated holistically i.e.
  is the research important or is it not. Sometimes
  some aspects of the research may be more useful
  than others but this question is essentially a
  holistic one.
• Rigor is normally evaluated on an issue by issue
  basis. This requires comment of individual
  aspects of the research. But there is also a
  holistic view. There may be some tension between
  the individual evaluations and a holistic view of
  the research. The sum of the parts may be
  different to the whole.

20
Relevance

• In information science Relevance is the extent
  to which some information is pertinent,
  connected, or applicable to the matter at hand.
  It represents a key concept in the fields of
  documentation, information science, and
  information retrieval. Pehcevski and Larsen
• Although academic relevance is today of essential
  importance to research, surprising little has
  been written about it.
• It used to have been assumed that if the research
  is good then it is academically relevant and
  this attitude has only recently been challenged.
• The main drivers of this challenge to relevance
  may be seen in the debate surrounding Mode 1 and
  Mode 2 research.

21
A stumbling block issue

• Previously relevance was not a visible issue. If
  the research and his/her supervisor thought that
  the question was important then that was all that
  was needed.
• Now the research has to convince the examiners or
  the reviewers that it is adequately relevant. An
  argument has to be made.
• A research dissertation/paper could be
  refused/rejected on the grounds that the
  research is not adequately relevant

22
Weak and strong relevance

• According to Blake Ives (US IS academic) there
  are two forms of relevance - weak relevance and
  strong relevance - http//www.misq.org/archivist/v
  ol/no20/issue3/edstat.html
• Weak relevance occurs when a research question
  addresses an issue which is known and documented
• Strong relevance requires an action research
  approach to answering the research question which
  produces concrete findings

23
A closer look at relevance

• The statement that a research question is
  relevant begs the question of relevant to whom?
  and in which way?
• A research question is relevant if there is-
• A distinct interest among the stakeholders but
  who are the stakeholders? A material stakeholder
  group
• The research question will provide interesting
  results and/or useful guidelines to professionals
• Improvements to practice i.e. there is clear
  application for the findins

24
How are relevant research questions to be
identified?

• Relevant research questions may be found in-
• the extant literature
• work place experiences
• special interest groups
• Business/professional/practitioner groups
• government requests
• general business press
• Remember that a relevant question leads to a
  solution for someone
• Is there pure or basic research in BM
  Studies?

25
What do we need to find a relevant research
question?

• Imagination!
• .. But also needed is imagination to create
  from these hints the great generalizations to
  guess at the wonderful, simple, but very strange
  patterns beneath them all, and then to experiment
  to check again whether we made the right guess.
  Richard Feynman, Six Easy Pieces, p2, Penguin
  Books, London (1995)
• Finding a suitable research question should not
  be rushed.
• Activities which can help to find relevant
  research questions include-
• Talking to knowledgeable informants
• Focus groups
• Qualitative questionnaires
• Blue sky and scenario sessions
• Talking to your supervisor/s
• Literatureacademic and popular press

26
A caveat!

• The relevance will be judged by both academics
  and by practitioners but academics will/may have
  a greater say than practitioners.
• The degree of relevance is often tested by the
  So What? or Whats new ....... and who cares?
  question. This question needs to be directly
  answered in a research dissertation and in a peer
  reviewed academic paper.
• Relevance or the lack thereof will always be
  known post hoc or a posteriori but you have to
  address this issue on an a priori basis

27
The time dimension

• Relevance has a time dimension.
• What is relevant today may not be relevant
  tomorrow.
• The time value of a research question is a
  function of the field of study and the topic and
  directly influences the relevance of the research.

28
The argument

• Relevance may not be taken for granted.
• The researcher needs to argue that the research
  is relevant because it-
• answers a well known /important question/s
• addressed a non-obscure question
• will produce a useful result usually a paradigm
• will allows several other questions to be
  explored
• it will attract people and funding to the
  university or business school
• etc

29
The argument is one of the key issues for
establishing relevance and rigor
30
Assessing relevance

• Feedback from informants and peers is the most
  effective way of ensuring that the research is
  relevant.
• It is useful to have in mind the need to test
  this relevance occasionally through out the
  duration of the research
• Also test the findings by asking the stakeholders
  of the results of the research are likely to be
  useful to them.

31
Journal titles......Relevant?

• Stark, S., Chernyshenko, O., Guenole, N.
  (Accepted).  Can subject matter expert ratings of
  statement extremity be used to streamline the
  development of unidimensional pairwise preference
  scales. (18 no ?)  Organizational Research
  Methods
• Chernyshenko, O., Stark, S., Guenole, N.
  (2007). Can the discretionary nature of certain
  performance criteria lead to differential
  prediction across cultural groups? (15 yes or no
  answer) International Journal of Selection and
  Assessment, 15, 175-184
• Be careful of papers with very long titles and
  many authors.