Introduction to Design Research: a Methodological Background for Scientific Work

1
Introduction to Design Research a
Methodological Background for Scientific Work
2
Outline

• Motivation
• Types of research
• Design Research Basics
• Evaluation in Design Research
• Conclusion

3
Motivation

• Motivation for research
• pure research enhance understanding of phenomena
• instrumentalist research a problem needs a
  solution
• applied research a solution needs application
  fields
• Motivation for research methodology
• (qualitatively) control research process
• validate research results
• compare research approaches
• respect rules of good scientific practice

4
Research A Definition

• Research
• an activity that contributes to the understanding
  of a phenomenon Kuhn, 1962 Lakatos, 1978
• phenomenon a set of behaviors of some
  entity(ies) that is found interesting by a
  research community
• understanding knowledge that allows prediction
  of the behavior of some aspect of the phenomenon
• activities considered appropriate to the
  production of understanding (knowledge) are the
  research methods and techniques of a research
  community
• paradigmatic vs multi-paradigmatic communities
  (agreement on phenomena of interest and research
  methods)

5
Scientific Disciplines

• Types of research Simon, 1996
• natural sciences phenomena occurring in the
  world (nature or society)
• design sciences sciences of the artificial
• all or part of the phenomena may be created
  artificially
• studies artificial objects or phenomena designed
  to meet certain goals
• social sciences structural level processes of a
  social system and its impact on social processes
  and social organization
• behavioural sciences the decision processes and
  communication strategies within and between
  organisms in a social system

6
phenomena
design sciences
Semantic Web (CS)
activities
Owen,1997
7
Design research basics

• Process model
• Artifact types
• result of the research work
• Artifact structure
• content of the research approach
• Evaluation
• evaluation criteria
• evaluation approach

8
Process model

• a problem-solving paradigm
• seeks to create innovations that define the
  ideas, practices, technical capabilities, and
  products through which the analysis, design,
  implementation, and use of information systems
  can be effectively and efficiently accomplished
  Tsichritzis 1997 Denning 1997

9
Design research process
knowledge flows
operation and goal knowledge
circumscription
process steps
Awareness of problem
Suggestion
Development
Conclusion
Evaluation
logical formalism
abduction
deduction
Takeda,1990
10
Artifacts

• are not exempt from natural laws or behavioral
  theories
• artifacts rely on existing "kernel theories" that
  are applied, tested, modified, and extended
  through the experience, creativity, intuition,
  and problem solving capabilities of the
  researcher Walls et al. 1992 Markus et al.
  2002

11
Design research outputs March Smith, 1995

• Constructs
• conceptual vocabulary of a problem/solution
  domain
• Methods
• algorithms and practices to perform a specific
  task
• Models
• a set of propositions or statements expressing
  relationships among constructs
• abstractions and representations
• Instantiations
• constitute the realization of constructs, models
  and methods in a working system
• implemented and prototype systems
• Better theories
• artifact construction

Thesis output
12
Design research outputs
constructs better theories models
emergent theory about embedded phenomena
abstraction
models methods constructs better theories
abstraction
knowledge as operational principles
abstraction
artifact as situated implementation
instatiations methods constructs
Purao , 2002
13
Examples

• Open up a new area
• Provide a unifying framework
• Resolve a long-standing question
• Thoroughly explore an area
• Contradict existing knowledge
• Experimentally validate a theory
• Produce an ambitious system
• Provide empirical data
• Derive superior algorithms
• Develop new methodology
• Develop a new tool
• Produce a negative result

14
Artifact structure

• Structure of the artifact
• the information space the artifact spans
• basis for deducing all required information about
  the artifact
• determines the configurational characteristics
  necessary to enable the evaluation of the
  artifact

Content of the thesis
15
Evaluation criteria

• Evaluation criteria
• the dimensions of the information space which are
  relevant for determining the utility of the
  artifact
• can differ on the purpose of the evaluation

Test cases
16
Evaluation approach

• Evaluation approach
• the procedure how to practically test an artifact
• defines all roles concerned with the assessment
  and the way of handling the evaluation
• result is a decision whether or not the artifact
  meets the evaluation criteria based on the
  available information.

Testing method
17
Evaluation approach (2)

• Quantative evaluation
• originally developed in the natural sciences to
  study natural phenomena
• approaches
• survey methods
• laboratory experiments
• formal methods (e.g. econometrics)
• numerical methods (e.g. mathematical modeling)

18
Evaluation approach (3)

• Qualitative evaluation
• developed in the social sciences to enable
  researchers to study social and cultural
  phenomena
• approaches
• action research
• case study research
• ethnography
• grounded theory
• qualitative data sources
• observation and participant observation
  (fieldwork)
• interviews and questionnaires
• documents and texts
• the researchers impressions and reactions

19
Constructs
20
Methods
21
Models
22
Instantiations
23
Conclusion

• Good research results require a careful
• design of the research methodology and
• considerable evaluation efforts

24
Research Concepts
25
Agenda

• Research Basics
• What research is and is not
• Where research comes from
• Research deliverables
• Methodologies
• Research process
• Quantitative versus qualitative research
• Questions

26
Research Basics

• What research is and isnt
• Research characteristics
• Research projects and pitfalls
• Sources of research projects
• Elements of research proposals
• Literature reviews

27
What Research Is Not

• Research isnt information gathering
• Gathering information from resources such books
  or magazines isnt research.
• No contribution to new knowledge.
• Research isnt the transportation of facts
• Merely transporting facts from one resource to
  another doesnt constitute research.
• No contribution to new knowledge although this
  might make existing knowledge more accessible.

28
What Research Is

• Research is

the systematic process of collecting and
analyzing information (data) in order to increase
our understanding of the phenomenon about which
we are concerned or interested.1
29
Research Characteristics

• Originates with a question or problem.
• Requires clear articulation of a goal.
• Follows a specific plan or procedure.
• Often divides main problem into subproblems.
• Guided by specific problem, question, or
  hypothesis.
• Accepts certain critical assumptions.
• Requires collection and interpretation of data.
• Cyclical (helical) in nature.

30
Research Projects

• Research begins with a problem.
• This problem need not be Earth-shaking.
• Identifying this problem can actually be the
  hardest part of research.
• In general, good research projects should
• Address an important question.
• Advance knowledge.

31
Research Project Pitfalls

• The following kinds of projects usually dont
  make for good research
• Self-enlightenment.
• Comparing data sets.
• Correlating data sets.
• Problems with yes / no answers.

32
High-Quality Research(1 of 2)

• Good research requires
• The scope and limitations of the work to be
  clearly defined.
• The process to be clearly explained so that it
  can be reproduced and verified by other
  researchers.
• A thoroughly planned design that is as objective
  as possible.

33
High-Quality Research(2 of 2)

• Good research requires
• Highly ethical standards be applied.
• All limitations be documented.
• Data be adequately analyzed and explained.
• All findings be presented unambiguously and all
  conclusions be justified by sufficient evidence.

34
Sources of Research Problems

• Observation.
• Literature reviews.
• Professional conferences.
• Experts.

35
Stating the Research Problem

• Once youve identified a research problem
• State that problem clearly and completely.
• Determine the feasibility of the research.
• Identify subproblems
• Completely researchable units.
• Small in number.
• Add up to the total problem.
• Must be clearly tied to the interpretation of the
  data.

36
Hypotheses

• Hypotheses are tentative, intelligent guesses as
  to the solution of the problem.
• There is often a 1-1 correspondence between a
  subproblem and a hypothesis.
• Hypotheses can direct later research activities
  since they can help determine the nature of the
  research and methods applied.

37
Delimitations

• All research has limitations and thus certain
  work that will not be performed.
• The work that will not be undertaken is described
  as the delimitations of the research.

38
Definitions

• Define each technical term as it is used in
  relation to your research project.
• This helps remove significant ambiguity from the
  research itself by ensuring that reviewers, while
  they may not agree with your definitions, at
  least know what youre talking about.

39
Assumptions

• Assumptions are those things that the researcher
  is taking for granted.
• For example a given test instrument accurately
  and consistently measures the phenomenon in
  question.
• As a general rule youre better off documenting
  an assumption than ignoring it.
• Overlooked assumptions provide a prime source of
  debate about a research projects results.

40
Importance of the Study

• Many research problems have a kind of theoretical
  feel about them. Such projects often need to be
  justified
• What is the research projects practical value?
• Without this justification, it will prove
  difficult to convince others that the problem in
  question is worth study.

41
Research Proposals

• Research proposals are documents that describe
  the intended research including
• Problem and subproblems.
• Hypotheses.
• Delimitations.
• Definitions.
• Assumptions.
• Importance.
• Literature review.

42
Literature Review

• A literature review is a necessity.
• Without this step, you wont know if your problem
  has been solved or what related research is
  already underway.
• When performing the review
• Start searching professional journals.
• Begin with the most recent articles you can find.
• Keep track of relevant articles in a
  bibliography.
• Dont be discouraged if work on the topic is
  already underway.

43
Literature Review Pitfalls(1 of 2)

• Be very careful to check your sources when doing
  your literature review.
• Many trade magazines are not peer reviewed.
• Professional conferences and journals often have
  each article reviewed by multiple people before
  it is even recommended for publication.
• The IEEE and ACM digital libraries are good
  places to start looking for legitimate research.

44
Literature Review Pitfalls(2 of 2)

• The Internet can be a good source of information.
  It is also full of pseudo-science and poor
  research.
• Make sure you verify the claims of any
  documentation that has not been peer reviewed by
  other professionals in the computing industry.

45
Processes Methodologies

• Research Process.
• Common Methodologies.
• Methodology Comparison.

46
Research Process

• Research is an extremely cyclic process.
• Later stages might necessitate a review of
  earlier work.
• This isnt a weakness of the process but is part
  of the built-in error correction machinery.
• Because of the cyclic nature of research, it can
  be difficult to determine where to start and when
  to stop.

47
Step 1 A Question Is Raised

• A question occurs to or is posed to the
  researcher for which that researcher has no
  answer.
• This doesnt mean that someone else doesnt
  already have an answer.
• The question needs to be converted to an
  appropriate problem statement like that
  documented in a research proposal.

48
Step 2 Suggest Hypotheses

• The researcher generates intermediate hypotheses
  to describe a solution to the problem.
• This is at best a temporary solution since there
  is as yet no evidence to support either the
  acceptance or rejection of these hypotheses.

49
Step 3 Literature Review

• The available literature is reviewed to determine
  if there is already a solution to the problem.
• Existing solutions do not always explain new
  observations.
• The existing solution might require some revision
  or even be discarded.

50
Step 4 Literature Evaluation

• Its possible that the literature review has
  yielded a solution to the proposed problem.
• This means that you havent really done research.
• On the other hand, if the literature review turns
  up nothing, then additional research activities
  are justified.

51
Step 5 Acquire Data

• The researcher now begins to gather data relating
  to the research problem.
• The means of data acquisition will often change
  based on the type of the research problem.
• This might entail only data gathering, but it
  could also require the creation of new
  measurement instruments.

52
Step 6 Data Analysis

• The data that were gathered in the previous step
  are analyzed as a first step in ascertaining
  their meaning.
• As before, the analysis of the data does not
  constitute research.
• This is basic number crunching.

53
Step 7 Data Interpretation

• The researcher interprets the newly analyzed data
  and suggests a conclusion.
• This can be difficult.
• Keep in mind that data analysis that suggests a
  correlation between two variables cant
  automatically be interpreted as suggesting
  causality between those variables.

54
Step 8 Hypothesis Support

• The data will either support the hypotheses or
  they wont.
• This may lead the researcher to cycle back to an
  earlier step in the process and begin again with
  a new hypothesis.
• This is one of the self-correcting mechanisms
  associated with the scientific method.

55
Common Methodologies

• Methodologies are high-level approaches to
  conducting research.
• The individual steps within the methodology might
  vary based on the research being performed.
• Two commonly used research methodologies
• Quantitative.
• Qualitative.

56
Methodology Comparison

• Quantitative
• Explanation, prediction
• Test theories
• Known variables
• Large sample
• Standardized instruments
• Deductive

• Qualitative
• Explanation, description
• Build theories
• Unknown variables
• Small sample
• Observations, interviews
• Inductive

57
References

• Leedy P. D. and Ormrod J. E., Practical Research
  Planning and Design, 7th Edition. 2001.

58
Questions?
59
REFERENCES

• DFG Rules of Good Scientific Practice available
  at www.dfg.de, last seen September 2005
• Tsichritzis, D. "The Dynamics of Innovation,"
  Beyond Calculation The Next Fifty Years of
  Computing, Copernicus, 1997, pp. 259-265
• Denning, P.J. "A New Social Contract for
  Research," Communications of the ACM (402),
  February 1997, pp. 132-134
• Simon, H.A. The Sciences of the Artificial, 3rd
  Edition, MIT Press, Cambridge, MA, 1996
• Markus, M.L., Majchrzak, A., and Gasser, L., "A
  Design Theory for Systems that Support Emergent
  Knowledge Processes," MIS Quarterly (263),
  September, 2002, pp. 179-212
• Walls, J.G., Widmeyer, G.R., and El Sawy, O.A.
  "Building an Information System Design Theory for
  Vigilant EIS," Information Systems Research
  (31), March 1992, pp. 36-59
• Kuhn, T.S. The Structure of Scientific
  Revolutions, 3rd Edition, University of Chicago
  Press, 1996
• March, S.T. and Smith, G. Design and Natural
  Science Research on Information
• Technology, Decision Support Systems (154),
  December 1995, pp. 251-266
• Lakatos, I. The Methodology of Scientific
  Research Programmes, John Worral and Gregory
  Currie, Eds., Cambridge, Cambridge University
  Press, 1978
• Wikipedia available at www.wikipedia.org, last
  seen Semptember 2005
• Purao, S. Design Research in the Technology of
  Information Systems Truth or Dare. GSU
  Department of CIS Working Paper. Atlanta, 2002
• Leedy P. D. and Ormrod J. E., Practical Research
  Planning and Design, 7th Edition. 2001.