Scientific Research

1
Scientific Research

• Robert O. Briggs
• Delft University of Technology
• University of Arizona
• bbriggs_at_groupsystems.com
• Tucson, AZ 85721

2
Todays program

• Introductions
• Epistemology
• The Philosophy of Science
• The Scientific Approach

3
Which is BobsExciting Secret Identity?

• Bob Briggs
• Facilitated the surrender of Napoleon at Waterloo
• Invented the Internet
• Sang with Elvis Presley in concert

4
Photographic Evidence
Elvis
Bob
5
Three Ways to Think About Academia

• The Philosophical
• The Pragmatic
• The Publishable

6
EpistemologyThe Philosophical View

• The study of the nature of knowledge
• Presuppositions
• Foundations
• Extent
• Validity

7
EpistemologyThe Philosophical View

• A way of knowing
• A way of creating knowledge

8
Prevailing Epistemologies

• Interpretivism
• Criticalism
• Positivism

9
Interpretivism

• Creating knowledge about
• The inferences people draw from and the meanings
  people ascribe to the words and actions of
  others.
• Key Assumption
• There is no objective reality

10
Positivism (Science)

• Creating knowledge about
• Cause-and-effect
• Key Assumption
• There is an objective reality

11
Criticalism

• Creating knowledge about
• The nature of and resolution of deep social ills
• Key Assumption
• Deep social ills exist

12
Epistemology Myths

• Positivism and Interpretivism are mutually
  exclusive world views
• Objective Reality vs. No Objective Reality?
• What is Reality?

13
Epistemology Myths

• Positivists skew studies to find the result they
  want
• Interpretivists dont believe in gravity

14
Epistemology Myths

• Interpretivism is qualitative
• Positivism is quantitative

15
Epistemology Myths

• An epistemology is something you are
• Im an interpretivist
• Im a positivist

16
Pragmatic Epistemology

• A set of mental disciplines
• To keep us from drawing (and then publishing)
  bone-headed conclusions

17
Pragmatic Interpretivism

• A set of mental disciplines
• To keep us from drawing bone-headed conclusions
• About the inferences and meanings people ascribe
  to the words and actions of others

18
Pragmatic Positivism

• A set of mental disciplines
• To keep us from drawing bone-headed conclusions
• About patterns of cause-and-effect

19
Publishable Positivism

• Report of a study on the causes of a
  phenomenon-of-interest that
• Provides convincing arguments that
• The conclusions may not be bone-headed

20
The Philosophy of Science
21
Positivist Assumptions

• Regular patterns of causation
• Independent from human mind
• Knowable

22
The Boundaries of Science

• If its not about cause-and-effect
• Its not science
• Period.

23
Goals of Science

• Create causal models for phenomena of interest
• Test the usefulness of those models
• Use those models to increase the likelihood
  people will survive and thrive.

24
Why should you care about Positivist Science?
25
Why Should You care?

• Good science will
• Make it more likely that people will survive and
  thrive
• Make you work smart
• Get you published in the good journals
• Bad Science will
• Harm others
• Waste effort, time, and money
• Embarrass you for years

26
The Phenomenon of Interest

• In the world of cause-and-effect
• The phenomenon-of-interest is the EFFECT
• The EFFECT is what you seek to explain
• The EFFECT is what you seek to improve
• The EFFECT is the outcome you measure

27
The three most exciting words in science are,
Gee, thats funny- Issac Azimov
28
The Positivist Disciplines

• Phenomenon-of-Interest
• Who Cares?
• Theory
• Hypotheses
• Research Methods
• Analysis

29
The First Discipline

• Explicitly Define
• The Phenomenon of Interest

30
The First DisciplineDefine The Phenomenon of
Interest

• Explicitly
• In writing
• Refine the definition as your understanding
  deepens
• Challenge your definition continuously

31
Explicitly Define thePhenomenon of Interest

• Satisfaction
• First definition
• The degree to which needs are fulfilled
• Measures
• I am satisfied
• My needs are fulfilled
• I feel satisfied
• Better definition
• An affective arousal with a positive valance in
  response to the judgment that needs have been or
  will be satisfied
• Measures
• I feel satisfied with
• gave me a feeling of satisfaction
• I feel good about

32
Phenomenon Vs Domain

• The phenomenon-of-interest is the OUTCOME you
  hope to improve measurably
• Productivity
• Creativity
• The domain is the setting in which the outcome
  manifests
• Requirements Engineering
• Project Management

33
Phenomenon vs. DomainThe Pragmatic View

• You study the phenomenon of interest
• Dont ever forget it
• You sell the domain
• To funding agencies
• To reviewers
• To readers

34
The Second Discipline

• Who Cares?!?

35
Who Cares?!?

• Why is this phenomenon-of-interest is worthy of
  study?

36
Philosophical Who cares?

• Science must increase the likelihood that people
  will survive and thrive
• Society provides the scarce resources for
  scientific enquiry. You must be able to justify
  your use of them.

37
Pragmatic Who Cares

• Your reviewer just had a much better paper
  rejected by the same journal

38
Publishable Who Cares?

• 1. The phenomenon of interest is worth studying
• 1.1 People are more likely to survive and thrive
  if we understand the cause of this phenomenon
• 1.2 The existing literature does not fully
  explain the causes of this phenomenon

39
Publishable Who cares?

• You must define explicitly the phenomenon of
  interest in the first or second paragraph
• Its your anchor for all that follows

40
Good Who Cares? 1.1

• Organizations exist to create value for
  stakeholders
• Organizations operate under risk
• Mitigate risk, the organization may survive
• Internal risk assessments can mitigate risk
• Risk assessments must be run by groups
• If we can make risk assessment groups more
  productive, we may increase that people will
  survive and thrive!
• Productivity is.
• This study examines the use of GSS to make risk
  assessment groups more productive.

41
Bad Who cares 1.1

• Organizations do risk assessments frequently
• We studied collaborative risk assessment workshops

42
Ugly Who Cares 1.1

• We collected some data about risk assessment
  workshops

43
Good Who Cares 1.2

• Connolly et al (1992) showed that productivity of
  brainstorming teams could be improved by making
  them anonymous.
• However, Johnson and Stephens (2003) showed
  better productivity when brainstorming teams were
  identified
• A causal theory of productivity might be useful
  for explaining these seemingly disparate results,
  and might allow the development of even better
  brainstorming techniques.
• This paper offers such a theory

44
Bad Who Cares 1.2

• Jones (1983) said nothing has been done about
  productivity
• Smith (1978) called for more research on
  productivity
• Johnson (1981) studied productivity among factory
  workers
• I studied productivity among brainstorming groups

45
Ugly Who Cares 1.2

• I searched 3 on-line databases and browsed 6 web
  search engines and only found 2 articles on this
  topic.
• Little is known about this topic
• Nobody has studied this topic yet.

46
Publishable PositivismThe Opening Argument
Section 1. Who Cares?!? Argument This
phenomenon is worth studying. 1.1 People will
be better off if we understand this
phenomenon 1.2 Current literature does not yet
fully explain it
47
The Third Discipline

• Theory

48
Theory

• A causal model of the phenomenon-of-interest
• Drives all subsequent activity
• Hypotheses
• Experimental design
• Measures
• Analysis
• Conclusions

49
Data have no meaning except with respect to the
theory from which they spring
Todays Message
50
Goals of Science

• Create causal models for phenomena of interest
  (Theory)
• Test the usefulness of the models (Experiment)
• Use those models to increase the likelihood
  people will survive and thrive. (Application)

51
Anything Missing?
52
Anything Missing?
Truth
53
Positivist Perspective
Science True
Science Useful
54

• A useful model is better than Truth

55
Useful Is Better Than True
56
Name the Phenomenon
Bobezite Block
57
Describe the Phenomenon
A
Bobezite Block
B
58
Explore the Phenomenon
A
Bobezite Block
B
59
Explore the Phenomenon
A
Bobezite Block
Bobezite Block
Bobezite Block
Bobezite Block
Bobezite Block
B
60
Describe the dynamics of the phenomenon
61
A Useful Model
One Gear
62
Truth
One Thousand Gears
63
When does the Model Become Useful?
64
When you want todo something new
65
Therefore

• For matters of cause-and-effect
• A useful model (Theory)
• is better than Truth

66

• An experiment, without a Theory is meaningless

67
What is a Theory?

• An excuse to not do anything meaningful?
• Pie-in-the-sky disconnect from reality?

68
There is nothingmore usefulthan a good theory
69
What is a theory?

• Causal Model
• Internally Consistent
• Explains and/or predicts
• Proposes mechanisms of causation
• Testable

70
Structure of a Theory

• Axioms
• Propositions

71
Axioms

• Assumptions about the fundamental nature of the
  universe
• Axioms are received

72
Example Axioms

• Axiom 1
• Human attention is limited
• Axiom 2
• All action is purposeful for goal attainment

73
Axioms Are Received

• Source is irrelevant
• Feynmans Inspiration

74
Propositions

• Functional Statements of cause-and-effect that
  must be logically true if the axioms are true

75
Propositions are...

• Causal
• Composed of constructs
• Without empirical content

76
Useful Propositions


Productivity
Effort
Goal Congruence
1
2
-
3
Distraction

• Proposition 1 Productivity is a function of
  effort
• Proposition 2 Effort is a function of goal
  congruence
• Proposition 3 Effort is an inverse function of
  distraction

77
Mathematical Propositions

• P ?(E)
• Where
• P Productivity
• G Goal Congruence
• E -?(D)
• Where
• E Effort
• D Distraction

78
Problematic Propositions
79
Qualities of a Good Theory

• Parsimony
• Explanation/Prediction
• Boundaries

80
Pragmatic Theory

• You usually start with propositions and work
  backward to axioms
• You usually start badly and get better
• You use someone elses theory whenever you can
• Your technology is probably not in your theory

81
Pragmatic Theory

• A good theory will get you to the moon and back
  safely on the first try
• Good theory will do more to save you from drawing
  bone-headed conclusions than any other discipline
  of positivism

82
Publishable PositivismAlternative Wordings for
Propositions

• Y is a function of Z
• Z causes Y
• Z determine Y
• The more Z you do, the more Y you get
• Z has a positive influence on Y

83
Publishable Positivism
Section 2. Theory Argument I understand what
causes Z If we assume X to be the case, then it
must be that Proposition 1 Y is a function
of Z.
84
The Fourth Discipline

• Hypotheses

85
The Fourth DisciplineHypotheses

• Comparative statements
• Some explicitly stated measurable outcome
• Compared across at least two treatments
• Logically derived from propositions
• Tests the proposition
• Empirical content
• An answer to a research question

86
Example Hypothesis

• H1 Brainstorming teams with access to an
  automated social-comparison-feedback graph will
  produce more unique ideas than teams with no
  automated graph

87
Example Hypothesis

• H2 During brainstorming, the more we pound
  randomly on the walls, the fewer ideas a team
  will produce.

88
Problematic Hypotheses

• H3 Groups using richer media will exhibit higher
  levels of cohesion initially

89
Problematic Hypotheses

• H4 On negotiation tasks, face-to-face groups
  will outperform computer mediated groups, will
  experience less process difficulty, than
  computer-mediated groups, and will have more
  favorable reactions to their group task
  performance, interaction process, and
  communication medium

90
Publishable Positivism
Section 3. Hypotheses Argument This theory is
testable If, as Propositon 1 posits, Y is a
function of Z, then it must be that
H1. People using Technology-1 will
score higher on the Y-test than do
people using Technology-2.
91
The Fifth Discipline

• Research Methods

92
An experiment without a theory is meaningless
Todays Message
93
Experiment

• Compare outcomes
• Different treatments
• Control other possible causes

94
Experimental Inquiry
95
Investigative Inquiry
Population 1
Results
One Treat- ment

Compare
Population 2
Results
96
Positive Results mean...

• Manipulation caused difference
• Hypothesis has support
• Theory has support

97
Negative Results Mean

• Experiment Flawed?
• Hypothesis Flawed?
• Propositions Flawed?
• Axioms Broken?

98
The Only Scientific Truth

• The Model is No Good

99
Publishable Positivism
Section 4. Methods Argument I found a
reasonable way to test the hypotheses 4.1 My DV
instantiates the phenomenon of interest 4.2 My
IV instantiates a causal construct 4.3 My
approach would reveal a difference if there were
one 4.4 There are few alternative explanations
for any difference discovered
100
An Experiment without a theory is meaningless
101
Phenomena Large, Odd-Smelling Boxes
102
Scientific Instrument Drill
103
Collecting Data without A Theory
104
Collecting Data Without A Theory
105
Collecting Data Without A Theory
106
Collecting Data With a Theory
107
Collecting Data With a Theory
108
Collecting Data With a Theory
109
Collecting Data With a Theory
110
Collecting Data With a Theory
111
A Physicist Uses the Elephant Theory


Fission!
112
A Farmer Uses the Elephant Theory
113
A Farmer Uses the Theory
114
There is nothingmore usefulthan A Good Theory
115
An Experiment without a theory is meaningless
116
Data have no meaning except in reference to the
theory from which they spring.
117
Kinds of Causal Theories

• Descriptive
• Predictive
• Explanatory

118
Descriptive Model

• What factors impact the length of pins?
• Pin-length factors
• - Social Tone (Parties)
• - Bob-Presence
• - ?

119
Predictive Causal Model
How can we predict the length of B? The length of
B is directly proportional to the length of A
120
Predictive Causal Model
What about the Hacksaw Experiments?
A
Bobezite Block
B
121
Explanatory Model
Why is the length of B proportional to the
length of A? A and B are linked by a gear.
A
B
122
Another View of Theory

• Three out of four kinds of theories are
  dangerous

123
Levels of Theory

• A - Fully Axiomatized
• B - Building or Broken
• C - Construct Theory
• D - Descriptive Theory

124
A - Level Theory

• All Axioms in place
• Many propositions expressed
• Extensive, unequivocal empirical support

125
A-Level Theory

• F M A
• A good theory
• gets you to the moon
• on your first try.

126
B-Level Theory

• Some axioms in place
• Some propositions
• Little empirical support
• Danger - some unknown effects

127
C-Level Construct Theory

• Assert that a construct exists
• Find a way to measure it
• Danger You always will find a way to measure it

128
C-Level Studies

• Communication Apprehension Instrument
• Measure different groups
• Compare to other constructs

129
C-Level Studies

• Locus of Control
• 3,000,000 study
• Disastrous result

130
D-Level Descriptive Theory

• Describe Characteristics
• Taxonomy, Framework
• Dangers
• Over Aggregation
• Infinite regression

131
An Input-Process-Output Model of Group Outcomes
from GSS Use
Team
Task
Process
Outcome
Technology
Context
From Nunamaker, et al. (1991)
132
Endlessly Divisible Constructs

• Characteristics of the team
• Structure
• Leadership Style
• Power differences
• Norms
• Intra group process
• History
• Cohesiveness
• Heterogeneity
• Etc. Etc.

133
Infinite Regression

• Conclusion The phenomenon cant be studied
• Better Conclusion Im asking the wrong question

134
The Experiment
135
Points to Ponder

• You dont have to measure cause, you only have
  to manipulate it.
• You must measure every effect
• You must have a theoretical explanation for every
  effect

136
Experimental Model
137
Investigative Inquiry
Population 1
Results
One Treat- ment

Compare
Population 2
Results
138
Investigative Inquiry
PTA Members
Loved It
Eat At Joes

Compare
Non-PTA Members
So-So
139

• PTA Causes Change in Taste?
• Joe was charismatic principal.

140
Experimental Logic

• If every thing else is the same, the difference
  MUST be caused by my treatments.

141
Science and Technology

• You do not study technology
• You study the effects to which technology can be
  applied

142
Science and Technology

• Every PRESCRIPTION implies an underlying model
  of cause-and-effect

143
The Dangers of Match and Fit Theories

• The quality of the building depends on the fit
  between the plan and the purpose

144
The Dangers of Match and Fit Theories

• Every Match or Fit theory implies one or more
  underlying models of cause-and-effect
• But does not bother to articulate them

145
Experimental Design

• Construct Validity
• Statistical Validity
• Internal Validity
• External Validity

146
Construct Validity

• Am I measuring the construct I think Im
  Measuring?
• Thermometer to measure time?
• Theory drives measures

147
Statistical Validity

• Are statistics interpreted meaningfully
• Theory Drives Statistics

148
Internal Validity

• Did my treatment really cause the difference I
  observed?

149
Threats to Internal Validity

• Unfavorable Comparison
• Group receiving Poor treatment stops trying

150
Threats to Internal Validity

• Between-group competition
• Group receiving the poor treatment makes extra
  effort to excel

151
Threats to Internal Validity

• The Hawthorne Effect
• Paying attention to people affects their
  performance.

152
Control for Hawthorne Effect
Pay Attention to Both Groups
Treat- Ment
Group1
Control Group
153
Threats to Internal Validity

• Novelty Effect
• New situations stimulate performance.
• Control Longitudinal Study

154
Threats to Internal Validity

• Maturation
• Perhaps the effect occurred simply because the
  subjects got older.

155
Control for Maturation
Treat- Ment
Group1
Control Group
Measure Here
156
Threats to Internal Validity

• History
• Something happens during the experiment that
  causes the effect

157
Control for History
Treat- Ment
Group1
Control Group
Measure Here
158
Threats to Internal Validity

• Reactive Measures
• Somehow the initial measuring process causes the
  effect

159
Control for Reactive Measures
Treat- Ment
Group1
Control Group
Measure Here
160
Threats to Internal Validity

• Calibration
• differences caused by shifts in instrument
  calibration over the course of the study.

161
Control for Calibration
Treat- Ment
Group1
Control Group
Measure Here
162
Classic Books

• Campbell Stanley
• Cook Campbell

163
External Validity

• To what population do my results apply?
• Generalizability

164
Theory Drives

• Hypothesis
• Measures
• Treatments
• Statistics

165
Scientific Method

• Discover Phenomenon
• Theorize
• Hypothesize Fastest Falsifications
• Experiment
• Conclude
• Apply

166
Selling Your Science Getting Published

• Introduction Who cares?
• Theory Says Who?
• Hypotheses Prove it!
• Design Are you sure?
• Results Did you get it?
• Discussion So What?
• Conclusions Theory Good?

167
Truth

• Powerful theory will outperform powerful
  statistics every time!

168
Truth

• There is No Perfect Study
• You must pilot your study

169
Truth

• No Theory is made or broken by a single study

170
Remember

• Experiments without theories are meaningless

171
Remember

• Data Have No Meaning except in reference to the
  theory from which they spring

172
(No Transcript)