Scientific Communication CITS7200

1
Scientific Communication CITS7200

• Lecture 9
• Designing and Analysing Experiments

2

• Scientific method concerns hypothesis,
  experiment, and validation
• A hypothesis is a supposition or conjecture put
  forth to account for known facts
• Until it is tested, it is not a theory, nor a law

3

• Hypothesis A logical but unproven explanation
  for a given set of facts
• It is an educated guess

4

• Observation I observe that I always have exactly
  one unmatched sock in my laundry basket coming
  out of my drier
• Example from Greg Anderson

5
Hypotheses

• The CIA takes them from my basket
• Someone on board the SS Enterprise beams them
  from my basket
• The drier eats them
• The drier is a gateway that sucks socks into
  another dimension
• Someone in the house wears only one sock at a
  time
• Someone in the house cant throw both socks into
  the basket

6

• Theory A hypothesis which has been tested
  numerous times and found to explain previous
  observations and make accurate predictions about
  future observations
• To a scientist, you have a theory when there is
  no more reasonable doubt
• Lay person theory guess

7

• Law A theory that has been tested extensively
  and has never been disproven in any test

8

• In computing, experiments are used to verify
  hypotheses about algorithms
• The hypotheses might relate to the performance of
  a specific task, or to the computational
  efficiency or scalability
• Experiments must be repeatable

9
Reader can expect some of

• The steps that make up the algorithm
• The input and output, and the internal data
  structures used by the algorithm
• The scope of application of the algorithm and its
  limitations

10

• The properties that will allow demonstration of
  correctness, such as preconditions,
  postconditions, and loop invariants
• A demonstration of correctness
• A complexity analysis, for both space and time
  requirements
• Experiments confirming the theoretical results

11
Stating the hypothesis

• Must be clear, precise and unambiguous
• Clarify what is not being tested
• i.e. limit the hypothesis to what is interesting
  and provable

12
Example

• Suppose P-lists are a well-known data structure
  used in a range of applications, but particularly
  as an in-memory search structure that is fast and
  compact. Suppose you develop a new data structure
  called Q-lists. You do a formal complexity
  analysis and discover that both structures
  exhibit the same asymptotic complexity in space
  and time, but on all the tests you have run, the
  Q-lists perform better than the P-lists.

13

• Q-lists are superior to P-lists
• Such a hypothesis must apply in all cases, in all
  applications, under all conditions, and for all
  time
• Can never be verified experimentally
• A single counterexample destroys it

14

• As an in-memory search structure for large data
  sets, Q-lists are faster and more compact than
  P-lists

15
Occams Razor

• One should not increase, beyond what is
  necessary, the number of entities required to
  explain anything
• "Pluralitas non est ponenda sine neccesitate
• Entities should not be multiplied unnecessarily"

16

• If you have two theories that both explain the
  observed facts, then you should use the simpler
  theory until more evidence comes along
• Conjurors
• Crop circles
• UFOs

17

• The Razor doesn't tell us anything about the
  truth or otherwise of a hypothesis, but rather it
  tells us which one to test first
• The simpler the hypothesis, the easier it is to
  shoot down

18
Developing hypotheses

• Original hypothesis might not be correct
• Dont alter incrementally
• Q-lists are superior to P-lists, except for data
  sets of size greater than 2,000,000
• Hypothesis is confirmed once it can make
  successful predictions

19
Distinguish between

• The algorithm worked on our data
• The algorithm was predicted to work on any data
  in this class, and this prediction has been
  confirmed on our data

20

• Tests should be blind
• Hypothesis should not be tested on the data from
  which it arose

21
Argumentation

• Need an argument relating your hypothesis to the
  evidence
• First identify the conclusion
• Next identify the premises
• Is the argument valid?

22

• An argument is valid iff its premises logically
  imply its conclusions - that is, if it is
  impossible for its conclusions to be false if its
  premises are true
• An argument is also sound if additionally its
  premises are true

23

• Today is Friday and thus tomorrow is a holiday.
• Valid but not sound

24

• Outside of mathematics it is very difficult to
  develop arguments that stand up to the rigour of
  logical analysis
• Often difficult or impossible to list all the
  premises

25

• Principle of Charity if the step from an invalid
  argument to a valid argument requires a
  suppressed premise that is known to be true, or
  likely to be true, or non-controversial, then
  assume that the argument contains that premise

26

• Principle of Minimality if the argument needs
  only some particular assumption to become valid,
  then do not assume anything beyond that

27

• A valid argument can fail to be sound in only one
  way one of its premises is false
• Play the adversary with your own arguments

28
Example

• The new string hashing argument is fast because
  it doesn't use multiplication or division

29

• Why are multiplication and division an issue?
• On most machines they use several cycles, or they
  may not be implemented in hardware at all. The
  new algorithm instead uses two exclusive-or
  operations per character and a modulo in the
  final step. I agree that for pipelined machines
  with floating-point accelerators the difference
  might not be great.

30

• Modulo isn't always in hardware either.
• True, but it is required only once.

31

• So there is also an array lookup? That can be
  slow.
• Not if the array is cache resident.

32

• What happens if the hash table size is not 28?
• Good point. This function is most effective for
  hash tables of size 28, 216 etc.

33

• Are there any improbable consequences if your
  hypothesis is true?
• Does your hypothesis displace or contradict some
  currently-held belief?
• Does your hypothesis cover all the observations
  explained by some currently-held belief?

34
Scrivens steps of analysis

• Clarification of meanings, including the use of
  dictionaries and encyclopaedias where
  appropriate. Removing ambiguities of terms.
  Identifying and labelling propositions.

35

• Identification of conclusions, stated and
  unstated.

36

• Portrayal of the argument in graphical form.

37

• Formulation of unstated premises. This is the
  most difficult step it requires creativity and
  relevant background knowledge to find the
  premises most appropriate to produce a valid
  argument. This is also the step most prone to
  bias and emotion.

38

• Criticism of the inferences and premises. Try to
  find counterexamples or, in the case of inductive
  arguments, evidence of bias.

39

• Introduction of other relevant arguments.

40

• Overall evaluation. How good is your argument?
  How much would you bet on it? At what odds?

41
Inductive arguments

• Deductive arguments require the extremely strong
  condition that it is strictly impossible for the
  conclusion to be false.
• Inductive arguments are more common in science,
  and still require analysis.

42
Sherlock Holmes

• Holmes deduces that the passenger sitting
  opposite him in the train is a teacher, because
  he has chalk on his hands.

43
The argument

1. This man has white powder on his hands.
2. White powder that looks like chalk is chalk.
3. This man has chalk on his hands.
4. Most people with chalk on their hands are
   teachers.
5. This man is a teacher.

44

• As a deductive argument, it is flawed.
• As an inductive argument at Holmes time, it is
  not bad.

45

• To criticise the inductive strength of an
  argument you must show that the truth of the
  premises would fail to make the conclusion highly
  probable.

46
Types of inductive argument

• Direct inference This involves inferring from the
  proportion of individuals in a class having some
  property the probability of a particular
  individual having that property. Such an
  inference is defeated by an additional biasing
  factor.

47

• Inverse inference This inference proceeds in the
  reverse direction from a sample of individuals to
  characteristics of the whole population. It is
  common in political surveys. Such inferences can
  be defeated if the sampling procedure can be
  shown to be biased.

48

• Analogies Analogies provide the basis for many
  inductive arguments. By drawing attention to
  similarities between structural or law-like
  features of two systems, some support may be
  found for claiming that a further, unobserved
  feature of one is likely to be similar to a
  corresponding feature of the other. Such
  arguments are defeated by the use of
  disanalogies, that is by pointing out how the
  systems differ.

49

• Correlations Some kinds of inductive reasoning
  which may support or undermine causal theories
  invoke the existence of (or failure of)
  correlations between observed variables. Such
  arguments can get very complicated and need to be
  treated with care.

50
Bayesian Evaluation Theory

• Bayes' theorem gives the relation between the
  probability of a hypothesis (conclusion) given
  its evidence, P(he), and its probability prior
  to any evidence, P(h), and its likelihood,
  P(eh).

51

• Make sure you dont confuse P(eh) and
  P(he)Assume that x is a personP(x speaks
  English) ? 15P(x is Australian) ? 0.3
• P(x speaks English x is Australian) ? ?P(x is
  Australian x speaks English) ? ?

52

• P(he) P(e) P(eh) P(h)
• ?
• P(he) P(eh) P(h) / P(e)

53

• When several alternative hypotheses are competing
  for our belief, we test them by deducing
  consequences of each one, then conducting
  experimental tests to observe whether or not
  those consequences actually occur

54

• If a hypothesis predicts that something should
  occur, and that thing does occur, it strengthens
  our belief in the truthfulness of the hypothesis.
  Conversely, an observation that contradicts the
  prediction would weaken (or destroy) our
  confidence in the hypothesis

55

• Often predictions involve probabilities one
  hypothesis might predict a certain outcome has a
  30 chance of occurring, while a competing
  hypothesis might predict a 50 chance of the same
  outcome.
• The occurrence or non-occurrence of the outcome
  would shift our relative degree of belief from
  one hypothesis toward another. Bayes theorem
  provides a way to calculate these "degree of
  belief" adjustments.

56
Example

• Suppose a woman is the daughter of a carrier of
  haemophilia, and therefore is known to have a
  50/50 chance of being a carrier herself. If she
  subsequently has a normal child, how does this
  affect the likelihood that she is a carrier?

57

• Originally Pr(C) Pr (NC) 50
• Pr(CN) Pr(NC) x Pr(C) / Pr(N)
• 0.5 x 0.5 / 0.75 33
• Pr(NCN) Pr(NNC) x Pr(NC) / Pr(N)
• 1.0 x 0.5 / 0.75 66
• Thus, after the evidence her chances are now 66
  of being NC

58

• Evidence will support a hypothesis if and only if
  the likelihood ratio
• l(eh) P(eh) / P(eh)
• is greater than one

59

• Imagine a disease which occurs in 0.1 of the
  population
• A test for this disease is 99 accurate
• P(positive test ill) 0.99
• P(negative test healthy) 0.99
• What is P(ill positive test)?

60

• Consider a sample of 100,000 people
• 100 ill people have 99 positive
  tests
• 99,900 healthy people have 999 positive tests
• So P(ill positive test) 99/(99999) ? 9
• Remember P(positive test ill) 99!

61
Logical fallacies

• Equivocation Equivocation occurs when a term in
  an argument can have more than one meaning, and
  the argument relies on the appearance of validity
  of a term that is actually being used in another
  way

62

• Killing is immoral every religion and system of
  law acknowledges this
• Every system of law and religion treats
  complicity in a killing as immoral also
• Eating meat is complicity in a killing
• Therefore, eating meat is immoral

63

• Affirming the consequent A valid deductive
  inference is modus ponens
• If A, then B
• A
• Therefore, B

64

• It is possible to run the conditional in reverse,
  by denying the consequent this results in modus
  tollens
• If A, then B
• Not B
• Therefore, not A

65

1. If you wish to compete, you must have completed a
   negative drug test
2. You did not complete the test
3. Therefore, you cannot compete

66

• Claim Every card with a vowel on one side has an
  even number on the other
• Given four cards displaying C, E, 8, 3
• Which card or cards needs to be turned over to
  verify the claim?

67

• Affirming the consequent attempts to use modus
  ponens in reverse, but without denying the
  consequent. It is blatantly illogical, and yet
  there are times when it makes sense
• If A, then B
• B
• Therefore, A

68
Affirming the consequent

• I do not tell you that Schlesinger, Stevensens
  number one man, number one brain trust, I dont
  tell you hes a Communist. I have no information
  on that point. But I do know that if he were a
  Communist he would also ridicule religion as
  Schlesinger has done.

69

• Red Herrings and Straw Men
• Red herrings are arguments or facts that are
  irrelevant to the main issue under discussion but
  are brought up in order to distract people from
  that issue

70

• In arguing that Schlesinger is a Communist, when
  challenged McCarthy goes on to say Communists
  are taking over Washington and we must do
  everything we can to stop them.

71

• Straw men are what you invent when you violate
  the principle of charity the attribution to your
  opponents of arguments which they in fact do not
  endorse

72

• Appeals to Authority Improper appeals to
  authority occur when
• the authority invoked pertains to some other
  domain
• (Sports stars endorse headache tablets)

73

• the expert opinion is widely disputed by other
  experts in the field
• (Smoking doesnt cause cancer)

74

• the expert has a history of offering unreliable
  opinions
• (Cancer researcher working for a tobacco company)

75

• Ad Hominem Ad hominem attacks are those against
  the person rather than the argument

76

• Stereotyped Reasoning and Prejudice
• Everybody argues using stereotypes, and mostly
  this is not a bad thing. Inappropriate uses of
  stereotypes involve the failure to adjust them in
  the light of new information. Such behaviour is
  called prejudice.

77

• For more examples (there are many!)www.philosop
  hicalsociety.com/Logical Fallacies.htmwww.skepti
  creport.com/tools/logicfallacies.htmwww.fallacyf
  iles.orgwww.adamsmith.org/logicalfallaciesProo
  fsforP.pdf

78
Evidence

• analysis or proof
• modelling
• simulation
• experiment

79

• An analysis or proof is a formal argument that
  the hypothesis is correct

80

• A model is a mathematical description of the
  hypothesis
• Usually there is a demonstration that the
  hypothesis and the model correspond

81

• A simulation is usually an implementation or
  partial implementation of a simplified form of
  the hypothesis, in which the difficulties of a
  full implementation are side stepped by omission
  or approximation

82

• An experiment is a full test of the hypothesis,
  based on an implementation of the proposal and on
  real (or at least realistic) data

83
Fair experiments

• Tests should be fair not designed to support
  your hypothesis