The psychology of knights and knaves

1
The psychology of knights and knaves

• Lance J. Rips,
• University of Chicago,
• 1989

2
Knights and Knaves

• (1)   We have three inhabitants, A, B, and C,
  each of whom is a knight or a knave. Two people
  are said to be of the same type if they are both
  knights or both knaves. A and B make the
  following statements
• A B is a knave
• B A and C are of the same type.
• What is C?
• (Smullyan, 1978, p.22)

3
Protocol evidence

• Subjects attempted to solve problems by
  considering specific assumptions
• Worked forward from their assumptions
• Subjects sometimes forgot assumptions

4
Protocol evidence
5
Computational model

• Based on the idea that people deal with deduction
  problems by applying mental-deduction rules like
  those of formal natural deduction systems

6
Computational model

• Subjects performance predicted on a deduction
  problem in terms of length of required derivation
  and availability of rules
• The shorter the derivation and more available the
  rules, the faster and more accurate subjects
  should be

7
Computational model

• knight(x) x is a knight, knave(x) x is a
  knave
• says(x,p) person x uttered sentence p
• Rule 1
• says(x, p) and knight(x) entail p.
• Rule 2
• says(x,p) and knave(x) entail NOT p.
• Rule 3
• NOT knave(x) entails knight(x)
• Rule 4
• NOT knight(x) entails knave(x).

8
Computational modelPROLOG Program

• Stores logical form of sentences in problem and
  extracts names of individuals (A, B, and C)
• Assumes first-mentioned individual is a knight,
  knight(A)
• Draws as many inferences as possible from
  assumption
• If contradictory sentences (knight(B) and
  knave(B)) it abandons assumption that
  first-mentioned individual is a knight and
  continues with assumption knave(A)

9
Computational model PROLOG Program

• Revises rule ordering, rules successfully applied
  will be tried first on the next round
• Continues until it has found all consistent sets
  of assumptions about the knight / knave status of
  each individual

10
Computational modelPROLOG Program
11
Computational model PROLOG Program

• All rules operate forward
• Assumes subjects error rates and response time
  depend on length of derivations

12
Experiment 1
Rule 5 (AND Elimination) p AND q entails p,
q. Rule 6 (Modus Ponens) IF p THEN q and p
entail q Rule 7 (DeMorgan-1) NOT (p OR q)
entails NOT p AND NOT q Rule 8
(DeMorgan-1) NOT (p AND q) entails NOT p OR NOT
q
13
Experiment 1

• Rule 9 (Disjunctive Syllogism-1)
• p OR q and NOT p entail q.
• p OR q and NOT q entail p.
• Rule 10 (Disjunctive Syllogism-2)
• NOT p OR q and p entail q.
• p OR NOT q and q entail p.
• Rule 11 (Double Negation Elimination)
• NOT NOT p entails p.

14
Experiment 1Method

• Submitted puzzles to the PROLOG program and
  counted the number of inference steps it needed
  to solve them
• 34 problems
• Six problems had 2 speakers, 28 had 3
• 2 speaker problems had 3 or 4 clauses
• 3 speaker problems had 4, 5, or 9 clauses

15
Experiment 1Method

• 4 clause, 3 speaker problems
• (2) A says, C is a knave.
• B says, C is a knave.
• C says, A is a knight and B is a knave.
• (3) A says, B is a knight.
• B says, C is a knave or A is a knight.
• C says, A is a knight.

16
Experiment 1 - Subjects

• 34 subjects
• 3 groups of 10 to 13 individuals
• University of Arizona Undergraduates
• English Speakers, no formal logic courses
• 10 subjects stopped working on the problems after
  15 minutes

17
Experiment 1 Results and Discussion

• None of the subjects solved the most difficult
  problem and 35 solved the easiest.
• 24 of problems predicted to be easier, 16 of
  problems predicted difficult.
• Program used a mean of 19.3 steps in solving
  simpler problems, 24.2 steps on the more
  difficult problems.
• Core subjects solved 32 of the easier problems
  and 20 of more difficult problems.

18
Experiment 1Results and Discussion
Percentage of Correct solutions in Experiment 1
as a function of the number of inference steps
used by the model
19
Experiment 1Results and Discussion

• 3-speaker, 9-clause outlier
• (4) A says, Were all knaves.
• B says, A, B, or C is a knight.
• C says, A, B, or C is a knave.

20
Experiment 1Results and Discussion

• Prediction that subjects would score higher on
  puzzles with smaller number of inference steps
  consistent with findings.

21
Experiment 1Results and Discussion

• Binary Connectives
• says(A, ((knave(A) AND knave(B)) AND knave(C ))
• N-ary Connectives
• AND(knave(A), knave(B), knave(C ))

22
Experiment 2

• Predict the amount of time subjects take to reach
  a correct solution based on the number of steps
  the model needs to find a correct answer.

23
Experiment 2

• Problems were simplified as longer problems
  produced longer and more variable times
• More difficult problems also resulted in less
  correct answers.
• Tighter control on the form of the problems
• Eliminate irrelevant effects of problem wording
  and response.

24
Experiment 2

• Modified rules to allow program to solve a wider
  variety of problems
• Rules 9 and 10 (Disjunctive Syllogism)
• Allowed the program to infer p from any of the
  following
• OR(knight(x), p) and knave(x)
• OR(knave(x), p) and knight(x)
• OR(p, knight(x)) and knave(x) and
• OR(p, knave(x)) and knight(x)

25
Experiment 2Method

• Subjects viewed the problems on a monitor and
  responded using a response panel.
• Monitor presented subjects with feedback about
  accuracy of their answer and amount of time taken.

26
Experiment 2Method
27
Experiment 2Method
28
Experiment 2Method

• Submitted problems to the natural-deduction
  program and chose 12 of the groups based on
  output.
• Each group had same output but differed in the
  number of inference steps required to solve
• Column 1 (small) 13.1 steps
• Column 2 (small) 13.0 steps
• Column 3 (large) 16.4 steps

29
Experiment 2Method

• The prediction is that the large step problems
  within each row will result in longer response
  times and more errors.

30
Experiment 2Subjects

• 53 University of Chicago Undergraduates
• Native English speakers, no formal logic
• 5 bonus minus 10 cents per trial on which they
  made an error
• Discarded data from subjects who made errors on
  more than 40 of trials
• 30 subjects succeeded

31
Experiment 2Results and Discussion

• The problems with a larger number of predicted
  inference steps took longer for the subjects to
  solve.
• Subjects took 25.5s to 23.9s to solve the two
  types of small-step problems, but 29.5s on the
  large-step problems.

32
(No Transcript)
33
Experiment 2Results and Discussion

• Error Rates
• 1st Small step 15.8
• 2nd Small step 9
• Large step 14.4

34
Experiment 2Results and Discussion

• Knight-knave Problems
• Took longer to solve and most difficult
• Knight-knight 24.8s 14.4 errors
• Knight-knave 29.4s 17.5
• Knave-knight 24.0s 8
• Knave-knave 26.8s 12.2
• But only a small difference in the number of
  steps necessary for the program to solve.

35
Experiment 2Results and Discussion

• Attributed increase in knight-knave problems to
  the small-step items
• Subjects incorrectly assume character is lying
  when they state I am a knave
• This would result in knave(A)-knight(B) response

36
Experiment 2Results and Discussion

• Effects of negatives
• Subjects took longer to read and comprehend
  negative sentences
• The model adds extra steps are necessary to
  transform these negatives to positives
• Rule 3 NOT(knave(x)) to knight(x)
• 23.4s to solve no negative problems with 10.6
  error rate
• 27.2 to solve problems with one negative with
  13.9 error rate

37
General DiscussionNatural-deduction model

• People carry out deduction tasks by constructing
  mental proofs
• Represent information
• Make further assumptions
• Draw inferences
• Make conclusions on basis of derivation

38
General DiscussionNatural-deduction model

• The knights and knaves problems extend model
  compared to previous experiments which judge
  validity of arguments
• Depend on logical properties but do not have
  premise-conclusion format

39
General DiscussionNatural-deduction model

• Protocol
• Participants followed assume-and-deduce strategy
• Experiment 1
• Predict probability of subjects solving a set of
  moderately complex and varied puzzles
• Experiment 2
• Response times increased with the number of
  inference steps

40
General DiscussionNatural-deduction model

• Limitations
• A large minority found the simpler problems to be
  extremely difficult and performed below chance
  level of performance
• Results were interpreted using only the
  natural-deduction framework

41
General Discussion

• Subjects who did not complete the task
• Large variation
• Experiment 1 some achieved 80 correct, other
  subjects missed all

42
General Discussion

• Individual Differences
• OR Introduction
• Avoided problems dependent on OR Introduction
• Lack of availability of Knight-knave rules
• Subjects do not understand that what a knight
  says is true and what a knave says is false

43
General DiscussionAlternative Theories

• Deduction by heuristic
• By responding knave if a character says I am a
  knave and responding knight otherwise
• Results in 25 correct versus obtained 87
• No apparent non-logical short cuts

44
General DiscussionAlternative Theories

• Deduction by pragmatic schemas
• Knights and knaves does not follow the real world
  schema
• Very few situations in which people always tell
  the truth or always lie
• May help with Wason selection task (permission /
  restrictions)
• But no case for people using schemas on most
  deduction problems

45
General DiscussionAlternative Theories

• Deduction by mental models
• Subject surveys model for potential conclusion
  and if found attempts to find a counter example
  by altering the model.
• If no counterexample found the subject adopts
  initial conclusion as correct.
• If counterexample is found, conclusion is
  rejected and another conclusion is examined.
• Continues until acceptable conclusion is found or
  it is decided that no conclusion is valid.

46
General DiscussionAlternative Theories

• (1)   We have three inhabitants, A, B, and C,
  each of whom is a knight or a knave. Two people
  are said to be of the same type if they are both
  knights or both knaves. A and B make the
  following statements
• A B is a knave
• B A and C are of the same type.
• What is C?

47
General DiscussionAlternative Theories

• Subject use tokens for each character.
• knightA
• knaveB
• knaveC
• Conclusion that C is a knave, continue with
  counterexamples.

48
General DiscussionAlternative Theories

• knaveA
• knightB
• knaveC
• Since conclusion stands in both then C is a knave.

49
General DiscussionAlternative Theories

• None of the speak aloud subjects mentioned tokens
• Could be a difficulty with describing mental
  models.
• The theory does not account for the process that
  produces and evaluates the model

50
General DiscussionAlternative Theories

• Deny that it is due to mental inference rules or
  non-logical heuristics
• What cognitive mechanism is responsible for these
  insights?
• Could be put together in a haphazard manner and
  checked for consistency.
• Fails to give a good account of systematic
  protocols
• Shifts burden of explanation to consistency
  checker

51
QA

• Questions? Thoughts?

52
General Discussion

• Natural-deduction explains where the items come
  from using intermediate sentences
• Challenge to mental modelers