Research%20Methods%20Project

1
Research Methods Project

• Joshua Warren
• Dr. Black
• 5/1/06

2
Case Based Reasoning

• Case-based reasoning is the ability to solve a
  new problems based on generalizations from
  examples of prior occurrences. It is the ability
  to analogize and use those generalizations to
  make predictions about future cases.
• Comprehension of cases
• Generalization
• Rule by analogy

3
Purpose

• Is there a correlation between legal education
  and case-based reasoning skills?
• Hypothesis On an assessment of case-based
  reasoning, law students should perform
  differently from psychology students.

4
Test of CBR

• Presents Cases
• Asks hypotheticals with new combinations of
  facts.
• THIS TEST HAS NO PSYCHOMETRIC RELIABILITY OR
  VALIDITY

5
Sample

• N20 (10 per group)
• Group Psych - psychology doctoral students in
  cognitive studies and developmental psychology
• Group Law - subjects who are near completing law
  school or recently completed law school.

6
METHOD

• Subjects were provided a short quiz consisting of
  one page of cases and three multiple choice
  hypotheticals to solve.
• The quiz was sent via email as a MSWord .doc file
  and subjects were asked to spend 10-15 minutes
  completing the three questions and then email
  back answers.

7
ill-suited Problem

• There is not always enough information to solve
  the problem.
• Some situations are like the previous ones and
  some are new and different.
• There are rules that govern but we dont know
  what they are.
• Sometimes we can deduce the rules, sometimes we
  can induce the rules and sometime we can just
  intuit the rules.
• This is law / life? (do we create our answers?)

8
The Arbitrary Rules of this Game

• SciFi fails no matter what.
• Harry was successful from 1970-1990 but has
  failed since
• Westerns win awards but tend not to make money.
• Love stories only make money if George is in them
• Phillip started out as a critical success from
  1970-1990 but then sold out and made money from
  1990 on.

9
Question 1

• It is the year 2008 and you are Phillips agent.
  You only make money if his movie makes 30 million
  dollars. Which of the following movies do you
  try to cast him in?
• A sci-fi
• A war movie.
• A western
• A love-story
• Explain Why and Why Not.

10
Answer 1

• A and D will fail because sci-fis always fail
  and because love-stories are only successful with
  George. Phillip is on a hot streak post-1990, so
  his success is very well guaranteed by B. A
  western is possible but they tend not to make
  money

11
Question 1 responses
12
Question 2

• It is the year 2008 and you are a Hollywood
  Producer, your goal is to make money for a Big
  Hollywood studio. Which of these movies will make
  30 million dollars for the studio?
• A war movie starring Harry.
• A mystery starring Phillip.
• A sci-fi starring Roger.
• A love-story starring George.
• Explain Why and Why Not.

13
Answer 2

• A is failed because Harry is a flop now. B
  should succeed because Phillip is doing well. C
  will fail because sci-fis fail. D succeeds
  because love stories with George make money.

14
Question 2 responses
15
Question 3

• It is the year 2008 and you are an Independent
  Producer, your goal is to win awards, what movie
  do you make next? Why?
• A western starring Phillip.
• A western starring Roger.
• A sci-fi starring George.
• A war movie starring Harry.
• Explain Why and Why Not

16
Answer 3

• Phillip is a sell out now but westerns tend to
  win awards so its a risk but possible. Its
  more likely with Roger (b) because Roger is not a
  sellout so this will not decrease the likelihood
  of awards (rule-interaction). C is doomed as are
  all sci-fis. And D is doomed because Harry is a
  wash-up.

17
Question 3 responses
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
Possible Results Evaluated

• If law students perform better this is evidence
  that law school improves CBR or that students
  with high CBR self-select into law school
• If other students outperform law students then it
  is possible that other factors uncontrolled in
  the study effect CBR more than law school
  education (like gender or race or age or
  explanatory style).still a possibility here
• If both groups perform equally (or not
  statistically different) then either the test
  lacks power to detect or there may be a ceiling
  effect.

24
Arbitrary Rules Identified

• SciFi fails no matter what.
• 1A because Phillip had never chance to make
  sci-fi movie yet sci-fi would be closest them to
  an adventure movie.
• Harry was successful from 1970-1990 but has
  failed since
• Why?
• P3D - Harry has a unique ability to make a war
  movie.
• P3D - One bad movie is not a trend
• L3A - has Harry really lost it (is his life a
  wreck ? Is he off the wagon?), and would this
  affect his career. Because Harry has a history
  of making more great movies in war, I would pick
  him as long as his life isnt a wreck.
• Westerns win awards but tend not to make money.
• Love stories only make money if George is in them
• Phillip started out as a critical success from
  1970-1990 but then sold out and made money from
  1990 on.
• The Roger Factor
• L3A Roger is no longer an issue
• L3A Philip is historically the highest award
  winner Philips big name could be a draw
• L3B the critics have become disenfranchised
  with Philip

25
But remember is it a valid test???

• There is nothing but facial validity.
• Future research should find a validity measure
• The test is not correlated to any criterion score
  nor was it constructed by experts in the field.
  It has no construct or criterion validity and its
  content validity is questionable.
• No matter what the results are, they provide weak
  evidence of anything.

26
Results

• The law student sample was slightly more likely
  to see the best answer to the ill- suited
  problem.
• The statistical difference is unclear and will
  require a larger study with more subjects and
  thus a greater power.

27
Other problems

• The sample is really small so even if significant
  the result may be a fluke because its not a
  random sample of any population and the groups
  are likely more representative as groups of my
  friends then as groups of their respective
  populations.

28
Future research

• Redesign the survey to get clearer responses and
  better explanations. Differences in question
  type should be magnified.
• Need much bigger sample.
• Need to get better statistical controls.
• NEED a validity measure.