Scientific Reasoning Competency Assessment in Higher Education in Virginia

1
Scientific Reasoning Competency Assessment in
Higher Education in Virginia

• Harold Geller
• hgeller_at_gmu.edu
• Department of Physics and Astronomy
• George Mason University

2
What Im Going to Talk About

• SCHEV Mandate for Core Competencies
• Scientific Reasoning Competency
• Basis of GMU Response
• Definition
• Standards
• Methodology
• Analysis and Presentation
• Commentary

3
SCHEV Mandate for Core Competencies

• The State Council of Higher Education for
  Virginia (SCHEV)
• Meeting on core competencies in November 2001
• See state issued guidelines
• Original deadlines modified in 2002
• Responses required were delayed to 2003

4
Scientific Reasoning Competency

• Not easily defined
• One approach
• Hazen and Trefil (1991)
• Ability to comprehend science article in
  newspaper
• Does this really define scientific reasoning?
• Can this be left to the media?
• Is this definition an end-all?
• What grade level does this represent?
• GMU approach
• See GMU Assessment Proposal

5
Basis of GMU Response

• Issues hashed out in committee
• Definition
• Implementation
• Which students take exam
• How many students take it
• How administered
• Ability to measure across disciplines
• Lowest common denominator
• Content independence

6
GMU Definition of Scientific Literacyby
committee

• Based upon 8 defined learning goals
• Identify different scientific ways of knowing
• Develop and test hypotheses
• Interpret data
• Recognize primary and secondary sources
• Utilize qualitative and quantitative techniques
• Recognize power and limitations of science
  process
• Communicate among scientists and general public
• Recognize role of science in decision making

7
GMU Standardsby committee

• To be set and established by committee consisting
  of
• All faculty that teach general education science
  courses
• Members of the Scientific Reasoning Competency
  Assessment Committee

8
GMU Methodologyby committee

• Develop common test items
• Reflect goals in definition
• Pilot questions
• Use in final examinations of sample of general
  education science courses
• Count as part of grade in class
• Analyze results of pilot testing
• Develop standards based upon results in sample of
  classes

9
As Part of Methodology(a syllabus addendum)by
committee

• Sample supplement to all general education
  science classes syllabi
• One of the goals of the general education
  program at George Mason University is to develop
  students competent in scientific reasoning. This
  course will help you achieve that goal. In order
  to judge university success in reaching this
  goal, a test will be given in late March or early
  April that will focus on scientific reasoning.
  This required multiple choice test will take
  approximately half an hour or so to complete and
  will count toward your grade.

10
Sample Question (vetoed)by Geller

• Johannes Kepler (1571-1630) derived his three
  Laws of Planetary Motion solely from the
  observations of Tycho Brahe (1546-1601). This
  represents which of the following ways of
  scientific knowing?
• A inductive
• B deductive
• C empirical
• D theoretical
• E none of the above

11
Sample Question (vetoed)by Geller

• The above graphs represent a phase diagram for
  water and carbon dioxide. Using the phase
  diagram for carbon dioxide, what phase would
  carbon dioxide be found at a pressure of 0.5
  atmosphere and a temperature of -78.5o C?
• A liquid
• B gas
• C solid
• D any of the above is possible
• E none of the above

12
Sample Question (vetoed)by Geller

• The above graphs all represent experimental
  values observed over a period of time. The graph
  that best represents a value that is increasing
  exponentially over time is which one of the above
  graphs?
• A Graph A
• B Graph B
• C Graph C
• D Graph D
• E None of the above

13
Sample Question (vetoed)by Geller

• Stefan-Boltzmann's Law relates the energy output
  of an object to its temperature. The energy is
  proportional to the temperature (in Kelvin) to
  the fourth power. Therefore, if the temperature
  (in Kelvin) of an object is doubled, its energy
  output will increase by how many times?
• A 2
• B 4
• C 8
• D 16
• E 32

14
Sample Questions (affirmed)Article from
Washington Post
Tumors Prevalent in Anacostia's Fish Rate Rivals
Highest Recorded for U.S. River Vehicle
Pollution Blamed By Monte Reel Fish in the
Anacostia River have cancerous tumor rates that
are as high as ever documented in an American
river, and a U.S. government-led study to be
published next month links the tumors to
pollution caused by vehicle emissions and
runoff. Fifty to 68 percent of mature brown
bullhead catfish collected in 2001 from three
parts of the river in the city had liver tumors,
most of which were cancerous, according to the
study led by the U.S. Fish and Wildlife Service.
In addition to the liver tumors, 13 to 23 percent
of the bullheads had skin tumors, scientists
found
15
Sample Questions (affirmed)based upon article
from Washington Post

1. Which one of the following graphs is consistent
   with the data presented in the article on tumors
   of Bullhead Catfish at a specific point in the
   Anacostia river? (for competency 3)

16
Sample Questions (affirmed)based upon article
from Washington Post

• Which of the following hypotheses are justified
  given the data in the article? You expect that
  fish who eat insects in the same river will
  (competency 2)
• Have the same level of tumors as brown bullhead
  catfish because they are living in the same
  environment.
• Have more tumors because the insects also pick up
  the air pollution.
• Have fewer tumors than bullhead catfish because
  they do not eat from the river bottom.
• Cant be predicted because what the fish eat does
  not matter.

17
GMU Analysis and Presentationby committee

• Present aggregate data to SCHEV
• Percentages of high proficiency
• Percentages of acceptable proficiency
• Percentages of low proficiency
• Adjust as needed
• Determination to be made in pilot testing

18
Commentaryby Geller

• How best to test scientific literacy (a.k.a.
  assessment)
• Essays, multiple choice, short answer, mixes
• Problems with need to apply across disciplines
• Lowest common denominator
• Content independence
• How important is content?
• Comparison with state examinations
• Standards of Learning (Virginia)
• Comparison with reading comprehension
• New York Times is aimed to be 9th grade reading
  level
• Washington Post aims at 8th grade reading level

19
Commentaryby Geller

• Value of tests
• Measure of higher education or lower education?
• Added burden to all lower level science classes
• What if students take more than one of these
  introductory science courses?
• Some students take astronomy as general elective.
• This can skew statistics.
• Unfunded mandates
• More attention to something other than the course
  itself?
• Why is this a college/university requirement?
• Failure of lower education?

20
Acknowledgements

• GMU Office of Institutional Assessment
• Karen Gentemann
• Ying Zhou
• Center for Teaching Excellence
• Laurie Fathe
• GMU Scientific Literacy Competency Committee
• Department representatives
• Faculty volunteers
• Department Colleagues
• Robert Ehrlich, Robert Ellsworth, Maria
  Dworzecka, Greg Foster (Chemistry), Joe
  Weingartner