To Use or Not To Use: The Case of the Program for International Student Assessment PISA and How the

1
To Use or Not To Use The Case of the Program
for International Student Assessment (PISA) and
How the United States of America Deals With Its
Implications Compared to Other CountriesWerner
W. WittmannUniversity of Mannheim, Germany
Chair Patrick E. McKnight (George Mason
University) Evaluation 2006The Consequences of
Evaluation Portland, Oregon Nov. 1th 4th
2
Outline

• The OECD-PISA assessments
• Demands on the workforce
• Knowledge profile components and their
  distributions using international benchmarks
• Potential consequences I The case of academic
  psychology
• Potential consequences II The case of the
  economy
• The role of foreign-born graduate students,
  postdocs, faculty and workforce
• Summary and (pinc?) conclusions

3
Program of International Student Assessmentrun
by OECD-Paris

• PISA2000 focused on reading skills, but assessed
  math and science knowledge as well.
• PISA2003 focused on math, but assessed reading,
  science and problem solving skills as well.
• PISA2006 focused on science, but assessed reading
  and math knowledge as well.
• Assessed is the knowledge of 15 year old students
  in representative samples for most OECD plus
  additional countries.
• The knowledge items had been developed by
  consensus of international expert groups, what
  they believe is relevant and important in real
  life in a globalized( flat?) world.
• The assessments allow to map the knowledge
  profiles of the incoming workforce of today and
  tomorrow.

4
PISA Results

• Disastrous results for some countries e.g.
  Germany and the United States of America.
• Excellent results for some North European and
  Asian countries, e.g. Finland, Japan and good
  results for Canada.
• Heated political discussions about the causes and
  reasons, especially in Germany,Austria and
  Switzerland.
• Not much discussions and excitement in the USA
• Why this? Are there no consequences to be taken
  out of these evaluations?

5
The danger of Brunswik-asymmetry in validation
strategies
6
The danger of asymmetry in validation strategies
7
Demands on the workforce, related implications
for policies and for the economy.

• CP SNOWs distinction of the two cultures
• Buz Hunts 1995 question Will we be smart
  enough? Cognitive demands on the incoming work
  force.
• Camilla Benbow and David Lubinskis focus on
  tilted profiles in aptitude and achievement

8
Tilted Profiles
9
Tilted Profiles
VERBAL
QUANT
University of .... (e.g. Harvard School of
Education)
10
Tilted Profiles
QUANT
VERBAL
University of .... (e.g. MIT, Caltech,
Georgiatech)
11
Tilted Profiles
QUANT
VERBAL
University of .... (e.g. your University ?)
12
Tilted Profiles as demands of the workplace
VERBAL
QUANT
13
Tilted Profiles as demands of a different
workplace
QUANT
VERBAL
14
Even Profiles of still another one
QUANT
VERBAL
15
Applying a three-group classification to the
PISA2003 data in terms of level and scatter/shape
of the knowledge profiles.

• Two factor unrotated principal component analysis
  applied to PISA2003 reading and math scales lead
  to a general knowledge component mapping overall
  level of reading and math both approx. equally
  weighted.
• The second component contrasted reading and math
  and is the scatter/shape component, with low
  scores mapping math tiltedness and high scores
  verbal tiltedness.
• Both components were banded into three equally
  sized groups, leading to a high, medium and low
  level group at the first component and a verbal
  tilted, an even and a math tilted group at the
  second component.
• Expected benchmark numbers derived by analyzing
  all participating countries are 33.33 at the
  margins and 11.11 within each cell, assuming
  both component are normally distributed and
  orthogonal.

16
Tilted Profiles
17
Tilted Profiles
18
Tilted Profiles
19
Pisa2003-profiles Level and shape group
percentages
The following results are for 32 OECD
Countries Percents of total count all PISA_OECD
countries Tiltedness/shape (rows) by level
(columns) of PISA-profiles
lower middle upper Total
N third
third third Verbal 11.0 12.4 9.9 33.3 74
713 Even 10.6 11.1 11.6 33.3 74690 Quant 11.7 9.
8 11.8 33.3 74691 Total 33.3 33.3 33.3 100.0
N 74691 74690 74713 224094
20
Pisa 2003-profiles Level and shape group
percentages
The following results are for United States
Percents of total count Tiltedness (rows) by
level of profile (columns) m male f
female low middle high
Total N Verbal f
18.0 24.3 21.7 64.0 1738 m
6.6 10.1 7.5 24.2 662 Even f
10.5 10.1 9.4 29.9 811 m 17.8 15.0 12.8 45.7 1
252 Quant f 3.1 1.7 1.3 6.1 166 m
14.2 8.1 7.9 30.1 826 Total f
31.6 36.0 32.4 100.0 m 38.6 33.1 28.3 100.0
N f 858 978 879 2715 m 1057 908 775 2740
21
Pisa 2003-profiles Level and shape group
percentages
The following results are for Australia
Percents of total count Tiltedness (rows) by
level of profile (columns) m male f
female low middle high
Total N Verbal f
9.4 20.0 26.9 56.3 3500 m
4.8 7.0 7.3 19.2 1216 Even f
6.0 9.1 16.5 31.6 1963 m 8.1 11.7 16.8 36.6
2319 Quant f 3.0 3.2 6.0 12.1 753 m
13.0 12.5 18.6 44.2 2800 Total f
18.4 32.3 49.4 100.0 m 26.0 31.3 42.7 100.0
N f 1142 2005 3069 6216 m
1647 1981 2707 6335
22
Pisa 2003-profiles Level and shape group
percentages
The following results are for New Zealand
Percents of total count Tiltedness (rows) by
level of profile (columns) m male f
female low middle high
Total N Verbal f
7.5 17.0 27.9 52.4 1166 m
3.2 5.6 9.3 18.2 416 Even f
8.0 10.6 15.3 33.9 755 m 7.8 10.5 17.2 35.5
811 Quant f 4.8 3.5 5.4 13.6 303 m
14.3 13.0 19.0 46.3 1059 Total f
20.4 31.0 48.6 100.0 m 25.3 29.1 45.5 100.0
N f 453 690 1081 2224 m 579 666 1041 2286
23
Pisa 2003-profiles Level and shape group
percentages
The following results are for United Kingdom
Percents of total count Tiltedness (rows) by
level of profile (columns) m male f
female low middle high
Total N Verbal f
9.3 17.7 19.3 46.3 2257 m
3.0 5.5 5.9 14.4 671 Even f
10.3 13.5 17.8 41.5 2023 m 10.4 15.5 17.4 43.3
2020 Quant f 3.0 4.1 5.0 12.2 592 m
13.4 13.1 15.8 42.3 1972 Total f
22.7 35.2 42.1 100.0 m 26.8 34.1 39.1 100.0
N f 1104 1716 2052 4872 m
1248 1590 1825 4663
24
Pisa 2003-profiles Level and shape group
percentages
The following results are for Canada Percents
of total count Tiltedness (rows) by level of
profile (columns) m male f female low
middle high Total
N Verbal f 7.3 18.8 21.9 48.1 6609 m
2.4 5.3 5.3 13.0 1754 Even f
6.8 13.4 17.7 37.9 5210 m
8.7 13.0 15.2 36.8 4958 Quant f
3.2 4.5 6.4 14.0 1929 m 14.6 15.6 20.0 50.2
6757 Total f 17.3 36.7 46.0 100.0 m
25.6 33.9 40.5 100.0 N f 2379 5043 6326 13748
m 3454 4562 5453 13469
25
Profile tiltedness PISA 2003 OECD_CountriesHigh
Level Group only
11 OECD-Benchmark
11 OECD-Benchmark
26
A Prominent Tilted Profile !Do the USA no longer
educate people like him?
27
Too few in quantitative psychology.The
subdiscipline sees shrinking numbers, but growing
opportunities.  BY REBECCA CLAY (APA
Monitor,2005)

• "There aren't enough of us quantitative people,
  and many of us are getting to be more senior,"
  says Appelbaum, a psychology professor at the
  University of California , San Diego . "We're now
  getting to the point where the first piece of
  mail we open is the one that has the balance in
  the retirement account."
• A supply/demand mismatch
• it wasn't always like this.
• In an influential 1990 article in the American
  Psychologist (Vol. 45, No. 6, pages 721734),
  Leona S. Aiken, PhD, Stephen G. West, PhD, and
  colleagues describe the heyday of quantitative
  psychology. Psychologists of the 1960s, they
  write, saw themselves as leaders in statistical,
  measurement and design issues. Psychology
  departments often had quantitative specialists,
  and graduate students were well equipped to
  handle the quantitative aspects of their
  research. By 1990, that legacy had faded along
  with the number of students aware of, interested
  in and able to enter the field.
• "A lot of the major quantitative programs over
  the years have died," says West. "We're one of
  three larger programs in psychology in the
  country, and we produced one PhD this year."
• Why is it so hard to attract students to
  quantitative psychology? One problem is the fact
  that quantitative psychology faculty are "few and
  far between," says Aiken, who chairs the Arizona
  State University quantitative concentration in
  psychology. Another is the paucity of American
  students with sufficient interest or ability in
  math. Even those who are qualified may not pursue
  quantitative training, since undergraduate
  mentors and counselors are often unfamiliar with
  the subdiscipline. Some even steer students away.
  Meanwhile, some quantitative programs recruit
  highly qualified international students. On
  graduation many of these students return home
  others remain in North America but often take
  research rather than teaching positions.

28
Wide-ranging implications

• Take teaching "There's a shortage of faculty
  trained in quantitative measurement today that's
  likely to lead to an even greater problem in the
  next generation of faculty,"says Paul Nelson,
  PhD, deputy executive director for education at
  APA.
• Because of that shortfall in training,
  researchers and clinicians alike often lack the
  skills they need to interpret ever more
  sophisticated science. Grant review committees
  now increasingly demand a methodological expert
  on the project team.
• Finding qualified reviewers for journal
  submissions is another problem.
• "APA has over 45 editors, and they all have in
  their Rolodexes the name and address of the same
  24 people--almost all of whom are over the age of
  60," says APA Publisher Gary R. VandenBos, PhD.
• Attracting students
• Fears about the shortage prompted APA's
  Publications and Communications Board to propose
  the creation of a joint task force with APA's
  Board of Scientific Affairs and Board of
  Educational Affairs. The three boards will
  discuss the possibility of a formal proposal when
  they meet in November.

29
Economic ContributionWho contributes most, high
level verbal tilted or high level quant tilted
workforce?

• Solow's model of economic growth, often known as
  the neo-classical growth model, allows the
  determinants of economic growth to be separated
  out into increases in inputs (labour and capital)
  and technical progress. Using his model, Solow
  calculated that about four-fifths of the growth
  in US output per worker was attributable to
  technical progress.
• Since Solow's initial work in the 1950s, many
  more sophisticated models of economic growth have
  been proposed, leading to varying conclusions
  about the causes of economic growth. In the 1980s
  efforts have focused on the role of technological
  progress in the economy, leading to the
  development of endogenous growth theory (or new
  growth theory).

30
The Pisa data,IQ,oil,socialism and economic
prosperity

• Earl B. Hunt (Buz), University of Washington
  hinted me to the controversial LynnVanhanen book
  about IQ and the wealth of nations and so we
  related the PISA data to what was available from
  LV.
• Using all countries(32) from PISA2003 we found
  the level factor to be most predictive of
  gdp/c.(Hunt and Wittmann,2006, paper under review
  at Intelligence).
• In the meantime I have restricted the number of
  countries to 24 most developed ones and found the
  tilt factor to be more predictive than the level
  factor. This is no surprise because the most
  developed ones do not differ so much in the level
  (restriction of range), they differ more on tilt!

31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
Tentative Robust Path-Model(ranks) using 24
countries
AUT BEL CAN CHE CZE DNK ESP FIN FRA GBR GER GRC HU
N IRL ITA JPN NLD NOR POL PRT SVK SWE USA
36
Extended path-model regressing the residualized
gdp/c-res onto school variables, a tribute to
James Coleman
AUS AUT BEL CAN CHE CZE DNK ESP FIN FRA GBR GER G
RC HUN IRL ITA JPN NLD NOR POL PRT SVK SWE USA
STMORALE Student moral index TCMORALE Teacher
moral index TEACHBEHA Teacher behavior
index TCSHORT Teacher shortage SCMATEDU
Quality of educational materials IQ_COUNT
Country IQ from LynnVanhanen GDP/c_res
Logrealgdp from LynnVanhanen with oil and
socialism dummies partialled out
37
Rising Above The Gathering Storm Energizing and
Employing America for a Brighter Economic Future
Committee on Prospering in the Global Economy of
the 21st Century An Agenda for American Science
and Technology, National Academy of Sciences,
National Academy of Engineering, Institute of
Medicine6 x 9, 512 pages, 2006.
1) Increase America's talent pool by vastly
improving K-12 mathematics and science education
2) Sustain and strengthen the nation's
commitment to long-term basic research 3)
Develop, recruit, and retain top students,
scientists, and engineers from both the U.S. and
abroad and 4) Ensure that the United States is
the premier place in the world for innovation.
38
Share of foreign-born scientists and engineers in
U.S. SE occupations, by degree level 1990 and
2000
39
Composition of academic doctoral SE workforce
by race/ethnicity, sex, and citizenship at
degree conferral 19892003
40
Policy Implications of International Graduate
Students and Postdoctoral Scholars in the United
States Commitee on Policy Implications of
International Graduate Students and Postdoctoral
Scholars in the United States, Board on Higher
Education and Workforce, National Research
Council ISBN 0-309-09613-8, 196 pages, 6 x 9,
paperback (2005)

• In 1966, 78 percent of SE doctorates were
  US-born and 22 percent were foreign-born.
  In 2000, 61 percent
  were US-born and 39 percent were
  foreign-born.
• In 2003, international students earned 38 percent
  of the US awarded SE doctorates and
• 
  58.9 percent of the engineering
  doctorates.
• Among SE postdoctoral scholars, the share of
  temporary residents has increased from
  
• 
  37 percent in 1982 to
  
• 
  59 percent in 2002.
• More than one-third of US Nobel laureates are
  foreign-born.
• Nearly half the doctorate-level staff and 58
  percent of the
• postdoctoral, research, and clinical fellows at
  the National Institutes of Health campus are
  foreign nationals.
• For SE occupations, data from the 2000 US
  Census indicate that
• about 38 percent of doctorate-level employees are
  foreign-born, compared
• with 24 percent in 1990.
• Of the SE tenure-track and tenured faculty, 19
  percent are foreign-born
• in engineering fields,foreign-born
  hold 36 percent of faculty positions.

41
PhD students and post-docs

• The number of doctorate recipients with SE
  postdoctoral appointments at U.S. universities
  more than doubled in the past two decades.
• Noncitizens account for most of the increase in
  SE postdocs during the period.
• Noncitizens accounted for 58 of SE postdocs in
  2003.
• About two-thirds of SE postdocs are in the
  biological/medical/other life sciences.

42
Origin of foreigners earning U.S. SE doctorates
19832003
43
Foreign student plans to stay in United
Statesafter receipt of U.S. SE doctorate
19832003
44
Change in faculty composition

• A recent study further delineates the changing
  demographics. Among postdoctoral scholars, the
  participation rate among temporary residents has
  increased from 37.4 percent in 1982 to 58.8
  percent in 2002.
• Similarly, the share of foreign-born faculty who
  earned their doctoral degrees at US universities
  has increased from 11.7 percent in 1973 to 20.4
  percent in 1999.
• In engineering fields, the share increased from
  18.6 percent to 34.7 percent in the same period.

45
The importance of foreign-born scientists and
engineers to the SE enterprise in the United
States continues to grow.

• Twenty-five percent of all college-educated
  workers in SE occupations in 2003 were foreign
  born.
• Forty percent of doctorate degree holders in SE
  occupations in 2003 were foreign born.
• Among all doctorate holders resident in the
  United States in 2003, a majority in computer
  science (57), electrical engineering (57),
  civil engineering (54), and mechanical
  engineering (52) were foreign born.

46

• Foreign-born scientists and engineers constituted
  23 of scientists and engineers with U.S.
  doctorates in academic employment in 2003. This
  lower bound estimate of foreign-born doctorate
  holders excludes doctorates from foreign
  institutions.
• The share of foreign-born doctorate holders was
  more than double that in 1973, when it stood at
  11.
• Academic employment of foreign-born doctorate
  holders was highest in the computer sciences and
  engineering (44 and 40, respectively), followed
  by mathematics (33), the physical sciences
  (25), and the life sciences (22).

47
Quality of International Graduate Students

• How can quality of international graduate
  students be assessed? Several factors play a
  substantial role in graduate-student admissions
  decisions. Among them are selectivity of the
  institution, applicant Graduate Record
  Examination (GRE) scores and undergraduate grade
  point average, undergraduate major, prior
  research experience, and quality of the
  applicants undergraduate institution. The match
  between the research interests of the applicant
  and those of departmental faculty also plays a
  role. Studies show that many admissions
  committees make implicit adjustments in the GRE
  verbal score for applicants from
  non-English-speaking coun-

48

• From 1993 to 2001, for example, numbers of US
  citizens and permanent resident graduate students
  dropped by 10 percent (with numbers of white
  males dropping by 26 percent) while numbers of
  temporary resident graduate students rose by 31
  percent. National Science Board. 2004. Science
  and Engineering Indicators 2004 (NSB 04-1).
  Arlington, VA National Science Foundation, p.
  2-16.

49
    EQS PATH-Diagram PISA_causalmodel1     List
of the 24 countries used for this model
analysis   AUT BEL CAN CHE CZE DNK ESP FIN FRA GB
R GER GRC HUN IRL ITA JPN NLD NOR POL PRT SVK SWE
USA                 STMORALE Student moral index
SCMATEDU Quality of educational
materials PROPMATH proportion of math teachers A
detailed description of these indices is to be
found in VariableInformation_Schoolindices.doc IQ_
COUNT Country IQ from LynnVanhanen ZRE2
Logrealgdp from LynnVanhanen with oil and
socialism dummies partialled out.
AUS
50
The importance of foreign-born scientists and
engineers tothe SE enterprise in the United
States continues to grow.
t Twenty-five percent of all college-educated
workers in SE occupations in 2003 were foreign
born. t Forty percent of doctorate degree holders
in SE occupations in 2003 were foreign born. t
Among all doctorate holders resident in the
United States in 2003, a majority in computer
science (57), electrical engineering (57),
civil engineering (54), and mechanical engineerin
g (52) were foreign born.
51
(No Transcript)
52
(No Transcript)
53
Change in first-time full-time graduate
enrollment in SE, by citizenship status
200103
54
Summary and anticipated consequences of evaluation

• There are some warning signs on the wall,who
  reads, understands the consequences and transfers
  them into effective policies?!
• What is wrong with K-12 (and maybe K-16)
  education?
• Is it a paradox that the odds of foreign borns to
  get a top-notch US-education and finally a higher
  level position in some major areas are higher
  than the odds of those born into this country?
• Who is left behind?
• Fortunately there are now many at NSF and NAS
  who are focussing these problems.

55
Apologies

• Sorry, if parts of my presentation seemed PINC to
  you, but a real noncitizen friend tried talking
  straight to you.
• Thanks for not throwing wrotten tomatoes on me
  and thank you for your attention.