Quality assessment in the Humanities

1
Quality assessment in the Humanities Social
sciencesa conceptual framework and its
application to bibliometry

• Jaak Billiet
• Royal Flemish Academy of Arts Sciences
  (Belgium)
• Social Methodology K.U. Leuven
• Lecture prepared for meeting of Academia
  Europaea, Oslo June 20th, 2005

2
introduction

• Option maximize likelihood of not overlapping
  with Wim Blockmans intro (documents of KNAW
  Judging research and KVAB Bibliometry in HS
  very close in main conclusions)
• Origin of Flemish document and differences in
  conception
• A conceptual framework valid measurement in
  comparative settings main concepts applied to
  quality assessment of sciences
• Some accents in the Flemish document

3
Origin of KVAB document (1)

• Prior experiences
• Work in ESF on European citation index for human
  sciences and manifold of problems in this
  enterprise
• Problems met in workgroup of Flemish
  Inter-university Council in preparing European
  ranking system of journals for Flemish situation
• Problems met in research evaluations (teams,
  individuals) that were often dominated by
  principles of bio-medical sciences
• Preparation of documents
• Workgroup of 13 members of Flemish Academy (Human
  sciences plus observers from exact sciences and
  language literature) chaired by M. Storme
  (jurist) and edited by H. De Dijn (Philosopher)
• Text prepared during six meetings discussion of
  proposal in the class of human sciences of
  Academy and in board.

4
Origin of KVAB document (2)

• Structure of the document
• Two parts and English Summary
• Part 1. View on bibliometrically guided quality
  assessment of publication output in the human
  sciences a very critical view on the
  application of of citations, impact factors,
  and ISI ranking in line of Dutch report
  (KNAW) (cross-references)
• Part 2. examples (applications) in several
  disciplines by small teams of academics-
  Philosophy - law- History -
  Sociology-Literature - Dutch language
  literatur
• Institute for Scientific Information
  (Philadelphia)

5
2. A conceptual framework valid measurement in
comparative setting (1)

• basic idea quality assessment measurement
• Concepts originating from my own work in social
  methodology cross cultural equivalence of
  latent variables measured by multiple indicators
  (cfr. Applied in data quality assessment of ESS)
• These concepts seem very useful for understanding
  and locating the dissatisfaction of human
  social scientists with the application of
  bibliometric procedures of exact sciences
• Main concepts.

6
A conceptual framework valid measurement in
comparative setting (2)

• main concepts (1)
• - distinction between theoretical validity and
  measurement validity (apart from
  reliability)- sharp distinction between the
  intended concept (complex concept of quality) ,
  the measured concept (obtained quality score),
  and the indicators used to measure the intended
  concept (citations, impact factors)- the
  relative importance of a true score, but
  crucial importance of equal measurement error
  between groups (universities, disciplines,
  individual scientists) if one wants to compare-
  the concept of invariant measurement between
  groups (concept of quality and measured concept
  have the same meaning over units universities,
  disciplines..)

7
A conceptual framework valid measurement in
comparative setting (3)

• main concepts (2)
• - the concept of interaction between measurement
  instrument, measurement error and
  characteristics of research units (difference in
  error on quality measurement according to
  discipline under investigation)
• - concept of unobtrusive measurement.
  Assumption of no effect of measurement
  instrument on what is being measured (no
  reactivity). Is this the case in application of
  bibliometric measures?
• Note these concepts are developed in the
  context of quantitative (statistical) methodology
  but they are by analogy also applicable to
  qualitative methodology (different
  operationalisation)
• further explanation in next figure

8
A conceptual framework valid measurement in
comparative setting (4)

• Measurement model
• idea of causal relationship between latent
  variable (LV) and four observed indicators (OV),
  and between measured latent variable and
  theoretical (intended) concept (TC)
• measurement validity theoretical validity?
  random error
  validity parameter (equal for all
  disciplines?)
• e
• ov1
• e
• ov2
• method e LV ?
  TC
• effect ov3
• e
• ov4
• assumed to be 0 or equal over groups

9
A conceptual framework valid measurement in
comparative setting (5)

• Example comparison of research quality of a
  team in bio-medical science, Dutch literature,
  and Political science using as observed
  indicators - impact factors - of publications
  in top ranked ISI journals - amount research
  budget - of PhD students

10
A conceptual framework valid measurement in
comparative setting (6)

• Conclusions in the two reports assumptions not
  correct for these disciplines- no measurement
  invariance of indicators between the
  disciplines- strong difference in systematic
  measurement error (method effect)- reflection
  needed about correspondence between measured
  concept and intended concept of research quality
  (relation between measured latent variable and
  concept between disciplines theoretical
  validity)- risk of reactivity effect of
  measurement on published output (example some
  young political scientists neglect studies in
  Dutch for Flemish audience and are likely to
  report only in English ISI journals production
  of salami publications cut one study in several
  parts in order to have more publications).

11
A conceptual framework valid measurement in
comparative setting (7)

• conclusions (continued)
• From the viewpoint of theoretical validity very
  serious danger in measurement difference between
  theoretical concept, the latent variable and its
  observed indicators dissapeared, and the observed
  indicator coincides with the intented theoretical
  conceptresearch quality IS what one measures by
  ISI !!! ( operationalism)
• in report of KNAW
• serious reflection on the meaning of research
  quality
• complex multidimensional concept that can have
  different meanings in different disciplines
  (problem how to compare over disciplines in
  order to allocate budgets???). Quality is not
  only publicaton output.
• interesting lists of quality indicators dependent
  of kind of audiences

12
3. Accents in KNAW document (1)(as far as they
are not covered by previous speaker)

• The idea of a forum language. Not necessarily
  English, can vary according to (sub)disciplines
• Serious heterogeneity between disciplines and
  sub-disciplines asks for qualified approach
  according to sub-discipline, and taking the
  mission state of research groups into account
  (problem comparability over research
  groups)example for yearly evaluation of
  professors, K.U. Leuven used the ISI for exact
  sciences and branches in psychology and economy,
  but uses Sociological Abstracts and Academic
  bibliography for sociology
• Method bias not only applied to differences
  between disciplines but also between paradigmas.
  Example experimental psychology performs better
  with ISI assessment than phenomenological
  psychology

13
3. Accents in KNAW document (2)

• About BOOKS, books and books
• no uniform evaluation system for all disciplines
  
• two final reflections
• clarity in the communication to young pre-docs
  and post-docs they should be sure that the rules
  that they have learned and oriented their work
  should be the rules that will be applied to them
  in later evaluations in the whole of Europe
  (fairness)
• Asking for a more qualified system depending on
  mission state and on disciplines does not mean
  that publishing in ISI should be avoided!!! (in
  my field reviewers in ISI journals are top
  specialists and one can learn much from their
  evaluations, even in case of rejection)

14
3. Accents in KNAW document (3)

• Finally
• one may expect serious reflection, discussion,
  and even research on the theoretical and
  measurement validity of the quality indicators
  before they are applied in evaluation systems and
  in research policy