Quality assessment in the Humanities & Social sciences: a conceptual framework and its application to bibliometry Jaak Billiet Royal Flemish Academy of Arts & Sciences (Belgium) Social Methodology K.U. Leuven Lecture prepared for meeting of Academia Europaea, Oslo June 20th, 2005

Bibliometry in Human & Social Sciences 2 introduction Option: maximize likelihood of not overlapping with Wim Blockmans intro (documents of KNAW Judging research… and KVAB Bibliometry in HS… very close in main conclusions) 1.Origin of Flemish document and differences in conception 2.A conceptual framework: valid measurement in comparative settings… main concepts applied to quality assessment of sciences 3.Some accents in the Flemish document

Bibliometry in Human & Social Sciences 3 1.Origin of KVAB document (1) Prior experiences Work in ESF on European citation index for human sciences and manifold of problems in this enterprise Problems met in workgroup of Flemish Inter-university Council in preparing European ranking system of journals for Flemish situation Problems met in research evaluations (teams, individuals) that were often dominated by principles of bio-medical sciences Preparation of documents Workgroup of 13 members of Flemish Academy (Human sciences plus observers from exact sciences and language & literature) chaired by M. Storme (jurist) and edited by H. De Dijn (Philosopher) Text prepared during six meetings + discussion of proposal in the class of human sciences of Academy and in board.

Bibliometry in Human & Social Sciences 4 Origin of KVAB document (2) Structure of the document Two parts and English Summary Part 1. View on bibliometrically guided quality assessment of publication output in the human sciences = a very critical view on the application of # of citations, impact factors, and ISI* ranking = in line of Dutch report (KNAW) (cross-references) Part 2. examples (applications) in several disciplines by small teams of academics - Philosophy - law - History - Sociology -Literature - Dutch language & literatur * Institute for Scientific Information (Philadelphia)

Bibliometry in Human & Social Sciences 5 2.A conceptual framework: valid measurement in comparative setting (1) basic idea: quality assessment = measurement Concepts originating from my own work in social methodology: cross cultural equivalence of latent variables measured by multiple indicators (cfr. Applied in data quality assessment of ESS) These concepts seem very useful for understanding and locating the dissatisfaction of human & social scientists with the application of bibliometric procedures of exact sciences Main concepts: ….

Bibliometry in Human & Social Sciences 6 A conceptual framework: valid measurement in comparative setting (2) main concepts (1) - distinction between theoretical validity and measurement validity (apart from reliability) - sharp distinction between the intended concept (complex concept of quality), the measured concept (obtained quality score), and the indicators used to measure the intended concept (citations, impact factors…) - the relative importance of a true score, but crucial importance of equal measurement error between groups (universities, disciplines, individual scientists) if one wants to compare - the concept of invariant measurement between groups (concept of quality and measured concept have the same meaning over units: universities, disciplines…..)

Bibliometry in Human & Social Sciences 7 A conceptual framework: valid measurement in comparative setting (3) main concepts (2) - the concept of interaction between measurement instrument, measurement error and characteristics of research units (difference in error on quality measurement according to discipline under investigation) - concept of unobtrusive measurement. Assumption of no effect of measurement instrument on what is being measured (no reactivity). Is this the case in application of bibliometric measures? Note: these concepts are developed in the context of quantitative (statistical) methodology but they are by analogy also applicable to qualitative methodology (different operationalisation) further explanation in next figure

Bibliometry in Human & Social Sciences 8 A conceptual framework: valid measurement in comparative setting (4) Measurement model idea of causal relationship between latent variable (LV) and four observed indicators (OV), and between measured latent variable and theoretical (intended) concept (TC) measurement validity theoretical validity? random error validity parameter (=equal for all disciplines?) e ov1 e ov2 method eLV TC effect ov3 e ov4 assumed to be 0 or equal over groups

Bibliometry in Human & Social Sciences 9 A conceptual framework: valid measurement in comparative setting (5) Example: comparison of research quality of a team in bio-medical science, Dutch literature, and Political science using as observed indicators - impact factors - # of publications in top ranked ISI journals - amount research budget - # of PhD students

Bibliometry in Human & Social Sciences 10 A conceptual framework: valid measurement in comparative setting (6) Conclusions in the two reports: assumptions not correct for these disciplines - no measurement invariance of indicators between the disciplines - strong difference in systematic measurement error (method effect) - reflection needed about correspondence between measured concept and intended concept of research quality (relation between measured latent variable and concept between disciplines = theoretical validity) - risk of reactivity: effect of measurement on published output (example: some young political scientists neglect studies in Dutch for Flemish audience and are likely to report only in English ISI journals; production of salami publications = cut one study in several parts in order to have more publications).

Bibliometry in Human & Social Sciences 11 A conceptual framework: valid measurement in comparative setting (7) conclusions (continued) - From the viewpoint of theoretical validity very serious danger in measurement: difference between theoretical concept, the latent variable and its observed indicators dissapeared, and the observed indicator coincides with the intented theoretical concept research quality IS what one measures by ISI !!! (= operationalism) in report of KNAW: serious reflection on the meaning of research quality = complex multidimensional concept that can have different meanings in different disciplines (problem = how to compare over disciplines in order to allocate budgets???). Quality is not only publicaton output. interesting lists of quality indicators dependent of kind of audiences

Bibliometry in Human & Social Sciences Accents in KNAW document (1) ( as far as they are not covered by previous speaker) The idea of a forum language. Not necessarily English, can vary according to (sub)disciplines Serious heterogeneity between disciplines and sub-disciplines: asks for qualified approach according to sub-discipline, and taking the mission state of research groups into account (problem = comparability over research groups…) example: for yearly evaluation of professors, K.U. Leuven used the ISI for exact sciences and branches in psychology and economy, but uses Sociological Abstracts and Academic bibliography for sociology… Method bias not only applied to differences between disciplines but also between paradigmas. Example: experimental psychology performs better with ISI assessment than phenomenological psychology

Bibliometry in Human & Social Sciences Accents in KNAW document (2) About BOOKS, books and books… no uniform evaluation system for all disciplines two final reflections: clarity in the communication to young pre-docs and post-docs… they should be sure that the rules that they have learned and oriented their work should be the rules that will be applied to them in later evaluations in the whole of Europe… (fairness) Asking for a more qualified system depending on mission state and on disciplines does not mean that publishing in ISI should be avoided!!! (in my field: reviewers in ISI journals are top specialists and one can learn much from their evaluations, even in case of rejection)

Bibliometry in Human & Social Sciences Accents in KNAW document (3) Finally: one may expect serious reflection, discussion, and even research on the theoretical and measurement validity of the quality indicators before they are applied in evaluation systems and in research policy…