Bibliometrics: coming ready or not CAUL, September 2005 Cathrine Harboe-Ree

Information kit Introduction Introduction Assessing Quantitative Performance Indicators Workshop Assessing Quantitative Performance Indicators Workshop –Reading list Citation Analysis in Research Evaluation, Henk F Moed Citation Analysis in Research Evaluation, Henk F Moed Managing Australian Research Output for Increased Return on Investment: the Role of Open Access Institutional Repositories, Cathrine Harboe-Ree Managing Australian Research Output for Increased Return on Investment: the Role of Open Access Institutional Repositories, Cathrine Harboe-Ree Research Indicator Databases available from Thomson Scientific (ISI) Research Indicator Databases available from Thomson Scientific (ISI)

Introduction What is bibliometrics? What is bibliometrics? Why is bibliometrics important? Why is bibliometrics important? What’s bibliometrics got to do with us? What’s bibliometrics got to do with us?

Quantitative Performance Indicators Workshop ARC Linkage project ARC Linkage project –Range of performance indicators to assess research performance –Assessment of each indicator for validity, etc –Evaluation of different performance measures on institutions, groups and individuals –Applicability of field-specific weights to counter inequalities ANU, UQ, DEST ANU, UQ, DEST Chief Investigator – Linda Butler Chief Investigator – Linda Butler 16 May workshop to test various indicators 16 May workshop to test various indicators

Quantitative Performance Indicators Citations to non-source publications Citations to non-source publications Journal publications classified by impact quartiles Journal publications classified by impact quartiles Identification of highly cited publications Identification of highly cited publications Book publications weighted by publisher prestige Book publications weighted by publisher prestige Honours, awards and prizes Honours, awards and prizes Election to learned academies and professional academic associations Election to learned academies and professional academic associations Conferences Conferences Service to journals Service to journals Visiting Fellowships Visiting Fellowships Office Bearers in learned academies and professional academic associations Office Bearers in learned academies and professional academic associations Copyright Agency Limited (CAL) data Copyright Agency Limited (CAL) data Membership of grants committees Membership of grants committees

Quantitative Performance Indicators findings Number of publications as the sole indicator is not valid Number of publications as the sole indicator is not valid Some sort of peer review process is desirable Some sort of peer review process is desirable Several indicators should be used in combination Several indicators should be used in combination Multiple indicators should not be aggregated to a single number Multiple indicators should not be aggregated to a single number Quality measures have to be responsive to their fields Quality measures have to be responsive to their fields Individual or small unit assessment is expensive and inappropriate Individual or small unit assessment is expensive and inappropriate Journal impact factors of the ISI type are highly inappropriate Journal impact factors of the ISI type are highly inappropriate Process/technical errors can develop over time, invalidating all data Process/technical errors can develop over time, invalidating all data

Bibliography Linda Butler’s ARC reference list Linda Butler’s ARC reference list Henk F. Moed Citation analysis in research evaluation Henk F. Moed Citation analysis in research evaluation

Henk Moed: validity elements Formal Formal Open Open Scholarly founded Scholarly founded Supplemented with expert knowledge Supplemented with expert knowledge Carried out in clear policy context Carried out in clear policy context Explicit statement of basic notions of scholarly quality Explicit statement of basic notions of scholarly quality Enlightening rather than formulaic Enlightening rather than formulaic

Henk Moed: Thomson ISI Coverage excellent in physics, chemistry, molecular biology, biochemistry, biological sciences, clinical medicine Coverage excellent in physics, chemistry, molecular biology, biochemistry, biological sciences, clinical medicine Good in applied and engineering sciences, biological sciences, geosciences, mathematics, medicine and health related social sciences Good in applied and engineering sciences, biological sciences, geosciences, mathematics, medicine and health related social sciences Moderate in sociology, political science, anthropology, education and humanities Moderate in sociology, political science, anthropology, education and humanities Affected by Affected by –non-source material (books, conference proceedings –Language and national barriers

Henk Moed: findings Alternative data sources and methodologies should be explored Alternative data sources and methodologies should be explored The expertise of specialists is important The expertise of specialists is important There is no single ‘perfect’ indicator of journal performance There is no single ‘perfect’ indicator of journal performance Thomson ISI journal impact factors are not a reliable guide Thomson ISI journal impact factors are not a reliable guide Data accuracy is a key problem Data accuracy is a key problem The substantive content of work should be taken into account The substantive content of work should be taken into account The level of aggregation is crucial The level of aggregation is crucial Both citation analysis and peer review have strengths and limits Both citation analysis and peer review have strengths and limits A study of RAE questions its ability to identify ‘top’ research departments A study of RAE questions its ability to identify ‘top’ research departments Including more sources (ie web) does not necessarily lead to more valid assessment of scholarly contribution Including more sources (ie web) does not necessarily lead to more valid assessment of scholarly contribution

Open access impact Strong consistent evidence that open access availability has a significant impact on citation rates (50-250%) Strong consistent evidence that open access availability has a significant impact on citation rates (50-250%) Citations themselves are not a measure of quality Citations themselves are not a measure of quality Scholars value citedness Scholars value citedness RQF could consider it more RQF could consider it more University ranking systems using journal impact factors University ranking systems using journal impact factors –Should use citation factors Tension between ‘quality’ and ‘accessibility’ and “impact” Tension between ‘quality’ and ‘accessibility’ and “impact”

Thomson Scientific databases Essential Science Indicators Essential Science Indicators Institutional Citation Reports Institutional Citation Reports University Science Indicators University Science Indicators National Science Indicators National Science Indicators High Impact Papers High Impact Papers Journal Citation Reports Journal Citation Reports Web Citation Index Web Citation Index

Thomson Scientific databases Linda Butler Linda Butler –Uses raw data –Cleans it –Maps to RFCD codes –Cautions against use by people who don’t understand the data –Is concerned that there are not enough people who can interrogate the data