1. Oslo University College, Norway
Central R&D assessment indicators: Scientometric and Webometric Methods
Peter Ingwersen
Royal School of LIS
Denmark –
Oslo University College, Norway

2. Agenda Scientific Communication: Scientometrics: Webometrics
Classic & present models
Scientometrics:
Publication Analyses
Publication Point evaluation (’Norwegian Model’)
Citation Analyses
Crown Indicators (research profile weighting)
Hirsch Index (h-Index)
Webometrics
Web Impact Factors; Issue tracking - mining
Concluding remarks
Ingwersen
2011

3. Scientific communication 1 – Classic Model (prior to Web / open access)
Research idea & activities
TechnicalResearch report
Peers
Archive
Conf.
Papers
(Peer reviewed)
Journal
Articles
Library
index
Domain databases
Citation
database
Time
Un-published
non-peer review
informal com.
Ingwersen
2011

4. Scientific communication 1 – Present Model (incl. Web / open access)
Research idea & activities
TechnicalResearch reports
Working
papers
Peers
Inst. Repositories
Open access journals
Conf.
Papers
(Peer reviewed)
Journal
Articles
Time
Un-published
public
Non Peer review
Full text
Domain database
- Web of
Science
- Scopus
Google (Scholar)
Academic Web Search Engines
Ingwersen
2011

5. Scientific communication 2 – What ’is’ scientific information?
Blogs …
Teaching material
Working papers
Research reports
Student output
Searchable on Open Web
Conference Papers
Posters, Abstracts (peer reviewed)
Collaboratory round tables
Open Access
- Journals (peer reviewed)
- Inst. Repositories
(Duplicates/versions)
Partly searchable on Open Web
Confidence in information source?
Qualified knowledge source
(Domain dependent )
Restricted Access
- Journal articles
(peer reviewed)
Authoritative source
- Research Monographs
Ingwersen
2011

6. Examples of Publication analysis
Ranking most productive
Countries in a field
Journals in a field
Institutions or universities; departments or goups
(Exponential Bradford-like distributions)
Counting scientific publications in
Academic fields / disciplines
Countries, regions, universities, departments
Counting number of articles OVER TIME
Time series
Ingwersen
2011

7. Typical time series
Ingwersen
2011

8. Productivity Growth
Ingwersen
2011

9. Publication Growth – all fields = index 1: China=14,114 p.; EU=736,616 p; USA=887,039 p. India = 65,250 (98,598) publ.
Ingwersen
2011

10. Publication success ‘points’
As done in Norway:
Articles from the journals considered among the 20 % best journals in a field: 3 points
Articles from other (peer reviewed) journals: 1 point
Conference papers (peer reviewed): .7 points
Monographs (int.publisher): 8 points
Monographs (other publishers): 5 points
Fractional counts; points used for funding distribution
Covers all research areas, incl. humanities, for all document types
Ingwersen
2011

11. One cannot use the publication points for DIRECT comparison
Between universities or countries
Or applied to individual researchers
Recent detailed article on the system:
Schneider, J.W. (2009) An outline of the bibliometric indicator used for performance-based funding of research institutions in Norway. European Political Science, 8(3), p
Ingwersen
India 2010

12. However: Publication Point Indicators established!
Elleby, A., & Ingwersen, P. Publication point indicators: A comparative case study of two publication point systems and citation impact in an interdisciplinary context. Journal of Informetrics, 4 (2010):
doi: /j.joi
Ingwersen
India 2010

13. Publication Point Indicators 2
Comparing the vectors of ideal cumulated PP (= expected success gain) with the actually obtained PP, for the same publication (types), providing a ratio that can be normalized: nPPI: The normalized Publication Point Index
Comparisons between institutionas can be done at specific ranges of publication vector values through their nPPI.
Ingwersen
India 2010

14. Cumulated Publ. Point Indicator the DIIS example (n=70)
Ingwersen
Tutorial 2011

15. Citation Analyses
Diachronic (forward in time) … or Synchronous (back in time – like ISI-JIF)
Observing: how older research is received by current research (ISI+Scopus: always peer reviewed sources)
Citation indicators:
Time series (like for publications)
Citation Impact (Crown Indicators)
Citedness
Ingwersen
2011

16. Absolute Citation Impact
Ingwersen
2011

17. ‘Crown indicators’
Normalized impact-indicators for one unit (center/university/country) in relation to research field globally:
JCI : Journal Crown Indicator
FCI : Field Crown Indicator – both provide an index number
Ingwersen
2011

18. Journal Crown Indicator
The ratio between:
- the real number of citations received for all journal articles in a unit from a year, and
- the diachronic citation impact of the same journals used by the unit, covering the same period (= the expected impact).
ONE WOULD NEVER APPLY THE ISI-JIF!!
Since it only signifies the AVERAGE (international) impact of an article made in a synchronous way
2011
2010
2009
2008
2007
Ingwersen
2011

19. Journal Impact Factor - ISI
Synchroneous method:
For 2010: Analysis done in Febr-April, 2011 for …
1) all citations given in 2010 to journal X
for articles+notes+letters in journal X,
2) Published in previous two years:
2011
2010
2009
2008
2007
Ingwersen
2011

20. Field Crown Indicator - FCI
Normalisation must be weighted in relation to the observed unit’s publication profile:
Like a ’shadow’ unit (country)
An example of this weighting for India:
Ingwersen
2011

21. Research profile as weight for impact calculation (as ‘shadow country’)
Ingwersen
2011

22. Research profile (China) as weight for impact calculation (as ‘shadow country’)
Ingwersen
2011

23. A small European country with very different profile
Ingwersen
2011

24. Example of
research
profile with
FCI-index score
Ingwersen
2011

25. Summary: Different indicators – one given period
Σc/Σp / ΣC/ΣP – Globally normalized impact:
For single fields it is OK to use!
If averaged over all subject areas: quick’n dirty!: all areas have the same weight! – thus:
Σc / Σ(C/Parea x parea ) = FCI:
Standard Field Crown Indicator (FCI) for ’profile’ of subject areas for a local unit (country/university) – via applying it as global profile, like a kind of ’shadow unit’.
Made as ratio of sums of citations over publications (weights)
(If done as sum of rations divided by fields: all fields equal)
Ingwersen
2011

26. Ageing of journals or articles
Cited half-life - diachronic:
Acumulate citations forward in time by year:
yrs
Citations
Acum: †
1/2 life= 132/2 = 66 = ca. 4,2 years
Ingwersen
Tutorial 2011

27. Ageing of journals or articles – 2
Ingwersen
Tutorial 2011

28. Hirsch Index (2005)
A composite index of publications and citations for a unit (person, group, dept. …) in a novel way:
H is the number of articles given a number of citations larger or equal to h.
A person’s h-index of 13 implies that he/she among all his/her publications has 13, that at least each has obtained 13 citations.
The index is dependent on research field and age of researcher. Can be normalized in many ways.
Ingwersen
Tutorial 2011

29. Criticism of Citation Analyses
Formal influences not cited
Biased citing
Informal influences not cited
Self-citing – may indeed improve external cits.! Different types of citations –
Variations in citation rate related to type of publication, nationality, time period, and size and type of speciality – normalization?
Technical limitations of citation indexes and domain databases
Multiple authorship – fractional counting/article level
Ingwersen
Tutorial 2011

30. Reply: van Raan (1998) Different biases equalizes each other
If researchers simply do most of their referencing ”in a reasonable way” and if a sufficient number of citations are counted, realiable patterns can be observed.
It is very unlikely that all researchers demonstrate the same biases (e.g. all researchers consciously cite research, which does not pertain to their field)
Ingwersen
Tutorial 2011

31. Google Scholar Does not apply PageRank for ranking but citations
Contains conference papers and journal articles (??)
Workable for Computer Science and Engineering (and Inf. Sc.)
Requires a lot of clean-up!
Apply (Publish or Perish) for better analysis on top of GS
Google Scholar may provide the h-index for persons
Ingwersen
2011

32. infor-/biblio-/sciento-/cyber-/webo-/metrics
L. Björneborn & P. Ingwersen 2003
infor-/biblio-/sciento-/cyber-/webo-/metrics
informetrics
bibliometrics
scientometrics
cybermetrics
webometrics
Ingwersen
2011

33. Link terminology basic concepts
L. Björneborn & P. Ingwersen 2003
B has an outlink to C; outlinking : ~ reference
B has an inlink from A; inlinked : ~ citation
B has a selflink; selflinking : ~ self-citation
A has no inlinks; non-linked: ~ non-cited
E and F are reciprocally linked
A is transitively linked with H via B – D H is reachable from A by a directed link path
A has a transversal link to G : short cut
C and D are co-linked from B, i.e. have co-inlinks or shared inlinks: co-citation
B and E are co-linking to D, i.e. have co-out-links or shared outlinks: bibliog.coupling A B E G C D F H
co-links
Ingwersen
2011

34. Ingwersen
2011

35.
Ingwersen
2011

36. Search engine analyses
See e.g. Judith Bar-Ilan’s excellent longitudinal analyses
Mike Thelwall et al. in several case studies
Scientific material on the Web:
Lawrence & Giles (1999): approx. 6 % of Web sites contains scientific or educational contents
Increasingly: the Web is a web of uncertainty
Allen et al. (1999) – biology topics from 500 Web sites assessed for quality:
46 % of sites were ”informative” – but:
10-35 % inaccurate; % misleading
48 % unreferenced
Ingwersen
2011

37. The Web-Impact Factor Ingwersen, 1998
Intuitively (naively?) believed as similar to the Journal Impact Factor
Demonstrate recognition by other web sites - or simply impact – not necessarily quality
Central issue: are web sites similar to journals and web pages similar to articles?
Are in-links similar to citations – or simply road signs?
What is really calculated?
DEFINE WHAT YOU ARE CALCULATING: site or page IF
Ingwersen
2011

38. The only valid webometric tool: Site Explorer Yahoo Search …
If one enters (old valid) commands like:
Link:URL or Domain: topdomain (edu, dk) or Site:URL you are transferred to:
Or find it via this URL
The same facilities are available in click-mode, as one starts with a given URL:
Finding ‘all’ web pages in a site
Finding ‘all’ inlinks to that site/those pages
Also without selflinks! – this implies …
Ingwersen
2010 Åbo

39. … to calculate Web Impact Factors
But one should be prudent in interpretations.
Note that external inlinks is the best indicator of recognition
Take care of how many sub-domains (and pages) that are included in the click analysis.
Results can be downloaded
Ingwersen
2010 Åbo

40. Possible types of Web-IF:
E-journal Web-IF
Calculated by in-links
Calculated as traditional JIF (citations)
Scientific web site – IF (by link analyses)
National – regional (some URL-problems)
Institutions – single sites
Other entities, e.g. domains
Best nominator: no. of staff – or simply use external inlinks
Ingwersen
2011

41. Web-links like citations?
Kleinberg (1998) between citation weights and Google’s PageRank:
Hubs ~ review article: have many outlinks (refs) to:
Authority pages ~ influential (highly cited) documents: have many inlinks from Hubs!
Typical: Web index pages = homepage
with self-inlinks = Table of contents
Ingwersen
2011

42. Reasons for outlinking …
Out-links mainly for functional purposes
Navigation – interest spaces…
Pointing to authority in certain domains? (Latour: rhetoric reasons for references-links)
Normative reasons for linking? (Merton)
Do we have negative links?
We do have non-linking (commercial sites)
Ingwersen
2011

43. Some additional reasons for providing links
In part analogous to providing references (recognition)
And, among others,
emphasising the own position and relationship (professional, collaboration, self-presentation etc.)
sharing knowledge, experience, associations …
acknowledging support, sponsorship, assistance
providing information for various purposes (commercial, scientific, education, entertainment)
drawing attention to questions of individual or common interest and to information provided by others (the navigational purpose)
Ingwersen
2011

44. Other differences between references, citations & links
The time issue:
Aging of sources are different on the Web:
Birth, Maturity & Obsolescence happens faster
Decline & Death of sources occur too– but
Mariages – Divorse – Re-mariage – Death & Resurrection … & alike liberal phenomena are found on the Web! (Wolfgang Glänzel)
Ingwersen
2011

45. Issue tracking – Web mining
Adequate sampling requires knowledge of the structure and properties of the population - the Web space to be sampled
Issue tracking of known properties / issues may help
Web mining the unknown is more difficult, due to
the dynamic, distributed & diverse nature
the variety of actors and minimum of standards
the lack of quality control of contents
Web archeology – study of the past Web
Ingwersen
2011

46. Nielsen Blog Pulse Observes blogs worldwide by providing:
Trend search – development over time of terms/concepts – user selection!
Featured trends – predefined categories
Coversation tracker – blog conversations
BlogPulse profiles – blog profiles
Look into:
Ingwersen
2011

47. Home > Tools Trend Search
Ingwersen
2011

48. Concluding remarks: Future
With open access we can foresee a nightmare as concerns tracking qualified and authoritative scientific publications, aside from the citation indexes because of
Lack of Bibliographic control (what is original – vs. parallel and spin-off versions & crab?) over many institutional repositories – and mixed on the web with all other document types incl. Blogs (web 2.0) – Google Scholar(?) … Google Books (?)
Ingwersen
2011

49. Concluding remarks
One may be somewhat cautious on Web-IF applications without careful sampling via robots due to its incomprehensiveness and what it actually signifies
One might also try to investigate more the behavioural aspects of providing and receiving links to understand what the impact might mean
and how/why links are made
Understand the Web space structure better
Design workable robots, downloading & local analyses
Ingwersen
2011

50. References
Allen, E.S., Burke, J.M., Welch, M.E., Rieseberg, L.H. (1999). How reliable is science information on the Web? Science, 402, 722.
Björneborn, L., Ingwersen, P. (2004). Towards a basic framework for webometrics. Journal of American Society for Information Science and Technology, 55(14):
Brin, S., Page, L. (1998). The anatomy of a large scale hypertextual web search engine. Computer Networks and ISDN Systems, 30(1-7),
Elleby, A., Ingwersen, P. Publication Point Indicators: A Comparative Case Study of two Publication Point Systems and Citation Impact in an Interdisciplinary Context. Journal of Informetrics, 2010, 4, p
Hirsch, J.E. (2005): An index to quantify an individual’s scientific research output. PNAS, 102:
Ingwersen
2011

51. References 2
Jepsen, E.T., Seiden, P., Ingwersen, P., Björneborn, L., Borlund, P. (2004). Characteristics of scientific Web publications: Preliminary data gathering and analysis. Journal of American Society for Information Science and Technology, 55(14):
Lawrence, S., Giles, C. L. (1999). Accessibility and distribution of information on the Web. Nature, 400,
Li, X.M., Thelwall, M., Musgrove, P.,, Wilkinson, D. (2003). The relationship between the WIFs or inlinks of Computer Science Departments in UK and their RAE ratings or research productivities in Scientometrics, 57(2),
Moed, H. (2005). Citation Analysis in Research Evaluation. Springer.
Thelwall, M. & Harrier, G. Do the Web sites of higher rated scholars have significantly more online impact? JASIST, 55(2),
SearchEngineShowdown.com
Van Raan, A. se: - for publications
Ingwersen
2011