1. BASE: Institutional Repositories Bielefeld Academic Search Engine (BASE): an End-user Oriented Institutional Repository Search Service Dirk Pieper/Friedrich Summann Bielefeld UL

2. BASE: Institutional Repositories Part 1: Institutional Repository Servers BASE: concept and content Creating a special view on institutional repository server collections Demo: BASE user-interface and further visions Part 2: OAI dataflow, BASE dataflow Repository information in registries OAI harvesting problems Further developments of BASE Overview:

3. BASE: Institutional Repositories Definition: “A digital collection capturing and preserving the intellectual output of a single or multi-university community.” (Raym Crow, http://www.arl.org.sparc/IR/ir.html) IR servers exist of course also outside the university community IR servers appear as simple web sites, database systems with OAI interface, … Institutional Repository Servers:

4. BASE: Institutional Repositories BASE uses Fast Data Search BASE contains intellectual selected resources with focus on OAI-Servers but also web crawled content BASE displays result lists as bibliographic data and full text hits BASE frontend is written in PHP using the search API from Fast Data Search BASE offers sorting, search refinement and search history BASE: concept and content

5. BASE: Institutional Repositories Search API Pipeline QUERY & RESULT PROCESSING DOCUMENT PROCESSING Pipeline FILE TRAVERSER FILTER SEARCH INDEX FILES CONNECTORS TUNING, ADMINISTRATION and DEBUGGING WEB CRAWLER BASE: concept and content

6. BASE: Institutional Repositories BASE: concept and content At present 2,7 mio documents in 189 collections, 15 of them web crawled data

7. BASE: Institutional Repositories Projekt Gutenberg-DE Internet Library of Early Journals Oxford Various Institutional Repositories Springer Link Metadata Cornell HistMath Fulltext Crawl University Michigan Historical Math CiteSeerZentralblatt Mathematik Bielefeld Univ: Math. Preprints ArXivOPAC UL Bielefeld Ifo Institute Munich Zeitschriften der Aufklärung (Bielefeld UL) BASE: concept and content

8. BASE: Institutional Repositories Special view on IR server collections Collections are listed in configuration file [ftubirmingham] url = "http://eprints.bham.ac.uk/" desc_de = "The Univ. of Birmingham: Eprints Archive" desc_en = "The Univ. of Birmingham: Eprints Archive" descdd_de = "Birmingham Univ." descdd_en = "Birmingham Univ."  Collections can be clustered for user-interface, e.g. “Institutional Repositories Europe” consists of [ftubarcelona], [ftubath], [ftubristol], [ftuhelsinki], …  Parametric search possible  Frontend is ready for multi view (independent views with own configuration and layouts on the same backend)

9. BASE: Institutional Repositories Try your search on Google Scholar... Vision: search in Google Scholar

10. BASE: Institutional Repositories Check citations (citing articles) in Google Scholar... Vision: check citations in Google Scholar

11. BASE: Institutional Repositories OAI-Data Harvesting BASE Internal Index (FAST) OPAC Article Database Dissertations, monographs (fulltext) Articles (fulltext) PubMed, Euclid, ArXiv, CiteSeer, Citebase, DOAJ articles All ressources (texts, images, video,references.... OAI dataflow at Bielefeld UL

12. BASE: Institutional Repositories OAI-Data Web Pages Database Records Harvesting Pre-Processing Processing Internal Index (FAST) User interface (PHP) BASE dataflow

13. BASE: Institutional Repositories  Eprints Registry (607)  Openarchives.org (383)  DSpace Registry (28)  Directory of Open Archive Repositories (324)  Univ. of Illinois Registry (1000) Repository information in registries

14. BASE: Institutional Repositories 2 16 12 55 14 6 33 4 2 18 1 7 3 3 USA 76 Canada 13 South America 2 Africa 2 India 3 Australia 11 New Zealand 1 3 OAI-compliant univ. repositories in BASE

15. BASE: Institutional Repositories  OAI Registry Watcher (Bielefeld UL, Perl)  Open Source Harvester (FS Consulting, Perl with modifications)  XML Validator and Repairer (Bielefeld UL, based on Perl XML modules  OAI Harvest Watcher (Bielefeld UL, Perl)  OAI Resource Updater (Bielefeld UL, Perl) Tools for the Harvesting Environment

16. BASE: Institutional Repositories  Repositories do not response or deliver Error Messages  Data contain only References without any Fulltext  Links to the Document do not work  Access to fulltext is restricted  XML file is not well-formed  Field content varies OAI harvesting challenges

17. BASE: Institutional Repositories http://xxx.xxx.uni-xxxxx.de/publications/ ELibD905_diplom_allnoch.pdf Barry Wellman,Jeffrey Boase,Kakuko Miyata Barry Wellman,Jeffrey Boase,Kakuko Miyata The Mobile-izing.... Talk P. Bruzzone Bruzzone Pierluigi Reproductive Biology and Endocrinology 2004, 2:52 doi:10.1186/1477-7827-2-52 2004-07-05 Review http://www.rbej.com/content/2/1/52 OAI Harvesting: Problems in Practice 1

18. BASE: Institutional Repositories EN: 9910 ENG: 771 En: 566 Eng: 1 English: 24084 English (United States): 63 English and Greek: 1 English and Russian: 1 English/Japanese: 1 English; Russian: 1 English=en: 1 Translation into English: 2 en: 1279115 en-CA: 865 en-US: 3 en-es: 5 en-us: 8 en;: 2 en_UK: 618 en_US: 18456 eng: 186787 eng : 92 eng + dut: 2 eng;: 17 eng; fre; ger;: 141.... OAI Harvesting: Problems in Practice 2 - Variations of

19. BASE: Institutional Repositories  Standard repository software is great - for OAI harvesting as well  Small collections – small problems  Getting the related fulltext is complicated  Libraries produce better metadata  Data aggregation may produce problems  Writing e-mails helps - sometimes Some Rules from Harvesting Practice

20. BASE: Institutional Repositories  Search form (working)  HTTP calls (working)  Web Service (in development)  Federated Search (Vascoda) (in discussion) Further Developments: BASE Interfaces

21. BASE: Institutional Repositories <form action="http://www.base-search.net/index.php" method="post" accept-charset="UTF-8"> Local Integration: Search Form

22. BASE: Institutional Repositories Thank you!