Evaluating the Portal: The Story Behind the Numbers Melody Burton, Queen’s University Toni Olshen, York University Ontario Scholars Portal OCUL Forum December 8, 2004

Why Evaluation? Feedback to OII funders Understand who, where, and why the digital resources are used Supplement usage numbers to answer the key question: What is the impact of portal content on research at Ontario academic libraries?

Today’s Presentation Overview of MINES: Measuring the Impact of Networked Electronic Services - Toni Overview of Qualitative Research: Focus groups - Melody Questions?

MINES Methodology What user groups use OSP? What specific resources are used? From where? How do users learn about OSP? Are there differences in the use of digital resources based on the user's location? Why use OSP? (sponsored research? Instruction? patient care?) Does use differ by discipline? user group? location?

MINES Methodology Web-based surveys conducted over the course of a year for each institution Activated during randomly selected 2-hour survey periods each month as users access one of OSP’s journals Mandatory, short, and anonymous

MINES Survey Form – Five Questions and a Comment Box

Survey Form Survey form determined : users’ status Discipline (affiliation) location or where accessed from purpose of use (sponsored research, instruction, patient care, course work) how the resource was identified (bibliography, colleague, librarian, important journal in field etc.)

Definition of Usage for MINES A successful search connecting the user to an article of interest Unique to Scholars Portal because of consortial server setup

Want to see your library’s MINEs data?

MINES Output: MAY –AUGUST 2004 – 5223 respondents

Very Preliminary Findings – Subject Affiliation Applied Sciences Business Education Environmental Studies Fine Arts Humanities Law Medical Health Sciences Social Sciences 804 17.5% 146 3.2 176 3.8 160 3.5 22 .5 93 2.0 21 .5 1341 29.2 1031 22.4 673 14.6 129 2.8

Very Preliminary Findings – User Status Faculty GraduateProfessional Undergraduate Library Staff Staff Other 764 16.6% 2068 45.0 1039 22.6 47 1.0 427 9.3 251 5.5

Very Preliminary Findings – Location Library Off-Campus On-Campus ( but not in the library) 578 12.6% 1978 43.6 2040 44.4

Very Preliminary Findings- Purpose of Use Sponsored research Other non-sponsored research Teaching Course work Patient care Other activities 2189 47.6% 919 20.0 278 6.0 686 14.9 143 3.1 381 8.3

Cross Tabulations Purpose of use by affiliation, user status, location, why Location by affiliation, user status, purpose of use, why Why by affiliation, user status, location, purpose of use

Cross Tabulations

Location and Purpose of Use

Location and Why – Important Journal

What is the relationship of usage data to MINES findings?

Additional Qualitative Data Survey respondent comments What does the range of institutional experiences reveal? What anecdotal data can faculty and students add to the development of OSP?