1. Evaluating the Portal: The Story Behind the Numbers
Melody Burton, Queen’s University
Toni Olshen, York University
Ontario Scholars Portal
OCUL Forum
December 8, 2004

2. Why Evaluation? Feedback to OII funders
Understand who, where, and why the digital resources are used
Supplement usage numbers to answer the key question:
What is the impact of portal content on research at Ontario academic libraries?

3. Today’s Presentation Overview of MINES:
Measuring the Impact of Networked Electronic Services - Toni
Overview of Qualitative Research:
Focus groups - Melody
Questions?

4. MINES Methodology What user groups use OSP?
What specific resources are used?
From where?
How do users learn about OSP?
Are there differences in the use of digital resources based on the user's location?
Why use OSP? (sponsored research? Instruction? patient care?)
Does use differ by discipline? user group? location?

5. MINES Methodology
Web-based surveys conducted over the course of a year for each institution
Activated during randomly selected 2-hour survey periods each month as users access one of OSP’s journals
Mandatory, short, and anonymous

6. MINES Survey Form – Five Questions and a Comment Box

7. Survey Form Survey form determined : users’ status
Discipline (affiliation)
location or where accessed from
purpose of use (sponsored research, instruction, patient care, course work)
how the resource was identified (bibliography, colleague, librarian, important journal in field etc.)

8. Definition of Usage for MINES
A successful search connecting the user to an article of interest
Unique to Scholars Portal because of consortial server setup

9. Want to see your library’s MINEs data?

10. MINES Output: MAY –AUGUST 2004 – 5223 respondents

11. Very Preliminary Findings – Subject Affiliation
Applied Sciences
Business
Education
Environmental Studies
Fine Arts
Humanities
Law
Medical Health
Sciences
Social Sciences %

12. Very Preliminary Findings – User Status
Faculty
GraduateProfessional
Undergraduate
Library Staff
Staff
Other %

13. Very Preliminary Findings – Location
Library
Off-Campus
On-Campus ( but not in the library) %

14. Very Preliminary Findings- Purpose of Use
Sponsored research
Other non-sponsored research
Teaching
Course work
Patient care
Other activities %

15. Cross Tabulations
Purpose of use by affiliation, user status, location, why
Location by affiliation, user status, purpose of use, why
Why by affiliation, user status, location, purpose of use

16. Cross Tabulations

17. Location and Purpose of Use

18. Location and Why – Important Journal

19. What is the relationship of usage data to MINES findings?

20. Additional Qualitative Data
Survey respondent comments
What does the range of institutional experiences reveal?
What anecdotal data can faculty and students add to the development of OSP?