1. Digital Library Evaluation Flora McMartin Broad Based Knowledge flora.mcmartin@gmail.com

2. e-VALU-ation What do you, as an organization VALUE? What do you, as an organization VALUE? Service to your community? Service to your community? Is access enough? Is access enough? Education - Learning - Teaching Education - Learning - Teaching Does it matter what you call it? Importance of language Does it matter what you call it? Importance of language Where you start? Where you start? Organizational Metaphor Organizational Metaphor Walter M. Miller’s A Canticle for Leibowitz Walter M. Miller’s A Canticle for Leibowitz Neal Stephenson’s The Diamond Age Or, a Young Lady's Illustrated Primer Neal Stephenson’s The Diamond Age Or, a Young Lady's Illustrated Primer Open system vs. Closed system Open system vs. Closed system In a collaboration: What if your values differ? In a collaboration: What if your values differ?

3. A Simple Evaluation Model Describe program (Mission, goals, objectives) Design Evaluation Conduct EvaluationInterpret Results Revise program’s activities, goals, etc. (Formative Evaluation) Measure Success of Impact of Program (Summative Evaluation)

4. Evaluation Building Blocks Types of Evaluation PlanningProcess Improvement or Impact IntentPracticeResults Mission Mission Goals Goals Objectives Objectives Activities Activities Programs Programs Outcomes Outcomes Products Products Systems approach: evaluation deals with aspects of performance related to goals - effectiveness, efficiency or both

5. Logic is behind it all Evaluating: transforming teaching and learning in undergraduate and graduate institutions, is like evaluating…. Evaluating: transforming teaching and learning in undergraduate and graduate institutions, is like evaluating…. World Peace You can evaluate the programs, activities that have been undertaken to accomplish goals, mission and vision You can evaluate the programs, activities that have been undertaken to accomplish goals, mission and vision GGG AAAAAAA V M

6. Designing an Evaluation Program goals - what you want to accomplish Program goals - what you want to accomplish Identify Stakeholders Identify Stakeholders Develop GOOD evaluation questions Develop GOOD evaluation questions Match method to question Match method to question Ethical considerations Ethical considerations Collect data Collect data Analyze data Analyze data Report results - use results Report results - use results

7. GOOD Evaluation Questions Feasible (resources, access?) Feasible (resources, access?) Clear (what behavior/activity/product?) Clear (what behavior/activity/product?) Significant (how will it help your DL to know this?) Significant (how will it help your DL to know this?) Ethical (people as subjects vs. participants) Ethical (people as subjects vs. participants) 3 dangerous ideas/words/concepts you  oughta  wanta

8. Evaluation Elements of a DL ResourcesHoldingsSelectionOrganizationStructure MetadataAccessPersistenceManagementInterfaces SearchServicesAvailabilityDisseminationDelivery ReferralUseUsersCommunitySecurity PrivacyCostsOperationsCollaboration?????

9. Levels of Evaluation, I User Centered User Centered Social - how well does a DL support the needs, demands, practices of a community? Social - how well does a DL support the needs, demands, practices of a community? Institutional - how well does a DL support organizational purpose Institutional - how well does a DL support organizational purpose Individual - how well does a DL support information needs, tasks, activities of individuals? Individual - how well does a DL support information needs, tasks, activities of individuals?

10. Levels of Evaluation, II System System Engineering - how well do hardware, networks, etc., perform? Engineering - how well do hardware, networks, etc., perform? Processing - how well do procedures, techniques, algorithms, operations perform? Processing - how well do procedures, techniques, algorithms, operations perform? Content - how well is the collection represented, organized, structured and managed? Content - how well is the collection represented, organized, structured and managed?

11. Levels of Evaluation, III Interface - how well does the DL’s interface support access, searching, navigation, browsing? Interface - how well does the DL’s interface support access, searching, navigation, browsing? Can be asked of both users and systems Can be asked of both users and systems Organizational - how well is the consortium, set of partners working Organizational - how well is the consortium, set of partners working

12. What we know about evaluating DLs DL evaluation is complex, undefined and difficult DL evaluation is complex, undefined and difficult There is no one framework or model to ‘fit all’ There is no one framework or model to ‘fit all’ Much work has been done Much work has been done Usability, User Needs, Workflow Usability, User Needs, Workflow More needs to be done More needs to be done Sustainability Sustainability Impact Impact Multiple methods is a must -triangulation Multiple methods is a must -triangulation Collaboration - how to do it? When? Around what? Talk about complexity…… Collaboration - how to do it? When? Around what? Talk about complexity……

13. Triangulation Value of peer review User Survey Users SAY they highly value peer reviews Webmetrics # hits on peer reviews Observation How users use site, do they SEE peer reviews?

14. Consortiums: Complicating Factors Diverse goals; diverse priorities Diverse goals; diverse priorities Different technologies; different infrastructure; different definitions Different technologies; different infrastructure; different definitions Different activities; different services; different features Different activities; different services; different features

15. Resources Saragevic, T. (2000).. Library Trends. Saragevic, T. (2000).. Library Trends. Digital Library Federation. (1998). A working definition of digital library. Found at: http://www.diglib.org/about/dldefinition.htm Digital Library Federation. (1998). A working definition of digital library. Found at: http://www.diglib.org/about/dldefinition.htm http://www.diglib.org/about/dldefinition.htm McMartin, F. & Morgan, G. (2005). Research and evaluation methods for information technology professionals. New Media Consortium Annual Conference Preoceedings, Honolulu, Hawaii. McMartin, F. & Morgan, G. (2005). Research and evaluation methods for information technology professionals. New Media Consortium Annual Conference Preoceedings, Honolulu, Hawaii.