1. Evaluation of the NSDL and Google for Obtaining Pedagogical Resources Frank McCown, Johan Bollen, and Michael L. Nelson Old Dominion University Computer Science Department Norfolk, Virginia, USA ECDL 2005 September 21, 2005

6. Useful K-12 Educational Content The Entire Web

7. ECDL 20057 Related Work Many studies compare web search engines’ ability to find “relevant” documents Human evaluators (typically students) are used Top 10 or 20 results are evaluated None have used NSDL or evaluated educational usefulness of search results Sumner said educators expect DLs to save them time over using web search engines because DLs filter their content 1 1 T. Sumner et al.: Understanding Educator Perceptions of “Quality” in Digital Libraries, (JCDL 2003)

8. ECDL 20058 Virginia Standards of Learning BIO.3The student will investigate and understand the chemical and biochemical principles essential for life. Key concepts include a) water chemistry and its impact on life processes; b) the structure and function of macromolecules; c) the nature of enzymes; … http:// www.pen.k12.va.us/VDOE/Superintendent/Sols/home.shtml

11. ECDL 200511 Evaluators

12. ECDL 200512 Query Ratings Were the search terms well chosen? 1 = Strongly agree 2 = Agree 3 = Neutral 4 = Disagree 5 = Strongly disagree Average rating 2.08 Median rating 2

13. ECDL 200513 Link Ratings Were the search results useful for educating a student about the Learning Statement? 1 = Strongly agree 2 = Agree 3 = Neutral 4 = Disagree 5 = Strongly disagree 6 = N/A Wilcoxon signed rank test revealed a statistically significant difference (p < 0.05) between the NSDL and Google ratings.

14. ECDL 200514 Calculate Precision Google’s precision = 145/380 = 38.2% NSDL’s precision = 57/334 = 17.1%

15. ECDL 200515 Search Result Ratings All domains

16. ECDL 200516 Search Result Ratings Median ratings for each domain

17. ECDL 200517 Ranking of Search Results How well did Google and NSDL rate their search results? Performed Spearman Rank correlation between rankings and ratings Google: rho=0.125, p=0.001 ** NSDL: rho=0.057, p=0.173

18. ECDL 200518 How Did this Happen? Why did Google perform better than NSDL? Google has more stuff 8 billion pages vs. a few million? But Google’s stuff is not screened for educational quality Google and NSDL show different results 38 queries  6 duplicate results in the top 10 results 25% of NSDL results not indexed by Google

19. ECDL 200519 How Did this Happen? More NSDL results are inaccessible or login- protected (NSDL = 9.3%, Google = 5.3%) 81% of inaccessible NSDL scores were from http://nsdl.lon-capa.org/ http://nsdl.lon-capa.org/ Some NSDL resources were not for K-12 students 17% of NSDL results were from http://arXiv.org/http://arXiv.org/

20. ECDL 200520 Improvements for NSDL Provide advanced search features Provide the ability to target the grade level appropriateness of information Rank results based on relevance

21. ECDL 200521 Thank You Questions? Evaluation results, data files, slides: http://www.cs.odu.edu/~fmccown/research/nsdl_google/