1. Assessing Tools that Measure Information Literacy (IL) Skills: Could Project SAILS Prove Useful for Law Libraries? AALL/Wolters Kluwer (Aspen) Grant –Funded Research Co-P.I.s: Molly Brownfield & Dennis Kim-Prieto Rutgers Law Library – Newark

2. Could SAILS Prove Useful? Brownfield & Kim-Prieto Rutgers – Newark Assessing Assessment Tools? Assumption: Information Literacy (IL) is measurable. Therefore: 1.Instructional Law Librarians will want to measure IL. Project SAILS 2.Current state-of-the-art assessment is Project SAILS instrument: can it work for law libraries? 3.We think not: here’s why.

3. Could SAILS Prove Useful? Brownfield & Kim-Prieto Rutgers – Newark What is Project SAILS? It is an: Item Response Theory (IRT)-based assessment tool, targeting information literacy skills which are based on ACRL Information Literacy Competency Standards for Higher Education. It has been funded and supported by: Institute of Museum and Library Services, Association of Research Libraries, Ohio Board of Regents, Academic Library Association of Ohio, and Kent State University.

4. Could SAILS Prove Useful? Brownfield & Kim-Prieto Rutgers – Newark Why did we choose Project SAILS? Ease of use: –SAILS –SAILS uses graphical user interface (GUI) –accessible through standard internet browsers –SAILS –SAILS performs all analyses SAILSExternal validity: SAILS results positively correlate with subject SAT/ACT scores. Price: $3.00 for each subject with a cap of $2,000.00 per total administration. SAILSSAILS is explicitly keyed to measure ACRL’s IL standards.

5. Could SAILS Prove Useful? Brownfield & Kim-Prieto Rutgers – Newark What did we do with Project SAILS? SAILSWe administered the SAILS instrument to 28 1Ls and 29 3Ls (n=57). Using law students in Human Subjects Research requires two things: –Institutional Review Board (IRB) approval & –CA$H!!! IRB approval generally requires certification. Your institution will have its own IRB, but rules tend to be standardized per federal research funding regulations.

6. Could SAILS Prove Useful? Brownfield & Kim-Prieto Rutgers – Newark What is IRT, and how does it work? Sample Question One You need to get information on an event that took place two days ago. Where are you most likely to find information about the event? CHOOSE ONE ANSWER Book Dissertation Journal article Magazine Newspaper

7. Could SAILS Prove Useful? Brownfield & Kim-Prieto Rutgers – Newark What is IRT, and how does it work? Sample Question Two You need a case that addresses the following issues: employment discrimination and intentional infliction of emotional distress. Which boolean operator should you use? CHOOSE ONE ANSWER And (&) Near (within /n number of words) Same Sentence ( /s) Not (NOT) Or (OR)

8. Could SAILS Prove Useful? Brownfield & Kim-Prieto Rutgers – Newark IRT against Classical Test Theory (CTT)? CTT weighs both items the same, despite item differences in their level of difficulty. Correct answers on Question 1 do not reveal much about subject IL, since most should get it right. Correct answers on Question 2 are likely to reveal more, since not as many subjects will get it right. CTT does not distinguish the value of each item, and weighs all items on a test the same. IRT distinguishes item weights in analyzing responses, which reveals more about subjects’ IL.

9. Could SAILS Prove Useful? Brownfield & Kim-Prieto Rutgers – Newark Sample Questions from Project SAILS

10. Could SAILS Prove Useful? Brownfield & Kim-Prieto Rutgers – Newark Radcliff, Carolyn J., Joseph A Salem, Jr., Julie A. Gedeon, and Lisa G. O'Connor. 2007. “Project SAILS Sample Test Questions.”. Last Accessed 3 May 2008

11. Could SAILS Prove Useful? Brownfield & Kim-Prieto Rutgers – Newark What did we find with Project SAILS? R-N Law Students outperformed institution-type benchmark on the following skill sets: Developing a Research Strategy Selecting Finding Tools Searching Using Finding Tool Features Evaluating Sources Documenting Sources Understanding Economic, Legal, and Social Issues

12. Could SAILS Prove Useful? Brownfield & Kim-Prieto Rutgers – Newark What else did we find? Skill results for R-N Law Students in performance order: Using Finding Tool Features (Best Performance!) Retrieving Sources Documenting Sources Evaluating Sources Developing a Research Strategy Selecting Finding Tools Searching Understanding Economic, Legal, and Social Issues (Worst Performance?!)

13. Could SAILS Prove Useful? Brownfield & Kim-Prieto Rutgers – Newark Data from SAILS administration found … RU Law Students Students from Doctoral-granting Institutions Students from All Institutions Standard 1: Nature & Extent 655 ±26584 ±1576 ±1 Standard 2: Efficient Access 640 ±18577 ±1571 ±1 Standard 3: Evaluation & Incorporation 639 ±21576 ±1570 ±1 Standard 5: Socio-legal Issues 623 ±18566 ±1558 ±1

14. Could SAILS Prove Useful? Brownfield & Kim-Prieto Rutgers – Newark No significant difference btw. 1L & 3L IL! SAILS SAILS Skill SetsRU 1L ScoresRU 3L Scores Research Strategy643 ±31643 ±35 Finding Tools629 ±56643 ±45 Searching628 ±36631 ±35 Finding Tool Feat.683 ±57715 ±63 Retrieving Sources667 ±67674 ±61 Eval. Sources641 ±29667 ±35 Doc’ting. Sources662 ±37674 ±39 Econ., Leg., Soc. Is.625 ±32624 ±29

15. Could SAILS Prove Useful? Brownfield & Kim-Prieto Rutgers – Newark Our Observations SAILSOur administration did not show any significant distinction between 1L and 3L SAILS performance; yet, We know that 3Ls have had more training or exposure to legal research practices than 1Ls have. SAILSAs the current standard (SAILS) does not adequately reflect law student performance, Accurate Law Student Information Literacy (LSIL) assessment requires a subject-specific tool.

16. Could SAILS Prove Useful? Brownfield & Kim-Prieto Rutgers – Newark Our Recommendations SAILSAs a cohort measure, and not an individual measure, SAILS data only assesses skill sets between and within institutions, and does not assess skill sets of particular students. SAILSSAILS provides a useful model and valuable experience in assessing law students, but does not test the subject-specific research abilities that instructional law librarians teach. We need a subject-specific IL assessment instrument if we want to measure LSIL.

17. Could SAILS Prove Useful? Brownfield & Kim-Prieto Rutgers – Newark What can academic law libraries do next? Law libraries have a vested interest in measuring Law Student IL (LSIL). Law schools and their libraries can contribute the following toward creating an adequate measure: –Subject pools, –Research support, and –A consensus on IL: what subject-specific skills make a law student more or less literate when evaluating information and performing research?

18. Could SAILS Prove Useful? Brownfield & Kim-Prieto Rutgers – Newark Thank you for your interest! Questions? Principal Investigators: Molly Brownfield, Duke Law Library brownfield@law.duke.edu Dennis Kim-Prieto, Rutgers-Newark Law Library dprieto@kinoy.rutgers.edu