Julie Gedeon, Coordinator of Assessment Barbara Schloman, Project Director Libraries and Media Services Profiling Students’ Understanding: Using TRAILS to Assess 9 th Grade Information Literacy Skills OELMA Conference 2007, Dayton, Ohio

1.TRAILS Development Overview 2.TRAILS-9 Results: School Year Your Feedback Presentation Objectives

ILILE goals: –To foster successful collaboration among K-12 teachers and library media specialists –To advance library and information literacy in the K- 12 school curriculum. Institute for Library and Information Literacy Education Federally funded through the Institute of Museum and Library Services (IMLS) and the U.S. Department of Education.

High School – College Collaborations Transitioning to College (T2C) ( TRAILS: Tool for Real-Time Assessment of Information Literacy Skills ( University Libraries’ ILILE Projects

Goal: To develop a tool to assist library media specialists in measuring the information literacy competencies of students. Specific objectives: –Standards-based –Available on the Web at no cost –Easy to administer –Insure privacy –Report outcomes by student and by class TRAILS

Coming in 2008! TRAILS-6 Recurring request from librarians. Potential to track the development of information literacy competencies in students across their K-12 school years and into college, providing the opportunity for more targeted instruction at all levels. Available funding; expandable online system.

TRAILS-6 Development Timeline 2007: June-July: item development August: item refinement; develop pilot assessments October: pilot testing items November-December: analyze data; revise items; develop final assessments January 2008: go live

TRAILS Development Process Reviewed standards for information literacy: –Ohio Academic Content Standards: Identified all Ohio standards, benchmarks, indicators that relate to information literacy at the 6 th /9 th grade level. –AASL Information Power: Reviewed standards and indicators.

Information Literacy Categories Standards were categorized into five measurable information literacy categories: Develop topic Identify potential sources Develop, use, and revise search strategies Evaluate sources and information Recognize how to use information responsibly, ethically, and legally

Developing Assessment Items Establishing Priority Competencies Within each category the information literacy skills common across the standards were identified and prioritized. Developing Objectives Student learning objectives were developed to address the priority competency areas. Writing Items Items were then written to address the objectives for each category.

Testing and Revising Items Field Testing of Items, Followed by Revisions –Are the items understandable as written? –Are they measuring what was intended? TRAILS-9: Field testing of items by volunteer library media specialists working with a small number of 9 th grade students. TRAILS-6: Sample assessments being administered to 6 th grade students in over 130 schools nationwide during October 2007.

Available Assessments TRAILS-9: Two 30-item general assessments covering all five of the information literacy categories. Two sets of 10-item assessments for each of the five categories. TRAILS-6: ?? Offer general assessments of differing lengths (20- or 25- items)??

TRAILS-9 Use TRAILS-9 went live in January 2006: 400 account holders by April; 1,000 by June. In 2006/2007 school year : Over 2,500 account holders; administered to more than 32,500 students. Currently : 3,200 + account holders Geographic distribution : All 50 states plus the District of Columbia and the Virgin Islands; more than 40 countries Feedback : Influences improvements to items and user interface.

TRAILS-9 Results All Students Data From General Assessment 1 (30 items) school year 18,323 students Findings Higher grades performed better than lower grades Overall assessment slightly easy for target audience –4% of 9 th grade top students “unmeasured” –Works great for 6 th grade

6 th Grade Person-Item Map 6 th grade student scores are within the bounds of the TRAILS-9 item measures. GA1gr6PImap.txt

9 th Grade Person-Item Map Some 9 th grade students were not measured because the range of difficulty of TRAILS-9 items was inadequate. More difficult information literacy items are needed. GA1gr9PImap.txt

TRAILS-9 Results Ohio Students General Assessment 1, Ohio 6,929 students –6 th 149 –7 th 202 –8 th 857 –9 th 4,038 –10 th 328 –11 th 866 –12 th 212 –Other 277

Performance by Category Category% correct 5. Recognize how to use information responsibly, ethically, and legally 74% 2. Identify potential sources70.4% 3. Develop, use, and revise search strategies60.7% 1. Develop topic55.6% 4. Evaluate sources and information49.5%

1. Develop topic— difficult 4. When you research a topic, it is important to know its relationship to other concepts. Which word or phrase represents the broadest (most general) subject under which all of the other topics would fit? studying (22%) note taking (19%) strategies for academic success (38%) rewriting your notes (4%) group work (6%) get started early (6%) share contact information (5%)

1. Develop topic—easy 3. Read the original topic and the revised topic. Decide if the revised topic is broader (less specific) or narrower (more specific) than the original topic. Initial Topic: Describe the elements needed in an auditorium to reduce echoes. Revised Topic: Describe the elements needed in an auditorium for good sound quality. Broader (63%) Narrower (37%)

2. Identify potential sources—difficult 6. Which one of these resources would most likely have a current article on steroid use in professional baseball? a school library catalog (9%) a science database such as Access Science (16%) a newspaper database such as Newspaper Source (56%) a biography database such as Gale Biography Resource Center (7%) a biography database such as American National Biography (12%)

2. Identify potential sources—easy 9. You just learned your favorite author has a new book out. Where could you most likely check out a copy? hospital library (1%) public library (94%) government library (4%) museum library (1%)

3. Develop, use, revise search strategies—difficult 18. If you wanted to search for a topic that has several synonyms (for example, young people, adolescents, teenagers, teens), which Boolean operator would you use? adj (20%) and (30%) near (14%) not (4%) or (32%)

3. Develop, use, revise search strategies—easy 16. If you need to know what chapters are in a book, which part of the book provides the best information? cover of the book (1%) glossary (6%) introduction (5%) table of contents (88%)

4. Evaluate sources and information—difficult 19. The following is a quote from an article you found for your research paper on stem cell research. "Stem cell research offers the possibility of understanding many different diseases. Until this research is well-funded, the number of Americans with health insurance is not expected to change." Which of the following is an accurate assessment of this information? This is credible information because it is written as a quote, indicating this is an expert opinion. (21%) This is credible information because it presents current scientific understanding. (37%) This is not credible information because the relationship between the statements is not logical. (32%) This is not credible information because the author is biased against stem cell research. (9%)

4. Evaluate sources and information—easy 22. You hear on a radio talk show that Mad Cow Disease may have been found in the United States. How might you best determine the truth of this statement? Call for a transcript of the program from the radio station. (4%) Check the fbifiles.com Web site for information the government itself might not release to the public. (8%) Discuss the news with co-workers who might have heard the program. (5%) Look up the topic at the American Council on Beef Web site for current news. (22%) Search for Mad Cow Disease on the U.S. Department of Agriculture Web site. (61%)

5. Recognize how to use information responsibly, ethically, legally--difficult 28. Which of the following concepts makes it legally wrong for government agencies to deny you access to official information under most circumstances? Fair use (8%) Freedom of information (45%) Intellectual freedom (15%) Intellectual property (7%) Right to privacy (25%)

5. Recognize how to use information responsibly, ethically, legally--easy 29. Plagiarism means: quoting someone else’s work and giving that person credit (10%) writing a short story with a group of classmates (5%) the use of another's original words or ideas as though they were your own (83%) developing a handbook on copyright laws (3%)

TRAILS-9 Results: Seeking Your Input Review handout with 3 easy and 3 difficult items. Why do you think students would find these items easy or difficult? At the end of the discussion, please leave your sheets with us!

The “Lifers” on the TRAILS Project: Barbara Schloman, TRAILS Project Director Julie Gedeon, TRAILS Assessment Coordinator Wendy Torrence, TRAILS Item Development Coordinator David Bird, Software Engineer Extraordinaire We thank you for your interest and ongoing support!

Wiki for TRAILS We welcome your use of TRAILS and your comments and suggestions.