The Evaluation of the Web-based ARTIST Ann Ooms, Joan Garfield, Bob delMas – University of Minnesota Assessment Resource Tools for Improving Statistical Thinking (ARTIST) The goal of the Web-based ARTIST is to help teachers asses statistical literacy, statistical reasoning and statistical thinking. The ARTIST website provides a variety of assessment resources for teaching first courses in statistics and contains:  Item Database: a collection of over 1,200 items organized by topic and learning outcome  11 Assessment Topic Scales and the Comprehensive Assessment of Outcomes in a first Statistics Course (CAOS)  Information, guidelines, and examples of alternative assessments  Information on implementation issues related to designing, administering, and evaluating assessments  Direct links to or copies of articles on assessment in statistics  References and links to other related assessment resources  Copies of conference papers and presentations on the ARTIST project and handouts from ARTIST mini-courses  Information on upcoming ARTIST events  Information about ways to participate as a scale tester Funded by NSF CCLI-ASA The Iterative Evaluation Model for Improving Online Educational Resources ComponentsMethods and instruments Component 1 Evaluation planning Meetings to plan evaluation Creation of the pre evaluation interview protocols Pre evaluation interviews ARTIST users and non ARTIST users Reporting on pre evaluation interview results Component 2a Evaluation of web design Creation of the survey for ARTIST users Creation of the observation protocol for frequent Item Database users Creation of the observation protocol for first time Item Database users Reporting on web design Component 2b Evaluation of web content The survey for ARTIST users Reporting on web content Component 3 Evaluation of use of the educational resource The survey for ARTIST users Creation of the interview protocol for frequent ARTIST users Reporting on use of the educational resource Component 4 Evaluation of educational impact Creation of the interview protocol for frequent ARTIST users Reporting on educational impact Evaluation Instruments Pre Evaluation Interview Protocol ARTIST Users and non-ARTIST Users Survey for ARTIST UsersSurvey for Non-ARTIST UsersInterview Protocol for Frequent ARTIST Users Observation Protocol for Frequent Item Database Users Observation Protocol for First- Time Item Database Users The interviews were conducted to collect information about today’s instructors of statistics assessment practices. The information was used to tailor the other evaluation instruments. Participants were statistics instructors known by the educational resource directors. The survey was created and administered to collect information about the web design, the web content, and implementations in educational settings. All registered ARTIST users teaching introductory statistics in the US, who had accessed the website since August 2004, and who were teaching that semester were invited. The survey was created and administered to collect information about the non-users’ characteristics. Invitations for submitting the survey were sent by to the following list serves: isostat group, ASA Stat ed, and the sigmaa listserv. The interviews for frequent ARTIST users were conducted to collect information about implementations in educational settings and educational impact. Participants were selected based on the number of ARTIST topic scales and ARTIST items they had administered to their students. The observations were conducted to collect information about the navigatability of the Item Database. We wanted to learn where the Assessment Builder was difficult to use, even for experienced users. Participants were asked to conduct a think-aloud while performing certain tasks, such as create a test, add a question to that test, remove a question. The observations were conducted to collect information about the navigatability of the Item Database. We wanted to learn where the Item Database was difficult to use for first-time users. Participants were asked to conduct a think-aloud while performing certain tasks, such as create a test, add a question to that test, remove a question. Evaluation Findings 1.Web design: concrete recommendations have been made to improve the web design 2.Web content: recommendations have been made to include large data sets to the Item Database and to add additional item topics. 3.Use: Many statistics educators are actively using ARTIST and value the resource. The main reasons for statistics educators to not use ARTIST are time and misconceptions about what ARTIST has to offer. Because of the evaluation, several non-users became more aware of goal of ARTIST and started using ARTIST since. 4.Educational impact: instructors are changing assessment practices but there is no impact yet on student outcomes because, so far, newly developed ideas have not been implemented by instructors. Contact information Ann Ooms ARTIST Component 2 Evaluation of educational value Component 4 Evaluation of educational impact Component 4a Evaluation of impact on instructors’ perceptions Component 4b Evaluation of impact on educational environment Component 4c Evaluation of impact on student outcomes Component 4d Evaluation of sustainability of impact Component 1 Evaluation planning Component 2a Evaluation of web design Component 2b Evaluation of web content R e p o r t Component 3 Evaluation of use of the educational resource R e p o r t