C R E S S T / U C L A Paper presented at AERA New Orleans—April 2000 Using Technology to Assess Students’ Web Expertise Davina C. D. Klein CRESST/UCLA Louise Yarnall Center for Technology in Learning, SRI International Christina Glaubke CRESST/UCLA C R E S S T / U C L A

2 Technology Focus: WWW  Over 1.5 billion Web pages currently available, increasing at rate of 1.9 million pages per day  At least 40% of U.S. classrooms linked to Internet  With 9 million children using Internet at school  Most common classroom use for WWW is research  3rd most common use of computers at school  Has surpassed drill-and-practice software  Nearly 90% of teachers perceive classroom WWW access as valuable or essential for their teaching

C R E S S T / U C L A 3 Research Approach  Study experienced Web users  Identify set of measures to be used to assess students’ fluency with the World Wide Web  Web Expertise Assessment (WEA)  Web-based, authentic performance assessment  Tasks require students to navigate through large information space searching for relevant information and bookmarking relevant findings  All measures logged and coded

C R E S S T / U C L A 4 WEA Search Task  Imagine you are learning about the U.S. presidents in your history class. Your teacher has asked you to write a report about what presidents said during their speeches when first elected to office. She has asked you to find out which presidents spoke of the importance of an educational system available to all without charge.  Use WEA to find this information for your report.  Find as many useful pages as you can.  Bookmark pages by clicking on the Add Bookmark button near the top of your screen.  You may bookmark as many useful pages as you think necessary.

*

C R E S S T / U C L A 11 WEA Background Questionnaire  Used to evaluate students’ WWW background knowledge  Paper-and-pencil survey  Students rated statements on scale of 1 (“I really don’t agree”) to 5 (“I really agree”)  e.g., “The information on the World Wide Web is not very useful”

C R E S S T / U C L A 12 Participants  120 middle and high school students  Students had strong technology background  Students had access to WWW in class  Students were familiar with navigating the World Wide Web

C R E S S T / U C L A 13 Methods  Students completed WWW background questionnaire  Students trained on WEA  Students given 20 minutes to complete WEA search task  Reminded to bookmark relevant pages  Data coded with high interrater reliabilities (ranging from )

C R E S S T / U C L A 14 Evaluating WEA Performance

C R E S S T / U C L A 15 Creating WEA Scales  Navigational strategies (  =.88)  Number of times back button used  Number of steps in search  Number of revisited information pages  In general, students navigated well  Students used back often (M = 17) for orientation  Students revisited over one third of info pages visited, orienting themselves in Web space (M = 4)  Students completed many steps (M = 93)

C R E S S T / U C L A 16 Creating WEA Scales (con’t.)  Finding ability (  =.86)  Average bookmark score  Quality of bookmark set  Efficiency of search  In general, students able to find information  Average bookmark peripherally relevant to task (M = 1.8 on 0-3 scale)  Quality of bookmark response set good (M = 1.8 on 0-3 scale)  About one fifth of pages bookmarked appropriately (efficiency M =.21)

C R E S S T / U C L A 17 Creating WEA Scales (con’t.)  Background Web knowledge (  =.76)  WWW helpful in finding information  Information on WWW not very useful (r)  Information on WWW is accurate/correct  Not a lot of detailed information on WWW (r)  In general, students familiar with Web  Students agreed that WWW is helpful in finding information (4.2)  Students disagreed that information on WWW is not useful (2.1)  Students were neutral/agreed that information on WWW is accurate (3.1)  Students disagreed that there is not a lot of detailed or in-depth information on WWW (2.3)

C R E S S T / U C L A 18 Creating WEA Scales (con’t.)  Searching expertise (  =.71)  Number of good searches  Quality of keyword searching set  Number of redirected searches  In general, students had difficulty searching (consistent with literature)  Number of good searches low (M = 2.0)  Quality of keyword searching set rather poor (M = 1.7 on 0-3 scale)  Students redirected searches, browsing search output before selection (M = 2.3)

C R E S S T / U C L A 19 Results  Work explored constructs underlying students’ Web fluency  Students demonstrated their abilities to search, navigate, and find information  Analyses identified important individual measures and we coded these reliably  Composite indicators from factor analysis make sense theoretically

C R E S S T / U C L A 20 Next Steps  Future research will focus on validity of assessment  Can we distinguish between expert and novice Web users with WEA?  Is WEA sensitive to instruction?  Once validity established, use WEA to examine effects of Internet usage  Link assessment and instruction  Create guidelines for teaching the Web effectively

C R E S S T / U C L A 21 For More Information  Visit our Web site at:  m  Available:  Overheads of this presentation  Full paper  And much, much more...