Mining for Problem- solving Styles in a Virtual World Brian M. Slator, Dept. of Computer Science Donald P. Schwert and Bernhardt Saini- Eidukat, Dept. of Geosciences; North Dakota State University, Fargo, ND

NDSU WWWIC World Wide Web Instructional Committee Jeff Clark Paul JuellDonald Schwert Philip McClean Brian Slator Bernhardt Saini-Eidukat Alan White WWWIC faculty supported by large teams of undergraduate and graduate students WWWIC’s virtual worlds research supported by NSF grants DUE and EAI

The Geology Explorer Project Educational Game designed to provide authentic learn-by-doing experience Exploration of a spatially oriented virtual world Practical, field oriented, expedition planning and decision making Scientific problem solving (i.e., a “hands on” approach to the scientific method

Balancing Pedagogy with Play Games have the capacity to engage! Powerful mechanisms for instruction Illustrate real-world content and structure Promote strategic maturity (“learning not the law, but learning to think like a lawyer”)

Advantages of Virtual Worlds Collapse virtual time and distance Allow physical or practical impossibilities Participate from anywhere Interact with other users, virtual artifacts, and software agents Multi-user collaborations and competitive play

Technical Approach Networked, internet based, client-server simulation UNIX-based MOO (Multi-User Dungeon, Object Oriented) Java-based clients (text version - telnet based; graphical version in development)

The Game Planet Oit - similar to Earth, but opposite the Sun Students “land” on Oit to undertake exploration Authentic Geoscience goals - e.g., to locate, identify, and report valuable minerals

The Simulation ~50 places: desert, cutbank, cave, etc. ~100 different rocks and minerals ~15 field instruments: rock pick, acid bottle, magnet, etc. ~Software Tutors: agents for equipment, exploration, and deduction

Real World of Planet Earth

The Geology Explorer: Planet Oit Game Scenario You are a geologist. Explore this new planet. Authentic geologic goals. - Locate and report valuable minerals. Must learn geologic content.

The Geology Explorer 50 Places 50 Places 90 Different Rocks and Minerals 90 Different Rocks and Minerals 15 Field Instruments 15 Field Instruments 25 Laboratory Instruments 25 Laboratory Instruments Software Tutors Software Tutors

Maps of Planet Oit

The Geology Explorer

Virtual Field Instruments

Rejects the notion of standardized multiple choice tests Pre-game narrative-based survey short problem-solving stories students record their impressions and questions Similar post-game survey with different but analogous scenarios Surveys analyzed for improvement in problem-solving Subjective Assessment

Assessment Not “multiple choice” recall Content specific: Problem solving, Hypothesis formation, Diagnostic reasoning

Assessment by Scenarios Assess computer literacy PreTest: Present scenario, students propose course of action or solution Engage in learning experience Control vs Virtual PostTest: Present similar scenario, student response Analysis of assessment data

The Geology Explorer: Assessment Protocol, Fall, 1998 Pre-course Assessment: 400+ students Computer Literacy Assessment: (244 volunteers) Divide by Computer Literacy and Geology Lab Experience Geomagnetic (Alternative) Group: (122 students) Geology Explorer Geology Explorer Treatment Group: (122 students) Non-Participant Control Non-Participant ControlGroup: (150 students, approx.) Completed Completed (78 students) Non- completed Non- completed (44 students) Completed Completed (95 students) Non- completed Non- completed (27 students) Post-course Assessment: 368 students

Diagnostic Tutors 1. Equipment tutor 2. Exploration tutor 3. Science tutor Detects when a student makes a wrong guess and why (i.e. what evidence they are lacking); or when a student makes a correct guess with insufficient evidence (i.e. a lucky guess) Intelligent Software Tutoring Agents

Tutors are Needed In Virtual Environments: Students can join from any remote location They can log in at any time of day or night Human tutors cannot be available at all times to help Students can foul things up and not know why

Information is readily available The simulation can track actions The simulation can generate warnings and explanations Tutor “visits” are triggered by user action Tutors are Needed In Virtual Environments:

Student interact with the intelligent tutoring agent Students can ignore advise and carry on at their own risk

Learning Style Complete history record for (#11347) as of Mon Mar 5 21:03: Central Standard Time Sep14/09:39 assigned original goal: Sphalerite Goal Sep14/09:39 connected TO MOO Sep14/09:44 entered YOUNG MOUNTAINS (#132) Sep14/09:44 Equipment Tutor says needed Streak Plate to find Sphalerite Goal [...] Sep14/09:49 purchased Streak Plate (#12191) for $0.5 [...] Sep14/09:50 entered YOUNG MOUNTAINS (#132) Sep14/09:51 entered CAVE (#341) Sep14/09:51 entered CAVE (#275) Sep14/09:51 Exploration Tutor says overlooked goal in Cave of Sphalerite Goal Sep14/09:52 entered CAVE (#341) [...] Sep14/09:53 entered ROCK MUSEUM (#594) Sep14/09:54 entered THE MINERAL COLLECTION (#1796) [...] Sep14/10:04 entered CAVE (#275) Sep14/10:04 Exploration Tutor says overlooked goal in Cave of Sphalerite Goal [...] Sep14/10:11 streak yellowish brown resinous vein (#1998) (#1998) with Streak Plate (#12191) (#12191) results: "yellow" Sep14/10:18 reported yellowish vein (#1998) as Native Gold (#657) scoring 0 points { , "Geology Tutor", #1840, #341, "Said, 'Native Gold has a yellow metallic appearance. But the yellowish brown resinous vein (#1998) does not.'"} Sep14/10:19 reported goal yellowish vein (#1998) as Sphalerite (#560) Sep14/10:19 Previously assigned goal solved -- new goal assigned Sep14/10:19 assigned Native Copper Goal scoring 100 points [...]

Learning Style Patterns we noticed: analytical approach: frequent reference to on-line help, conducting sequences of experiments, deliberative: many experiments pattern-matching approach: exploring far and wide in search of their goals: many movements “brute force” approach: simply visiting location after location and identifying everything: many reports

Learning Style ReportsMovesExperiments average st. dev min5190 max

rme10-ME5-Me4 r-e 8--E4rM-2 r-- 5R-E4r-E2 -m- 5R--3RmE2 -me 4RME e 4RM-3Rm-1 rm- 4-M-2-mE1 rmE1 R-e1 Rme1 Total 40 (49.4%) 24 (29.6%) 17 (21.0%) Note: R = many reports; r = few reports; M = many moves; m = few moves; E = many experiments; e = few experiments. Example: “-Me” means normal reporting, many moves, below normal experiments (where normal is within one-half standard deviation from the mean). Consistently normal or below normal activity Consistently normal or above normal activity Mixed problem-solving activity Learning Style

A wide range of approaches are supported Questions: Are some of the “pattern matchers” really “curious explorers? Are some of the “pattern matchers” really “curious explorers? Is there such a thing as TOO much experimentation? Is there such a thing as TOO much experimentation? Will software tutors effect what we’re seeing? Will software tutors effect what we’re seeing? How can the game encourage a more analytical approach? How can the game encourage a more analytical approach? Are students “gaming” the system? Are students “gaming” the system?

World Wide Web Instructional Committee (WWWIC) North Dakota State University Fargo ND