Research on Authentic Assessment Using a Virtual World for Learning Geology Bernhardt Saini-Eidukat 1, Donald Schwert 1, Brian Slator 2, Lisa Daniels 3, and Jeffrey Terpstra 4 1 Dept. of Geosciences; 2 Dept. of Computer Science; 3 School of Education; 4 Department of Statistics North Dakota State University, Fargo, ND

NDSU WWWIC World Wide Web Instructional Committee Paul JuellDonald Schwert Philip McCleanBrian Slator Bernhardt Saini-EidukatAlan White Jeff ClarkLisa Daniels Jeff Terpstra WWWIC faculty supported by large teams of undergraduate and graduate students WWWIC’s virtual worlds research supported by NSF grants DUE , EAR , DUE , ITR and EPSCoR , and Department of Education FIPSE P116B011528

The Geology Explorer Project Educational Game designed to provide authentic learn-by-doing experience Exploration of a spatially oriented virtual world Practical, field oriented, expedition planning and decision making Scientific problem solving (i.e., a “hands on” approach to the scientific method)

Balancing Pedagogy with Play Games have the capacity to engage! Powerful mechanisms for instruction Illustrate real-world content and structure Promote strategic maturity (“learning not the law, but learning to think like a lawyer”)

Technical Approach Networked, internet based, client-server simulation UNIX-based MOO (Multi-User Dungeon, Object Oriented) Java-based clients (text and graphical versions)

The Setting Planet Oit - similar to Earth, but opposite the Sun Students “land” on Oit to undertake exploration Authentic Geoscience goals - e.g., to locate, identify, and report valuable minerals

The Simulation ~50 places: desert, cutbank, cave, etc. ~100 different rocks and minerals ~15 field instruments: rock pick, acid bottle, magnet, etc. ~Software Tutors: agents for equipment, exploration, and deduction

Graphical Navigation Database Map of Planet Oit

Carrying out Investigations: Players Identify Outcrops Using Tests Results of Tests (in this case, acid reactivity) Detail Images

Carrying out Investigations: Players Identify Outcrops Using Tests

Because each mineral in the rock can be distinguished, the test can be performed on each mineral. Each returns whether it is acid reactive or not. Coarse- grained rocks If the rock contains more than 10% calcite, it fizzes actively when acid is poured on it. If the rock contains between 0 and 10 percent calcite, its powderized form will fizz. Also, if the rock contains more than 25% dolomite, its powder will fizz actively. Finally, if the rock contains no calcite or dolomite, it will not fizz. Fine-grained rocks If the property acid_reactive is 2, the mineral fizzes actively when acid is poured on it. If the property acid_reactive is 1, and a powder is made of the mineral, the powder will fizz actively. If the acid_reactive property is 0, the mineral does not fizz. Minerals Most other objects in the game have a acid_reactive property of 0. Pouring acid on them will cause them not to fizz. Players are an exception to this rule. You are unable to pour acid on other players. Other objects Example of Programming Actions: Acid Bottle: "pour acid bottle on substance"

Deductive Tutors Equipment tutor Exploration tutor Science tutor Detects when a student makes a wrong guess and why (i.e. what evidence they are lacking); or when a student makes a correct guess with insufficient evidence (i.e. a lucky guess) Intelligent Software Tutoring Agents Students/players interact with the intelligent tutoring agents Players can ignore advice and carry on at their own risk

Virtual Mapping SS LS SS Schist Marble Qzite Schist Qtzite Dike SchistGneiss LS Marble Qzite SS Schematic map - not to scale Locate yourself and outcrops using topo maps and aerial photos General rock identification Create a Geologic Map

Virtual Mapping Tutorials on Concepts of Geologic Mapping are Available in the Environment A Player (appearance can be changed)

Virtual Mapping Base Maps for Adding Data and Creating a Geologic Map “aerial photo”

Virtual Mapping Base Maps for Adding Data and Creating a Geologic Map Topographic map

Virtual Mapping Outcrop Locations are Shown with Markers Rock Type Markers Show Locations of Identified Outcrops You Are Here

Player Creates a Geologic Map Based on Outcrop Locations, and Can Get Immediate Feedback Virtual Mapping

Virtual Mapping: Formative Assessment

Virtual Mapping: Assessment Computer Scoring of Your Map

Rejects the notion of standardized multiple choice tests Pre-game narrative-based survey short problem-solving stories students record their impressions and questions Similar post-game survey with different but analogous scenarios Surveys analyzed for improvement in problem-solving Assessment Using Scenarios: Summative

You are in northern Manitoba on a fishing trip. Fishing has been good. At dawn on the day of your departure you cut across country to a remote bay and have more good luck: you catch your limit of walleye! Assessment Using Scenarios

On the way back to the lodge, you stumble across a heavy, moss-covered rock on the shoreline, flipping it over. Looking down, you see the underside shines with a brilliant metallic yellow. You pick up the rock and lug it back to the lodge with you. Assessment Using Scenarios

When it’s time to leave, the pilot notes that you are 45 pounds overweight...exactly the weight of the rock you found. She says, "At $8 a pound, eh, you owe $360." As a geologist, what do you do? List the things you would consider in your decision. List the questions you would ask yourself, and give the reasons behind those questions. Assessment Using Scenarios

Assessment Method General Problem Solving (60 pts) 1.1 Form a Hypothesis: 20 pts 1.2 Suitable Test: 20 pts Decide to Test Experimental Design, Comparison, Etc. First Specific Test Second Specific Test 1.3 Conclusions based on evidence: 20 pts Stated and correct with correct explanation Stated and correct Stated and correct with incorrect explanation Implied and correct Stated and incorrect Just implied Implied and incorrect ?

Specific Information (24 pts) 2.1 Color/Luster: 6 pts Mention testing for Describe how Specific properties 2.2 Malleability/Hardness: 6 pts (ditto) 2.3 Specific Gravity: 6 pts (ditto) 2.4 Other Test : 6 pts (ditto) Assessment Method

Expert Knowledge (16) 3.1 Current price of gold: 5 pts High knowledge Some knowledge No knowledge 3.2 Specific Geologic Conditions : 5 pts (ditto) 3.3 Judges prerogative: 6 pts ??

Validation: Multiple Graders Assessment Protocol Pearson Product Moment Correlation Coefficent between graders Grader 1Grader 2Sample SizeCoefficient jennifertarico kimaarico kimajennifert

Current Assessment Research: Scenario Order; Team vs. Solo play StudentsScen 9/10Scen 10/9Total Teams of Solo Total Preliminary Data from 2004 – scoring is ongoing! Number of Observations Used Here: 190

Conclusions: Scenario Order; Team vs. Solo play Scenario Combination SoloTeamP-valSignif Scen 9/ NS Scen 10/ S Net Improvement for 2004 The response variable is Net Improvement based on one grader, “Kelly”. The data were analyzed as a two-factor factorial design with interaction, based on ordinary least squares methodology.

Geology Explorer Development Team Graphics Acey Olson (Rendered Landscapes) Rebecca Potter (Rendered Images, Icons, and Landscapes) Dan Reetz (Rendered Images, 3D) Shannon Tomac (Rendered Landscapes, Icons, & Images, Comic Book) Daniel Turany (Rock/Mineral Photographs) Staff John Bauer; Java Client (GUMI-bare and GUMI-game) Design and Implementation Nem Schlecht Mark Tinguely Brad Vender; MOO-side HTML server Students Graduate Research Assistants: Kim Addicott; Assessment Otto Borchert; Software tutors, Java client development Bob Cosmano; Makes Java Go; Client Implementation Krista Dauner; Tutoring Scripts Rahul Devabhaktuni; Hydrology (Water) on Oit Kuo-Di Jian; Java Tool Integration Yongxin "George" Jia; Java/LambdaMOO Hierarchy Tool Guy Hokanson; Software tutors, dynamic HTML Bhaskar Marthi; Instrument verbs Vidyalatha Nagareddy; Java/LambdaMOO Entity Tool Shanhong Wu; Java3D Undergraduate Research and Independent Study Students: Bryan Bandli; Assessment and Evaluation Chris Fish; Endgame Design and Implementation Kelly Gorz; Assessment and Evaluation Julia Karst-Grey; Geologic Content and Assessment Lucas Koehntop; Java consulting Ned Kruger; Assessment and Evaluation Acey Olson; Bryce Landscapes, Tutoring, Player History, Interface Design Carson Rittel; Content Quality Control, Help Database Joy Turnbull; Content Quality Control, Help Database Dean Vestal; Content Quality Control, Help Database, Interface Design Mindy Sue Vogel; Geologic & Quality evaluation, Alternative exercises K.C. Vorthmann; Geologic Content and Assessment Jeff Walsh; Geologic Content and Assessment Jane Willenbring; Assessment and Evaluation Acknowledgments: Mark Tinguely, who saved our world when its universe imploded, and Dave Schmidt for the name: Planet Oit. Aric Olsen; Geologic Content and Assessment Jennifer Thorstad; Geologic Content and Assessment

World Wide Web Instructional Committee (WWWIC) North Dakota State University Fargo ND