Applying agent technology to evaluation tasks in e-learning environments

Selby Markham, Jason Ceddia & Judy Sheard CSSE, Monash Colin Burvill & John Weir Mechanical and Manufacturing Engineering, U M Bruce Field Mechanical Engineering, Monash Linda Stern & Leon Sterling CSSE, UM

The PEDANT project grew out of a desire/need to understand how current students deal with electronic materials.

Aims to investigate The relationship between the way students use on-line and interactive educational tools and the quality of their learning experience. Using automated, agent-oriented software tools

The educational tools SiMLED A simulator for exploring engineering principles. Engineering

Algorithms in Action : AIA An interactive, visual tool for exploring programming algorithms CSSE The educational tools

MOMUS Tutor A graphics-based tutorial system on mechanical engineering principles Engineering The educational tools

Web Industrial Experience Resource : WIER A Web-based resource to support students in the Industrial Experience project CSSE The educational tools

What is a software agent? A bit like TRON

What is a software agent? Your computer Virus Checker

What is a software agent? It has a purpose It can be given intelligence It is responsive It can be designed to learn It acts independently

What is a software agent? It has a purpose It is responsive It can act independently

It has a purpose To monitor software output To collect appropriate output To organise that output

It is responsive The information can be analysed Notifications can be given

It can act independently When initiated it needs no support

It can be given intelligence (Phase 2 of the project)

But does it have Ethics?

But why use agents?

Software tools create issues that are not easily addressed by self-report techniques or observational techniques.

Have you tried to analyse the log data from a Web-based tool? The extent of the data is mind-boggling

Have you tried to trace the informational path used by the learner? How often have you been unable to trace your own path when working on the Web?

The development of agent technology is a productive direction for achieving both short- term and long-term goals

Agent technology is technologically compatible with the teaching/learning tasks. Agent technology can provide data about what the learner is doing rather than what he/she remembers that he she was doing. Agent technology provides a tool with a highly pervasive ability to carry out functional formative assessment.

Conceptualising the basic process

Educator Learner Software Learning Objectives Learner Motives Evaluation Learning Tasks Learner Behaviour Inferred Behaviour Responses Learner Outcomes Measures

Educator Learner Software Learning Objectives Learner Motives Evaluation Learning Tasks Learner Behaviour Inferred Behaviour Responses Learner Outcomes Measures 2003

Educator Learner Software Learning Objectives Learner Motives Evaluation Learning Tasks Learner Behaviour Inferred Behaviour Responses Learner Outcomes Measures 2003 We will have achieved the primary aim of developing: A rationale for defining pedagogical elements Prototypical agents that can monitor the software Have analysis models that can apply to evaluation tasks

Educator Learner Software Learning Objectives Learner Motives Evaluation Learning Tasks Learner Behaviour Inferred Behaviour Responses Learner Outcomes Measures 2003 The evaluation component: The evaluation of the functional usability will have begun The format for evaluating the processes being used by students - a broader view of formative assessment

Educator Learner Software Learning Objectives Learner Motives Evaluation Learning Tasks Learner Behaviour Inferred Behaviour Responses Learner Outcomes Measures There are many more tasks to be done to fill in the underlying matrix

The agent in evaluation

Provides are means of monitoring process and relating process to educational goals

This creates what can be called a process evaluation – as defined in the broader evaluation literature. This is similar to formative evaluation but more as originally conceptualised by Scriven.

The current process model for the project

e.g.

Action (User, Type, Time, [parameter list] )

action (user code, page, time [URLdata])

Action (User, Type, Time, [parameter list] ) action (user code, page, time [URLdata]) download (403, file_manager, , [doc_rep: 0 Requirements-Model.doc])

The practical meaning could be Learning tasks that the software enables: exploring a particular area accessing a particular resource

Learner behaviours: passive lost wandering directed interaction linear, orderly interaction exploration (at different levels of depth)

FIN