Evaluating Mobile LearningMLearn08 Workshop 07/10/08 G. Vavoula 1/17 Define Research Questions & Evaluation Audiences Giasemi Vavoula University of Leicester

Evaluating Mobile LearningMLearn08 Workshop 07/10/08 G. Vavoula 2/17 What do we want to know (and why) What is the research question? Who cares? Who are the stakeholders/audiences who will utilise the outcomes, and in what ways?

Evaluating Mobile LearningMLearn08 Workshop 07/10/08 G. Vavoula 3/17 A lack of aspirin does not necessarily mean that there is a headache … A lack of research does not necessarily mean that there is a need for research (Randolph 2008)

Evaluating Mobile LearningMLearn08 Workshop 07/10/08 G. Vavoula 4/17 Defining Research Questions Definition of Research Question stems from identified research problem Types of research problems: Scholarly research problems How can learning and psychological theories inform mobile learning practice, and vice versa? What are the best methods for mobile learning research and development, and how can they be improved? What are the best methods to implement and improve the utility of mobile learning innovations? How effective is a mobile learning intervention and how can this effectiveness be improved? What factors moderate the effectiveness of a mobile learning intervention? Evaluation problems More local in scope, aim to answer questions of importance to stakeholders: –Should a certain mobile learning initiative be continued or abandoned? –Have certain initiative objectives been met? Development problems Development of interventions / best practices: –How to develop new / adapt existing mobile learning interventions to solve current educational problems

Evaluating Mobile LearningMLearn08 Workshop 07/10/08 G. Vavoula 5/17 Purposes of research Identification Description Comparison Correlation Experimentation Causal explanation Become aware of the phenomenon under study, its contexts and constructs. E.g.: Identify types of learner reaction to mobile learning intervention

Evaluating Mobile LearningMLearn08 Workshop 07/10/08 G. Vavoula 6/17 Purposes of research Identification Description Comparison Correlation Experimentation Causal explanation Describe quantitatively or qualitatively the attributes of the phenomenon.

Evaluating Mobile LearningMLearn08 Workshop 07/10/08 G. Vavoula 7/17 Purposes of research Identification Description Comparison Correlation Experimentation Causal explanation How instances of descriptions differ. E.g.: Male vs. Female learner reactions

Evaluating Mobile LearningMLearn08 Workshop 07/10/08 G. Vavoula 8/17 Purposes of research Identification Description Comparison Correlation Experimentation Causal explanation Multiple instances of comparison are examined to establish relationships between variables. E.g.: Time spent using technology and learner reactions

Evaluating Mobile LearningMLearn08 Workshop 07/10/08 G. Vavoula 9/17 Purposes of research Identification Description Comparison Correlation Experimentation Causal explanation (or causal description) Special case of comparison where one variable is controlled by researcher. E.g.: New vs. old version of software and learner reactions

Evaluating Mobile LearningMLearn08 Workshop 07/10/08 G. Vavoula 10/17 Purposes of research Identification Description Comparison Correlation Experimentation Causal explanation Explain why/how of causal relationship by examining phenomenon in detail, referring to related theory, and comparing with previous evidence.

Evaluating Mobile LearningMLearn08 Workshop 07/10/08 G. Vavoula 11/17 Purposes of mobile learning research Focus Purpose ProcessOutcome Identification What (new) happensTypes of outcome Description How it happensWhat is learned Comparison How process variesHow outcomes vary Correlation Variables influencing process Variables influencing outcomes Experimentation Manipulate variables to optimise process Manipulate variables to inflict better outcomes Causal explanation Why a variable value results in optimised process Why a variable value results in better outcomes Observable Objective Capture-able Hidden Subjective Obscured

Evaluating Mobile LearningMLearn08 Workshop 07/10/08 G. Vavoula 12/17 Purposes of mobile learning research Focus Purpose ProcessOutcome Identification What (new) happensTypes of outcome Description How it happensWhat is learned Comparison How process variesHow outcomes vary Correlation Variables influencing process Variables influencing outcomes Experimentation Manipulate variables to optimise process Manipulate variables to inflict better outcomes Causal explanation Why a variable value results in optimised process Why a variable value results in better outcomes Observable Objective Capture-able Hidden Subjective Obscured

Evaluating Mobile LearningMLearn08 Workshop 07/10/08 G. Vavoula 13/17 Purposes of mobile learning research Focus Purpose ProcessArtifacts Identification What (new) happensTypes of artifact Description How it happensWhat are they like Comparison How process variesHow artifacts vary Correlation Variables influencing process Variables influencing artifact quality Experimentation Manipulate variables to optimise process Manipulate variables to produce better artifacts Causal explanation Why a variable value results in optimised process Why a variable value results in better artifacts Observable Objective Capture-able Observable Objective Capture-able

Evaluating Mobile LearningMLearn08 Workshop 07/10/08 G. Vavoula 14/17 Purposes of mobile learning research Focus Purpose ProcessArtifactsPerceptions Identification What (new) happensTypes of artifactTypes of perceptions Description How it happensWhat are they likeDescription Comparison How process variesHow artifacts varyHow perceptions vary Correlation Variables influencing process Variables influencing artifact quality Variables influencing perceptions Experimentation Manipulate variables to optimise process Manipulate variables to produce better artifacts Manipulate variables for more positive perception Causal explanation Why a variable value results in optimised process Why a variable value results in better artifacts Why a variable value results in positive… Observable Objective Capture-able Observable Objective Capture-able Semi-observable Subjective Articulable

Evaluating Mobile LearningMLearn08 Workshop 07/10/08 G. Vavoula 15/17 Audiences / Stakeholders Research community How fundamental processes of learning can be mediated, enhanced and transformed Designers How a combination of technologies and activities can best be developed to address problems and provide new learning opportunities Stakeholders: Policy makers, learners, teachers… Learning gains or changes, either through comparison with existing approaches, or by showing how mobile learning can create radically new opportunities, such as linking people in real and virtual worlds (Adapted from Sharples, forthcoming)

Evaluating Mobile LearningMLearn08 Workshop 07/10/08 G. Vavoula 16/17 Audiences / Stakeholders What do they want to know? Basic research knowledge Degree of success of designs Possibilities for improvement Emerging practices / opportunities … How will the research outcomes be used? Inform design Inform theory building Inform policy Inform practice …

Evaluating Mobile LearningMLearn08 Workshop 07/10/08 G. Vavoula 17/17 Points to take away… Consider audiences and their needs Identify research problem and define research focus Formulate research questions