Software Systems for e- learning Evaluation Slavi Stoyanov

Design research Design of a software application Iterative improvements Evaluation embedded in the design and development process Validity & Reliability

Traditional approach Evaluation of the final product Usability Cognitive walking-through Measuring effect

Cases ALFANET SMILE Maker IPSS_EE DIPSEIL Learning style models, modes and controls

SMILE 1(effect of concept mapping for problem solving) Concept mapping as a problem solving tool –Knowledge representation –Knowledge elicitation –Knowledge reflection –Knowledge creation

SMILE 1(cont) Graphical convention vs problem solving instruction Graphical instruction: nodes, links, labels Problem solving instruction: set of heuristics and more concrete techniques that support the cognitive processes of knowledge elicitation, knowledge representation, knowledge reflection, and knowledge creation in each of the problem- solving phases

SMILE 1 (cont) Some examples for CPS techniques Some guidelines that support knowledge elicitation during idea generation phase are as follows: “Look at the map analysis of situation that just has been made. Start to formulate solution by scratch, as many as possible. Write down everything that pops-up to your mind without any judgment.” An example for knowledge changing heuristic during the idea generation is the following: “Take randomly one of the marginal concepts and put it at the very central place of the map. Try to reconfigure the map from this new perspective. Use the new vision as a stimulus for a free association in order to generate as many ideas as possible. Play with labels. Randomly select a pair of nodes and change the links’ label. Use this as a provocation for producing as many solutions as you can.”

SMILE 1 (cont) Research questions What is the effect of type of concept mapping instruction on solving ill-structured problems? What is the effect of individual differences on the construction of concept maps given an ill- structured problem-solving situation?

SMILE 1 (cont) Experimental design: factorial experimental design with a post-test control group Independent variables –Type of instruction (classical concept mapping and SMILE) –Learning style (activist, theorist, reflector, pragmatist) Dependent variables: broad perception, divergence

SMILE 1(cont) Broad perception Fluency – number of nodes, number of links Flexibility –Variety of nodes (facts, data, metaphors, feelings) –Variety of links (descriptive, structural, causal, metaphorical)

SMILE 1 (cont) Divergence –Fluency (number of ideas) –Flexibility – variety of ideas (ready-made solutions, elaboration, unconventional solutions)

SMILE 1(cont) Hypotheses Hypothesis I: the experimental group using the new method for concept mapping instruction will score significantly higher on mapping production than the control group, which applies the classical concept mapping instruction method. Hypothesis II: individual differences in learning styles will predict differences in mapping production and will generate an interactive effect with the type of instruction.

SMILE 1 (cont) Participants: 32 students, randomly assigned to two groups Instruments –LSQ (Honey & Mumford, 1992) –Reflective questionnaire –Mapping production Procedure

SMILE 1 (cont) Results Experimental group significantly better on: Fluency –Number of nodes –Variety of nodes –Variety of links Divergence (number of ideas, variety of ideas) No significant difference on styles

SMILE II SMILE vs Inspiration (explicit vs implicit problem solving support Dependent measurements –Solving a case –Mapping production –Reflection of the participants Experimental design: ‘randomly assigned experimental and control groups with post- test only’

SMILE II (cont) Subjects – 47 Instruments –Reflective questionnaire (method, learning environment, user interface) Procedure

SMILE II (cont) Results SMILE significantly better on solution of a case (F(1, 45) = 5.897, p =.019); flexibility of nodes - p =.024; variety of labels – p =.018; originality of ideas – p =.042) No significant difference on number of ideas. Strong positive relationship between final solution and broad perception and divergence.

SMILE II Results perceived effectiveness (reflective questionnaire) SMILE superior on knowledge representation, reflection and creation. No significant difference on knowledge elicitation

PSS in higher education Electronic Performance Support System (integrates conceptually and defines operationally performance, support and technology system) …just-in-time, just-enough, and just-at-the-point- of need computer support for an effective and efficient job performance. “…people who do not know what they are doing can do it as if they did” (Gery, 2002, p. 29).

EPSS for higher education Focus on active learning Acquisition and application of skills The immense power of technology in addressing instructional issues Appropriate representation and filtering of learning resources Integrative approach for operationalizing performance support

Performance Defining a set of authentic problems and constituting tasks related to a specific working environment Shifting the focus from the lower levels of the learning taxonomy such as knowledge and understanding, towards its higher levels such as solving real-world problems Applying adequate summative performance- oriented assessment methods.

Support Designing a sequence of easy-to-complex tasks Creating opportunities for deliberate practicing these tasks Gradually diminishing the amount of support (scaffolding) Providing variety of instructional stimuli (resources) Allowing constant access to learning resources Giving formative performance feedback Adapting instruction to level of knowledge and learning style of students.

System Using recent developments of information and communication technologies (ICT) Performance support should be embedded into the interface and functionality of the application System depends on how comprehensively performance and support are defined and how well they are operationalized in the architecture and the interface of a system

Pilot study Post-test only experimental group design 9 students in Physics Engineering Traditional teaching vs PSS (IV) Dependent measurements –Performance achievements –Students’ attitudes Instruments –Reflective questionnaire (18 items: 9/9) –Performance test

Analysis and Results A paired-samples t-test Statistically significant increase in performance of students from Time 1 when they worked under traditional settings (M = 8.4, SD = 1.8) to Time 2 when they used PSSL [M = 9.7, SD = 0.7, t(8) = 2.63, p<.05]. The eta squared statistic (.46) indicated a large effect size.

Experimental Study Independent variable: method of instruction with two levels: traditional teaching and PSSL Dependent variables: performance of students on tasks and students’ reflections on the instructional method Control variable: students’ experience with computers Post-test with a control and an experimental group research design

PSS Experimental Study Participants – 40 (‘Information technology for physicists’) Measurement Instruments –Attitude questionnaire –Reflective questionnaire –Performance test

Analysis and Results One-way analysis of variance significant test (ANOVA) with a confidence alpha level of.05 confirmed the hypothesis that the experimental group using the performance support system scored significantly higher then the control group, which worked under the traditional instructional conditions – F (1, 38) = 9.875, p =.003)

Analysis and Results (cont) The independent-samples t-test indicated a difference between the experimental group (M = 4.03, SD =.53) and the control group (M = 3.53, SD =.58) on the learning-by-computer sub-scale as the experimental group scored higher [t(38) = 2.81, p <.05)]. Multiple regression analysis - significance due to applying PSSL was still quite stable [F (2, 37) = 5.061, p =.011.

Conclusions PSSL is an effective instructional approach in higher engineering education. PSSL is more effective in the practical implementation of the ideas related to support and system, but less effective for the performance part. A performance support system for educational purposes creates opportunities for structuring learning resources in a particular way, but it is up to instructors to select concrete learning content and to structure it in a particular way. A promising idea is structuring the information resources as particular categories such as background information (definitions, mental models, theoretical frameworks), examples (work-out examples, simulations, demonstrations), and procedures (guidelines, techniques and tools.

DIPSEIL issues (cont) The effectiveness of the implementation of the idea of learning style adaptation in a performance support system is not affected directly by the component interface structure consisting of advisory component, an information component and a training component, but rather by the structure of the content and learning activities. Evaluation of the effectiveness of adaptive software applications in most of the cases does not apply powerful research designs. The research design in the evaluation of performance support system in education do not control for possible effect of learning style as it is expected that performance systems might better support some of the learning style, namely activist and pragmatist.

Adaptation in PSSL The definition of PSSL implies the idea of embedded adaptation through accommodating learning style within the structure of content and learning activities. The DIPSEIL PSSS structures learning content into: background information (definitions, theoretical models and frameworks), examples (work-out examples, demonstrations, simulations), procedures (step-by-step approaches, techniques, instruments,). The system requires deliberate practicing of learning tasks. Developing a versatile learning strategy requires students to apply the cognitive mechanism of copying behaviour.

Research Questions (DIPSEIL) What is the effect of adaptive performance support system on learning achievements of students? Is the effect of performance support system different for learning styles? Assumptions: –working with the adaptive performance support system (DIPSEIL) will result in a better performance compared to non-adaptive performance support system (IPSS-EE) –The adaptive performance support system (DIPSEIL) will improve the performance of students across all learning style.

Design, variables Factorial experimental design with post- test measurement Adaptive (DIPSEIL) vs non-Adaptive (IPSS_EE) system (InV) Learning styles (4) Performance achievements (DV) 46 students (Informatics for Physics Engineers)

Results The adaptive PSSL is superior to the non- adaptive one (F(1,38) = 7.7, p =.009 ); large effect size (partial eta squared is.17) Non-significant main effect of learning style on performance achievements of students, [F (38, 3) =.056], p =.98, eta squared.004 No significant statistical difference is found on the interaction effect between method of instruction and learning style, [F (38, 3) =.9], p =.015, eta squared,.015

Method and LS

Conclusion Embedded adaptation is an effective and efficient way for improving performance of learners – a way of practically resolving (a) the contradiction between adapting to instruction vs adapting to learners; and (b) the issue of preferential vs compensational modes of adaptation Embedded adaptation serves equally well all learning styles. Technological development of adaptive software applications based on learning style should reflect the developments in the contemporary learning style paradigm and the theories of adaptive instruction.

Adaptive models, modes and controls Conceptual opertionalisation of learning style –Level vs style –Process vs style –Preferable vs observable behaviour –Measurement instruments

Adaptive approaches Preferential vs Compensation adaptation Pre-assessment vs Embedded adaptation Design-time vs Run-time adaptation

Research questions What is the effect of matching, compensating, and monitoring adaptive instructional approaches on complex learning? Is there any effect of learning styles on learning achievements in complex learning situations?

Research Design Factorial –IDvs: Adaptive approaches (Matching, Compensating, Monitor) and Learning Style (Activist, Reflector) –Dv: learning achievements of students Participants: 49 divided into 3 groups (Matching, Compensating, Monitoring)

Measurement Instruments Achievement test (testlet): 10 items Revised by Kirton & de Ciantis (1996) LSQ (Honey & Mumford, 1992) Activist-Reflector scale

Results Two-ways ANOVA No statistically significant main effect for groups Mean score of Monitor group higher than Activist and Reflector groups No main effect for learning styles Reflectors more comfortable with Preferential adaptation; Activists feel better with Compensation adaptation