KAJO – E-xcellence assessment experiences E-xcellence seminar, Oulu Eetu Pikkarainen

2008 Quick Scan Self assessment questionnaire in Webropol: Report, planning the next phase, new articulation of the objectives

2009 Team discussion Same questionnaire in Webropol Now responses from almost all types of stakeholders

Comments Very demanding and hard questionnaire Some questions hard to understand and very broad Orientation: what is the institution? Here University: (University has strategies) Part of the question suit for one type of stakeholders (e.g. students)

STRATEGIC MANAGEMENT

1.The institution has e-learning policies and a strategy for development of e-learning that are widely understood and integrated into the overall strategies for institutional development and quality improvement. Policies include both infrastructure and staff development. University in a state of change and its strategic planning is maybe a little fragmented. Campus Futurus ( is a network organisation that is responsible for drawing up and developing the strategy for the educational use of the ICT at the University of Oulu. The wide understanding and integration to action is at the moment maybe not as good as could be hoped. Partially Adequate

2.The resourcing of developments in e-learning curricula take into account any special requirements over and above the normal requirements for (non-e) curricula. These will include items such as equipment purchase, software implementation, recruitment of staff, training and research needs, and technology developments. These are planned at university level in the above mentioned strategy. The needs for pedagogically justified resourcing are not always as flexible as could be hoped because resourcing and maintaining of equipment and software is mainly centralised. In KAJO project we could get extra resourcing for important web conferencing tool (Adobe Connect Pro, ACP). Partially Adequate

3.The institution has a management information system which is reliable, secure and effective for the operation of the e-learning systems adopted. The study and student management information system is in state of change in open university and new system will be the same as in whole other university and many other universities in Finland (OODI). This is a great betterment. (The staff management system is separate.) Partially Adequate

4.When e-learning involves collaborative provision the roles and responsibilities of each partner are clearly defined through operational agreements and these responsibilities are communicated to all participants. This is especially under work in KAJO project. We use process modelling to define roles and tasks of all participants. (Earlier there may have been some small occasional confusion.) Largely Adequate

CURRICULUM DESIGN

5.E-learning curricula conform to qualification frameworks, codes of practice, subject benchmarks and other institutional or national quality requirements in the same way as non-e curricula. In KAJO project the non e-curriculum has been fitted to e-learning and the main goal has been that quality may not decrease. Actually quality has been tried to increase by using these E- xcellence benchmarks. For non e-curriculum there does not exist so detailed frameworks in national and local levels and quality manuals for teaching are just under work in university. Largely Adequate

6.Curricula are designed in such a way as to allow maximum flexibility for the learner with respect to time, place and pace of learning, consistent with the satisfactory achievement of learning outcomes and integration with other (non-e) learning activities. Use of formative and summative assessment needs to be appropriate to the curriculum design. Flexibility has been increased but offering web conferencing lectures and demand to attend in network discussions force some time restrictions for courses. (More flexibility would put quality in danger.) In programme level there will be possibility to start and finish flexibly if the new management information system will not prevent it. (What does that last sentence mean here?) Partially Adequate

7.Curriculum design ensures that appropriate provision is made for the acquisition of general educational objectives and the integration of knowledge and skills across the programme of study. When blended learning is used, the contribution of e-learning components to the development of educational objectives needs to be made clear. In KAJO project the general educational objectives are taken into account. Usually in university studies today the special learning outcomes may take too much weight. The effect of different e-learning tools and methods to general educational objectives should generally be discussed more. Largely Adequate

8.Curricula are designed in such a way as to require broad participation in an on-line academic community. As well as student-student and student-tutor interactions this includes, where appropriate, interaction with external professionals and/or involvement in research and professional activities. Until now not much interaction with external professionals has been offered. This will be an aim in higher levels of studies. Now the communities consist mainly of students, tutors and teachers. Partially Adequate

COURSE DESIGN

9.Each course includes a clear statement of learning outcomes in respect of both knowledge and skills. Outcomes are of such a nature as to be attainable through e-learning, augmented as and when necessary by face-to- face provision. In a blended-learning context there is an explicit rationale for the use of each component in the blend. Learning outcomes has been defined for each course, but same definitions go for e-leaning and non e-learning alternatives of courses. E- learning and blended model is developed rather for more flexibility and efficiency than for some specific outcomes. Largely Adequate

10.Learning outcomes, not the availability of technology, determine the means used to deliver course content and there is reasoned coherence between learning outcomes, the strategy for use of e-learning, the scope of the learning materials and the assessment methods used. Inside e-learning the means to deliver courses are selected so that they would best support learning objectives and outcomes. Partly the availability of technology is restricting the selection. Those components has been planned to create a coherent whole. Largely Adequate

11.Learning outcomes are comparable with those of courses delivered by other means. The defined leaning outcomes are same in same courses. Also the attained leaning outcomes are believed to be at least same, but it should be researched more strictly in future. Largely Adequate

12.Courses should be designed in such a way as to : –foster active learning –facilitate individual study and the development of study skills –support the development and interaction of learning communities –place the learner in control of time, place and pace of learning wherever possible –recognise the diversity of learners and build on their strengths and background –make appropriate provision for persons with disabilities –be sensitive in their use of materials to the cultural diversity present amongst learners –require learners to reflect on, evaluate and provide feedback on course contents and requirement This has been the goal of KAJO development work. The most serious deficiencies are in flexibility of time (se above) and recognition of diversity of learners and their possible disabilities and cultural backgrounds. The feedback system should take these problems up when they will appear and then they will be addressed. Largely Adequate

13.Interactions between students and with tutors (both synchronous and asynchronous) are facilitated by a variety of means including , telephone, group forums etc. to allow both individual and group interactions. Access to tutors is designed to be on a regular and sufficient basis known to both tutors and learners. At the minimum level of engagement tutors provide learners with timely expert advice on course issues or materials and individual feedback on assignments. Tutors and their interaction with students is one main development area of KAJO project. Telephone is not regularly used but during interaction periods tutor is accessible by and VLE. Perhaps tutors should be resourced with even more time for each student in future. Fully Adequate

14.Course design, development and evaluation involve individuals or teams with expertise in both academic and technical aspects. Integral to the course design process are mechanisms for trialling materials and incorporating feedback into the final product. This is realised quite well during the project phase, and plans are being made for future. Feedback from students is most important, but in design phase course plans are discussed in a team of stakeholders. Largely Adequate

15.Learning materials are designed with a sufficient level of interactivity to enable active student engagement and to enable them to test their knowledge, understanding and skills at regular intervals. Where self-study materials are meant to be free-standing, they are designed in such a way as to allow learners on-going feedback on their progress through self- assessment tests. Self-assessment tests are used in some courses, but mainly the interaction and learning feedback is tried to ensure by tutors’ work. Partially Adequate

16.Course materials conform to explicit guidelines concerning layout and presentation and are as consistent as possible across a programme. This has been tried for with a special publishing platform which is especially designed for learning materials (Apumatti). In addition some uniform editorial guidelines have been followed. Largely Adequate

17.Course materials including the intended learning outcomes, are regularly reviewed, up-dated and improved using feedback from stakeholders as appropriate. This is also realised quite well during the project phase, and plans are being made for future about yearly updates. Largely Adequate

18.Courses provide both formative and summative assessment components. Summative assessment are explicit, fair, valid and reliable. Appropriate measures are in place to prevent impersonation and/or plagiarism, especially where assessments are conducted on-line. Summative assessments are mainly by essays, and their evaluation is as explicit and valid as in non e-learning curricula. Impersonation and plagiarism are tried to prevent by continuous interaction in courses. In future we will get anti- plagiarism software (Urgund). Perhaps assessment should be made more explicit and formative assessment increased. Largely Adequate

COURSE DELIVERY

19.The technical infrastructure maintaining the e-learning system is fit for purpose and supports both academic and administrative functions. Technical specifications are based on a survey of stakeholder requirements and involve realistic estimates of system usage and development. Partly technical infrastructure is based on historical reasons and availability of technology: some tools (VLE) just happens to bee in wide use in university and thus strategically supported. Partly it is reasonably renewed and functioning very well (ACP). Partially Adequate

20.The reliability and security of the delivery system have been rigorously tested beforehand and appropriate measures are in place for system recovery in the event of failure or breakdown. Systems are proved to be reliable in action. In office hours recovery is fast because maintenance is centralised in university. (One unlikely problem is that if ACP system should break down during evening lectures, then teaching must be transferred to new time. Largely Adequate

21.Appropriate provision has been made for system maintenance, monitoring and review of performance against the standards set and against improvements as these become available. This is managed centralised by IT Administration Services of the university. Largely Adequate

22.The Virtual Learning Environment is appropriate for the pedagogical model adopted and for the requirements of all users. It should is integrated with the institution's management information system as far as possible. Used VLE (Optima) does support well neither the pedagogical model nor the integration on information, but it is only platform at the moment which is maintained by university IT Administration Services. So we have no resources for better system. In the e-learning strategy of university there is a plan to offer better platform by 2012 Partially Adequate

23.The VLE provides information and services to all users in a logical, consistent and reliable way. All users are confident that the VLE's systems for communication and provision of information are secure, reliable and, where appropriate, private. In this respect the VLE functions quite reliably. Largely Adequate

STAFF SUPPORT

24.Materials and information accessible through the VLE are regularly monitored, reviewed and updated. The responsibility for this is clearly defined and those responsible are provided with appropriate and secure access to the system to enable revision and updating to occur. Teachers are responsible to review and update VLE as part of their normal course planning. Largely Adequate

25.All staff with academic, media development and administrative roles is be able to support the development and delivery of e-learning programmes without themselves being technical experts. The institution ensures that appropriate training and support is provided for these staff and that this training is enhanced in the light of system developments. Training in educational technology is offered widely in the university; also technical expert help is available. In this project and programme tutors and teachers are familiarised as needed. Largely Adequate

26.Pedagogic research and development are regarded as high status activities within institutions with a commitment to high quality e- learning. There are mechanisms within these institutions for the dissemination of good practice in support of e-learning (including good practice developed elsewhere and/or through consortia), and for the training or mentoring of new staff in such practice. Career development incentives reflect an e-learning culture. This is realised quite well also in the university level. (Although in university level it may be suspected that it does not widely reach all staff). In our project and programmes good practises all disseminated in team meetings. Largely Adequate

27.The institution ensures that issues of staff workload and any other implications of staffs' participation in e-learning programmes (such as intellectual property rights over programme materials) are taken proper account of in the management of courses or programmes. Copyright contracts are made beforehand on study materials. Teacher workload is taken into account in work plans and contracts. Some ambiguity exists about how the amounts of work in e-learning an non e-learning curricula are compared. Largely Adequate

28.Institutions ensure that adequate administrative support (including effective management information systems) is available to academic staff, particularly part-time tutors/mentors. Because information management systems are not well integrated and partly new staff does not always easily get the information it would need. Partially Adequate

STUDENT SUPPORT

29.Prospective students are provided with a clear picture of what will be involved in pursuing the e-learning programme and the expectations that will be placed on them. This includes information on technical (system and VLE) requirements, requirements concerning background knowledge and skills, the nature of the programme, the variety of learning methods to be used, the nature and extent of support provided, assessment requirements, fees, etc. A comprehensive and extensive info material is planned. At the moment they get the information in face to face info meetings. Largely Adequate

30.E-learning students are provided with the equivalent of a student handbook setting out their rights and responsibilities, those of their institution, a full description of their course or programme, and information on the ways in which they will be assessed. This will be part of the planned info material. At the moment the information is a little bit scattered. The students of open university get a study guide handbook, but it is a little bit more fitted to non e-learning curricula. Largely Adequate

31.E-learning students have access to learning resources and learner support systems which, although they may be provided through different means, are the equivalent of those available to campus-based students. These include: –access to library resources –support for the development of key skills (including support for e- learning skills, collaborative working on-line and contributing to on-line communities which are key skills in an e-learning context) –advice and counselling over choice of courses and progression through the programme –an identified academic contact, tutor and/or mentor who will provide constructive feedback on academic performance and progression –access to help desk, administrative support and advisory services –opportunities to provide and receive formal feedback on their experience on the course –procedures to handle and resolve any difficulties or disputes which may arise. Now at last via new study information system also open university students will get similar access to library resources than basic students of university. Other resources are offered well. The greatest problem is that students are passive to give feedback and so maybe also to take the available support. Some activation methods must be developed. Largely Adequate

32.Students are provided with clear and up-to-date information on the range of support services available and how these may be accessed. This will be part of the planned info material. Largely Adequate

33.The expectations on students for their participation in the on-line community of learners are made clear both in general terms and in relation to specific parts of their course or programme. In addition to info material and general activation, the participation in group interaction is an explicit part of study methods in courses. Largely Adequate

Thank you! Comments and questions!?