Project VILAROB TR1-LEO Second Meeting Warsaw, (WP2) presentation of ISAR Analysis prepared by ISAR partners. Marcin Słowikowski

TR1-LEO Second Meeting Warsaw, Content As in the user requirements identification stage, three basic types of questions have been included in the questionnaire, as follows. Factual questions, such as identification of the problems that the users had when accessing the ISAR system. Classification questions, such as the definition of respondents’ content delivery preferences. Attitude/belief questions, such as definition of the learning requirements of the users and whether these have been met.

TR1-LEO Second Meeting Warsaw, The questions have also been classified into seven additional groups according to their content. These include the following. Technology usage – The range of services and tools offered by the ISAR platform are included in this group. Delivery – The way the content is delivered to the users is compared to traditional techniques and related problems are identified. Content – That group refers to the content of the lessons. The content is judged according to its consistence, attractiveness, and quality. Impact and learning outcomes – The learning objectives of the participants are analysed and a first indication of the impact of the ISAR platform is acquired. Usability – Questions in this group are related to the use of the ISAR system, including presentation and overall functionality aspects. General impression – Each of the previous groups has been focused to a specific aspect of the ISAR platform. This group of questions instead identifies how the users perceived the ISAR platform as a whole and how they believe this platform could be further improved. Generation / update – Specific questions addressing Training Organisations & Education Organisations (Target Group IV: Trainers and Consultants): Questions in this group are to analyse if the ISAR services and methodology support material will meet requirements of this groups regarding creation and updated of course material.

TR1-LEO Second Meeting Warsaw, Data analysis  The questionnaire has been completed by 91 people. 11 of them are members of target group I (SME manager), 15 of them are members of target group II (technical staff), 8 of them are members of target group III (less qualified and unemployed workers), 7 of them are members of target group IV (Trainers and Consultants), and 50 of them are members of target group V (Students and unemployed highly qualified people).

TR1-LEO Second Meeting Warsaw, Data analysis The questionnaire has been disseminated in the following countries: Poland, Germany, Slovakia, UK, Romania, Spain, and Greece. As already shown by first evaluation of early prototype of the ISAR solution, the majority of users prefers to communicate with other users via (39%) and chatroom (40%). Especially students and unemployed highly qualified people (48% of target group V) prefer to use a chatroom for communication between users. Both features preferred by majority of users of all five target groups are implemented in full ISAR system solution.

TR1-LEO Second Meeting Warsaw, Data analysis Most of the respondents had no problems when accessing the ISAR system. Furthermore, some of the respondents from all target groups detected slow network connection (23%), resulting also in multimedia viewing problems (20%), although ISAR pages were redesigned for low bandwidth connections. However, the server load may need to be distributed to different location. Moreover, interface problems still existing at early prototype phase (e.g. layout, colours and complaints about the background colour of the ISAR system) could be almost eliminated for full ISAR solution.

TR1-LEO Second Meeting Warsaw, Data analysis The respondents have also reported their delivery preferences after using the ISAR system. The ISAR platform corresponds to their preferences as approx 2/3 of them study at times suited to their individual needs and prefer learning at their own pace. Another interesting fact is that more than half of all respondents from all target groups like face-to-face communication with the teacher, which may explain the interest chat rooms and in message boards. Moreover, the importance of adding interactive content to the ISAR platform is noted by all of the respondents.

TR1-LEO Second Meeting Warsaw, Data analysis A worrying fact is that especially users of target groups I-III have reported production errors (e.g. typographical mistake) in the ISAR lessons. The content should therefore be continuously and thoroughly checked for errors. Most of the respondents do not disagree with the statements that the ISAR course has good structure and informative material. Only 14% of trainers and 18 % of students disagree on that. Moreover, 2/3 of all users believe that the course has an appropriate difficulty level, while further 30% neither agree nor disagree with that statement. The point where the ISAR course maybe needs to be improved is its multimedia resource as only 56% of users from all target groups believe that the multimedia is of high quality, while further 39% neither agree nor disagree with that statement.

TR1-LEO Second Meeting Warsaw, Data analysis The examples included in the ISAR lessons are very useful according to 100% of the respondents from target group I-III (overall 97%) while 29% of trainer would like to improve them. 44% of the users would like to have more examples included into the future, whereby 73% of students and unemployed highly qualified people (target group V) don’t ask for this. This fact emphasis the need for target group specific courses and should be noticed for further improvements. More or less all participants (87%) think that the questions at the end of each lesson to be relevant to the taught subject. The questions are also considered attractive (65%) and accurate (72%) most of the respondents. However, the quality (71%) and even more the attractiveness of the questions should be continuously improved.

TR1-LEO Second Meeting Warsaw, Data analysis The wide range of the learning objectives is another indication of the multi- cultural and multi-disciplinary nature of the respondents. Most of them visited the ISAR system to acquire new knowledge, while other goals included enhanced academic and practical skills, and advance motivation. Even more encouraging is that 100% of all participants from all five target groups report that their learning objectives have been met. The respondents have also been asked questions related to the general impact of the ISAR platform. They believe that the ISAR system will lead to increased understanding of A&RM systems (62%) and will help users become more efficient with their jobs (66%). However, a large percentage (20%) does not agree nor disagree with above statements. Moreover, a large percentage (64%) believe that ISAR system will lead to more people studying A&RM related subjects. This indicates that the full ISAR solution could be improved but at the same time that their is still some further improvement potential to make ISAR content and its delivery more interesting for the users. Possible solutions would be to add more interactive media.

TR1-LEO Second Meeting Warsaw, Data analysis - Usability One of the most important factors for advancing the use of the ISAR solution is its usability potential. The target groups should be able to use the ISAR system with minimum training and as fast as possible. In this connection the full ISAR system could convinced all users, given that all of them (100%) were able to start using the ISAR system with only the on-screen instructions and further 92% of them were able to predict the result of clicking on each button or link. Moreover, 92% believes that provided graphics and multimedia assist them in noticing and learning the content rather than simply entertaining. The ease of use of the system is also reflected in a related question where 77% of the respondents grade the ISAR system as being easy or very easy to use. Moreover, 67% of the respondents say that ISAR solution provides a informative glossary and further 19% characterise it mediocre.

TR1-LEO Second Meeting Warsaw, Data analysis – Usability Another important factor is the visual design/layout of the ISAR system. As in previous questions that feature has received comparatively lower marks than other usability features (56% of the respondents only say that the visual design is good while 31% characterise it mediocre). Since the ISAR system is an e-learning solution, the media integration is also important. The results indicate that the content is well coordinated with the multimedia (63% of the respondents say that the integration is good) but it could be further improved (24% of the participants claim that the integration is mediocre). Overall 83% of the participants have been satisfied with the usability features of the ISAR system.

TR1-LEO Second Meeting Warsaw, Data analysis - General impression 73% of the respondents say that the content of ISAR system is that the is more practical and reflective, 67% that it is easier to use and follow, 64% that it is more focused than traditional teaching methods, 63% of them that it is more flexible, 62% that it is more user friendly, and 60% visually more stimulating. The respondents believe that the course could be improved by make course activities more stimulating (17%), improve the examples used in the course (15%), update content of course (14%) and increasing the multimedia used in the course (13%), 91% of the people who responded to the questionnaire will recommend this course to their colleagues. At the last question, respondents could add any additional thoughts and questions they may had. Their comments are representative of the aspects that need to be improved, including the visual design/layout and interactive/multimedia content.

TR1-LEO Second Meeting Warsaw, Conclusion 100% of all participants from all five target groups report that their learning objectives have been met and that they are able to start using the ISAR system with only the on-screen instructions. All services that are required by the users are already included in the ISAR system. The examples included in the ISAR lessons are very useful according to 96% of the respondents while 44% requires more examples to be included in the future. 87% of the participants think the questions at the end of each lesson to be relevant to the taught subject. Therefore, questions and examples are very important and should be continuously increased based on user feedbacks.

TR1-LEO Second Meeting Warsaw, Conclusion The contents of the ISAR courses should be checked for production errors. The respondents believe that the ISAR system will lead to increased understanding of A&RM systems (62%) and will help users become more efficient with their jobs (66%). Overall 83% of the participants have been satisfied with the usability of the ISAR system. 91% of the ISAR users are going to propose the ISAR system to their colleagues, which means that they consider it as a useful application.