1. http://calibrate.eun.org WP4: Validation Andrea Kárpáti, ELU (karpatian@axelero.hu)karpatian@axelero.hu) In cooperation with Roger Blaimire, EUN (roger.blamire@eun.org)roger.blamire@eun.org

2. http://calibrate.eun.org CALIBRATE Review 19 December 2006 2 Validation framework Levels of validation – S ystems – Institutional - school level – Pedagogical - teacher/student level, educational usability – Technical - performance of project deliverables versus school insfastructure: technical usability – Economic – sustainability, dissemination – Cultural/Linguistic – how content, technology, inherent teaching style style etc. Travel Methods of validation – Usability tests: National (pre)pilots, (inter) national workshops, (inter) national discipline based collaborative groups – Interviews: semi-structured individual, focus groups – (Pre)pilots, school based usability tests – Workshops: (inter)national

3. http://calibrate.eun.org CALIBRATE Review 19 December 2006 3 Institutional / school level Assessment criteria e-maturity change in teaching and in collaboration dissemination involvement of wider community change in school ICT culture communication and management) Assessment instruments ICT Competence Inventory (ICT/CI) Educational Strategies Questionnaire (ESQ) Focus group interviews (teachers, heads, students of same school)

4. http://calibrate.eun.org CALIBRATE Review 19 December 2006 4 Pedagogical - teacher/student level Authoring, adapting, modifying, sharing of LO Implementation in classroom teaching Impact on practice e.g. the role of teachers, teaching strategies, teaching styles Discipline-related experiences – Math, Science, Environmental Science, English as Foreign Language. Lesson plans – online template (lesson type, content, ICT and traditional tools used, pedagogical strategy, evaluation methods etc. Blog – online diary on working with Calibrate tools and their utilization: experiences, problems, desires for further development National and international fore for discipline-based teacher groups Focus group interviews (teachers of the same discipline from different schools)

5. http://calibrate.eun.org CALIBRATE Review 19 December 2006 5 Technical - performance of Calibrate tools and the LRE How deliverables work in practice Implications for ICT resourcing in the school Localization issues 1: technical aspects Technical support Integration with existing ICT technology Adaptability to school needs Economic Cost for schools: integration with existing environment, training, technical support (Pre)pilots Blogs International forum for piloting teachers SUSTAINABILITY STUDY I: – school visits with interviews one year after project ends – ICT infrastructure survey at the beginning of the project and one year after finish – Interview with school leadership

6. http://calibrate.eun.org CALIBRATE Review 19 December 2006 6 Cultural/Linguistic Localization issues 2: cultural aspects Support for multiple languages Intercultural (what aspects of national educational culture does the project support and what not?) Trans-national collaboration Culture of education (attitude to innovation, ethos of profession, private and public sector) Attitude of stakeholders (students, researchers, support staff) International workshop for teachers with focus group interviews and collaborative LO design experiments Interviews with national educational policy makers experts for (ICT policy, European harmonization of curricula, SUSTAINABILITY STUDY II: – Attitude surveys (participating teachers, heads and peers at piloting schools, other stakeholders) – On-site observations – peer review visits

7. http://calibrate.eun.org CALIBRATE Review 19 December 2006 7 Timeline for Validation 1 January 2006: disciplinary areas for validations selected, curriculum analyses January – March 2006: selection of piloting school / teachers May 2006. 1. pre-pilot May-September 2006: elaboration / discussion of validation framework September – November 2006: teacher training where needed, pilot of validation instruments October – December 2006: elaboration of training material for validating teachers December 2006: 2. pre-pilot, finalisation of validation plan and instruments

8. http://calibrate.eun.org CALIBRATE Review 19 December 2006 8 Timeline for Validation 2 January 2007: teacher training courses end with module on CMS, with special focus on LRE and Toolbox February – May 2007: Validation Round 1: school usability studies followed up through lesson plans, interviews and site visits June-July 2007: summary of VR1, further development / eventual improvement of tools, planning for VR2 September – December 2007: VR2 January – February 2008: summary of VR2, finalisation of Calibrate tools, suggestions for dissemination

9. http://calibrate.eun.org CALIBRATE Review 19 December 2006 9 Prepilot 1: First impressions Aim: test basic functionalities of LRE Methods: guided walkthrough, structured questionnaire for individual walkthrough followed by plenary discussion – all video/audio-taped Participants: 2 teachers / country from 6 countries, LRE and Toolbox designers, WP4 coordinators Results: design and help improved, relevance of functions for teachers clarified, further functions planned – co-evolutionary design

10. http://calibrate.eun.org CALIBRATE Review 19 December 2006 10 Prepilot 2: User test Experienced users (teachers of the 1th pilot) and newcomers to Calibrate tools piloted, 2x2 teachers / 10 piloting countries Self-report questionnaire filled out while performing tasks in the LRE Topics: – Ease of use (navigation, performance of tasks) – Visual aspects (pleasingness, communication) – Pedagogical aspects (relevance of tools / content) – Suggestions for tool design – General attitudes / feelings / motivational level changes (moods) while using tools Summary by ELU will compare „expert” and novice” users’ attitudes, usage patterns, levels of involvement, critical observations and satisfaction

11. http://calibrate.eun.org CALIBRATE Review 19 December 2006 11 Teacher Training for Calibrate Methods and content varies acc. to participating country / teachers selected. Common core: using learning content management systems incl. LRE and Toolbox – a unit developed by ELU Phase 1, preparation for pilot: collaborative learning in school-based, national groups, theme: ICT in education Phase 2, pilot: knowledge building community of teachers – discipline-based work in (inter)national groups Course types: – accredited national training courses – „crash course” on LRE and Toolbox – EPICT – the European Pedagogical ICT Licence

12. http://calibrate.eun.org CALIBRATE Review 19 December 2006 12 Educational Strategies Questionnaire F1: Aims and objectives of school education – present and future – Values, attitudes and beliefs about teaching and education – The Role of school F2: ICT at school: function and mission – Existing good practice / problems – ICT development plan F3. The role / task / mission of the teacher F4 Own educational practice – Teaching environment / infrastructure – Paradigms, strategies, methods – Preparation for teaching F5. Opinion / plans / wishes about in-service training F6. Innovation experiences F7. Personal efficiency / resilience / coping with change

13. http://calibrate.eun.org CALIBRATE Review 19 December 2006 13 ICT Competency Inventory (ICT-CI) A self report questionnaire piloted in a project measuring ICT readiness in about 150 schools of a district in Budapest. The ICT-SI is an index based measuring tool. (The concept of indexing is used for the data reduction method employed: several variables are composed in one index.) Our test comprises the following indices: Index (1) – School access to ICT Index (2) – Home access to ICT Index (3) – ICT related attitudes Index (4) – ICT related competencies Index (5) – Educational use of ICT

14. http://calibrate.eun.org CALIBRATE Review 19 December 2006 14 Four domains for every index: „low” (value range: 0,01–1), „medium” (value range 1,01–2) „high” (value range 2,01–3) 0 value, indicating the non-existence of any interpretable data for the index. (E. g., low level ICT access means sporadic, rare access while 0 level indicates no access to a PC). Items correlated with the presence / absence of certain personality traits measured by the California Personality Inventory (CPI) to identify teacher profiles that tend to success or fail with ICT Individual training / coaching needs

15. http://calibrate.eun.org

16. http://calibrate.eun.org CALIBRATE Review 19 December 2006 16 Personality tests for better validation results? Coaching -. A generally accepted performance enhancement method in industry and commerce, very rare in education EPICT (European Pedagogical ICT Licence), www.epict.org – nationalization and adaptation: usability at stakewww.epict.org CPI was used to define basic personality traits relevant for the world of work (sociability, innovativeness, coopeartion, assertiveness etc.) CPI results were correlated with ICT Competence Inventory and Education Strategies Questionnaires LEARNER PROFILES defined – facilitators could improve mentoring and customize assessment BFQ to be used for Calibrate in Hungary and offered for other participants for consideration

17. EPICT pilot in Hungary: Strong correlation between personality traits and ICT skills, attitudes and competence 1. School ICT access 2. Home ICT access 3. ICT attitudes 4. ICT compe- tences 5. Educational use of ICT Sociability0,154-0,1040,1530,222*0,123 Social skills0,1200,0900,1390,275**0,015 Self respect0,0590,0390,0550,0940,034 Gen. wellbeing (Wb) 0,219*-0,0570,218*0,256**-0,074 Self control0,133-0,0310,222*0,120-0,064 Tolerance0,1780,0000,354**0,351**0,007 Establ. good relationships 0,179-0,0300,233*0,262**-0,149 Building a community -0,0930,200*-0,118-0,1390,230*

18. http://calibrate.eun.org CALIBRATE Review 19 December 2006 18 Intellectual efficiency (Ie) 0,1830,0160,238 * 0,332**0,002 Flexibility (Fl) 0,1630,0140,1620,306**0,126 Feminine qualities (Fm -0,0670,076- 0,020 -0,1030,114 Extraversion (Ex) 0,151-0,0330,1620,271**0,052 Reliability (Re) 0,1470,0530,1340,228*0,055 Emotional stability (Es) 0,114-0,0300,1150,226*0,046 Intellect (In)0,150-0,0160,1420,208*0,029 1. School ICT access 2. Home ICT access 3. ICT attitudes 4. ICT com- petences Educational Use of ICT Pearson two-tailed correlation, significance levels: *: 0.05, **: 0,01

19. http://calibrate.eun.org CALIBRATE Review 19 December 2006 19 Correlation analysis of CPI and ICI findings, questionnaires and semi-structured interviews, show personality features that are associated with success and failure to acquire ICT skills on a level sufficient for educational use Some of these variables may be affected by specially targeted mentoring methods, teaching environment course content (best practice examples) custom-made skills training. Based on EPICT adaptation results in Hungary, in-service training of teachers may be improved to suit not only the initial level of ICT skills but also the mindset and personality of teachers.