1. Quality Assurance Strategies: Evaluating Learning Resources Presented by: Karin Lundgren-Cayrol In collaboration with: Gilbert Paquette and Suzanne Lapointe for CICE & GTN-Normétic for CICE & GTN-Normétic I2LOR WORKSHOP ON REPOSITORIES Montreal, November 8 2006

2. OverviewOverview Goal and Objectives Basic Concepts –Learning Ressources (LR) –LR Repository Types –Quality Assurance Strategy –Quality Dimensions Integration/Development Processes LR Life-Cycle and Quality Assurance Strategies Conclusion

3. Goals and Objectives Provide a set of Quality Assurance Strategies to ensure LOR sustainability Objectives Establish quality assurance strategies Provide pertinent evaluation criteria Provide guidelines and principles Repository Managers & Developers LO designers LO users

4. What’s a Learning Resource ? Video Learning Design CD- ROM Software Electronic book SCORM Object Image Any entity, digital or non-digital, which is intended to be, or may be used for the purpose of learning, education or training Granularity

5. Quality Assurance Strategy? List of Stakeholders Repository Type User Profile List of Needs Metadata Profile Integrating a Repository Organisational and Technical Decisions Identify Quality Assurance Strategies 1.Design and Produce 2.Integrate into LR 3.Reuse and Revise 1.Design and Produce 2.Integrate into LR 3.Reuse and Revise ISO 9000: 2000 Quality Assurance is the planned and systematic activities put in place to ensure quality requirements for a product or service will be fulfilled. Document all processes.

6. Quality Dimensions Instructional QualityInstructional Quality Content clarity and conciseness, instructional strategies aligned to the learning objectives, appropriate media according to target audience, etc… Media and Ergonomical QualityMedia and Ergonomical Quality User-friendliness, motivating, visually attractive, built-in accessibility features, etc.. Technical QualityTechnical Quality Technical interoperability and robustness, metadata schema and tagging procedures, conformance to standards

7. Process preceeding the integration/development of a Learning Object Repository The LR Life-cycle and how to identify Quality Assurance Strategies

8. Strategies at the Organisational Level Integrating a LR Stakeholder support Institutional Support Technological Infrastructure Policy on copyrights Define Goal of LR IdentifyActors Specify Metadata schema IdentifyNeeds Specify Resource Types Goal Stakehol der list Resource Types & Expertise LISTof User Profile Repository Type Repository Type Stakeholders Stakeholders User Profile User Profile List of Needs List of Needs List of Resource Types List of Resource Types Metadata Profile Metadata Profile

9. LR Life-Cycle and Identifying Quality Strategies Design/Production Deposit and metadata referencing Adaptation/Reuse/Assignment Life-cycle of a LR Before inclusion During contribution After inclusion Pedagogical/MediaQuality Technical Quality All dimensions REVISION

10. LR Repository Types and Main Charactersitics Private RepositoriesPrivate Repositories –Quality = personal usefulness of the LO –Metadata schema –Recommendation systems (social/collaborative tagging/folksomy) Community repositoryCommunity repository –Quality = Member ship and peer reviewed LO’s –Domain specific interest, practice or discipline –Protects the consumer trough membership –May use domain ontology Public RepositoryPublic Repository –Quality = Annotation and Right’s Management –Metadata validation system and automatic capturing –Propose evaluation criteria for different resources –Provide evaluation, recommendation and annotation systems

11. SummarySummary Graphical designer: Julie Aubin, 2006

12. Too Much is Overkill Too Little Just Kills Karin Lundgren-Cayrol Suzanne Lapointe Gilbert Paquette CICE, TÉLUQ - UQAM Give me a chance!

13. During Contribution Strategies here could be referring to all types of quality dimensions: -Pedagogical and Ergonomical Quality by enforcing Peer Review -Controlled contribution But if Strategies has been applied at the design/production stage, strategies here normally only need to refer to -The quality of the metadata -Interoperability of the Learning Resource and its Metadata Record

14. During Contribution Multi-actor expertise for metadata –Peer Review Strategies: Involving Content Expert / Author /ID Expert Title, LR language, key words, description, type, version, contributors, intended end user, learning context Text mining algorithms can help –Metadata Quality Strategies Library technician Version, classification, rights, relations metadata Overall respect of the standard used –Interoperability Computer Technicians Format, size, required conditions, Installation Remarks, meta-metadata, record language General Principle: Reduce form-filling: use wizards, smart automatic or semi-automatic computer agents

15. During Contribution Demand Membership for contributors: –Gives responsability and motivation; –Make contributors visible by ‘Business Card’ portfolio, chat groups etc. –Suggest domain specific Community of Practices Make sure that the author provides the following infos: Rights –Degree of Pedagogical Reusability (Rights) –Content quality indications (Method,checklist, evaluation used etc) –Interface quality (idem) –How it might be an efficient learning or teaching tool

16. Learning Design as Agregated Objects Basic Resources Tools Documents Media Elements Actors Units of Learning Operations Scenarios Processes

17. Completeness or Usability

18. Before Inclusion Strategies that refers to Instructional Design of the Learning Resource and aims at: -Pedagogical Quality -Ergonomical Quality Demands specilized ID methods if LR is a Unit of Learning compliant with : -IMS LD -SCORM -Other like WEBCT, BlackBoard etc..

19. Some Guidance Principles Design and Production Strategies –Use a solid and adapted ID method –Support Collaborative design, that is let specialists apply their expertise –Clearly identify knowledge and user competencies –Favor pedagogical strategies putting the learner in the center –Apply evaluation criteria during development and implement at least one learner/peer evaluation cycle –Be informed about Access4All production principles

20. Instuctional Engineering Method Knowledge Modeling 210 Knowledge modeling principles 212 Knowledge model 214 Target competencies 310 Learning units content 410 Learning instruments content 610 Knowledge and competency management Instructional Modeling 220 Instructional principles 222 Learning events network 224 Learning units properties 320 Instructional scenarios 322 Learning activities properties 420 Learning instruments properties 620 Actors and group management Materials Modeling 230 Media principles 330 Development infrastructure 430 Learning materials list 432 Learning materials models 434 Media elements 436 Source documents 630 Learning system / resource management Delivery Modeling 240 Delivery principles 242 Cost-benefit analysis 340 Delivery planning 440 Delivery models 442 Actors and user’s materials 444 Tools and telecommunication 446 Services and delivery locations 540 Assessment planning 640 Maintenance / quality management Problem definition 100 Training system 102 Training objectives 104 Target Learners 106 Actual situation 108 Reference documents MISA

21. Content Expert Instructional Designer Media Specialist Delivery Specialist Project Leader Collaborative Design Note: Interactive Objets are Software and answers to software quality criteria Basic principle: Build or integrate objects that you can quality certify

22. The LORI Evaluation Criteria LORI The LORI Evaluation Criteria LORI Content Quality Veracity, accuracy, balanced presentation of ideas, and appropriate level of detail Goal Alignment Alignment among learning goals, activities, assessments, and learner characteristics Feedback and Adaptability Adaptive content or feedback driven by differential learner input or learner modeling Motivation Ability to motivate, and stimulate the interest or curiosity of, an identified population of learners Visual Design Design of visual and auditory information for enhanced learning and efficient mental processing Usability Ease of navigation, predictability of the user interface, and the quality of UI help features Accessibility Support for learners with disabilities Reusability Ability to port between different courses or learning contexts without modification Standard conformance Adherence to international standards and specifications Nesbit, Belfer & Vargo, 2003

23. LORI instrument Adding: General quality or the rate (X/45) or % ? Other criteria ?

24. Accessibility4AllAccessibility4All The TILE PROJECT : http://barrierfree.ca/tile/index.htmhttp://barrierfree.ca/tile/index.htm Checklist of Checkpoints for Web Content Accessibility Guidelines 1.0

25. After Inclusion in the Repository Refers to strategies that provide: Retrieval quality –Free text search –Browsing by Classification Schemes or Typologies –Ontological search –Federated search Easy display of information –Descriptive resumé –Full meta data record –How many used it –What do they think Peer reviews and evaluations from actual users/reusers. –Simple Annotation systems –Social tagging/bookmarking and Folksomy –Collaborative filtering techniques –Recommendation algorithms

26. After Inclusion Strategies List of new and innovative high quality LRs Suggestions for: –Revisions to authors/designers for improvements –Renewal of resource –New resource

27. MERLOTMERLOT

28. Metadata display

29. PALOMA WEB Federated search displays Source Repository

30. EDNA Online

31. SEARCH ARIADNE DISPLAY

32. FORMIST/deLicio.usFORMIST/deLicio.us Social Tagging Information

33. SMETESMETE Technical information Rights information Reviews information