1. 1 QA Focus – Supporting JISC's Digital Library Programmes Quality Assurance For Digital Library Programmes: A Practical Workshop Brian Kelly UK Web Focus UKOLN Email: B.Kelly@ukoln.ac.uk http://www.ukoln.ac.uk/qa-focus/ Workshop Aims Learn about QA for digital library work Hear about QA Focus's activities Identify QA requirements in metadata & service deployment Work on solutions Agree on follow-up actions Workshop Aims Learn about QA for digital library work Hear about QA Focus's activities Identify QA requirements in metadata & service deployment Work on solutions Agree on follow-up actions

2. 2 QA Focus – Supporting JISC's Digital Library Programmes Timetable 14:00Why Do We Need QA? / What Is QA Focus? [BK] 14:20What Can Go Wrong In Your Project? [ALL] 14:50Report back [Group Reporters] 15:15Coffee 15:30The QA Focus Approach [BK] 15:45QA For Metadata – An Example [MG] 16:05Embedding QA in your Project [ALL] 16:30Report Back [Group Reporters] 16:40What Next? (Get some answers and win a prize!) 17:00Finish (?)

3. 3 QA Focus – Supporting JISC's Digital Library Programmes About Us The workshop organised by: QA Focus – a JISC-funded advisory service provided by UKOLN and AHDS which supports JISC's digital library programmes (5/99, and now FAIR and X4L) Brian Kelly, QA Focus project manager (and UK Web Focus) Additional support provided by:  Marieke Guy (UKOLN): formerly QA Focus officer and now SPP and E-Prints UK project manager  Gareth Knight (AHDS): QA Focus officer Note Amanda Closier (UKOLN) will be working on QA Focus from January

4. 4 QA Focus – Supporting JISC's Digital Library Programmes Aims of the Session The aims of the QA session are: To provide participants will an awareness of the importance of quality assurance (QA) in ensuring that project deliverables are (a) functional (b) interoperable (c) widely accessible and (d) re-purposable To find out about QA Focus's resources and gain an understanding of the QA Focus methodology To make use of the QA Focus methodology in an area of your work To have an opportunity to look and evaluate QA Focus resources To agree on areas to be addressed by projects

5. 5 QA Focus – Supporting JISC's Digital Library Programmes Why Do We Need QA? Why Is QA Needed Now? In the past JISC had standards-based philosophy but: Approach of "let a 1,000 flowers bloom" Diversification as no killer application then (it could have been Gopher!) Things are different now: The Web is the killer application There is now a need for interoperability (not just self- contained independent Web sites) JISC Programmes increasingly intended as service development and not 'speculative' research There is a need for publicly-funded services to be seen to provide value for money Introduction

6. 6 QA Focus – Supporting JISC's Digital Library Programmes Technical Areas Addressed The areas addressed by QA Focus are: Standards Digitisation Access (Web) Metadata Software Service Deployment The areas of standards, digitisation and access/Web have been addressed and a wide range of advice is currently available. In this session we will focus on the areas of metadata, software and service deployment The areas of standards, digitisation and access/Web have been addressed and a wide range of advice is currently available. In this session we will focus on the areas of metadata, software and service deployment

7. 7 QA Focus – Supporting JISC's Digital Library Programmes Group Exercise 1 Split into a number of small groups Choose a reporter Address the question "What can go wrong with my project?" The focus should be primarily on technical challenges (i.e. not 'all the staff leave') Address primarily the areas of metadata, software and service deployment E

8. 8 QA Focus – Supporting JISC's Digital Library Programmes Report Back – Exercise 1 What can go wrong with your project? What have you identified?

9. 9 QA Focus – Supporting JISC's Digital Library Programmes Coffee Break We should now break for coffee

10. 10 QA Focus – Supporting JISC's Digital Library Programmes What Can Go Wrong? We have previously identified the following: A lack of understanding of appropriate standards and best practices A failure to make use of appropriate implementation architectures A lack of awareness of failure to comply with appropriate standards and best practices A lack of awareness of checking & validation tools and procedures (and their limitations) Using quality control rather than implementing quality assurance procedures QA Focus is addressing these high level areas

11. 11 QA Focus – Supporting JISC's Digital Library Programmes Approach Taken By QA Focus Two possible approaches to avoiding problems: Enforce Inspect all project’s work Strict auditing, with penalties for no-compliance Encourage Developmental:  Working with community  Explaining reasons for compliance  Documenting examples of best practices  Providing advice on implementation & monitoring  Developing a QA methodology  Seeking to embed QA in working practices Introduction The latter approach is felt to be preferable, especially in a HE context

12. 12 QA Focus – Supporting JISC's Digital Library Programmes QA Focus Activities QA Focus activities: Focus groups identified key concerns (in areas of standards, implementation and service deployment) Surveys profiled approaches, best practices and common problems Provision of focussed advice primarily addressing:  Standards and best practices  Implementation issues  Compliance checking Case studies describing examples of best practices Development of a QA methodology Validation of approach Embedding QA methodology within projects

13. 13 QA Focus – Supporting JISC's Digital Library Programmes Survey Findings Surveys of compliance of 5/99 project entry points helped to profile community and identify common problems and best practices Surveys

14. 14 QA Focus – Supporting JISC's Digital Library Programmes Documentation: Advice Advice Advisory briefing documents are being produced These are: Brief, focussed documents Informed by findings of the surveys Over 50 briefing documents have been published to date Advisory briefing documents are being produced These are: Brief, focussed documents Informed by findings of the surveys Over 50 briefing documents have been published to date

15. 15 QA Focus – Supporting JISC's Digital Library Programmes Documentation: Case Studies Case Studies have been commissioned These are: Written by projects themselves (helping to achieve 'buy-in') Describe the solution adopted to a particular problem Include details of lessons learnt – not just a press release! Over 30 published Case Studies have been commissioned These are: Written by projects themselves (helping to achieve 'buy-in') Describe the solution adopted to a particular problem Include details of lessons learnt – not just a press release! Over 30 published Advice

16. 16 QA Focus – Supporting JISC's Digital Library Programmes Surveys For FAIR & X4L Programmes QA Focus now supports JISC's FAIR and X4L programmes We have developed a self- assessment interface for projects http://www.ukoln.ac.uk/qa-focus/ surveys/fair/web-self-assessment/ http://www.ukoln.ac.uk/qa-focus/ surveys/fair/web-self-assessment/ QA For Web The aim is that projects will (a) check and, if necessary, fix their home page; (b) check/fix other key pages and (c) address underlying causes of problems "I have found the webpage very useful for testing the technical compliance and accessibility of our website" "Many thanks for this - what a really useful tool!" "I have found the webpage very useful for testing the technical compliance and accessibility of our website" "Many thanks for this - what a really useful tool!"

17. 17 QA Focus – Supporting JISC's Digital Library Programmes Putting it Together We have: Explained why projects should implement standards and best practices (they don't always know) Provide case studies on approaches by projects Described implementation architectures Described various checking tools But how do we ensure projects : Check deliverables for themselves Have systematic checking procedures There is a need to describe: Tools & architectures for projects for check compliance with standards & best practices Embed systematic QA procedures But how do we ensure projects : Check deliverables for themselves Have systematic checking procedures There is a need to describe: Tools & architectures for projects for check compliance with standards & best practices Embed systematic QA procedures Tools

18. 18 QA Focus – Supporting JISC's Digital Library Programmes The QA Focus Methodology The QA Focus methodology which we are advising projects to deploy: Based on well-established QA principles Seeks to be lightweight and achievable Seeks to benefit the projects themselves, as well as interoperability benefits to JISC programmes Consists of: Documented policies Procedures for monitoring compliance with policies Audit trails Sharing and collaboration QA Focus Methodology See

19. 19 QA Focus – Supporting JISC's Digital Library Programmes Applying QA – Web Standards Policy:The Web site will use XHTML 1.0 and CSS 2.0 standards Architecture:The Web site will be based on XHTML templates and SSI Monitoring:New and updated pages validated using,validate and,cssvalidate. Every month,rvalidate will be used Exceptions:HTML derived automatically (e.g. Save As HTML in PowerPoint) need not comply with standards. The files will be stored in a standard directory to enable such files to be excluded from checks QA Focus Methodology

20. 20 QA Focus – Supporting JISC's Digital Library Programmes Applying QA – Accessibility (1) Policy:The Web site will strive to attain WAI A guidelines. Consistent accessibility shortcuts will be used. An accessibility policy will be published. Architecture:The Web site will be based on XHTML templates which comply with WAI A. Monitoring:New and updated pages will be validated using,bobby. A monthly batch checker will be used and audit reports published (to enable any trends to be spotted). Exceptions:A list of permitted exceptions will be provided. QA Focus Methodology Note that University of Edinburgh Library have a useful Web accessibility policy

21. 21 QA Focus – Supporting JISC's Digital Library Programmes Applying QA – Accessibility (2) Policy:The organisation has no accessibility policy Authors are free to implement their own accessibility shortcuts (if at all) Architecture:No centralised policy covering authoring tools or architecture will be provided Monitoring:No monitoring will be carried out If you don't have a written policy, the unwritten policy may well be frightening!

22. 22 QA Focus – Supporting JISC's Digital Library Programmes Self Assessment Toolkit Further Deliverables We are developing a self-assessment toolkit for projects to use, by individual projects or across project clusters The toolkit consists of: Self-assessment questionnaires Examples of QA procedures Documented examples of use of testing tools Advice on standards and best practices Case studies FAQs … Next Steps

23. 23 QA Focus – Supporting JISC's Digital Library Programmes Self Assessment Toolkit (1) Next Steps

24. 24 QA Focus – Supporting JISC's Digital Library Programmes Self Assessment Toolkit (2) The toolkit is also available in paper format, suitable for use in workshops, meetings, etc. This approach will be used in today's QA workshop Toolkits on Metadata, Software and Service Deployment will be used

25. 25 QA Focus – Supporting JISC's Digital Library Programmes QA For Metadata – An Example Presentation By Marieke Guy

26. 26 QA Focus – Supporting JISC's Digital Library Programmes Group Exercise 2 Embedding Quality Assurance Into Your Project Work In your groups use the QA framework to define: Appropriate policies Appropriate checking procedures Audit trails In the areas of metadata, software and service deployment You may use the metadata, software and service deployment toolkits.

27. 27 QA Focus – Supporting JISC's Digital Library Programmes Report Back – Exercise 2 Please provide a brief report back on: Any significant issues which arose How you envisage addressing the issues covered in the exercise? What additional support you may need

28. 28 QA Focus – Supporting JISC's Digital Library Programmes Comments On Exercises Your project deliverables may not be interoperable because: You were unclear of the purpose of the metadata / software or there were differing views across project partners You chose inappropriate standards for your purpose or your level or expertise, funding, timescales, etc. Your modelling of your metadata (or database) meant that it was not interoperable with others You chose an inappropriate implementation architecture You failed to implement appropriate cataloguing, etc. rules You had cataloguing rules, but no processes for spotting problems, deviations, ambiguities, etc. You haven't allocated sufficient resources to implement the above You will need to implement QA procedures to address such issues

29. 29 QA Focus – Supporting JISC's Digital Library Programmes Hype QA implemented Thoughts On Timeliness of QA Early adopters Despondency Realism & service deployment Maturity Software Digitisation Web Metadata Web Services Semantic Web Maturity Software Digitisation Web Metadata Web Services Semantic Web Note that QA for metadata is at an early stage. We are currently still developing and evaluating the potential for technologies such as OAI, IMS, …

30. 30 QA Focus – Supporting JISC's Digital Library Programmes Next Steps – Metadata QA for metadata is at early stages (little documented work found). Thoughts on QA for metadata: SyntaxCheck syntax on embedded DC Cataloguing RulesEnsure projects have appropriate cataloguing rules InteroperabilityEnsure metadata can interoperate with third parties Fitness for purposeEnsure metadata is fit for its purpose Change controlEnsure architecture for managing metadata can cope with change control (cf. Exploit Case Study) Next Steps See and

31. 31 QA Focus – Supporting JISC's Digital Library Programmes Next Steps – Software QA for software development is mature, but there may be a need to engage in culture change within the sector Thoughts on QA for software: SpecificationProcess for developing systems spec, agreeing spec with stakeholders, etc. MethodologyDocumentation of systems development methodology (cf UML case study) TestingDocumentation of testing methodology DeploymentAwareness of deployment challenges Next Steps See for example

32. 32 QA Focus – Supporting JISC's Digital Library Programmes Next Steps – Service Deployment QA for service deployment is important to protect investment in project funding (but there is a lack of awareness in some cases). Development ArchitectureEarly public documentation of development architecture Service ArchitectureAwareness of likely service architecture and liaison with service SecurityAwareness of security concerns LicensingLicensing issues Scenarios Software developed using cool open software environment – but JISC Service has no knowledge of environment. Resources digitised, but copyright clearance not documented. Service refuses to take delivery due to liability concerns. Scenarios Software developed using cool open software environment – but JISC Service has no knowledge of environment. Resources digitised, but copyright clearance not documented. Service refuses to take delivery due to liability concerns. Next Steps

33. 33 QA Focus – Supporting JISC's Digital Library Programmes Approaches For Web Examples of areas which may be considered for QA for Web is illustrated. We intend to provide such advice on solutions for metadata, software an service deployment once we know the answers and have obtained case studies. Web Toolkit The Web toolkit seeks to address quality assurance issues for the provision of a Web site. 1. Purpose Of Your Web Site Have you identified the purpose of your Web site? There can be a number of roles for your project Web site: information about the project; for use by project partners; access to project deliverables, etc. You may wish to take different approaches depending on the different roles (e.g. use of standards; access conditions; preservation policy, etc.) For further information see: briefing-15, 2. Standards For Your Web Site Have you chosen the standards to be used on your Web site?? Use of open standards such as HTML and CSS are important in order to maximise access to resources, ensure resources function correctly and to facilitate reuse of your Web site. For further information see: briefing-01, briefing-35, … case-study-02, case-study-05, … Web Toolkit The Web toolkit seeks to address quality assurance issues for the provision of a Web site. 1. Purpose Of Your Web Site Have you identified the purpose of your Web site? There can be a number of roles for your project Web site: information about the project; for use by project partners; access to project deliverables, etc. You may wish to take different approaches depending on the different roles (e.g. use of standards; access conditions; preservation policy, etc.) For further information see: briefing-15, 2. Standards For Your Web Site Have you chosen the standards to be used on your Web site?? Use of open standards such as HTML and CSS are important in order to maximise access to resources, ensure resources function correctly and to facilitate reuse of your Web site. For further information see: briefing-01, briefing-35, … case-study-02, case-study-05, …

34. 34 QA Focus – Supporting JISC's Digital Library Programmes What Next? (1) QA Focus materials: Available for you to use Feedback needed Book token will be provided for 2 people providing best feedback (see note) QA for metadata, software and service deployment Currently under development Your input today is valuable Why not contribute a case study?

35. 35 QA Focus – Supporting JISC's Digital Library Programmes What Next? (2) Embedding QA In Your Work There is a need for QA to ensure interoperability and protect JISC's investment QA Focus have recommended a self-assessment approach, rather than external checkers This is reliant on projects implementing appropriate QA procedures You will need to consider how to embed QA into your project work Please complete the form in which you'll list your action plans This is for your personal use. Please complete the form in which you'll list your action plans This is for your personal use.

36. 36 QA Focus – Supporting JISC's Digital Library Programmes What Next? (3) Future Programmes QA Focus is recommending that for future JISC programmes: Programme calls require projects to indicate the approaches they will take to QA Initial reports to JISC should describe in more detail the projects:  QA methodology  Criteria for standards  Intended service deployment environment Your current project can provide a valuable opportunity to gain experiences in these areas

37. 37 QA Focus – Supporting JISC's Digital Library Programmes Conclusions We have described: The approaches to QA which have been taken by the QA Focus project The resources which have been developed for the community by QA Focus A systematic methodology for QA You have: Addressed the QA requirements for your project Started to employ the QA methodology Identified areas for you to address Identified areas in which you can contribute to the community Time for a drink!