1. UCA International Users Group The Quality Assurance Program prepared by the UCA ® International Users Group Testing Subcommittee UPDATED: 28 July 2008 Jack Robinson: UCA International Users Group Secretary Bruce Muschlitz: Chair Testing Subcommittee Marco C. Janssen: editor of the QAP

2. International Users Group Mission Advancing interoperability for the utility enterprise by providing a global forum in which stakeholders can work cooperatively as members of a common organization to support international standards: Educational Activities Improvement Process for International Standards Testing and Certification Promotional Activities

3. International Users Group Charter Not-for-Profit International Consortium Utilities and Vendors working together in support of international standards Over 109 Members From 26 Countries: IEC TC 57 Members (Editors and Conveners of IEC 61850, CIM and OpenAMI/DR) Vendors of Control and Substation Automation Products Utility Buyers of 61850, CIM and OpenAMI/DR Products Testers and Certification Providers Consultants, Universities, Research Facilities

4. Broad Technical Focus Utility enterprises are complex and cross many functional boundaries Promoting interoperability in the utility enterprise requires a similarly broad focus UCA IUG uniquely offers members an organization to promote standardization across several functional areas

5. UCA IUG – Technical Focus Communications –IEC 61850 – Substation Automation –IEC 61850 based extensions for wind, hydro, distributed generation, etc. Application Integration –Common Information Model (CIM) IEC 61970 and IEC 61968 –Generic Interface Definition (GID) IEC 61970 Metering and Demand Response –OpenAMI – Information/data models, reference designs and interoperability guidelines based on IntelliGrid and IEC standards. ONE MEMBERSHIP

6. The Users Group and Testing The Users Group strongly supports testing Focus for testing is now on devices and systems developed in conformance with IEC Standards Users Group works closely with the IEC for problem resolution and tracking (e.g. WG 10 members are all in the IEC 61850 Technical Issues Users Group Subcommittee) Testing procedures, requirements and guidelines: –Quality Assurance of performed activities –Traceability of results

7. Testing Subcommittee IEC 61850 Test Procedures WG OpenDR Test Procedures WG Testing TSC Org Chart CIM Test Procedures WG

8. AEP A. Clark EPRI J.Hughes Co-Chair EnerNex B.Muschlitz Co-Chair KCA J.Robinson Secretary Triangle/ MicroWorks G.Schimmel SISCO H.Falk CIM Member DR Member Testing Subcommittee Testing SC

9. Committee / Working Group Editors/ Representatives Testing Quality Assurance Program Editor: Marco Janssen (UTInnovation) CIM Document Editors Tester Accreditation Task Force (unique to each area) IEC 61850 Document Editors: –Tester System Accreditation: Willem Strabbing (KEMA) –Server Product Test Procedures: Richard Schimmel (KEMA) –Client Product Test Procedures: Richard Schimmel (KEMA)

10. Testing Web Site Support Testing Subcommittee / WG Areas (Meeting Agenda’s / Minutes, Draft Documents, Issues Discussion, Action Items, Tasks, Events) Member Only Testing Documents (Corporate Access) –Quality Assurance Program and Addendums –Product Test Procedures –Tester Accreditation Procedures Public Documents –QAP Overview/ Testing White Papers –Tester Certification (IEC 61850) –Testing Completed (IEC 61850) (Posted Certificates)

11. The Users Group and Quality Assurance for Testing A Quality Assurance Program (QAP) for IEC Product Testing and Test System Accreditation and Recognition is a major benefit to the members and is expected to save time and money for participants The QAP allows for the continuous improvement of test centers, test systems and the international standards The Users Group QAP identifies the means to quickly resolve IEC specification problems and, through IEC coordination, provides the vehicle for later update of IEC standards following IEC charter rules

12. Concepts of the QAP Feedback is essential –To keep all informed –To make sure standards are correct and well understood –To quickly resolve problems uncovered during testing Process is decentralized –All activities relating to testing are performed individually by parties that may be other than the UCA ® International Users Group –UCA ® International Users Group plays a major role in coordinating and managing information dissemination The process provides value for all who participate, and a strategy to bring all stakeholders together (users, vendors, testers, consultants).

13. QAP - The Development Process IEC TC57 Working Groups Product Development Test System Development Test Labs Projects Test SystemsTested Products Success Specifications Test Cases Products

14. QAP - The Quality Assurance Process IEC TC57 Working Groups Vendor Manufacturer System Integrator Test System Developers Test Labs Projects (Users) All Parties Need Feedback Difficult to Coordinate QAP Realistic Coordination

15. Standards Improvement Technical Issues (TISSUES) TISSUES Initiator emails 1st Proposal “green” TISSUES “yellow” TISSUES Final Proposal 100% approval Resp. GoE member will create 1st proposal & distribute to GoE After discussion a final proposal is distributed to TISSUES group for vote. Selected and distributed to the responsible member of Group of Experts (GoE) Could be a short term fix IEC TC57 WG10

16. The Contents of the QAP Roles of: –Standards organization –Product development –Test cases, test system development, test systems and test laboratories –Projects Rules, requirements and procedures for exchange of information between them –Input from and to: Vendors, manufacturers and system integrators for product development Test laboratories Vendors, manufacturers and system integrators for projects Users for projects

17. The contents of the QAP (cont'd) QAP Maintenance Problem reporting and handling –Standard problem report –Formal procedure for initial evaluation, proposals for resolution and resolution and formal reporting of solutions and determine which developers are involved –Short term recommendation to user(s) and vendor(s) on how to proceed –Process to upgrade the documents and/or test systems associated with the problem Mechanism for identification, registration and publication of interoperability problems and the resolutions within various revisions/errata of IEC standards as well as specific revisions of the test systems and lab certified devices

18. The contents of the QAP (cont'd) Organization and infrastructure –Users Group Executive and Technical Committees –Audits Authority to organize and/or perform audits at testing organizations participating in the UCA ® International Users Group QAP To verify the set up, implementation and use of the required quality system. Periodically or upon request from a UCA ® International Users Group member with notice and mutually agreeable time schedule and agenda. –Confidentiality of documents and information In general testing results owned and controlled by testers and vendors Results reported back to UCA ® International Users Group and are considered public information for dissemination to Members Field problems carry the level of confidentiality as designated by the reporting company

19. The contents of the QAP (cont'd) Organization and infrastructure –Information sharing Amongst UCA ® International Users Group Members in accordance with the confidentiality guidelines Primary sharing vehicle is the Users Group web site under access and password control Public areas available to any one who visits the web site * Confidential areas open to Users Group Members Private Forum open to committee or working group members * = www.ucaiug.mobi

20. The contents of the QAP (cont'd) Procedures, requirements and guidelines for the realization of tests including definitions for:  test configurations  conformance test procedures / steps / results  the conformance test process  standard test procedure groups  IEC 61850 conformance blocks  test case list  test report  test procedures  control of testing organizations and systems  planning of tests  test definitions and set up  formal and informal testing  the use of test tools  re-testing  self testing  procurement steps  test architecture / system configuration  test architecture for conformance  what is to be conformance tested?

21. The contents of the QAP (cont'd) Test Procedures –Detailed test procedures for IEC 61850 Device Testing Used by testing organizations Maintained by the UCA ® International Users Group Contains Abstract Test Cases, based on IEC 61850 Part 10 –Certification done by test centers, users and test vendors (No direct certification of devices by Users Group) –Any qualified facility may issue a “Product Certificate” –Users Group maintains certified devices list on the web site –Users / vendors that submitted the device for test own the detailed test report / results

22. The contents of the QAP (cont'd) Control of Conformance and Interoperability –Starts with accreditation and recognition of testing organizations –No minimum conditions IEC 61850 protocol functionality to be supported in order to get a certificate –Certificate remains valid as long as the owner follows the Users Group guidelines specified in the QAP –Users Group maintains record of all tested and all tested and certified products and systems including the references to the documents and their respective versions that the tests performed were based upon –Arbitrator role for the Testing Committee

23. The contents of the QAP (cont'd) Resolution of Problems resulting from Testing –Testing must be documented, repeatable, and results comparable across tested equipment / applications –QAP provides process for problem tracking, analysis, and resolution of tests, standards, and implementation agreements –Users Group assists testing organizations in Maintaining problem reports Resolving specification conflicts Modifying the standards as problems are uncovered Maintaining lists of certified products Ensuring the testing facilities keep up their qualifications per approved test procedures

24. The contents of the QAP (cont'd) Resolution of Problems resulting from Testing –Access to QAP and problem information, along with key technical detail for users, product developers and integrators –Status evaluation of own devices against a ‘non-aligned’ technical resource that presents an unbiased interpretation of the problems and possible resolutions –Participation in the problem feedback process by Approved Testers –Information on problems uncovered in the field by vendors who receive device certificates from approved testers –Vendors and testers shall include in their users and quality assurance manuals, a problem identification, resolution and reporting process that includes the UCA ® International Users Group feedback step

25. The contents of the QAP (cont'd) Improvement Process –In accordance with the ISO 9000 series of standards, the UCA ® International Users Group strives for continuous process improvement –The community including members and non-members of the UCA ® International Users Group are invited to use the mechanisms for providing feedback –In accordance with the procedures laid down in the QAP the UCA ® International Users Group will facilitate improvements.

26. Other Available Testing Documents Test Center Accreditation and Recognition Procedure for IEC 61850 Device Testing (Test System Accreditation) Conformance test for IEC 61850 Servers (Device Test Procedures) Testing of CIM Products (Draft by the CIM group) CIM Compliance Document/ Conformance Testing Requirements Conformance test for IEC 61850 Clients

27. Test System Accreditation Describes the procedure to qualify test centers Conditions that test centers should comply to in order to certify IEC 61850 based products and attach an Users Group label to the tested product To Date: There are 3 Users Group approved Test Centers for IEC 61850 Products: KEMA, Nari-Relays and ABB Tested device vendors (receiving certificates) participate in the Users Group Quality Assurance Program

28. Device Test Procedures Conformance test procedures and results in general and in detail (IEC 61850 Example)

29. Testing Milestones AEP/UG Testing Workshop (May 05: Demo of Testing Capability, Tools, and Rollout of the QAP) QAP Version 2.6 (March 2007) –Update for General Quality Assurance Program –Addendum for IEC 61850 (June 2008) –Addendum for CIM and OpenAMI/DR (in preparation) Tester Certification IEC 61850 –KEMA –ABB (Recognized) –Nari-Relays (Recognized) Device Testing IEC 61850 Server Procedures Rev 2.2 (October 2007) Testing Completed (IEC 61850) for 12 Vendors and 79 Products (posted on Web Site)

30. Testing Current Plans QAP Version 2.X –Addendum for CIM –Addendum for OpenAMI/DR Tester Certification IEC 6185 Device Testing IEC 61850 Server Procedures Use Rev 2.2 (after October 2008) Device Testing IEC 61850 Client Rev 1.0 (by early 2008) Future Documents: Product Mark, Implementation Agreements, Testing Specifications, Test Procedures for CIM and OpenAMI/DR

31. Bottom-Line Benefits Test and certification activities that lower costs for users Provide input to international standards activities at no additional cost and no travel Members only access to: –Resolution of technical issues related to standards –Industry experts via a global help desk –Technical documents and best practices from existing users –On-line discussion forums Discounts on paid seminars and exhibitions

32. Conclusions The available documents provide comprehensive means to assure the quality of the testing process Testing will be well documented, repeatable, and results shall be comparable across tested equipment/applications. The QAP provides an appropriate process through which to perform problem tracking, analysis, and resolution of tests, standards, and implementation agreements The UCA ® International Users Group provides a vehicle for deployment and testing of high-quality products based on open standards for the Electric Utility Industry

33. To Join The UCA International Users Group Go To: www.ucaiug.mobi or E-mail kay@ucausersgroup.org or call (919) 847-2241www.ucaiug.mobi kay@ucausersgroup.org For Public Documents and Information: www.ucaiug.mobi or E-mail bruce@ucausersgroup.org or jack@ucausersgroup.org www.ucaiug.mobi bruce@ucausersgroup.org jack@ucausersgroup.org