Boot Camp - Conformity March 7, 2011 SF Bay Area

Boot Camp - Conformity Overview Org Chart <month year> doc.: IEEE 802.15-<doc#> Boot Camp - Conformity Overview Org Chart What does this Working Group do? Previous meetings – Knoxville, San Francisco, DC, Detroit, Fort Lauderdale Terms we use NIST Activities Our Activities Slide 2 Bruce Muschlitz, EnerNex Page 2 <author>, <company>

UCAIug Org Chart (simplified) Technical Oversight Committee IEC61850 Testing CIM OSG Security Comms Conformity Edge Enterprise System SGIP TCC

Boot Camp – Conformity Goals and Activities of the working group <month year> doc.: IEEE 802.15-<doc#> Boot Camp – Conformity Goals and Activities of the working group Coordinate and create Conformity Task Forces based on needs. Edge Conformity Enterprise Conformity Security Conformity Provide overall guidance Propose/Review task force deliverables Slide 4 Bruce Muschlitz, EnerNex Page 4 <author>, <company>

Boot Camp – Conformity Previous Meeting (Knoxville) <month year> doc.: IEEE 802.15-<doc#> Boot Camp – Conformity Previous Meeting (Knoxville) Organized Group Introduced 61850-10 as one models Stressed conformance != interoperabilty Explained abstract vs. detailed tests Introduced “virtual” test environments Slide 5 Bruce Muschlitz, EnerNex Page 5 <author>, <company>

Boot Camp – Conformity Previous Meeting (San Francisco) <month year> doc.: IEEE 802.15-<doc#> Boot Camp – Conformity Previous Meeting (San Francisco) Continued Group Organization Discussed “plug-fest” – won’t do this Discussed how others do this: ISO Guide 65 IEC 17011 and 17025 www.rabnet.com (superseded by ANSI-ASQ Accreditation Board Discussed Product Mark (logo) Recognized: 61850 Testing, SGIP TCC Slide 6 Bruce Muschlitz, EnerNex Page 6 <author>, <company>

Boot Camp – Conformity Previous Meeting (McLean) <month year> doc.: IEEE 802.15-<doc#> Boot Camp – Conformity Previous Meeting (McLean) Organized Security Conformity Re-organized by Horizontal teams Discussed interaction with SGIP TCC Recognized Edge/ENT might work better as (Physical) Device/ (Middleware) Interface Slide 7 Bruce Muschlitz, EnerNex Page 7 <author>, <company>

Boot Camp – Conformity Previous Meeting (Detroit) <month year> doc.: IEEE 802.15-<doc#> Boot Camp – Conformity Previous Meeting (Detroit) NIST TCC IPRM and CPRM coordination Abstract Test Case (ATC) template defined Slide 8 Bruce Muschlitz, EnerNex Page 8 <author>, <company>

Boot Camp – Conformity Previous Meeting (Ft.Lauderdale) <month year> doc.: IEEE 802.15-<doc#> Boot Camp – Conformity Previous Meeting (Ft.Lauderdale) Worked with SGIP SGTCC to complete IPRM (IPRM=Interoperability Process Reference Manual) Agreed to align CPRM with NIST SGIP TCC IPRM Collaborating with EPRI to establish requirements-based testing activity Agreed Security Conformity Group to work collaboratively with SGIP CSWG Slide 9 Bruce Muschlitz, EnerNex Page 9 <author>, <company>

Boot Camp – Conformity Common Terms <month year> doc.: IEEE 802.15-<doc#> Boot Camp – Conformity Common Terms Conformance – meets spec? Interop – plays well with others? Positive tests – does it work right? Negative tests – recovers gracefully? Black Box tests – no inside knowledge White Box tests – view algorithms Slide 10 Bruce Muschlitz, EnerNex Page 10 <author>, <company>

Boot Camp – Conformity NIST SGIP Collaboration <month year> doc.: IEEE 802.15-<doc#> Boot Camp – Conformity NIST SGIP Collaboration SGIP Test and Certification Committee Interoperability Process Reference Manual Published Q4 2010 Promotion during 2011 through WG8 End to End Testing group Work closely with Edge and Enterprise Conformity groups http://collaborate.nist.gov/twiki-sggrid/bin/view/SmartGrid/WebHome Slide 11 Phil Beecher, PG&E Page 11 <author>, <company>

Boot Camp – Conformity Background Material <month year> doc.: IEEE 802.15-<doc#> Boot Camp – Conformity Background Material http://osgug.ucaiug.org http://www.ucaiug.org Slide 12 Bruce Muschlitz, EnerNex Page 12 <author>, <company>

Boot Camp – Conformity Questions? <month year> doc.: IEEE 802.15-<doc#> Boot Camp – Conformity Questions? Slide 13 Phil Beecher, PG&E Page 13 <author>, <company>

Edge / Enterprise Conformity Boot Camp

Edge / Enterprise Conformity Activity Certification Process Reference Manual Test Methodology and Abstract Test Cases 29 December 2018 Conformity Boot Camp 15 15

What is the CPRM? Overview of device and system requirements Identifies best practice for product (device and system) protocol design Describes testing of specific technologies identified by OpenSG membership Describes the process used to define and maintain the quality of a Certification Program Companion document to SGTCC IPRM 29 December 2018 Conformity Boot Camp 16 16

CPRM Guiding Principles Open standards based Clean, layered architecture Robust certification program Focussed on application programming interface, not specific applications Layered conformance testing Performance testing considerations Economically viable 29 December 2018 Conformity Boot Camp 17 17

CPRM Organisational Requirements Identifies organisational structure to support a robust certification and interoperability testing program for products Addresses “devices” (incorporating hardware) and “system applications” (software only) Considerations: Program management Test laboratory qualification Logo management Change control Dispute resolution 29 December 2018 Conformity Boot Camp 18 18

Program Overview 29 December 2018 Conformity Boot Camp 19 19 Interoperability Test and Certification Authority P R O G A M   Certification Program Manager Device / Product Certification Body Lead Laboratory Testing Organizations E Q S / L I C Y SSO / SDO Periodic Normalization Industry Implementers 29 December 2018 Conformity Boot Camp 19 19

SG System Components 29 December 2018 Conformity Boot Camp 20 20

Context of Individual Test Suites 29 December 2018 Conformity Boot Camp 21 21

Abstract Test Documents Contents / methods based on X291 OSI Conformance Testing Methodology and Framework for Protocol Recommendations for ITU-T Applications – Abstract Test Suite Specification Each document summarises requirements for: Test Methodology or Methodologies PICS proformas Test suites Abstract Test Cases Other information? 29 December 2018 Conformity Boot Camp 22 22

General Model for Multi-Party Testing Lower Tester Control Function TCP LT3 UT3 LT2 TCP UT2 LT1 TCP UT1 PCO PCO PCO Y-ASPs IUT (P) PDUs PCO (P) PDUs PCO PCO (P) PDUs X-ASPs X-Service-Provider 7 September 2010 Abstract Test Cases

Current Status CPRM Test Case Documents Completed recirculation ballot Available as companion document to SGTCC IPRM describing organisational and testing requirements to help achieve interoperable products Test Case Documents Enterprise Conformity project underway OpenHAN – working on test methodologies, test harness functionality Need to start OpenADE and OpenADR 29 December 2018 Conformity Boot Camp 24 24

EPRI (CIM) Test Methodology to OpenSG Systems Test Methodology designed to compatible with: IEC CIM 61968 SGSystems (AMI-Ent, OpenHAN, OpenADR, OpenADE, etc.) Test process defined, documented and repeatable Interop test using Virtual Lab Environment 29 December 2018 Conformity Boot Camp 26 26

Agenda for Thursday, March 10th Overview Security Conformance & Charter Overview TCC and CSWG Testing & Certification Subgroup Review Security Testing Methodology

Review Security Conformity TF Charter Establish security conformance requirements for laboratories desiring to certify smart grid components and systems and; Establish clear scoping boundaries, perform research to identify existing models, and propose a high-level philosophy of approach. Chair: Bobby Brown, EnerNex Vice-chair: needed (Sandy Bacik)

Review CSWG Testing & Certification Is NISTIR 7628 Testable / Actionable? Is AMI Security Profile 2.0 Testable / Actionable? SGIP TCC Coordination Tasks Miscellaneous Tasks

Testing & Metrics GAO Report – “no metrics for evaluating cyber security” Open Source Security Testing Methodology Manual (OSSTMM) by Institute for Security and Open Methodologies

OSSTMM Purpose Test conducted thoroughly Test included all necessary channels Posture for test complied with laws and regulations Results are measurable Results are consistent and repeatable Results contain only facts derived from tests themselves?

Security Test Audit Report Serves as proof of a factual test Holds Analyst responsible for test Provides clear result to client Provides comprehensive overview Provides understandable metrics

Outward Support CSWG Testing & Certification Sub-group SG Security CyberSec-Interop