Lead from the front Texas Nodal 1 EDS 3 Release 5: SCED Phase 1 Testing Aug 14, 2007

Lead from the front Texas Nodal 2 Terminology: Black Box vs. White Box Testing Black box testing takes an external perspective of the test system to derive test cases. These tests can be functional or non-functional, though usually functional. The test designer selects valid and invalid input and determines the correct output. There is no knowledge of the test object's internal structure.  “Simulate Market Participant interaction” White box testing uses an internal perspective of the system to design test cases based on internal structure. It requires programming skills to identify all paths through the software. The tester chooses test case inputs to exercise paths through the code and determines the appropriate outputs.  “Validate Integration is working correctly”

Lead from the front Texas Nodal 3 MMS EDS Phase 1 Release Testing The EDS team will use a Black Box approach to test the EDS system simulating the Market Participant interaction. The set of valid and invalid test cases that will be executed during the phase 1 Support Window will be run simulating 3 QSEs to ensure system stability. Any critical functionality issues that would negatively affect phase 1 progress will be fixed prior to phase 1 while non-critical functionality issues will be prioritized along with internal testing variances.  The goal is to complete testing and critical issue fixes prior to 8/24. EWS Notification UI Validation MP

Lead from the front Texas Nodal 4 ERCOT Internal Functional and Integration Testing The INT team will use a White Box testing approach to perform functional testing against the External Web Services infrastructure. Internal checkpoints and database validation checkpoints In addition, the team will test the Integration of the Web Services with the SCED (MMS) system. Test submission and validation rules through EWS This testing will use valid test cases as well as extensive set of invalid test cases to test the validation rules. MMS EWS UI Validation Engine MP Notification Transformation

Lead from the front Texas Nodal 5 EDS Readiness Test Cases Examples Positive Test Cases for EDS 3 Submission Items (3PO, COP, OS, Inc/Dec) –Submit, Query, Update, Cancel* –Some complexity (multiple: resources, curves, intervals, statuses, limits, capacity)* –Validate synchronous response and asynchronous notification –Validate through MMS User Interface Negative Test Cases –Validate QSEs can only view and submit on their own resources –Create XML submissions to fail key validation rules (valid resource, limits, status, required items, invalid dates, …) –Validate synchronous response and asynchronous notification provide error messages * Some variations only apply to certain submission items

Lead from the front Texas Nodal 6 Internal Functional and Integration Testing Examples Business Test Cases –Positive Path Testing – white box (approx 20%) Same submission items for EDS 3 Extensive complexity variations (multiple: resources, curves, intervals, statuses, limits, capacity) Extensive variations of validation rules and optional fields Over-writing behavior Validate Notification Service –Negative Path Testing for each submission item (approx 80%) Invalid data input types (market, trading day, QSE, resource name, ownership, start date/time, end date/time, date variations, expiration date/time) Rejection due to overlapping offers, submission time, no energy offer curve, missing elements, … Invalid energy offer curve components (above/below cap, non-monotonically increasing, resource specific floor/cap violation) Invalid resource (not combined cycle, RMR) Review asynchronous error notifications

Lead from the front Texas Nodal 7 Internal Functional and Integration Testing Examples Technical Test Cases –EWS Security Testing Invalid: source, User Id, DUNS number Test: digital signatures, certificate, certificate revocation, timestamp and nonce Replay detection (valid / invalid / null nonce, created time) –EWS Transformation Testing Validate each submission item (submit, update, query, cancel*) Validate database entries –EWS XSD validation Validate optional fields –General Payload compression, format tag, verb / noun, mRID usage for all submission items

Lead from the front Texas Nodal 8 Internal Functional and Integration Testing Examples Technical Test Cases (cont.) –MMS input data validation Validate each submission item (submit, update, query, cancel*) Validate optional fields –Notification Service Data Load Failure (staging & destination tables) Offer and Bid Set Acceptance / Errors –System Status Web Service Web Service valid, error –Get mRID Web Service Web Service valid, error mRID usage for all submission items

Lead from the front Texas Nodal 9 Estimated Number of Test Cases & Scripts Business Test Cases –Test Cases Approx 320 –Test Scripts Approx 700 Technical Test Cases –Test Cases Approx 120 –Test Scripts Approx 250 Approximately 20% are planned to be executed prior to EDS 3 kickoff on 8/24

Lead from the front Texas Nodal 10 Questions ?