Alberta Reliability Standards Compliance Working Group Meeting Minutes are noted on the summary slide for each section June 19, 2012

Agenda Topics 1.Introduction 2.Presentation of discussion plan, method and scope 3.Presentation on compliance monitoring audit process issues 4.Initial discussion on solutions 5.Meeting debrief

Plan Review feedback and observations obtained from previous year’s audit experience Provide input on possible solutions to identified process issues Timelines 1st meeting (today) CWG review issues Gather input from CWG 2 nd Meeting – DRAFT (end July or early August) AESO presents solutions based on first meeting input CWG refines solutions 3 rd Meeting – FINAL (End August or early September) AESO presents updated solutions as per 2 rd meeting refinements 4 th meeting if required (End September or early October) Publish guidelines – one month following final meeting

Method, Scope Method Go over the issues in order of the compliance monitoring audit process sections (guideline process map) As a group: discuss each issue focusing on: impact, broadness and provide input on possible solutions Feedback will be used by the AESO to update the audit guideline Scope Scope is limited to: The compliance monitoring audit process Existing, clearly defined issues. Existing constraints (legislation and mandate) Issues outside of the scope will be tabled and brought forward at a later date or other forum.

Process sections

Identified Compliance monitoring audit issues (13) (numbers 7.x refer to the audit guideline sections) Process SectionIssues GEN Information / Guidelines Publish audit schedule Changes to audit schedule Workshop /Notification 3 7.4Pre-audit conference0 7.5 Gather audit evidence Submit audit evidence 2 Process SectionIssues 7.7 Review for completeness Assess audit evidence 0 7.9Issue draft report Post-audit conference Review draft audit report Issue final audit report Referral to MSA and retain evidence 0

Defined Issues General: Information/Guidelines 1.Applicability Assessments Participants/CMP -Process change to reflect updated applicability sections -Complete 2.Acknowledgement and Consent Participants/CMP -Simplification to meet business need -End August 3.Guideline overly complexParticipants -Need to define specific issues -CWG Input

Defined Issues Audit Guideline - 7.2: Change to audit schedule 4.Request for deadline extensionsCMP -Requests for deadline extensions were unreasonable -CWG Review (from December 2011 CWG)

Defined Issues Audit Guideline - 7.3: Workshop / Notification 5.Lessons learned and timing Participants -When can lessons learned be shared? -CWG Input 6.Publish generic index templateParticipants -Can a generic index be published in advance? -Related to #9 7.Asset identificationParticipants/CMP -Need to identify an asset list prior to audit -CWG Input

Defined Issues Audit Guideline - 7.6: Submit audit evidence 8.Bad index of evidence files CMP -Difficulties in finding evidence -Created standard index template (Attachment 4) -Related to #9 (from December 2011 CWG) 9.Audit index VS RSAWsParticipants/CMP -What method can be used for indexing evidence? -CWG Input

Defined Issues Audit Guideline - 7.7: Review for completeness 10.Initial review too detailedCMP -Some assessment was done during completeness review -CWG Review (From December 2011 CWG) 11.How can we help entities better manage information requests? Participants/CMP -Should we proposed tentative timelines at this stage? -CWG Input 12.Additional request content is sometimes unclearParticipants -Requests are not always clear resulting in multiple follow-up -CWG Input

Defined Issues Audit Guideline : Review draft audit report 13.Re-issue of draft reportCMP -Extent of commenting on re-issued draft report is unclear -CWG Input

Additional issue brought forward by a participant NEW: Difficulties with overlap of self-certification and audit -What are the consequences of self-certifying compliance if there is a subsequent draft audit report showing a suspected contravention? -Discussion was held at the meeting with follow-up at the next.

Discussion on issues WORKSHEETS Each worksheet will show: 1.Title 2.Background 3.Current status 4.Possible constraints After a quick review of the issue we will begin discussions: 1.Gather thoughts on impact and broadness 2.Discuss possible solutions

Discussion on issues SUMMARY At the end of the discussion, we will: 1.Write a summary of the main points for all to see 2.Validate that the summary contains all pertinent discussions NOTE: It is important that we validate the summary as these points will be used to create the solutions that will be presented at the 2 nd meeting.

General 3 Guideline overly complex Background During an end of compliance monitoring audit survey a market participant stated that the guideline was not used because it is overly complex. Current status “Overly complex” is unclear and we need input Possible Constraints The purpose of the guideline is to define the procedures for planning, executing, and reporting with respect to each compliance monitoring audit. It has the dual purpose of providing policy/rules and procedures (“how to”). During a compliance monitoring audit one is more likely to simply look for the procedure which is currently mixed with the policy/rules. It is possible to change the guideline however, both elements must remain.

3 Audit Guideline overly complex Our thoughts The guideline(s) could be changed as follows: 1.Separate policy/rules from the “How to” 2.Create a simplified “how to” section 3.Use FAQs for missed items -Update annually with FAQ transfer to guideline 4.Roll-out update to all participants (newsletter) 5.Send to new participants Group discussion… What are the issues, impact and broadness?

3 Audit guideline overly complex SUMMARY The CWG: Agrees with the approach to split the “how to” and policy There is always room for improvement in the audit guidelines

Audit guideline - 7.2: Change to audit schedule 4 (Review) Requests for deadline extensions were unreasonable Background Some MPs requested extensions of all or most deadlines and did not provide sufficient reasons. Guideline states “reasons beyond the market participant’s control” which is not clear. Most reasons were related to internal scheduling or change of staff. Current status As discussed at the December 2011 CWG: The guideline is looking for “exceptional” reason. Entity needs to build a case We will communicate intent through compliance audit training sessions We will change guideline to provide better guidance

Audit guideline - 7.3: Workshops / Notification 5 Lessons learned and timing Background We have been asked by entities to share lessons learned. “Lessons learned” are shared during our audit training sessions - however, this is only useful to scheduled participants which may not be timely for all. Current status “Lessons learned” are provided during the audit training of a scheduled entity. Possible Constraints “Lessons learned” cannot be identified in the audit report - under rule 12, only suspected contraventions are identified.

5 Lessons learned and timing Our thoughts As discussed at the December 2011 CWG meeting: We have modified training to focus on market participant needs Emphasis is placed on key areas (index, pointing to evidence, use of attestations and relating evidence to requirements/measures (location within a file) Further to this, we will: Include a “lessons learned” module at the Stakeholder Information Sessions Provide a review of expectations at RSAW training New: Ad hoc Old: Request stakeholder input on interest and timing Group discussion… Does this work?

5 Lessons learned and timing SUMMARY The CWG: Agrees that a module at the stakeholder information sessions for lessons learned on the general audit process would be valuable There is a need for RSAW sessions to go over auditor expectations: Start with statistically problematic When possible group by standards areas Possibly include AESO technical SME at the session Suggest once a year for problem standards

Audit guideline - 7.3: Workshops / Notification 6 Publish generic index template Background Entities are requested to fill out the index template when submitting audit evidence. The index is sent out as an attachment to the notification letters. Some entities requested that a copy of the index template be published well in advance of the audit. Current status We provide a copy filtered by entity type as an attachment to the notification letter. A generic (non-filtered) copy is provided upon request. Possible Constraints This issue is related to #8 (difficulties for auditors in finding evidence) and with #9 (use of RSAWs or Index)

6 Publish generic index template Our thoughts We have no issues with publishing a generic index – however, since the need for this depends on our decision to use RSAWs or not, we propose that 6, 8 and 9 be discussed as one issue. Group discussion… Does this work?

6 Publish generic index template SUMMARY The CWG: Agreed to discuss this issue with issue #9

Audit guideline - 7.3: Workshops / Notification 7 Need to identify an asset list prior to audit (Can extend to relationships and other information) Background The need for information depends on individual standard requirements and on entity structure. As an example, a participant requested that we provide a list of assets that will be included in the audit. Current status Information is requested and obtained during the audit. Possible Constraints Since the need for information varies by standard and by entity structure, requests for lists cannot be generic. As an example, an entity may have different CIP procedures for different plants within their organisation – in such a case, a list of operating personnel is required for each plant (the asset) so that the yearly provision of the procedure can be verified.

7 Need to identify an asset list prior to audit? Our thoughts Lists are not limited strictly to assets. The lists are required to perform certain assessments. Requests depend on individual standard requirements and entity structure which may include but is not limited to: Functional entity type (who owns, who operates and who performs PP function) Asset list to verify inclusion or exclusion Possible solutions: Introduce pre-audit questionnaire – similar concept to WECC. More information could be requested in the RSAW. Group discussion… Impact (timing), broadness, alternatives?

7 Need to identify an asset list prior to audit? SUMMARY The CWG: Is in favour of using a pre-audit questionnaire There will be a possible need for training

Audit guideline - 7.6: Submit audit evidence 8 Bad index of evidence files Background Based on the 2011 Q2 and Q3 audit experience, it was found that the suggested index format as described in appendix B of the audit guideline, did not provide clear guidelines on what was required. Further to this, entities were allowed to submit a variety of index styles. Having a variety of index structures led to difficulties for auditors to find the actual evidence files and prolonged the audit schedule by necessitating requests for clarifications and re-submittal of additional evidence. Current status As per the December 2011 CWG meeting, we now provide a index template filtered by entity type as an attachment to the notification letter. A generic (non-filtered) copy is provided upon request. Possible Constraints The auditors need the entity to point to the evidence document/page for each applicable requirement.

8 Bad index of evidence files Our thoughts Since the need for an index depends on our decision to use RSAWs or not, we propose that 6, 8 and 9 be discussed as one issue. Group discussion… Does this work?

8 Bad index of evidence files SUMMARY The CWG: Agreed to discuss this issue with issue #9

Audit guideline - 7.6: Submit audit evidence 9 Audit index VS use of RSAWs Background Issue numbers 6 and 8 outlined the difficulties that the auditors were having in finding evidence documents because of bad indexes. Entities suggested that perhaps the RSAW could be submitted. A pilot project enabling submission of both the RSAW and index (as back-up) was provided to requesting entities for 2012 Q1 and Q2. Current status As per the December 2011 CWG meeting, we now provide an index template filtered by entity type as an attachment to the notification letter. A generic (non-filtered) copy is provided upon request. As a pilot project, RSAW submission with index back-up is permitted when requested. Possible Constraints The intent is that the index or RSAW be used to point the auditor to the evidence document/page related to the requirement/measure of the standard. For efficiency, only one method should be used.

9 Audit Index VS use of RSAWs Our thoughts The pilot project of using RSAWs was quite successful. There were no major difficulties in finding evidence. Most submitted RSAWs used imbedded document links which proved to be quite efficient. In one case the link did not work however the document was found using the index back-up. Alternatively an additional request could have been made. The “Description” column was sometime used to give or state evidence. The intent of the column is to describe the evidence (such as paragraph or page number). The purpose of using the RSAW should be the same as the index – to point the auditor directly to the evidence. The advantage of the RSAW is that the requirement, measure and assessment step are all aligned in one document and page row. Group discussion… Index VS RSAW? Impact and broadness?

9 Audit Index VS use of RSAWs SUMMARY The CWG: Agrees with using the RSAW instead of the index template Notes: Language in RSAW could be clearer Separate the request from auditor functions o Side note: New RSAW are formatted this way

Audit guideline - 7.7: Review of completeness 10 (Review) Initial review too detailed Background (Internal issue) The completeness review went beyond a simple check that all documents were received and usable. Some assessment was being performed during the completeness review and then repeated in more detail during the assessment phase. Some entities were confused by the similar nature of requests during the completeness review and assessment phases. Current status As discussed at the December 2011 CWG: The guideline will be changed to state “Administrative review” so as to align with language used in self-certification We will limit the “administrative review” to the intended checks We will better communicate the intent through training workshops

Audit guideline - 7.7: Review of completeness 11 How can we help entities better manage information requests? Background Some participants have asked us to give them an idea of when to expect additional information requests in order to better plan SME resources during the audit assessment phase. Current status Although auditors try to provide an estimate of when additional requests will be available, it is not part of the process to provide an audit assessment schedule Possible Constraints Information request timelines may depend on volume of material which is not known until the initial “administrative review” is completed. Once the volume is known, a tentative schedule of when additional requests for each standard type is possible – however, there are too many variables to be able to determine in advance when the draft or final report will be completed.

11 Need to establish an additional request schedule Our thoughts Establishing a schedule of when information requests will help entities and us to better prepare internal resources Any schedule would be on a tentative basis and could not be guaranteed Possible inclusions in process: After the “administrative review”, the AESO will provide a tentative schedule of when the entity can expect information requests. The entity would provide any possible constraints The AESO will make “best effort” to accommodate entity needs. Group discussion… Work for you? Impact and broadness?

11 Need to establish an additional request schedule SUMMARY The CWG: Agrees with the approach of providing a schedule for additional requests: Schedule is a best guess Group by standard area Possibly update tentative schedule on known changes Ex: Where it conflicts with the constraints

Audit guideline - 7.7: Review of completeness 12 Additional request content is sometimes unclear Background A participant stated that additional requests are not always clear and require additional follow-up to be understood. This leads to difficulty in tracking requests, confusion and inefficiencies. Further to this it is not clear how an entity can request clarifications. Current status Process is not sufficiently detailed. Possible Constraints Requests are made so that the auditor is satisfied that all available evidence has been received. Multiple requests may still be necessary.

12 Additional request process not efficient Our thoughts Establishing a more detailed process may help clarify request content. Auditors will use a template for requests Template to include all key information Standard / Requirement Background of request and any related documents Entities to use the template to respond Space provided for response Space provided to attach documents Group discussion… Work for you? Impact and broadness?

12 Additional request process not efficient SUMMARY The CWG: Agrees that using a template is good Entity should communicate any need for clarifications and the AESO will re-word the request if required.

Audit guideline : Review draft audit report 13 Extent of commenting on re-issued draft report Background The guideline allows for re-issue of the draft report: “In the event a re-evaluation of previously submitted audited evidence is deemed appropriate, and results in revised audit findings or significant changes to the draft audit report, a new draft audit report will be issued and the registered market participant will be provided ten business days to review and submit their comments in writing to the compliance monitor.” It is unclear however as to what extent comments may be provided. Current status No changes have been made to section 7.11 of the guideline. Possible Constraints None identified

13 Extent of commenting on Re-issued draft report Our thoughts Comments on a re-issued draft report should be limited to items that have changed. The guideline wording would be changed to: “In the event a re-evaluation of previously submitted audited evidence is deemed appropriate, and results in revised audit findings or significant changes to the draft audit report, a new draft audit report will be issued and the registered market participant will be provided ten business days to review and submit their comments on the changes in writing to the compliance monitor.” Group discussion… Work for you? Impact and broadness?

13 Extent of commenting on Re-issued draft report SUMMARY The CWG: Agrees with the approach of adding “on the changes” to this section of the audit guideline.

Next steps We will send power point summaries to all CWG members Create solutions based on today’s results Set next CWG meeting End July / Early August Date: Will send out survey to members on possible dates Time: Afternoon, mid-week is good. Purpose AESO to present solutions based on first meeting input CWG to refine

Thank You Any questions or comments?