1. REVĪZIJAS IESTĀDES JAUNĀ LOMA – REZULTĀTU UN IZPILDES RĀDĪTĀJU AUDITS Olga Guza, vecākā eksperte, ES fondu revīzijas departaments, Finanšu ministrija

2. Content of the presentation Output and result indicators Goals at different stages and performance reserve Auditing approaches and requirements Available data, IT system, reporting and information from beneficiaries Timetable and extent of auditing

3. Output and result indicators Indicators and corresponding targets established in order to assess progress in programme implementation aimed at achievement of objectives as the basis for: –monitoring –evaluation –review of performance Those indicators include: –output indicators relating to the operations supported –result indicators relating to the priority concerned Defined in the programme

4. Indicators in the programme ProgrammePriority axe Investment priority Specific objective Indicators Specific objective Investment priority Specific objective Priority axe Investment priority Specific objective In Latvia: 1 OP 12 priorities 33 investment priorities 72 objectives

5. Structure of indicators IndicatorsResult Baseline value (at the beginning of the programming period) Target value (2023) OutputBaseline is zero Target value (2023) 121 result indicators in OP of Latvia 135 output indicators in OP of Latvia

6. Example Priority axe - Environmental protection and resource efficiency Investment priority - To preserve, protect, promote and develop the natural and cultural heritage Specific objective - To preserve, protect and develop the important cultural and natural heritage, as well as the development of related services Output indicators: 1. Number of supported sites of natural and cultural heritage Baseline 0; Target 15 sites 2. Increase in expected number of visits to supported sites of cultural and natural heritage and attractions Baseline 0; Target 140 000 visits Result indicator: Spent nights in tourist accommodation establishments in Latvian territory during the year Baseline: 3 775 195 nights Target: 3 978 152 – 4 079 631 nights

7. Performance review EC in cooperation with MS, shall undertake a review of the performance of the programmes in each MS in 2019 (the 'performance review') –with a view to monitoring progress towards the objectives and targets set for each priority over the course of the 2014 - 2020 programming period –to ensure that the budget of the Union is not used in a wasteful or inefficient way Performance review will be carried out with reference to the performance framework set out in the respective programme

8. Performance framework Performance framework consists of up to four different types of indicators: financial, output and result indicators and key implementation steps OP Result indicatorsOutput indicators Performance framework Subset of result indicatorsSubset of output indicatorsFinancial indicatorsKey implementation steps 121 → 0 135 → 31

9. Performance framework in the OP Programme Priority axe Investment priority Specific objective Indicators Specific objective Indicators Investment priority Specific objective Priority axe Investment priority Specific objective Performance framework Progress is reviewed twice during the programming period against the milestones and targets set in the programme.

10. Goals at different stages 2018: Milestones are intermediate targets, directly linked to the achievement of the specific objective of a priority The milestones are intermediate targets set for indicators to be achieved by 31 December 2018 and to be assessed in 2019 2023: Targets The targets themselves are set to be achieved by 31 December 2023 and their accomplishment will be assessed at the closure of the programme period in 2025

11. Performance reserve 6 % of ERDF, ESF and Cohesion Fund under the Investment for Growth and Jobs goal, EAFRD and to measures financed under shared management in accordance with the EMFF Regulation Established in the Partnership Agreement and programmes Established in order to facilitate the focus on performance and attainment of the objectives of the Union strategy for smart, sustainable and inclusive growth

12. Allocation of Performance reserve 2019 MS submits Annual Implementation Report EC + MS carries out Performance Review Use of Performance Framework from programme Priority achieved its milestone if: no more than two indicators → all indicators ≥ 85% of their milestone value by the end of 2018 three or more indicators → all indicators except for one ≥ 85% of their milestone value and one indicator ≥ 75% of its milestone value EC adopts decision (Implementing Act) Priorities which have achieved milestones – Performance reserve deemed allocated with Decision of EC Priorities which have not achieved milestones – MS should propose reallocation of Performance reserve

13. Consequences of poor performance (after Performance review) Suspension of payments Serious failure in achieving milestones relating only to the financial and output indicators Failure is due to clearly identified implementation weaknesses, which EC had previously communicated to MS MS has failed to take the necessary corrective action to address such weaknesses At the end of 2018: If 2 indicators, any of indicators < 65% of milestone value If > 2 indicators, any 2 of indicators < 65% of milestone value

14. Consequences of poor performance (after EC examination of the final implementation report) Financial corrections Serious failure to achieve the targets relating only to financial indicators, output indicators Failure is due to clearly identified implementation weaknesses, which EC had previously communicated to MS MS has failed to take the necessary corrective action to address such weaknesses At the end of 2023: If 2 indicators, any of indicators < 65% of target value If > 2 indicators, any 2 of indicators < 65% of target value

15. Auditing approaches and requirements – Designation procedure 2 ways of carrying out audit work: –Full audit work in accordance with EC guidance and checklist –But if management and control system is essentially the same as for the 2007-13 programming period + audit evidence of its effective functioning during that period is present → conclusion that the relevant criteria are fulfilled without carrying out additional audit work In both cases - for the new criteria (including procedures to ensure reliability of data on indicators/milestones/progress of the OP in achieving its objectives), audit work will have to be performed in order to assess the compliance in these areas

16. During designation procedure Check whether following procedures are in place and address following issues? physical progress of the product/service compliance with the terms and the conditions of the grant agreement compliance with the output and result indicators Procedures and checklists for the management verifications – checking reality of the project Starting from 1 July 2015: system ensures reliable and relevant data relating to indicators and milestones (broken down by gender where required) and on the progress of OP in achieving its objectives Procedures for IT system to collect, record and store data on each operation procedures to confirm the reliability of data relating to indicators, milestones and the progress of programme Procedures to draw up the management declaration, report on the controls and the annual summary

17. Auditing approaches and requirements – system audits and audits of operations Wording of the Opinion: –«In my opinion, and based on the audit work performed … the management and control system put in place function properly» –«The audit work carried out does not put in doubt the assertions made in the management declaration» Wording of the Management declaration – «Furthermore, I/we confirm the reliability of data relating to indicators, milestones and the progress of the operational programme required under article 125(2)(a) of Regulation (EU) No 1303/2013»

18. System audits The audit scope include verification that the relevant authorities properly ensure reliability of data relating to: –output indicators –milestones (financial and output indicators, key implementation steps) –progress of the programme in achieving its objectives Audit Authority should assess: –whether effective controls are implemented over collecting, summarizing and reporting the related data –whether the reported compiled figures reconcile with the source data

19. Relevant key requirements – Key requirement 2 AC 2.1:The MA drew up, for approval by the Monitoring Committee, appropriate selection procedures and criteria that ensure the contribution of operations to the achievement of the specific objectives and results of the relevant priority AC 2.4: All applications/projects are evaluated in accordance with the applicable criteria NoMain risks (examples)Expected controlsTesting procedures 1.Approved selection procedures and criteria are not appropriate All selection procedures and criteria prepared by IB are reconciled by MA All selection procedures and criteria are approved by the Monitoring Committee Control test for process of preparation and approval of selection procedures and criteria 2.Approved selection procedures and criteria are changed during the ongoing selection process resulting in non equal treatment of applicants 3. Decisions taken on the acceptance or rejection of applications are not adequate resulting in ineligible expenditure Applications are evaluated using the selection criteria /scoring approved by the Monitoring Committee Evaluators of the applications possess the required expertise and independence Control test for a sample of applications Substantive testing (re-performance) for a sample of applications (take both approved and rejected applications) Review submitted appeals

20. Relevant key requirements – Key requirement 4 AC 4.1: The management verifications include Administrative verifications and On-the-spot verifications of operations AC 4.3: Written procedures and comprehensive checklists should exist to be used for the management verifications in order to detect any material misstatements NoMain risks (examples)Expected controls Testing procedures 1. Risk of suspension of payments if management verifications do not timely reveal a serious failure in achieving milestones / targets relating to financial and output indicators Beneficiaries' application for reimbursement templates are adjusted in order to enable for timely and correct reporting on indicators Administrative verifications include verification on: application for reimbursement includes check on the progress in the attainment of indicators final application for reimbursement includes check on information on the actual contribution to the output and results indicators provided by the beneficiary On-the-spot verifications include verification on: correctness of the data communicated by the beneficiaries in relation to the indicators correct understanding of the indicator by the beneficiary and the values reported correctness of inputting information on indicators into the IT system by the beneficiary Review of the checklists Control test for a sample of administrative and on-the-spot verifications performed Substantive testing for a sample of administrative and on-the-spot verifications performed 2. Written procedures and checklists for verifications are not detailed enough in order to ensure quality and reliability of the data on indicators

21. Relevant key requirements – Key requirement 5 AC 5.1: The detailed supporting documents for operations are kept at the appropriate management level (including progress in achieving outputs and results and monitoring reports) The audit trail shall allow data in relation to output indicators for the operation to be reconciled with targets and reported data and result for the programme NoMain risks (examples)Expected controlsTesting procedures 1. The data, aggregated related to indicators and target values at investment priority, priority or programme level is not timely, complete and reliable Reconciliation performed for the data submitted to the monitoring committee and EC (AIR) Re-performance of reconciliation

22. Relevant key requirements – Key requirement 6 AC 6.1: The existence of IT system capable to collect, record and store data on each operation, including data relating to indicators and milestones and on the progress of the programme in achieving its objectives AC 6.2: Adequate procedures are in place to allow for the aggregation of the data where this is necessary for the purposes of evaluation, audits, as well as for payment applications and accounts, annual summaries, annual implementation and final reports, including reports on financial data, submitted to the Commission

23. Thematic audits System audits relating to key requirements targeted to specific thematic areas should be indicated in the Audit Strategy, which may include: –Thematic audit on the reliability of data relating to indicators and milestones and on the progress of the operational programme in achieving its objectives provided by the managing authority to the monitoring committee –Thematic audit on the functioning and security of IT systems

24. Audits of operations The audits of operations should cover: –that for expenditure declared to EC outputs and results underpinning payments to the beneficiary have been delivered –participant data or other records related to outputs and results are consistent with the information submitted to EC –required supporting documentation demonstrates an adequate audit trail

25. Available data Ex-ante evaluation IT system Information from beneficiaries Reports at National level (including information submitted by the managing authority to the monitoring committee) Annual Implementation Reports

26. Ex-ante evaluation Information on appraisal of: –the relevance and clarity of the proposed programme indicators –how the expected outputs will contribute to results –whether the quantified target values for indicators are realistic, having regard to the support from the Funds –the suitability of the procedures for monitoring the programme and for collecting the data necessary to carry out evaluations –the suitability of the milestones selected for the performance framework

27. IT system Data on indicators: common and programme specific output indicators (including measurement unit, target value, achievement level for each calendar year) common and programme specific result indicators (including measurement unit, baseline value, target value, achievement level for each calendar year) Member States shall ensure that no later than 31 December 2015, all exchanges of information between beneficiaries and authorities can be carried out by means of electronic data exchange systems

28. National reports (Latvia’s example) MA Responsible Bodies (10 entities) Cooperation Body (CFCA) Monitors achievement of output and result indicators at the level of specific objectives Monitors achievement of output and result indicators at the level of projects Carries out administrative and on-the-spot checks Participate in on-the- spot checks Enters data to MIS Enters data to MIS on result indicators MIS Reports to MA Reports Provides info to Monitoring Committee Prepares annual plan on achievement of outputs and results (milestones and targets) Prepares AIR

29. Annual Implementation Reports Year of submission of AIR 2016201720182019 Content of AIR Information on implementation of OP and its priorities by reference to the financial data, common and programme-specific indicators and quantified target values, including changes in the value of result indicators Additional content Information on milestones defined in the performance framework Additional content No Achieving the objectives of OP No Achieving the objectives of OP

30. Extent of auditing ISA 330 Controls that have not changed from previous audits: The auditor’s can make decision on whether to rely on audit evidence obtained in previous audits for controls that: (a) have not changed since they were last tested; and (b) are not controls that mitigate a significant risk, But the auditor shall test the controls at least once in every third year, and shall test some controls each year to avoid the possibility of testing all the controls on which the auditor intends to rely in a single audit period with no testing of controls in the subsequent two audit periods

31. Timetable of auditing – multiannual (example) Events 1st AIR IT system 1st AIR with milestones Milestones to be achieved Performanc e review Year 20152016201720182019 Audit work Designation procedure IT system (KR6) MA functions (+ delegated) Indicators thematic … Audit work KR2 system (+KR5) KR4 system (+KR5) ……

32. Timetable of auditing – one year Events 5.07 – AA receives list of expend. 30.10 – AA receives draft accounts 31.01. AA receives MD Timeline January - June July August - December January Audit work System audits Sampling Audits of operations Audit of accounts

33. Conclusions Auditing approach should follow a common audit methodology for the Funds Key focus on data related with financial and output indicators Include specific information in the Audit Strategy Audit during: –designation process –system audits (including thematic audits, IT audits) –audit of operations Part of the audit opinion of the audit authorities

34. Thank you for your attention! Olga Guza (+371) 6 708 3864 olga.guza@fm.gov.lv