ProZorro : Innovations in Digital Ex-Ante Monitoring Tools for Public Procurement in Ukraine Eliza Niewiadomska - 2019

Contents of presentation Ex-Ante monitoring in Ukraine - the challenges and opportunities EBRD Technical Assistance Project Solution Benefits Lessons learned and next steps 26 July, 2019

1. The procurement monitoring challenge in Ukraine Monitoring and audit of public procurement is necessary to ensure compliance with public procurement law and curb corruption and waste by deterrence and detection. However, the high volume of procurement transactions in Ukraine made this impossible to do manually resulting in low detection rates or overreliance on retrospective audits. 26 July, 2019

1. The monitoring opportunity in Ukraine Development of the ProZorro e-Procurement system presented an opportunity to implement real time data driven monitoring and prevention through proactive intervention. Enabled data processing algorithms in line with preselected risk management framework and risk assessment needs to be conceived. Pioneer the use the Open Contracting Data Standard for monitoring and risk based auditing 26 July, 2019

1. Ex-ante monitoring as a means to mitigate risks Key principles for effective ex-ante procurement monitoring in Ukraine: Monitoring body should be independent; All monitoring activities should be open an accessible to the public when officially published; Trigger for open monitoring case should be automated compliance and automated risk indicators; 26 July, 2019

1. Ex-ante monitoring as a means to mitigate risks (cont.) Key principles for effective ex-ante monitoring in Ukraine: Monitoring decision should be evidence based, risk-based and data-driven; Monitoring function should be introduced by the law; Methodology of monitoring should be approved with the policy maker; Methodology should consist of procedures selection rules and processes which should be evidence-based and data-driven; 26 July, 2019

2. EBRD Technical Co-Operation Project Concept EBRD developed a technical co-operation project with the Government of Ukraine to introduce: Digital ex-ante monitoring to improve compliance, efficiency and effectiveness of public procurement; Data-driven, evidence-based and risk-based approach to procurement monitoring; Digitalized monitoring procedure in Ukraine’s e-procurement system ProZorro using the Open Contracting Data Standard (OCDS); Risk mitigation in public procurement and optimize efficiency of the procedures. 26 July, 2019

2. EBRD Technical Co-Operation Project Scope of Work Development of: The data-driven ex-ante monitoring methodology & automated risk indicators; Automated procurement procedures with risk prioritisation concept for selection of high risk public procedures for ex-ante monitoring; Automated risk indicators engine prototype and risk procedures prioritization engine developed on a proof of-concept basis. 26 July, 2019

2. Legal and technical implementation of the ex-ante monitoring solution January 2018 - an amendment to the Public Procurement Law was adopted and ex-ante monitoring was introduced; State Audit Service of Ukraine became responsible to conduct ex-ante audit for all public procurement procedures; September 2018 official methodology of compliance indicators, risk indicators and risk-prone procedures prioritization was approved by the Ministry of Economy and Trade of Ukraine and State Audit Service. 26 July, 2019

3. ProZorro ex-ante monitoring solution overview 26 July, 2019

3. Procedures covered by the ex-ante monitoring solution PROCUREMENT METHOD PLANNING NOTICE BID PROPOSALS AWARD CONTRACT SIGNING CONTRACT EXECUTION Open tender above EU threshold 4 2 6 5 Open tender 8 Below threshold open tender Negotiation 1 Direct purchasing (below threshold reporting) TOTAL   16 17 26 July, 2019

3. Legal and technical implementation of the ex-ante monitoring solution October 2018 - ProZorro launched new modules in pilot mode: Indicators engine Prioritization engine Cabinet of State Audit Service October 2018 official procurement monitoring started by the State Audit Service of Ukraine 26 July, 2019

3. Operational performance of the system Procedures risk indicators online calculation - every 10 minutes In average 280 000 times per day indicators are recalculated by the engine 150 000 times procedures are verified against indicators algorithms daily In average 25 000 indicators statuses are changed per day Risky procedures online queueing - once an hour Risky procedures online prioritisation - once an hour 26 July, 2019

4. Ex-ante monitoring solution benefits Enables real-time manipulation of the complete procurement dataset. Provides the big picture in public procurement system for risk allocation. Enables monitoring throughout the year, supporting a smoother process. Presents real-time results for the entire procurement system as well as for the monitoring function when required. Facilitates procurement risk analysis, transaction and controls efficiency assessment. Keeps monitoring procurement transactions during its whole lifecycle and even while you sleep.

4. Ex-ante monitoring benefits (cont.) Developed for Ukraine - can be suitable for procurement sectors worldwide and is open source. As an automated OCDS based monitoring solution, it enables smarter & intelligent procurement transaction monitoring at any stage of the procurement process for any purposes and by any interested authority. For example, the solution can be effectively utilized by: Official authorities for ex- ante public procurement monitoring. Supreme Audit Institutions - to plan and execute audits. Internal auditors - to evaluate and improve the effectiveness of risk management, control, and governance processes. Civil societies and watchdogs involved in public procurement monitoring.

4. Ex-ante monitoring benefits (cont.) Official monitoring authorities are now able to find abnormal activity within the public procurement much quicker, more accurately and react appropriately. Already positive signs that the solution will enable profiling and intelligent targeting of procurement activities that should be monitored at any or all stages of the procurement cycle. The existence of the solution has had a positive impact on the perception of public procurement in Ukraine. 26 July, 2019

5. Lessons learned so far Successful implementation of a risk-based monitoring methodology is dependent on: People: Government stakeholders are struggling with the implementation of the public procurement risk management system. Lesson: Necessary to enhance risk management capacity of the officials responsible for the monitoring and introduce periodical risk assessment of the public procurement system. 26 July, 2019

5. Lessons learned so far (cont) Process: To build the uniformed automated risk-based monitoring approach to the procurement transactions in Prozorro system was challenging due to the risk specific nature of the project and the source of the data. Lesson: The next phase should incorporate this experience into the methodology for the roll out. 26 July, 2019

5. Lessons learned so far (cont) Technologies: Data quality and availability is vitally important for any automated risk-based data analytical approach. Lesson: The technologies should support up to the date monitoring processes and users experience to make it relevant (risk engine + auditors cabinet) 26 July, 2019

5. What is next for automated ex-ante monitoring? Evaluate pilot results and amend the solution with lessons learned if any. Enhance risk management capacity of the officials responsible for the monitoring and introduce periodical risk assessment of the public procurement system in Ukraine. This will enable continuous development of the red flags and risk indicators. Amend methodology with observed pilot suggestions. Introduce the policy for automated monitoring solution periodic validation. Assess indicators performance, implement indicators in a pipeline (more then 50 at the moment).

RANKS TRANSACTION READS TRANSACTIONS PREPARES DATA ONLINE REFERENCING AGGREGATES RISKS ORDERS TRANSACTIONS DELIVERS RESULTS