1. Risk based selection for On the Spot Control at Agricultural and Rural Development Agency Miklós Lelkes Central Physical Control Department Agricultural and Rural Development Agency (ARDA) Budapest, Hungary Control Methods Workshop 2010 13-15 April Ispra Italy

2. Control Methods Workshop 2010, Ispra Italy Selection for physical control Start point: 100 % of claims Selection of control sample (cost effectiveness) (e.g. min. 5% control rate) –Random selection:20-25% (representative overview) –Risk analysis: 75-80% (financial risk of EU) –Direct selection To have effective control methods Agricultural and Rural Development Agency (ARDA) Budapest, Hungary

3. Control Methods Workshop 2010, Ispra Italy Defining the risk factors and weighting First year(s) –Based on expert appraisal Evaluation of the results of the Control  update of the selection method Changes in the category limits Changes in the scoring Changes in the weighting Agricultural and Rural Development Agency (ARDA) Budapest, Hungary

4. Definition of a risk factor (example) small / large cases are with a higher risk Agricultural and Rural Development Agency (ARDA) Budapest, Hungary Average parcel size Risk score

5. Control Methods Workshop 2010, Ispra Italy Evaluation of the effectiveness of the risk analysis (2007) ~ 241.000 claims For > 80 measures 4th year of Hungary/ARDA in the EU  Huge amount of information in the IACS  Need for special solution for deriving information from the DB  Data mining software/technique Agricultural and Rural Development Agency (ARDA) Budapest, Hungary

6. Control Methods Workshop 2010, Ispra Italy The goal of Data Mining ?  Not to find the perfect model for a certain problem  but to find the optimal model for a certain problem that is: Robust Generalizes well Easy to understand Provides insight into drivers of the problem Easy to implement Agricultural and Rural Development Agency (ARDA) Budapest, Hungary

7. Control Methods Workshop 2010, Ispra Italy Risk Analysis#1 Create an abstract mathematical model that behaves coherently with regard to risky farmers. Generalize risk patterns for automatic detection -Identify typical patterns -Score applications for probability of non- compliance Agricultural and Rural Development Agency (ARDA) Budapest, Hungary Model 1 Model 2 trainingvalidation

8. Control Methods Workshop 2010, Ispra Italy Risk Analysis#2 Modeling techniques: -Predictive Modeling: Decision Trees Neural Networks Regression Scorecards Pre-requirements: -confirmed historical cases as input data (non-compliance flag) Result: Risk score Agricultural and Rural Development Agency (ARDA) Budapest, Hungary

9. Control Methods Workshop 2010, Ispra Italy Data mining / estimation models Target variable:Errors (bad / good) Agricultural and Rural Development Agency (ARDA) Budapest, Hungary Average rate of anomalies % of anomaly as a function of sampling rate (with population ordered by decreasing rate of anomaly)

10. Control Methods Workshop 2010, Ispra Italy Data mining with SAS in Hu Pilot in 2007 –Selection of OTSC sample of SAPS by data mining Operational from 2008 –All area and animal based subsidies 2nd year of operational work in 2009 –Extend use to all measures Agricultural and Rural Development Agency (ARDA) Budapest, Hungary

11. Control Methods Workshop 2010, Ispra Italy Interesting first results (2007) Less categories in the factors Some criterion were not relevant! (but in 2007 they were used because of the regulation) Agricultural and Rural Development Agency (ARDA) Budapest, Hungary Area size>0.3 and <1.0 -5 >1.0 and <10.0 3 >10.0 Age>18 and <401 >40 and <608 >603 High risk = low score

12. Control Methods Workshop 2010, Ispra Italy Operational work from 2008 The Integrated Risk Analysis System includes: Interface to –IACS System (direct access) –Hungarian ovine and caprine I&R System –Hungarian bovine I&R System –Hungarian porcine I&R System –Farmers Registry Risk datamart Analytical models (approx. 15 analytical models) –Area based measures (e.g. SAPS); –Agricultural Environment Protection Program based measures; –Rural Development (investment projects) based measures Agricultural and Rural Development Agency (ARDA) Budapest, Hungary

13. Control Methods Workshop 2010, Ispra Italy Data mining Statistical analysis Architecture of Integrated Risk Analysis System Place of operational work (system for transactions) Utilization Extraction data quality integration transformation IACS Datawarehouse Source External Datamart Risk management Information for management Web reporting Uniform handling of metadata Agricultural and Rural Development Agency (ARDA) Budapest, Hungary

14. Control Methods Workshop 2010, Ispra Italy Models Decision tree –Easy to understand (if not too complicated) –Big groups Regression Neural network –Normally the best prediction result –The result can not be interpreted (“black box”) Scorecard 14 Agricultural and Rural Development Agency (ARDA) Budapest, Hungary

15. Control Methods Workshop 2010, Ispra Italy Scorecard Easy to work with, can be calculated “by hand” Easy to interpret Good for non linear variables (i.e. age) Not a problem, if the variable has strange distribution Sometimes can result in big groups Not a group of factors, but a uniform model! 15 Agricultural and Rural Development Agency (ARDA) Budapest, Hungary

16. Control Methods Workshop 2010, Ispra Italy Data Mining Model Training and Scoring Target Variable Prediction Analysis Scoring Score Rules (Score Code) Converting a complex control result in a binary format (black or white) Should be defined by PA Agricultural and Rural Development Agency (ARDA) Budapest, Hungary

17. Control Methods Workshop 2010, Ispra Italy Operational work from 2008 The Integrated Risk Analysis System includes (cont.): –Scoring lists –Evaluation statistics (Random and Risk based) –OLAP reports –Ad-hoc reporting and analysis interface Agricultural and Rural Development Agency (ARDA) Budapest, Hungary SAS® Enterprise Guide SAS® Enterprise Miner 5

18. Control Methods Workshop 2010, Ispra Italy Results Analysis of results (e.g.: interpretation of scorecards) Model documentation –Automatic report in Enterprise Miner –Auditable documentation of data mining process Documentation of all selection procedures Review of model quality Statistics for EU (OLAP cubes) Agricultural and Rural Development Agency (ARDA) Budapest, Hungary

19. Control Methods Workshop 2010, Ispra Italy Risk factors found relevant by data mining Agricultural and Rural Development Agency (ARDA) Budapest, Hungary 200720082009 Total amount of application--- Total area change in the area Number of physical blocks Number of physical blocks in 2007 --- Number of physical blocks in 2006 Number of parcels number of parcels change in number of parcels from previous year change in number of parcels from 2005 to 2006 --- change in number of parcels from 2004 to 2005 --- Number of different land use type--- average parcel size--- underclaimed reference parcel underclaimed reference parcel in the previous year ---

20. Control Methods Workshop 2010, Ispra Italy Risk factors found relevant by data mining Agricultural and Rural Development Agency (ARDA) Budapest, Hungary 200720082009 percentage of risky parcels in the application (close to minimum parcel size) --- joint cultivation--- grassland in the application---grassland in the application percentage of orchard and vineyard--- percentage of orchard and vineyard compactness of the farm--- change of application complexity--- result of earlier controls --- TOP-UP crops in the application--- Gender of applicant --- Age of applicant Other area based applications ---

21. Control Methods Workshop 2010, Ispra Italy Results by mean of efficiency Cumulative lift: 1.5-4 times higher compared to random sample –Fine tuning of criterion and weighting (0.2-3 lift increase) –Selection of variables (0.1-1 lift increase) –Proposal of new variables to include (0.1-0.8 lift increase) –Global optimization of cross-validation effects (0.1-0.5 lift increase) Agricultural and Rural Development Agency (ARDA) Budapest, Hungary Lift = ratio of anomalies in risk sample over random sample

22. Control Methods Workshop 2010, Ispra Italy Some objectives found by OTSC The quality of application could be better  level of irregularities in the random sample still high ?(although it is better from year to year) –Higher control rate?  better risk assessment (in terms of % of anomaly)! Rate of irregularities in classical field inspection sample is significantly higher than in RS sample (research needed) –Difference in technique –Difference in selection / population Agricultural and Rural Development Agency (ARDA) Budapest, Hungary

23. Control Methods Workshop 2010, Ispra Italy Sliding window Agricultural and Rural Development Agency (ARDA) Budapest, Hungary Blocks  farmers  risk calculation per window, other info

24. Control Methods Workshop 2010, Ispra Italy Various shapes Agricultural and Rural Development Agency (ARDA) Budapest, Hungary 30 x 30,30 x 42,10 x 30,30 x 10,10 x 10…

25. Control Methods Workshop 2010, Ispra Italy Efficiency Agricultural and Rural Development Agency (ARDA) Budapest, Hungary Net over gross ratio constraint may be opposite to risk?

26. Control Methods Workshop 2010, Ispra Italy Conculsion for the effective risk analysis Worth to use data mining solution Yes, we need RS, but the level of anomalies in RS sample is a question Better not to use all farms in the RS zone –20-25% use of VHR? –Site shape? –Quota for MS? Cost of data mining solution? –Expensive –But cheaper, than a flat rate correction! Agricultural and Rural Development Agency (ARDA) Budapest, Hungary

27. Control Methods Workshop 2010, Ispra Italy Thank you for your attention! Agricultural and Rural Development Agency H-1095 Soroksári út 22-24. www.mvh.gov.hu Central Physical Control Department Tel.: + 36 1 301-2409 Fax:+ 36 1 301-2444 E-mail: lelkes.miklos@mvh.gov.hu