1. Sampling bias in multi-agent simulation (MAS) models Buysse, J 1., Frija, A 1., Van der Straeten, B 1., Nolte, S. 1, Lauwers, L. 1,2, Claeys, D. 2 and Van Huylenbroeck G. (1)Ghent University, Department of Agricultural Economics (2)Institute for Agricultural and Fisheries Research, Merelbeke, Belgium 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics

2. Overview of the presentation - Why MAS models? - Problem statement: the sampling bias in MAS - Objectives - Methodology - Results - Perspectives 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics

3. Why MAS models? - Heterogeneity of opportunities and constraints at the individual level -Accurate estimation of policy distributional effects - Accurate estimation of agents interactions (spatial effects, TC, propensity of innovation, etc.) - But… 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics

4. Problem statement - The need for full population data - In case of sampling, farms in sample cannot interact with the real-world farm 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics

5. Problem statement 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics Farmer i Full population Farmer j

6. Problem statement 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics Farmer i Sample Farmer k

7. Problem statement -Systematic bias when TC between agents are simulated in MAS - Most MAS empirical models rely on sample data - Future large scale MAS models on sampled data ! 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics

8. Problem statement Illustration of sampling bias on real model (Van der Straeten et al. 2010) - A MAS model used to simulation PR exchange between 30,000 farmers in Flanders - The bias is correlated to the sample size 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics BootstrapNumber of repetitions Average cost simulated SDAverage cost simulated/Average cost of population S= 100 (0.26%)1003210.881200.77+ 59 % S= 200 (0.52%)1002698.58666.24+ 33 % S= 500 (1.31%)1002581.09755.91+ 28 % S= 750 (2%)1002571.13536.30+ 27 % Full population (100%)-2016.49--

9. Objectives ‣ To test, illustrate, and quantify the sampling biases resulting in cases of existence of TC ‣ To develop and to discuss mechanisms that can remove such sampling biases 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics

10. Methodology - Simplified MAS model - minimizes transport costs of emissions and the cost of emission abatement: Minimize Σ n (Σ m c nm τ nm + ω n p) s.t. e n + Σ m τ mn - Σ m τ nm ≤ r n + ω n where ‣ n and m are farm indices, ‣ τ nm is the amount of transported emission form n to m, ‣ ω n is the amount of emission abatement of agent n, ‣ e n is the amount of emission of farm n, ‣ r n is the amount of emission rights of farm n, ‣ c nm is the transport cost per transported emission from farm n to m, ‣ p is the penalty per overused emission right. 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics

11. Methodology -Applied on synthetic population data (500 farmers): average cost per farm: 27.74 ‣ Emission (e n ) random sampled from normal distribution, ‣ Emission right (r n ) random sampled from normal distribution, ‣ Transport costs (c nm ) is random sampled from uniform distribution - We bootstrap on different sample sizes: 20-500 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics

12. Results 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics Average cost of 27.7 Variations

13. Results - The order of magnitude of the sampling bias can be very large - Nonlinear effect of the sample size on the bias Cause: - Subsamples do not always satisfy the real population balance - Motivation for sampling bias correction via macrobalance coefficients - The amount of emission is smaller than the total amount of emission rights (Σn en < Σn rn) 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics

14. Results 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics Average cost of 27.7 Average costs of samples

15. Remove remaining bias with calibration - Calibration is the comparison of two measurements: -the measurement of a device with known correctness: full population model -is used to correct another measurement made by another device: sample model -Once calibrated, the second device can make correct measurements: sample model can be used in for simulations - Resampling data is used to estimate the calibration function: prediction of the bias as a function of the sample size 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics

16. Results coefficients of the polynomial of the simulated average costs on the sample size 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics EstimateStd. Errort valuePr(>|t|) (sample size)^06,11E+052,68E+0422.754< 2e-16*** (sample size)^1-2,53E+042,62E+03-9.657< 2e-16*** (sample size)^25,64E+029,35E+016.0321.88e-09*** (sample size)^3-7,46E+001,69E+00-4.4031.11e-05*** (sample size)^46,24E-021,78E-023.5050.000464*** (sample size)^5-3,42E-041,16E-04-2.9440.003267** (sample size)^61,24E-064,84E-072.5630.010450* (sample size)^7-2,95E-091,29E-09-2.2860.022337* (sample size)^84,42E-122,13E-122.0760.037970* (sample size)^9-3,78E-151,98E-15-1.9110.056064. (sample size)^101,41E-187,91E-191.7780.075497.

17. Results 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics Average cost of 27.7

18. Conclusion - Macrobalance correction is very useful -Only macrobalance is necessary -Also usefull in models without heterogenous interactions - Calibration correction is promising -such corrections are not possible if we do not have full population data -necessity to assign correction factors based on information available in sample datasets - Corrected sampling in MAS is important - more complex analysis become possible - more datasets at sample level could be used - MAS can be applied in large scale empirical models 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics

19. Further research - Check for: -Impact on variance -Impact of changes in model structure -Impact of using synthetic full population as calibration reference - Search for: -Calibration correction without availability of full population (see first attempts in paper) 122 nd EAAE Seminar – Ancona 16- 18 February 2011 UGent – Faculty of Bioscience Engineering– Department Agricultural Economics

20. THANK YOU