1. Dr. Russ Congalton & Kamini Yadav GFSAD30 Meeting, Menlo Park 19 th -21 st January, 2016 Reference Data Collection & Accuracy Assessment: Some Results The Error Matrices

2. Theory vs. Practice 2

3. Accuracy by Continent  Australia  Africa  North America  Europe  South America  Asia Each Continent Has Different Characteristics:  Differences in land and water resources including soil and terrain conditions,  Topology and climatology (such as annual precipitation and thermal zones), and  Variability in growing periods of different crops. Each Continent has different suitability for growing crops and thus it is necessary to consider some common agronomical and ecological matching zones for assessing agriculture resources and potential to estimate cropland extent and its types. 3

4. Assumptions and Issues  Assumed:  Create crop/no crop map as first level of stratification before beginning crop type mapping  Map continent by some type of agricultural eco-zone to reduce variability and improve accuracy  Every team mapping the same 8 crop types we agreed on  Issues:  Must determine the appropriate population to sample  Must then balance the sample size based on proportion of area in crop vs. non-crop  Must account for both commission and omission error  Important Note: Conducting this assessment was highly dependent on using HSI for validation samples 4

5. Different Accuracy Assessment Strategies by Continent Each Continent has been treated differently to perform accuracy assessment  Australia: Buffering Crop/No-Crop region using Euclidean Distance between the crop/no-crop pixels  Africa: Considering IIASA crop proportion in each AEZ and asymptote behavior of number of samples to determine the sample proportion in each zone  North America: Minimum number of crop samples in each agro- ecological Zone (AEZ) 5

6. Accuracy Assessment: Australia 6

7. Generating more No-Crop samples for Australia  The Error Matrix was generated from 1,118 ground samples collected by Pardha in August 2014 in Australia  The error matrix and accuracy estimates were not statistically valid and balanced  The samples for Crop and No-Crop were neither balanced nor proportional to crop/no-crop area in GCE v.2 Map 7

8. Crop/No-Crop Area Proportionality GCE v.2 ClassPixel Count (PC) Area sq. m. (PC * 250*250) Area % Cropland 16205071387816937500 Cropland 26363457397716062500 Cropland 31051096569312500 Cropland 4 18932511832812500 Cropland 5 23723514827187500 Cropland 655076134422562500 Total Cropland1365095885318487500012.06316576 No-Crop621946000000088 Total Area 7072644875000 Class No. of Samples Sample %Crop Area % Crop10611.4101184112 No crop82388.5898815988 Total929 Samples in Crop/No-Crop Maintain proportional samples to GCE v.2 Crop/No-Crop area 8 1.Croplands, RF, all crops 2.Croplands, RF, Pastures 3.Croplands, Irr., all crops 4.Croplands, Irr., Pastures 5.Croplands, Irr., Orchards 6.Croplands, Fallow

9. No-Crop Samples 1,118 reference samples (1/3 rd ground collected samples) were received Only 36 samples were of No-Crop out of 1,118 samples Additional 800 random samples have been generated in No-Crop region of Australia The center part of Australia has been removed to avoid sample because there is almost no possibility for cropland 9

10. Crop Samples Generate samples separately for Crop and No-Crop Regions 106 Crop samples randomly selected from 1,082 ground collected samples to balance the proportion with No-Crop samples (i.e.,36 original +787 out of 800 additional samples) 10

11. Buffer GCE v.2 Map Cropland Area  GOAL: Include omission error  Procedure:  Generate Euclidean distance layer from GCE v.2 Crop to No-Crop  Calculate distance of Crop pixels from No-Crop pixels  Within Australia bound, the distance layer had the range from 0-24 pixels or map units  Two buffer zones (buffer 1 and buffer 2 ) were generated using the range of 0 -1 and 0-2 map units  700 and 800 Random samples have been generated (250x250m) for Buffer 1 and 2 respectively which are proportional to cropland area using Google Earth Imagery 11

12. Buffered GCE v.2 Cropland Map 12

13. Crop Buffer 1 13

14. Samples proportional to Crop Area in Buffer 1 Crop Buffer 0-1 ED Area (sq. m.)Area %No. of SamplesSample % Crop 85318487500015.9022007610317.25293 No Crop451201512500084.09 59785.28571 Total (Buffer Area 1) 5365200000000 700 The Error Matrix *ED- Euclidean Distance in map units 14 Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland55157078.57% No-Crop4858263092.38% Sum Points 103597700 Producer Accuracy 53.40%97.49% 91.00%

15. Crop Buffer 2 15

16. Samples proportional to Crop Area in Buffer 2 Crop Buffer 0-2 ED Area (sq. m.)Area %No. of SamplesSample % Crop 85318487500013.0893314482 11.42061 No Crop566498512500087 718 89.75 Total (Buffer Area 2) 6518170000000 800 *ED- Euclidean Distance in map units 16 Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland58318965.17% No-Crop2468771196.62% Sum Points 82718800 Producer Accuracy 70.73%95.68% 93.13% The Error Matrix

17. Result: The Error Matrix Unbalanced samples Balanced by Crop Proportion 17

18. Crop/No-Crop Accuracy Matrix using Crop Buffers (Population) Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland55157078.57% No-Crop4858263092.38% Sum Points 103597700 Producer Accuracy 53.40%97.49% 91.00% Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland58318965.17% No-Crop2468771196.62% Sum Points 82718800 Producer Accuracy 70.73%95.68% 93.13% Crop Buffer 2 (0-2 Euclidean Distance) Crop Buffer 1 (0-1 Euclidean Distance) 18

19. Conclusion/Summary 19 Class description Histogram SMT-2014 Percent of croplands No of samples % of samples Name#Pixels%#% 1. Croplands, rainfed, SC (season 1 &2), all crops620507145.5% 839 56.4% 2. Croplands, rainfed, SC, pastures636345746.6% 152 10.2% 3. Croplands, irrigated, SC, DC (Season1 &2), all crops1051090.8% 64 4.3% 4. Croplands, irrigated, SC, pastures1893251.4% 63 4.2% 5. Croplands, irrigated, continuous, orchards2372351.7% 64 4.3% 6. Croplands, fallow5507614.0% 141 9.5% 7. Noncropland 165 11.1% Total croplands13650958100.00%1488100.00% Information provided by Pardha about increasing the reference data promotionally with crop area How was the percent of cropland types determined? Classified map or some other independent source of reference The percent of the area covered by non-croplands?? To decide number of No-Crop samples and their distribution ISSUE: The percent of Class 1 and Class 2 are almost same, but the percent of number of samples does not match with this area proportionality – working with Pardha on this now Number of samples for fallow are more than the area extent ISSUE: Methods of augmenting additional reference samples need to discussed

20. Accuracy Assessment: Africa 20

21. Africa Cropland Products  L1: Cropland Extent map, 2014 with Irrigated, Rain-fed, No-crop class labels  L2: Crop intensity, 2014 (Limited reference data)  L3: Crop Dominance, 2003-2014 (Limited reference data)  Agro-ecological layer provided, but this layer was not used for the mapping. 21

22. Training Data for Crop Mapping  Curt Reynold’s Data  Visual interpretation from Google Earth  Ground data from Murali  Corn samples in South Africa  Sugarcane samples  Irrigation samples in Egypt 22

23. Distribution of Ground Collected Validation Samples 23  Irrigated, Rain-fed Samples from Mutlu  Mali Ground Collected Data  LULC Independent Dataset  East Africa Dataset  Samples collected by different independent projects

24. Issue : Some of the “Validation Samples” have been already used for Training e.g., Malawi Data Used in Training 24

25. Issue : Redundant, overlapping and spatially auto-correlated 25 Cleaned-up ground collected Validation Data Still we have uneven distribution of Reference Samples in each zone to perform accuracy assessment

26. Agro-Ecological Zones in Africa ZonesGrowing Days Zone 10 Days Zone 21-59 Days Zone 360-119 Days Zone 4120-179 Days Zone 5180-239 Days Zone 6240-299 Days Zone 7300-364 Days Zone 8365-365+ Days 26

27. Assessment Performed by Ag-Eco Zone  Determination of number of samples needed per zone  Analysis of proportion of crop/no crop by zone  Initial use of hybrid crop probability layer  Evaluated using 50, 100, 150, 200, and 250 samples per zone  Highly dependent on using HSI for the reference data  Produced error matrices by zone and total for Africa Zone 3 27

28. Hybrid Crop Probability Layer (IIASA, Fritz et al. 2015) Zone 1Zone 2Zone 3Zone 4Zone 5Zone 6Zone 7Zone 8 1 (0-10)% 13395501073780023018902907160368117019553801388330455027 2 (10-20)% 555.66270115.311694716642119553521259814558731918.5 3 (20-40)% 1423.556349828976150873048186327467715637546333.8 4 (40-60)% 2613.4742245.13384334662784377042827531589833878.59 5 (60-80)% 5765.3443595.222224425634121921111975038844.9303.831 6 (80-100)% 2598.4368755.913905245781.871798.98418.693049.5299.4985 Weighted crop 9459.3364168401.38659200.8767397.6741642.3426260.9221182.627586.94 Zone Area 13525501102750034087304351530508842028543501891840537726 Weighted % 0.6991.52719.33817.63514.57514.93311.6915.130 Total Crop 12956.452288209.511064371443551.81406111.9898196.69502839.4282534.2195 Crop % 0.9572.61332.45833.17327.63331.46726.57915.348 28

29. Crop/No-Crop Proportion in each zone Zones Crop Samples No-Crop Samples Zone Area (Sq. Km.) Sample Proportion (%) IIASA Total Crop Proportion (%) Zone 1 149 1352550 2 0.957 Zone 2 248 11027500 4 2.613 Zone 3 644 3408730 12.00 32.458 Zone 4 1436 4351530 28.00 33.173 Zone 5 941 5088420 18.00 27.633 Zone 6 446 2854350 8.00 31.467 Zone 7 644 1891840 12.00 26.579 Zone 8 149 537726 2.00 15.348 Sample Proportion % Samples Zone 1Zone 2Zone 3Zone 4Zone 5Zone 6Zone 7Zone 8 502.004.0012.0028.0018.008.0012.002.00 1002.004.0018.3619.1917.78.087.217.69 1502.004.0016.2118.1215.066.716.125.38 2002.004.0015.1516.0812.245.535.084.15 2502.004.0014.5215.2611.795.224.453.37 29 Zone 1Zone 2Zone 3Zone 4Zone 5Zone 6Zone 7Zone 8 IIASA Weighted % 0.6991.52719.33817.63514.57514.93311.6915.130

30. Crop Samples Proportion in Agro-ecological Zones of Africa 30 Asymptote level does not match/reach the IIASA crop Proportion

31. Results: Error Matrices Reference Data CroplandNo-Crop Sum PointsUser Accuracy Map Data Cropland24153961.54% No-Crop1219420694.17% Sum Points 36209245 Producer Accuracy 66.67%92.82% 88.98% Reference Data CroplandNo-Crop Sum PointsUser Accuracy Map Data Cropland11122347.83% No-Crop1819221091.43% Sum Points 29204233 Producer Accuracy 37.93%94.12% 87.12% Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland10 2050.00% No-Crop1019820895.19% Sum Points 20208228 Producer Accuracy 50.00%95.19% 91.23% Zone 3 Zone 4 Zone 5Zone 8 Zone 7 Zone 6 Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland42666.67% No-Crop324124498.77% Sum Points 7243250 Producer Accuracy 57.14%99.18% 98.00% Reference Data CroplandNo-Crop Sum PointsUser Accuracy Map Data Cropland471136.36% No-Crop222823099.13% Sum Points 6235241 Producer Accuracy 66.67%97.02% 96.27% Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland35837.50% No-Crop1027428496.48% Sum Points 13279292 Producer Accuracy 23.08%98.21% 94.86% 31

32. Result: Overall Accuracy for Africa Zones Crop Samples No-Crop Samples Zone Area (Sq. Km.) Overall Accuracy % Zone 1 149 1352550 100.00 Zone 2 248 11027500 100.00 Zone 3 36209 3408730 88.98 Zone 4 29204 4351530 87.12 Zone 5 20208 5088420 91.23 Zone 6 13279 2854350 94.86 Zone 7 6235 1891840 96.27 Zone 8 7243 537726 98 Zones Crop Samples No Crop Samples Total Samples Zone Area (Sq. Km.) Area ratioOverall Accuracy, OA %Area Ratio * OA Zone 3 36209245 3408730 0.18798907888.98 16.72727 Zone 4 29204233 4351530 0.23998383987.12 20.90739 Zone 5 20208228 5088420 0.28062280891.23 25.60122 Zone 6 13279292 2854350 0.15741540894.86 14.93243 Zone 7 6235241 1891840 0.10433365496.27 10.0442 Zone 8 7243250 537726 0.02965521398 2.906211 Total 305126461 91.11872 % 32 Almost No-Crop zones Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland575110852.78% No-Crop571,3281,38595.88% Sum Points 1141,3791,493 Producer Accuracy 50.00%96.30% 92.77% The Error Matrix

33. Conclusion  Each agro-ecological zone (AEZ) had different crop area proportionality which then requires sample distribution calculation  The number of samples in each AEZ stabilizes at different sample sizes  Important to assess accuracy in each zone as results vary based on complexity of the AEZ.  Easy to use HSI for Crop/No Crop Reference Data, BUT to generate or collect Reference Data for validation of Crop Intensity and Crop Dominance Products we will need Ground Collected Data 33

34. Accuracy Assessment of Cropland Products of North America 34

35. Cropland Data for North America  Resampled CDL map of 250m pixel resolution with 7 translated crop types  Zone wise Validation samples for each crop type  Classified mosaic map with 7 crop types  13 Agro-ecological Zone Map 35 Classified Crop Types Alfalfa Corn-Soybean Rice Cotton Potato Wheat-Barley Other Crops

36. Steps to perform accuracy assessment of North America crop type maps:  The accuracy assessment will be performed using the reference data from resampled 250m Cropland Database Layer (CDL) for the year 2008.  The reference data will consist of randomly generated 250m homogeneous samples.  The composite labels (i.e. combined crop types of the classified map) will be compared with the combined CDL reference labels.  The accuracy estimates will be provided for each agro-ecological zone in North America.  The accuracy assessment will be provided in the form of error matrix with overall accuracy, producer’s accuracy and user’s accuracy. 36

37. Step 1: Crop Proportion in each Zone Zone area (Sq. Km.) Total Crop (Sq. Km.) Total Crop % Zone 171 Zone 2203914.752727.65031.33 Zone 31057468.313734.9771.29 Zone 41478721.44102524.126.93 Zone 5739426.24107577.6714.54 Zone 6468497.27129056.6927.54 Zone 7356419.89107180.230.07 Zone 8716200.7125188.217.47 Zone 9732081.03135162.2218.46 Zone 101203562.33233386.1419.39 Zone 11693321.1376521.8911.03 Zone 12125167.2111132.958.89 Zone 138584.862759.638.84 7783436.1521044952.313.42 37

38. Sample Proportion in each Zone Total Crop %No Crop %Crop SamplesNo-Crop SamplesTotal Samples Zone 1 Zone 2 1.33764247798.6623575210478887992 Zone 3 1.29885472798.701145271801370713887 Zone 4 6.93329502393.0667049818524572642 Zone 5 14.5488034185.4511965918710601247 Zone 6 27.54694672.453054159423582 Zone 7 30.0713296369.92867037220513733 Zone 8 17.4794858582.520514151879021089 Zone 9 18.4627403981.537259612039241127 Zone 10 19.3912798880.608720122179251142 Zone 11 11.0370053288.9629946815912861445 Zone 12 8.89446205691.10553794949501044 Zone 13 8.84848236391.1515176463637700 Total 13.4286.581,95831,67233,630 38

39. Crop/No-Crop Error Matrices Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland75138885.23% No-Crop297,8757,90499.63% Sum Points 1047,8887,992 Producer Accuracy 72.12%99.84% 99.47% Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland1454,2984,4433.26% No-Crop359,4099,44499.63% Sum Points 18013,70713,887 Producer Accuracy 80.56%68.64% 68.80% Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland15759174820.99% No-Crop281,8661,89498.52% Sum Points1852,4572,642 Producer Accuracy 84.86%75.95% 76.57% Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland16524541040.24% No-Crop2281583797.37% Sum Points1871,0601,247 Producer Accuracy 88.24%76.89% 78.59% Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland1389323159.74% No-Crop2133035194.02% Sum Points 159423582 Producer Accuracy 86.79%78.01% 80.41% Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland17910828762.37% No-Crop4140544690.81% Sum Points 220513733 Producer Accuracy 81.36%78.95% 79.67% 39 Zone 2 Zone 3 Zone 4 Zone 5 Zone 6 Zone 7

40. Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland11929841728.54% No-Crop6860467289.88% Sum Points 1879021,089 Producer Accuracy 63.64%66.96% 66.39% Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland15521336842.12% No-Crop4871175993.68% Sum Points 2039241,127 Producer Accuracy 76.35%76.95% 76.84% Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland19022941945.35% No-Crop2769672396.27% Sum Points 2179251,142 Producer Accuracy 87.56%75.24% 77.58% Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland12231243428.11% No-Crop379741,01196.34% Sum Points 1591,2861,445 Producer Accuracy 76.73%75.74% 75.85% Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland8131239320.61% No-Crop1363865198.00% Sum Points 949501,044 Producer Accuracy 86.17%67.16% 68.87% Reference Data CroplandNo-CropSum PointsUser Accuracy Map Data Cropland5720926621.43% No-Crop642843498.62% Sum Points 63637700 Producer Accuracy 90.48%67.19% 69.29% 40 Zone 8 Zone 9 Zone 10 Zone 11 Zone 12 Zone 13 Contd.

41. Result: Overall Accuracy Crop Samples No-Crop SamplesNo. of SamplesZone AreaArea % Zone 210478887992 203914.75 2.619879 Zone 31801370713887 1057468.3 13.58626 Zone 418524572642 1478721.4 18.99848 Zone 518710601247 739426.24 9.500084 Zone 6159423582 468497.27 6.019212 Zone 7220513733 356419.89 4.579252 Zone 81879021089 716200.7 9.201684 Zone 92039241127 732081.03 9.405714 Zone 102179251142 1203562.3 15.46326 Zone 1115912861445 693321.13 8.90773 Zone 12949501044 125167.21 1.608137 Zone 1363637700 8584.862 0.110298 Total7783365.2 Reference Data Cropland No- CropSum Points User Accuracy Map Data Cropland1,5836,9218,50418.61% No-Crop37524,75125,12698.51% Sum Points1,95831,67233,630 Producer Accuracy 80.85%78.15% 78.31% Crop SamplesNo-Crop SamplesNo. of SamplesZone AreaArea ratioOverall Accuracy, OA %Area ratio * OA Zone 2 10478887992203914.750.02619999.472.605994 Zone 3 18013707138871057468.30.13586368.89.347348 Zone 4 185245726421478721.40.18998576.5714.54714 Zone 5 18710601247739426.240.09500178.597.466116 Zone 6 159423582468497.270.06019280.414.840049 Zone 7 220513733356419.890.04579379.673.64829 Zone 8 1879021089716200.70.09201766.396.108998 Zone 9 2039241127732081.030.09405776.847.22735 Zone 10 21792511421203562.30.15463377.5811.9964 Zone 11 15912861445693321.130.08907775.856.756513 Zone 12 949501044125167.210.01608168.871.107524 Zone 13 636377008584.8620.00110369.290.076425 Total 336307783365.2175.72815% 41 Overall accuracy with 33,630 samples in 13 Zones

42. Spatial distribution of disagreement in Crop/No-Crop map of North America 42

43. Conclusion/Summary  The accuracy assessment of crop and no-crop class shows that there is a commission error in the Cropland class  Need to check the percent of cropland vs. no-cropland area in North America with reference to CDL  Need to have a Crop/No-Crop extent map  Need to check the accuracy of crop types in each zone based on their area proportionality 43

44. 44

45. 45