26. Classification Accuracy Assessment 3/16/01 Accuracy Assessment FR 4262

26. Classification Accuracy Assessment 3/16/01 Shrub & Herbaceous Forest Cultivated Water Urban – % Impervious 100 Landsat Classification of Twin Cities Metro Area Land Cover How accurate is this classification? Miles 4.5 9 18 27 36 FR 4262

26. Classification Accuracy Assessment 3/16/01 Introduction Classification accuracy accuracy of a classified image compared to an independent, reference or standard sometimes called “thematic” accuracy Why do accuracy assessment? FR 4262

History of Accuracy Assessment 26. Classification Accuracy Assessment 3/16/01 History of Accuracy Assessment Largely overlooked with interpretation of aerial photography Time honored skill and assumed correct Quantitative assessment approaches are relatively new and have been primarily associated with classification of digital satellite data Early work started in the mid-1970’s FR 4262

Phases or Levels of Accuracy Assessment 26. Classification Accuracy Assessment 3/16/01 Phases or Levels of Accuracy Assessment “It looks good” Non-site specific comparison of area of each class Site specific assessment by comparison to known areas and determination of overall percent correct Use of an error matrix and statistics derived from it FR 4262

26. Classification Accuracy Assessment 3/16/01 Accuracy vs. Precision Accuracy = agreement between a standard (assumed to be correct) and a classification Precision = detail or specificity; variance Bias = consistent difference between estimated and true values Accurate (unbiased) and precise X X X X X Precise, but not accurate (i.e., biased) Accurate (unbiased), but not precise X X X X FR 4262

Sources of Classification Errors 26. Classification Accuracy Assessment Sources of Classification Errors 3/16/01 Data acquisition errors Sensor performance, stability, view angle, atmosphere Data processing errors Misregistration Scene-dependent errors Resolution, mixed pixels Misclassification Errors of omission and commission Inaccurate reference data Temporal inconsistencies between reference map and imagery Incorrectly identified data used for training the classifier; a sure cause of misclassification If reference data class is incorrect in the map used for accuracy assessment, it will be counted as an error even if the pixels are correctly classified FR 4262

Boundaries and Mixed vs. Pure Pixels 26. Classification Accuracy Assessment 3/16/01 Boundaries and Mixed vs. Pure Pixels FR 4262

Example of Commission and Omission Errors 26. Classification Accuracy Assessment 3/16/01 Example of Commission and Omission Errors Forest Non-Forest True Boundary FR 4262

Example of Commission and Omission Errors, cont. 26. Classification Accuracy Assessment 3/16/01 Example of Commission and Omission Errors, cont. Classified as Non-Forest Classified as Forest FR 4262

Example of Commission and Omission Errors, cont. 26. Classification Accuracy Assessment 3/16/01 Example of Commission and Omission Errors, cont. Forest Non-Forest Non-Forest Classified as Forest Classified as Forest Classified as Non-Forest = CORRECT CLASSIFICATION = INCORRECT CLASSIFICATION = CORRECT CLASSIFICATION True Boundary Errors of Commission (of Forest) and Omission (of Non-Forest) FR 4262

Factors Affecting Classification Accuracy 26. Classification Accuracy Assessment 3/16/01 Factors Affecting Classification Accuracy Parcel size and shape Small and/or narrow parcels have higher proportion of mixed pixels Number and uniformity of classes Spectral-radiometric “contrast” among classes Adequacy of training data Quality of reference data compatibility with imagery in terms of date, scale, resolution, classification system FR 4262

Error Characteristics 26. Classification Accuracy Assessment 3/16/01 Error Characteristics Errors are not randomly distributed spatially or among classes more likely to be systematic and preferential Often errors are not spatially isolated points, but occur in groups of varied size and shape i.e., the entire field or stand may be misclassified Errors may be related to the parcels to which they pertain e.g., may occur at edges of fields FR 4262

Pre-Accuracy Assessment Considerations 26. Classification Accuracy Assessment 3/16/01 Pre-Accuracy Assessment Considerations How is the map information presented? Classification scheme Discrete vs. continuous data Minimum mapping unit? FR 4262

Pre-Accuracy Assessment Considerations 26. Classification Accuracy Assessment 3/16/01 Pre-Accuracy Assessment Considerations Reference data Source (maps, photos, field data, images…) Existing vs. new data Photo vs. ground Resolution vs. classification Assumption of 100% accuracy How to collect? (procedures, cost) When to collect (before, during, after project)? Objectivity and Consistently Independence (equal probability) Collection consistency Inter-interpreter variation Quality control Spatial autocorrelation FR 4262

26. Classification Accuracy Assessment 3/16/01 Using existing data vs. collecting new data Existing maps and data are seldom current May have employed a different classification system Are often of unknown accuracy Proceed with caution when using existing data and maps FR 4262

Accuracy Assessment Sampling Designs 26. Classification Accuracy Assessment 3/16/01 Accuracy Assessment Sampling Designs Area-based assessment Simple random Systematic (simple, offset) Stratified random (by class or other) Stratified systematic (e.g. by quad) Cluster sampling Multi-stage sampling FR 4262

Comparison of Non-site Specific and Site Specific Measures of Accuracy 26. Classification Accuracy Assessment 3/16/01 Comparison of Non-site Specific and Site Specific Measures of Accuracy Classified or Interpreted Images Reference Map Non-Site Specific Site Specific Area of A = 200 Area of A = 198 Area of A = 180 Which is the better classification? Class A FR 4262

Example of Reference Sections for Training and Accuracy Assessment 26. Classification Accuracy Assessment 3/16/01 Example of Reference Sections for Training and Accuracy Assessment Stratified systematic design with sections as sampling units FR 4262

Units of Comparison for Site Specific Accuracy Assessment 26. Classification Accuracy Assessment 3/16/01 Units of Comparison for Site Specific Accuracy Assessment Pixels Clusters of pixels (e.g., 3 x 3 block) Neighborhoods of pixels e.g., crop fields or forest stands “Wall-to-wall” comparisons a problem because of mixed pixels FR 4262

26. Classification Accuracy Assessment 3/16/01 Common Mistakes Inappropriate reference source Resolution, time, quality Samples not representative Too few / unequal probabilities Pre-screening reference areas Sampling homogeneous areas “Fudging” samples “Correcting” data Rejection of “outliers” and “bad data” Autocorrelation Invalid sampling scheme (statistically: bias ) Inter-interpreter variation Non-blind reference data collection Not planning for accuracy assessment Not reporting enough info FR 4262

Comparing Classifications to Training Data 26. Classification Accuracy Assessment 3/16/01 Comparing Classifications to Training Data Not a rigorous (good) approach will likely be biased training data are generally more accurately classified than other (independent) data Instead need an independent sample of reference data to compare classification to FR 4262

26. Classification Accuracy Assessment 3/16/01 Group Exercise Describe a valid sampling scheme to assess the accuracy of a single 2011 Level 2 (i.e. several broad classes with one level of sub-classes) Landsat-based land cover classification of the Twin Cities Metro Area. FR 4262