Accuracy Assessment of Thematic Maps THEMATIC ACCURACY

What is Accuracy Assessment? Map Accuracy -- The proportion of agreement between a classified map and reference data assumed to be correct. Thematic Precision – The level of detail that is mapped. A map distinguishing lodgepole pine, Douglas fir, Ponderosa, etc. is more precise (but probably less accurate!) than a map just showing Forest. Accuracy and precision are DIFFERENT!

Spatial Accuracy vs. Thematic Accuracy  Thematic accuracy is how well the class names on the map correspond to what is really on the ground.  Spatial accuracy quantifies errors in the locations of boundaries.  Thematic and spatial accuracy are related but are usually treated separately.

Steps for performing thematic accuracy assessment: 1) Develop a sampling scheme 2) Collect reference data 3) Compare reference data to classified map 4) Compute accuracy metrics and deliver to map users

Sampling Schemes  Reference locations (same thing as training sites) must be unbiased  Reference locations must be large enough to find with certainty on your classified image  You must ensure that you can correctly identify the types at your reference sites  You must visit LOTS of reference locations  They MUST be different than the training sites used to create the original classified image.

Collecting Reference Data  Same as for original training site data! Field, other images, personal knowledge, etc.

Quantifying Accuracy – Comparing Mapped Types to Reference Types  Contingency Tables = Error Matrices = Confusion Matrices  Traditional Accuracy Statistics are calculated using the error matrix

Contingency Table or Error Matrix  Error matrix is an n x n array where n is the number of classes  Rows: reference (“correct”) data (n rows)  Columns: mapped classes (n cols)  Note that rows and columns can be switched, so you have to pay attention!

Error Analyses - Example  Create a map with 3 thematic classes (water, forest, and urban)  Collect 95 ground reference data  (Water 33, Forest 39, and Urban 23)  Compare those locations to those places in the map  Generate Error Matrix.

Confusion Matrix Reference data Classified image Water ForestUrbanTotal Water Forest Urban Total

Classification Accuracy  Lots of ways to look at the thematic accuracy of a classification  Overall accuracy  Errors of omission  Errors of commission  User’s accuracy  Producer’s accuracy  Accuracy statistics (e.g., Kappa)  Fuzzy accuracy

Overall Accuracy  Of all of the reference sites, what proportion were mapped correctly?  Easiest to understand but least amount of information for map users and map producers (us).

Overall Accuracy Reference data Classified image WaterForestUrbanTotal Water Forest Urban Total Correctly classified: = 74 Total number reference sites = 95 Overall accuracy = 74 / 95 = 77.9%

Off-diagonal Elements  The off-diagonal elements of a contingency table tell us the most about how to improve our remote sensing classification!  Should spend lots of time examining ERRORS to figure out what went wrong

Errors of Omission  The type on the ground is not that type on the classified image – the real type is OMITTED from the classified image.

Omission Error Reference data Classified image WaterForestUrbanTotal Water Forest Urban Total For water: = / 33 = 36% For forest: = 8 8 / 39 = 20% For urban: = 1 1 / 23 = 4%

Errors of Commission  A type on the classified image is not that type on the ground – the type is COMMITTED to the classified image.

Commission Error Reference data Classified image WaterForestUrbanTotal Water Forest Urban Total For water: = 6 6 / 27 = 22% For forest: = 6 6 / 37 = 16% For urban: = 9 9 / 31 = 29%

Producer’s Accuracy  Map accuracy from the point of view of the map maker (PRODUCER).  How often are real features on the ground correctly shown on the map?

Producer’s Accuracy Can be computed (and reported) for each thematic class Producer accuracy = 100 – omission error Water: 100 – 36 = 64% Forest: 100 – 20 = 80% Urban: 100 – 4 = 96% Or PA = #correct/row total

User’s Accuracy  Accuracy from the point of view of a map USER (not a map maker).  How often is the type the map says should be there really there?

User’s Accuracy Can be computed (and reported) for each thematic class User accuracy = 100 – commission error Water: 100 – 22 = 78% Forest: 100 – 16 = 84% Urban: 100 – 29 = 71% Or UA = #correct/column total

Another example forestbushcropurbanbarewaterunclassProduc er acc forest bush crop urban bare water User acc overall 73.15%

Ok. Now, wing one on the board.

Accuracy Assessment -- Summary  Absolutely critical to any remote sensing based mapping project  Expensive – should be included in the budget from the start  Requires collection of accurate reference data either in the field or from higher resolution data  Analysis should include many aspects of accuracy to give users more information about the product.