Geographic data validation

Index Basic concepts Why do we need validation? How to assess geographic data Initial checks Intermediate checks Advanced checks Some final considerations

Index Basic concepts Why do we need validation? How to assess geographic data Initial checks Intermediate checks Advanced checks Some final considerations

Basic concepts Quality Faithful representation of a feature Quality of data related to quality of output GIGO principle Data have the potential to be used in ways unforeseen when collected. The value of the data is directly related to the fitness for a variety of uses.

Basic concepts Fitness-for-use The suitability of a set of data for a specific purpose A.K.A. usability Should not be confused with quality Quality: Abstract Usability: Specific Low-quality dataset may be of a high usability

Basic concepts Precision o Closeness of repeated measurements to a given value, either correct or not Accuracy o Closeness of a measurement to the true value

Precision vs Accuracy

Basic concepts Precision o Closeness of repeated measurements to a given value, either correct or not Accuracy o Closeness of a measurement to the true value Precision is an intrinsic value Accuracy depends on knowing the true value of the variable Data validation: assessing the accuracy Compare against a reference value

Index Basic concepts Why do we need validation? How to assess geographic data Initial checks Intermediate checks Advanced checks Some final considerations

Why do we need validation?

This was a striking example, but more subtle issues can (and actually do) happen We need to develop techniques and methodologies to explore the data In other words, we need to validate the data Validating gives a sense of the reliability of the records, and clues on how to improve it

Index Basic concepts Why do we need validation? How to assess geographic data Initial checks Intermediate checks Advanced checks Some final considerations

How to assess? Depending on the aim of the assessment, different techniques Remember that high quality datasets are more likely to show high fitness-for-use Ideally, check for quality If we know the purpose, check for its fitness

How to assess? Work with geographic information a la DarwinCore Work with individual records as well as collections of data Start with the most basic pieces of information Look for coherence with other pieces of information If not, why? Make modifications of information to see if they fit In more advanced levels, make use of available taxonomic or temporal information

How to assess? Tools Spreadsheet: Microsoft Excel, LibreOffice Calc… o Well-known environment o Visually easy Open Refine o Spreadsheet-like, but with some enhanced features Scripts o Database scripts: work directly at the source o Other programming language: enhanced capabilities GIS software o Often linked with other tools, such as spreadsheets or scripts

Visualizations Visual exploration of record set Useful for a first-level assessment Primary visualization for geographic data: maps Next picture has several issues that can be detected using a map…

Coordinate transposition This happens when latitude is stored in longitude field and vice-versa Usually difficult to detect on a one-by-one basis But when looked at the whole picture…

Zero vs Null One of the most common issues Storing 0 (zero) instead of leaving the field empty This happens with some data management systems Latitude 0 and longitude 0 are stored meaning “unknown coordinates” But we do not know that, that is not what the standard says

Negation Forgetting or altering the positive/negative of the coordinates Usually forgetting the minus sign The most common source: transforming from DMS to DD, without taking “W” or “S” into account

Check against country The easiest way of checking these issues is to check if the coordinates fall inside the specified country… Of course, if we have a country value to check against Two ways Use GIS software Use webservices like geonames (we will see this in the openRefine session)

Georeferencing Intermediate check If we have locality information and coordinates, we can check if they match Georeferencing is a tough task, and prone to uncertainties, so some level of imprecision is to be expected Make good use of the “uncertainty” fields in DarwinCore! But still…

, Anahuac NWR (UTC 049) Grandville POINT( ) Marine Nature Study Area 78º 47’ 52” S; 35º 50’ 31” E Stewart Park POINT( ) Backyard My Habitat , Wilderness Park, north of 14th St Delaney Conservation Area 57.3, 11.9

Multi-domain checks Using information from different sources to check quality Especially use taxonomic information to improve geospatial data Most basic example: check data against range map If point falls inside range map of the specified species, OK Sometimes, temporal information is useful

Index Basic concepts Why do we need validation? How to assess geographic data Initial checks Intermediate checks Advanced checks Some final considerations

Considerations NEVER modify the original data Data cleaning is a human task, and thus, it is not error- free Information we believe is wrong may be right Make an “improved copy” of the data Or “flag” the records as inaccurate Re-share the improvements With the community: so that others don’t have to re- invent the wheel With the original owners of the data: so that they can correct the errors at the source