URBDP 591 A Lecture 17: Mistakes that Scientists Make Objectives Evaluating Empirical Research Learning from Mistakes Mistakes in Research Design Mistakes in Data Analysis Summary of Research Mistakes

Common mistakes in research Formulation of research problem puts off research problem selection until after courses are completed uncritically accepts first research idea that comes along selects a problem that is too broad or too narrow (too broad most common) fails to consider methods and analysis needed for study Research design and methods uses too small a sample size to detect true differences weakens research by changing design in order to make data collection easier attempts research in too short a time span fails to carry out a pilot study or adequately testing instruments and procedures

Reviewing the literature hurries literature review for sake of starting study concentrates on research findings to exclusion of other areas of concern, e.g. methods fails to accurately delimit search process Gathering data incorrect or inadequate sampling methods neglects the power of randomization lacks understanding regarding the instrumentation or variances in measurement Use of statistical tools inappropriate for proposed analysis uses one procedure when several can be applied overemphasizes the importance of small differences that are statistically significant disregards the assumptions needed for statistical tools

Formulation of research problem Research design and methods Reviewing the literature Gathering data Use of statistical tools Types of mistakes in research

Science The standard picture: Experiments and observes Constructs theory Makes predictions Compares the prediction to the experiments and observations Publishes the theory and the experiments

Science in Theory If predicted values  observed values, then there is confidence in the theory If predicted values  observed values, then there is no confidence in the theory Other scientists check the theory and the experiments and propose new experiments and improvements to the theory

Science in Practice In practice, science is more complicated Experiments have errors Experiments may be difficult to replicate Comparisons may be unreasonable Some scientists are more careful than others Some are arrogant, use science to get power

Data collected without a well defined problem Taking existing data and trying to develop a research question Ambiguous objectives Failing to review the literature Ad hoc research A sound basis for research Clear assumptions Recognize limitations Alternative rival hypothesis Common Mistakes in Research

Research Question Characteristics of a problem It should ask about a difference or relationship between two or more variables It should be clearly and unambiguously stated It should be stated in question form (or, alternatively, in the form of an implicit question such as, the purpose of this study was to determine whether…..) It should be testable by empirical methods; that is,it should be possible to collect data to answer the question(s) It should not represent a moral or ethical position

Research proposal Constructing a hypothesis It should talk about the relationship or difference between two or more variables It should be stated clearly and unambiguously in the for of a declarative sentence It should be testable; that is, it should be possible to restate it in an operational form that can then be evaluated based on the data

Good Research Statement of the purpose or problem Rationale and theoretical base Question(s) to be answered Hypotheses or objectives Design and procedure Assumptions Limitations Delimitations Definition of terms

Types of Error in Data Attribute accuracy Positional accuracy Conceptual accuracy Logical accuracy

Positional Accuracy and Precision Applies to both horizontal and vertical positions - x, y, z Function of the scale at which spatial database was created The mapping standards employed by the United States Geological Survey specify that: "requirements for meeting horizontal accuracy as 90% of all measurable points must be within 1/30th of an inch for maps at a scale of 1:20,000 or larger, and 1/50th of an inch for maps at scales smaller than 1:20,000"

r=3.33

Accuracy Standards for Various Scale Maps 1:1,200 ± 3.33 feet 1:2,400 ± 6.67 feet 1:4,800 ± feet 1:10,000 ± feet 1:12,000 ± feet 1:24,000 ± feet 1:63,360 ± feet 1:100,000 ± feet

Spatial objects are in "probable" locations within a certain area

False Accuracy and False Precision Result from interpreting spatial information beyond the levels of accuracy and precision in which they were created

Conceptual Accuracy and Precision Inaccuracies and imprecision may be inherent in the conceptual design of the database. Users may use inappropriate categories or misclassify information. For example, classifying cities by income level would probably be an ineffective way to study atmospheric pollution.

Logical Accuracy and Precision Information can be employed illogically. An example could include performing mathematical operations on categorical data. GIS and statistical packages are typically unable to warn the user if inappropriate comparisons are being made or if data are being used incorrectly

1. Obvious sources of error 2. Errors resulting from natural variations or from original measurements. 3. Errors arising through processing. Sources of inaccuracy in spatial information

Obvious Sources of Error Age of data Areal or Temporal Cover Spatial or Temporal Scale Density of Observations Relevance Format Accessibility Cost

Age of Data Data sources may simply be too old to be useful or relevant to current research projects Past collection standards may be unknown, non-existent,or not currently acceptable Much of the information base may have subsequently changed over time

Areal and Temporal Cover Data on a given area or time frame may be completely lacking, or only partially available Uniform, accurate coverage may not be available User must decide whether further collection of data is required

Spatial and Temporal Scale The ability to show detail in time and space is determined by its scale Scale restricts type, quantity, and quality of data One must match the appropriate scale to the level of detail required in the project

Density of observation The number of observations within an area is a guide to data reliability and should be known by the data user. If the contour line interval on a map is 40 feet, resolution below this level is not accurately possible. If the pixel size of a landsat image is 30 m, resolution below 30 m is not accurately possible. If the temporal resolution of a dataset is a year, monthly resolution is not accurately possible.

An example of surrogate data are electronic signals from remote sensing that are use to estimate land cover. The data is being obtained by an indirect method. Sensors on the satellite do not "see" trees, but only certain digital signatures typical of trees and vegetation. Sometimes these signatures are recorded by satellites even when trees and vegetation are not present. Relevance When the desired data regarding a site or area cannot be obtained, "surrogate" data may have to be used instead. A valid relationship must exist between the surrogate and the phenomenon it is used to study.

Format Methods of formatting digital information for transmission, storage, and processing may introduce error in the data. Examples are: Rasterizing a vector map Vectorizing a raster map Digitizing & scanning

Accessibility Accessibility to data is not equal. What is open and readily available in one country or agency may be restricted, classified, or unobtainable in another. Also access to the quality of data may vary across agencies and data sets.

Costs and Copyrights Extensive and reliable data is often quite expensive to obtain or convert. Copyrights also may limit data access and quality control.

Errors Resulting from Natural Variation or from Original Measurements Positional accuracy Accuracy of content Sources of variation in data

Positional Accuracy Spatial analysts can accurately place well-defined objects and features such as... roads, buildings, boundary lines Less discrete boundaries such as vegetation or soil type may reflect the estimates of the surveyor Many entities lack sharp boundaries in nature and are subject to interpretation Faulty or biased field work, map digitizing errors and conversion, and scanning errors can all result in inaccurate maps for GIS projects.

Errors in quantitative accuracy may occur from faulty instrument calibration used to measure specific features such as altitude, soil or water pH, or atmospheric gases. Mistakes made in the field or laboratory may be undetectable in the a research project unless the user has conflicting or corroborating information available. Accuracy of content

Sources of Variation in Data Variations in data may be due to measurement error introduced by: - faulty observation - biased observers - mis-calibrated or inappropriate equipment

Errors Arising Through Processing Numerical Errors Errors in Topological Analysis Classification and Generalization Problems Digitizing and Geocoding Errors

The Problems of Propagation and Cascading Research projects usually involve operations on many sets of data Inaccuracy, imprecision, and error may be compounded data analyses that employ many data sources - in two ways: Propagation one error leads to another Cascading erroneous, imprecise, and inaccurate information will skew a result

In the absence of a Data Quality Report, ask questions about undocumented data before you use it. What is the age of the data? Where did it come from? In what medium was it originally produced? What is the areal coverage of the data? To what map scale was the data digitized? What projection, coord. system, and datum were used in maps? What was the density of observations used for its compilation? How accurate are positional and attribute features? Does the data seem logical and consistent? Do cartographic representations look "clean?" Is the data relevant to the project at hand? In what format is the data kept? How was the data checked? Why was the data compiled? What is the reliability of the provider?

Components of Data Quality recognized by National Standard for Digital Data Quality 1.Lineage Narrative of source materials used & procedures applied Parameters of projections and transformations 2.POSITIONAL ACCURACY Usually the component identified with "accurate" maps National Map Accuracy Standards: 90% of well-defined points within.02 3.ATTRIBUTE ACCURACY Error in attribute value Categories: reported as misclassification matrix 4.LOGICAL CONSISTENCY Amount that the data fits into the expected structure tests based on internal evidence within database 5.COMPLETENESS Exhaustiveness of coverage

Summary  Learning … from mistakes Before you Start doing research Theoretical Assumptions Checking Sources Collecting Information Research Methods and Statistical Analysis Research Instruments (Survey) Interpreting Results Creating and Maintaining Records Communicating research results  Learning … from mistakes Before you Start doing research Theoretical Assumptions Checking Sources Collecting Information Research Methods and Statistical Analysis Research Instruments (Survey) Interpreting Results Creating and Maintaining Records Communicating research results