Inter-Rater or Inter- Observer Reliability

Description Is the extent to which two or more individuals (coders or raters) agree. Inter- Rater reliability addresses the consistency of the implementation of a rating system.

What value does reliability have to survey research? Surveys tend to be weak on validity and strong on reliability. Surveys tend to be weak on validity and strong on reliability. Survey research presents all subjects with a standardized stimulus, and so goes a long way toward eliminating unreliability in the researcher's observations. Survey research presents all subjects with a standardized stimulus, and so goes a long way toward eliminating unreliability in the researcher's observations.

How used in quantitative research? is generally perceived as a means of verifying coherence in the understanding of a certain topic. not necessarily required to engage deeply into the material in order to obtain an understanding of the study’s findings for rating purposes.

How used in qualitative research? During data collection, inter-observer reliability is demonstrated when two or more researchers' independently collected data mirror each other. During data analysis, it is demonstrated when independently functioning researchers agree upon data segments to be coded, categories to be used, the placing of data segments into the same categories, and the interpretations drawn from examination of classified data segments.

An example of how it is used Let's say you had 100 observations that were being rated by two raters. For each observation, the rater could check one of three categories. Imagine that on 86 of the 100 observations the raters checked the same category. In this case, the percent of agreement would be 86%. Let's say you had 100 observations that were being rated by two raters. For each observation, the rater could check one of three categories. Imagine that on 86 of the 100 observations the raters checked the same category. In this case, the percent of agreement would be 86%.

Direct link to on-line JAE article where reliability are addressed. ull/strokeaha;26/1/46 ull/strokeaha;26/1/46 ull/strokeaha;26/1/46 ull/strokeaha;26/1/46 ERN/pub/technical_reports/isern pdf ERN/pub/technical_reports/isern pdf ERN/pub/technical_reports/isern pdf ERN/pub/technical_reports/isern pdf

References Goodwin, L. D., & Goodwin, W. L. (1984). Are validity and reliability "relevant“ in qualitative evaluation research? Evaluation & the Health Professions, (7) Marques, J. F.,The Application of Inter-rater Reliability as a Solidification Instrument in a Phenomenological Study Retrieved 10/19/06 from Retrieved 10/19/06 from Retrieved 10/19/06 from