interrater reliability

n. The level of a test's measurement error attributed to differences in the ratings, scores, or observations provided by different evaluators of the same event or phenomenon. When using people to evaluate or observe an event, there may be differences between true scores and recorded scores due to human error and/ or perception; as such, interrater reliability (also called interscorer or interobserver reliability) is used to examine the extent to which different observers give consistent estimates, evaluations, or ratings of the same phenomenon. While there are various ways of calculating interrater reliability (such as percentage of agreement between raters or use of z score estimates of agreement), the most appropriate method for calculating interrater reliability is application of the kappa statistic, which reports interrater reliability as a proportion of complete agreement, taking into consideration corrections for chance agreement. Interrater reliability based on the kappa statistic ranges from 1 (perfect agreement) to -1 (less agreement than predicted by chance); kappa scores above 0.40 are generally considered satisfactory.

- BJM