This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Yeaton, W. H. Articles by Wortman, P. M. Search for Related Content PubMed Articles by Yeaton, W. H. Articles by Wortman, P. M. Social Bookmarking                   What's this?

On the Reliability of Meta-Analytic Reviews

The Role of Intercoder Agreement

University of Michigan

Paul M. Wortman

State University of New York

The article addresses the issue of intercoder reliability in meta-analyses. The current practice of reporting a single, mean intercoder agreement score in meta-analytic research leads to systematic bias and overestimates the true reliability. An alternative approach is recommended in which average intercoder agreement scores or other reliability statistics are calculated within clusters of coded variables. These clusters form a hierarchy in which the correctness of coding decisions at a given level of the hierarchy is contingent on decisions made at higher levels. Two separate studies of intercoder agreement in meta-analysis are presented to assess the validity of the model.

Evaluation Review, Vol. 17, No. 3, 292-309 (1993)
DOI: 10.1177/0193841X9301700303


CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

This article has been cited by other articles:

				L. King and B. Snook Peering Inside a Canadian Interrogation Room: An Examination of the Reid Model of Interrogation, Influence Tactics, and Coercive Strategies Criminal Justice and Behavior, July 1, 2009; 36(7): 674 - 694. [Abstract] [PDF]

				M. A. Campbell, S. French, and P. Gendreau The Prediction of Violence in Adult Offenders: A Meta-Analytic Comparison of Instruments and Methods of Assessment Criminal Justice and Behavior, June 1, 2009; 36(6): 567 - 590. [Abstract] [PDF]

				B. Snook, J. Eastwood, P. Gendreau, C. Goggin, and R. M. Cullen Taking Stock of Criminal Profiling: A Narrative Review and Meta-Analysis Criminal Justice and Behavior, April 1, 2007; 34(4): 437 - 453. [Abstract] [PDF]

				S. A. French and P. Gendreau Reducing Prison Misconducts: What Works! Criminal Justice and Behavior, April 1, 2006; 33(2): 185 - 218. [Abstract] [PDF]

				W. N. Welsh and G. Zajac A Census of Prison-Based Drug Treatment Programs: Implications for Programming, Policy, and Evaluation Crime Delinquency, January 1, 2004; 50(1): 108 - 133. [Abstract] [PDF]

				P. Gendreau, C. Goggin, and P. Smith Is the PCL-R Really the "Unparalleled" Measure of Offender Risk?: A Lesson in Knowledge Cumulation Criminal Justice and Behavior, August 1, 2002; 29(4): 397 - 426. [Abstract] [PDF]

				W. H. Yeation, J. C. Langenbrunner, J. M. Smyth, and P. M. Wortman Exploratory Research Synthesis: Methodological Considerations for Addressing Limitations in Data Quality Eval Health Prof, September 1, 1995; 18(3): 283 - 303. [Abstract] [PDF]

				N. A. Day, D. R. Dunt, and S. Day Maximizing Response to Surveys in Health Program Evaluation At Minimum Cost Using Multiple Methods: Mail, Telephone, and Visit Eval Rev, August 1, 1995; 19(4): 436 - 450. [Abstract] [PDF]

				A. J. Petrosino Specifying Inclusion Criteria for a Meta-Analysis: Lessons and Illustrations From a Quantitative Synthesis of Crime Reduction Experiments Eval Rev, June 1, 1995; 19(3): 274 - 293. [Abstract] [PDF]