BioSpace Collaborative

Academic/Biomedical Research
News & Jobs
Biotechnology and Pharmaceutical Channel Medical Device and Diagnostics Channel Clinical Research Channel BioSpace Collaborative    Job Seekers:  Register | Login          Employers:  Register | Login  

NEWSLETTERS
Free Newsletters
Archive
My Subscriptions

NEWS
News by Subject
News by Disease
News by Date
PLoS
Search News
Post Your News
JoVE

CAREER NETWORK
Job Seeker Login
Most Recent Jobs
Search Jobs
Post Resume
Career Fairs
Career Resources
For Employers

HOTBEDS
Regional News
US & Canada
  Biotech Bay
  Biotech Beach
  Genetown
  Pharm Country
  BioCapital
  BioMidwest
  Bio NC
  BioForest
  Southern Pharm
  BioCanada East
  US Device
Europe
Asia

DIVERSITY

PROFILES
Company Profiles

INTELLIGENCE
Research Store

INDUSTRY EVENTS
Research Events
Post an Event
RESOURCES
Real Estate
Business Opportunities

PLoS By Category | Recent PLoS Articles
Mathematics - Science Policy

Heterogeneity of Inter-Rater Reliabilities of Grant Peer Reviews and Its Determinants: A General Estimating Equations Approach
Published: Wednesday, October 31, 2012
Author: Rüdiger Mutz et al.

by Rüdiger Mutz, Lutz Bornmann, Hans-Dieter Daniel

Background

One of the most important weaknesses of the peer review process is that different reviewers’ ratings of the same grant proposal typically differ. Studies on the inter-rater reliability of peer reviews mostly report only average values across all submitted proposals. But inter-rater reliabilities can vary depending on the scientific discipline or the requested grant sum, for instance.

Goal

Taking the Austrian Science Fund (FWF) as an example, we aimed to investigate empirically the heterogeneity of inter-rater reliabilities (intraclass correlation) and its determinants.

Methods

The data consisted of N?=?8,329 proposals with N?=?23,414 overall ratings by reviewers, which were statistically analyzed using the generalized estimating equations approach (GEE).

Results

We found an overall intraclass correlation (ICC) of reviewer? ratings of ??=?.259 with a 95% confidence interval of [.249,.279]. In humanities the ICCs were statistically significantly higher than in all other research areas except technical sciences. The ICC in biosciences deviated statistically significantly from the average ICC. Other factors (besides the research areas), such as the grant sum requested, had negligible influence on the ICC.

Conclusions

Especially in biosciences, the number of reviewers of each proposal should be increased so as to increase the ICC.

  More...

 

//-->