Unlock ways to share data on peer review

Peer review is the defining feature of scholarly communication. In a 2018 survey of more than 11, 000 researchers, 98% said that they considered peer review important or extremely important for ensuring the quality and integrity of scholarly communication.

Computational Models That Matter During a Global Pandemic Outbreak

The COVID-19 pandemic is causing a dramatic loss of lives worldwide, challenging the sustainability of our health care systems, threatening economic meltdown, and putting pressure on the mental health of individuals (due to social distancing and lock-down measures). The pandemic is also posing severe challenges to the scientific community, with scholars under pressure to respond to policymakers’ demands for advice despite the absence of adequate, trusted data. Understanding the pandemic requires fine-grained data representing specific local conditions and the social reactions of individuals. While experts have built simulation models to estimate disease trajectories that may be enough to gu…

Hidden connections: Network effects on editorial decisions in four computer science journals

Abstract This paper aims to examine the influence of authors’ reputation on editorial bias in scholarly journals. By looking at eight years of editorial decisions in four computer science journals, including 7179 observations on 2913 submissions, we reconstructed author/referee-submission networks. For each submission, we looked at reviewer scores and estimated the reputation of submission authors by means of their network degree. By training a Bayesian network, we estimated the potential effect of scientist reputation on editorial decisions. Results showed that more reputed authors were less likely to be rejected by editors when they submitted papers receiving negative reviews. Although th…

Fragments of peer review: A quantitative analysis of the literature (1969-2015)

This paper examines research on peer review between 1969 and 2015 by looking at records indexed from the Scopus database. Although it is often argued that peer review has been poorly investigated, we found that the number of publications in this field doubled from 2005. A half of this work was indexed as research articles, a third as editorial notes and literature reviews and the rest were book chapters or letters. We identified the most prolific and influential scholars, the most cited publications and the most important journals in the field. Co-authorship network analysis showed that research on peer review is fragmented, with the largest group of co-authors including only 2.1% of the wh…

Do editors have a silver bullet? an agent-based model of peer review

This paper presents an agent-based model of peer review that looks at the effect of different editorial policies of referee selection. We tested four author/referee matching scenarios as follows: random selection of referees, selection of referees with a similar status to submission authors, selection of higher-skilled and lower skilled referees. We tested these scenarios against three types of referee behaviour, i.e., fair, unreliable and strategic and measured their implications for the quality and efficiency of the process. Results show that in case of randomness of referee judgment, any editorial policy is detrimental for peer review. If referees behave strategically, certain matching p…

Author response: Large-scale language analysis of peer review reports

Assessing Peer Review by Gauging the Fate of Rejected Manuscripts: the case of the Journal of Artificial Societies and Social Simulation

This paper investigates the fate of manuscripts that were rejected from JASSS-The Journal of Artificial Societies and Social Simulation, the flagship journal of social simulation. We tracked 456 manuscripts that were rejected from 1997 to 2011 and traced their subsequent publication as journal articles, conference papers or working papers. We compared the impact factor of the publishing journal and the citations of those manuscripts that were eventually published against the yearly impact factor of JASSS and the number of citations achieved by the JASSS mean and top cited articles. Only 10% of the rejected manuscripts were eventually published in a journal that was indexed in the Web of Sci…

Attitudes of referees in a multidisciplinary journal: An empirical analysis

This paper looks at 10 years of reviews in a multidisciplinary journal, The Journal of Artificial Societies and Social Simulation (JASSS), which is the flagship journal of social simulation. We measured referee behavior and referees' agreement. We found that the disciplinary background and the academic status of the referee have an influence on the report time, the type of recommendation and the acceptance of the reviewing task. Referees from the humanities tend to be more generous in their recommendations than other referees, especially economists and environmental scientists. Second, we found that senior researchers are harsher in their judgments than junior researchers, and the latter ac…

Publishing: Journals could share peer-review data

The effect of publishing peer review reports on referee behavior in five scholarly journals

To increase transparency in science, some scholarly journals are publishing peer review reports. But it is unclear how this practice affects the peer review process. Here, we examine the effect of publishing peer review reports on referee behavior in five scholarly journals involved in a pilot study at Elsevier. By considering 9,220 submissions and 18,525 reviews from 2010 to 2017, we measured changes both before and during the pilot and found that publishing reports did not significantly compromise referees’ willingness to review, recommendations, or turn-around times. Younger and non-academic scholars were more willing to accept to review and provided more positive and objective recommend…

When Competition Is Pushed Too Hard. An Agent-Based Model Of Strategic Behaviour Of Referees In Peer Review

This paper examines the impact of strategic behaviour of referees on the quality and efficiency of peer review. We modelled peer review as a process based on knowledge asymmetry and subject to evaluation bias. We built two simulation scenarios to investigate largescale implications of referee behaviour and judgment bias. The first one was inspired by “the luck of the reviewer draw” idea. In this case, we assumed that referees randomly fell into Type I and Type II errors, i.e., recommending submissions of low quality to be published or recommending against the publishing of submissions which should have been published. In the second scenario, we assumed that certain referees tried intentiona…

The peer review game: an agent-based model of scientists facing resource constraints and institutional pressures

This paper looks at peer review as a cooperation dilemma through a game-theory framework. We built an agent-based model to estimate how much the quality of peer review is influenced by different resource allocation strategies followed by scientists dealing with multiple tasks, i.e., publishing and reviewing. We assumed that scientists were sensitive to acceptance or rejection of their manuscripts and the fairness of peer review to which they were exposed before reviewing. We also assumed that they could be realistic or excessively over-confident about the quality of their manuscripts when reviewing. Furthermore, we assumed they could be sensitive to competitive pressures provided by the ins…