0000000000362170
AUTHOR
Mario Paolucci
A simulation of disagreement for control of rational cheating in peer review
Understanding the peer review process could help research and shed light on the mechanisms that underlie crowdsourcing. In this paper, we present an agent-based model of peer review built on three entities - the paper, the scientist and the conference. The system is implemented on a BDI platform (Jason) that allows to define a rich model of scoring, evaluating and selecting papers for conferences. Then, we propose a programme committee update mechanism based on disagreement control that is able to remove reviewers applying a strategy aimed to prevent papers better than their own to be accepted (rational cheating). We analyze a homogeneous scenario, where all conferences aim to the same leve…
Reputation or peer review? The role of outliers
We present an agent-based model of paper publication and consumption that allows to study the effect of two different evaluation mechanisms, peer review and reputation, on the quality of the manuscripts accessed by a scientific community. The model was empirically calibrated on two data sets, mono- and multi-disciplinary. Our results point out that disciplinary settings differ in the rapidity with which they deal with extreme events—papers that have an extremely high quality, that we call outliers. In the mono-disciplinary case, reputation is better than traditional peer review to optimize the quality of papers read by researchers. In the multi-disciplinary case, if the quality landscape is…
A Proposal for Agent Simulation of Peer Review
Peer review lies at the core of current scientific research. It is composed of a set of social norms, practices and processes that connect the abstract scientific method with the society of people that apply the method. As a social construct, peer review should be understood by building theory-informed models and comparing them with data collection. Both these activities are evolving in the era of automated computation and communication: new modeling tools and large bodies of data become available to the interested researcher. In this paper, starting from abstract principles, we develop and present a model of the peer review process. We also propose a working implementation of a subset of t…
Mechanism change in a simulation of peer review: from junk support to elitism
Abstract Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results. While it might seem obvious that science would regulate itself scientifically, the consensus on peer review is eroding; a deeper understanding of its workings and potential alternatives is sorely needed. Employing a theoretical approach supported by agent-based simulation, we examined computational models of peer review, performing what we propose to call redesign, that is, the replication of simulations using different mechanisms. Here, we show that we are able to obtain the high sensitivity to rational cheating that is present in literature. In additio…
Towards an Agent-Based Model for the Analysis of Macroeconomic Signals
This work introduces an agent-based model for the analysis of macroeconomic signals. The Bottom-up Adaptive Model (BAM) deploys a closed Walrasian economy where three types of agents (households, firms and banks) interact in three markets (goods, labor and credit) producing some signals of interest, e.g., unemployment rate, GDP, inflation, wealth distribution, etc. Agents are bounded rational, i.e., their behavior is defined in terms of simple rules finitely searching for the best salary, the best price, and the lowest interest rate in the corresponding markets, under incomplete information. The markets define fixed protocols of interaction adopted by the agents. The observed signals are em…