Peer evaluation with graph neural networks

Abstract

Peer assessment systems are rising in various social contexts, such as peer grading in large (online) classrooms, peer review in conferences, peer art evaluation, etc. However, peer assessments might not be as accurate as expert assessments, thus rendering these systems unreliable. Peer assessment systems’ reliability is influenced by factors such as peers’ assessment ability, manipulation and strategic assessment behaviors, and the peer assessment setup (e.g., peer assessing group work or individual work of others). In this work, we first model peer assessment as multi-relational weighted networks that can represent a variety of peer assessment setups, and capture conflicts of interest and strategic behaviors. Leveraging our peer assessment network model, we introduce a graph convolutional network which can learn assessment patterns and user behaviors to more accurately predict expert evaluations. Our extensive experiments on real and synthetic datasets demonstrate the efficacy of our proposed approach, which outperforms existing peer assessment methods.