Social or Individual Disagreement? Perspectivism in the Annotation of Sexist Jokes

The purpose of this paper is to show that the disagreement expressed in the data does not come from individual differences but from diverse and sometimes conflicting, social positions. Using a medium size dataset, 210 sexist jokes and 76 annotators, we test the hypothesis that, from a certain point (size of 12 in our data), adding more subjects to the annotation process does not increase the disagreement. We also measure the attitudes of subjects in sexism, introducing a new scale of Hostile Neosexism, and the consistent or inconsistent behaviour of annotators regarding their attitudes. We propose that perspectives are a combination of attitudes and behaviours, and we explore how they affect inter-rater agreement and which will be the number of annotators that we need to include all the perspectives in an annotation strategy.