JOKER

CLEF 2023 JOKER Track:

Automatic Wordplay Analysis

Tasks, evaluation setup, and metrics

In the edition, we propose three shared tasks:

Task 1: Pun detection in English, French, and Spanish. We will use the corpora which were a side result of JOKER-2022 [1] and the augmented data of SemEval-2017 [2] as well as standard evaluation metrics for binary classification (precision, recall, accuracy)
Task 2: Pun location and interpretation in English, French, and Spanish. We will use the extended corpus of JOKER-2022 Task 1. We will evaluate location based on the accuracy and the interpretation based on paronyms and synonyms.
Task 3: Pun translation from English to French and to Spanish. We will use the extended version of the corpus of JOKER-2022 Task 3 and we will continue to evaluate the generated translation manually.
Open Task: in English, French, and Spanish. In addition, we encourage the use of our data for tasks related to computational wordplay and humour. These could take the form of experiment reports on humour perception, evaluation, wordplay generation, or user studies. Register for any of the aforementioned shared task if you are interested in this open task.

References

[1] L. Ermakova et al., “Overview of the CLEF 2022 JOKER Task 3: Pun Translation from English into French”, in Proceedings of the Working Notes of CLEF 2022: Conference and Labs of the Evaluation Forum, 2022.
[2] T. Miller, C. Hempelmann, and I. Gurevych, “SemEval-2017 Task 7: Detection and Interpretation of English Puns”, in Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada, 2017, p. 58‑68. doi: 10.18653/v1/S17-2005.

This project has received a government grant managed by the National Research Agency under the program "Investissements d'avenir" integrated into France 2030, with the Reference ANR-19-GURE-0001.

JOKER is supported by The Human Science Institute in Brittany (MSHB)