Picture for Sabine Weber

Sabine Weber

Language Models Are Poor Learners of Directional Inference

Add code
Oct 10, 2022
Tianyi Li, Mohammad Javad Hosseini, Sabine Weber, Mark Steedman

Figure 1 for Language Models Are Poor Learners of Directional Inference
Figure 2 for Language Models Are Poor Learners of Directional Inference
Figure 3 for Language Models Are Poor Learners of Directional Inference
Figure 4 for Language Models Are Poor Learners of Directional Inference

We examine LMs' competence of directional predicate entailments by supervised fine-tuning with prompts. Our analysis shows that contrary to their apparent success on standard NLI, LMs show limited ability to learn such directional inference; moreover, existing datasets fail to test directionality, and/or are infested by artefacts that can be learnt as proxy for entailments, yielding over-optimistic results. In response, we present BoOQA (Boolean Open QA), a robust multi-lingual evaluation benchmark for directional predicate entailments, extrinsic to existing training sets. On BoOQA, we establish baselines and show evidence of existing LM-prompting models being incompetent directional entailment learners, in contrast to entailment graphs, however limited by sparsity.

* Findings of EMNLP 2022 
Viaarxiv icon
Github IconAccess Paper or Ask Questions
Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon

Cross-lingual Inference with A Chinese Entailment Graph

Add code
Mar 11, 2022
Tianyi Li, Sabine Weber, Mohammad Javad Hosseini, Liane Guillou, Mark Steedman

Figure 1 for Cross-lingual Inference with A Chinese Entailment Graph
Figure 2 for Cross-lingual Inference with A Chinese Entailment Graph
Figure 3 for Cross-lingual Inference with A Chinese Entailment Graph
Figure 4 for Cross-lingual Inference with A Chinese Entailment Graph

Predicate entailment detection is a crucial task for question-answering from text, where previous work has explored unsupervised learning of entailment graphs from typed open relation triples. In this paper, we present the first pipeline for building Chinese entailment graphs, which involves a novel high-recall open relation extraction (ORE) method and the first Chinese fine-grained entity typing dataset under the FIGER type ontology. Through experiments on the Levy-Holt dataset, we verify the strength of our Chinese entailment graph, and reveal the cross-lingual complementarity: on the parallel Levy-Holt dataset, an ensemble of Chinese and English entailment graphs outperforms both monolingual graphs, and raises unsupervised SOTA by 4.7 AUC points.

* Accepted to Findings of ACL 2022 
Viaarxiv icon
Github IconAccess Paper or Ask Questions
Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon