Alert button

MoqaGPT : Zero-Shot Multi-modal Open-domain Question Answering with Large Language Model

Add code
Bookmark button
Alert button
Oct 20, 2023
Le Zhang, Yihong Wu, Fengran Mo, Jian-Yun Nie, Aishwarya Agrawal

Figure 1 for MoqaGPT : Zero-Shot Multi-modal Open-domain Question Answering with Large Language Model
Figure 2 for MoqaGPT : Zero-Shot Multi-modal Open-domain Question Answering with Large Language Model
Figure 3 for MoqaGPT : Zero-Shot Multi-modal Open-domain Question Answering with Large Language Model
Figure 4 for MoqaGPT : Zero-Shot Multi-modal Open-domain Question Answering with Large Language Model

Share this with someone who'll enjoy it:

Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon

Multi-modal open-domain question answering typically requires evidence retrieval from databases across diverse modalities, such as images, tables, passages, etc. Even Large Language Models (LLMs) like GPT-4 fall short in this task. To enable LLMs to tackle the task in a zero-shot manner, we introduce MoqaGPT, a straightforward and flexible framework. Using a divide-and-conquer strategy that bypasses intricate multi-modality ranking, our framework can accommodate new modalities and seamlessly transition to new models for the task. Built upon LLMs, MoqaGPT retrieves and extracts answers from each modality separately, then fuses this multi-modal information using LLMs to produce a final answer. Our methodology boosts performance on the MMCoQA dataset, improving F1 by +37.91 points and EM by +34.07 points over the supervised baseline. On the MultiModalQA dataset, MoqaGPT surpasses the zero-shot baseline, improving F1 by 9.5 points and EM by 10.1 points, and significantly closes the gap with supervised methods. Our codebase is available at https://github.com/lezhang7/MOQAGPT.

* Accepted into EMNLP2023 Findings  
View paper onarxiv icon

Share this with someone who'll enjoy it:

Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon