UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities

👀 TL;DR UniversalRAG is a novel RAG framework that retrieves across multiple modalities and granularities by introducing a modality-aware routing mechanism that dynamically identifies the most appropriate modality-specific corpus for each query, effectively addressing the limitations posed by modality gaps and fixed-granularity retrieval.

Concept. Illustration of limitations of existing RAG methods (Left), and the proposed UniversalRAG framework (Right). Specifically: (A) RAG with Single Modality struggles to handle queries requiring modalities other than one in the corpus; (B) RAG with Single Granularity lacks flexibility in granularity, resulting in overly fine or overly coarse information; (C) RAG with Single Unified Corpus causes modality gaps that bias retrieval toward the modality of the query; (D) Our UniversalRAG overcomes these limitations via a modality- and granularity-aware routing mechanism over diverse corpora.

Retrieval-Augmented Generation (RAG) has shown substantial promise in improving factual accuracy by grounding model responses with external knowledge relevant to queries. However, most existing RAG approaches are limited to a text-only corpus, and while recent efforts have extended RAG to other modalities such as images and videos, they typically operate over a single modality-specific corpus.

In contrast, real-world queries vary widely in the type of knowledge they require, which a single type of knowledge source cannot address. To address this, we introduce UniversalRAG, a novel RAG framework designed to retrieve and integrate knowledge from heterogeneous sources with diverse modalities and granularities. Specifically, motivated by the observation that forcing all modalities into a unified representation space derived from a single aggregated corpus causes a modality gap, where the retrieval tends to favor items from the same modality as the query, we propose a modality-aware routing mechanism that dynamically identifies the most appropriate modality-specific corpus and performs targeted retrieval within it.

Also, beyond modality, we organize each modality into multiple granularity levels, enabling fine-tuned retrieval tailored to the complexity and scope of the query.

We validate UniversalRAG on 8 benchmarks spanning multiple modalities, showing its superiority over various modality-specific and unified baselines.

Comparison of average scores across different RAG methods, including baselines and UniversalRAG (Ours). UniversalRAG, by adaptively routing each query to the most suitable modality- and granularity-specific corpus (e.g., no-retrieval, paragraph, document, image, clip, video), consistently outperforms baselines that rely on either a single modality-specific corpus or a unified embedding corpus, highlighting the effectiveness of leveraging multiple modalities of data through adaptive selection.

Overall results comparing diverse RAG methods. Our UniversalRAG is represented by the colored cells, which includes a train-free router approach with GPT-4o, and trained router models with DistilBERT and T5-Large. By adaptively selecting the optimal retrieval modality for each query, UniversalRAG matches the performance of oracle retrieval selection and surpasses all baselines on average.

Effect of retrieval granularity on the performance across two benchmarks: HotpotQA, which requires multi-hop reasoning over documents, and LVBench, whose queries are typically answerable with short video segments. UniversalRAG with granularity consistently outperforms the model without granularity, highlighting that incorporating different levels of granularity enhances UniversalRAG’s ability to adaptively handle a wide range of user questions.

Traditional RAG models are limited by modality-specific retrieval, often missing relevant knowledge from other modalities. UniversalRAG (Ours) dynamically routes each query to the most appropriate modality, enabling more accurate retrieval and response generation.

The user query requires complex retrieval involving multiple entities. ParagraphRAG retrieves limited context centered around a single entity, leading to an incorrect answer. UniversalRAG (Ours) routes the query to the document corpus and retrieves richer document-level information, allowing it to capture both relevant entities and generate the correct response.

The user query requires only a segment of the video to determine the answer. VideoRAG retrieves a broad range of frames across the video, which may include irrelevant content or miss key frames, leading to an incorrect response. UniversalRAG (Ours) routes the query to the clip-level granularity, retrieving more focused and relevant visual information, enabling it to generate the correct response.

BibTeX

@article{yeo2025universalrag,
  title={UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities},
  author={Yeo, Woongyeong and Kim, Kangsan and Jeong, Soyeong and Baek, Jinheon and Hwang, Sung Ju},
  journal={arXiv preprint arXiv:2504.20734},
  year={2025}
}

UniversalRAG

Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities

Abstract

Main Results

Effect of Multi-granularity

Case Study

BibTeX