EY's multimodal RAG with knowledge graphs aims to ground enterprise LLMs and improve accuracy

Most enterprise RAG systems only retrieve text, but critical information often lives in charts, diagrams, and engineering drawings. EY has published a multimodal RAG framework that retrieves both text and illustrations and connects them through a knowledge graph. The goal is to produce answers that are more complete, contextualized, and easier to verify.

What happened

EY's framework separates text and visual content into different ingestion pipelines. Text is segmented and enriched with keyword extraction and named-entity resolution. Illustrations are assigned descriptive metadata using captions, nearby text, bounding-box analysis, OCR, and language-model descriptions.

The system stores text and illustrations in separate vector indexes. Each content element becomes a node in a weighted knowledge graph with explicit relationships between passages and illustrations. EY describes three methods for building those relationships: deterministic keyword matching, semantic similarity based on embeddings, and machine-learning-based inference for implicit associations. A "gleaning" process resolves missing links and ambiguous entities.

Retrieval happens in stages. The system first performs a similarity search against the appropriate modality-specific index. It then uses the resulting identifiers to traverse neighboring nodes in the knowledge graph. A multimodal re-ranker orders the retrieved passages and illustrations before inserting them into the LLM prompt. This enables multi-hop reasoning without requiring the LLM to infer every connection.

EY argues that chunking methods, embedding models, relationship-building techniques, reranking strategies, and retrieval scope should be configurable rather than hard-coded. The work grew out of limitations EY encountered in client projects, said Dipanjan Sengupta, EY Global Delivery Services Consulting distinguished technologist and AI engineering leader.

Why AI builders should care

Enterprise RAG systems that ignore visuals miss a large portion of the information in documents. An industrial company may keep essential specifications in engineering drawings, while a life sciences company may rely heavily on graphs. EY's methodology links illustrations back to relevant text, resulting