NeoMME: an efficient Multimodal-native and Multilingual Encoder
Original reporting by Hugging Face
NeoMME refers to a novel family of multilingual multimodal encoders designed to efficiently process both text and images within a unified architecture. Unlike many contemporary visual language models that rely on separate pretrained vision components and causal language decoders, NeoMME employs a single bidirectional Transformer to process raw image patches and text tokens from scratch. This innovative approach eliminates the computational overhead associated with traditional multi-tower designs, enabling a more streamlined and resource-efficient foundation for multimodal understanding. Trained with a masked discrete-diffusion objective, NeoMME comes in 260M and 800M parameter sizes, offering versatility for various applications.
Optimized Retrieval
This architecture underpins NeoMME-Retriever, a fine-tuned version specifically developed for visual document retrieval. NeoMME-Retriever generates both dense and late-interaction embeddings in a single forward pass, providing flexibility and powerful ranking capabilities. Benchmarks demonstrate that the 260M model achieves top-tier retrieval performance on ViDoRe v3 among models under 800M parameters, while simultaneously offering significantly faster document encoding speeds—nearly doubling the throughput of comparable models. Furthermore, the researchers introduced hierarchical token pooling and asymmetric quantization, drastically reducing the storage footprint for high-resolution late-interaction embeddings by up to 255 times, all while preserving over 95% of retrieval quality. NeoMME models and their checkpoints are openly available on Hugging Face.
NeoMME marks a significant advance in multimodal AI by consolidating image and text processing within a single, long-context bidirectional Transformer, entirely trained from scratch. This innovative architecture, free from the overhead of separate pretrained vision towers or causal language models, yields a highly efficient foundation. The fine-tuned NeoMME-Retriever variant achieves competitive visual document retrieval performance, outperforming similarly sized models while offering impressive encoding throughput and dramatically reducing late-interaction embedding storage for high-resolution documents—a 255x compression at minimal quality loss. By open-sourcing NeoMME and its practical implementations, the project empowers practitioners to integrate these efficiencies into their own systems.
Future Multimodal Landscapes The implications of NeoMME’s design philosophy are profound. Its success in unifying multimodal understanding without relying on large, disparate pretrained components suggests a more streamlined and resource-efficient paradigm for AI development. This approach not only lowers computational costs and energy footprints but also broadens accessibility, making powerful multimodal capabilities attainable for a wider range of applications and organizations. Specifically, NeoMME-Retriever's ability to handle high-resolution visual documents efficiently will be transformative for visual Retrieval-Augmented Generation (RAG) systems, enabling richer, context-aware interactions that leverage intricate visual cues often lost in traditional text-only extraction. As a multilingual, multimodal foundation encoder, NeoMME sets a new benchmark for practical, scalable, and globally applicable AI, pointing towards a future where unified models are the norm, fostering innovation across diverse industries reliant on complex document understanding.
Frequently asked questions
- What is NeoMME and how does its architecture differ from typical visual language models?
- NeoMME is a family of multilingual, multimodal encoders that uses a single bidirectional Transformer to process both text tokens and raw image patches. Unlike many generative visual language models, it does not rely on a separate pretrained vision tower or a causal language model, being trained from scratch with a masked discrete-diffusion objective for efficiency and direct multimodal understanding.
- How does NeoMME enhance the efficiency of large-scale visual document indexing and retrieval?
- NeoMME-Retriever significantly boosts efficiency by offering high-speed document encoding, processing pages nearly twice as fast as comparable models. For late-interaction embeddings, it employs hierarchical token pooling and asymmetric quantization, reducing storage footprint by up to 255 times while maintaining over 95% of retrieval quality, making high-resolution retrieval practical.
- What is visual RAG and how does NeoMME contribute to this advanced information retrieval process?
- Visual RAG (Retrieval-Augmented Generation) retrieves entire document page images, rather than just extracted text, to leverage visual context like layouts and diagrams. NeoMME-Retriever generates the necessary embeddings for these images, enabling accurate retrieval of relevant pages for subsequent processing by visual language models, enhancing the quality of generated answers.