Multimodal Foundation Models: Research Digest

Literature digest: Multimodal foundation models

Multimodal foundation models (MFMs) are large-scale models trained on diverse data modalities—such as text, image, video, and audio—to serve as general-purpose backbones for downstream tasks. Recent work has shifted from narrow, modality‑specific architectures toward unified, generalist systems that can reason and generate across multiple modalities. This digest synthesizes key themes from recent surveys, evaluations, and application‑oriented studies.

Architectures and generalization

“Multimodal foundation models: From specialists to general‑purpose assistants” surveys the evolution from vision‑language specialists to broad‑purpose MFMs, emphasizing unified architectures and scalable pre‑training. “Towards artificial general intelligence via a multimodal foundation model” proposes a foundation model pre‑trained with large multimodal data via self‑supervised learning, aiming to capture cross‑modal structure for AGI‑relevant capabilities. “Evolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models” traces the hierarchical progression from LLMs to large multimodal models, highlighting architectural and training innovations that enable cross‑modal generalization. “Advances in multimodal adaptation and generalization: From traditional approaches to foundation models” contrasts traditional multimodal settings with modern foundation‑model‑based adaptation, underscoring how MFMs improve transfer and robustness across domains.

Efficiency, evaluation, and specialized domains

“A survey of resource‑efficient LLM and multimodal foundation models” reviews techniques for reducing the computational footprint of MFMs, including efficient architectures and training/serving strategies. “HEMM: Holistic evaluation of multimodal foundation models” introduces a benchmark suite to systematically assess MFMs across diverse capabilities, exposing gaps in current evaluation practices. “InternVideo2: Scaling foundation models for multimodal video understanding” focuses on video‑centric MFMs, showing how multimodal‑friendly encoders and temporal modeling improve video understanding. “Intern‑s1: A scientific multimodal foundation model” presents a scientific‑domain MFM that jointly updates all parameters during multimodal continual pre‑training, aiming to support complex scientific workflows.

Open problems

• Designing truly generalist MFMs that generalize across many modalities and tasks without catastrophic forgetting.
• Developing evaluation frameworks that capture holistic, real‑world capabilities beyond narrow benchmarks.
• Reducing compute and memory costs while preserving or improving multimodal performance.
• Improving adaptation and generalization of MFMs to low‑resource and domain‑specific settings.
• Ensuring robustness, safety, and interpretability of MFMs in high‑stakes applications such as science and healthcare.

Key papers

1. Multimodal foundation models: From specialists to general-purpose assistants — C Li,Z Gan,Z Yang,J Yang,L Li,L Wang…
2. Towards artificial general intelligence via a multimodal foundation model — N Fei,Z Lu,Y Gao,G Yang,Y Huo,J Wen,H Lu…
3. Hemm: Holistic evaluation of multimodal foundation models — PP Liang,A Goindani,T Chafekar…
4. Evolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models. — Z Chen,L Xu,H Zheng,L Chen,A Tolba…
5. A survey of resource-efficient llm and multimodal foundation models — M Xu,W Yin,D Cai,R Yi,D Xu,Q Wang,B Wu…
6. Advances in multimodal adaptation and generalization: From traditional approaches to foundation models — H Dong,M Liu,K Zhou,E Chatzi…
7. Internvideo2: Scaling foundation models for multimodal video understanding — Y Wang,K Li,X Li,J Yu,Y He,G Chen,B Pei…
8. Intern-s1: A scientific multimodal foundation model — L Bai,Z Cai,Y Cao,M Cao,W Cao,C Chen…
9. Vip5: Towards multimodal foundation models for recommendation — S Geng,J Tan,S Liu,Z Fu,Y Zhang
10. Many-shot in-context learning in multimodal foundation models — Y Jiang,J Irvin,JH Wang,MA Chaudhry…

Papers via the AISA Scholar API; synthesis by the AISA LLM layer. 2026-06-23.