AgentFlare

Scholar · multimodal foundation models

Multimodal Foundation Models: Research Digest

10 foundational papers with 1,284 combined citations, plus the latest arXiv preprints.

Foundational papers101,284 citations
Most citedSeed-x: Multimodal models with unifie…385 citations · 2024
arXiv, Sep 202664 papers+8% vs Oct 2025
Newest preprint2026-10-27

AISA Scholar API · arXiv

arXiv submissions per monthPapers matching abs:multimodal AND abs:"foundation model" · last 12 complete months
monthly arXiv submissions: Oct 2025 59 → Sep 2026 64; high 87 (Jun 2026), low 49 (Feb 2026)46617590Oct 2025Mar 2026Sep 2026
402 · Pay-per-Crawl$0.02

You're reading the full report — free for humans and search engines. The same content ships as a structured JSON dataset (normalized facts, sources and citation arrays, no HTML parsing) so AI agents can consume it directly — metered per fetch via the open HTTP 402 payment protocol.

⚡ Agents: fetch .data.json · $0.02 → agents: crawlerAutoPrice=true · verify at https://esa.aisa.one/api/v1/access/verify

What’s new on arXiv

A12026-10-27cs.CV

Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?

Jialv Zou, Xiaotong Li, Zhengrong Yue, Xu Huang, Yaxin Luo, Wenhao Yan, Haodong Li, Wenyu Liu, Xinggang Wang

Discrete visual tokens represent vision in a discrete form analogous to language, offering a native interface for multimodal modeling under a shared next-token prediction objective. However, quantization error is widely assumed to cause unavoidable semantic loss, fundamentally limiting the semantic fidelity of discrete visual tokens and positioning continuous visual representations as an intrinsic upper bound. Using multimodal understanding as a proxy for semantic fidelity, we revisit this assumption and find quantization error unreliable, as discrete representations with substantially larger errors can achieve better understanding. Instead, the effective bit budget of discrete indices emerges as a key determinant of downstream performance. However, existing quantizers eventually saturate as additional codebooks contribute progressively less semantic information. As one approach to alleviating this saturation, we introduce Factorized Vector Quantization, which quantizes complementary, fixed-dimensional views of the visual feature with symmetric parallel codebooks. It continues to improve at larger bit budgets, ultimately matching continuous visual representations in multimodal understanding. Trained on industrial-scale multimodal data, our model achieves state-of-the-art performance among discrete-representation models, outperforming UniAR and Emu3-Chat by 22.8% and 59.2%, respectively, on a nine-metric average. To our knowledge, it is also the first discrete-representation model competitive with strong continuous-representation models such as Qwen3-VL, including on fine-grained OCR tasks. We further identify the key principles and a practical recipe enabling this competitiveness. These results and insights support the potential of discrete visual representations to underpin large-scale native multimodal foundation models. Code and models are available at https://github.com/hustvl/FVQ

A22026-10-26eess.IV

When Slides Met Scans: Cross-Attention for Renal Cancer Prognosis

Anusha Krishnamoorthy, Basudha Pal

Histopathology and computed tomography (CT) provide complementary views of renal cancer. We connect these modalities through bidirectional cross-attention between 16 histology prototypes and up to 32 kidney-centered CT tokens from frozen foundation models. In 210 TCGA-KIRC patients with 46 disease-specific survival (DSS) events, 5-fold evaluation yielded a C-index of 0.760±0.081 and 3-year AUROC of 0.773 (95\% CI 0.678--0.860), the highest averages among evaluated unimodal and fusion baselines. External testing in 180 CPTAC-CCRCC patients yielded DSS C-index of 0.629 with CT withheld. Attribution analysis linked predictions to image regions while exposing acquisition artifacts. These findings support further study of cross-scale fusion and independent multimodal validation.

A32026-10-26cs.LG

TimeWorld: Towards Action-Conditioned Multimodal World Models for Time Series Forecasting

Hang Ni, Weijia Zhang, Hao Liu

Time series forecasting supports decision-making in web applications such as online retail and network management, where planned actions can reshape future dynamics. World models provide a natural paradigm for predicting such action-dependent changes beyond extrapolating historical patterns. However, existing time series world models (TSWMs) typically rely on fixed, domain-specific covariate schemas that limit flexible intervention specification, while multimodal forecasters treat text primarily as passive predictive context. To investigate these gaps, we introduce TSWM-Bench, a cross-domain benchmark for multimodal TSWMs spanning seven real-world datasets, with textual interventions grounded in recorded action--outcome data. Our evaluation reveals that conventional multimodal forecasters underperform history-only time series foundation models (TSFMs) despite dataset-specific training, whereas simple text-prefix adaptations improve forecasting across all evaluated TSFM backbones. These findings suggest that pretrained TSFMs encode implicit world knowledge of temporal dynamics, offering a strong foundation for action-conditioned modeling beyond conventional multimodal fusion. Building on this insight, we propose TimeWorld, a plug-in framework that adapts pretrained TSFMs through explicit action-response modeling. Specifically, the Prototype-Guided Latent Modulation (PLM) module grounds textual actions in temporal dynamics by composing representative response prototypes to modulate future representations based on the action and historical patterns. The Action Contrastive Learning (ACL) component strengthens action-specific response modeling by contrasting these updates under alternative interventions sharing the same history, distinguishing action types while preserving relative intensity relationships.

A42026-10-24cs.CV

PathKGen: Pathology Knowledge Assisted MLLM for Explainable Histopathology

Nadun Rajapaksha, Sandun Herath, Devnith Wijesinghe, Isiri Amani Withanawasam, Hasindri Watawana, Ranga Rodrigo

Whole Slide Image (WSI) analysis serves as the fundamental method for the clinical diagnosis of cancer in digital histopathology. Recent advances in Multimodal Large Language Models (MLLMs) have enabled state-of-the-art (SOTA) automated histopathology report generation. However, MLLMs reason purely through distributions encoded in their neural parameters, limiting the amount of knowledge and reasoning logic they can contain given the model scale. Their black-box reasoning poses another challenge to the human interpretation of intermediate reasoning. We address these issues by utilizing knowledge graphs to inject external structured knowledge for MLLMs to reason over, while making this reasoning component visible. Knowledge graphs are capable of partially externalizing the containment of medical knowledge, an area that remains underexplored in pathology report generation. We introduce PathKGen, a knowledge graph-assisted MLLM for histopathology report generation. PathKGen has two main branches: (1) a visual path that learns visual token inputs to the MLLM using pathology foundation models, and (2) a knowledge path that learns knowledge token inputs by extracting key morphological features of WSIs through querying a pathology-specific knowledge graph. A comprehensive evaluation on WSI-Bench dataset shows that PathKGen surpasses previous SOTA methods while revealing diagnostic reasoning as visually intuitive graphs. Due to its modular design, the proposed knowledge path is complementary to vision-aligned MLLMs and can be extended to other applications by curating domain-specific knowledge graphs to generate knowledge tokens.

A52026-10-23cs.CV

Multimodal Generative AI Assists Comprehensive Echocardiographic Interpretation

Yi Qin, Marawan Elbatel, Shanshan Song, Taoran Huang, Jiarong Guo, Ningshu Li, Lehan Wang, Chenxu Zhao, Jiexin He, Machteld J. Boonstra, St…

Echocardiography is the cornerstone of cardiovascular diagnostics, yet its global utility is constrained by a critical shortage of experienced interpreters. Current artificial intelligence (AI) solutions remain limited, typically operating as opaque "black boxes" or focusing on narrow tasks that fail to capture the holistic, guideline-driven reasoning of a clinician. Here we introduce EchoChat, a generative multimodal vision-language model that performs comprehensive echocardiographic analysis to support clinician interpretation. By emulating the cognitive workflow of a cardiologist, EchoChat integrates vast medical knowledge with a chain-of-thought reasoning mechanism to explicitly ground its conclusions in visual evidence. Trained on echocardiography-instruction pairs from 46,358 examinations, our model establishes a new state-of-the-art across diverse cohorts from six international medical centers. It supports the broadest diagnostic scope of 33 cardiac conditions and findings, including 15 newly added conditions for which it achieves an average F1 of 73.7% across available external evaluations. It surpasses existing best-performing foundation models in cardiac condition recognition (improving average recall by 13.54% on shared tasks) and quantification (average R2 score increase of 0.275 on shared tasks). Furthermore, in a source-blinded echocardiographer reader study, echocardiography reports co-generated with EchoChat were rated higher in diagnostic accuracy and completeness by two senior echocardiographers, and improved diagnostic performance from 72.78% to 82.75% in terms of F1 score over routine-reader only reports. These results demonstrate that integrating clinical reasoning into multimodal AI can democratize specialist expertise and transform the standard of cardiovascular care.

A62026-10-23cs.CL

On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models

Leon Sick, Lukas Hoyer, Yannick Strümpler, Christian Reisswig, Timo Ropinski, Federico Tombari

On-policy self-distillation enables post-training foundation models without relying on stronger external teachers. However, existing approaches rely on ground-truth question-answer pairs, human-curated problem sets, or other supervised signals. In this work, we reformulate on-policy self-distillation as a self-training framework, driven by self-generated question-answer pairs and privileged reasoning demonstrations. We push supervision to the limit: guided solely by coarse, high-level topic descriptions, the model synthesizes its own curriculum of queries and privileged reasoning traces without access to any human-curated prompts or answers. We find that self-distillation on fully synthetic privileged information enables a model to improve across multiple domains while retaining its capabilities on domains it is not self-trained on. We demonstrate the generality of this framework across text and multimodal domains, achieving consistent self-improvement on standard language benchmarks and Visual Question Answering. Our findings demonstrate that foundation models possess rich latent capabilities that can be amplified without external task supervision or curated problem distributions.

A72026-10-23cs.CV

4DMemory: Benchmarking Long-Horizon Spatiotemporal Memory with Harness-Augmented Agents

Yanhao Jia, Yi Zhu, Yiran Zhong, Weigao Sun, Feida Zhu, Pengkun Jiao, Sihan Yang, Ming Ma, Steven Hoi, Erik Cambria

Humans efficiently process multimodal streams spanning hours to years, building persistent, fine-grained spatiotemporal memories that support reasoning about and predicting long-horizon events. In contrast, existing benchmarks focus on short- and medium-context tasks, and most foundation models struggle to preserve fine-grained information over long horizons. To evaluate and improve long-horizon spatiotemporal reasoning, we introduce the LongVideo dataset and the 4DMemoryBench. LongVideo comprises about 900 hours of video and over 5,000 manually reviewed questions covering visual, textual, and spatiotemporal reasoning across diverse context lengths. We further develop the 4DMemory Harness, which integrates four complementary skills, 22+ specialized tool models, and a model-specific retrieval-augmented generation system. Evaluating 13 leading models, we find that standalone foundation models struggle with long-horizon tasks, while the 4DMemory Harness improves all of them by over 50\%. Compared to previous benchmark, the same models exhibit a performance drop of more than 10\% on 4DMemoryBench. We hope 4DMemory will facilitate the development of spatiotemporal agents capable of addressing complex, long-horizon tasks.

A82026-10-23cs.CV

LapaFM: A Laparoscopic Foundation Model via Multi-Stage Continual Learning

Xiaobo Zhu, Shiwei Wu, Yu An, Yuechen Tao, He Sun, Song Zhang, Jianbo Wang, Xiaojing Wang, Junwei Liu, Wei Guo, Yanfei Shao, Xuan Zhao, Jia…

Artificial intelligence-based analysis of intraoperative laparoscopic videos has the potential to improve surgical decision-making and precision. However, effective surgical assistance requires models that can generalize across diverse intraoperative tasks. Here we present LapaFM, a foundation model for laparoscopic surgery trained on more than 20,000 minutes of surgical videos using a multi-stage continual learning strategy that mitigates catastrophic forgetting and inter-task interference. LapaFM learns multimodal visual representations and supports multiple tasks, including smoke removal, surgical phase recognition, instrument and polyp segmentation, and lesion identification. The model was evaluated on 13 retrospective and two prospective cohorts collected from different imaging devices. LapaFM consistently outperformed task-specific models and existing foundation models across datasets, and also generalized to gastrointestinal endoscopy tasks, including polyp detection and classification. Notably, it recovered over 90% of smoke-obscured frames and achieved clinician-level or superior performance in lymph node metastasis prediction and prospective polyp classification.

Foundational papers

  1. 2024
    A survey of resource-efficient llm and multimodal foundation models284 citations
  2. 2024
    Seed-x: Multimodal models with unified multi-granularity comprehension and generation385 citations
  3. 2025
    Mmada: Multimodal large diffusion language models321 citations
  4. 2026
    How well does gpt-4o understand vision? evaluating multimodal foundation models on standard computer vision tasks25 citations
  5. 2026
    Yuvion vl: A multimodal foundation model for adversarial content and ai safety1 citations
  6. 2026-10
    Latest on arXiv: Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?cs.CV
Citations per paperCitation counts via the AISA Scholar API
citations: R1 Mmada: Multimodal lar… 321, R2 A survey of resource-… 284, R3 Seed-x: Multimodal mo… 385, R4 Scaling spatial intel… 82, R5 How well does gpt-4o … 25, R6 Comp: Continual multi… 15, R7 Yuvion vl: A multimod… 1, R8 End-to-end multimodal… 9, R9 Efficient and effecti… 34, R10 Development and valid… 128R1 Mmada: Multimodal lar…321R2 A survey of resource-…284R3 Seed-x: Multimodal mo…385R4 Scaling spatial intel…82R5 How well does gpt-4o …25R6 Comp: Continual multi…15R7 Yuvion vl: A multimod…1R8 End-to-end multimodal…9R9 Efficient and effecti…34R10 Development and valid…128
R12025321 citations

Mmada: Multimodal large diffusion language models

L Yang, Y Tian, B Li, X Zhang, K Shen et al.

… multimodal diffusion foundation models designed to achieve superior performance across diverse domains such as textual reasoning, multimodal … thereby enhancing the model’s ability …

R22024284 citations

A survey of resource-efficient llm and multimodal foundation models

M Xu, W Yin, D Cai, R Yi, D Xu, Q Wang, B Wu et al.

… To support the growth of these large models in a scalable … topics from cutting-edge model architectures and training/serving … posed by large foundation models and to potentially inspire …

R482 citations

Scaling spatial intelligence with multimodal foundation models

Z Cai, R Wang, C Gu, F Pu, J Xu et al.

… Despite remarkable progress, multimodal foundation models … explore scaling up multimodal foundation models to cultivate … modal foundations including visual understanding models (ie, …

How this page is made. Foundational papers and citation counts come from academic search via the AISA Scholar API, with abstracts read from each paper’s page; recent preprints and monthly volumes come from the public arXiv API (metadata CC0). The digest is written by a language model from those abstracts only — every theme links the papers it draws on, and figures not in the inputs are removed.

Sources & citations

  1. Mmada: Multimodal large diffusion language models 321 citations
  2. A survey of resource-efficient llm and multimodal foundation models 284 citations
  3. Seed-x: Multimodal models with unified multi-granularity comprehension and generation 385 citations
  4. Scaling spatial intelligence with multimodal foundation models 82 citations
  5. How well does gpt-4o understand vision? evaluating multimodal foundation models on standard computer vision tasks 25 citations
  6. Comp: Continual multimodal pre-training for vision foundation models 15 citations
  7. Yuvion vl: A multimodal foundation model for adversarial content and ai safety 1 citations
  8. End-to-end multimodal pathology foundation model with clinical dialogue 9 citations
  9. Efficient and effective adaptation of multimodal foundation models in sequential recommendation 34 citations
  10. Development and validation of a multimodal multitask vision foundation model for generalist ophthalmic artificial intelligence 128 citations
  11. Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals? arXiv · 2026-10-27
  12. When Slides Met Scans: Cross-Attention for Renal Cancer Prognosis arXiv · 2026-10-26
  13. TimeWorld: Towards Action-Conditioned Multimodal World Models for Time Series Forecasting arXiv · 2026-10-26
  14. PathKGen: Pathology Knowledge Assisted MLLM for Explainable Histopathology arXiv · 2026-10-24
  15. Multimodal Generative AI Assists Comprehensive Echocardiographic Interpretation arXiv · 2026-10-23
  16. On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models arXiv · 2026-10-23
  17. 4DMemory: Benchmarking Long-Horizon Spatiotemporal Memory with Harness-Augmented Agents arXiv · 2026-10-23
  18. LapaFM: A Laparoscopic Foundation Model via Multi-Stage Continual Learning arXiv · 2026-10-23

Related