AgentFlareScholar · mixture of experts architectures
Mixture-of-Experts Architectures: Research Digest
10 foundational papers with 8,768 combined citations, plus the latest arXiv preprints.
AISA Scholar API · arXiv
You're reading the full report — free for humans and search engines.
The same content ships as a structured JSON dataset (normalized facts, sources and
citation arrays, no HTML parsing) so AI agents can consume it directly — metered per fetch via the open
HTTP 402 payment protocol.
crawlerAutoPrice=true · verify at https://esa.aisa.one/api/v1/access/verify
What’s new on arXiv
CRISP: Co-designing Pruning and Scheduling for Efficient MoE Inference on Edge Servers
Congwei Zhang, Youming Tao, Yifei Zou, Yu Yang, Ruirui Zhang, Xiaoyu Zhang, Falko Dressler, Xiuzhen Cheng, Dongxiao Yu
Mixture-of-Experts (MoE) inference on memory-constrained edge platforms is often dominated by expert-weight movement rather than computation. Although each token activates only a small subset of experts, routing is input-dependent, so all experts must remain accessible and memory movement dominates latency. Existing approaches such as uniform pruning and expert dropping overlook two properties of MoE execution: expert combinations are highly skewed in frequency, and experts contribute asymmetrically within each combination. We present CRISP, a combination-aware framework that co-designs pruning and runtime scheduling for efficient MoE serving on memory-constrained edge platforms. CRISP reorders neurons by importance so one stored expert can support multiple execution widths, assigns differentiated execution to hot, warm, and cold combinations based on frequency and expert role, and exploits heterogeneous widths to overlap expert loading with computation. Across Mixtral-8x7B and DeepSeek-MoE-16B, CRISP improves throughput by up to 2.1× at the same memory budget while limiting accuracy loss to 3.6\% relative to full-width inference. Our results highlight that efficient edge MoE serving requires jointly optimizing which expert capacity is preserved and how expert execution is scheduled.
Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data
Geunhyoung Kim, Sung Ju Hwang
Mixture-of-Experts (MoE) models are bottlenecked by loading routed-expert weights from memory during interactive decoding at small batch sizes. Because most intermediate channels in a routed expert contribute little to any single token, channel selectors predict which channels matter and read only their weights, typically keeping channels above a threshold calibrated on offline data. We show that channel selection needs no calibration data. First, a calibrated threshold does not fix how much a selector reads, because the number of channels passing it varies across inputs; Prox with thresholds calibrated for an 18% read budget actually reads only 12.9% on average across our benchmarks. Second, top-k keeps each token's k highest-scoring channels and needs no calibration data. At the calibrated budget, a selector's top-k version matches or beats its threshold version on the same score in 40 of 43 comparisons at 13% and 18% budgets, including all 23 on generation tasks. The top-k versions also avoid the repetition-loop collapse that calibrated thresholds cause on Gemma 4. Calibrating a threshold thus provides neither of the two things selectors use it for: a known read budget and a consistent accuracy gain at that budget. We furthermore introduce a reader built entirely from the model's weights, which screens channels with a ternary sketch and recovers the skipped ones with a low-rank map. Across three post-trained MoE models, our reader retains 98.1% of full-model accuracy at 17.8% of routed-expert reads, with Triton kernels running MoE blocks 1.60 to 1.90x faster than optimized vLLM at batch one and up to 2.69x at larger batches on the same three models.
Agent-Conditioned Routing: Diverse Reasoning Paths Within One Frozen Model
Ananth Kalyanasundaram, Ankush Kadu, Athira Gopal, Aswanth Krishnan
Generating several solutions can expose alternative methods, but additional trajectories often paraphrase one dominant approach. We investigate whether the conditional capacity of a frozen sparse mixture-of-experts (MoE) model can provide such diversity from one problem prompt, without user-assigned roles, prescribed methods, fine-tuning, or model copies. Agent-Conditioned Routing (ACR) assigns m parallel trajectories persistent, untrained shared/private expert-eligibility masks while preserving the learned, input-dependent token router. On a screened 126-question multi-mode mathematics and physics benchmark, with model judges reading the reported derivations in full, we count the number of problems yielding at least two distinct primary methods among soft-correct solutions in four trajectories. ACR reaches approximately twice as many such problems as the strongest temperature setting under GLM 5.3 (37.3 versus 18.7), and 1.67 times as many under GPT-5.6 Sol (45.00 versus 27.00), at matched output budgets. A blinded four-expert audit on a stratified sample of 30 problems also finds more multi-method solutions with ACR on both backbones, with lower correctness. Matched-eligibility ablations show modest multi-method count advantages for ACR, while token-resampled masks attain higher correctness. An informed sequential prompt elicits more diversity still, using previous solutions and explicit requests for alternative methods. ACR adds about 160 KiB of mask state. In a three-problem microbenchmark capped at 1,024 tokens, its latency and peak VRAM usage remain within 3% of batched temperature sampling.
GradCO: Behavior-Aware Gradient-Conflict Offloading for Text-Based Person Anomaly Search
Zhiheng Fu, Zixu Li, Zhiwei Chen, Jiale Huang, Fangxu Liu, Jinhe Bi, Yupeng Hu
Text-based Person Anomaly Search (TPAS) retrieves a target pedestrian using natural-language descriptions of appearance and normal or anomalous behavior. Unlike conventional text-based person retrieval, TPAS must preserve identity cues while distinguishing fine-grained behaviors. It presents two challenges: same-identity, different-behavior hard negatives cause gradient conflict during granular alignment, while strong appearance cues can obscure weak behavioral evidence during aggregation. We propose GradCO (behavior-aware Gradient-Conflict Offloading), comprising three complementary components. Behavior-Aware Mixture-of-Experts (B-MoE) forms appearance, normal-behavior, and anomalous-behavior streams whose roles are learned from retrieval objectives without expert-specific semantic labels. Conflict Offload Learning (COL) uses pair-conditioned offload anchors in similarity computation to reduce the conflicting-gradient burden on the main text representations. Hierarchical Intent Alignment (HIA) aggregates expert similarities and features using query-conditioned weights. Experiments on five TPAS/TIPR benchmarks show GradCO significantly surpasses SOTA methods and validate our design.
EnergyCal: Taming Loss Spikes in Muon-Family Optimizers for Sparse MoE Training
Masato Fujitake
Sparse mixture-of-experts (MoE) language-model training can exhibit large loss spikes with Muon-family optimizers. We introduce EnergyCal, a lightweight rule that attenuates routed-expert updates according to the normalized squared Frobenius norm of their finite-step Newton-Schulz directions. EnergyCal leaves the spectral transform unchanged and adds no persistent optimizer state. We evaluate it on a 4B-parameter MoE model. Against a development-selected, lower-learning-rate MuonH recipe, EnergyCal-MuonH reduces mean thresholded loss-spike burden by 99.51% and final geometric-mean perplexity by 4.06%, with a modest throughput cost. With plain Muon at matched nominal learning rate and weight decay, the reduction in mean spike burden averages 95.98% across the two tested exponents, accompanied by small endpoint improvements. These findings support stabilization both with and without Hyperball in the evaluated setting.
SpecLST: Spectral-Enhanced Long-Short Term Spatio-Temporal Multimodal Modeling for Traffic Forecasting
Kenan Kang, Matthew M. Y. Kuo, Weihua Li
Modeling multi-scale dynamics in urban road networks is essential for intelligent transportation systems but is often hindered by the computational cost of long-range historical context and the challenge of fusing heterogeneous modalities. Existing spatio-temporal frameworks face a trade-off between sequence length and memory overhead, while frequently neglecting the transient effects of exogenous factors such as road incidents and weather. To address these limitations, we propose SpecLST, a frequency-enhanced long-short-term spatio-temporal multimodal framework. SpecLST leverages a fast Fourier transform-based frequency-domain representation module to efficiently encode multi-week historical dependencies, bypassing the memory bottlenecks of standard attention mechanisms over raw long sequences. Heterogeneous context is integrated through attention-based fusion, followed by a sparse Mixture-of-Experts backbone with graph augmentation for spatio-temporal representation learning. Additionally, a frequency-enhanced residual head isolates high-frequency transients, improving sensitivity to abrupt traffic anomalies. At the 60-minute horizon, SpecLST achieves mean relative error reductions of 14.10% in MAE, 10.35% in RMSE, and 15.78% in MAPE across eight neural forecasting baselines and four datasets, with the dataset- and metric-specific average reductions ranging from 5.63% to 22.21%.
Physical structure as an organizing principle for universal Hamiltonian learning
Mingkang Liu, Zhanghao Zhouyin, Guanghui Wang, Daye Zheng, Lei Shen, Qiangqiang Gu
Density functional theory (DFT) gives the electronic structure of materials, but its cost limits large-supercell calculations and materials screening. Machine-learned Hamiltonians bypass the self-consistent cycle but are usually system-specific; universal ones could cover new compositions and disordered solids. Such models must generalize across chemistry and keep downstream properties physical, while staying inexpensive for large cells. Here we introduce UniTB, a universal E(3) equivariant Hamiltonian representation framework that integrates physical priors rooted in density-functional tight-binding theory with deep learning. UniTB's network, which corrects a prior Hamiltonian, generalizes across chemistry through a mixture of experts routed by that prior and a charge-equilibration head as in self-consistent-charge tight binding. A projection bounds band shifts from the prior, suppressing spurious states and keeping downstream properties physical. We provide the fastest SO(2) tensor-product GPU kernels for the network. The variant UniTB-Flash assembles its prior from two-center tables at O(N) cost for N atoms, against O(Nlog N) on a grid. Trained on 29,242 crystals, UniTB reaches a 0.48 meV matrix-element error on nonmagnetic test crystals spanning 68 elements. Without amorphous training data, both models agree with DFT optics and conductivity of amorphous Ge-Sb-Te. UniTB-Flash ranks candidates with no significant difference from UniTB, whose property values are more accurate; together they screen 16,195 crystals for five application domains of industrial relevance.
DuplexTune: Coupled-State Replay for Memory-Efficient Mixture-of-Experts Adaptation
Paul Raj, Simon Caldwell, Daniel Whitmore, Mutlu Mete
Sparse expert execution does not remove the stored states needed to adapt a mixture-of-experts language model. We present DuplexTune, a framework for replaying coupled hidden states through pretrained attention and expert modules. Trainable width bridges connect two residual streams to the backbone. Bridge warm-up precedes backbone adaptation, with router weights kept frozen. We analyze the implicit state-recovery equation and identify the convergence and replay conditions needed for reconstruction. On Qwen1.5-MoE-A2.7B, the single-H800 evaluation reports 38.8 GB peak memory, versus 64.8 GB for supervised fine-tuning with checkpointing, and throughput of 25.1 versus 19.9 samples/s. DuplexTune achieves 67.2% MMLU, 75.6% GSM8K, and 7.69 MT-Bench, with a weaker multilingual score than the checkpointed baseline. The results characterize a configuration-dependent tradeoff among state storage, throughput, and adaptation quality.
Foundational papers
- 2017
- 2022
- 2025
- 2026
- 2026
- 2026
- 2026
- 2026-10
A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications
S Mu, S Lin
… more efficient and scalable architectures capable of meeting … challenges is the Mixture of Experts (MoE) architecture, which … mixture algorithm called DeepME (Deep Mixture Experts) for …
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
N Shazeer, A Mirhoseini, K Maziarz, A Davis et al.
… We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine …
Towards understanding the mixture-of-experts layer in deep learning
Z Chen, Y Deng, Y Wu, Q Gu et al.
… For the MoE, we consider a mixture of 4 experts with a linear gating network. For the expert/single model architectures, we consider a CNN with 2 convolutional layers (architecture …
A survey of mixture of experts models: Architectures and applications in business and finance
S Joshi
… Since its beginning, the Mixture of Experts architecture has … up of a single, big architecture in which all the elements are … The MoE architecture addresses the problem by making the …
The evolution of mixture-of-experts architectures in large language models: Routing, topology, load balancing, and expert parallelism
J Li
… modern Mixture-of-Experts systems along five coupled dimensions: expert granularity, expert … We describe eight architectural milestones as a dependency graph with six mainline …
More: 3d visual geometry reconstruction meets mixture-of-experts
J Gao, Z Wang, X Fang, X Ren et al.
… We propose MoRE, a dense visual foundation model featuring a mixture-of-experts architecture … In Stage 2, we incorporate mixture-of-experts to further train the model for robust and …
Towards greater leverage: Scaling laws for efficient mixture-of-experts language models
C Tian, K Chen, J Liu, Z Liu, Z Zhang et al.
… Furthermore, our 256-expert architecture features significantly higher sparsity and finer granularity than OLMoE’s, a key structural difference that, along with varying training parameters, …
X-moe: Enabling scalable training for emerging mixture-of-experts architectures on hpc platforms
Y Yuan, A Gupta, J Li, S Dash, F Wang et al.
Emerging expert-specialized Mixture-of-Experts (MoE) architectures, such as DeepSeek-MoE, deliver strong model quality through fine-grained expert segmentation and large top-k …
Infrastructure Economics of Sparse Mixture-of-Experts in Cloud-Native NLP: Benchmarking Cost, Accuracy, and Performance
P Patel
… In conclusion, sparse Mixture-of-Experts architectures represent a significant advancement in NLP infrastructure, turning what was once a purely scaling challenge into an optimization …
Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
W Wan, J Han, W Zhang, C Jin
… architectural optimization that is itself a central contribution of this work: a reusable, cost-efficient experimental paradigm for systematically studying MoE architectural … MoE architectures …
How this page is made. Foundational papers and citation counts come from academic search via the AISA Scholar API, with abstracts read from each paper’s page; recent preprints and monthly volumes come from the public arXiv API (metadata CC0). The digest is written by a language model from those abstracts only — every theme links the papers it draws on, and figures not in the inputs are removed.
Sources & citations
- A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications 244 citations
- Outrageously large neural networks: The sparsely-gated mixture-of-experts layer 7,949 citations
- Towards understanding the mixture-of-experts layer in deep learning 388 citations
- A survey of mixture of experts models: Architectures and applications in business and finance 3 citations
- The evolution of mixture-of-experts architectures in large language models: Routing, topology, load balancing, and expert parallelism 1 citations
- More: 3d visual geometry reconstruction meets mixture-of-experts 15 citations
- Towards greater leverage: Scaling laws for efficient mixture-of-experts language models 39 citations
- X-moe: Enabling scalable training for emerging mixture-of-experts architectures on hpc platforms 14 citations
- Infrastructure Economics of Sparse Mixture-of-Experts in Cloud-Native NLP: Benchmarking Cost, Accuracy, and Performance 115 citations
- Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization 0 citations
- CRISP: Co-designing Pruning and Scheduling for Efficient MoE Inference on Edge Servers arXiv · 2026-10-27
- Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data arXiv · 2026-10-27
- Agent-Conditioned Routing: Diverse Reasoning Paths Within One Frozen Model arXiv · 2026-10-27
- GradCO: Behavior-Aware Gradient-Conflict Offloading for Text-Based Person Anomaly Search arXiv · 2026-10-27
- EnergyCal: Taming Loss Spikes in Muon-Family Optimizers for Sparse MoE Training arXiv · 2026-10-27
- SpecLST: Spectral-Enhanced Long-Short Term Spatio-Temporal Multimodal Modeling for Traffic Forecasting arXiv · 2026-10-26
- Physical structure as an organizing principle for universal Hamiltonian learning arXiv · 2026-10-26
- DuplexTune: Coupled-State Replay for Memory-Efficient Mixture-of-Experts Adaptation arXiv · 2026-10-26