Scaling Native Multimodal Pre-From Scratch: A Comprehensive Analysis

一、论文概述

项目	内容
标题	Scaling Native Multimodal Pre-Training From Scratch
作者	Not explicitly extracted (references to authors appear in citations)
机构	Not explicitly extracted
论文	arXiv:2607.22043 (https://arxiv.org/abs/2607.22043)
代码	Not explicitly provided in HTML version
发布	July 24, 2026 (based on HTML generation date)
许可	Not specified in available content

二、核心思想

问题定义

Large language models (LLMs) have demonstrated remarkable reasoning and generation capabilities, yet their reliance on text-only pre-training inherently constrains their perception of the multimodal physical world. While multimodal pre-training mitigates this limitation, the current dominant paradigm relies on late fusion — coupling a pre-trained language model with a separately-pretrained vision encoder via a projection layer. This introduces a fundamental asymmetry: vision and language representations are learned independently from distinct data distributions under different optimization objectives.

To resolve this asymmetry, researchers have explored native multimodal pre-training, where models are trained from scratch on interleaved multimodal data. However, a critical practical question remains unanswered: How should computational resources be optimally allocated between model size and training token count when training such models from scratch? Specifically, do native multimodal objectives follow analogous scaling laws to unimodal language models, and do their optimal allocations align or conflict?

解决方案概述

This paper establishes compute-optimal scaling laws for native multimodal pre-training by investigating how resources should be distributed between model size N (activated non-embedding parameters) and total training tokens D under a fixed computational budget C. The work reveals that while both language and multimodal objectives scale predictably, they exhibit fundamentally different allocation behaviors — the language objective’s optimal allocation is largely invariant to data composition, whereas the multimodal objective is highly sensitive to the ratio of multimodal versus text data in the mixture. Through complementary estimation methodologies (IsoFLOP profiles and training-curve envelopes), the authors derive precise power-law relationships that characterize this divergence, enabling construction of a joint Pareto frontier that maps compute budgets and data mixtures to optimal parameter and token configurations.

三、技术架构

整体框架图

The architecture follows a standard decoder-only Transformer design enhanced with Mixture-of-Experts (MoE) components, eliminating traditional vision encoders entirely in favor of direct patch embedding projections. Images are mapped directly into continuous patch embeddings through a single projection layer, then processed alongside text tokens within the shared transformer capacity.

Model Family: Six configurations ranging from A71M to A3B parameters:

Model	Layers	Hidden Size	FFN Hidden Size	Attention Heads	Query Groups	KV Channels	Total Experts	Activated Experts	Expert FFN Hidden	Shared Expert Hidden
A71M	11	640	2048	8	4	128	128	8	256	256
A128M	15	768	2048	12	-	-	128	8	256	256
A340M	19	1152	3072	16	-	-	128	8	384	384
A590M	23	1280	4096	20	-	-	128	8	512	512
A874M	27	1536	4096	24	-	-	128	8	512	512
A3B	48	2048	6912	32	-	-	128	8	768	768

Note: First layer is dense; remaining layers use MoE.

核心公式

Compute Constraint Definition (Equation 1): The computational budget follows the standard approximation from scaling law literature:

$C = 6 N D$

where $C$ represents total FLOPs, $N$ is the number of activated non-embedding parameters, and $D$ is the total number of training tokens. The factor of 6 accounts for forward and backward pass operations.

Optimization Objective: Given the lack of reliable multimodal validation metrics, the authors optimize based on smoothed training loss as a proxy for test loss:

\arg\min_{N,D \text{ s.t. } \mathrm{FLOPs}(N,D)=C} L(N,D) \tag{1}

where $L(N,D)$ denotes the final pre-training loss as a function of model size and token count.

Compute Law (Equation 3): Along the lower envelope of achievable losses per compute, the relationship follows a power law with an irreducible floor:

L(C) = E + \left(\frac{C_c}{C}\right)^{\beta} \tag{3}

where $E$ represents the irreducible loss floor, $C_c$ is the critical compute scale, and $\beta$ is the decay exponent. These parameters are fitted via least squares in logarithmic space using log-sum-exp parameterization to enforce strict positivity.

Allocation Power Law (Equation 2): Both language and multimodal objectives demonstrate that optimal resource allocation adheres to power-law relationships:

N_{\text{opt}}(C) \propto C^{a}, \qquad D_{\text{opt}}(C) \propto C^{b} \tag{2}

where $a$ and $b$ are the scaling exponents governing model size and token count respectively. These exponents differ significantly between the language and multimodal objectives, reflecting their distinct dependencies on data composition.

模型组件

Key Architectural Components:

1. Patch Embedding Layer: Replaces traditional vision encoders; projects raw image pixels directly into continuous patch embeddings. This enables seamless integration of visual features into the shared transformer space without modality-specific preprocessing pipelines.

2. Decoder-only Transformer Base: Standard autoregressive language modeling architecture extended to handle interleaved multimodal sequences.

3. Mixture-of-Experts (MoE): Each non-first layer employs sparse expert activation (8 out of 128 total experts per layer) with separate and shared expert feedforward networks. This provides large parameter capacity while maintaining manageable computation per token during training.

4. Vision Token Masking: During loss computation, only text tokens contribute to the gradient; vision tokens are masked out following practices from prior work. This prevents the multimodal signals from dominating the gradient updates while still allowing effective cross-modal representation learning.

5. Muon Optimizer: The optimization strategy utilizes Muon optimizer with weight decay of 0.1 and gradient clipping threshold of 1.0, combined with a warmup-stable learning rate schedule.

训练流程

Hyperparameters (Table 1 from Appendix):

Setting	Value
Optimizer	Muon
Weight Decay	0.1
Gradient Clipping	1.0
Learning Rate Schedule	Warmup-stable
Global Batch Size	16M sequences
Sequence Length	4096
Warmup Steps	2000

Multi-stage Learning Rates by Model Size:

• A71M: 2.6e-3
• A128M: 1.9e-3
• A340M: 1.1e-3
• A590M: 8.2e-4
• A874M: 6.8e-4
• A3B: 3.0e-4

Data Composition Strategy:

• Fixed text corpus: 250B tokens (web pages, books, academic papers, diverse domains)
• Variable multimodal data: Ratios $r \in \{0, 0.1, 0.2, 0.3\}$ representing proportion of multimodal versus text tokens
• Text-heavy ratios favored at larger scales due to shifting optima

四、核心创新

Innovation	Description	Evidence
Decoupled Objective Analysis	First systematic investigation separating language vs. multimodal scaling behavior within native multimodal pre-training framework	IsoFLOP profiles show distinct parabolic minima patterns for each objective
Composition-Variant Scaling Law	Demonstration that multimodal allocation exponent strongly depends on data mixture ratio, unlike language objective which remains nearly invariant	Empirical measurement shows substantial decline in optimal model size exponent as $r$ increases from 0.1 to 0.3
Joint Pareto Frontier Construction	Methodology mapping trade-off between language loss and multimodal loss across varying data ratios, enabling concrete deployment configurations	Visualized in Figure 7 showing steep trade-off curve
Dual Estimator Framework	Use of both IsoFLOP profiles and training-curve envelopes as independent validation mechanisms reducing artifact concerns	Close agreement between methods confirms results reflect inherent data properties rather than fitting artifacts
Positive Cross-Modal Transfer Effect	Empirical evidence that native multimodal pre-training improves pure-text spatial reasoning tasks despite training including masked vision tokens	SpatialEval accuracy measurements show clear improvement over text-only baseline

五、代码实现分析

The implementation leverages a transformer-based MoE architecture with specialized handling for multimodal data. Key implementation details include:

• Architecture: Decoder-only transformer with MoE layers applied after the initial dense layer, enabling parameter-efficient scaling
• Loss Function: Computed exclusively on text tokens with vision tokens masked during backpropagation
• Learning Rate Schedule: Warmup-stable approach with model-size-specific rates (decreasing with scale)
• Batching: Very large global batch size of 16M sequences facilitates stable gradient estimation at scale
• Sequence Length: Fixed at 4096 tokens accommodating both textual and visual content within single sequence context

六、实验结果

基准测试

Text Benchmark Suite (16 benchmarks, various shot settings):

• Aggregate: MMLU-Redux (5-shot), MMLU-Pro (5-shot), AGIEval-en (3-shot), SuperGPQA (5-shot)
• Coding: HumanEval+ (0-shot), MBPP+ (3-shot)
• Mathematics: GSM8K (4-shot, CoT), MATH (4-shot, CoT)
• Logic Reasoning: BBH (3-shot, CoT), SpatialEval (1-shot)
• Knowledge: NaturalQuestions (5-shot), TriviaQA (5-shot)
• Commonsense Reasoning: Hellaswag (10-shot), SIQA (0-shot), PIQA (0-shot), WinoGrande (5-shot)

Multimodal Benchmark Suite (23 benchmarks):

• Aggregate: MMStar, MMMU, MMMU-Pro, MME, MMBench
• VQA: VQAv2, TextVQA
• STEM: MathVista, MathVerse, ScienceQA
• Document Understanding: HallusionBench, LogicVista, AI2D, ChartQA
• Vision Knowledge: MMBench-CC, SimpleVQA
• Counting: CountBench, CountQA
• Spatial Reasoning: RealWorldQA, CV-Bench, OmniSpatial, SEAM, SpatialEval

消融实验

Text Capability Preservation (Figure 8): Across varying multimodal data ratios $r \in \{0, 0.1, 0.2, 0.3\}$ with fixed 250B text token budget, average accuracy on 16 text benchmarks remains consistently near 44-45% across all ratios, demonstrating that introducing multimodal data does not compromise core language capabilities.

Multimodal In-Context Learning Emergence (Figures 10-13):

• Requires sufficient model scale to benefit from few-shot prompting (3-shot gains negligible at smallest model sizes)
• Gains increase steadily with total training tokens $D$
• Effects concentrate primarily on spatial-reasoning benchmarks
• Training progression shows upward trend in few-shot performance margins

与现有方法对比

Late-fusion approaches (CLIP + LM coupling) suffer from fundamental asymmetry — modalities learn independently from disparate distributions under conflicting objectives. Native pre-training eliminates this by establishing unified representational space from scratch through simultaneous joint optimization of shared parameters. The key quantitative difference revealed is that while language objectives maintain stable scaling exponents regardless of mixture ratio, multimodal objectives exhibit strong sensitivity — requiring more tokens relative to parameters as data becomes denser.

计算前沿分析

The compute frontier exhibits predictable power-law behavior where minimal achievable loss $L(C)$ decreases according to Equation (3). For the language objective, scaling exponents $a$ and $b$ remain approximately constant across different values of $r$, indicating that optimal parameter-to-data ratio stays consistent regardless of accompanying multimodal content. In contrast, for the multimodal objective, increasing $r$ substantially reduces the optimal model size exponent, reflecting that denser multimodal data makes parameter expansion less efficient relative to simply collecting more training examples.

七、相关工作

Multimodal Pre-Training Paradigms: Two primary approaches exist:

1. Late fusion: Coupling pre-trained language models with vision encoders via projection layers (CLIP, SigLip variants). Efficient but suffers from asymmetric representation learning.
2. Native pre-training: Training from scratch on interleaved multimodal data (EMu, other emerging works). Enables deep cross-modal integration but lacked systematic scaling characterization prior to this work.

Compute-Optimal Scaling Laws: Building upon Kaplan et al. (2020) and Hoffmann et al. (2022)‘s foundational work on language model scaling, this paper extends the framework to the multimodal setting with two novel methodological contributions: treating language and multimodal objectives as separate but coupled optimization problems, and deriving the resulting joint Pareto frontier that captures the trade-off inherent in data mixture selection.

八、总结

核心贡献

1. Independent Scaling Analysis: First systematic characterization revealing that language and multimodal objectives follow distinctly different allocation laws within native multimodal pre-training frameworks.

2. Pareto Frontier Formulation: Computationally derived efficient frontier specifying exact combinations of model size, text token count, and multimodal token count for any given compute budget and target data ratio.

3. Empirical Positive Transfer Demonstration: Quantitative evidence that native multimodal pre-training preserves text capabilities while enhancing spatial reasoning abilities and enabling multimodal in-context learning emergence.

4. Methodological Framework: Established dual-estimator protocol combining IsoFLOP profiling with training-envelope extraction providing robust scaling law identification immune to individual run artifacts.

技术影响

This research provides the essential infrastructural foundation for predictably scaling native multimodal foundation models. By quantifying how optimal resource allocation depends on data composition, practitioners can now make principled decisions about model architecture choices, dataset curation priorities, and compute budget distribution when developing new multimodal systems. The observed divergence between language and multimodal scaling behaviors suggests that future work should consider potentially asymmetric investment strategies — maintaining larger model capacities when building heavily multimodal corpora, or prioritizing data collection when focusing on language-centric applications.

局限性

1. Scale limitations: Analysis restricted to models up to 3B active parameters (A3B), leaving open questions about behavior at substantially larger scales (10B+, 100B+).

2. Modality scope: Evaluation limited to single image-text data family; extension to video, audio, and other sensory modalities remains unexplored.

3. Proxy metric dependence: Scaling laws derived from training loss minimization rather than direct task-performance metrics; while justified by lack of standardized multimodal evaluation protocols, ultimate alignment with downstream task quality requires further investigation.

4. Architecture specialization: Results specific to decoder-only transformer with MoE; generalizability to alternative architectural designs (encoder-decoder, hybrid attention mechanisms, etc.) warrants additional study.

九、参考资源

• arXiv Paper: https://arxiv.org/abs/2607.22043
• arXiv HTML Version: https://arxiv.org/html/2607.22043v1
• PDF Download: https://arxiv.org/pdf/2607.22043v1

Key Citations:

• deepmind2026gemini, anthropic2026claude, openai2026gpt — Large language models foundations
• shukor2025scalingnmm — Prior work on scaling native multimodal pre-training
• cui2025emu35 — EMu multi-modality research
• radford2021clip, zhai2023siglip, tschannen2025siglip2 — Vision encoders / late-fusion baselines
• kaplan2020oaiscaling — Original compute scaling laws
• hoffmann2022chinchilla — IsoFLOP and training-envelope methodology

Figures Extracted (from arXiv HTML):

• Figure 1: IsoFLOP curves (language objective) — x1.png
• Figure 2: Training curve envelope (language objective) — x2.png
• Figure 3: Compute-optimal allocation (language objective) — x3.png
• Figure 4: IsoFLOP curves (multimodal objective) — x4.png
• Figure 5: Training curve envelope (multimodal objective) — x5.png
• Figure 6: Compute-optimal allocation (multimodal objective) — x6.png
• Figure 7: Joint Pareto frontier (trade-off between multimodal loss and language loss) — x7.png
• Figure 8: Text capabilities preserved under native multimodal pre-training — x8.png
• Figure 9: Multimodal pre-training enhances pure-text spatial reasoning — x9.png
• Figure 10: Multimodal in-context learning emerges with model scaling — x10.png

Tables (Appendix):

• Table 1: Pre-training hyperparameters across model family
• Tables 2-8: Detailed benchmark performance at different token budgets
• Tables 9-12: Final text performance scaling across model family
• Tables 13-15: Few-shot in-context learning results