Microsoft Reveals MAI-Base-2 Training Recipe: 2.8 Trillion Parameters Across 16,000 GPUs in 38 Days

The Microsoft AI team has disclosed the pretraining engineering details of its next-generation frontier foundation model, MAI-Base-2. This Mixture-of-Experts (MoE) model features 80 billion active parameters and 2.8 trillion total parameters. On a cluster of 16,000 NVIDIA GB200 GPUs, the team completed full-scale training on 50 trillion tokens over 38 days, integrating native visual understanding directly into the base architecture.

Total parameters and training volume expanded, yet each token still activates only a small fraction of parameters; vision was also integrated into the base model starting from the pretraining stage.

In ultra-large-scale training, approaches selected in small-scale experiments may overfit once scaled up; expert architectures that theoretically conserve compute can also become bottlenecked by VRAM allocation and cross-GPU communication. Microsoft addressed these issues in an interconnected manner: first having small experiments reproduce the data repetition patterns of large-scale training, then redesigning expert scheduling, the optimizer, and the data pipeline. What the report discloses is not merely model scale, but how these choices shaped actual training.

How Does a Pretrained Base Model Differ from a Chatbot?

Many people who use AI daily are accustomed to chatting with ChatGPT or Copilot, easily conflating "releasing a large model" with "upgrading a chatbot." However, MAI-Base-2 is a pretraining base model (Base Model) that builds representations of language and world knowledge primarily through self-supervised learning on massive corpora, without yet undergoing post-training procedures such as supervised fine-tuning (SFT) or reinforcement learning (RL).

The intuitive distinction lies in the interaction paradigm: the core objective of a pretraining base model is to predict the next token based on preceding context, excelling at continuing text from inputs; conversational models used daily, by contrast, undergo post-training guidance to understand instructions and respond in structured question-and-answer format. Microsoft's own previous generation serves as an example: MAI-Base-1 became MAI-Thinking-1 following post-training.

Base models can be evaluated on downstream tasks, but directly comparing cross-entropy loss across different models is confounded by variations in tokenizer granularity—for the exact same text, different tokenizers produce varying token counts, making direct per-token loss comparisons unfair.

To address this, Microsoft adopted a tokenizer-agnostic metric in its evaluations: Bits-Per-Byte (BPB).

(BPB normalizes the model's text prediction loss by the original byte count, measuring the theoretical information bits required per byte when the model compresses unseen text. A lower BPB indicates that the model predicts the probability distribution of the test set more accurately.)

On independent held-out test sets spanning Code, custom third-party Question Answering (QA), Science and Engineering (STEM), and Mathematics (Math), MAI-Base-2 achieved BPB scores of 0.2203, 0.4162, 0.3653, and 0.3561, respectively, outperforming the other baseline models included in the comparison.

How Was "4 to 12 Times Higher Efficiency" Calculated?

The 4.2x to 12.3x reported by Microsoft refers to the computational efficiency gain of the training recipe, benchmarked against DeepSeek-V4-Pro-Base.

Actual training costs cannot be inferred directly from FLOPs, and MAI-Base-2 ultimately consumed significantly more training FLOPs than DeepSeek-V4-Pro-Base. "4.2–12.3x" is an iso-BPB compute efficiency (FLOPs-Efficiency) comparison provided by Microsoft based on fitted curves.

Directly comparing final evaluation results can easily conflate the efficiency of a training recipe with total compute consumed. To assess recipe efficiency independently, the research team fitted separate scaling law curves across models of varying sizes, projecting the theoretical compute required by the recipe to reach the same BPB as DeepSeek-V4-Pro-Base: approximately 1/12.3 of the latter on code tasks, 1/4.2 on QA tasks, and roughly 1/8.3 and 1/6.6 on STEM and math tasks, respectively. Compared to the previous-generation MAI-Base-1, recipe efficiency improved by 6.3x to 12.7x.

This fitted curve includes the final empirical measurement point of MAI-Base-2, placing the baseline comparison point within the fitted range of known models (making it an interpolation rather than an extrapolation). Rather than reducing total compute expenditure, Microsoft leveraged the more efficient recipe to train a larger model totaling 2.8 trillion parameters.

Small Models Must Also Undergo the "Repeated Learning" of Large Models

Before embarking on ultra-large-scale training, engineering teams need to explore architectures and hyperparameters on smaller models. To this end, Microsoft constructed a multi-scale experimental ladder while maintaining a fixed TPP (Training Tokens per active Parameter) across rungs, scaling model parameter counts and training data volumes proportionally together.

However, utilizing the entire dataset directly in small-scale experiments introduces distortion in data repetition epochs.

For instance, suppose a specific data source is repeated 8 times during full training on 50T tokens, while a small-scale experiment has a budget of only 5T tokens. If the full dataset is supplied directly to the small model, it might not even complete a single pass over that data source. Lacking multi-epoch repetition, the small-scale experiment would favor under-regularized architectures or hyperparameter configurations, resulting in overfitting when scaled up to large-scale, multi-epoch training.

To resolve this bias, Microsoft implemented a Source Scaling mechanism: in lower-tier small model experiments, unique sample volume is artificially constrained in proportion. Taking that data source as an example, a 5T token small-scale experiment samples only 1/10 of its unique data while cycling through it 8 times, thereby maintaining a stable ratio of unique data to parameters under fixed TPP.

Beyond data scale, hyperparameter selection is similarly complex. In large model training, multiple hyperparameters often interact, such that altering one shifts the optimal value of another. Taking learning rate (LR) and weight decay (WD) as examples, the former governs per-step update magnitude while the latter constrains weight growth; in Microsoft's experimental setup, model performance proved most sensitive to their product, LR × WD. This interaction causes high-performing hyperparameter combinations to concentrate tightly within a narrow, tilted diagonal band on the (log LR, log WD) plane.

Searching along a standard rectangular grid on the (log LR, log WD) plane leaves many experimental points outside this narrow band, wasting compute while failing to densely sample optimal combinations. The team therefore transformed the search coordinate system, laying out grids in (log LR, log (LR × WD)) space to cover the favorable configurations concentrated in the band more effectively. By training models across diverse model sizes and TPPs and fitting two-dimensional quadratic surfaces to evaluation results to derive analytical extrema, they formulated an empirical rule for MAI-Base-2 hyperparameters:

LR* × WD* = 17.04 / steps

log LR* = −1.51 + 0.175 log TPP − 0.573 log steps

where steps represents total training steps, and TPP is training tokens per active parameter. In the formula, LR* × WD* is inversely proportional to training steps, reflecting that the timescale of decoupled weight decay corresponds to a fixed fraction of training steps; meanwhile, the exponent -0.573 for LR* with respect to steps is close to -0.5, remaining consistent with the empirical rule where doubling batch size typically corresponds to scaling learning rate by √2 when other training conditions remain unchanged.

If Only a Small Fraction of Parameters Is Used, Why Can It Still Run Slowly?

MAI-Base-2 features 102 layers. Each layer contains 1 persistent shared expert and dynamically activates 7 out of 512 routed experts. This means a token activates only a subset of computational branches as it passes through a layer; across the network's 2.8 trillion total parameters, a single token actually activates approximately 80 billion parameters.

When selecting architectures, Microsoft fixed the target at identical evaluation loss and compared training compute (FLOPs) and wall-clock time separately. Choices were primarily guided by wall-clock time because it reflects the throughput existing clusters can actually deliver; if a design saves compute but runs slower, the discrepancy points to where underlying infrastructure must subsequently be optimized.

Sparse MoE accommodates larger total parameter capacity with fewer FLOPs per step. Prior modeling experiments demonstrated that placing MoE on every layer saves more compute than alternating dense FFN and MoE layers; in actual hardware execution, however, dynamic routing induces VRAM fragmentation, while fine-grained experts heighten scheduling and inter-GPU communication overhead. In previous-generation MAI-Base-1, limited underlying MoE implementation efficiency meant that alternating MoE and dense FFN every other layer was actually optimal in wall-clock time. For MAI-Base-2, the team resolved VRAM utilization and expert scheduling bottlenecks, lifting Model FLOPs Utilization (MFU) in MoE compute layers from 24% to 35%, making it viable to deploy MoE across all 102 layers. Across 38 days of full training, average cluster-wide MFU reached 21% (benchmarked against each GB200's theoretical BF16 peak of 2.5 PFLOPS, excluding activation recomputation and optimizer operations), with a goodput of 89% (measuring the proportion of time training progressed productively). These correspond to effective computational efficiency and effective operational uptime, respectively, and must not be conflated with raw GPU utilization.

A key improvement boosting MoE layer efficiency was multi-round dropless MoE (Multi-round Dropless MoE) scheduling.

MoE routers learn from data, resulting in uneven loads across experts. When certain experts receive too many tokens, VRAM can easily overflow. In traditional practices, setting a fixed threshold to drop overflowing tokens degrades model training performance according to experiments; conversely, provisioning excessive VRAM for each expert based on worst-case imbalances leaves memory idle under normal conditions, crowding out memory that could otherwise reduce activation recomputation.

Microsoft's approach divides experts on each Expert Parallel Rank into fixed expert groups (Expert Groups), assigning each group a fixed single-round token capacity:

1. Intra-group capacity sharing: When an expert receives more tokens than its capacity, it first borrows idle capacity from other experts in the same group.
2. Multi-round pipelined fallback: If total expert group capacity still cannot accommodate all tokens, overflow tokens proceed to supplementary rounds for continued processing, avoiding token dropping. Supplementary rounds incur additional compute and lower MFU.
3. Communication-computation overlap: Dispatch, compute, and collect are pipelined and overlapped across expert groups and rounds. In forward propagation, aside from partially exposed latency in initial dispatch and final collect, communication largely runs in parallel with expert computation and shared expert computation; in backward propagation, input dispatch overlaps with layer activation recomputation, while gradient collection overlaps with weight gradient computation.

Expert token capacity itself represents a configurable trade-off: allocating larger capacity buffers consumes more VRAM, whereas setting capacity too low risks increasing supplementary rounds and extra compute; in actual training, this capacity is typically tuned to complete within a single round under steady-state conditions.

Additionally, the team initiated All-to-All communication directly from GPUs, swapped tokens over NVLink via symmetric memory, and fused per-layer load balancing loss into a single global reduction, reducing underlying scheduling overhead brought by fine-grained experts.

How to Make Parameter Updates Faster and More Stable?

In large-scale cluster training, optimizer design and numerical stability are critical to ensuring smooth training progression.

In MAI-Base-2, Microsoft adopted the Muon optimizer for 2D matrix parameters outside embedding layers, while retaining AdamW for parameters such as embedding layers. When scaling to this magnitude, the team found that compared to AdamW, Muon delivered more stable training dynamics and substantial efficiency gains. At the heart of Muon is the Newton–Schulz iteration algorithm, which orthogonalizes momentum for weight matrices through multiple matrix multiplications.

However, this encountered a core conflict in large-scale distributed training: to conserve memory, ZeRO architectures shard optimizer momentum across GPUs; yet the Newton–Schulz algorithm relies on full 2D matrix multiplications and cannot execute directly on isolated shards. To address this, the team implemented batched momentum resharding during optimizer steps: each GPU first updates FP32 momentum shards locally and casts them to BF16, then reconstructs full momentum matrices via batched All-to-All exchanges so each GPU holds a subset of complete matrices; GPUs then locally execute symmetry-aware Newton–Schulz kernels to perform orthogonalization; finally, a second round of All-to-All sends updated results back to original shards, which are cast back to FP32 and applied to weights.

Following optimizations such as resharding metadata caching and prefetching, the Muon phase averaged 165 ms, accounting for 2.1% of iteration step time. Theoretical limits estimated from communication bottlenecks stood at roughly 55 ms, with empirical time at about 3x the theoretical figure (the gap primarily stemming from straggler nodes, kernel launch overhead, load imbalances, and memory copies).

Another technique stabilizing training was activation gradient clipping (Activation Gradient Clipping). In standard pretraining, clipping is typically applied to the global norm of weight gradients across an entire batch. When rare anomalous samples appear in a batch, global clipping scales down normal sample gradients across the batch proportionally.

To mitigate this, Microsoft applied clipping to activation gradients (rather than activation outputs themselves) at specific positions in the network. This localized constraint helps alleviate the collateral reduction of whole-batch gradients caused by outlier gradients. In actual training, combined with optimization recipes including Muon, gradient norm fluctuations for MAI-Base-2 narrowed significantly compared to the prior generation, though median absolute gradient values were not uniformly lower throughout. Training loss curves in the report also indicate that generation two progressed more smoothly than generation one across full training; this represents an observation resulting from multiple combined recipe shifts.

Efficiency gains from this technique grew progressively with model scale, rising from approximately 2% on an 18-layer model to 28% on a 42-layer model.

Moreover, activation gradient clipping helped the team detect data anomalies. For example, in code corpora, an anomalous sample triggering clipping revealed Apache open-source license text unexpectedly truncated at distributed under, immediately followed by a concatenated pull request title # PR Title. The team utilized this to pinpoint and resolve a segmentation defect in the preprocessing pipeline.

Data Foundation: How Was the 50 Trillion Token Mix Assembled?

In data construction, the Microsoft report emphasized an in-house data pipeline and data provenance, stating that training data originated entirely from native collection and internal cleaning, without using distilled data from other models.

Collecting more web pages does not equate to learning proportionally more new information. For instance, if multiple similar web pages repeatedly recount the same event, page counts increase while net new information remains limited. Microsoft improved web parsing, quality classification, topic bucketing, and deduplication: cleaning web content, evaluating quality and topic affiliation, and curtailing duplicate material. On top of this, the team employed embedding-based diversity sampling to prioritize novel data based on semantic characteristics, further curbing repetition among similar content.

Compared to MAI-Base-1, unique tokens in the MAI-Base-2 corpus expanded by approximately 36%, while total training throughput reached 50 trillion tokens (including multi-epoch repetition).

Code remained the largest source; while its share in the training mix declined from 55% to 45%, unique token volume increased by 34%—a shrinking share did not mean less content. Across the entire 50 trillion token training pool, data source proportions were: code 45%, web content 12%, vision-only 14%, STEM 9%, PDF 9%, books 5%, math 4%, and other 2%.

The 14% shown in the chart corresponds to vision-only tokens, whereas the report notes that VLM (vision-language multimodal) corpora accounted for 18% of the 50 trillion total training volume (equivalent to roughly 9 trillion training tokens including repetition), containing 4.5 trillion unique VLM tokens. The two metrics use different accounting scopes; interleaved image-text documents and similar data may simultaneously fall under other sources in categorical breakdowns.

Unlike the text-only architecture of previous-generation MAI-Base-1, MAI-Base-2 transitioned to an encoder-free native multimodal pretraining architecture, processing visual tokens directly within the base model. Training inputs included image-text pairs and documents with interleaved text and images; corpora covered chart comprehension, spatial reasoning, computer operation, sheet music reading, and other tasks. (These represent tasks covered by the training corpora; the report did not publish standalone benchmark scores for them.)

Microsoft Aims to Master Data, Models, and Systems in Tandem

The pretraining practices of MAI-Base-2 demonstrate the value of full-stack software-hardware co-optimization:

Through joint tuning across data pipelines, model architecture, and cluster infrastructure, the engineering team achieved higher learning efficiency within a given compute budget, leveraging a small-scale experimental ladder to provide relatively dependable guidance for large-scale training.

In business and application terms, Microsoft noted that fully autonomous data construction and provenance aim to support future model weight licensing to trusted partners through the Foundry platform. Currently, MAI-Base-2 remains in post-training and has not been opened to the public, nor have associated API or pricing details been announced; meanwhile, preparatory work for the next-generation MAI-Base-3 is already underway.