Gemma 4 Technical Report: How an Open Model Fights Bigger Ones With "Small Params + Thinking + Lean Memory"
Gemma 4's throughline is one sentence: use three engineering tracks to push "capability" and "cost" in the favorable direction at the same time.
Track one lifts intelligence: thinking mode (reason first, then answer). Track two cuts cost: local/global attention ratio, p-RoPE, KV sharing, K=V, QAT quantization, MTP speculative decoding. Track three expands modality: native image/text/audio, plus a "no separate encoder" unified architecture for the 12B. Every model size, table, and Arena rank below is evidence for this one sentence.
Thinking Mode
The main engine behind big math/code/STEM gains; almost every default eval has it on.
Long Context + Quantization + Speculative Decoding
KV can't blow up as context grows; 31B goes from ~64GB down to ~19GB; decoding speeds up too.
Native Multimodal + Unified 12B
Most sizes use a frozen encoder; the 12B ingests image patches and audio slices directly — two fewer big weight blobs.
- Family: E2B (effective 2.3B), E4B (effective 4.5B), 12B, 26B-A4B MoE (~4B active), 31B dense. Apache 2.0.
- Arena Text (2026-06-19): 31B Elo 1451 — reportedly #1 open dense model; the higher ranks nearby are mostly hundred-B-to-trillion-scale MoEs.
- Vs. Gemma 3 27B: STEM/code gaps are huge, but in the tables 4th-gen defaults to thinking while 3rd-gen 27B is mostly non-thinking — this shouldn't be read as a pure architecture win.
- On-device narrative: E2B closes in on 27B-class capability on several fronts (per official framing); audio encoder disk footprint goes from 390MB to 87MB.
Which Question in the Open-Source Race Is This Answering
By 2026 the open leaderboards were often dominated by giant MoEs. Gemma 4 picked a different lane: dense small-to-mid size + thinking + deployment-friendly.
If you only remember "another 31B just shipped" from this report, you'll miss what it's actually contesting. At the time, the top of Arena was crowded with open models at 300B, 700B, 1T, even 1.6T total params — mostly MoEs. Gemma rewrote the story as:
So the report's "core evidence" isn't a single SOTA point — it's a combination: the dense 31B holding its own against bigger MoEs (human preference); small models still fitting on-device (memory table + audio encoder size); long context without brute-forcing KV (RULER / LOFT).
Pick a Size First, See Who It Was Designed to Be
Click the five model tabs below. Same family, very different jobs.
E2B: On-Device Entry Point, Effective ~2.3B
Uses per-layer embeddings (same lineage as Gemma 3n) — total params are larger, but the "effective" count is 2.3B. Ships with its own 150M vision + 305M audio encoder. The report compares it to Gemma 3 27B, arguing "roughly an order of magnitude fewer params, close on many fronts."
E4B: On-Device Workhorse, Effective ~4.5B
Same playbook — small encoders + aggressive quantization. On vision evals, the report writes that it's not below Gemma 3 27B on any listed item (e.g. MMMU Pro 52.6 vs 49.7, MATH-Vision 59.5 vs 46.0). Long context is already ahead of 27B on several fronts too.
12B: the Report's Architecture Experiment (Also a Real Shipping Model)
This is the most "research-flavored" part of the whole report: trained from scratch with no separate vision/audio encoder. Images go through one big matmul on 48×48×3 patches (~35M params) + 2D coordinate positions; audio is projected directly at 40ms/640-dim chunks. The point isn't showmanship — it's cutting the memory fragmentation and pipeline complexity that separate encoders bring.
26B-A4B: the Sparse Route, ~4B Active
A 26B-class total-param MoE with ~3.8–4B active. Arena Elo 1438 — mostly close to the 31B on static benchmarks, but drops more noticeably on agent/terminal-style tasks (Terminal Bench, some Tau2 splits). Good fit if you want "small active, big capacity" deployment rather than a pure dense inference stack.
31B: Dense Flagship, the Report's "Headline" Slot
This is the one holding the #1 open-dense Arena spot; AIME 89.2, LiveCodeBench 80, GPQA 84.3, Codeforces Elo 2150. It also gets QAT and MTP on the efficiency side. It's not "just another 30B" — it's Gemma's proof point that a dense mid-size model can still go toe-to-toe with giant open MoEs in 2026.
Core Mechanism One: Why Thinking Mode Rewrites the Scores
Skip this section and every "crushes Gemma 3 27B" table later will be misread.
Gemma 4's instruction models can write out a reasoning trace in a dedicated channel before the final answer. Put <|think|> in the system turn and it switches on. Formatting-wise, the pretraining end token is <eos>, the instruction model's is <turn|>; thinking content runs through <|channel>thought ...<channel|>.
What a Full Call Looks Like
System turn switches on thinking + optional tool declarations
E.g. declaring a function like search_train — the model only knows what it can call if this is declared.
User asks a question
Plain natural language, same as any other instruction model.
Model writes the thought channel first
Internal reasoning, breaking down the problem, checking constraints — the final answer the user sees comes after the thought.
Issues a tool_call when needed, then continues
The appendix's ticket-booking example: ask for the destination first, then call the fare-lookup function.
Same Weights, Thinking Converts "Latent Potential" Into Visible Score
The report doesn't give a full ablation table for "thinking off vs. on," but the task type tells the story: the more a task needs multi-step reasoning, the bigger the gap between 4th-gen and 3rd-gen's 27B.
The "does it follow instructions" gap on IFEval is relatively mild; the "does it need many steps of thought" gap on AIME/code is extreme. This suggests thinking mode is very likely a score engine, not just icing.
The Filter You Must Apply When Reading Table 5
Also, Arena is human blind-preference voting; static benchmarks are reproducible scores — the two leaderboards can't be converted into each other. Beyond the vendor's self-reported numbers, third-party reproductions will shift over time.
Core Mechanism Two: How Long Context Gets "Cheaper"
Click through the four steps to see how global KV gets cut by up to 37.5%.
As context grows, the KV cache during inference scales linearly with sequence length and quickly becomes the memory bill itself. Gemma 4 didn't invent "infinite-context magic" — it split attention into "most layers only look nearby + a few layers look globally," then squeezed the cache specifically on those global layers.
All-Global Attention: Every Token Has to Talk to All History
The longer the sequence, the more every layer has to store the full K/V. As you go from 32k to 128k to 256k, the memory curve gets steep fast. On-device and single-GPU deployments die here first — not from "how big the parameter file is," but from this.
Like a meeting where every sentence anyone says has to be broadcast to the entire room, and the whole thing has to be recorded and archived. Once the room gets big enough, the recording drive fills up first.
Local Sliding Window : Global = 5:1 (4:1 for E2B)
Carries over the Gemma 3 approach: most layers only do local attention (looking at a recent window), with a true global attention layer inserted every so often. Compute and cache overhead get diluted proportionally.
Positional encoding is also split: global layers use p-RoPE (p=0.25), local layers use plain RoPE; frequencies are 1M and 10k respectively. The idea: global layers need to handle long-range positional relationships, local layers handle short-range structure.
Three Cuts, All Aimed at Global-Layer KV
K = V (Global Layers)
Outside E2B/E4B, global-layer values reuse the keys directly, saving a whole separate V store.
KV Sharing
Ratios like 20/35 for E2B, 18/42 for E4B — layers share caches instead of each holding its own full set.
p-RoPE
Paired with the global-layer positional scheme, further compresses the shape of the global cache; the report sums this up as up to −37.5% global KV.
Stacked With QAT
The weights themselves shrink too: 31B goes 64→19.2GB; on-device E2B drops to ~0.8GB. The context bill and the weight bill get cut together.
Cache Got Cheaper — Did Long-Context Scores Collapse?
| Item | Context | 31B | E4B | G3 27B |
|---|---|---|---|---|
| RULER accuracy | 128k | 96.4 | 86.6 | 66.0 |
| LOFT text retrieval | 128k | 79.5 | 58.5 | 8.6 |
| GraphWalks F1 | <128k | 82.3 | 50.9 | 32.8 |
| MRCR 8-needle | 128k | 66.4 | 25.4 | 13.5 |
Table 9, etc., are non-thinking. The report's read: long context is a full generation ahead of gen 3; E4B already beats 27B on several items. The 31B also reports a full-book MTOB at ~256k.
Core Mechanism Three: Two Multimodal Routes, Why the 12B Gets Its Own Chapter
Toggle between: the "encoder route" most sizes take vs. the 12B's "no-encoder route."
Translator + Main Brain
E2B/E4B use a 150M ViT + 305M USM-family audio encoder; the bigger dense/MoE models use a 550M vision encoder. During pretraining the encoders are frozen, and the main model learns to read whatever continuous representations they output.
Variable aspect ratio → 16px patches → soft tokens
N_max can go up to 70 / 140 / 280 / 560 / 1120. Higher resolution eats more sequence length; Appendix Table 12 shows scores drop at lower resolution.
40ms Mel → Conformer → continuous vector
Vs. Gemma 3n: 680M → 305M params, on-disk quantized footprint 390 → 87MB, and it reports ~10–17% relative gains on transcription/translation.
This route is engineering-mature: the encoder can be quantized and optimized independently, while the main model focuses on language. The tradeoff is memory fragmentation and the deployment complexity of "three sets of weights."
The Main Brain Looks at Raw Material Directly
The 12B is trained from scratch: image patches go through one matmul (~35M params) + 2D coordinate positional encoding; audio at 16kHz is sliced into 40ms chunks, yielding 640-dim vectors, and projected straight into the LLM embedding space. No separate ViT/USM.
If the language model is large enough and the pretraining recipe is right, "light projection + lots of data" can replace "heavy encoder." Table 8's ASR/translation numbers are the report's proof: at least for supported languages, this route isn't just a toy.
That's why the 12B's audio/vision encoder columns in the parameter table are blank, yet it still shows up in multimodal evals. It's both an architecture probe inside the product line and a reproducible experiment point for the community.
Efficiency Not Yet Covered: the Draft Head and Quantization
Long context solves "storage" — decoding speed still needs "fast." Gemma 4 ships with an autoregressive MTP draft head: it takes the main model's last-layer activations and KV, and a 4-layer small Transformer cross-attends to the main model's cache to draft future tokens in sequence, for use in speculative decoding. Because it cross-attends to KV the main model already has, it doesn't need a separate MTP prefill, and the draft length can vary.
The on-device draft head also shrinks "project onto a 262k-token vocab" into "top-k on clusters first, then project" — cutting the final matrix from d×262000 down to d×4096, with a claimed acceptance rate close to the original. QAT constrains the weights to int2/int4+int8 or Q4_0 during training itself, avoiding the big accuracy drop that comes from "quantize after training."
Evidence Layer: Human Preference vs. Static Benchmarks
Two bodies of evidence answer two different questions. Toggle to switch.
Who Do People Actually Prefer (2026-06-19 snapshot)
The x-axis is a "scale intuition" sketch, not a strict log scale. The takeaway: the dense 31B has muscled into the Elo band of giant MoEs, while sitting far above its own predecessor, the 27B.
| Rank | Model | Elo | Type |
|---|---|---|---|
| 1 | Claude Fable 5 | 1508 | Closed-source benchmark |
| 36 | DeepSeek V4 Pro Thinking | 1458 | MoE 1.6T/49B |
| 43 | Gemma 4 31B | 1451 | Dense 31B |
| 61 | Gemma 4 26B-A4B | 1438 | MoE 26B/4B |
| 157 | Gemma 3 27B | 1366 | Dense 27B |
Where It's Strong and Weak on Benchmarks
On knowledge and hard-question sets, the 31B/26B clearly pull ahead of E2B/E4B. E2B's GPQA score of 43.4 already sits close to G3 27B's 42.4 — right in line with the "small model closes in on big model" narrative.
Code is one of the areas with the biggest visual jump from gen 3 to gen 4. Combined with thinking mode, it looks more like "trading reasoning budget for accuracy" than simply piling on more code corpus.
| Task | 31B | 26B-A4B | E2B | G3 27B |
|---|---|---|---|---|
| Terminal Bench Hard | 36.0 | 14.0 | 3.0 | 4.0 |
| Tau2 retail | 86.4 | 85.5 | 34.6 | 6.6 |
| Tau2 telecom | 69.3 | 43.0 | 19.7 | 3.1 |
On agent/tool-use tasks, 31B and 26B diverge: they're close on retail-style scenarios, but the 31B is clearly steadier on telecom and terminal tasks. In other words, "close on Arena" doesn't mean "interchangeable on every agent task."
Arena proves: the dense 31B can muscle into the human-preference band of giant open MoEs.
The static benchmark suite proves: thinking + new post-training push the STEM/code ceiling into a different tier.
The long-context and quantization tables prove: this isn't a "lab-only, doesn't care about deployment" model card — the efficiency stack is a first-class citizen.
Training and Safety: What the Report Says — and What It Deliberately Doesn't
Pretraining data cuts off at 2025-01, covering web, code, and images; E2B/E4B/12B also include audio. Filtered for contamination, unsafe content, and memorization risk. Post-training follows the Gemma 3 framework plus thinking; the data side strips PII/toxicity/incorrect self-identity/duplicates, and adds attribution, hedging, and refusal subsets.
On safety, the report states it follows Gemini-tier processes, checks against the Frontier Safety Framework, tests inherent behavior under no filtering conditions, and shows improvement across safety categories relative to the prior generation with a low unwarranted-refusal rate. The report does not publish a full safety breakdown table, nor total training token counts, detailed data mixture ratios, or a systematic ablation of thinking mode. Read it as an "architecture and results whitepaper," not a "fully reproducible recipe book."
TPU v5p/v6e, three-way sharding across data/sequence/replica; ZeRO-3; Pathways multi-pod. Slice-Granularity Elasticity brings failure recovery from "many minutes" down to "a few seconds." None of this changes the model definition, but it explains how they could push both the encoder route and the no-encoder route simultaneously across the whole size lineup.
The Framework You Should Walk Away With
Prioritize E2B/E4B: effective small param count + QAT + smaller audio encoder + 4:1/5:1 attention. Don't just look at "total params" marketing names.
31B is the main piece of evidence. Arena and STEM/code both check out at once — but also verify your own quantization backend can actually handle Q4_0 and the MTP draft head.
The encoder-free 12B is the chapter genuinely worth digging into: it's asking whether a heavy encoder is still mandatory.
Align first: thinking on or off, same-day Arena snapshot, dense vs. MoE, context length, and KV implementation. Tables with mismatched settings can differ by 20 points and still mean nothing.
Full key-table number lookup (expand)
Table 5 excerpt (thinking) 31B: MMLU Pro 85.2 / AIME 89.2 / LCB 80 / CF 2150 / GPQA 84.3 / IFEval 98.9 / MRCR 66.4 / Terminal 36 / Tau2 retail 86.4
Vision Table 6 (high resolution) 31B: MMMU Pro 76.9 / MATH-Vision 85.6 / InfographicVQA 92.0 / OmniDocBench 0.131↓
Memory Table 3 E2B 4.6→0.8; E4B 9.0→2.3; 12B 24→7.65; 26B-A4B 52/7.6→16.2/2.8; 31B 64→19.2 (GB, 32k text)
Vocabulary 262k; same tokenizer family as Gemini Team 2025.
Source: Gemma Team, Google DeepMind. "Gemma 4 Technical Report." arXiv:2607.02770v1. This page is a mechanism-focused explainer; all interactivity is pure CSS. For exact figures, defer to the original PDF. The previous version leaned on lists and tables; this version was rewritten around "one core sentence + three mechanisms deep-dived + two-sided evidence."
Gemma 4: Open Models Go From "Pile On Params to Get Big" to "Small Body, Big Thinking, Lean Memory — And Still Makes the First Tier"
Google DeepMind released the Gemma 4 technical report: instead of a trillion-param behemoth, it uses three moves — thinking, lean memory, and native image/audio understanding — to let a model that fits on a single GPU go head-to-head with the big ones. One page, one read, done.
↓ Read it in one page · one figure animates
Gemma 4 is a set of AI models Google is releasing for free, that anyone can download and run on their own computer — that's what "open-source model" means, as opposed to models you can only use online without seeing what's inside. Open source has been stuck on a deadlock for the past couple of years:
✘ Can't run it: the smarter an open model usually is, the bigger it is — often needing a rack of servers or several top-tier GPUs
"Parameters" are the model's brain capacity — bigger usually means smarter, but also bigger means it needs more room. "Room" here means VRAM (the memory on your graphics card) — the whole model has to fit inside it to run. A behemoth with hundreds of billions to a trillion-plus params needs hundreds of GB of VRAM; an ordinary person's single GPU (roughly ten-something to twenty-something GB) just can't hold it.
What the report is trying to prove comes down to one sentence: you don't have to build a behemoth — three engineering moves working together can make a small model both smart and runnable.
- Thinking: before answering, it works through the problem step by step on a scratchpad — this is the main driver behind the huge jump in math and coding scores.
- Lean memory: by compressing the model and changing how it remembers, the same flagship model went from needing 64GB of VRAM down to about 19GB; the smallest model can even fit on a phone.
- Native image/audio understanding: no bolt-on parts — the model itself can read images and listen to audio.
Of the three moves, "thinking" is the main engine behind the score jump — and also the easiest one to misread. Let's use a figure to make it click first.
"Thinking mode" means letting the model work through a problem step by step on a dedicated "scratchpad" before giving its answer. Same hard question, with or without the scratchpad — very different results:
Being smart is only half the job — it also has to run. The clearest way to see this move's payoff is to compare how much VRAM the same 31-billion-param flagship needs before and after compression:
64GB needs an enterprise-grade GPU or several cards strung together to build a server-room-class machine; 19.2GB fits on one high-end gaming GPU (around 24GB). The math: the same 31B model needs 64GB uncompressed, 19.2GB after quantization compression — roughly a third of the original. All figures above are self-reported by the vendor (Google DeepMind); no third-party reproduction yet.
can download it and
run it themselves
you'd need a whole server rack to run it.
→ less runnable
→ still runs fine
going toe-to-toe with hundred-billion-param giants
straight answer scores 20.8 → thinking scores 89.2
fits on one GPU.
- × All numbers are vendor self-tested
- × No third-party reproduction yet
- × "Crushes previous gen" has some thinking-mode water mixed in
a model that fits on one GPU has made it into the first tier.
Members can copy or download the full Markdown, and let their AI subscribe and ask.
Found this useful? Leave your email — every new explainer, the moment it's out. Unsubscribe anytime.