Research explainer · XiaoHu

Gemma 4 Technical Report: How open models punch up with small params, thinking, and memory thrift

Google DeepMind’s Gemma 4 report is not a feature checklist. It argues you can approach frontier human preference and STEM without trillion-scale MoEs—and still fit on device.
⬡ The core (read this first)

Gemma 4’s thesis is one sentence: three engineering stacks that push capability up and cost down at the same time.

Stack one raises intelligence: thinking mode (reason, then answer). Stack two cuts cost: local/global attention ratio, p-RoPE, KV sharing, K=V, QAT, MTP speculative decoding. Stack three expands modalities: native text/image/audio, plus a 12B encoder-free unified path. Every model size, table, and Arena rank is evidence for that sentence.

01 Raise intelligence

Thinking mode

Main engine of big gains on math, code, and STEM; default evals almost all leave it on.

02 Cut cost

Long context + quantization + speculative decoding

Keep KV from exploding as context grows; 31B shrinks ~64GB → ~19GB; decoding can also speed up.

03 Expand modalities

Native multimodal + encoder-free 12B

Most sizes use frozen encoders; 12B ingests image patches and audio chunks directly—two fewer heavy weight towers.

60-second skim
  • Family: E2B (eff. 2.3B), E4B (eff. 4.5B), 12B, 26B-A4B MoE (~4B active), dense 31B. Apache 2.0.
  • Arena Text (2026-06-19): 31B Elo 1451—leading open dense model per the report; higher open ranks are mostly 100B–T-scale MoEs.
  • Vs Gemma 3 27B: STEM/code gaps look huge, but Gemma 4 defaults to thinking while G3 27B is often non-thinking—do not read it as pure architecture win.
  • On-device story: E2B nears 27B-class scores on several axes (official); audio encoder disk 390MB→87MB.
⚑ Based on arXiv:2607.02770, Gemma 4 Technical Report. Numbers are vendor-reported. Interactives explain mechanisms; they do not replace the PDF tables.
01

Which open-source problem is it answering?

In 2026, open leaderboards are often ruled by huge MoEs. Gemma 4 takes another path: mid-size dense + thinking + deployability.

If you only remember “another 31B,” you miss the seat it wants. Top open Arena models then were often 300B, 700B, 1T, even 1.6T MoEs. Gemma rewrites the story as:

ProblemFrontier models are strong, but local/private deploys cannot host or afford them
→
Old open answerKeep stacking total params, or accept a clearly weaker tier
→
Gemma 4’s answerThinking for scores + architecture that saves KV/weights + full size ladder from edge to cloud

So the core evidence is not one SOTA point—it is the combo: dense 31B coexists with larger MoEs on human preference; small models still fit on device (memory table + audio encoder size); long context without brute-force KV (RULER / LOFT).

HOW THE THREE LINES LOCK Thinking mode AIME / Code / GPQA Lift problem-solving & reasoning Efficiency stack KV / QAT / MTP Cut VRAM & latency Native multimodal Image + audio + unified 12B Widen what it can ingest Outcome: small sizes can still contest human preference + STEM + long context
The three lines are not equal ad slots: thinking owns scores, efficiency owns fit-and-speed, multimodal owns input shape. Drop one and the story breaks.
02

Pick a size first—see who it was built for

Tap the five sizes below. Same family, very different jobs.

E2B: on-device entry, ~2.3B effective

Uses per-layer embeddings (Gemma 3n lineage): larger total params, but “effective” 2.3B. Ships 150M vision + 305M audio encoders. The report compares it to Gemma 3 27B and claims ~10× fewer params with several near-parity scores.

0.8 GBQuantized weight footprint (32k text, Table 3)
4:1Local/global attention (thriftier than the family’s 5:1)
AIME 37.5Still far above G3 27B’s 20.8 (thinking settings differ)
76MMTP drafter head for speculative decoding

E4B: on-device workhorse, ~4.5B effective

Same small-encoder + aggressive QAT path. On vision, the report says it meets or beats Gemma 3 27B on listed evals (e.g. MMMU Pro 52.6 vs 49.7, MATH-Vision 59.5 vs 46.0). Several long-context scores already top 27B.