ai/nemotron-3.5-super repository overview
Read our How to Run Nemotron 3.5 Super VL Guide!
See Unsloth Dynamic 3.0 GGUFs for our quantization benchmarks.
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B
Description
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 is the full-precision release of NVIDIA Nemotron 3.5 Super VL, a vision-language model for reasoning, coding, tool use, document understanding, and video understanding. The BF16 weights support customization, domain adaptation, and full-precision evaluation. For optimized deployment, see the NVFP4 release.
The model combines interleaved Mamba-2, Mixture-of-Experts (MoE), and attention layers with a C-RADIO vision encoder and a vision-language connector. It has 120B total parameters and 12B active parameters and supports context lengths up to 1M tokens. Multi-Token Prediction (MTP) enables native speculative decoding.
This model is ready for commercial or non-commercial use.
Model Summary
| Total Parameters | 120B (12B active) |
| Architecture | Mamba-2–Transformer hybrid Mixture of Experts with a C-RADIO vision encoder |
| Precision | BF16 (full-precision reference weights) |
| Context Length | Up to 1M tokens; deployment settings are listed below |
| Inputs / Outputs | Text, images, and video / text |
| Deployment | 4× H100 or 2× GB200 |
| Speculative Decoding | Native Multi-Token Prediction (MTP) |
| Reasoning Mode | Configurable through the chat template (enable_thinking=True/False) |
| Recommended Sampling | Text: temperature 1.0, top_p 0.95; vision: temperature 0.6, top_p 0.95 |
| Best For | Customization, domain adaptation, and full-precision evaluation |
| License | OpenMDW-1.1 |
Model Overview
Model Developer: NVIDIA Corporation
What is Nemotron?
NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialized AI agents.
License/Terms of Use
Governing Download Terms: Use of this model is governed by the OpenMDW License Agreement, version 1.1.
Benchmarks
We evaluated our model on the following benchmarks:
| Benchmark | Nemotron 3.5 SuperGA BF16 | Qwen3.5-122B-A10B | Qwen3.8-Flash-Next | DeepSeek-V4-Flash0731 | Nemotron 3 Ultra | Inkling-Small | GLM-5.3-Flash |
|---|---|---|---|---|---|---|---|
| GPQA-Diamond | 89.52 | 86.7 | 92.7 | 89.6 | 87.0 | 88.6 | 91.2 |
| HLE, text-only | 31.79 | 24.2 | 39.6 | 35.8 | 29.2 | 33.3 | 36.47 |
| MMLU-ProX | 76.91 | 83.2 | 83.0 | 85.5 | 83.1 | 84.0 | 83.0 |
| AA-Omniscience accuracy | 26.20 | 26.2 | 27.9 | 45.0 | 25.3 | 35.4 | 34.2 |
| AA-Omniscience hallucination rate | 30.0 | 84.5 | 41.3 | 95.9 | 22.1 | 59.6 | 30.3 |
| AA-Omniscience index | 4.03 | -36.1 | -1.9 | -7.8 | 8.8 | -3.1 | 14.3 |
| CritPt | 10.86 | 1.1 | — | 12.6 | 3.1 | 9.1 | — |
| APEX shortlist | 88.36 | 54.7 | 91.8 | 88.7 | 75.0 | 71.5 | 77.1 |
| LiveCodeBench v6 | 88.22 | 76.7 | 94.1 | 94.6 | 89.2 | 88.3 | 80.6 |
| SciCode, subtask | 47.13 | 39.8 | 46.6 | 49.3 | 42.0 | 48.3 | 50.05 |
| AA-LCR v1.1 | 68.62 | 75.1 | 78.4 | 79.7 | 75.2 | 77.3 | 79.5 |
| GraphWalks | 89.71 | 86.1 | 76.1 | 65.4 | 63.3 | 55.7 | 88.4 |
| IFBench | 70.86 | 67.6 | 75.6 | 76.2 | 82.9 | 79.3 | 57.7 |
| LMArena v3 proxy | 52.67 | 47.0 | 35.2 | 49.8 | 34.3 | 53.6 | 61.0 |
| WMT24++ | 84.80 | 86.3 | 86.2 | 85.5 | 82.7 | 84.6 | 86.1 |
| tau2-bench retail | 69.24 | 55.4 | 71.6 | 73.8 | 73.7 | 71.8 | 78.2 |
| tau2-bench airline | 76.50 | 63.5 | 80.8 | 83.5 | 79.8 | 62.7 | 81.8 |
| tau2-bench telecom | 73.19 | 87.2 | 90.4 | 98.2 | 83.0 | 90.1 | 98.5 |
| tau3-bench banking | 18.88 | 14.2 | 44.5 | 38.6 | 18.4 | — | 34.8 |
| GDPval-AA v2 | 1211 | 861 | 1561 | 1382 | 1148 | 1313 | 1563 |
| BrowseComp | 78 | 58.1 | 89.0 | 84.2 | 67.85 | 74.1 | 86.6 |
| SWE-bench Verified | 71.10 | 71.9 | 78.7 | 75.4 | 73.9 | 75.27 | 79.10 |
| SWE-bench Multilingual | 67.72 | 62.0 | 77.10 | 72.0 | 68.8 | 71.4 | 78.30 |
| SWE-bench Pro | 51.78 | 44.9 | 60.37 | 56.91 | 49.66 | 56.54 | 59.70 |
| Terminal-Bench 2.1 | 66.70 | 48.70 | 85.80 | 76.80 | 54.80 | 53.40 | 82.70 |
| BabyVision | 29.60 | 35.7 | 46.8 | N/A | N/A | 24.7 | 39.1 |
| MMLongBench-Doc | 62.82 | 63.7 | 67.9 | N/A | N/A | 49.06 | 65.8 |
| V* | 85.60 | 91.1 | 93.0 | N/A | N/A | 70.5 | 88.1 |
| OCR-Reasoning | 65.08 | 64.3 | 67.1 | N/A | N/A | 54.7 | 64.6 |
| MathVision | 78.16 | 83.3 | 87.5 | N/A | N/A | 78.8 | 82.5 |
| HLE, full set | 29.20 | 24.4 | 38.8 | N/A | N/A | 31.6 | 35.1 |
| OCRBench v2, EN | 68.12 | 68.3 | 67.9 | N/A | N/A | 54.2 | 61.8 |
| OCRBench v2, ZH | 59.15 | 67.3 | 68.3 | N/A | N/A | 52.4 | 56.8 |
| CharXiv reasoning | 80.82 | 83.5 | 85.8 | N/A | N/A | 77.8 | 81.9 |
| Video-MME v2 | 32.85 | 32.0 | 45.1 | N/A | N/A | — | 14.8 |
| OSWorld-Verified | 51.14 | 51.5 | 55.8 | N/A | N/A | — | 14.78 |
| BrowseComp [offline] | 78 | 58 | 89 | 83 | 44 (reported) | 74 | 86.6 |
| WebVoyager [offline] | 77.72 | 54.89 | 90.39 | not applicable | not applicable | 5.25 | 72.64 |
| AA-Briefcase-Lite [offline] | 1137 (-) | 1252 (-) | 40 | 551 | 591 | 22 | 1636 | 1324 | 76 | 1469 | 1277 | 80 | 665 (836) | 866 (874) | 47 | 765 | 684 | 17 | 1785 (1658) | 1428 (1408) | 76 |
| AutomationBench-AA [offline] | 32.44 | 27.25 | 72.06 | 63.86 | 23.28 | 32.83 | 70.17 |
| GDP.pdf [offline] | 6.6 (AA 6.6) | 8.8 (AA 3.4) | 17.4 (AA 15.6) | 10.8 (AA 11.0) | 8.4 (AA 5.0) | 8.6 (AA 11.2) | 14.8 (AA 15.4) |
| Terminal Bench 4 [offline] | 1 | 0 | 20.71 | 3.54 | 0 | 0.5 | 15.15 |
| DeepSWE [offline] | 24.65 | 0.4 | 55.57 | 42.3 | 3 | 18 | 57.77 / 63.4 (reported) |
We evaluated Nemotron 3.5 Super on a comprehensive suite of benchmarks spanning agentic capabilities, reasoning and knowledge, multimodal, conversational ability and instruction following, long-context understanding, and multilingual performance. All evaluation results for Nemotron 3.5 Super and baselines were collected via NeMo Gym. For reproducibility purposes, more details on the evaluation settings can be found in the NeMo Gym recipes folder and the corresponding reproducibility tutorial. The published recipes specify the benchmark-specific containers, prompts, inference parameters, parser configurations, and scoring settings used to produce the results.
There were a few benchmarks that were not yet onboarded in our open source tools, and for these we used either their official open source implementation or an internal scaffolding that we plan to open source in the future: BabyVision, CharXiv-RQ, MathVision, MMLongBench-Doc, OCR-Reasoning, OCRBench-v2, V*, Video-MME-v2.
Agentic Coding Benchmarks
Additional harness-level coding-agent results for SWE-Bench Pro and Terminal-Bench 2.1 are shown below.
Deployment Geography: Global
Use Case
This model is intended for developers who want to customize a post-trained vision-language model, adapt it to domain-specific data, develop quantized variants, or run full-precision inference and evaluation.
Release Date
Hugging Face - 10/15/2026 via Hugging Face
Reference(s)
Model Architecture
- Architecture Type: Mamba-2–Transformer hybrid Mixture of Experts (MoE) with Multi-Token Prediction (MTP)
- Network Architecture: Nemotron Hybrid MoE with 512 routed experts per MoE layer, a C-RADIO vision encoder, and a vision-language connector
- Number of Model Parameters: 120B total / 12B active
Model Design
The model combines sparse expert computation, attention, and Mamba-2 layers to support long-context processing. The vision encoder and connector turn visual inputs into representations processed by the language backbone. A shared MTP head predicts future tokens recursively for speculative decoding.
The model supports up to 1M tokens of context. The prompt, visual inputs, conversation history, tool results, and generated response share this context window. Individual deployment recipes may use a smaller context window to accommodate the available GPU memory and concurrency.
Training Methodology
Stage 1: Foundation and Multi-Token Prediction
The language backbone is initialized from Nemotron 3 Super Base. A shared MTP head is trained for 100B tokens while the backbone remains frozen, followed by 1T tokens of distillation from Nemotron 3 Ultra Base.
Stage 2: Continued Pre-Training and Mid-Training
The model undergoes 100B tokens of Indic continued pre-training, a 35B-token extension to 1M context, 67B tokens of software-engineering mid-training at 256K context, and a final 9B-token long-context recovery stage. Software-engineering data includes repositories, issue–pull-request pairs, commit diffs, build and CI logs, and agent trajectories.
Stage 3: Vision-Language Training
A C-RADIO vision encoder and connector extend the language backbone to visual inputs, including images, document pages, and video frames.
Stage 4: Supervised Fine-Tuning
A two-stage supervised fine-tuning pipeline jointly trains the language and vision components on vision, reasoning, code, tools, chat, safety, and long-context data. Training uses packed sequences of up to 524,288 tokens.
Stage 5: Reinforcement Learning and Distillation
Reinforcement learning with verifiable rewards, reinforcement learning from human feedback, and specialized teacher training improve reasoning, agentic, chat, and vision capabilities. Multi-teacher on-policy distillation consolidates these capabilities into a single model.
Stage 6: MTP Boosting
The MTP head is further trained against the frozen post-trained backbone to improve speculative-decoding acceptance.
Input
- Input Type(s): Text, images, and video
- Input Format(s): Text strings, images, and sampled video frames
- Video: mp4, up to 2 minutes.
- Image: Red, Green, Blue (RGB) (jpeg, png)
- Input Parameters:
- Video: Three-Dimensional (3D)
- Image: Two-Dimensional (2D)
- Text: One-Dimensional (1D)
- Other Properties Related to Input: Maximum context length up to 1M tokens. Documents can be supplied as page images with accompanying text.
Output
- Output Type(s): Text
- Output Format: String
- Output Parameters: One-dimensional sequences
- Other Properties Related to Output: Maximum context length up to 1M tokens.
Our AI models are designed and optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
Software Integration
- Runtime Engine(s): vLLM; SGLang
- Hardware:
- NVIDIA Blackwell
- NVIDIA Hopper
- Operating System(s): Linux
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
Training, Testing, and Evaluation Datasets
Training
Data Modality: Text, images, and video
Foundation Training Data Size: More than 20 trillion tokens
Image Training Data Size: 1 Million to 1 Billion Images
Text Training Data Size: More than 10 Trillion Tokens
Video Training Data Size: 10,000 to 1 Million Hours
Foundation Data Collection Period: 2013 to December 2025
Data Collection Method: Hybrid: automated, manually-collected, and synthetic
Labeling Method: Hybrid: automated, manually-labeled, and synthetic
The foundation corpus contains curated and synthetic text covering English, 19 other spoken languages, and 43 programming languages. Sources include webpages, dialogue, articles, code, and other written materials across math, science, legal, finance, and general knowledge domains.
The post-training corpus includes reasoning, coding, tool use, instruction following, structured outputs, chat, safety, and long-context tasks, together with image and video understanding. Synthetic trajectories are filtered for structural correctness, repetition, and task validity.
For more information about the datasets used to train this model, please see the Public Summary of Training Content
Detailed dataset information
Base Pre-Training Corpus (Nemotron 3 Foundation)
The foundation of the model is trained on the Nemotron 3 corpus, comprising the following datasets from the Nemotron Pretraining Datasets collection:
| Dataset Collection | Token Counts | Description |
|---|---|---|
| Nemotron-CC-v2 & v2.1 | 9.1T | A massive collection of English web data filtered from Common Crawl, including 2.5T+ tokens of new organic, translated, and synthetically rephrased content. |
| Nemotron-CC-Code-v1 | 427.9B | High-quality code tokens extracted from Common Crawl using the Lynx + LLM pipeline to preserve structure and equations. |
| Nemotron-Pretraining-Code-v1 & v2 & v3 | 1.7T | Curated GitHub code references with multi-stage filtering, deduplication, and large-scale synthetic code data. |
| Nemotron-CC-Math-v1 | 133.3B | High-quality math pre-training dataset preserving LaTeX formatting and mathematical structures. |
| Nemotron-Pretraining-Specialized-v1 & v1.1 & v1.2 & Nemotron-Pretraining-SFT-v1 | 660.0B | Synthetic datasets targeting specialized domains such as STEM reasoning and scientific coding. |
| Nemotron-Pretraining-Legal-v1 | 4.3B | Synthetic datasets targeting the legal domain. |
Public Datasets
…(truncated — see the full README on HuggingFace)
Tag summary
Content type
Unrecognized
Digest
sha256:9fc52fc60…
Size
82.6 GB
Last updated
about 16 hours ago
docker pull ai/nemotron-3.5-super