amdenterpriseai/aim-google-gemma-4-26b-a4b-it repository overview

⁠Gemma 4 26B A4B IT

Gemma 4 26B A4B IT is the instruction-tuned, sparse mixture-of-experts variant of Google's Gemma 4 (Gemma4ForConditionalGeneration). Its text tower has 30 layers with 128 experts routed top-8 (moe_intermediate_size 704), giving roughly 26.5B total parameters with about 4B active per token. Attention is heterogeneous: 25 sliding-window layers (window 1024, head_dim 256, 8 KV heads) interleaved with 5 full-attention layers (global_head_dim 512, 2 KV heads), and max_position_embeddings is 262144. A SigLIP-based vision encoder provides image input; audio_config is null in the published checkpoint, so this AIM serves text and image only.