fix: align RotaryEmbedding with Qwen2Moe pattern for transformers compat

by kashif HF Staff - opened 7 days ago

base: refs/heads/main

←

from: refs/pr/4

Discussion Files changed

+37

-32

fix: align RotaryEmbedding with Qwen2Moe pattern for transformers compatdfa9ac6c

kashif

7 days ago

Replace custom rope_type handling with the standard Qwen2Moe pattern: use compute_default_rope_parameters for "default" type, only look up ROPE_INIT_FUNCTIONS for non-default types. Also adds partial_rotary_factor support.

fix: call super()._init_weights() to match Qwen2Moe convention for transformers v577298929

fix: align _init_weights with Qwen2Moe using nn.init APIc555c2f1

Use create_bidirectional_mask for backend-agnostic attention mask handlingd8a5a92f

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Ready to merge

This branch is ready to get merged automatically.

· Sign up or log in to comment