bi8ddlmZmZmZddlZddlmZddlmZm Z ddl m Z ddl m Z ddlmZmZdd lmZmZdd lmZdd lmZdd lmZmZmZe j8eZGd dej>Z GddeeZ!y))AnyDictOptionalN) ConfigMixinregister_to_config)logging)LuminaFeedForward) AttentionLuminaAttnProcessor2_0)&LuminaCombinedTimestepCaptionEmbeddingLuminaPatchEmbed)Transformer2DModelOutput) ModelMixin)LuminaLayerNormContinuousLuminaRMSNormZeroRMSNormceZdZdZ ddededededededed ed ed d ffd Z ddejdejdejdejdejdejde e e e ffdZxZS)LuminaNextDiTBlocka A LuminaNextDiTBlock for LuminaNextDiT2DModel. Parameters: dim (`int`): Embedding dimension of the input features. num_attention_heads (`int`): Number of attention heads. num_kv_heads (`int`): Number of attention heads in key and value features (if using GQA), or set to None for the same as query. multiple_of (`int`): The number of multiple of ffn layer. ffn_dim_multiplier (`float`): The multiplier factor of ffn layer dimension. norm_eps (`float`): The eps for norm layer. qk_norm (`bool`): normalization for query and key. cross_attention_dim (`int`): Cross attention embedding dimension of the input text prompt hidden_states. norm_elementwise_affine (`bool`, *optional*, defaults to True), dimnum_attention_heads num_kv_heads multiple_offfn_dim_multipliernorm_epsqk_normcross_attention_dimnorm_elementwise_affinereturnNc t |||z|_tjt j |g|_t|d||z|rdnd||dddt |_ tj|j_ t||||z|rdnd||dddt |_ t|td|zdz |||_t#||| |_t'||| |_t'||| |_t'||| |_t'||| |_y) Nlayer_norm_across_headsh㈵>F) query_dimrdim_headrheadskv_headsepsbiasout_bias processorr)r inner_dimrr) embedding_dimrr)r(elementwise_affine)super__init__head_dimnn Parametertorchzerosgater r attn1Identityto_outattn2r int feed_forwardrnorm1r ffn_norm1norm2 ffn_norm2 norm1_context) selfrrrrrrrrr __class__s i/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/models/transformers/lumina_nextdit2d.pyr1zLuminaNextDiTBlock.__init__5sI 22 LL.A-B!CD  $//18-d%!,.   KKM  3//18-d%!,.   .%#+/*#1  '$;  !(G^_ShCZ[  (G^_$%8h[rs hidden_statesattention_maskimage_rotary_embencoder_hidden_states encoder_masktembcross_attention_kwargsc |}|j||\} } } } |jd| | |||d|} |j|}|jd| |||dd|}||jj j ddddz}| |z}|jd}|jjd|}|| jdj |j|zz}|j|j|d| jdzz}|| jdj |j|zz}|S)a Perform a forward pass through the LuminaNextDiTBlock. Parameters: hidden_states (`torch.Tensor`): The input of hidden_states for LuminaNextDiTBlock. attention_mask (`torch.Tensor): The input of hidden_states corresponse attention mask. image_rotary_emb (`torch.Tensor`): Precomputed cosine and sine frequencies. encoder_hidden_states: (`torch.Tensor`): The hidden_states of text prompt are processed by Gemma encoder. encoder_mask (`torch.Tensor`): The hidden_states of text prompt attention mask. temb (`torch.Tensor`): Timestep embedding with text prompt embedding. cross_attention_kwargs (`Dict[str, Any]`): kwargs for cross attention. )rGrJrHquery_rotary_embkey_rotary_embNr)r>r8rBr;r7tanhviewflattenr: unsqueezer@r=r?rA)rCrGrHrIrJrKrLrMresidualnorm_hidden_statesgate_msa scale_mlpgate_mlpself_attn_outputnorm_encoder_hidden_statescross_attn_outputmixed_attn_output mlp_outputs rEforwardzLuminaNextDiTBlock.forwardvs,!=AJJ}VZ<[9Hi%4:: ,"4)-+  %  &*%7%78M%N"&DJJ ,"<'-  %  . 0@0E0EaBPQ0RR,/@@-55b9, ))!,->?  8#5#5a#8#=#=#?$**]B[#[[ &&t~~m'DIL_L_`aLbHb'cd %(:(:1(=(B(B(Dt~~V`Ga(aa rF)T)N)__name__ __module__ __qualname____doc__r<floatboolr1r5Tensorrrstrrrc __classcell__rDs@rErr$s4)-?t ?t!?t ?t  ?t " ?t?t?t!?t"&?t ?tR<@9||9 9 ,, 9 %|| 9 ll 9ll9!)c3h 89rFrc eZdZdZgdZe ddedeedeedeedeed eed eed eed eed eedee dee deedeeddffd Z dde jde jde jde jde jde eefde jfdZxZS)LuminaNextDiT2DModelaa LuminaNextDiT: Diffusion model with a Transformer backbone. Inherit ModelMixin and ConfigMixin to be compatible with the sampler StableDiffusionPipeline of diffusers. Parameters: sample_size (`int`): The width of the latent images. This is fixed during training since it is used to learn a number of position embeddings. patch_size (`int`, *optional*, (`int`, *optional*, defaults to 2): The size of each patch in the image. This parameter defines the resolution of patches fed into the model. in_channels (`int`, *optional*, defaults to 4): The number of input channels for the model. Typically, this matches the number of channels in the input images. hidden_size (`int`, *optional*, defaults to 4096): The dimensionality of the hidden layers in the model. This parameter determines the width of the model's hidden representations. num_layers (`int`, *optional*, default to 32): The number of layers in the model. This defines the depth of the neural network. num_attention_heads (`int`, *optional*, defaults to 32): The number of attention heads in each attention layer. This parameter specifies how many separate attention mechanisms are used. num_kv_heads (`int`, *optional*, defaults to 8): The number of key-value heads in the attention mechanism, if different from the number of attention heads. If None, it defaults to num_attention_heads. multiple_of (`int`, *optional*, defaults to 256): A factor that the hidden size should be a multiple of. This can help optimize certain hardware configurations. ffn_dim_multiplier (`float`, *optional*): A multiplier for the dimensionality of the feed-forward network. If None, it uses a default value based on the model configuration. norm_eps (`float`, *optional*, defaults to 1e-5): A small value added to the denominator for numerical stability in normalization layers. learn_sigma (`bool`, *optional*, defaults to True): Whether the model should learn the sigma parameter, which might be related to uncertainty or variance in predictions. qk_norm (`bool`, *optional*, defaults to True): Indicates if the queries and keys in the attention mechanism should be normalized. cross_attention_dim (`int`, *optional*, defaults to 2048): The dimensionality of the text embeddings. This parameter defines the size of the text representations used in the model. scaling_factor (`float`, *optional*, defaults to 1.0): A scaling factor applied to certain parameters or layers in the model. This can be used for adjusting the overall scale of the model's operations. )patch_embeddernormffn_normN sample_size patch_size in_channels hidden_size num_layersrrrrr learn_sigmarrscaling_factorr cpt|||_||_||_| r|dzn||_||_||_||z|_||_ t|||d|_ tjtj||_t#t%|d| |_tj(t+|Dcgc]}t-||||| | | | c}|_t1|t%|dddd||z|j z|_||zd zd k(sJd ycc}w) Nr T)rtru embed_dimr)i)rvrFgư>)r.conditioning_embedding_dimr/r(r)out_dimrz+2d rope needs head dim to be divisible by 4)r0r1rsrtru out_channelsrvrr2ryrrpr3r4r5empty pad_tokenrmintime_caption_embed ModuleListrangerlayersrnorm_out)rCrsrtrurvrwrrrrrrxrrry_rDs rEr1zLuminaNextDiT2DModel.__init__sS$ &$&/:K!O &#6 #':: ,.!{kX\ ekk+&>?"HK.DW# mmz* #' &'    2%'*;'=$+d.?.??   22a71<k>kk<1 s D3rGtimesteprJrKrIrMc `|j||\}}} }|j|j}|j|||} |j }|j D]} | |||||| |}|j || }|jx} } | d\}}|jd}|| z|| zz}|ddd|fj||| z|| z| | |j}|jddddddjddjdd}|s|fSt| S) a Forward pass of LuminaNextDiT. Parameters: hidden_states (torch.Tensor): Input tensor of shape (N, C, H, W). timestep (torch.Tensor): Tensor of diffusion timesteps of shape (N,). encoder_hidden_states (torch.Tensor): Tensor of caption features of shape (N, D). encoder_mask (torch.Tensor): Tensor of caption masks of shape (N, L). )rLrMrNrQrr r~)sample)rptodevicerrirrrtsizerVrpermuterWr)rCrGrrJrKrIrM return_dictmaskimg_sizerLlayer height_tokens width_tokensheightwidth batch_sizesequence_lengthoutputs rErczLuminaNextDiT2DModel.forward#sl&;?:M:Mm]m:n7 tX'7+..}/C/CD&&x1F U#((* [[ E! %'=M  mT: (,6   "''* !]2u 7LM%a)9/)9&9:?? -/,1F Wceievev &&q!Q1a8@@AFNNqRST9 'v66rF)r r~i rNNr#TTig?)NT)rdrerfrg _skip_layerwise_casting_patternsrr<rrhrir1r5rjrrkrrcrlrms@rEroros+Z(N$$%%&%)$&-/&*%(.2$(&*"&-1*->l>lSM>lc] >l c] >l SM >l&c]>lsm>lc]>l%UO>l5/>ld^>l$>l&c]>l!>l !>l>lN2637||37,,37 %|| 37 ll 37  ,, 37!%S#X37 37rFro)"typingrrrr5torch.nnr3configuration_utilsrrutilsr attentionr attention_processorr r embeddingsrrmodeling_outputsrmodeling_utilsr normalizationrrr get_loggerrdloggerModulerrorTrFrErsi'& B)C8'QQ   H %KK\d7:{d7rF