bi BddlmZddlmZmZmZddlZddlmZddlm Z m Z ddl m Z m Z dd lmZdd lmZmZmZdd lmZdd lmZd dlmZmZe j6eZeGdde ZGddee ZGddeZ y)) dataclass)DictOptionalUnionN)nn) ConfigMixinregister_to_config) BaseOutputlogging)AttentionProcessor)-HunyuanCombinedTimestepTextSizeStyleEmbedding PatchEmbedPixArtAlphaTextProjection) ModelMixin)HunyuanDiTBlock)Tuple zero_modulec2eZdZUeej ed<y)HunyuanControlNetOutputcontrolnet_block_samplesN)__name__ __module__ __qualname__rtorchTensor__annotations__j/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/models/controlnets/controlnet_hunyuan.pyrr$s#ELL11r!rceZdZe ddedededeedeedededed ed ed ed ed edeffd Z e de ee ffdZ dee e ee fffdZe ddZ ddej&defdZxZS)HunyuanDiT2DControlNetModelconditioning_channelsnum_attention_headsattention_head_dim in_channels patch_size activation_fntransformer_num_layers mlp_ratiocross_attention_dimcross_attention_dim_t5pooled_projection_dimtext_len text_len_t5"use_style_cond_and_image_meta_sizect|||_||z|_t | | dz| d|_t jtj||z| tj|_ t|||||d|_ t|| || ||_t j g|_t j t%| dzdz Dcgc]H}t'|j|j(j*|t-|j| z| d d Jc}|_t1t j2|||_t%t7|j.D]>}t j2||}t1|}|j"j9|@ycc}w) N silu_fp32) in_features hidden_size out_featuresact_fn)dtype)heightwidthr( embed_dimr)pos_embed_type)r/seq_lenr-r2r rTF)dimr&r* ff_inner_dimr-qk_normskip)super__init__ num_heads inner_dimr text_embedderr Parameterrrandnfloat32text_embedding_paddingr pos_embedrtime_extra_emb ModuleListcontrolnet_blocksrangerconfigr&intblocksrLinear input_blocklenappend)selfr%r&r'r(r)r* sample_sizer7r+r,r-r.r/r0r1r2layer_controlnet_block __class__s r"rEz$HunyuanDiT2DControlNetModel.__init__*s( ,,/AA6..2,  ')ll KK;.0C5== Y' #$#!!  L "7 6/Q  "$r!2mm##9Q#>#BC  (, (G(G"/!$T^^i%?!@(;     'ryyk'JKs4;;'( .fn_recursive_add_processorssjv/282F2F^b2F2c dV:./#)#8#8#: U%+tfAhZ,@%T U r!)strrrModulerrrj)rYrcrarbrms @r"attn_processorsz+HunyuanDiT2DControlNetModel.attn_processorsvsm  c 588?? X\]`bt]tXu !//1 BLD& 'fj A Br! processorc Tt|jj}t|tr,t||k7rt dt|d|d|ddt dtjjffd |jD]\}}|||y) a2 Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.rarbct|drEt|ts|j|n#|j|j |d|j D]\}}|d|||y)N set_processorrgrh)ri isinstancedictrtpoprj)rarbrqrkrlfn_recursive_attn_processors r"rxzSHunyuanDiT2DControlNetModel.set_attn_processor..fn_recursive_attn_processorsxv/!)T2((3(($z7J)KL#)#8#8#: T%+tfAhZ,@%S Tr!N) rWrpkeysrurv ValueErrorrnrrrorj)rYrqcountrarbrxs @r"set_attn_processorz.HunyuanDiT2DControlNetModel.set_attn_processorsD((--/0 i &3y>U+BPQTU^Q_P`a005w6QRWQXXkm   Tc T588?? T!//1 ALD& 'fi @ Ar!c |j}|j}|j}|j}|j} |j } |j } |j} |j} |j}|j}|j}|j}|}|xs |j}||||||| | | | | ||||}|r<|j|jd}t j#d|d|S)N)r%r+r*r'r-r.r7r(r,r&r)rZr0r1F)strictz0controlnet load from Hunyuan-DiT. missing_keys: r)rRr*r'r-r.r7r(r,r&r)rZr0r1r+load_state_dict state_dictloggerwarning)cls transformerr%r+load_weights_from_transformerrRr*r'r-r.r7r(r,r&r)rZr0r1 controlnetkeys r"from_transformerz,HunyuanDiT2DControlNetModel.from_transformers"##,, #66$88!'!>!>(( (( $$ $88&& (( ??(( 5!7!X6;X;X"7#9'1 3#9## 3!## ),,[-C-C-Ee,TC NNMcRSfXV Wr!controlnet_condconditioning_scalec b|jdd\} }|j|}||j|j|z}|j||| | |j}|j\}}}|j |j d|jd}|j ||d}tj||gd}tj||gd}|jdj}tj|||j}d}t|jD]\}}||||| }||fz}d}t||j D]\}}||}||fz}|Dcgc]}||z }}| s|fSt#| Scc}w)  The [`HunyuanDiT2DControlNetModel`] forward method. Args: hidden_states (`torch.Tensor` of shape `(batch size, dim, height, width)`): The input tensor. timestep ( `torch.LongTensor`, *optional*): Used to indicate denoising step. controlnet_cond ( `torch.Tensor` ): The conditioning input to ControlNet. conditioning_scale ( `float` ): Indicate the conditioning scale. encoder_hidden_states ( `torch.Tensor` of shape `(batch size, sequence len, embed dims)`, *optional*): Conditional embeddings for cross attention layer. This is the output of `BertModel`. text_embedding_mask: torch.Tensor An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. This is the output of `BertModel`. encoder_hidden_states_t5 ( `torch.Tensor` of shape `(batch size, sequence len, embed dims)`, *optional*): Conditional embeddings for cross attention layer. This is the output of T5 Text Encoder. text_embedding_mask_t5: torch.Tensor An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. This is the output of T5 Text Encoder. image_meta_size (torch.Tensor): Conditional embedding indicate the image sizes style: torch.Tensor: Conditional embedding indicate the style image_rotary_emb (`torch.Tensor`): The image rotary embeddings to apply on query and key tensors during attention calculation. return_dict: bool Whether to return a dictionary. N) hidden_dtyper)r@r r )tembencoder_hidden_statesimage_rotary_emb)r)shaperMrVrNr:rHviewrcat unsqueezeboolwhererL enumeraterTziprPr)rY hidden_statestimesteprrrtext_embedding_maskencoder_hidden_states_t5text_embedding_mask_t5image_meta_sizestyler return_dictr;r<r batch_sizesequence_lengthr\block_res_samplesr[blockcontrolnet_block_res_samplesblock_res_sampler]samples r"forwardz#HunyuanDiT2DControlNetModel.forwards^&++BC0 }5 &(8(89X(YY "" .U]UcUc#  *B)G)G& OQ#'#5#5 $ ) )".F.L.LR.P Q$  $<#@#@_^`#a % +@BZ*[ab c#ii)<>T(U[]^1;;A>CCE % ,?AVX\XsXs t%dkk2 ELE5!&;!1 M!2]4D D  E(*$256GI_I_2` ^ . ./0@A +GK[J]+] ( ^ So'o1C(C'o$'o02 2&@\]] (ps F,)rXNNzgelu-approximate i(g@irMT)rNT g?NNNNNNNT)rrrr rSrrnfloatrrEpropertyrrrprr| classmethodrrrr __classcell__r^s@r"r$r$)s&'#%"$%)$(/&(#'&*%)37#I<"I<!I< I< c] I< SM I<I<!$I<I<!I<!$I< #I<I< !I<"-1#I<I.AE2Dd3PbKbFc2c,dA@nr''\%(" !%#]^ ]^ " ]^r!r$cZeZdZdZfdZ ddej defdZxZ S) HunyuanDiT2DMultiControlNetModela `HunyuanDiT2DMultiControlNetModel` wrapper class for Multi-HunyuanDiT2DControlNetModel This module is a wrapper for multiple instances of the `HunyuanDiT2DControlNetModel`. The `forward()` API is designed to be compatible with `HunyuanDiT2DControlNetModel`. Args: controlnets (`List[HunyuanDiT2DControlNetModel]`): Provides additional conditioning to the unet during the denoising process. You must set multiple `HunyuanDiT2DControlNetModel` as a list. cVt|tj||_y)N)rDrErrOnets)rY controlnetsr^s r"rEz)HunyuanDiT2DMultiControlNetModel.__init__Es MM+. r!rrc tt|||jD]O\} \}}}|||||||||| | | |  }| dk(r|}&td|dDcgc] \}}||z }}}|f}QScc}}w)r) rrrrrrrrrrrrr)rrr)rYrrrrrrrrrrrriimagescaler block_samplescontrol_block_samplescontrol_block_sample block_samples r"rz(HunyuanDiT2DMultiControlNetModel.forwardIs\.7s?L^`d`i`i7j-k A )A)uj&+! %#(&;$7)A'= /!1' M Av(5%?BBWXYBZ\ijk\l>m):,l)<7)%)*?(@%1 A4%$ )sA5r) rrr__doc__rErrrrrrs@r"rr8sJ /%(" !%#H% H% " H%r!r)! dataclassesrtypingrrrrrconfiguration_utilsr r utilsr r attention_processorr embeddingsrrrmodeling_utilsr#transformers.hunyuan_transformer_2drrrr get_loggerrrrr$rr r!r"rs"(( B(4 (A*   H % 2j2 2L^*kL^^Y%zY%r!