biMtddlmZddlmZddlmZmZmZmZm Z m Z ddl Z ddl Z ddl m Z mZddlmZmZddlmZmZdd lmZd d lmZmZmZmZmZmZmZd d lm Z m!Z!d d l"m#Z#d dl$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+d dl,m-Z-ddl.m/Z/ej`e1Z2eGddeZ3GddejhZ5GddejhZ6GddejhZ7 d;de8de8de8de8de8dee8d ee e8e e8fd!ee8d"ee8d#e9d$ee9d%ee9fd&Z: dGd2d3ejhZ?Gd4d5ejhZ@Gd6d7ejhZAd=d8ZBd9ZCd:ZDy)>) dataclass)gcd)AnyDictListOptionalTupleUnionN)Tensornn) ConfigMixinregister_to_config) BaseOutputlogging) apply_freeu)ADDED_KV_ATTENTION_PROCESSORSCROSS_ATTENTION_PROCESSORS AttentionAttentionProcessorAttnAddedKVProcessor AttnProcessorFusedAttnProcessor2_0)TimestepEmbedding Timesteps) ModelMixin)CrossAttnDownBlock2DCrossAttnUpBlock2D Downsample2D ResnetBlock2DTransformer2DModelUNetMidBlock2DCrossAttn Upsample2D)UNet2DConditionModel)ControlNetConditioningEmbeddingc eZdZUdZdZeed<y)ControlNetXSOutputa= The output of [`UNetControlNetXSModel`]. Args: sample (`Tensor` of shape `(batch_size, num_channels, height, width)`): The output of the `UNetControlNetXSModel`. Unlike `ControlNetOutput` this is NOT to be added to the base model output, but is already the final output. Nsample)__name__ __module__ __qualname____doc__r*r __annotations__e/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/models/controlnets/controlnet_xs.pyr)r)4sFFr1r)c eZdZdZ ddej dej dej deej deejf fd ZxZ S) DownBlockControlNetXSAdapterz}Components that together with corresponding components from the base model will form a `ControlNetXSCrossAttnDownBlock2D`resnets base_to_ctrl ctrl_to_base attentions downsamplercht|||_||_||_||_||_yN)super__init__r5r6r7r8 downsamplers)selfr5r6r7r8r9 __class__s r2r=z%DownBlockControlNetXSAdapter.__init__Fs6  (($'r1)NN) r+r,r-r.r ModuleListrConv2dr= __classcell__r@s@r2r4r4Bsh*/3+/ ( (mm (mm ( R]]+ ( bii( ( (r1r4cXeZdZdZdedej dej ffd ZxZS)MidBlockControlNetXSAdapterz|Components that together with corresponding components from the base model will form a `ControlNetXSCrossAttnMidBlock2D`midblockr6r7cLt|||_||_||_yr;)r<r=rGr6r7)r?rGr6r7r@s r2r=z$MidBlockControlNetXSAdapter.__init__Zs%   ((r1) r+r,r-r.r#r rAr=rCrDs@r2rFrFVs3))!8) )egerer))r1rFc<eZdZdZdej ffd ZxZS)UpBlockControlNetXSAdapterzwComponents that together with corresponding components from the base model will form a `ControlNetXSCrossAttnUpBlock2D`r7c0t|||_yr;)r<r=r7)r?r7r@s r2r=z#UpBlockControlNetXSAdapter.__init__ds (r1)r+r,r-r.r rAr=rCrDs@r2rJrJasB)R]]))r1rJbase_in_channelsbase_out_channelsctrl_in_channelsctrl_out_channels temb_channelsmax_norm_num_groupstransformer_layers_per_blocknum_attention_headscross_attention_dimadd_downsampleupcast_attentionuse_linear_projectionc Jd} g}g}g}g}t|tr|g| z}t| D]}|dk(r|n|}|dk(r|n|}|jt |||jt ||z||t ||z|t ||d|r3|jt|||z|||| | | t |||jt ||| rI|jt ||t||zd|d }|jt ||nd}ttj|tj|tj| }|rtj||_ |||_ |S) Nrr max_factorh㈵> in_channels out_channelsrPgroups groups_outepsr] num_layersrTrWrVnorm_num_groupsTopuse_convr^name)r5r6r7) isinstanceintrangeappendmake_zero_convr!find_largest_factorr"r r4r rAr8r>)rLrMrNrOrPrQ has_crossattnrRrSrTrUrVrWrcr5r8r7r6ir>down_block_componentss r2get_down_block_adapterrrisJGJLL.4(D'E 'R$ : #R/0Av+;L/0Av+;L N+;=MNO ,/??.+*+;>N+N[no./@M`a       "'%)<< 1;A>(;*?%5$78IVi$j   N+<>OPQG#RJ N+<>OPQ#  1 1DO`gk N+<>OPQ 8 g&]]<0]]<0 +-==+D(-9*  r1 base_channels ctrl_channelsc t||} t|||z||tt|||z||||| } t||} t | | | S)N rRr]r^rP resnet_groupsrTrSrWrV)r6rGr7)rmr#rnrrF) rsrtrPrQrRrSrTrVrWr6rGr7s r2get_mid_block_adapterrxsl"-?L&%A!M1"#)#m]]=Z*[]pq//3) H"-?L &L8bn oor1r^prev_output_channelctrl_skip_channelscg}d}t|D])}|dk(r|n|}|jt|||+tt j |S)Nr r)r7)rkrlrmrJr rA)r^ryrzr7rcrpresnet_in_channelss r2get_up_block_adapterr}se LJ : W45F0 N+=a+@BTUVW &2==3N OOr1c :eZdZdZe ddededeedede de eeefdeed eed ed eed e ed e eeefde dede ffd Z e ddede ede eede eede dedededeefdZdZxZS)ControlNetXSAdaptera A `ControlNetXSAdapter` model. To use it, pass it into a `UNetControlNetXSModel` (together with a `UNet2DConditionModel` base model). This model inherits from [`ModelMixin`] and [`ConfigMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). Like `UNetControlNetXSModel`, `ControlNetXSAdapter` is compatible with StableDiffusion and StableDiffusion-XL. It's default parameters are compatible with StableDiffusion. Parameters: conditioning_channels (`int`, defaults to 3): Number of channels of conditioning input (e.g. an image) conditioning_channel_order (`str`, defaults to `"rgb"`): The channel order of conditional image. Will convert to `rgb` if it's `bgr`. conditioning_embedding_out_channels (`tuple[int]`, defaults to `(16, 32, 96, 256)`): The tuple of output channels for each block in the `controlnet_cond_embedding` layer. time_embedding_mix (`float`, defaults to 1.0): If 0, then only the control adapters's time embedding is used. If 1, then only the base unet's time embedding is used. Otherwise, both are combined. learn_time_embedding (`bool`, defaults to `False`): Whether a time embedding should be learned. If yes, `UNetControlNetXSModel` will combine the time embeddings of the base model and the control adapter. If no, `UNetControlNetXSModel` will use the base model's time embedding. num_attention_heads (`list[int]`, defaults to `[4]`): The number of attention heads. block_out_channels (`list[int]`, defaults to `[4, 8, 16, 16]`): The tuple of output channels for each block. base_block_out_channels (`list[int]`, defaults to `[320, 640, 1280, 1280]`): The tuple of output channels for each block in the base unet. cross_attention_dim (`int`, defaults to 1024): The dimension of the cross attention features. down_block_types (`list[str]`, defaults to `["CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "DownBlock2D"]`): The tuple of downsample blocks to use. sample_size (`int`, defaults to 96): Height and width of input/output sample. transformer_layers_per_block (`Union[int, Tuple[int]]`, defaults to 1): The number of transformer blocks of type [`~models.attention.BasicTransformerBlock`]. Only relevant for [`~models.unet_2d_blocks.CrossAttnDownBlock2D`], [`~models.unet_2d_blocks.UNetMidBlock2DCrossAttn`]. upcast_attention (`bool`, defaults to `True`): Whether the attention computation should always be upcasted. max_norm_num_groups (`int`, defaults to 32): Maximum number of groups in group normal. The actual number will be the largest divisor of the respective channels, that is <= max_norm_num_groups. conditioning_channelsconditioning_channel_order#conditioning_embedding_out_channelstime_embedding_mixlearn_time_embeddingrSblock_out_channelsbase_block_out_channelsrTdown_block_types sample_sizerRrVrQrWct!||d}|ddz}|dvrtd|t|t| k7rtd|d| dt | t t fs| gt| z} t | t t fs| gt| z} t |t t fs|gt| z}t|t| k7rtd|d| dt|d|| |_|rt|||_ nd|_ tjg|_ tjg|_tjd|dd d |_t#|d|d|_|d}|d}t'| D]c\}}|}||}|}||}d |v}|t| d z k(}|jj)t+|||||||| |||| || | | et-|d|d|| d|d| d| ||_|dg}t'|D]/\}}|t|d z krd nd}|j1|g|z1t t3|}|d}t5t| D]V}|}||}t5d Dcgc]}|j7} }|jj)t9||| Xycc}w)Nr)rgbbgrz&unknown `conditioning_channel_order`: zbMust provide the same number of `block_out_channels` as `down_block_types`. `block_out_channels`: z. `down_block_types`: .zdMust provide the same number of `num_attention_heads` as `down_block_types`. `num_attention_heads`: conditioning_embedding_channelsrrr r& kernel_sizepadding CrossAttn) rLrMrNrOrPrQrorRrSrTrUrVrW)rsrtrPrRrSrTrVrWr)r^ryrz)r<r= ValueErrorlenrilisttupler'controlnet_cond_embeddingrtime_embeddingr rA down_blocksup_connectionsrBconv_inrmcontrol_to_base_for_conv_in enumeraterlrrrx mid_blockextendreversedrkpopr})"r?rrrrrrSrrrTrrrRrVrQrWtime_embedding_input_dimtime_embedding_dimrMrOrpdown_block_typerLrNrois_final_blockrzr^number_of_subblocks reversed_base_block_out_channelsprev_base_output_channel_ctrl_skip_channels_r@s" r2r=zControlNetXSAdapter.__init__$s0 #:1#= 4Q7!; &^ ;EF`Eabc c ! "c*:&; ;tvHuII_`p_qqrs 6u F,H+ICP`La+a (-e}=#6"7#>N:O"O -e}=#6"7#>N:O"O  " #s+;'< <vxKwLLbcsbttuv  *I,>q,AB"7* & "34LN`"aD "&D ==, mmB/yy$6q$9qRST +9:LQ:OQhijQk+l(4A6.q1"+,<"=  A0  7 : 0  21 5 '?:M#&6"7!";;N    # #&%5&7%5&7"4(;"/1Ma1P(;A(>(;A(>'5#5%5*?  6/1"5,R0,)Eb)I 3B 7 3B 7-"7  134();< LOA|/0144!   % %|n7J&J K  L ,09P0Q+R(UNetControlNetXSModela9 A UNet fused with a ControlNet-XS adapter model This model inherits from [`ModelMixin`] and [`ConfigMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). `UNetControlNetXSModel` is compatible with StableDiffusion and StableDiffusion-XL. It's default parameters are compatible with StableDiffusion. It's parameters are either passed to the underlying `UNet2DConditionModel` or used exactly like in `ControlNetXSAdapter` . See their documentation for details. Trrup_block_typesrrdrTrRrSaddition_embed_typeaddition_time_embed_dimrVrWtime_cond_proj_dim%projection_class_embeddings_input_dimrctrl_conditioning_channels(ctrl_conditioning_embedding_out_channelsctrl_conditioning_channel_orderctrl_learn_time_embeddingctrl_block_out_channelsctrl_num_attention_headsctrl_max_norm_num_groupsct0||dks|dkDr td|dkr |s td| | dk7r tdt|tt fs|gt |z}t|tt fs|gt |z}t|tt fs|gt |z}t|tt fs|gt |z}|}d|_tjd|ddd |_ t|d|| |_ tjd|ddd |_ t|d|d|_|d}|ddz}t!|dd d |_t%||| |_|rt%|||_nd|_| d|_d|_n$t!| d d |_t%|||_g}|d}|d}t/|D]^\}}|}||}|} ||}d|v}!|t |dz k(}"|j1t3||| |||||!|||||||||" | | `t5|d|d||||d|d|d|d| |  |_g}#t t9|}$t t9|}%t t9|}&|dg}'t/|D]/\}}(|t |dz krdnd})|'j;|(g|)z1t t9|}*|*d}(t/|D]\}}+|(},|*|}(|*t=|dzt |dz }-t?dD.cgc]}.|'jA}/}.d|+v}!|t |dz k(}"|#j1tC|-|(|,|/|||!|$||%||&||" | || tjD||_#tjD|#|_$tjJ|d||_&tjN|_(tj|dddd |_)ycc}.w)Nrr&z1`time_embedding_mix` needs to be between 0 and 1.zKTo use `time_embedding_mix` < 1, `ctrl_learn_time_embedding` must be `True` text_timezAs `UNetControlNetXSModel` currently only supports StableDiffusion and StableDiffusion-XL, `addition_embed_type` must be `None` or `'text_time'`.rr rrT)flip_sin_to_cosdownscale_freq_shift) cond_proj_dim)r]time_embed_dimrrLrMrNrOrPrdrrorRbase_num_attention_headsrrTrUrVrWr rsrtrPrdrrRrrrTrVrWr)r]r^ryrzrPresolution_idxrorRrSrT add_upsamplerVrdrW) num_channels num_groups)*r<r=rrirrrr]r rB base_conv_inr'r ctrl_conv_inrmrrbase_time_projrbase_time_embeddingctrl_time_embeddingbase_add_time_projbase_add_embeddingrrl ControlNetXSCrossAttnDownBlock2DControlNetXSCrossAttnMidBlock2DrrrminrkrControlNetXSCrossAttnUpBlock2DrAr up_blocks GroupNormbase_conv_norm_outSiLU base_conv_act base_conv_out)1r?rrrrrdrTrRrSrrrVrWrrrrrrrrrrrtime_embed_input_dimrrrMrOrprrLrNrorr rev_transformer_layers_per_blockrev_num_attention_headsrev_cross_attention_dimrzr^rreversed_block_out_channels up_block_typeryr]rrr@s1 r2r=zUNetControlNetXSModel.__init__sBB   !%7!%;PQ Q  !*Cjk k  */Bk/Qd 6u F,H+ICP`La+a (-e}=#6"7#>N:O"O -e}=#6"7#>N:O"O 2T5MB(@'ACHXDY'Y $#6 IIa);A)>AWXY)H,CA,FG"<* & IIa)@)CQR\]^+9:QRS:TVhijVk+l( 2!4+A.2'(:1(=tjkl#4 ,$  %'80(D $(,D $  &&*D #&*D #&/0GY]tu&vD #&78]_m&nD # .q13A6"+,<"=  A0  21 5 0  7 : '?:M#&6"7!";;N   0%5&7%5&7"0$3-E"/1Ma1P-Ea-H-Ea-H(;A(>'5#5%5*?  :9,R01"5(+%=)Eb)I%=b%A%=b%A 3B 7-"7   +/9U0V+W("&x0H'I"J"&x0C'D"E6a89()@A LOA|4599q   % %|n7J&J K  L '+84F+G&H#215 ). 9  A}". 6q9L5c!a%EWAX[\A\6]^KEJ1X"N#5#9#9#;"N "N'=8M#&8"9A"==N   . +!-(;':"0#$"/1QRS1T(?(B(?(B%3!3%5$3*?  8==5y1"$,,z2UNetControlNetXSModel.from_unet..s"# szWhen a controlnet is passed, none of these parameters should be passed: size_ratio, ctrl_block_out_channels, time_embedding_mix, ctrl_optional_kwargs.) rrrrrdrTrRrrrVrWrrrS)rrrrrrSrQctrl_r)rr conv_norm_outconv_outbase_ add_time_proj add_embeddingc3NK|]\}}tj||ywr;)r from_modulesrrcs r2rz2UNetControlNetXSModel.from_unet..7s)* 1 - 9 9!Q ?* #%c3NK|]\}}tj||ywr;)rrrs r2rz2UNetControlNetXSModel.from_unet..<s)( 1 + 7 71 =( r)rranyrritemsrr from_configgetattrload_state_dict state_dicthasattrrrrrrrr rAziprrrrrrrr)rrrrrrrparams_for_unetkvparams_for_controlnetrmodules_from_unetmoptional_modules_from_unets r2rzUNetControlNetXSModel.from_unets <  ,66j"9=QJ(24KM_au'v!m  -1KK,=,=,?XDAq1CW1a4XX151O1O-.! =G.fn_recursive_add_processorsnsdv/282F2F2H dV:./#)#8#8#: U%+tfAhZ,@%T U r1)rtorchr Modulerrr5)r?r1rhr0r8s @r2attn_processorsz%UNetControlNetXSModel.attn_processorscsm  c 588?? X\]`bt]tXu !//1 BLD& 'fj A Br1 processorc Tt|jj}t|tr,t||k7rt dt|d|d|ddt dtjjffd |jD]\}}|||y) a4 Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.rhr0ct|drEt|ts|j|n#|j|j |d|j D]\}}|d|||y)N set_processorr4r)rridictr?rr5)rhr0r<r6r7fn_recursive_attn_processors r2rAzMUNetControlNetXSModel.set_attn_processor..fn_recursive_attn_processorsxv/!)T2((3(($z7J)KL#)#8#8#: T%+tfAhZ,@%S Tr1N) rr;keysrir@rrr9r r:r5)r?r<countrhr0rAs @r2set_attn_processorz(UNetControlNetXSModel.set_attn_processor}sD((--/0 i &3y>U+BPQTU^Q_P`a005w6QRWQXXkm   Tc T588?? T!//1 ALD& 'fi @ Ar1c jtd|jjDr t}nmtd|jjDr t }n8t dt t|jj|j|y)ze Disables custom attention processors and sets the default attention implementation. c3@K|]}|jtvywr;)r@rrprocs r2rzCUNetControlNetXSModel.set_default_attn_processor..si4t~~!>>ic3@K|]}|jtvywr;)r@rrGs r2rzCUNetControlNetXSModel.set_default_attn_processor..sh$#==hrIzOCannot call `set_default_attn_processor` when attention processors are of type N) allr;valuesrrrnextiterrD)r?r<s r2set_default_attn_processorz0UNetControlNetXSModel.set_default_attn_processors i4K_K_KfKfKhi i,.I h$J^J^JeJeJgh h%IabfgklpmAmAmHmHmJhKcLbMN   *r1s1s2b1b2ct|jD]9\}}t|d|t|d|t|d|t|d|;y)aFEnables the FreeU mechanism from https://huggingface.co/papers/2309.11497. The suffixes after the scaling factors represent the stage blocks where they are being applied. Please refer to the [official repository](https://github.com/ChenyangSi/FreeU) for combinations of values that are known to work well for different pipelines such as Stable Diffusion v1, v2, and Stable Diffusion XL. Args: s1 (`float`): Scaling factor for stage 1 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. s2 (`float`): Scaling factor for stage 2 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. b1 (`float`): Scaling factor for stage 1 to amplify the contributions of backbone features. b2 (`float`): Scaling factor for stage 2 to amplify the contributions of backbone features. rPrQrRrSN)rrsetattr)r?rPrQrRrSrpupsample_blocks r2 enable_freeuz"UNetControlNetXSModel.enable_freeusQ$"+4>>!: . A~ ND" - ND" - ND" - ND" -  .r1chd}t|jD]3\}}|D])}t||st||dt ||d+5y)zDisables the FreeU mechanism.>rRrSrPrQN)rrrrrU)r? freeu_keysrprVrs r2 disable_freeuz#UNetControlNetXSModel.disable_freeusW- !*4>>!: 5 A~ 5>1-D1Q1]NAt4 5 5r1crd|_|jjD]1\}}dt|jj vs(t d|j|_|jD]%}t|ts|jd'|jty)u1 Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. This API is 🧪 experimental. NAddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.T)fuse) original_attn_processorsr;rrr@r+rmodulesrirfuse_projectionsrDr)r?rattn_processorr0s r2fuse_qkv_projectionsz*UNetControlNetXSModel.fuse_qkv_projectionss)-%!%!5!5!;!;!= v A~#n66??@@ !tuu v)-(<(<%lln 3F&),''T'2 3  5 78r1cT|j|j|jyy)uDisables the fused QKV projection if enabled. This API is 🧪 experimental. N)r^rD)r?s r2unfuse_qkv_projectionsz,UNetControlNetXSModel.unfuse_qkv_projectionss)  ( ( 4  # #D$A$A B 5r1r*timestepencoder_hidden_statescontrolnet_condconditioning_scale class_labels timestep_condattention_maskcross_attention_kwargsadded_cond_kwargs return_dict apply_controlc  |jjdk(rtj|dg}|2d|j |j z dz}|j d}|} tj| s|jjdk(}|jjdk(}t|tr%|s|rtjntj}n$|s|rtjntj}tj | g||j} n6t#| j$d k(r| dj |j} | j'|j$d } |j)| }|j |j }|jj*rN| rL|j-||}|j/||}|jj0d z}||z|d|z zz}n|j/|}d}|jj2n|jj2d k(rd | vrt5|j6d| j9d }d| vrt5|j6d| j9d}|j;|j=}|j?|j$d df}tj@||gd}|j |j }|jC|}n#t5d|jj2d|||zn|}|}|x}}gg}}|jE|} |jG|}|jI|}| || z }| r||jK||zz}|jM||jM||jND]9}!|!|||||| || \}}}"}#|jQ|"|jQ|#;|jS|||||| || \}}|jTD]@}$t#|$jV}%||% d}&||% d}'|d|% }|d|% }|$||&|'|||| ||  }B|jY|}|j[|}|j]|}| s|fSt_|S)a The [`ControlNetXSModel`] forward method. Args: sample (`Tensor`): The noisy input tensor. timestep (`Union[torch.Tensor, float, int]`): The number of timesteps to denoise an input. encoder_hidden_states (`torch.Tensor`): The encoder hidden states. controlnet_cond (`Tensor`): The conditional input tensor of shape `(batch_size, sequence_length, hidden_size)`. conditioning_scale (`float`, defaults to `1.0`): How much the control model affects the base model outputs. class_labels (`torch.Tensor`, *optional*, defaults to `None`): Optional class labels for conditioning. Their embeddings will be summed with the timestep embeddings. timestep_cond (`torch.Tensor`, *optional*, defaults to `None`): Additional conditional embeddings for timestep. If provided, the embeddings will be summed with the timestep_embedding passed through the `self.time_embedding` layer to obtain the final timestep embeddings. attention_mask (`torch.Tensor`, *optional*, defaults to `None`): An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. If `1` the mask is kept, otherwise if `0` it is discarded. Mask will be converted into a bias, which adds large negative values to the attention scores corresponding to "discard" tokens. cross_attention_kwargs (`dict[str]`, *optional*, defaults to `None`): A kwargs dictionary that if specified is passed along to the `AttnProcessor`. added_cond_kwargs (`dict`): Additional conditions for the Stable Diffusion XL UNet. return_dict (`bool`, defaults to `True`): Whether or not to return a [`~models.controlnets.controlnet.ControlNetOutput`] instead of a plain tuple. apply_control (`bool`, defaults to `True`): If `False`, the input is run only through the base model. Returns: [`~models.controlnetxs.ControlNetXSOutput`] **or** `tuple`: If `return_dict` is `True`, a [`~models.controlnetxs.ControlNetXSOutput`] is returned, otherwise a tuple is returned where the first element is the sample tensor. rr&)dimsNgmpsnpu)rdevicer)rg333333?r text_embedsz has the config param `addition_embed_type` set to 'text_time' which requires the keyword argument `text_embeds` to be passed in `added_cond_kwargs`time_idsz has the config param `addition_embed_type` set to 'text_time' which requires the keyword argument `time_ids` to be passed in `added_cond_kwargs`rdimzgControlNet-XS currently only supports StableDiffusion and StableDiffusion-XL, so addition_embed_type = z is currently not supported.)hidden_states_basehidden_states_ctrltembrfrhrlrkro) hidden_statesres_hidden_states_tuple_baseres_hidden_states_tuple_ctrlr{rfrhrlrkro)r*)0rrr9fliprr unsqueeze is_tensorrttyperirfloat32float64int32int64tensorrshapeexpandrrrrrrrr@getrflattenreshapeconcatrrrrrrlrrrrr5rrrr))(r?r*rerfrgrhrirjrkrlrmrnro timestepsis_mpsis_npurt_emb ctrl_temb base_tembinterpolation_paramr{aug_embrurv time_embeds add_embedscembh_ctrlh_basehs_basehs_ctrl guided_hintdown residual_hb residual_hcup n_resnetsskips_hbskips_hcs( r2rzUNetControlNetXSModel.forwardsp ;; 6 6% ?#jjsCO  %."3"3FLL"AAXMN+55a8N y)]]''50F]]''50F(E**0F (.& u{{ i[fmmTI  !Q &!$**6==9I$$V\\!_5 ##I. v||, ;; 0 0]00 FI00 FI"&++"@"@#"E 22Y!FYBY5ZZD++E2D ;; * * 2  [[ , , ;$55 ~~&'{|,// >K!22 ~~&'xy),,Z8H11(2B2B2DEK%--{/@/@/CR.HIK{K&@bIJ#tzz2J--j9Gyz~{F{F{Z{Zz[[wx ")!4tg~$%! r44_E ""6*""6*  " k !F d>>vFI[[[Fvv$$ (D7;#)#)&*#5'=-+ 8 4FFK NN; ' NN; ' (%%"&1#9)'(  .. BBJJI z{+H z{+Hk z*Gk z*G$-5-5&*#5'=-+ F  &((0##F+##F+9 !00r1)rr) UpBlock2Drrrrrrr&rNNTTNNrr rrFrrr)NNNNNr#N) NrNNNNNTT)$r+r,r-r. _supports_gradient_checkpointingrrrjr rr rrr=rr%rrrrr.propertyrr;rDrOrWrZrbrdr r9rr)rrCrDs@r2rrs (,$&(( &u)?)+6:?@67-115!%&*,0?C$'*+?P/4*/.<;<(*=|[c]|[ * |[c |["#J|["#|[#3c ?3|[',CsO&<|[#3c ?3|[ &c]!|[""*##|[$%|[& $'|[(%SM)|[*08}+|[."/|[0%(1|[238*3|[4*-5|[6$(7|[8"'s9|[:#(U3Z"8;|[<#&=|[|[|59&*9=.2/3t"t01tUO t "*$u+!6 t %UO t'tnttl#:c+=&=!>0 AE2Dd3PbKbFc2c,d AF + .u.%.U..2594 C$37.1/30415;??C "J1J1 eS01J1 %|| J1 "%,,/ J1 %UO J1u||,J1  -J1!.J1!)c3h 8J1$Dell):$;<J1J1J1 !5( )J1r1rcTeZdZ d dedededededededeeeeefd eed eed eed ed eedeeffd Ze de de fdZ d!dZ d"dededeedeedeedeedeeeefdeededeeeeedfeedfffdZxZS)#rrLrMrNrOrPrdrrRrrrTrUrVrWct|g}g}g}g}g}g}d}t| tr| g|z} t |D]}|dk(r|n|}|dk(r|n|}|j t |||j t|||||j t||z||t||z|t||d|r[|j t| || z|| || ||||j t| || z|| || ||t|||j t ||| rb|j t ||t|d|d |_ t||zd|d |_ |j t ||nd|_ d|_ tj||_tj||_|rtj|ndg|z|_|rtj|ndg|z|_tj||_tj||_d |_y) Nrrr]r^rPr_rYr[r\rbTrerfF)r<r=rirjrkrlrmr!rnr"r base_downsamplersctrl_downsamplersr rA base_resnets ctrl_resnetsbase_attentionsctrl_attentionsr6r7gradient_checkpointing)r?rLrMrNrOrPrdrrorRrrrTrUrVrWrrrrr7r6rcrpr@s r2r=z)ControlNetXSCrossAttnDownBlock2D.__init__s$      2C 8,H+IJ+V (z"9 VA346/?P 346/?P    /?AQ R S    0!2"/*      03C C!2"/.(+;;H` 33DQij  &&&0)-EE$5#?#B,?.C)9(7   &&&0)-EE$5#?#B,?.C)9(;.get_first_cross_attention@$##A&99!<BB Br1rr8TFr)r5r]r^ time_emb_proj in_featuresnorm1rrrr8rheadsrTrVrWr>rrrrrrrrr6r7)rrrrrLrMrNrOrPrctrl_num_groupsrorRrrrTrVrWrUrs r2rz-ControlNetXSCrossAttnDownBlock2D.from_modules=s C*11!4@@*2215BB  " "1 % 1 14D D +2215BB&..q1??KK #++A.44?? (00399DD >< 0 M+.~/H/H/K/^/^+_ ('@'P'V'V $'@'P'V'V $";N"K"_"_ 8HYY $2$=$=a$@$V$V !!M+/ ('+ $'+ $"& # $( !'44D@-/-/'&%4')E%=%= 3)-"7 & **>+A+A+L+L+NO **>+A+A+L+L+NO   ! ! 1 1.2K2K2V2V2X Y  ! ! 1 1.2K2K2V2V2X Y   # # 3 3N4O4OPQ4R4]4]4_ `  # # 3 3N4O4O4Z4Z4\ ] **>+F+F+Q+Q+ST **>+F+F+Q+Q+ST r1r#ch|jD] }d|_ |jg}t|jt j r|j|j|j|j|j|D]}|jD] }d|_  yr%TNF) r&r'rrirr rArlrr?r)r*r+s r2r(z3ControlNetXSCrossAttnDownBlock2D.freeze_base_paramss__& 'E"&E  '''( d**BMM :   d22 3  ! ! -   d44 5 ,D* ,&+# , ,r1ryr{rfrzrhrkrlencoder_attention_maskro.c L|'|jddtjd|} |} d} d} tt |j |j }tt |j|j}t |||j|jD]\\}}\}}}}| rtj| || gd} tjr |jr|j|| |} n || |} ||| ||||dd} | rPtjr |jr|j|| |} n || |} ||| ||||dd} | r| || |zz} | | fz} | | fz} |j |jd }|jd }| rtj| || gd} |j!| } | r|j#| } | r| || |zz} | | fz} | | fz} | | | | fS) NscaleSPassing `scale` to `cross_attention_kwargs` is deprecated. `scale` will be ignored.r0r&rwFrfrlrkrrnrr)rloggerwarningrrrrrrr6r7r9catis_grad_enabledr_gradient_checkpointing_funcrr)r?ryr{rfrzrhrkrlrrorrbase_output_statesctrl_output_states base_blocks ctrl_blocksb_resb_attnc_resc_attnb2cc2bs r2rz(ControlNetXSCrossAttnDownBlock2D.forwards " -%))'48Dtu##3t00$2F2FGH 3t00$2F2FGH := d&7&79J9J; , @ 6OUF_eVc3FCK#8a@$$&4+F+F::5&$Ovt,!*?+A#1+A % ((*t/J/J!>>ufdSF"640F%#.C/E'5/E$) F#f+0B"BB!3vi!? !3vi!? Y, @\  ! ! -##B'C##B'CFCK#8a@++F3F//7#f+0B"BB!3vi!? !3vi!? v13EEEr1) rrTr&r&r&rTFTr)NNrNNNT)r+r,r-rjrr r rr=rrr4rr(r rrrrrrCrDs@r2rrs "(*IJ2323-1#+004!t,t,t, t,  t,  t,t,#&t,'/uS%*_/E&Ft,#+3-t,#+3-t,&c]t,t,#4.t, (~!t,l?*>?Pl??B,*37/3.1+/;?37"ZF"ZFZF (/ ZF %V, ZF %UO ZF!(ZF!)c3h 8ZF!) 0ZFZF vvuVS[153EE FZFr1rceZdZ ddededeededededeedeed eed ed eeffd Zed ede fdZ ddZ dde de de dee dee deeeefdee dee dedee e ffdZxZS)rrsrtrPrdrrRrrrTrVrWc  t |t|||_t ||||| || | |_t |||z||t t|||z|| || |  |_t|||_ d|_ y)N)rRr]rPrwrTrSrWrVrvF) r<r=rmr6r# base_midblockrnr ctrl_midblockr7r) r?rsrtrPrdrrRrrrTrVrWr@s r2r=z(ControlNetXSCrossAttnMidBlock2D.__init__s +=-H4)E%') 3 8"7-  5)E% 5&'-M==#@AC[!4 8"7-  "+=-H&+#r1rrc |j}|j}|j}d}|j}|j}t |j dj}|jdjj} |jdjj} |jdjj} ||j} ||j} ||j}||j}|j dj }|||| | | || | ||| }|jj#|j%|j&j#|j%|j(j#|j%|jj#|j%|S)NcN|jdjdjSrr)rGs r2rzOControlNetXSCrossAttnMidBlock2D.from_modules..get_first_cross_attention-s$&&q)<>JJ "**1-33>> '//288CC#<]#K#Q#Q #<]#K#Q#Q 7 FZZ4]CTT - 8 8 ; Q Q'''&%4)E%=%= 3-"7   **<+B+B+DE ++M,D,D,FG ++M,D,D,FG **<+B+B+DE r1r#c|jD] }d|_ |jjD] }d|_ y)r%TFN)r&r'r)r?r)s r2r(z2ControlNetXSCrossAttnMidBlock2D.freeze_base_paramsSsJ__& 'E"&E  '''224 (E"'E  (r1ryr{rfrzrhrlrkrroc F|'|jddtjd|} |} |||||d} | r(tj| |j | gd} |j | fi| } | r*|j| fi| } | |j| |zz} | | fS)Nrr)r{rfrkrlrr&rw) rrrr9rr6rrr7) r?ryr{rfrzrhrlrkrrorr joint_argss r2rz'ControlNetXSCrossAttnMidBlock2D.forward^s " -%))'48Dtu##%:,&<&<   YY(9(9&(ABJF###F9j9 'T''=*=Fd//7:LLLFv~r1) Nrrr&r&r&rFTr)NrNNNT)r+r,r-rjrrr=rr#rFrr(r rrrrr rrCrDs@r2rrsv (,!(*,-2323-1!&042,2,2, } 2,  2, #& 2,'*2,#+3-2,#+3-2,&c]2,2, (~2,h...3..` ( 04.1;?+/37""""" & " %V, " %UO "!)c3h 8"!("!) 0"" vv~ "r1rc&eZdZ d!dedededeedededeeded ed ed ed ed eeffd Zede de fdZ d"dZ d#de dee dfdee dfde dee deedeeeefdee deedee dede fd ZxZS)$rr]r^ryrzrPrdrrRrSrTrrVrWc t|g}g}g}d}||_| |_t | t r| g|z} t |D]}||dz k(r|n|}|dk(r|n|}|jt||||jt||z||||sZ|jt| || z|| || || |tj||_ |rtj|ndg|z|_tj||_| rt!|d||_nd|_d|_||_y) Nr r&rrrbT)rgr^F)r<r=has_cross_attentionrSrirjrkrlrmr!r"r rAr5r8r7r$ upsamplersrr)r?r]r^ryrzrPrdrrorRrSrTrrVrWr5r8r7rcrpres_skip_channelsr|r@s r2r=z'ControlNetXSCrossAttnUpBlock2D.__init__sm"    #0 #6 2C 8,H+IJ+V (z" A01Z!^0C , 89Q!4L    /A!/DFX Y Z NN 25F F!-"/*  !!&+$(;;$0#?#B,?.C)9(7  ! :}}W- 7D"-- 34&S]J]MM,7 (S_`DO"DO&+#,r1 base_upblock ctrl_upblockc|j}d}|jdj}|jdj|z }|jdj|z }|Dcgc]}|j} }|jdjj } |jdj j} |j} t|drtd} t|jdj}||j}||j}||j}|jdj }n d} d}d}d}d}d}|j"du}||||| | | | | ||||||}|jj%|jj'| r3|jj%|jj'|r6|j"j%|j"dj'|jj%|j'|Scc}w)NcN|jdjdjSrrrs r2rzNControlNetXSCrossAttnUpBlock2D.from_modules..get_first_cross_attentionrr1rrr8TF)r]r^ryrzrPrdrrorRrSrTrrVrW)r7r5r^r]rrrrrrrr8rrrTrVrWrrr)rrrctrl_to_base_skip_connectionsrr^r]prev_output_channelsrctrl_skip_channelssrPrrrorRrSrTrVrWrrs r2rz+ControlNetXSCrossAttnUpBlock2D.from_moduless:(4(A(A% C$++A.;; "**2.::\I +33A6BB\Q6STq}}TT$,,Q/==II !))!,22== %44 < . M+.|/F/Fq/I/\/\+] (";L"I"O"O ";L"I"]"] 8FWW $0$;$;A$>$T$T !!M+/ ("& "& # $( !#..d: #% 42'&)')E 3 3%-"7 $  %%l&:&:&E&E&GH     , ,\-D-D-O-O-Q R     , ,\-D-DQ-G-R-R-T U **+H+S+S+UV ]Us%Ir#ch|jD] }d|_ |jg}t|jt j r|j|j|j|j|j|D]}|jD] }d|_  yr) r&r'r5rir8r rArlrrs r2r(z1ControlNetXSCrossAttnUpBlock2D.freeze_base_paramss__& 'E"&E  'll^ door}} 5   doo . ?? &   doo . ,D* ,&+# , ,r1r|r}.r~r{rfrhrlrk upsample_sizerroc |'|jddtjdtddxr+tddxrtddxr tddfd} t j j jt|t|D]\} }}}}| r||||zz }| ||\}}tj||gd }tjr jrj| ||}n | ||}|~|||||| d d }jj|| }|S) NrrrPrQrRrSc rDtj||jjjj S||fS)N)rPrQrRrS)rrrPrQrRrS)r| res_h_baseis_freeu_enabledr?s r2maybe_apply_freeu_to_subblockzMControlNetXSCrossAttnUpBlock2D.forward..maybe_apply_freeu_to_subblock0sK"''!wwwwwwww%j00r1r&rwFrr)rrrrrr5r8r7rr9rrrrr)r?r|r}r~r{rfrhrlrkrrrorresnetattnrr res_h_ctrlrs` @r2rz&ControlNetXSCrossAttnUpBlock2D.forwards " -%))'48Dtu D$ % *dD) *dD) *dD)  1:= LL OO    1 2 1 2 :   5FD#z:Z3E!EE (EmU_(` %M:!II}j&AqIM$$&4+F+F $ A A&-Y] ^ &}d ;  $!*?+A#1+A % !! ' 8 ?? & OOM=IMr1) rNTr&r&rTFTr)NrNNNNT)r+r,r-rjrrrr=rrrJrr(r r rrrrrrCrDs@r2rrs "(,,-#$#'!!&04E-E-E-! E- !I E-  E-E-! E-'*E-!E-!E-E-E- (~E-N8(:8Jd88t,.37.1;?+/'+37"GG',FCK&8G',FCK&8 G  G (/ G%UOG!)c3h 8G!(G }G!) 0GG Gr1rcFttj||ddS)Nr&r)r) zero_moduler rB)r]r^s r2rmrmas ryylAqI JJr1cn|jD]!}tjj|#|Sr;)r&r initzeros_)r0ps r2rres/     q Mr1cP|}||k\r|S|dk7r||z}|dk(r|S|dz}|dk7ryy)Nrr&r0)numberrZfactorresiduals r2rnrnksB F  A+F? q=M!  A+r1)rTr&r&rTFT)Nrr&r&rFTr;)E dataclassesrmathrtypingrrrrr r r9torch.utils.checkpointr r configuration_utilsrrutilsrrutils.torch_utilsrattention_processorrrrrrrr embeddingsrrmodeling_utilsrunets.unet_2d_blocksrrr r!r"r#r$unets.unet_2d_conditionr%rr' get_loggerr+rr)r:r4rFrJrjrrrrxr}rrrrrrmrrnr0r1r2rs":: B(,6';7   H %     (299(()")))))*,EF)*)-',,0X!X!X!X! X!  X! "# X!#+5eCj+A"BX!"#X!"#X!X!tnX!$D>X!|$()+())*)-""& p p pC= p"# p #& p "# p"# p p  pF P P PS  PH *kH VC 1J C 1LdFryydFN SbiiSl[RYY[|K r1