bi zddlmZddlmZmZmZmZmZmZddl Z ddl m Z ddl Z ddl mZmZddlmZmZddlmZddlmZmZmZmZmZmZd d lmZd d lmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%d d l&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0d d l1m2Z2ddl3m4Z4m5Z5m6Z6ejne8Z9eGddeZ:Gdde2eeeeZ;y)) dataclass)AnyDictListOptionalTupleUnionN) ConfigMixinregister_to_config)PeftAdapterMixinUNet2DConditionLoadersMixin)FromOriginalModelMixin)USE_PEFT_BACKEND BaseOutput deprecateloggingscale_lora_layersunscale_lora_layers)get_activation)ADDED_KV_ATTENTION_PROCESSORSCROSS_ATTENTION_PROCESSORS AttentionAttentionProcessorAttnAddedKVProcessor AttnProcessorFusedAttnProcessor2_0) GaussianFourierProjectionGLIGENTextBoundingboxProjectionImageHintTimeEmbeddingImageProjectionImageTimeEmbeddingTextImageProjectionTextImageTimeEmbeddingTextTimeEmbeddingTimestepEmbedding Timesteps) ModelMixin)get_down_block get_mid_block get_up_blockc4eZdZUdZdZej ed<y)UNet2DConditionOutputa The output of [`UNet2DConditionModel`]. Args: sample (`torch.Tensor` of shape `(batch_size, num_channels, height, width)`): The hidden states output conditioned on `encoder_hidden_states` input. Output of last layer of model. Nsample)__name__ __module__ __qualname____doc__r0torchTensor__annotations__c/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/models/unets/unet_2d_condition.pyr/r/:s FELLr9r/cbeZdZdZdZgdZdgZdgZe dade e e e e e ffde de d e d e d e d e ed e ede ede e e e fde e de e e e fde dededede e dede e e e fde e e e e e fde e e e de e de ede e e e fde e e e e fde d e d!e ed"e ed#e e d$e e d%e d&ed'e d(ed)ed*e e d+e ed,e ed-e e d.e d/e d0e e d1ed2e d3e e d4e ed5e f`fd6 Zd e ede ede e e e fde e de e e e fde e e e fde e e e e e e fde de de e e e e ffd7Zd)ede d e d ed*e d8e e e ff d9Zde ede e e e fde e fd:Zd!e eded$e e d0e e d;e dZd1ede fd?Zed8eeeffd@ZdAe eeeefffdBZdCZdbdDe ee ee ffdEZdFedGedHedIefdJZdKZ dLZ!dMZ"dNe#jHdOe e#jHee fd8e e#jHfdPZ%dNe#jHdQe e#jHd8e e#jHfdRZ&dSe#jHdTe#jHdUeee'fd8e e#jHfdVZ(dTe#jHdUeee'fd8e#jHfdWZ) dcdNe#jHdOe e#jHee fdTe#jHdQe e#jHdXe e#jHdYe e#jHdZe eee'fdUe eee#jHfd[e e e#jHd\e e#jHd]e e e#jHd^e e#jHd_e d8e e*e ffd`Z+xZ,S)dUNet2DConditionModela A conditional 2D UNet model that takes a noisy sample, conditional state, and a timestep and returns a sample shaped output. This model inherits from [`ModelMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). Parameters: sample_size (`int` or `Tuple[int, int]`, *optional*, defaults to `None`): Height and width of input/output sample. in_channels (`int`, *optional*, defaults to 4): Number of channels in the input sample. out_channels (`int`, *optional*, defaults to 4): Number of channels in the output. center_input_sample (`bool`, *optional*, defaults to `False`): Whether to center the input sample. flip_sin_to_cos (`bool`, *optional*, defaults to `True`): Whether to flip the sin to cos in the time embedding. freq_shift (`int`, *optional*, defaults to 0): The frequency shift to apply to the time embedding. down_block_types (`Tuple[str]`, *optional*, defaults to `("CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "DownBlock2D")`): The tuple of downsample blocks to use. mid_block_type (`str`, *optional*, defaults to `"UNetMidBlock2DCrossAttn"`): Block type for middle of UNet, it can be one of `UNetMidBlock2DCrossAttn`, `UNetMidBlock2D`, or `UNetMidBlock2DSimpleCrossAttn`. If `None`, the mid block layer is skipped. up_block_types (`Tuple[str]`, *optional*, defaults to `("UpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D")`): The tuple of upsample blocks to use. only_cross_attention(`bool` or `Tuple[bool]`, *optional*, default to `False`): Whether to include self-attention in the basic transformer blocks, see [`~models.attention.BasicTransformerBlock`]. block_out_channels (`Tuple[int]`, *optional*, defaults to `(320, 640, 1280, 1280)`): The tuple of output channels for each block. layers_per_block (`int`, *optional*, defaults to 2): The number of layers per block. downsample_padding (`int`, *optional*, defaults to 1): The padding to use for the downsampling convolution. mid_block_scale_factor (`float`, *optional*, defaults to 1.0): The scale factor to use for the mid block. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. act_fn (`str`, *optional*, defaults to `"silu"`): The activation function to use. norm_num_groups (`int`, *optional*, defaults to 32): The number of groups to use for the normalization. If `None`, normalization and activation layers is skipped in post-processing. norm_eps (`float`, *optional*, defaults to 1e-5): The epsilon to use for the normalization. cross_attention_dim (`int` or `Tuple[int]`, *optional*, defaults to 1280): The dimension of the cross attention features. transformer_layers_per_block (`int`, `Tuple[int]`, or `Tuple[Tuple]` , *optional*, defaults to 1): The number of transformer blocks of type [`~models.attention.BasicTransformerBlock`]. Only relevant for [`~models.unets.unet_2d_blocks.CrossAttnDownBlock2D`], [`~models.unets.unet_2d_blocks.CrossAttnUpBlock2D`], [`~models.unets.unet_2d_blocks.UNetMidBlock2DCrossAttn`]. reverse_transformer_layers_per_block : (`Tuple[Tuple]`, *optional*, defaults to None): The number of transformer blocks of type [`~models.attention.BasicTransformerBlock`], in the upsampling blocks of the U-Net. Only relevant if `transformer_layers_per_block` is of type `Tuple[Tuple]` and for [`~models.unets.unet_2d_blocks.CrossAttnDownBlock2D`], [`~models.unets.unet_2d_blocks.CrossAttnUpBlock2D`], [`~models.unets.unet_2d_blocks.UNetMidBlock2DCrossAttn`]. encoder_hid_dim (`int`, *optional*, defaults to None): If `encoder_hid_dim_type` is defined, `encoder_hidden_states` will be projected from `encoder_hid_dim` dimension to `cross_attention_dim`. encoder_hid_dim_type (`str`, *optional*, defaults to `None`): If given, the `encoder_hidden_states` and potentially other embeddings are down-projected to text embeddings of dimension `cross_attention` according to `encoder_hid_dim_type`. attention_head_dim (`int`, *optional*, defaults to 8): The dimension of the attention heads. num_attention_heads (`int`, *optional*): The number of attention heads. If not defined, defaults to `attention_head_dim` resnet_time_scale_shift (`str`, *optional*, defaults to `"default"`): Time scale shift config for ResNet blocks (see [`~models.resnet.ResnetBlock2D`]). Choose from `default` or `scale_shift`. class_embed_type (`str`, *optional*, defaults to `None`): The type of class embedding to use which is ultimately summed with the time embeddings. Choose from `None`, `"timestep"`, `"identity"`, `"projection"`, or `"simple_projection"`. addition_embed_type (`str`, *optional*, defaults to `None`): Configures an optional embedding which will be summed with the time embeddings. Choose from `None` or "text". "text" will use the `TextTimeEmbedding` layer. addition_time_embed_dim: (`int`, *optional*, defaults to `None`): Dimension for the timestep embeddings. num_class_embeds (`int`, *optional*, defaults to `None`): Input dimension of the learnable embedding matrix to be projected to `time_embed_dim`, when performing class conditioning with `class_embed_type` equal to `None`. time_embedding_type (`str`, *optional*, defaults to `positional`): The type of position embedding to use for timesteps. Choose from `positional` or `fourier`. time_embedding_dim (`int`, *optional*, defaults to `None`): An optional override for the dimension of the projected time embedding. time_embedding_act_fn (`str`, *optional*, defaults to `None`): Optional activation function to use only once on the time embeddings before they are passed to the rest of the UNet. Choose from `silu`, `mish`, `gelu`, and `swish`. timestep_post_act (`str`, *optional*, defaults to `None`): The second activation function to use in timestep embedding. Choose from `silu`, `mish` and `gelu`. time_cond_proj_dim (`int`, *optional*, defaults to `None`): The dimension of `cond_proj` layer in the timestep embedding. conv_in_kernel (`int`, *optional*, default to `3`): The kernel size of `conv_in` layer. conv_out_kernel (`int`, *optional*, default to `3`): The kernel size of `conv_out` layer. projection_class_embeddings_input_dim (`int`, *optional*): The dimension of the `class_labels` input when `class_embed_type="projection"`. Required when `class_embed_type="projection"`. class_embeddings_concat (`bool`, *optional*, defaults to `False`): Whether to concatenate the time embeddings with the class embeddings. mid_block_only_cross_attention (`bool`, *optional*, defaults to `None`): Whether to use cross attention with the mid block when using the `UNetMidBlock2DSimpleCrossAttn`. If `only_cross_attention` is given as a single boolean and `mid_block_only_cross_attention` is `None`, the `only_cross_attention` value is used as the value for `mid_block_only_cross_attention`. Default to `False` otherwise. T)BasicTransformerBlock ResnetBlock2DCrossAttnUpBlock2D UpBlock2Dnormr= sample_size in_channels out_channelscenter_input_sampleflip_sin_to_cos freq_shiftdown_block_typesmid_block_typeup_block_typesonly_cross_attentionblock_out_channelslayers_per_blockdownsample_paddingmid_block_scale_factordropoutact_fnnorm_num_groupsnorm_epscross_attention_dimtransformer_layers_per_block$reverse_transformer_layers_per_blockencoder_hid_dimencoder_hid_dim_typeattention_head_dimnum_attention_headsdual_cross_attentionuse_linear_projectionclass_embed_typeaddition_embed_typeaddition_time_embed_dimnum_class_embedsupcast_attentionresnet_time_scale_shiftresnet_skip_time_actresnet_out_scale_factortime_embedding_typetime_embedding_dimtime_embedding_act_fntimestep_post_acttime_cond_proj_dimconv_in_kernelconv_out_kernel%projection_class_embeddings_input_dimattention_typeclass_embeddings_concatmid_block_only_cross_attentioncross_attention_normaddition_embed_type_num_headsc1  tE|||_| td|xs|}|j || | | | ||||| |)dz dz}1t j || d|)|1|_|j|$| |||%\}2}3t|3|2||'|(|_ |j||| |j||||+|2|3 |j||0||||||+|2 |&d|_nt|&|_t j g|_t j g|_t'| t(r|.| }.| gt+|z} |.d }.t'|t,r|ft+|z}t'|t,r|ft+|z}t'|t,r|ft+|z}t'| t,r| gt+|z} t'|t,r|gt+|z}|-r|2dz}4n|2}4| d}5t/|D]\}6}7|5}8| |6}5|6t+| dz k(}9t1|7fid | |6d||6d|8d|5d|4d|9 d|d|d|d||6d||6d| d|d|d| |6d| d|!d|,d|"d |#d!|/d"||6||6n|5d#|}:|j"j3|:t5|fid|4d| d$d|d|d|d%|d|d$d|d$d|d$d|d|d&|.d| d|!d|,d|"d!|/d"|d$d#||_d|_t;t=| };t;t=|}|t;t=|n|}?t;t=| } |;d}5t/| D]\}6}@|6t+| dz k(}9|5}A|;|6}5|;t?|6dzt+| dz }8|9sd'}B|xj8dz c_nd }BtA@fid |=|6dzd|?|6d|8d|5d(Ad|4d)Bd|d|d*|6d|d|>|6d|<|6d|d|d| |6d| d|!d|,d|"d |#d!|/d"||6||6n|5d#|}C|j$j3|C|1t jB| d||+|_"t||_#nd|_"d|_#|*dz dz}Dt j | d||*|D|_$|jK|,|,y)-Na#At the moment it is not possible to define the number of attention heads via `num_attention_heads` because of a naming issue as described in https://github.com/huggingface/diffusers/issues/2011#issuecomment-1547958131. Passing `num_attention_heads` will only be supported in diffusers v0.19.) rHrJrKrLrMrTrUrVrYrZr*rr) kernel_sizepadding)rLrFrGrf)rQ post_act_fn cond_proj_dim)rTrW)rQr`rltime_embed_dimtimestep_input_dim)rqr_rTrWrFrGrlrwF num_layersrUrCrD temb_channelsadd_downsample resnet_eps resnet_act_fn resnet_groupsrTrZrNr[r\rKrarbrmrcrdrprYrPoutput_scale_factorroTprev_output_channel add_upsampleresolution_idx) num_channels num_groupseps)rmrT)&super__init__rB ValueError _check_confignnConv2dconv_in_set_time_projr'time_embedding_set_encoder_hid_proj_set_class_embedding_set_add_embeddingtime_embed_actr ModuleList down_blocks up_blocks isinstanceboollenint enumerater+appendr, mid_blocknum_upsamplerslistreversedminr- GroupNorm conv_norm_outconv_actconv_out_set_pos_net_if_use_gligen)FselfrBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrUrVrWrXrYrZr[r\r]r^r_r`rarbrcrdrerfrgrhrirjrkrlrmrnrorprqconv_in_paddingrwrxblocks_time_embed_dimoutput_channelidown_block_type input_channelis_final_block down_blockreversed_block_out_channelsreversed_num_attention_headsreversed_layers_per_blockreversed_cross_attention_dim%reversed_transformer_layers_per_block up_block_typerrup_blockconv_out_padding __class__sF r:rzUNet2DConditionModel.__init__sr &  *v 2G5G -)!51- 3)E1U1 3  *A-!3yy +A.NTc  .2-@-@ 1+!1 .A. **0  ),   "" 3+ #  !! -2W)1 "   *G$; 3++!2W)  ! ("&D "01F"GD ==,r* *D 1-51E.$8#9C@PR:SWZ[kWl:lxzNyOOefvewwxy -s3b>j*F x&4d;$%vww x?k 9r9returnc|dk(r@|xs|ddz}|dzdk7rtd|dt|dzdd||_|}||fS|dk(r*|xs|dd z}t|d|||_|d}||fSt|d ) Nfourierrrz2`time_embed_dim` should be divisible by 2, but is rF)set_W_to_weightlogrF positionalzJ does not exist. Please make sure to use one of `fourier` or `positional`.)rr time_projr()rrerLrFrGrfrwrxs r:rz#UNet2DConditionModel._set_time_projs ) +/L3Ea3H13LN!Q& #UVdUeef!ghh6!#UWfDN"0 111!L 0/L3Ea3H13LN&'9!',/B,!2,nHe"D !L 0"82DWhv"D !K /!*+BOU_!`D !23XZh!iD  G +!3Odr!sD  L 0!7hv!wD  ,)*=)>?JK -r9c|dvrMd}t|tr|}nt|ttfr|d}|dk(rdnd}t ||||_yy)N)gatedzgated-text-imageirrz text-onlyz text-image) positive_lenout_dim feature_type)rrrtupler position_net)rrmrTrrs r:rz/UNet2DConditionModel._set_pos_net_if_use_gligensb : :L-s32 /$?215 *8G*C;L ?)3FUa!D  ;r9ci}dtdtjjdttt fffd |j D]\}}||||S)z Returns: `dict` of attention processors: A dictionary containing all attention processors used in the model with indexed by its weight name. namemodule processorsct|dr|j||d<|jD]\}}|d||||S)N get_processor .processorr)hasattrrnamed_children)rrrsub_namechildfn_recursive_add_processorss r:rzIUNet2DConditionModel.attn_processors..fn_recursive_add_processorssdv/282F2F2H dV:./#)#8#8#: U%+tfAhZ,@%T U r9)strr5rModulerrr)rrrrrs @r:attn_processorsz$UNet2DConditionModel.attn_processorssm  c 588?? X\]`bt]tXu !//1 BLD& 'fj A Br9 processorc Tt|jj}t|tr,t||k7rt dt|d|d|ddt dtjjffd |jD]\}}|||y) a4 Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.rrct|drEt|ts|j|n#|j|j |d|j D]\}}|d|||y)N set_processorrr)rrdictrpopr)rrrrrfn_recursive_attn_processors r:rzLUNet2DConditionModel.set_attn_processor..fn_recursive_attn_processorsxv/!)T2((3(($z7J)KL#)#8#8#: T%+tfAhZ,@%S Tr9N) rrkeysrrrrr5rrr)rrcountrrrs @r:set_attn_processorz'UNet2DConditionModel.set_attn_processorsD((--/0 i &3y>U+BPQTU^Q_P`a005w6QRWQXXkm   Tc T588?? T!//1 ALD& 'fi @ Ar9c jtd|jjDr t}nmtd|jjDr t }n8t dt t|jj|j|y)ze Disables custom attention processors and sets the default attention implementation. c3@K|]}|jtvywN)rr.0procs r: zBUNet2DConditionModel.set_default_attn_processor..si4t~~!>>ic3@K|]}|jtvywr)rrrs r:rzBUNet2DConditionModel.set_default_attn_processor..sh$#==hrzOCannot call `set_default_attn_processor` when attention processors are of type N) allrvaluesrrrnextiterr)rrs r:set_default_attn_processorz/UNet2DConditionModel.set_default_attn_processors i4K_K_KfKfKhi i,.I h$J^J^JeJeJgh h%IabfgklpmAmAmHmHmJhKcLbMN   *r9 slice_sizec  g dtjjf fd |jD] }| t }|dk(r Dcgc]}|dz }}n |dk(r|dgz}t |t s||gzn|}t |t k7r=tdt |d|jd t d t d tt |D]&}||} |}|||kDstd |d|d dtjjdttf fd t t|}|jD] } || y cc}w)a Enable sliced attention computation. When this option is enabled, the attention module splits the input tensor in slices to compute attention in several steps. This is useful for saving some memory in exchange for a small decrease in speed. Args: slice_size (`str` or `int` or `list(int)`, *optional*, defaults to `"auto"`): When `"auto"`, input to the attention heads is halved, so attention is computed in two steps. If `"max"`, maximum amount of memory is saved by running only one slice at a time. If a number is provided, uses as many slices as `attention_head_dim // slice_size`. In this case, `attention_head_dim` must be a multiple of `slice_size`. rct|drj|j|jD] }| yNset_attention_slice)rrsliceable_head_dimchildren)rr$fn_recursive_retrieve_sliceable_dimssliceable_head_dimss r:rzVUNet2DConditionModel.set_attention_slice..fn_recursive_retrieve_sliceable_dimss@v45#**6+D+DE* <4U; .fn_recursive_set_attention_slice;sEv45**:>>+;<* D0 C Dr9) r5rrrrrrrconfigrangerrr) rrrnum_sliceable_layersdimrsizereversed_slice_sizerrrs @@@r:rz(UNet2DConditionModel.set_attention_slices! < <mmo 9F 0 8 9 ##67  /BBs#(BJB 5 -3J@J:W[@\)ZL8bl z?c"56 6$S_$5VDKK=cReNfMghQQTUhQiPjjkm  s:' VAa=D%a(CD3J 5.McURS!TUU  V DUXX__ DRVWZR[ D#8J#78mmo JF ,V5H I J=Cs E;s1s2b1b2ct|jD]9\}}t|d|t|d|t|d|t|d|;y)aFEnables the FreeU mechanism from https://huggingface.co/papers/2309.11497. The suffixes after the scaling factors represent the stage blocks where they are being applied. Please refer to the [official repository](https://github.com/ChenyangSi/FreeU) for combinations of values that are known to work well for different pipelines such as Stable Diffusion v1, v2, and Stable Diffusion XL. Args: s1 (`float`): Scaling factor for stage 1 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. s2 (`float`): Scaling factor for stage 2 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. b1 (`float`): Scaling factor for stage 1 to amplify the contributions of backbone features. b2 (`float`): Scaling factor for stage 2 to amplify the contributions of backbone features. r!r"r#r$N)rrsetattr)rr!r"r#r$rupsample_blocks r: enable_freeuz!UNet2DConditionModel.enable_freeuFsQ$"+4>>!: . A~ ND" - ND" - ND" - ND" -  .r9chd}t|jD]3\}}|D])}t||st||dt ||d+5y)zDisables the FreeU mechanism.>r#r$r!r"N)rrrgetattrr&)r freeu_keysrr'ks r: disable_freeuz"UNet2DConditionModel.disable_freeu^sW- !*4>>!: 5 A~ 5>1-D1Q1]NAt4 5 5r9crd|_|jjD]1\}}dt|jj vs(t d|j|_|jD]%}t|ts|jd'|jty)u1 Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. This API is 🧪 experimental. NAddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.T)fuse) original_attn_processorsritemsrrr1rmodulesrrfuse_projectionsrr)r_attn_processorrs r:fuse_qkv_projectionsz)UNet2DConditionModel.fuse_qkv_projectionsfs)-%!%!5!5!;!;!= v A~#n66??@@ !tuu v)-(<(<%lln 3F&),''T'2 3  5 78r9cT|j|j|jyy)uDisables the fused QKV projection if enabled. This API is 🧪 experimental. N)r1r)rs r:unfuse_qkv_projectionsz+UNet2DConditionModel.unfuse_qkv_projectionss)  ( ( 4  # #D$A$A B 5r9r0rc|}tj|s|jjdk(}|jjdk(}t |t r%|s|rtj ntj}n$|s|rtjntj}tj|g||j}n6t|jdk(r|dj|j}|j|jd}|j|}|j|j }|S)Nmpsnpu)dtypedevicerr=)r5 is_tensorr>typerfloatfloat32float64int32int64tensorrshapetoexpandrr=)rr0r timestepsis_mpsis_npur=t_embs r:get_time_embedz#UNet2DConditionModel.get_time_embeds y)]]''50F]]''50F(E**0F (.& u{{ i[fmmTI  !Q &!$**6==9I$$V\\!_5 y)v||, r9 class_labelscd}|j~| td|jjdk(r-|j |}|j |j }|j|j |j }|S)Nz9class_labels should be provided when num_class_embeds > 0rr?)rrrr]rrIr=)rr0rP class_embs r:get_class_embedz$UNet2DConditionModel.get_class_embeds    +# !\]]{{++z9#~~l;  ,V\\B ,,\:==FLL=QIr9embencoder_hidden_statesadded_cond_kwargscd}|jjdk(r|j|}|S|jjdk(rSd|vrt|jd|j d}|j d|}|j||}|S|jjdk(rd|vrt|jd|j d}d|vrt|jd |j d}|j |j} | j|jd d f} tj|| gd } | j|j} |j| }|S|jjd k(r@d|vrt|jd|j d}|j|}|S|jjdk(rTd|vsd|vrt|jd|j d}|j d} |j|| }|S)Nrr image_embedsz has the config param `addition_embed_type` set to 'text_image' which requires the keyword argument `image_embeds` to be passed in `added_cond_kwargs` text_embedsrz has the config param `addition_embed_type` set to 'text_time' which requires the keyword argument `text_embeds` to be passed in `added_cond_kwargs`time_idsz has the config param `addition_embed_type` set to 'text_time' which requires the keyword argument `time_ids` to be passed in `added_cond_kwargs`rrrrz has the config param `addition_embed_type` set to 'image' which requires the keyword argument `image_embeds` to be passed in `added_cond_kwargs`rhintz has the config param `addition_embed_type` set to 'image_hint' which requires the keyword arguments `image_embeds` and `hint` to be passed in `added_cond_kwargs`)rr^rrrgetrflattenreshaperHr5concatrIr=) rrTrUrVaug_emb image_embs text_embsrYrZ time_embeds add_embedsr\s r: get_aug_embedz"UNet2DConditionModel.get_aug_embeds ;; * *f 4(()>?GZY[[ , , <%66 ~~&'}~+..~>J)--m=RSI((J?GFE[[ , , ;$55 ~~&'{|,// >K!22 ~~&'xy),,Z8H,,X-=-=-?@K%--{/@/@/CR.HIK{K&@bIJ#syy1J((4G$#[[ , , 7%66 ~~&'xy+..~>J((4G[[ , , <%66&HY:Y ~~&'IJ+..~>J$((0D((T:Gr9c,|j,|jjdk(r|j|}|S|jZ|jjdk(rAd|vrt|jd|j d}|j||}|S|jY|jjdk(r@d|vrt|jd|j d}|j|}|S|j|jjdk(rkd|vrt|jdt |d r|j|j|}|j d}|j|}||f}|S) NrrrXz has the config param `encoder_hid_dim_type` set to 'text_image_proj' which requires the keyword argument `image_embeds` to be passed in `added_cond_kwargs`rz has the config param `encoder_hid_dim_type` set to 'image_proj' which requires the keyword argument `image_embeds` to be passed in `added_cond_kwargs` ip_image_projz has the config param `encoder_hid_dim_type` set to 'ip_image_proj' which requires the keyword argument `image_embeds` to be passed in `added_cond_kwargs`text_encoder_hid_proj)rrrXrrr]rri)rrUrVrXs r:process_encoder_hidden_statesz2UNet2DConditionModel.process_encoder_hidden_statess  ,1Q1QU`1`$($9$9:O$P !<%$; " " .4;;3S3SWh3h%66 ~~&'CD-00@L$($9$9:OQ]$^ !*%$) " " .4;;3S3SWc3c%66 ~~&'~-00@L$($9$9,$G !%$ " " .4;;3S3SWf3f%66 ~~&'ABt45$:T:T:`(,(B(BCX(Y%,00@L00>L%:L$I !$$r9 timestep_condattention_maskcross_attention_kwargsdown_block_additional_residualsmid_block_additional_residual$down_intrablock_additional_residualsencoder_attention_mask return_dictc  d|jz}d}d}|jddD]}||zdk7s d}n|2d|j|jz dz}|j d}| 2d| j|jz dz} | j d} |j j rd|zd z }|j|| }|j||}|j|| }|5|j jrtj||gd }n||z}|j|||}|j jdk(r|\}}tj||gd }|||zn|}|j|j|}|j!||}|j#|}|J|j%dd8|j'}|j)d}d|j*di|i|d<|#|j'}|j)dd }nd }t,r t/||| duxr| du}| du}|s| | t1dddd| } d}|f}|j2D]}t5|drE|j6r9i}|r"t9| dkDr| j)d|d<|d|||||| d|\}}n1|||\}}|r"t9| dkDr|| j)dz }||z }|r#d}t;|| D]\} }!| |!z} || fz}|}|j<t5|j<dr.|j<j6r|j=|||||| }n|j=||}|r>t9| dkDr0|j| djk(r|| j)dz }|r|| z}t?|j@D]\}"}#|"t9|j@dz k(}$|t9|#jB d}|dt9|#jB }|$s|r|d jdd}t5|#dr|#j6r|#|||||||| }|#||||}|jDr"|jE|}|jG|}|jI|}t,r tK||| s|fStM| S)!aK The [`UNet2DConditionModel`] forward method. Args: sample (`torch.Tensor`): The noisy input tensor with the following shape `(batch, channel, height, width)`. timestep (`torch.Tensor` or `float` or `int`): The number of timesteps to denoise an input. encoder_hidden_states (`torch.Tensor`): The encoder hidden states with shape `(batch, sequence_length, feature_dim)`. class_labels (`torch.Tensor`, *optional*, defaults to `None`): Optional class labels for conditioning. Their embeddings will be summed with the timestep embeddings. timestep_cond: (`torch.Tensor`, *optional*, defaults to `None`): Conditional embeddings for timestep. If provided, the embeddings will be summed with the samples passed through the `self.time_embedding` layer to obtain the timestep embeddings. attention_mask (`torch.Tensor`, *optional*, defaults to `None`): An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. If `1` the mask is kept, otherwise if `0` it is discarded. Mask will be converted into a bias, which adds large negative values to the attention scores corresponding to "discard" tokens. cross_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). added_cond_kwargs: (`dict`, *optional*): A kwargs dictionary containing additional embeddings that if specified are added to the embeddings that are passed along to the UNet blocks. down_block_additional_residuals: (`tuple` of `torch.Tensor`, *optional*): A tuple of tensors that if specified are added to the residuals of down unet blocks. mid_block_additional_residual: (`torch.Tensor`, *optional*): A tensor that if specified is added to the residual of the middle unet block. down_intrablock_additional_residuals (`tuple` of `torch.Tensor`, *optional*): additional residuals to be added within UNet down blocks, for example from T2I-Adapter side model(s) encoder_attention_mask (`torch.Tensor`): A cross-attention mask of shape `(batch, sequence_length)` is applied to `encoder_hidden_states`. If `True` the mask is kept, otherwise if `False` it is discarded. Mask will be converted into a bias, which adds large negative values to the attention scores corresponding to "discard" tokens. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.unets.unet_2d_condition.UNet2DConditionOutput`] instead of a plain tuple. Returns: [`~models.unets.unet_2d_condition.UNet2DConditionOutput`] or `tuple`: If `return_dict` is True, an [`~models.unets.unet_2d_condition.UNet2DConditionOutput`] is returned, otherwise a `tuple` is returned where the first element is the sample tensor. rFNrTr*g?)r0r)r0rPrr[)rTrUrVr)rUrVgligenobjsscalez2T2I should not use down_block_additional_residualsz1.3.0a?Passing intrablock residual connections with `down_block_additional_residuals` is deprecated and will be removed in diffusers 1.3.0. `down_block_additional_residuals` should only be used for ControlNet. Please make sure use `down_intrablock_additional_residuals` instead. ) standard_warnhas_cross_attentionadditional_residuals) hidden_statestembrUrlrmrq)r|r}r8)rUrlrmrq)r|r}res_hidden_states_tuplerUrm upsample_sizerlrq)r|r}r~r)r0)'rrHrIr= unsqueezerrErOrrSrnr5catrfr^rrjrr]copyrrrrrrrrzrziprrrresnetsrrrrr/)%rr0rrUrPrkrlrmrVrnrorprqrrdefault_overall_up_factorforward_upsample_sizerrrNrTrRrar\ gligen_args lora_scale is_controlnet is_adapterdown_block_res_samplesdownsample_blockr{ res_samplesnew_down_block_res_samplesdown_block_res_sampledown_block_additional_residualrr'rs% r:forwardzUNet2DConditionModel.forwards@%&t':':$:!!& <<$ C..!3(,%    %  ."3"3FLL"AAXMN+55a8N " -&'*@*C*CFLL*Q&QU]%] "%;%E%Ea%H " ;; * *Z#%F##6H#E!!%7((\(R  {{22iii 0b9Io$$+@Te%  ;; * *l :#MGTYY~15F&2cGm    *%%c*C $ B B"7K\!C!  f% " -2H2L2LXW[2\2h%;%@%@%B "044X>K068I8I8I8XK8X/Y "8 , " -%;%@%@%B "/33GSAJJ  dJ /5TAqFemqFq 9E ;CHgHs Dn$  4S 0J"( $ 0 0 2 ')>?DTDhDh')$#&J"Ka"OCgCkCklmCn()?@&6'"(*?#1+A+A '+'# '7VRU&V# #&J"Ka"OBFFqIIF "k 1 "+ 2. )+ &IL&(GJ cE%'E)>@^(^%-GK`Jb-b*  c &@ " >> %t~~'<=$..BdBd*?#1+A+A (4<=ALL$H$K$Q$QQ>BB1EE ;;F"+4>>!:  A~#dnn"5"99N0#n6L6L2M1M1OPK%;CYCY?Z>Z%[ ""&; 6r : @ @ D ~'<=.BdBd'"(,7*?+A"/#1+A ("(,7"/ / >   ''/F]]6*Fv&  j 19 $F33r9)0NrrFTr)CrossAttnDownBlock2Drr DownBlock2DUNetMidBlock2DCrossAttn)r@r?r?r?F)i@irrr*r*gsilu gh㈵>rr*NNNNFFNNNNFdefaultFrurNNNNr r NrFNN@)r) NNNNNNNNNT)-r1r2r3r4 _supports_gradient_checkpointing_no_split_modules _skip_layerwise_casting_patterns_repeated_blocksr rr rrrrrBrrrrrrrpropertyrrrrr rrr(r-r7r9r5r6rOrSrrfrjr/r __classcell__)rs@r:r<r<Gs[z(,$e(.x$/0>B$) $( )B%t9>)?34"#())+6:MNLP)-.256@D%*&+*.-115*.!&'0%*),#/,0/3+/,0 ?C'(-9=.2-/mypeCsCx$89:ypyp yp " yp  ypyp *yp! ypc yp $D%+$56!yp""#J#yp$ U3Z0%yp& 'yp(!&)yp*+yp,-yp."#/yp01yp2#3c ?33yp4',CsU5\,I&J5yp6/7uU3Z7H.I7yp8"#9yp:'sm;yp<"#uSz/2=yp>&eCsO&<=?yp@#AypB $CypD#3-EypF&c]GypH"*#IypJ#3-KypLMypN"%OypP#QypR"'SypT!UypV%SMWypX (}YypZ$C=[yp\%SM]yp^_yp`aypb08}cypdeypf"&gyph)1iypj'smkypl(+mypypv 2x*2xc 2x$D%+$56 2x "#J 2x  U3Z0 2x#3c ?32x',CsU5:=N,N&O2x/32x 2x&eCsO&<=2xh2 2 2 2  2  2 sCx28&)&sm&)#3c ?3&)"# &)P#("3-#(#(#3- #( 08} #(  #( #(J( ((+("*# (  (  (&c]("#(08}((T  SV c+=&=!>. AE2Dd3PbKbFc2c,d AD +?JeCd3i4G.H?JB.u.%.U..0592 Cll.3ELL%4L.M %,, 6ell(5<rs":: BD?mm(   (   H %   J     X4 35PRbX4r9