biS$ddlmZddlmZmZmZmZmZmZddl Z ddl m Z ddl Z ddl mZmZddlmZddlmZddlmZmZmZmZmZdd lmZmZdd lmZdd lm Z m!Z!m"Z"dd l#m$Z$dd l%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,e,jZe.Z/dZ0eGdde+Z1GddeeZ2GddeeeZ3 ddZ4 d dZ5Gdde jlZ7Gdde jlZ8Gdde jlZ9y)!) dataclass)AnyDictListOptionalTupleUnionN) ConfigMixinregister_to_config)UNet2DConditionLoadersMixin)get_activation)ADDED_KV_ATTENTION_PROCESSORSCROSS_ATTENTION_PROCESSORSAttentionProcessorAttnAddedKVProcessor AttnProcessor)TimestepEmbedding Timesteps) ModelMixin) Downsample2D ResnetBlock2D Upsample2D)Transformer2DModel) DownBlock2D UpBlock2D)UNet2DConditionOutput) BaseOutputloggingc|jd}|-|j|df}tj|||gd}|j |dd}|j |dd}tj|||gd}||fS)Nrdim)shapenew_onestorchconcatexpand) hidden_statesattention_mask sos_token eos_token batch_sizenew_attn_mask_steps k/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/pipelines/audioldm2/modeling_audioldm2.pyadd_special_tokensr1/s$$Q'J!+44j!_E'9>K]&^dfg  Q3I  Q3ILL)]I!FANM . ((cXeZdZUdZej ed<dZeejed<y)AudioLDM2ProjectionModelOutputa Args: Class for AudioLDM2 projection layer's outputs. hidden_states (`torch.Tensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states obtained by linearly projecting the hidden-states for each of the text encoders and subsequently concatenating them together. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices, formed by concatenating the attention masks for the two text encoders together. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. r*Nr+) __name__ __module__ __qualname____doc__r'Tensor__annotations__r+r LongTensorr2r0r4r4>s( <<15NHU--.5r2r4c eZdZdZe dfd Z d deejdeejdeejdeejfdZ xZ S) AudioLDM2ProjectionModela A simple linear projection model to map two text embeddings to a shared latent space. It also inserts learned embedding vectors at the start and end of each text embedding sequence respectively. Each variable appended with `_1` refers to that corresponding to the second text encoder. Otherwise, it is from the first. Args: text_encoder_dim (`int`): Dimensionality of the text embeddings from the first text encoder (CLAP). text_encoder_1_dim (`int`): Dimensionality of the text embeddings from the second text encoder (T5 or VITS). langauge_model_dim (`int`): Dimensionality of the text embeddings from the language model (GPT2). ct|tj|||_tj|||_tj tj||_ tj tj||_ tj tj||_ tj tj||_ ||_ |j;tjj tjd||f|_yy)Nr!)super__init__nnLinear projection projection_1 Parameterr'ones sos_embed eos_embed sos_embed_1 eos_embed_1use_learned_position_embeddingzeroslearnable_positional_embedding)selftext_encoder_dimtext_encoder_1_dimlangauge_model_dimrLmax_seq_length __class__s r0rAz!AudioLDM2ProjectionModel.__init__as ))$46HIII&8:LMejj1C&DEejj1C&DE<< 3E(FG<< 3E(FG.L+  . . :27((2D2D Q 2NCD3D / ;r2r*hidden_states_1r+attention_mask_1cF|j|}t|||j|j\}}|j1|j ddd|j zj ddd}|j|}t|||j|j\}}tj||gd}|||j|dd}n|||j|dd}||tj||gd}nd}t||S)N)r,r-rr!r#r")r*r+)rDr1rHrIrLpermuterNrErJrKr'catr&r4)rOr*rUr+rVs r0forwardz AudioLDM2ProjectionModel.forward~sB 6 (: >T^^t~~) % ~  . . :.66q!Q?$BeBeennoprsuvwO++O<,> -9I9IUYUeUe- ))  =/"BJ   !&6&B-66 bq8IKN  ',<,D-668KM   %*:*F"YY8H'IrRN!N-')  r2)NN)NNNN) r5r6r7r8r rArr'r9r;r[ __classcell__rTs@r0r>r>Rs  (, <1526597; &  -& "%,,/& !!1!12 & #5#3#34 & r2r>cFHeZdZdZdZe d8deededededede e d ee d e e d e ee efd e ed e ee efdede de deede de ee efde ee efde ee efdee ee efdedee deedede de deedee dee d eed!ed"ed#eed$efDfd% Z ed&ee effd'Zd(e eee efffd)Zd*Zd+Z d9d,ej,d-e ej,e efd.ej,d/eej,d0eej,d1eej,d2eee efd3eej,d4ed5eej,d6eej,d&e ee ffd7ZxZS):AudioLDM2UNet2DConditionModela A conditional 2D UNet model that takes a noisy sample, conditional state, and a timestep and returns a sample shaped output. Compared to the vanilla [`UNet2DConditionModel`], this variant optionally includes an additional self-attention layer in each Transformer block, as well as multiple cross-attention layers. It also allows for up to two cross-attention embeddings, `encoder_hidden_states` and `encoder_hidden_states_1`. This model inherits from [`ModelMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). Parameters: sample_size (`int` or `Tuple[int, int]`, *optional*, defaults to `None`): Height and width of input/output sample. in_channels (`int`, *optional*, defaults to 4): Number of channels in the input sample. out_channels (`int`, *optional*, defaults to 4): Number of channels in the output. flip_sin_to_cos (`bool`, *optional*, defaults to `False`): Whether to flip the sin to cos in the time embedding. freq_shift (`int`, *optional*, defaults to 0): The frequency shift to apply to the time embedding. down_block_types (`Tuple[str]`, *optional*, defaults to `("CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "DownBlock2D")`): The tuple of downsample blocks to use. mid_block_type (`str`, *optional*, defaults to `"UNetMidBlock2DCrossAttn"`): Block type for middle of UNet, it can only be `UNetMidBlock2DCrossAttn` for AudioLDM2. up_block_types (`Tuple[str]`, *optional*, defaults to `("UpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D")`): The tuple of upsample blocks to use. only_cross_attention (`bool` or `Tuple[bool]`, *optional*, default to `False`): Whether to include self-attention in the basic transformer blocks, see [`~models.attention.BasicTransformerBlock`]. block_out_channels (`Tuple[int]`, *optional*, defaults to `(320, 640, 1280, 1280)`): The tuple of output channels for each block. layers_per_block (`int`, *optional*, defaults to 2): The number of layers per block. downsample_padding (`int`, *optional*, defaults to 1): The padding to use for the downsampling convolution. mid_block_scale_factor (`float`, *optional*, defaults to 1.0): The scale factor to use for the mid block. act_fn (`str`, *optional*, defaults to `"silu"`): The activation function to use. norm_num_groups (`int`, *optional*, defaults to 32): The number of groups to use for the normalization. If `None`, normalization and activation layers is skipped in post-processing. norm_eps (`float`, *optional*, defaults to 1e-5): The epsilon to use for the normalization. cross_attention_dim (`int` or `Tuple[int]`, *optional*, defaults to 1280): The dimension of the cross attention features. transformer_layers_per_block (`int` or `Tuple[int]`, *optional*, defaults to 1): The number of transformer blocks of type [`~models.attention.BasicTransformerBlock`]. Only relevant for [`~models.unet_2d_blocks.CrossAttnDownBlock2D`], [`~models.unet_2d_blocks.CrossAttnUpBlock2D`], [`~models.unet_2d_blocks.UNetMidBlock2DCrossAttn`]. attention_head_dim (`int`, *optional*, defaults to 8): The dimension of the attention heads. num_attention_heads (`int`, *optional*): The number of attention heads. If not defined, defaults to `attention_head_dim` resnet_time_scale_shift (`str`, *optional*, defaults to `"default"`): Time scale shift config for ResNet blocks (see [`~models.resnet.ResnetBlock2D`]). Choose from `default` or `scale_shift`. class_embed_type (`str`, *optional*, defaults to `None`): The type of class embedding to use which is ultimately summed with the time embeddings. Choose from `None`, `"timestep"`, `"identity"`, `"projection"`, or `"simple_projection"`. num_class_embeds (`int`, *optional*, defaults to `None`): Input dimension of the learnable embedding matrix to be projected to `time_embed_dim`, when performing class conditioning with `class_embed_type` equal to `None`. time_embedding_type (`str`, *optional*, defaults to `positional`): The type of position embedding to use for timesteps. Choose from `positional` or `fourier`. time_embedding_dim (`int`, *optional*, defaults to `None`): An optional override for the dimension of the projected time embedding. time_embedding_act_fn (`str`, *optional*, defaults to `None`): Optional activation function to use only once on the time embeddings before they are passed to the rest of the UNet. Choose from `silu`, `mish`, `gelu`, and `swish`. timestep_post_act (`str`, *optional*, defaults to `None`): The second activation function to use in timestep embedding. Choose from `silu`, `mish` and `gelu`. time_cond_proj_dim (`int`, *optional*, defaults to `None`): The dimension of `cond_proj` layer in the timestep embedding. conv_in_kernel (`int`, *optional*, default to `3`): The kernel size of `conv_in` layer. conv_out_kernel (`int`, *optional*, default to `3`): The kernel size of `conv_out` layer. projection_class_embeddings_input_dim (`int`, *optional*): The dimension of the `class_labels` input when `class_embed_type="projection"`. Required when `class_embed_type="projection"`. class_embeddings_concat (`bool`, *optional*, defaults to `False`): Whether to concatenate the time embeddings with the class embeddings. T sample_size in_channels out_channelsflip_sin_to_cos freq_shiftdown_block_typesmid_block_typeup_block_typesonly_cross_attentionblock_out_channelslayers_per_blockdownsample_paddingmid_block_scale_factoract_fnnorm_num_groupsnorm_epscross_attention_dimtransformer_layers_per_blockattention_head_dimnum_attention_headsuse_linear_projectionclass_embed_typenum_class_embedsupcast_attentionresnet_time_scale_shifttime_embedding_typetime_embedding_dimtime_embedding_act_fntimestep_post_acttime_cond_proj_dimconv_in_kernelconv_out_kernel%projection_class_embeddings_input_dimclass_embeddings_concatc# t7|||_| td|xs|}t |t |k7rtd|d|dt | t |k7rtd| d|dt | t s)t | t |k7rtd| d|dt |ts)t |t |k7rtd|d|dt |ts)t |t |k7rtd |d|dt |tr)t |t |k7rtd |d|dt | ts)t | t |k7rtd | d|d|d z d z}#tj|| d||#|_ |dk(r'|xs| ddz}$t| d|||_ | d}%nt|dt|%|$||||_||tj ||$|_n|dk(rt|%|$||_nz|dk(rtj$|$|$|_nY|dk(r|! tdt|!|$|_n5|dk(r)|! tdtj&|!|$|_nd|_|d|_nt+||_tj,g|_tj,g|_t | t r| gt |z} t |tr|ft |z}t |tr|ft |z}t | tr| gt |z} t |tr|gt |z}|"r|$d z}&n|$}&| d}'t3|D]\}(})|'}*| |(}'|(t | d z k(}+t5|)fid| |(d||(d|*d|'d|&d |+ d!|d"|d#|d$||(d%||(d&| d'|d(| |(d)|d*|},|j.j7|,|d+k(r)t9|d,| d,|&||| ||d,|d,|||- |_ntd.|d/d|_tt?| }-tt?|}.tt?| }/tt?|}0tt?|}1tt?| } |-d}'t3|D]\}(}2|(t | d z k(}+|'}3|-|(}'|-tA|(d zt | d z }*|+sd0}4|xj<d z c_nd1}4tC|2fid|/|(d zd|1|(d|*d|'d2|3d|&d3|4d!|d"|d#|d$|0|(d%|.|(d'|d(| |(d)|d*|}5|j0j7|5|'}3|1tjD| d||4|_#t+||_$nd|_#d|_$| d z d z}6tj| d|| |6|_%y)5Na#At the moment it is not possible to define the number of attention heads via `num_attention_heads` because of a naming issue as described in https://github.com/huggingface/diffusers/issues/2011#issuecomment-1547958131. Passing `num_attention_heads` will only be supported in diffusers v0.19.z\Must provide the same number of `down_block_types` as `up_block_types`. `down_block_types`: z. `up_block_types`: .zbMust provide the same number of `block_out_channels` as `down_block_types`. `block_out_channels`: z. `down_block_types`: zfMust provide the same number of `only_cross_attention` as `down_block_types`. `only_cross_attention`: zdMust provide the same number of `num_attention_heads` as `down_block_types`. `num_attention_heads`: zbMust provide the same number of `attention_head_dim` as `down_block_types`. `attention_head_dim`: zdMust provide the same number of `cross_attention_dim` as `down_block_types`. `cross_attention_dim`: z^Must provide the same number of `layers_per_block` as `down_block_types`. `layers_per_block`: r!rXr) kernel_sizepadding positionalz6 does not exist. Please make sure to use `positional`.)rm post_act_fn cond_proj_dimtimestep)rmidentityrDzX`class_embed_type`: 'projection' requires `projection_class_embeddings_input_dim` be setsimple_projectionz_`class_embed_type`: 'simple_projection' requires `projection_class_embeddings_input_dim` be set num_layersrqrarb temb_channelsadd_downsample resnet_eps resnet_act_fn resnet_groupsrprsrkrtrhrwrxUNetMidBlock2DCrossAttnr") rqrarrroutput_scale_factorrxrprsrrtrwzunknown mid_block_type : z4. Should be `UNetMidBlock2DCrossAttn` for AudioLDM2.TFprev_output_channel add_upsample) num_channels num_groupseps)&r@rAr` ValueErrorlen isinstanceboolintlistrBConv2dconv_inr time_projrtime_embedding Embeddingclass_embeddingIdentityrCtime_embed_actr ModuleList down_blocks up_blocks enumerateget_down_blockappendr mid_blocknum_upsamplersreversedmin get_up_block GroupNorm conv_norm_outconv_actconv_out)8rOr`rarbrcrdrerfrgrhrirjrkrlrmrnrorprqrrrsrtrurvrwrxryrzr{r|r}r~rrrconv_in_paddingtime_embed_dimtimestep_input_dimblocks_time_embed_dimoutput_channelidown_block_type input_channelis_final_block down_blockreversed_block_out_channelsreversed_num_attention_headsreversed_layers_per_blockreversed_cross_attention_dim%reversed_transformer_layers_per_block up_block_typerrup_blockconv_out_paddingrTs8 r0rAz&AudioLDM2UNet2DConditionModel.__init__s V &  *v 2G5G  C$7 7nooAAUVdUeefg  ! "c*:&; ;tvHuII_`p_qqrs .5#>R:SWZ[kWl:lxzNyOOefvewwxy -s3Z5[0\-#H-A$BC4Q7 ). 9" 1 A}#&8"9A"==N"0 8;N7AE3GYCZ]^C^8_`M"# ##q(#$ #4Q7!;.SST-U*  ,  %8 4*$%.%A$C%A$C'<&:!%< "2!")@#H& NN ! !( +"0 E" 1J  &!#/2T\"D +62DM"&D  DM+a/A5 q !<_Vf  r2returnci}dtdtjjdttt fffd |j D]\}}||||S)z Returns: `dict` of attention processors: A dictionary containing all attention processors used in the model with indexed by its weight name. namemodule processorsct|dr|j||d<|jD]\}}|d||||S)N get_processor .processorr)hasattrrnamed_children)rrrsub_namechildfn_recursive_add_processorss r0rzRAudioLDM2UNet2DConditionModel.attn_processors..fn_recursive_add_processors!sdv/282F2F2H dV:./#)#8#8#: U%+tfAhZ,@%T U r2)strr'rBModulerrr)rOrrrrs @r0attn_processorsz-AudioLDM2UNet2DConditionModel.attn_processorssm  c 588?? X\]`bt]tXu !//1 BLD& 'fj A Br2 processorc Tt|jj}t|tr,t||k7rt dt|d|d|ddt dtjjffd |jD]\}}|||y) a4 Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.rrct|drEt|ts|j|n#|j|j |d|j D]\}}|d|||y)N set_processorrr)rrdictrpopr)rrrrrfn_recursive_attn_processors r0rzUAudioLDM2UNet2DConditionModel.set_attn_processor..fn_recursive_attn_processorEsxv/!)T2((3(($z7J)KL#)#8#8#: T%+tfAhZ,@%S Tr2N) rrkeysrrrrr'rBrr)rOrcountrrrs @r0set_attn_processorz0AudioLDM2UNet2DConditionModel.set_attn_processor0sD((--/0 i &3y>U+BPQTU^Q_P`a005w6QRWQXXkm   Tc T588?? T!//1 ALD& 'fi @ Ar2c jtd|jjDr t}nmtd|jjDr t }n8t dt t|jj|j|y)ze Disables custom attention processors and sets the default attention implementation. c3@K|]}|jtvywN)rTr.0procs r0 zKAudioLDM2UNet2DConditionModel.set_default_attn_processor..Wsi4t~~!>>ic3@K|]}|jtvywr)rTrrs r0rzKAudioLDM2UNet2DConditionModel.set_default_attn_processor..Ysh$#==hrzOCannot call `set_default_attn_processor` when attention processors are of type N) allrvaluesrrrnextiterr)rOrs r0set_default_attn_processorz8AudioLDM2UNet2DConditionModel.set_default_attn_processorSs i4K_K_KfKfKhi i,.I h$J^J^JeJeJgh h%IabfgklpmAmAmHmHmJhKcLbMN   *r2c  g dtjjf fd |jD] }| t }|dk(r Dcgc]}|dz }}n |dk(r|dgz}t |t s||gzn|}t |t k7r=tdt |d|jd t d t d tt |D]&}||} |}|||kDstd |d|d dtjjdttf fd t t|}|jD] } || y cc}w)a Enable sliced attention computation. When this option is enabled, the attention module splits the input tensor in slices to compute attention in several steps. This is useful for saving some memory in exchange for a small decrease in speed. Args: slice_size (`str` or `int` or `list(int)`, *optional*, defaults to `"auto"`): When `"auto"`, input to the attention heads is halved, so attention is computed in two steps. If `"max"`, maximum amount of memory is saved by running only one slice at a time. If a number is provided, uses as many slices as `attention_head_dim // slice_size`. In this case, `attention_head_dim` must be a multiple of `slice_size`. rct|drj|j|jD] }| yNset_attention_slice)rrsliceable_head_dimchildren)rr$fn_recursive_retrieve_sliceable_dimssliceable_head_dimss r0rz_AudioLDM2UNet2DConditionModel.set_attention_slice..fn_recursive_retrieve_sliceable_dimsss@v45#**6+D+DE* <4U; .fn_recursive_set_attention_slicesEv45**:>>+;<* D0 C Dr2) r'rBrrrrrrconfigrangerrr) rOrrnum_sliceable_layersr$rsizereversed_slice_sizerrrs @@@r0rz1AudioLDM2UNet2DConditionModel.set_attention_slicecs! < <mmo 9F 0 8 9 ##67  /BBs#(BJB 5 -3J@J:W[@\)ZL8bl z?c"56 6$S_$5VDKK=cReNfMghQQTUhQiPjjkm  s:' VAa=D%a(CD3J 5.McURS!TUU  V DUXX__ DRVWZR[ D#8J#78mmo JF ,V5H I J=Cs E;samplerencoder_hidden_states class_labels timestep_condr+cross_attention_kwargsencoder_attention_mask return_dictencoder_hidden_states_1encoder_attention_mask_1c p d|jzd} d} tfd|jddDrtj dd} |2d|j |j z d z}|jd}|2d|j |j z d z}|jd}| 2d| j |j z d z} | jd} |}tj|s|jjd k(}|jjd k(}t|tr%|s|rtjntj}n$|s|rtj ntj"}tj$|g||j }n6t'|jd k(r|dj |j}|j)|jd }|j+|}|j |j }|j-||}d}|j.| t1d|j2j4dk(r-|j+|}|j |j }|j/|j |j }|j2j6rtj8||gd}n||z}|||zn|}|j:|j;|}|j=|}|f}|j>D]@}tA|dr |jBr|||||||| | \}}n |||\}}||z }B|jD|jE||||||| | }tG|jHD]\}}|t'|jHdz k(}|t'|jJ d}|dt'|jJ }|s| r|djdd} tA|dr|jBr||||||| ||| |  }||||| }|jLr"|jM|}|jO|}|jQ|}| s|fStS|S)a The [`AudioLDM2UNet2DConditionModel`] forward method. Args: sample (`torch.Tensor`): The noisy input tensor with the following shape `(batch, channel, height, width)`. timestep (`torch.Tensor` or `float` or `int`): The number of timesteps to denoise an input. encoder_hidden_states (`torch.Tensor`): The encoder hidden states with shape `(batch, sequence_length, feature_dim)`. encoder_attention_mask (`torch.Tensor`): A cross-attention mask of shape `(batch, sequence_length)` is applied to `encoder_hidden_states`. If `True` the mask is kept, otherwise if `False` it is discarded. Mask will be converted into a bias, which adds large negative values to the attention scores corresponding to "discard" tokens. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.unets.unet_2d_condition.UNet2DConditionOutput`] instead of a plain tuple. cross_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the [`AttnProcessor`]. encoder_hidden_states_1 (`torch.Tensor`, *optional*): A second set of encoder hidden states with shape `(batch, sequence_length_2, feature_dim_2)`. Can be used to condition the model on a different set of embeddings to `encoder_hidden_states`. encoder_attention_mask_1 (`torch.Tensor`, *optional*): A cross-attention mask of shape `(batch, sequence_length_2)` is applied to `encoder_hidden_states_1`. If `True` the mask is kept, otherwise if `False` it is discarded. Mask will be converted into a bias, which adds large negative values to the attention scores corresponding to "discard" tokens. Returns: [`~models.unets.unet_2d_condition.UNet2DConditionOutput`] or `tuple`: If `return_dict` is True, an [`~models.unets.unet_2d_condition.UNet2DConditionOutput`] is returned, otherwise a `tuple` is returned where the first element is the sample tensor. rXFNc3.K|] }|zdk7yw)rNr<)rsdefault_overall_up_factors r0rz8AudioLDM2UNet2DConditionModel.forward..sMaq,,1Msz9Forward upsample size to force interpolation output size.Tr!gmpsnpu)dtypedevicer)rz9class_labels should be provided when num_class_embeds > 0rr"r#has_cross_attention)r*tembrr+rrr r )r*r)rr+rrr r ) r*rres_hidden_states_tuplerr upsample_sizer+rr r )r*rrr)r)*ranyr%loggerinfotor unsqueezer' is_tensorrtyperfloatfloat32float64int32int64tensorrr)rrrrrrurrZrrrrrrrrresnetsrrrr)rOrrrrrr+rrr r r forward_upsample_sizer timestepsis_mpsis_npurt_embembaug_emb class_embdown_block_res_samplesdownsample_block res_samplesrupsample_blockrrs @r0r[z%AudioLDM2UNet2DConditionModel.forwardsb%&t':':$:!!& M6<<;LM M KKS T$( !  %  ."3"3FLL"AAXMN+55a8N " -&'*@*C*CFLL*Q&QU]%] "%;%E%Ea%H " # /(),D,G,G ,U(UYa'a $'?'I'I!'L $ y)]]''50F]]''50F(E**0F (.& u{{ i[fmmTI  !Q &!$**6==9I$$V\\!_5 y) v||,!!%7    +# !\]]{{++z9#~~l;  ,V\\B ,,\:==FLL=QI{{22iii 0b9Io&2cGm    *%%c*Cf%#) $ 0 0 2 ')>?DTDhDh&6"(*?#1+A+A,C-E '# '7VRU&V# "k 1 " 2$ >> %^^&;-'='=(?)A$ F"+4>>!:  A~#dnn"5"99N0#n6L6L2M1M1OPK%;CYCY?Z>Z%[ ""&; 6r : @ @ D ~'<=.BdBd'"(,7*?+A"/#1+A,C-E ("(sKgt3 <   ''/F]]6*Fv&9 $F33r2)"NrrTr)CrossAttnDownBlock2Dr3r3rr)rCrossAttnUpBlock2Dr4r4F)i@ir5rXr!r!silu gh㈵>r5r!NFNNFdefaultrNNNNr r NF)NNNNNTNN)r5r6r7r8 _supports_gradient_checkpointingr rrrrrr r rApropertyrrrrrrr'r9rrr[r\r]s@r0r_r_sEN(,$&* $( )B%t9>)?34"#())+6:?@56@D&+*.*.!&'0#/,0/3+/,0 ?C(-Qb c]b b  b  b  b  *b ! b c b $D%+$56b "#J!b " U3Z0#b $ %b &!&'b ()b *"#+b ,-b .#3c ?3/b 0',CsO&<1b 2"#uSz/23b 4&eCsO&<=5b 6 $7b 8#3-9b :#3-;b <=b >"%?b @!Ab B%SMCb D (}Eb F$C=Gb H%SMIb JKb LMb N08}Ob P"&Qb b H c+=&=!>0 AE2Dd3PbKbFc2c,d AF + ?JL040415;?9= :>;?Q4 Q4 eS01Q4 %|| Q4 u||, Q4   - Q4!.Q4!)c3h 8Q4!) 6Q4Q4"*%,,!7Q4#+5<<"8Q4 $e+ ,Q4r2r_c |jdr|ddn|}|dk(rt|||||||| | | S|dk(rF| tdtdid|d|d |d |d |d |d |d|d| d| d| d| d| d|d|d|St|d)NUNetResr) rrarbrrrrrrkrxr3z>cross_attention_dim must be specified for CrossAttnDownBlock2Drrqrarbrrrrrrkrprsrtrhrwrx does not exist.r<) startswithrrr3)rrrarbrrrrrqrsrrprkrtrhrwrxs r0rrxs/&.=-G-G -Roab)XgO-'!#%')!''1$;   2 2  &]^ ^# ! )E $ &  (  *  " ( (  2 !4 !4 #8 "6 . %r) rrarbrrrrrrrxr4zVYZ>ZMM`LaalmprEnFmGH $7 z" A)*a+\K NN +!-"/"(#(?"/(;,  3234 !!&+$(;;$0#?,?,B(5.C-A)96I!6L6TdZ_   ! >-- 3}}W-  "  $t,Xjqu!D ',#!%D &+#r2r*rrr+rrr r c :d} t|j} t|j| z} ||n|}||n|}t| D]#} t j r|j r|j|j| ||}t|jD]O\} }| | dkr|}|}n| | dkDr|}|}nd}d}|j|j| | z| z||dd|||d}Qnr|j| ||}t|jD]D\} }| | dkr|}|}n| | dkDr|}|}nd}d}|j| | z| z||||dd}F| |fz} &|j|jD] }||} | |fz} || fS)Nr<r!rFr+rrr ) rr&rRrr'is_grad_enabledrT_gradient_checkpointing_funcrrprS)rOr*rrr+rrr r  output_statesrnum_attention_per_layerridxrpforward_encoder_hidden_statesforward_encoder_attention_mask downsamplers r0r[zCrossAttnDownBlock2D.forward@sB & "%doo"6*"D(?'J #Pe )@(K $Qg !z"+ =A$$&4+F+F $ A A$,,q/S`bf g 09$:R:R0S,C,*63!88M59O6,8S1W8O59Q68<59=6$($E$E,C(Cc(IJ%5.&6 % %M*!0 Q t D 09$:R:R0S,C,*63!88M59O6,8S1W8O59Q68<59=6$VDOOA8O4ORU4U$V%'5.K/M$) %  %M$*],<r9swishr7Tr!r5?r!TFFFNNNNNNNr5r6r7rr rrrAr'r9rrrr[r\r]s@r0r3r3sm ,- '0$ $ #")S,S,S, S,  S,  S,'*S,S,"%S,S,S,S,p(,8<15;?9=:>;?I,||I,u||$I, ( 5 I, !. I, !)c3h 8 I,!) 6I,"*%,,!7I,#+5<<"8I,r2r3cteZdZ ddedededededededed ed effd Z dd ejd e ejde ejde ejde e ee fde ejde ejde ejdejfdZ xZS)rrarrCrrqrrxrrrDct|d|_| |_| | nt |dzd} t | t r| f} t | ttfr(t| dkDrtd| dt| | |_ t||||| |||| |  g}g}t|D]o}tt| D]2}|jt| || z||| || ||| |dnd 4|jt||||| |||| |  qt!j"||_t!j"||_d|_y) NTrr7rFrGrHF)rarrprnrtrwrN)r@rArrsrrrrrQrrrprrrrrBrrRr&rT)rOrarrCrrqrrxrrrDrsrrprtrwr&rRrrUrTs r0rAz UNetMidBlock2DCrossAttn.__init__s$ #' #6 )6)B K[\L\^`Ha )3 /#6"8  )D%= 9cBU>VYZ>ZMM`LaalmprEnFmGH $7  '(+$$;+$7(    z" A3234 !!&+#'::$/#?,?,B(5.C)96I!6L6TdZ_    NN +!,"/"(#(?"/(;,   <-- 3}}W- &+#r2r*rrr+rrr r rc 6|jd||}t|jt|jdz z} ||n|}||n|}tt|jddD]#} t j r|j rt|jD]O\} } | | dkr|} |}n| | dkDr|} |}nd} d}|j|j| | z| z|| dd|||d}Q|j|j| dz||}t|jD]D\} } | | dkr|} |}n| | dkDr|} |}nd} d}|j| | z| z||| |dd}F|j| dz||}&|S)Nrr!FrW) r&rrRrr'rXrTrrprY)rOr*rrr+rrr r r[rr\rpr]r^s r0r[zUNetMidBlock2DCrossAttn.forwards$( Q t< "%doo"63t||;Lq;P"Q(?'J #Pe )@(K $Qg !s4<<+,-* IA$$&4+F+F09$:R:R0S,C,*63!88M59O6,8S1W8O59Q68<59=6$($E$E,C(Cc(IJ%5.&6 % %M(!% A A$,,qSTuBUWdfj k 09$:R:R0S,C,*63!88M59O6,8S1W8O59Q68<59=6$VDOOA8O4ORU4U$V%'5.K/M$) %  %M$!4 QU 3M4 H U* IXr2) r`r!r!rar9rbr7Tr!rcr5FFrdrer]s@r0rrsh ,- '0$ $ #!S,S,S, S,  S, '* S,S,"%S,S,S,S,p(,8<15;?9=:>;?A||Au||$A ( 5 A !. A !)c3h 8 A!) 6A"*%,,!7A#+5<<"8A Ar2rceZdZ ddedededededededed ed ed ed effd Z ddejde ejdfde ejde ejde e ee fde ede ejde ejde ejde ejfdZxZS)r4rarbrrrCrrqrrxrrrDct|g}g}d|_| |_t |t r|f}t |t tfr(t|dkDrtd|dt|||_ t|D]}||dz k(r|n|}|dk(r|n|}|jt||z|||| || | ||  tt|D]3}|jt| || z||||| |||||dnd 5tj ||_tj ||_|r/tj t'|d| g|_d|_yd|_d|_y) NTrrFrGr!rrHFrM)rPrb)r@rArrsrrrrQrrrprrrrrBrrRr&r upsamplersrT)rOrarbrrrCrrqrrxrrrDrsrprrrtrhrwr&rRrres_skip_channelsresnet_in_channelsrUrTs r0rAzCrossAttnUpBlock2D.__init__'s,  #' #6 )3 /#6"8  )D%= 9cBU>VYZ>ZMM`LaalmprEnFmGH $7 z" A01Z!^0C , 89Q!4L  NN 25F F!-"/"(#(?"/(;,  3234 !!&+$(;;$0#?,?,B(5.C-A)96I!6L6TdZ_   % B-- 3}}W-  mmZ tbn-o,pqDO',##DO&+#r2r*r.rrrrr+rr r c bt|j} t|j| z} | | n|} | | n|} t| D]@} |d}|dd}t j ||gd}t j r|jr|j|j| ||}t|jD]O\}}| |dkr|}|}n| |dkDr| }| }nd}d}|j|j| | z|z||dd|||d}Q|j| ||}t|jD]D\}}| |dkr|}|}n| |dkDr| }| }nd}d}|j| | z|z||||dd}FC|j|jD] }|||} |S)Nr"r!r#rFrW) rr&rRrr'rZrXrTrYrrprk)rOr*rrrrrr+rr r rr[rres_hidden_statesr\rpr]r^ upsamplers r0r[zCrossAttnUpBlock2D.forwardxsI& "%doo"6*"D(?'J #Pe )@(K $Qg !z". A 7 ; &=cr&B #!II}6G&HaPM$$&4+F+F $ A A$,,q/S`bf g 09$:R:R0S,C,*63!88M59O6,8S1W8O59Q68<59=6$($E$E,C(Cc(IJ%5.&6 % %M*!0 Q t D 09$:R:R0S,C,*63!88M59O6,8S1W8O59Q68<59=6$VDOOA8O4ORU4U$V%'5.K/M$) %  %M=. ` ?? &!__ H )- G  Hr2)r`r!r!rar9rbr7Tr!r5rcTFFF)NNNNNNNN)r5r6r7rr rrrAr'r9rrrrr[r\r]s@r0r4r4&s,- '0$ $ #")O,O,O,! O,  O,  O,O,'*O,O,"%O,O,O,O,j(,8<;?'+159=:>;?K||K"'u||S'8!9Ku||$ K ( 5 K !)c3h 8 K }K!.K!) 6K"*%,,!7K#+5<<"8Kr2r4) r!NNNNFFFr9)r!NNNFFFr9): dataclassesrtypingrrrrrr r'torch.nnrBtorch.utils.checkpointconfiguration_utilsr r loadersr models.activationsrmodels.attention_processorrrrrrmodels.embeddingsrrmodels.modeling_utilsr models.resnetrrr"models.transformers.transformer_2drmodels.unets.unet_2d_blocksrrmodels.unets.unet_2d_conditionrutilsrr get_loggerr5rr1r4r>r_rrrr3rr4r<r2r0rs":: B200DDDAC(   H % ) 6Z6 6&R z;R jN 4J =XN 4t"#%#6;F"#%#69r_,299_,DWbiiWt]]r2