biKBddlmZddlmZmZmZmZmZmZddl Z ddl m Z ddl Z ddl mZmZddlmZddlmZmZdd lmZdd lmZmZmZmZmZmZmZdd lm Z m!Z!dd l"m#Z#dd l$m%Z%ddl&m'Z'm(Z(m)Z)ejTe+Z,eGddeZ-Gdde#eeZ.y)) dataclass)AnyDictListOptionalTupleUnionN) ConfigMixinregister_to_config)UNet2DConditionLoadersMixin) BaseOutputlogging)get_activation)ADDED_KV_ATTENTION_PROCESSORSCROSS_ATTENTION_PROCESSORS AttentionAttentionProcessorAttnAddedKVProcessor AttnProcessorFusedAttnProcessor2_0)TimestepEmbedding Timesteps) ModelMixin)TransformerTemporalModel)UNetMidBlock3DCrossAttnget_down_block get_up_blockc0eZdZUdZej ed<y)UNet3DConditionOutputa The output of [`UNet3DConditionModel`]. Args: sample (`torch.Tensor` of shape `(batch_size, num_channels, num_frames, height, width)`): The hidden states output conditioned on `encoder_hidden_states` input. Output of last layer of model. sampleN)__name__ __module__ __qualname____doc__torchTensor__annotations__c/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/models/unets/unet_3d_condition.pyr"r"1s LLr,r"c$eZdZdZdZddgZe d2deededed e e d fd e e d fd e ed fd edede de deede dede ee efdee ee efdeef fd Z edee effdZde e eeefddfdZde eee efffdZd3deededdfd Zd!Zd"Zd#Zd$Zd%Zd&Z d4d'ej:d(e ej:e efd)ej:d*eej:d+eej:d,eej:d-eee efd.ee ej:d/eej:d0ede e e ej:ffd1Z!xZ"S)5UNet3DConditionModela8 A conditional 3D UNet model that takes a noisy sample, conditional state, and a timestep and returns a sample shaped output. This model inherits from [`ModelMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). Parameters: sample_size (`int` or `Tuple[int, int]`, *optional*, defaults to `None`): Height and width of input/output sample. in_channels (`int`, *optional*, defaults to 4): The number of channels in the input sample. out_channels (`int`, *optional*, defaults to 4): The number of channels in the output. down_block_types (`Tuple[str]`, *optional*, defaults to `("CrossAttnDownBlock3D", "CrossAttnDownBlock3D", "CrossAttnDownBlock3D", "DownBlock3D")`): The tuple of downsample blocks to use. up_block_types (`Tuple[str]`, *optional*, defaults to `("UpBlock3D", "CrossAttnUpBlock3D", "CrossAttnUpBlock3D", "CrossAttnUpBlock3D")`): The tuple of upsample blocks to use. block_out_channels (`Tuple[int]`, *optional*, defaults to `(320, 640, 1280, 1280)`): The tuple of output channels for each block. layers_per_block (`int`, *optional*, defaults to 2): The number of layers per block. downsample_padding (`int`, *optional*, defaults to 1): The padding to use for the downsampling convolution. mid_block_scale_factor (`float`, *optional*, defaults to 1.0): The scale factor to use for the mid block. act_fn (`str`, *optional*, defaults to `"silu"`): The activation function to use. norm_num_groups (`int`, *optional*, defaults to 32): The number of groups to use for the normalization. If `None`, normalization and activation layers is skipped in post-processing. norm_eps (`float`, *optional*, defaults to 1e-5): The epsilon to use for the normalization. cross_attention_dim (`int`, *optional*, defaults to 1024): The dimension of the cross attention features. attention_head_dim (`int`, *optional*, defaults to 64): The dimension of the attention heads. num_attention_heads (`int`, *optional*): The number of attention heads. time_cond_proj_dim (`int`, *optional*, defaults to `None`): The dimension of `cond_proj` layer in the timestep embedding. Fnormtime_embeddingN sample_size in_channels out_channelsdown_block_types.up_block_typesblock_out_channelslayers_per_blockdownsample_paddingmid_block_scale_factoract_fnnorm_num_groupsnorm_epscross_attention_dimattention_head_dimnum_attention_headstime_cond_proj_dimct#|||_| td|xs|}t |t |k7rt d|d|dt |t |k7rt d|d|dt |ts)t |t |k7rt d|d|dd}d}|d z d z}tj||d || |_ |d d z}t|d dd |_ |d }t||| ||_td||d d | |_tj"g|_tj"g|_t |tr|ft |z}|d }t)|D]T\}}|}||}|t |d z k(}t+|||||| | | | | |||d }|j$j-|Vt/|d|| | | | |d| d |_d |_t5t7|}t5t7|}|d }t)|D]\}}|t |d z k(}|}||}|t9|d zt |d z }|sd} |xj2d z c_nd} t;||d z||||| | | | | ||d|}!|j&j-|!|}| 1tj<|d | | |_tAd|_!nd|_d|_!|d z d z}"tj|d |||" |_"y)Na#At the moment it is not possible to define the number of attention heads via `num_attention_heads` because of a naming issue as described in https://github.com/huggingface/diffusers/issues/2011#issuecomment-1547958131. Passing `num_attention_heads` will only be supported in diffusers v0.19.z\Must provide the same number of `down_block_types` as `up_block_types`. `down_block_types`: z. `up_block_types`: .zbMust provide the same number of `block_out_channels` as `down_block_types`. `block_out_channels`: z. `down_block_types`: zdMust provide the same number of `num_attention_heads` as `down_block_types`. `num_attention_heads`: r rrr) kernel_sizepaddingT)r; cond_proj_dim)r@r?r3 num_layersr<F) rIr3r4 temb_channelsadd_downsample resnet_eps resnet_act_fn resnet_groupsr>r@r9dual_cross_attention) r3rJrLrMoutput_scale_factorr>r@rNrO) rIr3r4prev_output_channelrJ add_upsamplerLrMrNr>r@rOresolution_idx) num_channels num_groupsepssilu)#super__init__r2NotImplementedErrorlen ValueError isinstanceintnnConv2dconv_inr time_projrr1rtransformer_in ModuleList down_blocks up_blocks enumeraterappendr mid_blocknum_upsamplerslistreversedminr GroupNorm conv_norm_outrconv_actconv_out)$selfr2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rAconv_in_kernelconv_out_kernelconv_in_paddingtime_embed_dimtimestep_input_dimoutput_channelidown_block_type input_channelis_final_block down_blockreversed_block_out_channelsreversed_num_attention_heads up_block_typerRrSup_blockconv_out_padding __class__s$ r-rZzUNet3DConditionModel.__init__bs< &  *%v 2G5G  C$7 7nooAAUVdUeefg  ! "c*:&; ;tvHuII_`p_qqrs -s3.fn_recursive_add_processors,sdv/282F2F2H dV:./#)#8#8#: U%+tfAhZ,@%T U r,)strr(r`Modulerrr)rsrrrrs @r-attn_processorsz$UNet3DConditionModel.attn_processors!sm  c 588?? X\]`bt]tXu !//1 BLD& 'fj A Br, slice_sizec  g dtjjf fd |jD] }| t }|dk(r Dcgc]}|dz }}n |dk(r|dgz}t |t s||gzn|}t |t k7r=tdt |d|jd t d t d tt |D]&}||} |}|||kDstd |d|d dtjjdttf fd t t|}|jD] } || y cc}w)a Enable sliced attention computation. When this option is enabled, the attention module splits the input tensor in slices to compute attention in several steps. This is useful for saving some memory in exchange for a small decrease in speed. Args: slice_size (`str` or `int` or `list(int)`, *optional*, defaults to `"auto"`): When `"auto"`, input to the attention heads is halved, so attention is computed in two steps. If `"max"`, maximum amount of memory is saved by running only one slice at a time. If a number is provided, uses as many slices as `attention_head_dim // slice_size`. In this case, `attention_head_dim` must be a multiple of `slice_size`. rct|drj|j|jD] }| yNset_attention_slice)rrisliceable_head_dimchildren)rr$fn_recursive_retrieve_sliceable_dimssliceable_head_dimss r-rzVUNet3DConditionModel.set_attention_slice..fn_recursive_retrieve_sliceable_dimsKs@v45#**6+D+DE* <4U; .fn_recursive_set_attention_sliceqsEv45**:>>+;<* D0 C Dr,) r(r`rrr\r^rlr]configrangerr_rm) rsrrnum_sliceable_layersdimrzsizereversed_slice_sizerrrs @@@r-rz(UNet3DConditionModel.set_attention_slice;s! < <mmo 9F 0 8 9 ##67  /BBs#(BJB 5 -3J@J:W[@\)ZL8bl z?c"56 6$S_$5VDKK=cReNfMghQQTUhQiPjjkm  s:' VAa=D%a(CD3J 5.McURS!TUU  V DUXX__ DRVWZR[ D#8J#78mmo JF ,V5H I J=Cs E; processorc Tt|jj}t|tr,t||k7rt dt|d|d|ddt dtjjffd |jD]\}}|||y) a4 Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.rrct|drEt|ts|j|n#|j|j |d|j D]\}}|d|||y)N set_processorrrC)rr^dictrrr)rrrrrfn_recursive_attn_processors r-rzLUNet3DConditionModel.set_attn_processor..fn_recursive_attn_processorsxv/!)T2((3(($z7J)KL#)#8#8#: T%+tfAhZ,@%S Tr,N) r\rkeysr^rr]rr(r`rr)rsrcountrrrs @r-set_attn_processorz'UNet3DConditionModel.set_attn_processor}sD((--/0 i &3y>U+BPQTU^Q_P`a005w6QRWQXXkm   Tc T588?? T!//1 ALD& 'fi @ Ar, chunk_sizerc|dvrtd||xsd}dtjjdtdtffd |j D] }|||y) aX Sets the attention processor to use [feed forward chunking](https://huggingface.co/blog/reformer#2-chunked-feed-forward-layers). Parameters: chunk_size (`int`, *optional*): The chunk size of the feed-forward layers. If not specified, will run feed-forward layer individually over each tensor of dim=`dim`. dim (`int`, *optional*, defaults to `0`): The dimension over which the feed-forward computation should be chunked. Choose between dim=0 (batch) or dim=1 (sequence length). )rrz-Make sure to set `dim` to either 0 or 1, not rrrrct|dr|j|||jD] }|||yNset_chunk_feed_forward)rrrrrrrrrfn_recursive_feed_forwards r-rzOUNet3DConditionModel.enable_forward_chunking..fn_recursive_feed_forwardEv78---M* B)%SA Br,N)r]r(r`rr_r)rsrrrrs @r-enable_forward_chunkingz,UNet3DConditionModel.enable_forward_chunkingsu f LSERS S _1  Behhoo B3 BUX Bmmo ?F %fj# > ?r,cdtjjdtdtffd |j D] }|ddy)Nrrrct|dr|j|||jD] }|||yrrrs r-rzPUNet3DConditionModel.disable_forward_chunking..fn_recursive_feed_forwardrr,r)r(r`rr_r)rsrrs @r-disable_forward_chunkingz-UNet3DConditionModel.disable_forward_chunkingsM Behhoo B3 BUX Bmmo 7F %fdA 6 7r,c jtd|jjDr t}nmtd|jjDr t }n8t dt t|jj|j|y)ze Disables custom attention processors and sets the default attention implementation. c3@K|]}|jtvywN)rr.0procs r- zBUNet3DConditionModel.set_default_attn_processor..si4t~~!>>ic3@K|]}|jtvywr)rrrs r-rzBUNet3DConditionModel.set_default_attn_processor..sh$#==hrzOCannot call `set_default_attn_processor` when attention processors are of type N) allrvaluesrrr]nextiterr)rsrs r-set_default_attn_processorz/UNet3DConditionModel.set_default_attn_processors i4K_K_KfKfKhi i,.I h$J^J^JeJeJgh h%IabfgklpmAmAmHmHmJhKcLbMN   *r,ct|jD]9\}}t|d|t|d|t|d|t|d|;y)aFEnables the FreeU mechanism from https://huggingface.co/papers/2309.11497. The suffixes after the scaling factors represent the stage blocks where they are being applied. Please refer to the [official repository](https://github.com/ChenyangSi/FreeU) for combinations of values that are known to work well for different pipelines such as Stable Diffusion v1, v2, and Stable Diffusion XL. Args: s1 (`float`): Scaling factor for stage 1 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. s2 (`float`): Scaling factor for stage 2 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. b1 (`float`): Scaling factor for stage 1 to amplify the contributions of backbone features. b2 (`float`): Scaling factor for stage 2 to amplify the contributions of backbone features. s1s2b1b2N)rhrgsetattr)rsrrrrrzupsample_blocks r- enable_freeuz!UNet3DConditionModel.enable_freeusQ$"+4>>!: . A~ ND" - ND" - ND" - ND" -  .r,chd}t|jD]3\}}|D])}t||st||dt ||d+5y)zDisables the FreeU mechanism.>rrrrN)rhrgrgetattrr)rs freeu_keysrzrks r- disable_freeuz"UNet3DConditionModel.disable_freeusW- !*4>>!: 5 A~ 5>1-D1Q1]NAt4 5 5r,crd|_|jjD]1\}}dt|jj vs(t d|j|_|jD]%}t|ts|jd'|jty)u1 Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. This API is 🧪 experimental. NAddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.T)fuse) original_attn_processorsritemsrrr$r]modulesr^rfuse_projectionsrr)rs_attn_processorrs r-fuse_qkv_projectionsz)UNet3DConditionModel.fuse_qkv_projectionss)-%!%!5!5!;!;!= v A~#n66??@@ !tuu v)-(<(<%lln 3F&),''T'2 3  5 78r,cT|j|j|jyy)uDisables the fused QKV projection if enabled. This API is 🧪 experimental. N)rr)rss r-unfuse_qkv_projectionsz+UNet3DConditionModel.unfuse_qkv_projectionss)  ( ( 4  # #D$A$A B 5r,r#timestepencoder_hidden_states class_labels timestep_condattention_maskcross_attention_kwargsdown_block_additional_residualsmid_block_additional_residual return_dictc  d|jzd} d} tfd|jddDrtj dd} |2d|j |j z d z}|jd}|} tj| s|jjd k(}|jjd k(}t|tr%|s|rtjntj}n$|s|rtj ntj"}tj$| g||j } n6t'| jd k(r| dj |j} |jd}| j)|jd } |j+| }|j |j }|j-||}|j/|d |jd |z}|j/|d |jd |z}|j1d ddddj3|jd |zdf|jddz}|j5|}|j7|||dd }|f}|j8D]?}t;|dr|j<r|||||||\}}n||||\}}||z }A|#d}t?||D]\}}||z}||fz }|}|j@|jA||||||}| || z}tC|jDD]\}}|t'|jDdz k(}|t'|jF d}|dt'|jF }|s| r|djdd} t;|dr|j<r|||||| |||}||||| |}|jHr"|jI|}|jK|}|jM|}|dddfj3d|f|jddzj1d dddd}| s|fStO|S)a The [`UNet3DConditionModel`] forward method. Args: sample (`torch.Tensor`): The noisy input tensor with the following shape `(batch, num_channels, num_frames, height, width`. timestep (`torch.Tensor` or `float` or `int`): The number of timesteps to denoise an input. encoder_hidden_states (`torch.Tensor`): The encoder hidden states with shape `(batch, sequence_length, feature_dim)`. class_labels (`torch.Tensor`, *optional*, defaults to `None`): Optional class labels for conditioning. Their embeddings will be summed with the timestep embeddings. timestep_cond: (`torch.Tensor`, *optional*, defaults to `None`): Conditional embeddings for timestep. If provided, the embeddings will be summed with the samples passed through the `self.time_embedding` layer to obtain the timestep embeddings. attention_mask (`torch.Tensor`, *optional*, defaults to `None`): An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. If `1` the mask is kept, otherwise if `0` it is discarded. Mask will be converted into a bias, which adds large negative values to the attention scores corresponding to "discard" tokens. cross_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). down_block_additional_residuals: (`tuple` of `torch.Tensor`, *optional*): A tuple of tensors that if specified are added to the residuals of down unet blocks. mid_block_additional_residual: (`torch.Tensor`, *optional*): A tensor that if specified is added to the residual of the middle unet block. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.unets.unet_3d_condition.UNet3DConditionOutput`] instead of a plain tuple. cross_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the [`AttnProcessor`]. Returns: [`~models.unets.unet_3d_condition.UNet3DConditionOutput`] or `tuple`: If `return_dict` is True, an [`~models.unets.unet_3d_condition.UNet3DConditionOutput`] is returned, otherwise a `tuple` is returned where the first element is the sample tensor. rFNc3.K|] }|zdk7yw)rNr+)rsdefault_overall_up_factors r-rz/UNet3DConditionModel.forward..]sMaq,,1Msz9Forward upsample size to force interpolation output size.Trgmpsnpu)dtypedevicer)r)r output_sizer rFrP) num_framesrrhas_cross_attention) hidden_statestembrrrr)rrrr+)rrrr)rrres_hidden_states_tupler upsample_sizerrr)rrrrr)r#)(rkanyshapeloggerinfotor unsqueezer( is_tensorrtyper^floatfloat32float64int32int64tensorr\expandrcr1repeat_interleavepermutereshaperbrdrfrrziprjrhrgresnetsrprqrrr")rsr#rrrrrrrrrforward_upsample_sizer timestepsis_mpsis_npurrt_embembdown_block_res_samplesdownsample_block res_samplesnew_down_block_res_samplesdown_block_res_sampledown_block_additional_residualrzrr}rs @r-forwardzUNet3DConditionModel.forward!sl%&t':':$:!!& M6<<;LM M KKS T$( !  %."3"3FLL"AAXMN+55a8N y)]]''50F]]''50F(E**0F (.& u{{ i[fmmTI  !Q &!$**6==9I\\!_ $$V\\!_5 y) tzz*!!%7##JA399Q?DTDhDh&6"(*?#1)+A '# '7VRUbl&m# "k 1 " 2 + 6)+ &IL&(GJ GE%'E)>@^(^%*/D.FF*  G &@ " >> %^^&;-%'= $F ) 4;;F"+4>>!:  A~#dnn"5"99N0#n6L6L2M1M1OPK%;CYCY?Z>Z%[ ""&; 6r : @ @ D ~'<=.BdBd'"(,7*?"/#1)+A ("(,7"/) / @   ''/F]]6*Fv&a(("j)9FLLi@NN)Nr)NNNNNNT)#r$r%r&r' _supports_gradient_checkpointing _skip_layerwise_casting_patternsr rr_rrrr rZpropertyrrrrrrrrrrrrrr(r)rboolr"r __classcell__)rs@r-r/r/>s @(-$(.0@'A$&*- + /E !"#())+#'57@D,07| c]| |  |  S/ | c3h| ""#s(O#| $%| & '| (!&)| *+| ,"#-| ./| 0!1| 2"#uSz/23| 4&eCsO&<=5| 6%SM7| | |c+=&=!>0?JeCd3i4G.H?JT?JD AE2Dd3PbKbFc2c,d AD?(3-?S?Y]?: 7 + .2594 C$040415;?IM@D G4 G4 eS01G4 %|| G4 u||, G4   - G4!.G4!)c3h 8G4*2% 2E)FG4(0 '=G4G4 $eELL&99 :G4r,r/)/ dataclassesrtypingrrrrrr r(torch.nnr`torch.utils.checkpointconfiguration_utilsr r loadersr utilsrr activationsrattention_processorrrrrrrr embeddingsrrmodeling_utilsr!transformers.transformer_temporalrunet_3d_blocksrrr get_loggerr$rr"r/r+r,r-r:s ":: B2((6'H   H %  J   j 4:{4Oj 4r,