biddlZddlmZmZddlZddlmZddlmZmZddl m Z ddl m Z ddl mZdd lmZmZmZdd lmZdd lmZdd lmZej2eZd ZddZGddej<ZGddej<Z Gddej<Z!Gddej<Z"Gdde eZ#Gdde eZ$y)N)OptionalUnion)nn) ConfigMixinregister_to_config) ModelMixin) FeedForward) Attention)TimestepEmbedding Timestepsget_2d_sincos_pos_embed)Transformer2DModelOutput) AdaLayerNorm)loggingcd}||d|zz ks ||d|zzkDrtjdtj5|||z |z }|||z |z }|j d|zdz d|zdz |j |j |tjdz|j||j|||cdddS#1swYyxYw)Ncddtj|tjdz zdz S)N?@)matherfsqrt)xs h/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/pipelines/unidiffuser/modeling_uvit.pynorm_cdfz(_no_grad_trunc_normal_..norm_cdfs(dhhq499S>122c99zjmean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect.r)minmax) loggerwarningtorchno_graduniform_erfinv_mul_rradd_clamp_)tensormeanstdabrlus r_no_grad_trunc_normal_r1s: q1s7{q1s7{ 2 ;  a$h#% & a$h#% & A 1q519-   C$))C.() D  ! #+s BC$$C-c t|||||S)aFills the input Tensor with values drawn from a truncated normal distribution. The values are effectively drawn from the normal distribution :math:`\mathcal{N}(\text{mean}, \text{std}^2)` with values outside :math:`[a, b]` redrawn until they are within the bounds. The method used for generating the random values works best when :math:`a \leq \text{mean} \leq b`. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution a: the minimum cutoff value b: the maximum cutoff value Examples: >>> w = torch.empty(3, 5) >>> nn.init.trunc_normal_(w) )r1)r*r+r,r-r.s r trunc_normal_r39s "&$Q ::rc<eZdZdZ dfd ZdZxZS) PatchEmbedz2D Image to Patch Embeddingc t |||z||zz} ||_||_t j ||||f|||_|rt j|dd|_nd|_| |_ |jrLt|t| dzd} |jd| jjd d yy) N) kernel_sizestridebiasFgư>)elementwise_affineeps?pt) output_type pos_embedr) persistent)super__init__flatten layer_normrConv2dproj LayerNormnorm use_pos_embedrintregister_bufferfloat unsqueeze) selfheightwidth patch_size in_channels embed_dimrDrCr9rI num_patchesr? __class__s rrBzPatchEmbed.__init__Os +0CD  $II Z0HQ[bf    Y5dSDIDI*   / 3{C?O;P^bcI  ioo.?.I.I!.LY^ _ rc|j|}|jr!|jdjdd}|jr|j |}|j r||j zS|S)Nrr)rFrC transposerDrHrIr?)rNlatents rforwardzPatchEmbed.forwardnsd6" <<^^A&00A6F ??YYv&F   DNN* *Mr) rZrFTTT)__name__ __module__ __qualname____doc__rBrY __classcell__rUs@rr5r5Ls.% `> rr5c*eZdZdeffd ZdZxZS) SkipBlockdimct|tjd|z||_tj ||_y)Nr)rArBrLinear skip_linearrGrH)rNrerUs rrBzSkipBlock.__init__{s7 99QWc2LL% rcx|jtj||gd}|j|}|S)Nre)rhr#catrH)rNrskips rrYzSkipBlock.forwards3   UYY4yb9 : IIaLr)r]r^r_rJrBrYrarbs@rrdrdzs&C&rrdceZdZdZ ddedededeededeeded ed ed ed ed ededeffd Z ddZ xZ S)UTransformerBlockaS A modification of BasicTransformerBlock which supports pre-LayerNorm and post-LayerNorm configurations. Parameters: dim (`int`): The number of channels in the input and output. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. cross_attention_dim (`int`, *optional*): The size of the encoder_hidden_states vector for cross attention. activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward. num_embeds_ada_norm (:obj: `int`, *optional*): The number of diffusion steps used during training. See `Transformer2DModel`. attention_bias (:obj: `bool`, *optional*, defaults to `False`): Configure if the attentions should contain a bias parameter. only_cross_attention (`bool`, *optional*): Whether to use only cross-attention layers. In this case two cross attention layers are used. double_self_attention (`bool`, *optional*): Whether to use two self-attention layers. In this case no cross attention layers are used. upcast_attention (`bool`, *optional*): Whether to upcast the query and key to float32 when performing the attention calculation. norm_elementwise_affine (`bool`, *optional*): Whether to use learnable per-element affine parameters during layer normalization. norm_type (`str`, defaults to `"layer_norm"`): The layer norm implementation to use. pre_layer_norm (`bool`, *optional*): Whether to perform layer normalization before the attention and feedforward operations ("pre-LayerNorm"), as opposed to after ("post-LayerNorm"). Note that `BasicTransformerBlock` uses pre-LayerNorm, e.g. `pre_layer_norm = True`. final_dropout (`bool`, *optional*): Whether to use a final Dropout layer after the feedforward network. renum_attention_headsattention_head_dimcross_attention_dim activation_fnnum_embeds_ada_normattention_biasonly_cross_attentiondouble_self_attentionupcast_attentionnorm_elementwise_affine norm_typepre_layer_norm final_dropoutc ft|| |_|duxr| dk(|_||_| dvr|t d| d| dt |||||| r|nd| |_|| rt || s|nd||||| |_nd|_|jrt|||_ ntj|| |_ || r5|jr t||ntj|| |_ nd|_ tj|| |_t|||| |_y Nada_norm)r ada_norm_zeroz`norm_type` is set to zw, but `num_embeds_ada_norm` is not defined. Please make sure to define `num_embeds_ada_norm` if setting `norm_type` to .) query_dimheadsdim_headdropoutr9rrrx)rrrrrrr9rx)r:)rrsr|rArBrvuse_ada_layer_normr{ ValueErrorr attn1attn2rnorm1rrGnorm2norm3r ffrNrerprqrrrrsrtrurvrwrxryrzr{r|rUs rrBzUTransformerBlock.__init__f$ $8!#6d#B"_ U_H_, 5 5:M:U( 4KKT+UVX  %'7K 3QU-    *.C"?T$7Z^)+#!1DJDJ  " "%c+>?DJc>UVDJ  *.C **S"56\\#:QR J DJ\\#:QR c7-_lmrc@|jr1|jr|j||}n|j|}n|}||ni}|j|f|jr|nd|d|} |js0|jr|j| |} n|j| } | |z}|j |jr0|jr|j ||n|j |}n|}|j |f||d|} |js/|jr|j | |n|j | } | |z}|jr|j|}n|}|j|} |js|j| } | |z}|SN)encoder_hidden_statesattention_mask r{rrrrvrrrr) rN hidden_statesrrencoder_attention_masktimestepcross_attention_kwargs class_labelsnorm_hidden_states attn_output ff_outputs rrYzUTransformerBlock.forwards   &&%)ZZ x%H"%)ZZ %>"!.