bi ddlZddlmZmZmZmZddlZddlZddl m cm Z ddlm Z ddl mZddlmZmZddlmZ dd ej(d ed ed edededej(fdZ ddedeeeeeffdedededeej2dedej(fdZ ddedeeeeeffdedededej8f dZ ddeej2defdZddZddZ ddZ!d Z"d!Z#Gd"d#e jHZ%Gd$d%e jHZ&Gd&d'e jHZ'Gd(d)e jHZ( dd+ed,ed-ed.eeeefdeej2deej(eej(ej(fff d/Z) dd0eeeefd+edeej2deej(eej(ej(fffd1Z* ddeej2defd2Z+dd3Z,dd4Z-dd5Z.d6d ddd*ej^fd7ed8eej8efd+efd9Z0 dd:ej(d;eej(eej(fd,edZ1d:ej(fd?Z2Gd@dAe jHZ3GdBdCe jHZ4GdDdEe jHZ5GdFdGe jHZ6GdHdIe jHZ7GdJdKe jHZ8GdLdMe jHZ9GdNdOe jHZ:GdPdQe jHZ;GdRdSe jHZ<GdTdUe jHZ=GdVdWe jHZ>GdXdYe jHZ?GdZd[e jHZ@Gd\d]e jHZAGd^d_e jHZBGd`dae jHZCGdbdce jHZDGdddee jHZEGdfdge jHZFGdhdie jHZGGdjdke jHZHGdldme jHZIGdndoe jHZJdpZKGdqdre jHZLGdsdte jHZMGdudve jHZNGdwdxe jHZOGdydze jHZPGd{d|e jHZQGd}d~e jHZRGdde jHZSGdde jHZTGdde jHZUy)N)ListOptionalTupleUnion)nn) deprecate)FP32SiLUget_activation) AttentionF timesteps embedding_dimflip_sin_to_cosdownscale_freq_shiftscale max_periodreturnct|jdk(sJd|dz}tj| t j d|tj |jz}|||z z }t j|}|dddfj|dddfz}||z}t jt j|t j|gd}|r+t j|dd|df|ddd|fgd}|dzdk(r*tjjj|d }|S) a& This matches the implementation in Denoising Diffusion Probabilistic Models: Create sinusoidal timestep embeddings. Args timesteps (torch.Tensor): a 1-D Tensor of N indices, one per batch element. These may be fractional. embedding_dim (int): the dimension of the output. flip_sin_to_cos (bool): Whether the embedding order should be `cos, sin` (if True) or `sin, cos` (if False) downscale_freq_shift (float): Controls the delta between frequencies between dimensions scale (float): Scaling factor applied to the embeddings. max_period (int): Controls the maximum frequency of the embeddings Returns torch.Tensor: an [N x dim] Tensor of positional embeddings. r zTimesteps should be a 1d-arrayrr)startenddtypedeviceNdim)rr rr)lenshapemathlogtorcharangefloat32rexpfloatcatsincosr functionalpad) rrrrrrhalf_dimexponentembs V/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/models/embeddings.pyget_timestep_embeddingr/s76 y 1 $F&FF $!H$$u||XU]]9;K;K(H8&::;H ))H C AtG  " " $s47| 3C #+C ))UYYs^UYYs^4" =CiiQ \*C9H9 ,=>BGqAhh!!%%c<8 J? embed_dim spatial_size temporal_sizespatial_interpolation_scaletemporal_interpolation_scaler output_typecH|dk(rt|||||S|dzdk7r tdt|tr||f}d|zdz}|dz}t j |d|tj |z } t j |d|tj |z } t j| | d } t j| d } | jd d|d|dg} t|| d } t j ||tj |z } t|| d }| dddddf} | j|d| jd|z} |dddddf}|j|d|dzd }t j|| gd }|S)a Creates 3D sinusoidal positional embeddings. Args: embed_dim (`int`): The embedding dimension of inputs. It must be divisible by 16. spatial_size (`int` or `Tuple[int, int]`): The spatial dimension of positional embeddings. If an integer is provided, the same size is applied to both spatial dimensions (height and width). temporal_size (`int`): The temporal dimension of positional embeddings (number of frames). spatial_interpolation_scale (`float`, defaults to 1.0): Scale factor for spatial grid interpolation. temporal_interpolation_scale (`float`, defaults to 1.0): Scale factor for temporal grid interpolation. Returns: `torch.Tensor`: The 3D sinusoidal positional embeddings of shape `[temporal_size, spatial_size[0] * spatial_size[1], embed_dim]`. np)r2r3r4r5r6r"`embed_dim` must be divisible by 4r rrxyindexingrrptr7Nr output_sizer)_get_3d_sincos_pos_embed_np ValueError isinstanceintr!r"r#meshgridstackreshape!get_2d_sincos_pos_embed_from_grid!get_1d_sincos_pos_embed_from_gridrepeat_interleaverconcat)r2r3r4r5r6rr7embed_dim_spatialembed_dim_temporalgrid_hgrid_wgridpos_embed_spatialgrid_tpos_embed_temporal pos_embeds r.get_3d_sincos_pos_embedrYQs<d*%'(C)E   1}=>>,$$l3 I *"a\\,q/& NQl lF \\,q/& NQl lF >>&&4 8D ;;t #D <<A|A Q@ AD9:KT_cd\\-emm LOk kF:;Mvcgh*$1*5);;1*;*A*A!*D}*T<,AtQJ7+==Q,q/)q> 02CD"MI r0cXd}tdd|d|dzdk7r tdt|tr||f}d |zdz}|dz}t j |d tj |z }t j |dtj |z } t j| |} t j| d } | jd d |d |dg} t|| } t j |tj |z } t|| } | tjddddf} t j| |d } | ddtjddf} t j| |d|d zd } t j| | gd }|S)a Creates 3D sinusoidal positional embeddings. Args: embed_dim (`int`): The embedding dimension of inputs. It must be divisible by 16. spatial_size (`int` or `Tuple[int, int]`): The spatial dimension of positional embeddings. If an integer is provided, the same size is applied to both spatial dimensions (height and width). temporal_size (`int`): The temporal dimension of positional embeddings (number of frames). spatial_interpolation_scale (`float`, defaults to 1.0): Scale factor for spatial grid interpolation. temporal_interpolation_scale (`float`, defaults to 1.0): Scale factor for temporal grid interpolation. Returns: `np.ndarray`: The 3D sinusoidal positional embeddings of shape `[temporal_size, spatial_size[0] * spatial_size[1], embed_dim]`. z`get_3d_sincos_pos_embed` uses `torch` and supports `device`. `from_numpy` is no longer required. Pass `output_type='pt' to use the new version now.output_type=='np'0.33.0F standard_warnr:rr;r<r raxisrNr)r rFrGrHr9r"r#rIrJrKrLrMnewaxisrepeat concatenate)r2r3r4r5r6deprecation_messagerPrQrRrSrTrUrVrWrXs r.rErEs: ? !8-@PUV1}=>>,$$l3 I *"aYY|Abjj 9r?rrrB) r get_2d_sincos_pos_embed_nprGrHr!r"r#rIrJrKrLrOzeros) r2rhrirjrkrlrr7rerRrSrTrXs r.get_2d_sincos_pos_embedrosT<d C  %x1DTYZ)% 3   )S! *   Yq\& F Q<) # %    Yq\& F Q<) # %   >>&&4 8D ;;t #D <<Ay|Yq\: ;D1)T{[I\A%LL%++|Y.G"H)!TZ[\ r0c|dk(rd}tdd|dt||S|dzd k7r td t|dz|d | }t|dz|d | }t j ||gd }|S)aG This function generates 2D sinusoidal positional embeddings from a grid. Args: embed_dim (`int`): The embedding dimension. grid (`torch.Tensor`): Grid of positions with shape `(H * W,)`. Returns: `torch.Tensor`: The 2D sinusoidal positional embeddings with shape `(H * W, embed_dim)` r9z`get_2d_sincos_pos_embed_from_grid` uses `torch` and supports `device`. `from_numpy` is no longer required. Pass `output_type='pt' to use the new version now.r[r\Fr])r2rTrr embed_dim must be divisible by 2rBr r)r $get_2d_sincos_pos_embed_from_grid_nprFrMr!rO)r2rTr7reemb_hemb_wr-s r.rLrL!sd C  %x1DTYZ3  1};<< .i1nd1gS^ _E -i1nd1gS^ _E ,,u~1 -C Jr0c2|dk(rd}tdd|dt||S|dzd k7r td tj|dz|j tj }||d z z}d d|zz }|jd}tj||}tj|}tj|}tj||gd} |r1tj| dd|dzdf| ddd|dzfgd} | S)a* This function generates 1D positional embeddings from a grid. Args: embed_dim (`int`): The embedding dimension `D` pos (`torch.Tensor`): 1D tensor of positions with shape `(M,)` Returns: `torch.Tensor`: Sinusoidal positional embeddings of shape `(M, D)`. r9z`get_1d_sincos_pos_embed_from_grid` uses `torch` and supports `device`. `from_numpy` is no longer required. Pass `output_type='pt' to use the new version now.r[z0.34.0Fr])r2posrrrqr=@r1'rr rN) r $get_1d_sincos_pos_embed_from_grid_nprFr!r"rfloat64rKouterr'r(rOr&) r2rvr7rreomegaoutemb_sinemb_cosr-s r.rMrMBsd C  %x1DTYZ3iSQQ1};<< LLa %-- PE Y_E %, E ++b/C ++c5 !CiinGiinG ,,)q 1CiiQ Q 0013q:JIN:J7J3KLRST Jr0ct|tr||f}tj|dtj|d|z z |z }tj|dtj|d|z z |z }tj ||}tj |d}|jdd|d|dg}t||} |r3|dkDr.tjtj||g| gd} | S)a Creates 2D sinusoidal positional embeddings. Args: embed_dim (`int`): The embedding dimension. grid_size (`int`): The size of the grid height and width. cls_token (`bool`, defaults to `False`): Whether or not to add a classification token. extra_tokens (`int`, defaults to `0`): The number of extra tokens to add. interpolation_scale (`float`, defaults to `1.0`): The scale of the interpolation. Returns: pos_embed (`np.ndarray`): Shape is either `[grid_size * grid_size, embed_dim]` if not using cls_token, or `[1 + grid_size*grid_size, embed_dim]` if using cls_token rr_r r`r) rGrHr9r"r#rIrJrKrrrdrn) r2rhrirjrkrlrRrSrTrXs r.rmrmks.)S! * YYy|2:: 6)A,:R SVi iF YYy|2:: 6)A,:R SVi iF ;;vv &D 88Dq !D <<Ay|Yq\: ;D4YEI\A%NNBHHlI-F$G#SZ[\ r0c|dzdk7r tdt|dz|d}t|dz|d}tj||gd}|S)aC This function generates 2D sinusoidal positional embeddings from a grid. Args: embed_dim (`int`): The embedding dimension. grid (`np.ndarray`): Grid of positions with shape `(H * W,)`. Returns: `np.ndarray`: The 2D sinusoidal positional embeddings with shape `(H * W, embed_dim)` rrrqr r`)rFryr9rd)r2rTrsrtr-s r.rrrrs_1};<< 1aa IE 0aa IE ..%a 0C Jr0cr|dzdk7r tdtj|dztj}||dz z}dd|zz }|j d}tj d ||}tj |}tj|}tj||gd }|S) a, This function generates 1D positional embeddings from a grid. Args: embed_dim (`int`): The embedding dimension `D` pos (`numpy.ndarray`): 1D tensor of positions with shape `(M,)` Returns: `numpy.ndarray`: Sinusoidal positional embeddings of shape `(M, D)`. rrrqr_rwr1rxrzm,d->mdr r`) rFr9r"rzrKeinsumr'r(rd)r2rvr|r}r~rr-s r.ryrys1};<< IIi1nBJJ 7E Y_E %, E ++b/C ))IsE *CffSkGffSkG ..'7+! 4C Jr0cFeZdZdZ dfd ZdZdZxZS) PatchEmbeda 2D Image to Patch Embedding with support for SD3 cropping. Args: height (`int`, defaults to `224`): The height of the image. width (`int`, defaults to `224`): The width of the image. patch_size (`int`, defaults to `16`): The size of the patches. in_channels (`int`, defaults to `3`): The number of input channels. embed_dim (`int`, defaults to `768`): The output dimension of the embedding. layer_norm (`bool`, defaults to `False`): Whether or not to use layer normalization. flatten (`bool`, defaults to `True`): Whether or not to flatten the output. bias (`bool`, defaults to `True`): Whether or not to use bias. interpolation_scale (`float`, defaults to `1`): The scale of the interpolation. pos_embed_type (`str`, defaults to `"sincos"`): The type of positional embedding. pos_embed_max_size (`int`, defaults to `None`): The maximum size of the positional embedding. c bt|||z||zz} ||_||_| |_t j ||||f|||_|rt j|dd|_ nd|_ ||_ ||z||zc|_ |_ ||z|_ | |_| r| } nt| dz} | d|_y| dk(r\t#|| |j|jd}| rd nd}|j%d |j'j)d | yt+d | )N kernel_sizestridebiasFgư>)elementwise_affineeps?sincosrA)rlrkr7TrXr persistentzUnsupported pos_embed_type: )super__init__flatten layer_normpos_embed_max_sizerConv2dproj LayerNormnorm patch_sizeheightwidthrlrkrHrXroregister_bufferr% unsqueezerF)selfrrr in_channelsr2rrrrkpos_embed_typer num_patchesrhrXr __class__s r.rzPatchEmbed.__init__sI +0CD  $"4II Z0HQ[bf    Y5dSDIDI$"(J"68K TZ:-#6  *IK,-I  !!DN x '/..$($<$< I"4J  ioo.?.I.I!.LYc d;N;KLM Mr0c6|j td||jz}||jz}||jkDrtd|d|jd||jkDrtd|d|jd|j|z dz}|j|z dz}|jj d|j|jd }|dd|||z|||zddf}|j dd |j d }|S) z2Crops positional embeddings for SD3 compatibility.Nz.`pos_embed_max_size` must be set for cropping.zHeight (z/) cannot be greater than `pos_embed_max_size`: .zWidth (rr r)rrFrrXrKr)rrrtopleftspatial_pos_embeds r.cropped_pos_embedzPatchEmbed.cropped_pos_embed sJ  " " *MN N4??*( D++ +6("QRVRiRiQjjkl  4** *% OPTPgPgOhhij &&/A5''%/A5 NN221d6M6MtOfOfhjk-asV|1CTDSXLEXZ[.[\-55a=N=T=TUW=XY  r0cR|j|jdd\}}n8|jd|jz|jd|jz}}|j|}|jr!|j dj dd}|j r|j|}|j|j|jS|jr|j||}n|j|k7s|j|k7rht|jjd||f|j|j |j"d}|j%j'd}n |j}||zj|jS)Nrrr rA)r2rhrlrkrr7r)rrrrr transposerrrXtorrrrrorlrkrr%r)rlatentrrrXs r.forwardzPatchEmbed.forward#s_  " " ."LL-MFE"LL,?bAQUYUdUdAdEF6" <<^^A&00A6F ??YYv&F >> !99V\\* *  " "..vu=I{{f$ e(;3"nn2226%uo"nn(,(@(@!== $  &OO-77:  NN "&&v||44r0) rr<FTTr rN)__name__ __module__ __qualname____doc__rrr __classcell__rs@r.rrs<& 5Nn!.5r0rc*eZdZdZdfd ZdZxZS)LuminaPatchEmbedaz 2D Image to Patch Embedding with support for Lumina-T2X Args: patch_size (`int`, defaults to `2`): The size of the patches. in_channels (`int`, defaults to `4`): The number of input channels. embed_dim (`int`, defaults to `768`): The output dimension of the embedding. bias (`bool`, defaults to `True`): Whether or not to use bias. cvt|||_tj||z|z|||_y)N in_features out_featuresr)rrrrLinearr)rrrr2rrs r.rzLuminaPatchEmbed.__init__Ns8 $II"Z/+="  r0cR|j|dj}|jx}}|j\}}}}||z||z} } |j ||| || |j dddddd}|j d}|j|}|j dd}tj|jd|jdtj|j} || ||fg|z|d| d| fj ddjdfS) a Patchifies and embeds the input tensor(s). Args: x (List[torch.Tensor] | torch.Tensor): The input tensor(s) to be patchified and embedded. Returns: Tuple[torch.Tensor, torch.Tensor, List[Tuple[int, int]], torch.Tensor]: A tuple containing the patchified and embedded tensor(s), the mask indicating the valid patches, the original image size(s), and the frequency tensor(s). rrr:r r<rrN) rrrsizeviewpermuterrr!onesrint32r) rx freqs_cis patch_height patch_width batch_sizechannelrr height_tokens width_tokensmasks r.rzLuminaPatchEmbed.forwardWs%LL1- %)__4 {-.VVX* GVU&, &R| FF:w |\S^ _ g g q!Q1  IIaL IIaL IIaOzz!''!*aggaj AHHU  e_  * n}nm|m3 4 < 9= $(I:J ;;M]>]']`a'a$'*::=YY / NN 0 1 (  , ,  - - &--a3 +55 t''+5t~~UZ6    D$<$<$>!>?EEmT""r0 text_embeds image_embedsc |j|}|j\}}}}}|jz|jd|||}|j |}|j ||g|jdd}|j djdd}|j dd}n|j}|j} |jddddd}|j||| z| ||z|||z||}|jddddd ddd j dd j dd}|j |}tj||gd j} |js |jr|jr)|j|k7s|j |k7r t#d |dz |j$zdz} |j |k7s|j|k7s|j&| k7r |j)||| | j* } n |j,} | j/| j0} | | z} | S)a7 Args: text_embeds (`torch.Tensor`): Input text embeddings. Expected shape: (batch_size, seq_length, embedding_dim). image_embeds (`torch.Tensor`): Input image embeddings. Expected shape: (batch_size, num_frames, channels, height, width). Nrr r<rrr:rraIt is currently not possible to generate videos at a different resolution that the defaults. This should only be the case with 'THUDM/CogVideoX-5b-I2V'.If you think this is incorrect, please open an issue at https://github.com/huggingface/diffusers/issues.rr_)rrrrKrrrrrrr!r& contiguousrrrrrFrrrrrrr) rrrr num_frameschannelsrrpp_tembedspre_time_compression_framesrs r.rzCogVideoXPatchEmbed.forwardsfnn[1 :F:L:L7 J&%    $'//HfeLL99\2L,<,,Z]lFXFXYZY[F\]L'//2<L'//J#-sFaKEQJPQS[L(//1aAq!QGOOPQSTU]]^_abcL99\2L , 'Q *,   ) )T-S-S554;L;LPU;UY]YkYkouYu  ,6>T=\=\*\_`*` '""f,$$-%%)DD $ ? ?E#>v}}!@! !% 2 2 ),,6<<,@Mm+F r0)rNriTZ<1r:g?r1TTN)rrrrHrboolr%rr!rTensorrrrrs@r.rrysJ&*"*+#&-2.1*.26!/X/Xsm/X /X  /X  /X/X/X/X/X%(/X!/X&+/X',/X$(/X ,0!/X" #/Xdko# #03#DG#QYZ_ZfZfQg# #285<<8u||8r0rc eZdZ d dededededef fd Zdej dej d ej fd ZxZS) CogView3PlusPatchEmbedr hidden_sizertext_hidden_sizercht|||_||_||_||_||_tj||dzz||_ tj|||_ t|||d}|j|||}|jd|jdy)NrrA)rlr7rXFr)rrrrrrrrrrrrorKrr%)rrrrrrrXrs r.rzCogView3PlusPatchEmbed.__init__s &&$ 0"4IIkJM9;G #3[A+ +7IW[ %%&8:LkZ  [)//*;Nr0 hidden_statesencoder_hidden_statesrc|j\}}}}||jzdk7s||jzdk7r td||jz}||jz}|j||||j||j}|j ddddddj }|j|||z||jz|jz}|j |}|j|}tj||gd}|jd}|jd|d|fj||zd }tj||jf|j|j } tj| |gdd } || zj!|jS) Nrz0Height and width must be divisible by patch sizerr:r r<rrrr)N.)rrrFrrrrrr!r&rXrKrnrrrr) rrrrrrr text_lengthimage_pos_embedtext_pos_embedrXs r.rzCogView3PlusPatchEmbed.forwards-:-@-@* GVU DOO #q (EDOO,Cq,HOP P4??*(%**:wY^`d`o`op %--aAq!Q?JJL %**:v~wQUQ`Q`G`cgcrcrGrs  -0 $/D E #8-"HaP ,11!4 ..&&5&9AA&5.RTU $** +?3H3HQ`QgQg II~?QG R  )--m.A.ABBr0)ri rr rrrrHrr!rrrrs@r.rrs| $"% OOO O  O  O4CU\\C%,,C[`[g[gCr0rTthetause_real grid_typemax_sizec  |dur td|dk(r|\} } |\ tj| d| ddz zz |tj} tj| d| d dz z z |tj} tj|tj} tjddz zz |tj} n|dk(r}|\}}|\ tj||tj} tj||tj} tj|tj} n td|d z}|d zd z}|d zd z}t || |d }t || |d }t || |d } fd }|\}}|\}}|\}}|dk(r|d|d}}|d|d}}|d |d }}||||}||||}||fS)a RoPE for video tokens with 3D structure. Args: embed_dim: (`int`): The embedding dimension size, corresponding to hidden_size_head. crops_coords (`Tuple[int]`): The top-left and bottom-right coordinates of the crop. grid_size (`Tuple[int]`): The grid size of the spatial positional embedding (height, width). temporal_size (`int`): The size of the temporal dimension. theta (`float`): Scaling factor for frequency computation. grid_type (`str`): Whether to use "linspace" or "slice" to compute grids. Returns: `torch.Tensor`: positional embedding with shape `(temporal_size * grid_size[0] * grid_size[1], embed_dim/2)`. TzJ `use_real = False` is not currently supported for get_3d_rotary_pos_embedlinspacerr r=slicez%Invalid value passed for `grid_type`.r:r<)r rc&|ddddddfjdd}|ddddddfjdd}|ddddddfjdd}tj|||gd}|jzzd}|S)Nrr)expandr!r&r)freqs_tfreqs_hfreqs_wfreqs grid_size_h grid_size_wr4s r.combine_time_height_widthz:get_3d_rotary_pos_embed..combine_time_height_width{s!T4*+22  [" $4*+22 2{B $a*+22 ;B  gw 'R  K '+ 5r  r0N)rFr!rr#r"get_1d_rotary_pos_embed)!r2 crops_coordsrhr4r rrrrrstoprRrSrVmax_hmax_wdim_tdim_hdim_wrrrrt_cost_sinh_cosh_sinw_cosw_sinr(r'rrs! ` @@r.get_3d_rotary_pos_embedr,7sD>teffJ" t#, [ !Hd1gq1K?U[chcpcp  !Hd1gq1K?U[chcpcp mF%--P }  12]BMZ`hmhuhu  g  u#, [eF%--HeF%--HmF%--P@AA NE NQ E NQ E&eV54PG%eV54PG%eV54PG&LE5LE5LE5G^m,eN].Cu\k*E,;,?u\k*E,;,?u #E5% 8C #E5% 8C 8Or0rkc|\}}|\} } |\} } } tjd||dz z|z ||tj}tj|d|d| dz z| z | |tj}tj|d|d| dz z| z | |tj}|dz}|dz}|dz}t||| z |dd}t||| z |dd}t||| z |dd}||||||fS)Nrr r=r<TF)r rrepeat_interleave_real)r!rr#r)r2rrhr4rkr rrr rrinterpolation_scale_tinterpolation_scale_hinterpolation_scale_wrVrRrSr#r$r%rrrs r.get_3d_rotary_pos_embed_allegror2sTKE4(KJ]G02G ^^ =MA- . > V\didqdqF^^ a$q'[1_- ;[QW_d_l_lF^^ a$q'[1_- ;[QW_d_l_lF NE NE NE& v--UTbgG& v--UTbgG& v--UTbgG GWfff <r?rrNr) r _get_2d_rotary_pos_embed_npr!rr#rIrJrKr!get_2d_rotary_pos_embed_from_grid) r2rrhrrr7rerr rRrSrTrXs r.get_2d_rotary_pos_embedr7s2*d C  %x1DTYZ*%   KE4 ^^ a$q'Yq\A-.1=y|TZbgboboF^^ a$q'Yq\A-.1=y|TZbgboboF >>&&4 8D ;;t #D <<A/ 12/ 0D1)THUI r0c|\}}tj|d|d|ddtj}tj|d|d|ddtj}tj||}tj|d}|j ddg|j dd}t|||} | S) a RoPE for image tokens with 2d structure. Args: embed_dim: (`int`): The embedding dimension size crops_coords (`Tuple[int]`) The top-left and bottom-right coordinates of the crop. grid_size (`Tuple[int]`): The grid size of the positional embedding. use_real (`bool`): If True, return real part and imaginary part separately. Otherwise, return complex numbers. Returns: `torch.Tensor`: positional embedding with shape `( grid_size * grid_size, embed_dim/2)`. rF)endpointrr r`rNr4)r9rr#rIrJrKrr6) r2rrhrrr rRrSrTrXs r.r5r5s"KE4 [[q47IaL5PRPZPZ [F [[q47IaL5PRPZPZ [F ;;vv &D 88Dq !D <<A/ 12/ 0D1)THUI r0c`|dzdk(sJt|dz|djd|}t|dz|djd|}|rBtj|d|dgd}tj|d|dgd}||fStj||gd}|S)a Get 2D RoPE from grid. Args: embed_dim: (`int`): The embedding dimension size, corresponding to hidden_size_head. grid (`np.ndarray`): The grid of the positional embedding. use_real (`bool`): If True, return real part and imaginary part separately. Otherwise, return complex numbers. Returns: `torch.Tensor`: positional embedding with shape `( grid_size * grid_size, embed_dim/2)`. r:rrrr4r r)rrKr!r&)r2rTrrsrtr(r'r-s r.r6r6s q=A   $QQ+h E $QQ+h Eiiq58,!4iiq58,!4CxiiA. r0cV|dzdk(sJt|dz|||}t|dz|||}|j|d|dzdjd|dd}|jd||dzdj|ddd}tj||gdj d}|S)as Get 2D RoPE from grid. Args: embed_dim: (`int`): The embedding dimension size, corresponding to hidden_size_head. grid (`np.ndarray`): The grid of the positional embedding. linear_factor (`float`): The linear factor of the positional embedding, which is used to scale the positional embedding in the linear layer. ntk_factor (`float`): The ntk factor of the positional embedding, which is used to scale the positional embedding in the ntk layer. Returns: `torch.Tensor`: positional embedding with shape `( grid_size * grid_size, embed_dim/2)`. r:rr) linear_factor ntk_factorr rr)rrrcr!r&r)r2len_hlen_wr<r=rsrtr-s r.get_2d_rotary_pos_embed_luminar@7s$ q=A   #Q]z E $Q]z E JJuaa 3 : :1eQ JE JJq%a 3 : :5!Q JE ))UEN + 3 3A 6C Jr0@rrvc (|dzdk(sJt|trtj|}t|tj rtj |}||z}d|tjd|d||j|z zz |z }tj||}|jjdk(} | r|j}|r|r|jjdd|jddzj} |jjdd|jddzj} | | fS|rtj|j|jgd j} tj|j|jgd j} | | fStj tj"||} | S) a Precompute the frequency tensor for complex exponentials (cis) with given dimensions. This function calculates a frequency tensor with complex exponentials using the given dimension 'dim' and the end index 'end'. The 'theta' parameter scales the frequencies. The returned tensor contains complex values in complex64 data type. Args: dim (`int`): Dimension of the frequency tensor. pos (`np.ndarray` or `int`): Position indices for the frequency tensor. [S] or scalar theta (`float`, *optional*, defaults to 10000.0): Scaling factor for frequency computation. Defaults to 10000.0. use_real (`bool`, *optional*): If True, return real part and imaginary part separately. Otherwise, return complex numbers. linear_factor (`float`, *optional*, defaults to 1.0): Scaling factor for the context extrapolation. Defaults to 1.0. ntk_factor (`float`, *optional*, defaults to 1.0): Scaling factor for the NTK-Aware RoPE. Defaults to 1.0. repeat_interleave_real (`bool`, *optional*, defaults to `True`): If `True` and `use_real`, real part and imaginary part are each interleaved with themselves to reach `dim`. Otherwise, they are concateanted with themselves. freqs_dtype (`torch.float32` or `torch.float64`, *optional*, defaults to `torch.float32`): the dtype of the frequency tensor. Returns: `torch.Tensor`: Precomputed frequency tensor with complex exponentials. [S, D/2] rrr1rnpur rCrr)rGrHr!r"r9ndarray from_numpyrr{typer%r(rNrr'r&polar ones_like) rrvr rr<r=r. freqs_dtyperis_npu freqs_cos freqs_sinrs r.rrXsH 7a<<#sll3#rzz"s# J E uaa{3::VY\\]^ann  KKU #E \\  % 'F  *IIK11! TUYZHZ1[aac IIK11! TUYZHZ1[aac )## IIuyy{EIIK8bAGGI IIuyy{EIIK8bAGGI )##KK 6> r0rruse_real_unbind_dim sequence_dimcJ|r|\}}|dk(r|ddddddf}|ddddddf}n/|dk(r|ddddddf}|ddddddf}ntd|d|j|j|j|j}}|dk(r_|jg|jddddj d\}}t j| |gdjd} nd|d k(rP|jg|jddddj d \}}t j| |gd} ntd |d |j|z| j|zzj|j} | St j|jjg|jdddd} |jd}t j| |zjd} | j|S) a3 Apply rotary embeddings to input tensors using the given frequency tensor. This function applies rotary embeddings to the given query or key 'x' tensors using the provided frequency tensor 'freqs_cis'. The input tensors are reshaped as complex numbers, and the frequency tensor is reshaped for broadcasting compatibility. The resulting tensors contain rotary embeddings and are returned as real tensors. Args: x (`torch.Tensor`): Query or key tensor to apply rotary embeddings. [B, H, S, D] xk (torch.Tensor): Key tensor to apply freqs_cis (`Tuple[torch.Tensor]`): Precomputed frequency tensor for complex exponentials. ([S, D], [S, D],) Returns: Tuple[torch.Tensor, torch.Tensor]: Tuple of modified query tensor and key tensor with rotary embeddings. rNr z`sequence_dim=z` but should be 1 or 2.rrr<rz`use_real_unbind_dim=z` but should be -1 or -2.)rFrrrKrunbindr!rJrr&r%rview_as_complexr view_as_realtype_as) rrrrMrNr(r'x_realx_imag x_rotatedr}x_outs r.apply_rotary_embrXs!*S 1 dD!Q&'CdD!Q&'C Q dAtQ&'CdAtQ&'C~l^;RST T66!((#SVVAHH%5S " $&QYY< FFqII B &&QYY< .apply_1d_ropeskk#s#AtQM2kk#s#AtQM24v||B/14445vc6<Z7[BRC9"5 $~';';'='CCGG UUr0r<rrrr r)chunkr!r&) rr positionsrar&r'r(r)r*r+thws r.apply_rotary_emb_allegrorgsV6?2NUENUENUEggaRg GAq!a1ue4Aa1ue4Aa1ue4A 1a)$A Hr0c NeZdZ ddededededeef fd Zd dZxZS) TimestepEmbeddingrtime_embed_dimact_fnout_dim post_act_fncBt |tj||||_|tj||d|_nd|_t ||_||}n|}tj||||_|d|_ yt ||_ y)NFr) rrrrlinear_1 cond_projr actlinear_2post_act) rrrjrkrlrm cond_proj_dimsample_proj_biastime_embed_dim_outrs r.rzTimestepEmbedding.__init__s  +~?OP  $YY}kNDN!DN!&)  !( !/  .2DFVW   DM*;7DMr0c|||j|z}|j|}|j|j|}|j|}|j|j |}|Sr)rqrprrrsrt)rsample conditions r.rzTimestepEmbedding.forwardsl  dnnY77Fv& 88 XXf%Fv& == $]]6*F r0)siluNNNTr) rrrrHstrrrrrrs@r.ririsS %)888 8  8 c] 8@ r0ric jeZdZd dedededeffd ZdejdejfdZ xZ S) Timesteps num_channelsrrrcZt|||_||_||_||_yr)rrrrrr)rrrrrrs r.rzTimesteps.__init__s- (.$8! r0rrcvt||j|j|j|j}|S)N)rrr)r/rrrr)rrt_embs r.rzTimesteps.forwards8&     00!%!:!:**   r0)r ) rrrrHrr%rr!rrrrs@r.r~r~sAS4W\eh%,,r0r~c6eZdZdZ ddedeffd ZdZxZS)GaussianFourierProjectionz-Gaussian Fourier embeddings for noise levels.embedding_sizerc6t|tjt j ||zd|_||_||_|rH|`tjt j ||zd|_ |j|_|` yy)NFr) rrr Parameterr!randnweightr rW)rrrset_W_to_weightr rrs r.rz"GaussianFourierProjection.__init__,s~ ll5;;~#>#FV[\ .  \\%++n"="EUZ[DF&&DK r0c|jrtj|}|dddf|jdddfzdztjz}|j rAtj tj|tj|gd}|Stj tj|tj|gd}|S)Nrrr) r r!rr9pirr&r(r')rrx_projr}s r.rz!GaussianFourierProjection.forward;s 88 ! A1d7dkk$'22Q6>   ))UYYv. &0ABKC ))UYYv. &0ABKC r0)r1TTF) rrrrrHr%rrrrs@r.rr)s'7ns ! 05  r0rc4eZdZdZddedeffd ZdZxZS)SinusoidalPositionalEmbeddinga[Apply positional information to a sequence of embeddings. Takes in a sequence of embeddings with shape (batch_size, seq_length, embed_dim) and adds positional embeddings to them Args: embed_dim: (int): Dimension of the positional embedding. max_seq_length: Maximum sequence length to apply positional embeddings r2max_seq_lengthct|tj|j d}tj tjd|dt jd |z z}tjd||}tj||z|ddddddf<tj||z|ddddddf<|jd|y)Nr rrrApe) rrr!r"rr$rr rnr'r(r)rr2rpositiondiv_termrrs r.rz&SinusoidalPositionalEmbedding.__init__Ts <</99!<99U\\!Y:txx?P>PS\>\]^ [[NI 68h#671aA:8h#671aA: T2&r0cV|j\}}}||jddd|fz}|Sr)rr)rr_ seq_lengths r.rz%SinusoidalPositionalEmbedding.forward]s277:q ;J;' 'r0) rrrrrHrrrrs@r.rrHs  '#'s'r0rc:eZdZdZdedededeffd ZdZxZS)ImagePositionalEmbeddingsa Converts latent image classes into vector embeddings. Sums the vector embeddings with positional embeddings for the height and width of the latent space. For more details, see figure 10 of the dall-e paper: https://huggingface.co/papers/2102.12092 For VQ-diffusion: Output vector embeddings are used as input for the transformer. Note that the vector embeddings for the transformer are different than the vector embeddings from the VQVAE. Args: num_embed (`int`): Number of embeddings for the latent pixels embeddings. height (`int`): Height of the latent image i.e. the number of height embeddings. width (`int`): Width of the latent image i.e. the number of width embeddings. embed_dim (`int`): Dimension of the produced vector embeddings. Used for the latent pixel, height, and width embeddings. num_embedrrr2c8t|||_||_||_||_t j|j||_t j|j||_ t j|j||_ yr) rrrrrr2r Embeddingr- height_emb width_emb)rrrrr2rs r.rz"ImagePositionalEmbeddings.__init__{sn   ""<< :,,t{{I>djj)<r0cV|j|}|jtj|j|j j d|j}|jd}|jtj|j|j j d|j}|jd}||z}|j d|j|jzd}||ddd|jdddfz}|S)Nrr rr) r-rr!r"rrrrrrr)rindexr-rrpos_embs r.rz!ImagePositionalEmbeddings.forwardshhuo__U\\$++ell%S%X%XYZ\`\g\g%hi  ))!, NN5<< 5<<#P#U#UVWY]YcYc#de ''* y(,,q$++ ":B?GA~1~q011 r0rrs@r.rrcs8.=== =  =$r0rcLeZdZdZfdZddZddejfdZxZ S)LabelEmbeddinga7 Embeds class labels into vector representations. Also handles label dropout for classifier-free guidance. Args: num_classes (`int`): The number of classes. hidden_size (`int`): The size of the vector embeddings. dropout_prob (`float`): The probability of dropping a label. ct||dkD}tj||z||_||_||_yNr)rrrrembedding_table num_classes dropout_prob)rrrruse_cfg_embeddingrs r.rzLabelEmbedding.__init__sB (1,!||K:K,K[Y&(r0c|3FGr0rrc|jd}|j|}|j||jd}|j |}t j ||gdS)Nrrr r)rrrKrrr!r&)rrrrimage_text_embedss r.rzTextImageProjection.forwardsh &&q) !--l;-55j$B\B\^`ann[1 yy+[9qAAr0)rr r rs@r.rrs_#"#&%' H H H! H # H B5<< Bu|| Br0rcTeZdZ ddededeffd Zdej fdZxZS)ImageProjectionrrrct|||_tj||j|z|_tj ||_yr)rrrrrrrr)rrrrrs r.rzImageProjection.__init__sH %:"IIot7Q7QTg7ghLL!45 r0rc|jd}|j|j|jjj}|j ||j d}|j|}|S)Nrr)rrrrrrKrr)rrrs r.rzImageProjection.forwardsn!''* ((9J9J9Q9Q9W9W)XY #++J8R8RTVW yy. r0)rrrr rs@r.rrs? ##&%' 6 6! 6 # 6ELLr0rc@eZdZdfd Zdej fdZxZS)IPAdapterFullImageProjectionct|ddlm}|||dd|_t j ||_yNr  FeedForwardgelu)mult activation_fn)rr attentionrffrrr)rrrrrs r.rz%IPAdapterFullImageProjection.__init__s6 *o/BZ`aLL!45 r0rcB|j|j|Sr)rr)rrs r.rz$IPAdapterFullImageProjection.forwardsyy.//r0)rrrrrrr!rrrrs@r.rrs60ELL0r0rc@eZdZdfd Zdej fdZxZS)IPAdapterFaceIDImageProjectionct|ddlm}||_||_||||z|d|_tj||_ yr) rrrr num_tokensrrrrr)rrrrrrrs r.rz'IPAdapterFaceIDImageProjection.__init__sK *$#6 o/BZ/OVZjpqLL!45 r0rc|j|}|jd|j|j}|j |S)Nr)rrKrrr)rrrs r.rz&IPAdapterFaceIDImageProjection.forwards: GGL ! IIb$//4+C+C Dyy|r0)rrr r rrs@r.rrs6ELLr0rc(eZdZdfd ZddZxZS)CombinedTimestepLabelEmbeddingsct|tddd|_t d||_t ||||_y)NrTr rrrrrj)rrr~ time_projritimestep_embedderrclass_embedder)rrrclass_dropout_probrs r.rz(CombinedTimestepLabelEmbeddings.__init__sA "T`ab!2sS`!a,[-I[\r0c|j|}|j|j|}|j|}||z}|SNr_)rrrr)rtimestep class_labels hidden_dtypetimesteps_proj timesteps_emb conditionings r.rz'CombinedTimestepLabelEmbeddings.forwardsN1..~/@/@|/@/TU **<8 $|3 r0)g?rrrrrrrrs@r.rrs]r0rc$eZdZfdZdZxZS)"CombinedTimestepTextProjEmbeddingsct|tddd|_t d||_t ||d|_yNrTrrrr{rk)rrr~rrirPixArtAlphaTextProjection text_embedderrrpooled_projection_dimrs r.rz+CombinedTimestepTextProjEmbeddings.__init__*sB "T`ab!2sS`!a67Lmdjkr0c|j|}|j|j|j}|j |}||z}|Sr)rrrrr)rrpooled_projectionrrpooled_projectionsrs r.rz*CombinedTimestepTextProjEmbeddings.forward1sX1..~/@/@GXG^G^/@/_` !//0AB$'99 r0rrs@r.rr)slr0rc$eZdZfdZdZxZS)*CombinedTimestepGuidanceTextProjEmbeddingsct|tddd|_t d||_t d||_t||d|_yr) rrr~rrirguidance_embedderrrrs r.rz3CombinedTimestepGuidanceTextProjEmbeddings.__init__=sR "T`ab!2sS`!a!2sS`!a67Lmdjkr0c,|j|}|j|j|j}|j|}|j |j|j}||z}|j |} || z} | Sr)rrrrrr) rrguidancerrr guidance_proj guidance_embtime_guidance_embrrs r.rz2CombinedTimestepGuidanceTextProjEmbeddings.forwardEs1..~/@/@GXG^G^/@/_` x0 --m.>.>EVE\E\.>.]^ )L8!//0AB(+== r0rrs@r.rr<sl r0rc eZdZd dedededeffd Zdej dej dej d ej d ejd ej f d ZxZ S)&CogView3CombinedTimestepSizeEmbeddingsr condition_dimr timesteps_dimct|t|dd|_t|dd|_t |||_t||d|_y)NTrrrr{r) rrr~rcondition_projrirrcondition_embedder)rrrrrrs r.rz/CogView3CombinedTimestepSizeEmbeddings.__init__UsV " tjkl']TXopq!2}]j!k"; ))!T1*588A A-- BQ%    !wwr{  nn  ++,,  ++,, ++,,   DKK$4$4dkk6F6F HXHX#YZ      !KK.. ++**! "&*# $]]% &' 1*yy|r0rrrrrHrrrrs@r.rrus*#C#C#C#UX#r0rc0eZdZ dfd ZddZxZS)-HunyuanCombinedTimestepTextSizeStyleEmbeddingcFt|tddd|_t d||_tddd|_t||d||_||_ |r$tjd||_ d |z|z}n|}t||d z|d |_y) NrTrrrr)rrr r: silu_fp32)rrrrk)rrr~rrir size_projrpooler"use_style_cond_and_image_meta_sizerrstyle_embedderrextra_embedder)rrrseq_lenrr= extra_in_dimrs r.rz6HunyuanCombinedTimestepTextSizeStyleEmbedding.__init__s "T`ab!2sS`!a"T`ab- (ABW  3U/ -"$,,q-"@D "]25JJL0L7$%)&  r0c|j|}|j|j|}|j|}|jrp|j |j d}|j|}|j dd}|j|} tj||| gd} ntj|gd} ||j| z} | S)Nr_rr9r r) rrrr<r=r;rr>r!r&r?) rrrimage_meta_sizestylerrrrstyle_embedding extra_condrs r.rz5HunyuanCombinedTimestepTextSizeStyleEmbedding.forwards1..~/@/@|/@/TU "[[)>?  2 2"nn_-A-A"-EFO-00|0DO-222w?O#11%8O$6#Y_`aJ$6#7Q?J$t':'::'FF r0)rrTrrrs@r.r7r7s# +/  Dr0r7c&eZdZdfd ZdZxZS)&LuminaCombinedTimestepCaptionEmbeddingct|t|dd|_t |||_t jt j|t j||d|_ y)NTrrro) rrr~rrirr Sequentialrrcaption_embedder)rrrfrequency_embedding_sizers r.rz/LuminaCombinedTimestepCaptionEmbedding.__init__sh "14^a "3?Whs!t " LL, - II# ! r0cZ|j|}|j|j|j}|j j d}||zj d|j dz }|j|}|j|}||z} | SNr_rr r)rrrrr%rsumrM) rr caption_feat caption_mask time_freq time_embedcaption_mask_floatcaption_feats_pool caption_embedrs r.rz.LuminaCombinedTimestepCaptionEmbedding.forwardsNN8, ++ILL|?Q?QL,RS *//1;;B?*-??DDDKN`NdNdijNdNkk/22<@--.@A !M1 r0)rrGrrrs@r.rIrIs  " r0rIc eZdZ ddedededededdf fd Z dd ej d ejd ejd eejfd Z xZ S)%MochiCombinedTimestepCaptionEmbeddingrrrrjnum_attention_headsrNct|t|dd|_t |||_t ||||_tj|||_ y)NTrKrr)r[r2r) rrr~rrirMochiAttentionPoolr<rr caption_proj)rrrrrjr[rs r.rz.MochiCombinedTimestepCaptionEmbedding.__init__s\ "PTkno!2~^k!l( 3~Zg  IIn6KLr0rrencoder_attention_maskrc|j|}|j|j|}|j||}|j |}||z} | |fSr)rrrr<r^) rrrr_rrtime_embrr^rs r.rz-MochiCombinedTimestepCaptionEmbedding.forwardsgNN8, ))),,\,*JK![[)>@VW(()>? "44 \))r0)rrr) rrrrHrr!rrrrrrrs@r.rZrZs "#$ MM #M M  M ! M M,/3 *""* %||*!& * u{{+ *r0rZc4eZdZddededeffd ZdZxZS)TextTimeEmbedding encoder_dimrjrct|tj||_t |||_tj|||_tj||_ yr) rrrrnorm1AttentionPoolingpoolrrnorm2)rrdrjrrs r.rzTextTimeEmbedding.__init__ sN \\+. $Y < IIk>: \\.1 r0c|j|}|j|}|j|}|j|}|Sr)rfrhrri)rrs r.rzTextTimeEmbedding.forward's@ =1  -0  -0  =1 r0)@r5rs@r.rcrcs"2C222r0rccfeZdZddededeffd Zdej dej fdZxZS) TextImageTimeEmbeddingrrrjct|tj|||_tj ||_tj|||_yr)rrrrrr text_norm image_proj)rrrrjrs r.rzTextImageTimeEmbedding.__init__0sB >>Bn5))O^Dr0rrcr|j|}|j|}|j|}||zSr)rrorp)rrrtime_text_embedstime_image_embedss r.rzTextImageTimeEmbedding.forward6s>>>+6>>*:;!OOL9 #333r0)rrr9r rs@r.rmrm/s=EsE3E^aE 45<<4u||4r0rmcJeZdZddedeffd Zdej fdZxZS)ImageTimeEmbeddingrrjct|tj|||_tj ||_yr)rrrrrpr image_normrrrjrs r.rzImageTimeEmbedding.__init__Bs0 ))O^D,,~6r0rcJ|j|}|j|}|Sr)rprw)rrrss r.rzImageTimeEmbedding.forwardGs' OOL9 OO,=>  r0rr9r rs@r.ruruAs&7737 !ELL!r0rucbeZdZddedeffd Zdej dej fdZxZS)ImageHintTimeEmbeddingrrjcNt|tj|||_tj ||_tjtjddddtjtjddddtjtjdddddtjtjddddtjtjdddddtjtjddddtjtjdd dddtjtjd d dd|_ y) Nr<rr )paddingrr)r~r`rr:) rrrrrprrwrLrSiLUinput_hint_blockrxs r.rzImageHintTimeEmbedding.__init__Os ))O^D,,~6 " IIaQ * GGI IIb"a + GGI IIb"a1 5 GGI IIb"a + GGI IIb"a1 5 GGI IIb"a + GGI IIb#q!A 6 GGI IIc1a +! r0rhintcp|j|}|j|}|j|}||fSr)rprwr)rrrrss r.rzImageHintTimeEmbedding.forwardes< OOL9 OO,=>$$T* $&&r0rzr rs@r.r|r|Ns1  3 ,'ELL' 'r0r|c&eZdZdfd ZdZxZS)rgct|||_tjt j d||dzz |_tj|||j|_ tj|||j|_ tj|||j|_ ||_ ||jz|_ y)Nr rr_)rrrrrr!rrrrrrr dim_per_head)rrr2rrs r.rzAttentionPooling.__init__ps  $&LLQ 1JYX[^1[$\!ii 9DJJG ii 9DJJG ii 9DJJG "%7r0c |j\ }} fd}|jddjj|jz}t j ||gd}|j|}|j|}|j|}dtjtjjz } t jd|| z|| z} t j| jdj!| j} t jd| |} | j# ddj%dd } | ddd ddfS) Nc|jdjj}|jdd}|j jzdj}|jdd}|S)Nrr r)rrrrrK)rbsrs r.rz'AttentionPooling.forward..shape}sgr2t~~t/@/@AA Aq!A "t~~-r43D3DEA Aq!AHr0r Trrz bct,bcs->btsrz bts,bcs->bctrr)rr2rrrr!r&rrrrsqrtrrsoftmaxr%rFrKr) rrlengthrr class_tokenqkvrrars ` @r.rzAttentionPooling.forwardzsAFFHFE ffDf1D4M4M4P4PQRQXQX4YY II{A&A . $++k* + $++a. ! $++a. !DIIdii(9(9:;;na%iUCv||~26;;FLLI LL 3 IIb"a * *1a 0Aqzr0rrrs@r.rgrgms 8"r0rgc eZdZ d dededeeddffd Zeddd ejd ejdejfd Z d ejd ejdejfd Z xZ S)r]Nr[r2rrct||xs||_||_t j |d|z|_t j |||_t j ||j|_y)Nr) rrrr[rrto_kvto_qto_out)rr[r2rrs r.rzMochiAttentionPool.__init__sb $1 #6 YYy!i-8 IIi3 ii 4??; r0Frrrc^|jd|jdk(sJ|jd|jdk(sJ|dddddfj|j}||jddj dz }||zjd|}|S)a6 Pool tokens in x using mask. NOTE: We assume x does not require gradients. Args: x: (B, L, D) tensor of tokens. mask: (B, L) boolean tensor indicating which tokens are not padding. Returns: pooled: (B, D) tensor of pooled tokens. r rNr_Tr)min)rrrrQclamp)rrrpooleds r. pool_tokenszMochiAttentionPool.pool_tokenssvvayDIIaL(((vvayDIIaL(((Aq$J"""1dhh1dh399a9@@d(Aw7 r0c|jd}|ddddddfj}tj|dd}|j ||d}t j ||gd}|j|}|j|ddd f}||jz}|jdd|j|f}|jdd }|jd\} } |jd|j|f}|jd}tj|| | |d }|jdj!dd}|j#|}|S) aP Args: x (`torch.Tensor`): Tensor of shape `(B, S, D)` of input tokens. mask (`torch.Tensor`): Boolean ensor of shape `(B, S)` indicating which tokens are not padding. Returns: `torch.Tensor`: `(B, D)` tensor of pooled tokens. rN)r rT)r"rr rrr<rK) attn_maskr,)rrr[r*rr!r&rrr[ unflattenrrPrscaled_dot_product_attentionr4rr) rrrDrx_poolkvrhead_dimrrs r.rzMochiAttentionPool.forwardsV FF1ID$)*//1 EE)V48 !!!T4!8 IIvqkq )ZZ] IIa1g 000 \\!a!9!98D E \\!Q yy|1 KKD44h? @ KKN  * *1aiSV W IIaL A & KKNr0r) rrrrHrr staticmethodr!rr BoolTensorrrrs@r.r]r]s %) <  < <SM <  <DIu||5<<ell(**U-=-=*%,,*r0r]c|jdd\}}dtj||z z}|dj|j|j }||j dz}tj|j|jfd}|jdd d d dj|||dzd z}|S) z Args: embed_dim: int box: a 3-D tensor [B x N x 4] representing the bounding boxes for GLIGEN pipeline Returns: [B x N x embed_dim] tensor of positional embeddings Nrd)NNNr=rrrr r<r:) rr!r"rrrrrJr'r(rrK)r2boxr num_boxesr-s r.#get_fourier_embeds_from_boundingboxrs IIbqMJ %,,y)I5 6C   " "#**CII " FC  b! !C ++swwy#'')," 5C ++aAq! $ , ,ZIPQMTUDU VC Jr0c2eZdZdfd Z ddZxZS)GLIGENTextBoundingboxProjectionc 8t|||_||_||_|dzdz|_t |tr|d}|dk(rtjtj|j|j zdtjtjddtjtjd||_ tjjtj|jg|_n|dk(rtjtj|j|j zdtjtjddtjtjd||_tjtj|j|j zdtjtjddtjtjd||_tjjtj|jg|_tjjtj|jg|_tjjtj|j g|_y)Nrr:r text-onlyz text-image)rr positive_lenrlfourier_embedder_dim position_dimrGtuplerrLrrlinearsr!rrnnull_positive_feature linears_text linears_imagenull_text_featurenull_image_featurenull_position_feature)rrrl feature_type fourier_freqsrs r.rz(GLIGENTextBoundingboxProjection.__init__s ( $1!)A-1 gu %ajG ; &== $++d.?.??E  #s#  #w' DL*/););EKKIZIZH[<\)]D & \ ) "  $++d.?.??E  #s#  #w' !D "$ $++d.?.??E  #s#  #w' "D &+XX%7%7 TEVEVDW8X%YD "&+hh&8&8dFWFWEX9Y&ZD #%*XX%7%7 TEVEVDW8X%Y"r0c|jd}t|j|}|jj ddd} ||zd|z | zz}|U|j j ddd} ||zd|z | zz}|j tj||gd} | S|jd}|jd}|jj ddd} |jj ddd} ||zd|z | zz}||zd|z | zz}|jtj||gd}|jtj||gd}tj||gd} | S)Nrr r) rrrrrrrr!r&rrrr)rboxesmaskspositive_embeddings phrases_masks image_masksphrases_embeddingsimage_embeddingsxyxy_embedding xyxy_null positive_nullobjs text_null image_null objs_text objs_images r.rz'GLIGENTextBoundingboxProjection.forward0s#=T=V=VX]^..33Aq"= (%/1u9 2II  * 66;;Aq"EM#6"=Um@["[ << +>*OUW XYD& *33B7M%//3K..33Aq"=I0055aB?J"4m!Cq=GX\eFe!e /+=[T^@^^ ))%))5G4X^`*abI++EII7G6X^`,abJ99i4!R>R>TUXXYef #99:JKSST^`bc (599nFV5W]^+__L)Lr0F)rrrrrrrrrs@r.rrcs it i r0rc*eZdZdZdfd ZdZxZS)rz Projects caption embeddings. Also handles dropout for classifier-free guidance. Adapted from https://github.com/PixArt-alpha/PixArt-alpha/blob/master/diffusion/model/nets/PixArt_blocks.py cdt|||}tj||d|_|dk(rtj d|_nB|dk(rtj|_n#|dk(rt|_ntd|tj||d|_ y) NTr gelu_tanhtanh) approximater{r:zUnknown activation function: ) rrrrrpGELUact_1rr rFrs)rrrrrkrs r.rz"PixArtAlphaTextProjection.__init__s   &L k Z^_ [ V4DJ v DJ { "!DJ