biMddlmZmZmZmZddlZddlmZddlmcm Z ddl Zddl m Z mZddlmZddlmZddlmZdd lmZdd lmZdd lmZd d lmZmZej<eZ dZ!GddejDZ#GddejHZ%GddejLZ'GddejHZ(GddejHZ)GddejHZ*GddejHZ+GddejHZ,GddejHZ-Gd d!ejHZ.Gd"d#ee eZ/y)$)ListOptionalTupleUnionN) ConfigMixinregister_to_config)FromOriginalModelMixin)logging)apply_forward_hook)get_activation)AutoencoderKLOutput) ModelMixin) DecoderOutputDiagonalGaussianDistributionceZdZdZ d dededeeeeeeffdeeeeeeffdeeeeeeffddf fd Zd fd ZxZ S) QwenImageCausalConv3dau A custom 3D causal convolution layer with feature caching support. This layer extends the standard Conv3D layer by ensuring causality in the time dimension and handling feature caching for efficient inference. Args: in_channels (int): Number of channels in the input image out_channels (int): Number of channels produced by the convolution kernel_size (int or tuple): Size of the convolving kernel stride (int or tuple, optional): Stride of the convolution. Default: 1 padding (int or tuple, optional): Zero-padding added to all three sides of the input. Default: 0 in_channels out_channels kernel_sizestridepaddingreturnNct|||||||jd|jd|jd|jdd|jdzdf|_d|_y)N)rrrrrr rrrrr)super__init__r_padding)selfrrrrr __class__s q/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/models/autoencoders/autoencoder_kl_qwenimage.pyrzQwenImageCausalConv3d.__init__:sx #%#  a$,,q/4<<?DLLYZO]^aeamamnoap]prst   c<t|j}|`|jddkDrN|j|j}t j ||gd}|dxx|j dzcc<tj||}t|)|S)Nrr dim) listr todevicetorchcatshapeFpadrforward)r!xcache_xrr"s r#r1zQwenImageCausalConv3d.forwardNst}}%  4==#3a#7jj*G 7A,A.A AJ'--* *J EE!W wq!!r$)rrN) __name__ __module__ __qualname____doc__intrrrr1 __classcell__r"s@r#rr+s &4545 !!!3c3m 445 ! c5c3//0 ! sE#sC-001 ! !(""r$rc @eZdZdZd dededededdf fd Zd ZxZS) QwenImageRMS_norma A custom RMS normalization layer. Args: dim (int): The number of dimensions to normalize over. channel_first (bool, optional): Whether the input tensor has channels as the first dimension. Default is True. images (bool, optional): Whether the input represents image data. Default is True. bias (bool, optional): Whether to include a learnable bias term. Default is False. r( channel_firstimagesbiasrNc.t||sdnd}|r|g|n|f}||_|dz|_t j t j||_|r.t j t j||_ yd|_ y)N)rrr)rrg?) rrr>scalenn Parameterr,onesgammazerosr@)r!r(r>r?r@broadcastable_dimsr.r"s r#rzQwenImageRMS_norm.__init__dsy .4Y&.;*)*#*#X \\%**U"34 8rCrGr@)r!r2s r#r1zQwenImageRMS_norm.forwardns>{{1(:(:1DtzzQTXT^T^^aeajajjjr$TTF) r5r6r7r8r9boolrr1r:r;s@r#r=r=Xs> FCFFTFX\FimFkr$r=c"eZdZdZfdZxZS)QwenImageUpsamplea Perform upsampling while ensuring the output tensor has the same data type as the input. Args: x (torch.Tensor): Input tensor to be upsampled. Returns: torch.Tensor: Upsampled tensor with the same data type as the input. c\t||jj|Sr4)rr1floattype_as)r!r2r"s r#r1zQwenImageUpsample.forward}s#wqwwy)11!44r$)r5r6r7r8r1r:r;s@r#rPrPrs55r$rPc>eZdZdZdededdffd ZddgfdZxZS) QwenImageResampleax A custom resampling module for 2D and 3D data. Args: dim (int): The number of input/output channels. mode (str): The resampling mode. Must be one of: - 'none': No resampling (identity operation). - 'upsample2d': 2D upsampling with nearest-exact interpolation and convolution. - 'upsample3d': 3D upsampling with nearest-exact interpolation, convolution, and causal 3D convolution. - 'downsample2d': 2D downsampling with zero-padding and convolution. - 'downsample3d': 3D downsampling with zero-padding, convolution, and causal 3D convolution. r(moderNc  t|||_||_|dk(rAt j t ddt j||dzdd|_y|d k(rXt j t ddt j||dzdd|_t||dzd d |_ y|d k(rFt j t jd t j||dd|_y|dk(r[t j t jd t j||dd|_t||d dd|_ yt j|_y)N upsample2d)@rYz nearest-exact) scale_factorrVr rrr upsample3d)rrr)rrr downsample2d)rrrr)r r )r downsample3d)r rrr)rr) rrr(rVrD SequentialrPConv2dresampler time_conv ZeroPad2dIdentity)r!r(rVr"s r#rzQwenImageResample.__init__s9   < MM!zP #saxA6DM\ !MM!zP #saxA6DM33aT]^DN ^ #MM",,|*DbiiPSUXZ[djFklDM ^ #MM",,|*DbiiPSUXZ[djFklDM23YybklDNKKMDMr$rc |j\}}}}}|jdk(r||d} || d|| <|dxxdz cc<n|ddddt dddddfj} | jddkra|| \|| dk7rTt j || dddddddddfjdj| j| gd} | jddkrR|| M|| dk(rEt j t j| j| j| gd} || dk(r|j|}n|j||| }| || <|dxxdz cc<|j|d||||}t j|dddddddddddf|dddddddddddffd}|j|||dz||}|jd}|jddddd j||z|||}|j|}|j!|||jd|jd|jdjddddd }|jd k(r||d} || "|j|| <|dxxdz cc<|S|ddddddddddfj} |jt j || ddddddddddf|gd}| || <|dxxdz cc<|S) Nr\rReprr rKr'rr&r^)sizerVCACHE_Tcloner.r,r- unsqueezer*r+ zeros_likerbreshapestackpermuteraview) r!r2 feat_cachefeat_idxbcthwidxr3s r#r1zQwenImageResample.forwards5 1aA 99 $%qkc?*&+JsOQK1$K1whiA 56<<>G}}Q'!+ 30KPZ[^P_chPh"'))'_Q2q!^<FFqILLW^^\^eflm#}}Q'!+ 30KPZ[^P_chPh"'))U-=-=g-F-I-I'..-Y[b,cij"k!#%/ NN1- NN1jo>&-JsOQK1$K !Q1a3A Qq!Q1a'7%8!Aq!Q1&RW X IIaLkk!nkk*t+Q1 bAkk!Q1%002))A2)&1a  * *1a 3 IIaL Aq ) 1 1*t2CXvW\ ] IIaL FF:tXvu = IIaAq! $8|r$)r5r6r7r8rr1r:r;s@r#rr"s+r$rc DeZdZdZd dedededeffd Zddgfd ZxZ S) QwenImageMidBlockz Middle block for QwenImageVAE encoder and decoder. Args: dim (int): Number of input/output channels. dropout (float): Dropout rate. non_linearity (str): Type of non-linearity to use. r(r}r~ num_layersc Vt|||_t||||g}g}t |D]9}|j t ||j t||||;tj||_ tj||_ d|_ y)NF) rrr(rzrangeappendrrD ModuleList attentionsresnetsgradient_checkpointing) r!r(r}r~rrr_r"s r#rzQwenImageMidBlock.__init__Ys *#sG]KL z" UA   5c: ; NN1#sG]S T U-- 3}}W- &+#r$Nrc|jd|||}t|j|jddD]\}}|||}||||}|SNrr)rzipr)r!r2rprqattnresnets r#r1zQwenImageMidBlock.forwardhsi DLLOAz8 4 ab1AB 0LD&Gq*h/A  0 r$)rBrrrr;s@r#rrOs8 ,C ,% ,c ,`c ,%)A3 r$rcPeZdZdZddgddggdddfd effd Zd d gfd ZxZS)QwenImageEncoder3dao A 3D encoder module. Args: dim (int): The base number of channels in the first layer. z_dim (int): The dimensionality of the latent space. dim_mult (list of int): Multipliers for the number of channels in each block. num_res_blocks (int): Number of residual blocks in each block. attn_scales (list of float): Scales at which to apply attention mechanisms. temperal_downsample (list of bool): Whether to downsample temporally in each block. dropout (float): Dropout rate for the dropout layers. non_linearity (str): Type of non-linearity to use. r&rr r&r&r rMrBrr~c Lt|||_||_||_||_||_||_t||_ dg|zD cgc]} || z } } d} td| ddd|_ tjg|_tt!| dd| ddD]\} \} }t#|D]R}|jj%t'| ||| |vr$|jj%t)||} T| t+|dz k7s{|| rdnd}|jj%t-|| | d z} t/||d |_t3|d |_t||dd|_d |_ycc} w)Nr?rrr[rKr^r]rVrYrFr)rrr(z_dimdim_multnum_res_blocks attn_scalestemperal_downsamplerrrconv_inrDr down_blocks enumeraterrrrzrlenrUr mid_blockr=norm_outconv_outr)r!r(rrrrrr}r~udimsrCir{r|rrVr"s r#rzQwenImageEncoder3d.__init__s    ,&#6 *=9#$x0Aa00-QQAF ==,$-c$s)T!"X.F$G  A >* !  ''(>vwPW(XYK'$$++,CG,LM   !CMA%%)w]-[,\]DO&+#r$rc|jD]}| ||||}||}|j0||jd|||}|S|jd|}|S)aI Forward pass through the upsampling block. Args: x (torch.Tensor): Input tensor feat_cache (list, optional): Feature cache for causal convolutions feat_idx (list, optional): Feature index for cache management Returns: torch.Tensor: Output tensor r)rr)r!r2rprqrs r#r1zQwenImageUpBlock.forward sll F%1j(31I   ?? &%&DOOA&q*h?'DOOA&q)r$)rBNr) r5r6r7r8r9rRrrxrr1r:r;s@r#rrsd "'+#,,, ,  ,  } ,,<%)A3r$rcPeZdZdZddgddggdddfd effd Zd d gfd ZxZS)QwenImageDecoder3dak A 3D decoder module. Args: dim (int): The base number of channels in the first layer. z_dim (int): The dimensionality of the latent space. dim_mult (list of int): Multipliers for the number of channels in each block. num_res_blocks (int): Number of residual blocks in each block. attn_scales (list of float): Scales at which to apply attention mechanisms. temperal_upsample (list of bool): Whether to upsample temporally in each block. dropout (float): Dropout rate for the dropout layers. non_linearity (str): Type of non-linearity to use. rr&rr FTTrBrr~c t|||_||_||_||_||_||_t||_ |dg|dddzD cgc]} || z } } ddt|dz zz } t|| ddd|_ t| d||d|_tj g|_t%t'| dd| ddD]b\} \} }| dkDr| dz} d}| t|dz k7r || rd nd }t)| ||||| }|j"j+||^| d z} dt-d |_t|ddd|_d |_ycc} w)NrKrr rrrr[rr\rX)r{r|rr}rr~rYFr)rrr(rrrrtemperal_upsamplerrrrrrrrDr up_blocksrrrrr=rrr)r!r(rrrrrr}r~rrrCrr{r|rup_blockr"s r#rzQwenImageDecoder3d.__init__5s    ,&!2*=9#+2,(4R4.!@AAaAAaCMA-..-UDGQJ +47G]WXYr*$-c$s)T!"X.F$G  A 1u1!MCMA%%0A!0D , (-++ H NN ! !( +( / 4*'%@ -gq!QG &+#QBs E3Nrc ||d}|ddddt dddddfj}|jddkrY||Ttj||dddddddddfj dj |j|gd}|j|||}|||<|dxxdz cc<n|j|}|j|||}|jD] }||||}|j|}|j|}||d}|ddddt dddddfj}|jddkrY||Ttj||dddddddddfj dj |j|gd}|j|||}|||<|dxxdz cc<|S|j|}|Sr)rhrir.r,r-rjr*r+rrrrrr)r!r2rprqrwr3rs r#r1zQwenImageDecoder3d.forwardus  !1+C1whiA-.446G}}Q!# 3(C))Z_Q2q!^%D%N%Nq%Q%T%TU\UcUc%dfm$ntuv Q 30A%JsO QK1 K QA NN1j( 3 2HJ1A 2 MM!    a   !1+C1whiA-.446G}}Q!# 3(C))Z_Q2q!^%D%N%Nq%Q%T%TU\UcUc%dfm$ntuv aC1A%JsO QK1 K a Ar$rr;s@r#rr&s@  -#>,>,@%)A3#r$rcZeZdZdZdZeddgddggddgd gd f d ed ed eededee dee de dee dee ddffd Z d2de ede ede e de e ddf dZ d3dZd3dZd3dZdZd ej&fd!Ze d4d ej&d"e deeeeffd#Zd4d$ej&d"e fd%Zed4d$ej&d"e deeej&ffd&Zd'ej&d(ej&d)edej&fd*Zd'ej&d(ej&d)edej&fd+Zd ej&defd,Zd4d$ej&d"e deeej&ffd-Z d5d.ej&d/e d"e d0e ejBdeeej&ff d1Z"xZ#S)6AutoencoderKLQwenImagea A VAE model with KL loss for encoding videos into latents and decoding latent representations into videos. This model inherits from [`ModelMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). F`rr rrB)gy):gM Og^)gQ?gtVƿgZӼ?gBfjÿgU0*?gL F%u?gMg&†?gz6>׿g F%uȿg[ AcgMJ?gW2ıҿ)g_L@gNё\C?gQ@g?@g9#J{?g|a2U?gHPs@g0*@gJ{/L& @gJY8@g]C@g(?gK46?gS:?go_Ι@g-?base_dimrrrrrr} latents_mean latents_stdrNc t |||_||_|ddd|_t ||dz||||j||_t|dz|dzd|_t||d|_ t||||||j||_ dt|jz|_ d|_d|_d|_d|_d|_d|_|j*t)d|jj+Dnd|j *t)d |j j+Dndd |_y) NrKr rFc3<K|]}t|tywr4 isinstancer.0ms r# z2AutoencoderKLQwenImage.__init__..`A:a)>?`rc3<K|]}t|tywr4rrs r#rz2AutoencoderKLQwenImage.__init__..rr)decoderencoder)rrrrrrrr quant_convpost_quant_convrrrspatial_compression_ratio use_slicing use_tilingtile_sample_min_heighttile_sample_min_widthtile_sample_stride_heighttile_sample_stride_widthsummodules_cached_conv_counts) r!rrrrrrr}rrr"s r#rzAutoencoderKLQwenImage.__init__sT  #6 !4TrT!:) eai>;H`H`bi  0 519aH4UE1E) eX~{DDZDZ\c  *+c$2J2J.K)K&!  '*#%("*-&(+% ||'`I]I]I_``||'`I]I]I_`` $  r$rrrrcd|_|xs |j|_|xs |j|_|xs |j|_|xs |j|_y)aX Enable tiled VAE decoding. When this option is enabled, the VAE will split the input tensor into tiles to compute decoding and encoding in several steps. This is useful for saving a large amount of memory and to allow processing larger images. Args: tile_sample_min_height (`int`, *optional*): The minimum height required for a sample to be separated into tiles across the height dimension. tile_sample_min_width (`int`, *optional*): The minimum width required for a sample to be separated into tiles across the width dimension. tile_sample_stride_height (`int`, *optional*): The minimum amount of overlap between two consecutive vertical tiles. This is to ensure that there are no tiling artifacts produced across the height dimension. tile_sample_stride_width (`int`, *optional*): The stride between two consecutive horizontal tiles. This is to ensure that there are no tiling artifacts produced across the width dimension. TN)rrrrr)r!rrrrs r# enable_tilingz$AutoencoderKLQwenImage.enable_tilingsW0&<&[@[@[#%:%Xd>X>X")B)ddFdFd&(@(aDDaDa%r$cd|_y)z Disable tiled VAE decoding. If `enable_tiling` was previously enabled, this method will go back to computing decoding in one step. FN)rr!s r#disable_tilingz%AutoencoderKLQwenImage.disable_tilings  r$cd|_y)z Enable sliced VAE decoding. When this option is enabled, the VAE will split the input tensor in slices to compute decoding in several steps. This is useful to save some memory and allow larger batch sizes. TNrrs r#enable_slicingz%AutoencoderKLQwenImage.enable_slicings  r$cd|_y)z Disable sliced VAE decoding. If `enable_slicing` was previously enabled, this method will go back to computing decoding in one step. FNrrs r#disable_slicingz&AutoencoderKLQwenImage.disable_slicing s !r$cd}||j|_dg|_dg|jz|_||j|_dg|_dg|j z|_y)Nc`d}|jD]}t|ts|dz }|Sr)rrr)modelcountrs r# _count_conv3dz9AutoencoderKLQwenImage.clear_cache.._count_conv3ds7E]]_ a!67QJE Lr$r)r _conv_num _conv_idx _feat_mapr _enc_conv_num _enc_conv_idx _enc_feat_map)r!r s r# clear_cachez"AutoencoderKLQwenImage.clear_cachesd 't||4$..0*4<<8S"Vd&8&88r$r2c |j\}}}}}|jr/||jkDs||jkDr|j |S|j d|dz dzz}t |D]}dg|_|dk(r;|j|ddddddddddf|j|j}K|j|dddddd|dz zzdd|zzddddf|j|j} tj| gd}|j} |j | S)Nrr&rrprqr ) r.rrr tiled_encoderrrrrr,r-r) r!r2r num_framerriter_routout_encs r#_encodezAutoencoderKLQwenImage._encode$sG)*&1i ??(B(B BftOjOjFj$$Q' ' Y]q((u 0A"#D Avll1Q2A2q!^#4ASAS^b^p^plq||aAQU Oa!a%i7A=>#11!//$ iid Q/ 0ooc"  r$ return_dictc(|jrU|jddkDrC|jdDcgc]}|j|}}t j |}n|j|}t |}|s|fSt|Scc}w)a Encode a batch of images into latents. Args: x (`torch.Tensor`): Input batch of images. return_dict (`bool`, *optional*, defaults to `True`): Whether to return a [`~models.autoencoder_kl.AutoencoderKLOutput`] instead of a plain tuple. Returns: The latent representations of the encoded videos. If `return_dict` is True, a [`~models.autoencoder_kl.AutoencoderKLOutput`] is returned, otherwise a plain `tuple` is returned. rr) latent_dist)rr.splitrr,r-rr)r!r2rx_sliceencoded_slicesru posteriors r#encodezAutoencoderKLQwenImage.encode<s    QCD771:Ndll73NNN .)A QA03 < "y99OsBzc |j\}}}}}|j|jz}|j|jz}|jr||kDs||kDr|j ||S|j |j|} t|D]} dg|_ | dk(r>|j| dddd| | dzddddf|j|j} N|j| dddd| | dzddddf|j|j} tj | gd} tj dd} |j |s| fSt| S) Nrrrrr gr)minmaxsample)r.rrrr tiled_decoderrrr rr r,r-clampr) r!r#rrrrrtile_latent_min_heighttile_latent_min_widthr2rrrs r#_decodezAutoencoderKLQwenImage._decodeWsh)*&1i!%!\>\ \ ??(= =J`A`$$QK$@ @    #y! 0ASDNAvll1Q1q1u9a%:#;aeaoaolp||AaAAIq!&;$<bfbpbp|qiid Q/  0kk#4S1 6MC((r$c:|jr_|jddkDrM|jdDcgc]}|j|j}}t j |}n|j|j}|s|fSt|Scc}w)a Decode a batch of images. Args: z (`torch.Tensor`): Input batch of latent vectors. return_dict (`bool`, *optional*, defaults to `True`): Whether to return a [`~models.vae.DecoderOutput`] instead of a plain tuple. Returns: [`~models.vae.DecoderOutput`] or `tuple`: If return_dict is True, a [`~models.vae.DecoderOutput`] is returned, otherwise a plain `tuple` is returned. rrr()rr.rr.r)r,r-r)r!r#rz_slicedecoded_slicesdecodeds r#decodezAutoencoderKLQwenImage.decodeps    QJK''RS*Uwdll73::UNUii/Gll1o,,G: G,,Vs"Barr blend_extentc t|jd|jd|}t|D]J}|dddddd| |zddfd||z z z|dddddd|ddf||z zz|dddddd|ddf<L|S)Nrr&r.r)r!r4rrr5ys r#blend_vzAutoencoderKLQwenImage.blend_vs1772; \B |$ A Aq<-!*;Q!>?1q!>?1q >> !%!\>\ \$($B$BdFdFd$d!#'#@#@DDbDb#b -0II +.FF q&$"@"@A AC1eT%B%BC 3  ":>a"77 {+&A*+D&Av Arr1q43N3N/N+NPQTUX\XrXrTrPr!rs QU Oa!a%i7D$?$? ??D$>$> >> @  <<9K9KVZVhVh<iD??40DKK%& 599Tq12' 3( KK - .  o >FAsJ$S> h4q5<<QU AlKDq5<<AE D+FD!!$q!Q0J1J0JLeMeLe'e"fg h   uyy< = >ii +Aq!^m^]l],RS r$c |j\}}}}}||jz}||jz}|j|jz} |j|jz} |j|jz} |j |jz} |j|jz } |j|j z }g}t d|| D]}g}t d|| D]}|jg}t |D]q}dg|_|dddd||dz||| z||| zf}|j|}|j||j|j}|j|s|jtj|d|j||jg}t|D]\}}g}t|D]{\}}|dkDr|j!||dz ||| }|dkDr|j#||dz ||}|j|ddddddd|jd|j f}|jtj|dtj|dddddddd|d|f}|s|fSt%| S) a Decode a batch of images using a tiled decoder. Args: z (`torch.Tensor`): Input batch of latent vectors. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.vae.DecoderOutput`] instead of a plain tuple. Returns: [`~models.vae.DecoderOutput`] or `tuple`: If return_dict is True, a [`~models.vae.DecoderOutput`] is returned, otherwise a plain `tuple` is returned. rNrrr r'rKrr()r.rrrrrrrr rrr rr,r-rr:r<r)r!r#rrr>rr sample_height sample_widthr,r-rArBrCrDrErrFrGrrrIr2rJrKdecs r#r*z#AutoencoderKLQwenImage.tiled_decodes+,'''1j&%!?!?? t=== !%!\>\ \$($B$BdFdFd$d!#'#@#@DDbDb#b 22T5S5SS 0043P3PP q&";< AC1e%=> 3  "z*)A&'SDNQ1q1u9a!6L2L.LaRSVkRkNkklD//5D"ll4DNNUYUcUcldGKK( )  599Tq12 3 KK    o >FAsJ$S> t4q5<<QU AlKDq5<<AE D+FD!!$q!Q0P$2P2P0PRqTXTqTqRq'q"rs t   uyy< = >ii +Aq!^m^]l],RS6MC((r$r)sample_posterior generatorc|}|j|j}|r|j|}n|j}|j ||}|S)z Args: sample (`torch.Tensor`): Input sample. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`DecoderOutput`] instead of a plain tuple. )rQr%)r"rr)rVr3) r!r)rPrrQr2r!r#rOs r#r1zAutoencoderKLQwenImage.forwardsU KKN..   9 5A Akk!k5 r$)NNNN)rN)T)FTN)$r5r6r7r8 _supports_gradient_checkpointingr r9rrrRrNrrrrrrrr,Tensorrr rrrr"r.rr3r:r<rr* Generatorr1r:r;s@r#rrs(-$+#%*=%o$d8 8 8 * 8  8 %[ 8 "$Z8 8 5k8 %[8  8 8 x15/35948 b ( b (}b$,E? b #+5/ b  b<  !9 037::,0: "E*F$GG H::4))D)2- -4-5X]XdXdIdCe--0%,,cell%,,cell@ell@/B@D=)ell=)=)}^c^j^jOjIk=)D"' /3    EOO,  }ell* + r$r)0typingrrrrr,torch.nnrDtorch.nn.functional functionalr/torch.utils.checkpointconfiguration_utilsrr loadersr utilsr utils.accelerate_utilsr activationsrmodeling_outputsrmodeling_utilsrvaerr get_loggerr5loggerrhConv3drModuler=UpsamplerPrUrzrrrrrrr$r#ris*0/ B-8(2'<   H % *"BII*"Zk k4 5 5T TnGRYYGT*bii*Z$ $NhhVBryyBJrrjSZ6LSr$