bi*XddlZddlmZmZmZddlZddlmZddlmcm Z ddl m Z m Z ddlmZddlmZddlmZdd lmZdd lmZd d lmZmZej6eZGd dej<ZGddej@Z!Gddej@Z"Gddej@Z#Gddej@Z$Gddej@Z%Gddej@Z&Gddej@Z'Gddej@Z(Gdd ee Z)y)!N)OptionalTupleUnion) ConfigMixinregister_to_config)logging)apply_forward_hook)get_activation)AutoencoderKLOutput) ModelMixin) DecoderOutputDiagonalGaussianDistributionceZdZ ddededeeeedffdeeeedffdeeeedffdeeeedffded ed effd Zd Z d e jde jffd Z xZ S)EasyAnimateCausalConv3d in_channels out_channels kernel_size.stridepaddingdilationgroupsbias padding_modec t|tr|n|fdz}t|dk(s Jd|dt|tr|n|fdz}t|dk(s Jd|dt|tr|n|fdz}t|dk(s Jd|d|\} } } |\|_} }|\}}}| dz |z}|It j | dz |zd| z zdz }t j | dz |zd|z zdz }nt|t r|x}}ntsJ||_t j | dz |zd|z zdz |_ d|_ t|1|||||df||| y) Nrz#Kernel size must be a 3-tuple, got z instead.zStride must be a 3-tuple, got z Dilation must be a 3-tuple, got rr r) rrrrrrrrr) isinstancetuplelent_stridemathceilintNotImplementedErrortemporal_paddingtemporal_padding_origin prev_featuressuper__init__)selfrrrrrrrrrt_ksh_ksw_ksh_stridew_stride t_dilation h_dilation w_dilationt_padh_padw_pad __class__s n/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/models/autoencoders/autoencoder_kl_magvit.pyr*z EasyAnimateCausalConv3d.__init__$s&0 U%Ck+Z[I[ ;1$b(KK=Xa&bb$%fe46)a-6{aS#A&!SS)(E:8 a8}!Y%EhZy#YY!'dD,2) x-5* J Z' ?IIqJ6!h,G1LMEIIqJ6!h,G1LME  %# #EE& &&!&'+yy4!8z2IQQY\2Z^_1_'`$! #%#u%%  c|`d|_yNr(r+s r8_clear_conv_cachez)EasyAnimateCausalConv3d._clear_conv_cache^  !r9 hidden_statesreturnc "|j}|jtj|dddd|jdfd}|j |}|j |dddd|j dfj|_|jd}g}d}||jzdz|krat|)|dddd|||jzdzf}||jz }|j|||jzdz|kratj|dS|jdk(r>tj|jdddd|jdz df|gd}n#tj|j|gd}|j |}|j |dddd|j dfj|_|jd}g}d}||jzdz|krat|)|dddd|||jzdzf}||jz }|j|||jzdz|kratj|dS)Nr replicate)padmode)dtyper rdim)rFr(FrDr&tor>clonesizer)forwardr!appendtorchconcat)r+r@rF num_framesoutputsioutr7s r8rMzEasyAnimateCausalConv3d.forwardbs##    %EE1a!6!6: M *,,5,9M  " " $!.q!d6K6K5K5M/M!N!T!T!VD '++A.JGAd+++a/:=gomAq!a$BWBW>WZ[>[:[4[&\]T]]"s#d+++a/:=<<+ +}}! % ''10E0E0I.J.L(LM}]cd! !& d.@.@--PVW X ),,5,9M  " " $!.q!d6K6K5K5M/M!N!T!T!VD '++A.JGAd+++a/:=gomAq!a$BWBW>WZ[>[:[4[&\]T]]"s#d+++a/:=<<+ +r9)rrrrrTzeros)__name__ __module__ __qualname__r$rrboolstrr*r>rOTensorrM __classcell__r7s@r8rr#s 45.//001#8 8 8 3c3h/0 8 c5c?*+ 8 sE#s(O+, 8 U38_,-8 8 8 8 t"/,U\\/,ell/,/,r9rceZdZ d dedededededededeffd Zd ejd ejfd Z xZ S)EasyAnimateResidualBlock3Drr non_linearitynorm_num_groupsnorm_epsspatial_group_normdropoutoutput_scale_factorc t |||_tj|||d|_t ||_t||d|_ tj|||d|_ tj||_ t||d|_ ||k7r%tj||d|_||_ytj |_||_y)NT) num_groups num_channelsepsaffinerrr)r)r*renn GroupNormnorm1r nonlinearityrconv1norm2Dropoutrdconv2Conv3dshortcutIdentityrc) r+rrr`rarbrcrdrer7s r8r*z#EasyAnimateResidualBlock3D.__init__s #6 \\&$   +=9,[,TUV \\_<]enrs zz'* ,\T? rVrWrXr$rZfloatrYr*rOr[rMr\r]s@r8r_r_s $!#'%("5"5"5 "5  "5  "5!"5"5#"5HEU\\EellEr9r_c jeZdZd dedededeffd ZdejdejfdZxZ S) EasyAnimateDownsampler3DrrrrcLt|t||||d|_y)Nr)rrrrr)r)r*rconv)r+rrrrr7s r8r*z!EasyAnimateDownsampler3D.__init__s' +#,K`fpq  r9r@rAcTtj|d}|j|}|S)N)rrrr)rIrDrr+r@s r8rMz EasyAnimateDownsampler3D.forwards&m\:  -0 r9)rr r r ) rVrWrXr$rr*rOr[rMr\r]s@r8rrs@ C s  Z_ U\\ellr9rc zeZdZ d dededededef fd ZdZdejd ejfd Z xZ S) EasyAnimateUpsampler3Drrrtemporal_upsamplercc~t||xs|}||_||_t ||||_d|_y)N)rrr)r)r*rrcrrr()r+rrrrrcr7s r8r*zEasyAnimateUpsampler3D.__init__sG #2{ !2"4+#,K  "r9c|`d|_yr;r<r=s r8r>z(EasyAnimateUpsampler3D._clear_conv_cacher?r9r@rActj|dd}|j|}|jr;|j ||_|Stj|d|j sdnd}|S)Nrr r nearest) scale_factorrE)r rr trilinear)rI interpolaterrr(rcrs r8rMzEasyAnimateUpsampler3D.forwardss m)R[\  -0  ! !!!)%2" !" !!*,0,C,C! r9)rFT) rVrWrXr$rYr*r>rOr[rMr\r]s@r8rrsc "'#' """ "  " ! "&" U\\ ell r9rceZdZ ddedededededededed ed ed effd Zd ejdejfdZ xZ S)EasyAnimateDownBlock3Drr num_layersact_fnrarbrcrdreadd_downsampleadd_temporal_downsamplec t |tjg|_t |D]7} | dk(r|n|}|jj t|||||||| 9| r%| r#t||dd|_ d|_ d|_ y| r%| s#t||dd|_ d|_ d|_ yd|_ d|_ d|_ y) Nrrrr`rarbrcrdrerr)rrr rr) r)r*rl ModuleListconvsrangerNr_r downsamplerspatial_downsample_factortemporal_downsample_factor)r+rrrrrarbrcrdrerrrSr7s r8r*zEasyAnimateDownBlock3D.__init__s ]]2& z" A)*a+\K JJ  * +!-"($3%'9#(;    57 l`ajstD -.D *./D + $;7 l`ajstD -.D *./D +#D -.D *./D +r9r@rAcr|jD] }||} |j|j|}|Sr;)rrr+r@rs r8rMzEasyAnimateDownBlock3D.forward>sAJJ 0D /M 0    ' ,,];Mr9) rrrrTrrTTrr]s@r8rrs !#'%(#(,+0+0+0 +0  +0  +0+0!+0+0#+0+0"&+0ZU\\ellr9rceZdZ ddedededededededed ed ed effd Zd ejdejfdZ xZ S)EasyAnimateUpBlock3drrrrrarbrcrdre add_upsampleadd_temporal_upsamplec t |tjg|_t |D]7} | dk(r|n|}|jj t|||||||| 9| rt||| ||_ yd|_ y)Nrr)rrc) r)r*rlrrrrNr_r upsampler)r+rrrrrarbrcrdrerrrSr7s r8r*zEasyAnimateUpBlock3d.__init__Gs ]]2& z" A)*a+\K JJ  * +!-"($3%'9#(;    3"7#5 DN"DNr9r@rAcr|jD] }||} |j|j|}|Sr;)rrrs r8rMzEasyAnimateUpBlock3d.forwardqs=JJ 0D /M 0 >> % NN=9Mr9) rrrrFrrTTrr]s@r8rrFs !#(%(!&*("("(" ("  ("  ("("!("("#("(" $("TU\\ellr9rceZdZ d dedededededededeffd Zd ejd ejfd Z xZ S)EasyAnimateMidBlock3drrrrarbrcrdrec t |||nt|dzd}tjt ||||||||g|_t|dz D].} |j jt ||||||||0y)Nryrrr) r)r*minrlrr_rrrN) r+rrrrarbrcrdre_r7s r8r*zEasyAnimateMidBlock3d.__init__zs -<-H/cR]abRbdfNg]]* +!,"($3%'9#(;     zA~& A JJ  * +!,"($3%'9#(;   r9r@rAch|jd|}|jddD] }||} |S)Nrr)r)r+r@resnets r8rMzEasyAnimateMidBlock3d.forwards?% 1 m4 jjn 2F"=1M 2r9)rrrrTrrrr]s@r8rrys!#'%(*** *  *  *!**#*XU\\ellr9rceZdZdZdZdddgdddd dd f d ed ed eedfdeedfdededededeffd Z de jde jfdZ xZ S)EasyAnimateEncoderzp Causal encoder for 3D video-like data used in [EasyAnimate](https://huggingface.co/papers/2405.18991). TrSpatialDownBlock3DSpatialTemporalDownBlock3Drrrr rrFrrdown_block_types.block_out_channelslayers_per_blockrardouble_zrcc t|t||dd|_t j g|_|d} t|D]|\} } | } || } | t|dz k(}| dk(rt| | |||d| | d }n)| d k(rt| | |||d| | d  }ntd | |j j|~t|d ||| |ddd |_ | |_t j|d |d|_t#||_|rd|zn|}t|d |d|_d|_y)NrrrkrrrF) rrrrrarbrcrrrTUnknown up block type: rz)rrrrcrarbrdrerhrgrir )r)r*rconv_inrlr down_blocks enumerater r ValueErrorrNr mid_blockrcrm conv_norm_outr conv_actconv_outgradient_checkpointing)r+rrrrrrarrrcoutput_channelsrSdown_block_typeinput_channelsis_final_block down_blockconv_out_channelsr7s r8r*zEasyAnimateEncoder.__init__s" /{|j|}|jD]=}tjr|jr|j ||}6||}?|j |}|jro|jd}|jdddddjdd}|j|}|jd|dfjddddd}n|j|}|j|}|j|}|Srx)rrrOis_grad_enabledr_gradient_checkpointing_funcrrcrLr{r|rr}rr)r+r@rr~s r8rMzEasyAnimateEncoder.forwards ]3 ** :J$$&4+F+F $ A A*m \ *= 9  : }5  " "&++A.J)11!Q1a@HHANM ..}=M)33A B7GHPPQRTUWXZ[]^_M ..}=M m4  m4 r9rVrWrX__doc__ _supports_gradient_checkpointingr$rrZrYr*rOr[rMr\r]s@r8rrs(,$- /C !!#(R,R,R, S/ R,"#s(OR,R,R,R,R,!R,hU\\ellr9rceZdZdZdZdddgdddd d fd ed ed eedfdeedfdedededeffd Z de jde jfdZ xZ S)EasyAnimateDecoderzp Causal decoder for 3D video-like data used in [EasyAnimate](https://huggingface.co/papers/2405.18991). TrrSpatialUpBlock3DSpatialTemporalUpBlock3Drrrr rrFrrup_block_types.rrrarrcc t|t||dd|_t |d|||ddd|_t jg|_tt|} | d} t|D]\} } | } | | } | t|dz k(}| dk(rt| | |dz||d|| d }n,| d k(rt| | |dz||d|| d }ntd | |jj|||_t j"|d|d|_t'||_t|d|d|_d |_y)Nrzrrkrrr)rrrrarbrdrerF) rrrrrarbrcrrrTrr)r)r*rrrrrlr up_blockslistreversedrr rrrNrcrmrr rrr)r+rrrrrrarrcreversed_block_out_channelsrrS up_block_typerrup_blockr7s r8r*zEasyAnimateDecoder.__init__(s /{r)r+namemodules r8r>z%AutoencoderKLMagvit._clear_conv_cachesL ..0 +LD&&"9:((*&"89((*  +r9NrrrrrrrAc*d|_d|_d|_|xs |j|_|xs |j|_|xs |j |_|xs |j |_|xs |j|_|xs |j|_y)aX Enable tiled VAE decoding. When this option is enabled, the VAE will split the input tensor into tiles to compute decoding and encoding in several steps. This is useful for saving a large amount of memory and to allow processing larger images. Args: tile_sample_min_height (`int`, *optional*): The minimum height required for a sample to be separated into tiles across the height dimension. tile_sample_min_width (`int`, *optional*): The minimum width required for a sample to be separated into tiles across the width dimension. tile_sample_stride_height (`int`, *optional*): The minimum amount of overlap between two consecutive vertical tiles. This is to ensure that there are no tiling artifacts produced across the height dimension. tile_sample_stride_width (`int`, *optional*): The stride between two consecutive horizontal tiles. This is to ensure that there are no tiling artifacts produced across the width dimension. TN) rrrrrrrrr)r+rrrrrrs r8 enable_tilingz!AutoencoderKLMagvit.enable_tilings4&*#&*#&<&[@[@[#%:%Xd>X>X"*D*gHgHg')B)ddFdFd&(@(aDDaDa%-J-pdNpNp*r9cd|_y)z Disable tiled VAE decoding. If `enable_tiling` was previously enabled, this method will go back to computing decoding in one step. FN)rr=s r8disable_tilingz"AutoencoderKLMagvit.disable_tiling(s  r9cd|_y)z Enable sliced VAE decoding. When this option is enabled, the VAE will split the input tensor in slices to compute decoding in several steps. This is useful to save some memory and allow larger batch sizes. TNrr=s r8enable_slicingz"AutoencoderKLMagvit.enable_slicing/s  r9cd|_y)z Disable sliced VAE decoding. If `enable_slicing` was previously enabled, this method will go back to computing decoding in one step. FNr r=s r8disable_slicingz#AutoencoderKLMagvit.disable_slicing6s !r9x return_dictc D|jrK|jd|jkDs|jd|jkDr|j ||S|j |ddddddddddf}|g}t d|jd|jD]C}|j |dddd|||jzddddf}|j|Etj|d}|j|}|j|S)a Encode a batch of images into latents. Args: x (`torch.Tensor`): Input batch of images. return_dict (`bool`, *optional*, defaults to `True`): Whether to return a [`~models.autoencoder_kl.AutoencoderKLOutput`] instead of a plain tuple. Returns: The latent representations of the encoded images. If `return_dict` is True, a [`~models.autoencoder_kl.AutoencoderKLOutput`] is returned, otherwise a plain `tuple` is returned. rzrNrr rG) rshaperr tiled_encoderrrrNrOcatrr>)r+rr first_frameshrS next_framesmomentss r8_encodezAutoencoderKLMagvit._encode=s ?? d.I.I IQWWUW[[_[u[uMu$$QK$@ @||AaBQB1n$56 Nq!''!*d&G&GH "A,,qAq1t7X7X3X/XZ[]^)^'_`K HH[ ! " IIaQ //!$  r9c(|jrU|jddkDrC|jdDcgc]}|j|}}t j |}n|j|}t |}|s|fSt|Scc}w)a Encode a batch of images into latents. Args: x (`torch.Tensor`): Input batch of images. return_dict (`bool`, *optional*, defaults to `True`): Whether to return a [`~models.autoencoder_kl.AutoencoderKLOutput`] instead of a plain tuple. Returns: The latent representations of the encoded videos. If `return_dict` is True, a [`~models.autoencoder_kl.AutoencoderKLOutput`] is returned, otherwise a plain `tuple` is returned. rr) latent_dist)rrsplitrrOrrr )r+rrx_sliceencoded_slicesr posteriors r8encodezAutoencoderKLMagvit.encode[s    QCD771:Ndll73NNN .)A QA03 < "y99OsBzc |j\}}}}}|j|jz}|j|jz} |jr7|jd|kDs|jd| kDr|j ||S|j |}|j|ddddddddddf} | g} td|jd|jD]C} |j|dddd| | |jzddddf} | j| Etj| d} |s| fSt| S)Nrzrrrr rGsample)rrrrr tiled_decoderrrrrNrOrr)r+r#rr~rhrQheightwidthtile_latent_min_heighttile_latent_min_widthrdecrSrs r8_decodezAutoencoderKLMagvit._decodewsE>?gg; L*fe!%!\>\ \ ?? .D DPR VkHk$$QK$@ @   #||AaBQB1n$56 nq!''!*d&G&GH $A,,qAq1t7X7X3X/XZ[]^)^'_`K JJ{ # $ii#6MC((r9cZ|jr_|jddkDrM|jdDcgc]}|j|j}}t j |}n|j|j}|j|s|fSt|Scc}w)a Decode a batch of images. Args: z (`torch.Tensor`): Input batch of latent vectors. return_dict (`bool`, *optional*, defaults to `True`): Whether to return a [`~models.vae.DecoderOutput`] instead of a plain tuple. Returns: [`~models.vae.DecoderOutput`] or `tuple`: If return_dict is True, a [`~models.vae.DecoderOutput`] is returned, otherwise a plain `tuple` is returned. rrr%) rrrr-r&rOrr>r)r+r#rz_slicedecoded_slicesdecodeds r8decodezAutoencoderKLMagvit.decodes    QJK''RS*Uwdll73::UNUii/Gll1o,,G  : G,,Vs"B(ab blend_extentc t|jd|jd|}t|D]J}|dddddd| |zddfd||z z z|dddddd|ddf||z zz|dddddd|ddf<L|S)Nrrrrr)r+r3r4r5ys r8blend_vzAutoencoderKLMagvit.blend_vs1771:qwwqz<@ |$ A Aq<-!*;Q!>?1q!>?1qrr9r;)r+rrr~rhrQr(r) latent_height latent_widthr*r+tile_latent_stride_heighttile_latent_stride_width blend_height blend_widthrowsrSrowjtilertile_hkr result_rows result_rowrs r8rz AutoencoderKLMagvit.tiled_encodes>?gg; L*fe$"@"@@  > >> !%!\>\ \$($B$BdFdFd$d!#'#@#@DDbDb#b -0II +.FF q&$"@"@A AC1eT%B%BC !D7777D6666 8 $||DAqsAq,AB &q*d.O.OP/A"&,,tAq!a$BcBc>c:cefhi4i/j"kKMM+./yyQ/t,&&( 4 # !$ KK ) * o =FAsJ$S> P4q5<<QU AlKDq5<<AE D+FD!!$q!Q  'M"NO P   uyy; < =))KQ/1a-,0VWr9c |j\}}}}}||jz}||jz} |j|jz} |j|jz} |j|jz} |j |jz} |j|jz }|j|j z }g}t d|| D]}g}t d|| D]}|dddddd||| z||| zf}|j|}|j|ddddddddddf}|g}t d||jD]C}|j|dddd|||jzddddf}|j|Etj|d}|j|j||j|g}t|D]\}}g}t|D]{\}}|dkDr|j||dz |||}|dkDr|j!||dz ||}|j|ddddddd|jd|j f}|jtj|dtj|dddddddd|d| f}|s|fSt#|S)Nrrr rGryrr%)rrrrrrrrrrrNrOrr>rr9r;r)r+r#rr~rhrQr(r) sample_height sample_widthr*r+r?r@rArBrCrSrDrErFrtile_decrHrr1rIrJr,s r8r'z AutoencoderKLMagvit.tiled_decodes>?gg; L*fe!?!?? t=== !%!\>\ \$($B$BdFdFd$d!#'#@#@DDbDb#b 22T5S5SS 0043P3PP q&";< AC1e%=> $222111 3++D1 $||DArr1a,@A (>q*d.O.OP1A"&,,tAq!a$BcBc>c:cefhi4i/j"kKOOK01 ))H!4&&( 7#+ $, KK 1 2 o =FAsJ$S> t4q5<<QU AlKDq5<<AE D+FD!!$q!Q0P$2P2P0PRqTXTqTqRq'q"rs t   uyy; < =ii +Aq!^m^]l],RS6MC((r9r&sample_posterior generatorc|}|j|j}|r|j|}n|j}|j |j}|s|fSt |S)aa Args: sample (`torch.Tensor`): Input sample. sample_posterior (`bool`, *optional*, defaults to `False`): Whether to sample from the posterior. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`DecoderOutput`] instead of a plain tuple. )rPr%)r"rr&rEr2r) r+r&rOrrPrr!r#r,s r8rMzAutoencoderKLMagvit.forward,sf KKN..   9 5A Akk!n##6MC((r9)NNNNNN)rAN)T)FTN)$rVrWrXrrrr$rrZrrYr*r>rrr r rr rOr[rr rrr"rr-r2r9r;rr' GeneratorrMr\r]s@r8rrsi(,$!.B- + !"! &#'-T/T/T/ T/ "#s(O T/  S/ T/c3hT/$%T/&'T/()T/*+T/,!-T/T/l+15/34859489="q ( "q (}"q%-SM "q $,E? "q #+5/ "q(0"q "qH  !37,0 "E*F$GG H:37::,0: "E*F$GG H::6))D)E-Y^YeYeJeDf)4- -4-5X]XdXdIdCe--2%,,cell%,,cell2ell22I\2h;)ell;);)}^c^j^jOjIk;)@"' /3 ) )) ) EOO, ) }ell* + )r9r)*r"typingrrrrOtorch.nnrltorch.nn.functional functionalrIconfiguration_utilsrrutilsr utils.accelerate_utilsr activationsr modeling_outputsr modeling_utilsrvaerr get_loggerrVloggerrtrModuler_rrrrrrrrr9r8rbs  )) B8(2'<   H %n,biin,bCECEL ryy %RYY%P3RYY3l02990f1BII1hqqhvvrl)*kl)r9