bi1MddlmZddlmZmZmZmZddlZddlm cm Z ddlm Z ddl m Z mZddlmZddlmZdd lmZdd lmZd d lmZmZmZmZmZd d lmZd dlm Z ddl!m"Z"m#Z#m$Z$eGddeZ%Gddee Z&y)) dataclass)DictOptionalTupleUnionN)nn) ConfigMixinregister_to_config)ConsistencyDecoderScheduler) BaseOutput)apply_forward_hook) randn_tensor)ADDED_KV_ATTENTION_PROCESSORSCROSS_ATTENTION_PROCESSORSAttentionProcessorAttnAddedKVProcessor AttnProcessor) ModelMixin) UNet2DModel) DecoderOutputDiagonalGaussianDistributionEncoderceZdZUdZded<y)ConsistencyDecoderVAEOutputa2 Output of encoding method. Args: latent_dist (`DiagonalGaussianDistribution`): Encoded outputs of `Encoder` represented as the mean and logvar of `DiagonalGaussianDistribution`. `DiagonalGaussianDistribution` allows for sampling latents from the distribution. r latent_distN)__name__ __module__ __qualname____doc____annotations__p/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/models/autoencoders/consistency_decoder_vae.pyrr&s0/r%rc4teZdZdZdZe d7dedededede edfd e d e edfd ed ed edede de edfde edfdededededededededede edff0fd Z d8de fdZ dZ d Zd!Zed"eeeffd#Zd$eeeeefffd%Zd&Ze d8d'ej2d(e d"eee effd)Ze d9d*ej2d+eej<d(e d,ed"eee ej2ff d-Z d.ej2d/ej2d0ed"ej2fd1Z!d.ej2d/ej2d0ed"ej2fd2Z"d8d'ej2d(e d"eee ffd3Z# d:d4ej2d5e d(e d+eej<d"eee ej2ff d6Z$xZ%S);ConsistencyDecoderVAEaP The consistency decoder used with DALL-E 3. Examples: ```py >>> import torch >>> from diffusers import StableDiffusionPipeline, ConsistencyDecoderVAE >>> vae = ConsistencyDecoderVAE.from_pretrained("openai/consistency-decoder", torch_dtype=torch.float16) >>> pipe = StableDiffusionPipeline.from_pretrained( ... "stable-diffusion-v1-5/stable-diffusion-v1-5", vae=vae, torch_dtype=torch.float16 ... ).to("cuda") >>> image = pipe("horse", generator=torch.manual_seed(0)).images[0] >>> image ``` Fscaling_factorlatent_channels sample_sizeencoder_act_fnencoder_block_out_channels.encoder_double_zencoder_down_block_typesencoder_in_channelsencoder_layers_per_blockencoder_norm_num_groupsencoder_out_channelsdecoder_add_attentiondecoder_block_out_channelsdecoder_down_block_typesdecoder_downsample_paddingdecoder_in_channelsdecoder_layers_per_blockdecoder_norm_epsdecoder_norm_num_groupsdecoder_num_train_timestepsdecoder_out_channelsdecoder_resnet_time_scale_shiftdecoder_time_embedding_typedecoder_up_block_typesc|t|t|||||| | | |_t | | ||||||||||| |_t |_|j||jd|jdtjgddddddfd|jd tjgd dddddfdtjd |zd |zd |_d|_d|_|j"j$|_t)|j"j$t*t,fr|j"j$d n|j"j$}t/|d t1|j"j2d z zz |_d|_y)N)act_fnblock_out_channelsdouble_zdown_block_types in_channelslayers_per_blocknorm_num_groups out_channels) add_attentionrCrEdownsample_paddingrFrGnorm_epsrHnum_train_timestepsrIresnet_time_scale_shifttime_embedding_typeup_block_types)rCF) force_upcastmeans)gg:?gyD?glL?g N3^) persistentstds)g4?gn=?gr ^?gr` ?rrrg?)super__init__rencoderr decoder_unetr decoder_schedulerr register_buffertorchtensorrConv2d quant_conv use_slicing use_tilingconfigr+tile_sample_min_size isinstancelisttupleintlenrCtile_latent_min_sizetile_overlap_factor)selfr)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@ __class__s r&rVzConsistencyDecoderVAE.__init__IsV !9%5+53-   (/959+5%3 ;-$C ;1 "=!> 3MN U3   LLI J4QRTXZ^K^ _   ELL!OPQUWXZ^`dQderw  ))A$7_9LaP %)KK$;$;!$++11D%=A KK # #A &((  %( qSA_A_=`cd=d7e(f$g!#' r%r`c||_y)a Enable tiled VAE decoding. When this option is enabled, the VAE will split the input tensor into tiles to compute decoding and encoding in several steps. This is useful for saving a large amount of memory and to allow processing larger images. N)r`)rjr`s r& enable_tilingz#ConsistencyDecoderVAE.enable_tilings %r%c&|jdy)z Disable tiled VAE decoding. If `enable_tiling` was previously enabled, this method will go back to computing decoding in one step. FN)rmrjs r&disable_tilingz$ConsistencyDecoderVAE.disable_tilings 5!r%cd|_y)z Enable sliced VAE decoding. When this option is enabled, the VAE will split the input tensor in slices to compute decoding in several steps. This is useful to save some memory and allow larger batch sizes. TNr_ros r&enable_slicingz$ConsistencyDecoderVAE.enable_slicings  r%cd|_y)z Disable sliced VAE decoding. If `enable_slicing` was previously enabled, this method will go back to computing decoding in one step. FNrrros r&disable_slicingz%ConsistencyDecoderVAE.disable_slicings !r%returnci}dtdtjjdttt fffd |j D]\}}||||S)z Returns: `dict` of attention processors: A dictionary containing all attention processors used in the model with indexed by its weight name. namemodule processorsct|dr|j||d<|jD]\}}|d||||S)N get_processor .processor.)hasattrr|named_children)rxryrzsub_namechildfn_recursive_add_processorss r&rzJConsistencyDecoderVAE.attn_processors..fn_recursive_add_processorssdv/282F2F2H dV:./#)#8#8#: U%+tfAhZ,@%T U r%)strr[rModulerrr)rjrzrxryrs @r&attn_processorsz%ConsistencyDecoderVAE.attn_processorssm  c 588?? X\]`bt]tXu !//1 BLD& 'fj A Br% processorc Tt|jj}t|tr,t||k7rt dt|d|d|ddt dtjjffd |jD]\}}|||y) a4 Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.rxryct|drEt|ts|j|n#|j|j |d|j D]\}}|d|||y)N set_processorr}r~)rrcdictrpopr)rxryrrrfn_recursive_attn_processors r&rzMConsistencyDecoderVAE.set_attn_processor..fn_recursive_attn_processorsxv/!)T2((3(($z7J)KL#)#8#8#: T%+tfAhZ,@%S Tr%N) rgrkeysrcr ValueErrorrr[rrr)rjrcountrxryrs @r&set_attn_processorz(ConsistencyDecoderVAE.set_attn_processorsD((--/0 i &3y>U+BPQTU^Q_P`a005w6QRWQXXkm   Tc T588?? T!//1 ALD& 'fi @ Ar%c jtd|jjDr t}nmtd|jjDr t }n8t dt t|jj|j|y)ze Disables custom attention processors and sets the default attention implementation. c3@K|]}|jtvywN)rkr.0procs r& zCConsistencyDecoderVAE.set_default_attn_processor.. si4t~~!>>ic3@K|]}|jtvywr)rkrrs r&rzCConsistencyDecoderVAE.set_default_attn_processor..sh$#==hrzOCannot call `set_default_attn_processor` when attention processors are of type N) allrvaluesrrrnextiterr)rjrs r&set_default_attn_processorz0ConsistencyDecoderVAE.set_default_attn_processors i4K_K_KfKfKhi i,.I h$J^J^JeJeJgh h%IabfgklpmAmAmHmHmJhKcLbMN   *r%x return_dictc|jrK|jd|jkDs|jd|jkDr|j||S|jrU|jddkDrC|j dDcgc]}|j |}}tj|}n|j |}|j|}t|}|s|fSt|Scc}w)al Encode a batch of images into latents. Args: x (`torch.Tensor`): Input batch of images. return_dict (`bool`, *optional*, defaults to `True`): Whether to return a [`~models.autoencoders.consistency_decoder_vae.ConsistencyDecoderVAEOutput`] instead of a plain tuple. Returns: The latent representations of the encoded images. If `return_dict` is True, a [`~models.autoencoders.consistency_decoder_vae.ConsistencyDecoderVAEOutput`] is returned, otherwise a plain `tuple` is returned. )rrrr) r`shaperb tiled_encoder_splitrWr[catr^rr)rjrrx_sliceencoded_sliceshmoments posteriors r&encodezConsistencyDecoderVAE.encodes$ ?? d.G.G G177SU;Y]YrYrKr$$QK$@ @    QCD771:Ndll73NNN .)A QA//!$09 < *yAAOs C7z generatornum_inference_stepsc:||jjz|jz |jz }dt |jj dz z}t j|d|}|j\}}}} |jj||j|jjt|d|| f||j|jz} |jjD]} t!j"|jj%| | |gd} |j'| | j(d d d dd d d d f} |jj+| | | |j,}|} | }|s|fSt/| S) a Decodes the input latent vector `z` using the consistency decoder VAE model. Args: z (torch.Tensor): The input latent vector. generator (Optional[torch.Generator]): The random number generator. Default is None. return_dict (bool): Whether to return the output as a dictionary. Default is True. num_inference_steps (int): The number of inference steps. Default is 2. Returns: Union[DecoderOutput, Tuple[torch.Tensor]]: The decoded output. rrnearest)mode scale_factor)devicer )rdtyperdimNsample)rar)rRrTrgrCF interpolaterrY set_timestepsrinit_noise_sigmarr timestepsr[concatscale_model_inputrXrstep prev_sampler)rjrrrrr batch_size_heightwidthx_tt model_input model_outputrx_0s r&decodezConsistencyDecoderVAE.decode:sy*++ +djj 8DII ES!?!?@1DE MM!), G'(ww$ Avu ,,-@,U$$55 FE *iqwwWXW_W_9  ''11 A,,(>(>(P(PQTVW(XZ['\bcdK,,[!<CCArr1aKPL0055lAsIVbbKC   6MC((r%ab blend_extentct|jd|jd|}t|D]A}|dddd| |zddfd||z z z|dddd|ddf||z zz|dddd|ddf<C|S)Nrrminrrange)rjrrrys r&blend_vzConsistencyDecoderVAE.blend_vjs1771:qwwqz<@ |$ xAa\MA$5q89Q\AQ=QRUVWXZ[]^`aWaUbfgjvfvUwwAaAqjM xr%ct|jd|jd|}t|D]A}|dddddd| |zfd||z z z|dddddd|f||z zz|dddddd|f<C|S)Nr rr)rjrrrrs r&blend_hzConsistencyDecoderVAE.blend_hqs1771:qwwqz<@ |$ xAaA }q'889Q\AQ=QRUVWXZ[]^`aWaUbfgjvfvUwwAaAqjM xr%c t|jd|jz z}t|j|jz}|j|z }g}t d|j d|D]}g}t d|j d|D]`} |dddd|||jz| | |jzf} |j | } |j| } |j| b|j|g} t|D]\}}g} t|D]d\} } |dkDr|j||dz | | |} | dkDr|j|| dz | |} | j| ddddd|d|ff| jtj| dtj| d} t| }|s|fSt|S)aEncode a batch of images using a tiled encoder. When this option is enabled, the VAE will split the input tensor into tiles to compute encoding in several steps. This is useful to keep memory use constant regardless of image size. The end result of tiled encoding is different from non-tiled encoding because each tile uses a different encoder. To avoid tiling artifacts, the tiles overlap and are blended together to form a smooth output. You may still see tile-sized changes in the output, but they should be much less noticeable. Args: x (`torch.Tensor`): Input batch of images. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.autoencoders.consistency_decoder_vae.ConsistencyDecoderVAEOutput`] instead of a plain tuple. Returns: [`~models.autoencoders.consistency_decoder_vae.ConsistencyDecoderVAEOutput`] or `tuple`: If return_dict is True, a [`~models.autoencoders.consistency_decoder_vae.ConsistencyDecoderVAEOutput`] is returned, otherwise a plain `tuple` is returned. rrrr Nrr)rfrbrirhrrrWr^append enumeraterrr[rrr)rjrr overlap_sizer row_limitrowsirowjtile result_rows result_rowrrs r&rz"ConsistencyDecoderVAE.tiled_encodews(444D F4q5<<QU AlKDq5<<AE D,GD!!$q!ZiZ)'C"DE F   uyy; < =))KQ/09 < *yAAr%rsample_posteriorc|}|j|j}|r|j|}n|j}|j ||j}|s|fSt |S)a Args: sample (`torch.Tensor`): Input sample. sample_posterior (`bool`, *optional*, defaults to `False`): Whether to sample from the posterior. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`DecoderOutput`] instead of a plain tuple. generator (`torch.Generator`, *optional*, defaults to `None`): Generator to use for sampling. Returns: [`DecoderOutput`] or `tuple`: If return_dict is True, a [`DecoderOutput`] is returned, otherwise a plain `tuple` is returned. )rr)rrrrrr) rjrrrrrrrdecs r&forwardzConsistencyDecoderVAE.forwardsk* KKN..   9 5A Akk!yk1886MC((r%)g{P? silu)rT)DownEncoderBlock2Drrrr rrrF)i@ir)ResnetDownsampleBlock2Drrrrr gh㈵>rr scale_shiftlearned)ResnetUpsampleBlock2Drrr)T)NTr)FTN)&rr r!r"_supports_group_offloadingr floatrfrrboolrVrmrprsrupropertyrrrrrrrr[Tensorrrrr Generatorrrrrrr __classcell__)rks@r&r(r(4s$"'!( $6J!%5 $%()')$%&+6L5 +,#$()"'')+/$%/<+43 G^(^(^( ^(  ^( %*#s(O ^(^(#(S/^(!^(#&^( "%!^(""#^($ $%^(&%*#s(O'^((#(S/)^(4%(5^(6!7^(8#&9^(: ;^(<"%=^(>&)?^(@"A^(B*-C^(D&)E^(F!&c3hG^(^(B%%" !c+=&=!>0 AE2Dd3PbKbFc2c,d AF +37 B B,0 B *E2N,OO P B BD04 #$ ,) <<,)EOO,,) ,) ! ,) }eELL11 2 ,),)^%,,cell%,,cell 5Bell5B5BOjlqOqIr5Bt"' /3 )  ) ) ) EOO, ) }eELL11 2 )r%r()' dataclassesrtypingrrrrr[torch.nn.functionalr functionalrconfiguration_utilsr r schedulersr utilsr utils.accelerate_utilsrutils.torch_utilsrattention_processorrrrrrmodeling_utilsr unets.unet_2drvaerrrrr(r$r%r&r sn"// B58-('EE  0* 0  0Z)J Z)r%