bi@ddlmZddlmZmZddlZddlZddlm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZmZmZmZeGd d e ZeGd de ZGdde j2ZGdde j2ZGdde j2ZGdde j2ZGdde j2ZGdde j2ZGdde Z!Gdde Z"Gdd e j2Z#Gd!d"e j2Z$y)#) dataclass)OptionalTupleN) BaseOutput) randn_tensor)get_activation) SpatialNorm)AutoencoderTinyBlockUNetMidBlock2Dget_down_block get_up_blockc0eZdZUdZej ed<y) EncoderOutputz Output of encoding method. Args: latent (`torch.Tensor` of shape `(batch_size, num_channels, latent_height, latent_width)`): The encoded latent. latentN)__name__ __module__ __qualname____doc__torchTensor__annotations__\/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/models/autoencoders/vae.pyrr!s LLrrcXeZdZUdZej ed<dZeejed<y) DecoderOutputz Output of decoding method. Args: sample (`torch.Tensor` of shape `(batch_size, num_channels, height, width)`): The decoded output sample from the last layer of the model. sampleN commit_loss) rrrrrrrr r FloatTensorrrrrr.s( LL/3K%++,3rrceZdZdZ ddededeedfdeedfdeded ed effd Zd e jd e jfdZ xZ S)Encodera The `Encoder` layer of a variational autoencoder that encodes its input into a latent representation. Args: in_channels (`int`, *optional*, defaults to 3): The number of input channels. out_channels (`int`, *optional*, defaults to 3): The number of output channels. down_block_types (`Tuple[str, ...]`, *optional*, defaults to `("DownEncoderBlock2D",)`): The types of down blocks to use. See `~diffusers.models.unet_2d_blocks.get_down_block` for available options. block_out_channels (`Tuple[int, ...]`, *optional*, defaults to `(64,)`): The number of output channels for each block. layers_per_block (`int`, *optional*, defaults to 2): The number of layers per block. norm_num_groups (`int`, *optional*, defaults to 32): The number of groups for normalization. act_fn (`str`, *optional*, defaults to `"silu"`): The activation function to use. See `~diffusers.models.activations.get_activation` for available options. double_z (`bool`, *optional*, defaults to `True`): Whether to double the number of output channels for the last block. in_channels out_channelsdown_block_types.block_out_channelslayers_per_blocknorm_num_groupsact_fndouble_zc t|||_tj||dddd|_tj g|_|d} t|D]Y\} } | } || } | t|dz k(}t| |j| | | dd||| d }|jj|[t|dd|dd|d|d| |_ tj|d|d |_tj |_|rd |zn|}tj|d|dd |_d |_y)Nrr kernel_sizestridepaddingư>) num_layersr$r%add_downsample resnet_epsdownsample_padding resnet_act_fn resnet_groupsattention_head_dim temb_channelsdefault r$r5r7output_scale_factorresnet_time_scale_shiftr9r8r: add_attention num_channels num_groupsepsr r1F)super__init__r(nnConv2dconv_in ModuleList down_blocks enumeratelenrappendr mid_block GroupNorm conv_norm_outSiLUconv_actconv_outgradient_checkpointing)selfr$r%r&r'r(r)r*r+mid_block_add_attentionoutput_channelidown_block_type input_channelis_final_block down_blockconv_out_channels __class__s rrGzEncoder.__init__Tsw  0yy  q !   ==,,A."+,<"= 0 A*M/2N#&8"9A"==N'00)+#11#$$-#1" J    # #J /% 0*(*2. !$-1"5)1   \\7I"7MZiost  08A ,l "4R"8:KQXYZ &+#rrreturnc|j|}tjrL|jr@|jD]}|j ||}|j |j |}n*|jD] }||} |j |}|j|}|j|}|j|}|S)z*The forward method of the `Encoder` class.) rJris_grad_enabledrVrL_gradient_checkpointing_funcrPrRrTrU)rWrr^s rforwardzEncoder.forwardsf%  "t'B'B".. O :::vN O66t~~vNF#.. , #F+ ,^^F+F##F+v&v& r) rr)DownEncoderBlock2D@r siluTT) rrrrintrstrboolrGrrre __classcell__r`s@rr#r#<s2,C.3 !! $C,C,C, S/ C, "#s(O C,  C,C,C,C,Jellu||rr#ceZdZdZ ddededeedfdeedfdeded ed effd Z dd ejd e ejdejfdZ xZ S)Decodera The `Decoder` layer of a variational autoencoder that decodes its latent representation into an output sample. Args: in_channels (`int`, *optional*, defaults to 3): The number of input channels. out_channels (`int`, *optional*, defaults to 3): The number of output channels. up_block_types (`Tuple[str, ...]`, *optional*, defaults to `("UpDecoderBlock2D",)`): The types of up blocks to use. See `~diffusers.models.unet_2d_blocks.get_up_block` for available options. block_out_channels (`Tuple[int, ...]`, *optional*, defaults to `(64,)`): The number of output channels for each block. layers_per_block (`int`, *optional*, defaults to 2): The number of layers per block. norm_num_groups (`int`, *optional*, defaults to 32): The number of groups for normalization. act_fn (`str`, *optional*, defaults to `"silu"`): The activation function to use. See `~diffusers.models.activations.get_activation` for available options. norm_type (`str`, *optional*, defaults to `"group"`): The normalization type to use. Can be either `"group"` or `"spatial"`. r$r%up_block_types.r'r(r)r* norm_typec t|||_tj||dddd|_tj g|_|dk(r|nd} t|dd|d|dk(rdn||d|| | |_ tt|} | d } t|D]_\} }| }| | } | t|dz k(}t||jdz|| || d||| | | }|jj|| }a|dk(rt!|d | |_n tj$|d |d |_tj&|_tj|d |dd |_d|_y)Nr;rr-r.spatialr2groupr<r=r r3r$r%prev_output_channel add_upsampler5r7r8r9r:r?rArEF)rFrGr(rHrIrJrK up_blocksr rPlistreversedrMrNrrOr rRrQrSrTrUrV)rWr$r%rrr'r(r)r*rsrXr:reversed_block_out_channelsrYrZ up_block_typerxr]up_blockr`s rrGzDecoder.__init__s  0yy  r "   r*'0I'= 4 (*2. !1:g1EI91"5)'1  '+84F+G&H#4Q7 ). 9 1 A}"0 8;N#&8"9A"==N#0014/+$7!//$-#1+(1 H NN ! !( +"0 + 10  !!,-?-BM!RD !#;Ma;P]lrv!wD    "4Q"7qRST &+#rr latent_embedsrac|j|}tt|jj j }t jr_|jrS|j|j||}|j|}|jD]}|j|||}n=|j||}|j|}|jD] }|||} ||j|}n|j||}|j|}|j|}|S)z*The forward method of the `Decoder` class.)rJnextiterrz parametersdtyperrcrVrdrPtorRrTrU)rWrr upscale_dtypers rrezDecoder.forwards%f%T$..";";"=>?EE  "t'B'B66t~~v}]FYY}-F!NN \::8V][ \^^FM:FYY}-F!NN 9!&-8 9  ''/F'' >Fv&v& r) rrUpDecoderBlock2Drgr rirjrvTN rrrrrkrrlrGrrrrernros@rrqrqs0*?.3 !! $J,J,J,c3h J, "#s(O J,  J,J,J,J,^15# #  -#  #rrqcheZdZdZdededdffd ZdejdejfdZxZ S) UpSamplea& The `UpSample` layer of a variational autoencoder that upsamples its input. Args: in_channels (`int`, *optional*, defaults to 3): The number of input channels. out_channels (`int`, *optional*, defaults to 3): The number of output channels. r$r%raNc|t|||_||_t j ||ddd|_y)Nr r-r.)rFrGr$r%rHConvTranspose2ddeconv)rWr$r%r`s rrGzUpSample.__init__Is= &(((lPQZ[efg rxcRtj|}|j|}|S)z+The forward method of the `UpSample` class.)rrelurrWrs rrezUpSample.forwardSs! JJqM KKNr rrrrrkrGrrrernros@rrr>sHhhh  h%,,rrc zeZdZdZ d dededededdf fd Zd d ejdejfd ZxZ S) MaskConditionEncoderz) used in AsymmetricAutoencoderKL in_chout_chres_chr0raNc 4t |g}|dkDr6|dz}|dz}||kDr|}|dk(r|}|j||f|dz}|dkDr6g}|D]\}} |j| |j|ddg} |}tt |D]f} || } | dk(s| dk(r*| jt j || dddn)| jt j || ddd| }ht j| |_y)Nr-r r;rrr.r) rFrGrOrangerNrHrI Sequentiallayers)rWrrrr0channelsin_ch_r%_in_ch_out_chrlout_ch_r`s rrGzMaskConditionEncoder.__init___s3 qjq[FaZF{ OOVV, - aKFqj ' )OFG    ( )HRLO,s<() A"1oGAva biiQqZ[\] biiQqZ[\]F  mmV, rrci}tt|jD]O}|j|}||}||tt |j <t j|}Q|S)z7The forward method of the `MaskConditionEncoder` class.)rrNrrltupleshaperr)rWrmaskoutrlayers rrezMaskConditionEncoder.forwardses4;;'( AKKNEaA'(CE!''N# $ 1 A    r)irrros@rrrZsd #-#-#- #-  #-  #-JU\\rrceZdZdZ ddededeedfdeedfdeded ed effd Z dd ejd e ejde ejde ejdejf dZ xZ S)MaskConditionDecoderaThe `MaskConditionDecoder` should be used in combination with [`AsymmetricAutoencoderKL`] to enhance the model's decoder with a conditioner on the mask and masked image. Args: in_channels (`int`, *optional*, defaults to 3): The number of input channels. out_channels (`int`, *optional*, defaults to 3): The number of output channels. up_block_types (`Tuple[str, ...]`, *optional*, defaults to `("UpDecoderBlock2D",)`): The types of up blocks to use. See `~diffusers.models.unet_2d_blocks.get_up_block` for available options. block_out_channels (`Tuple[int, ...]`, *optional*, defaults to `(64,)`): The number of output channels for each block. layers_per_block (`int`, *optional*, defaults to 2): The number of layers per block. norm_num_groups (`int`, *optional*, defaults to 32): The number of groups for normalization. act_fn (`str`, *optional*, defaults to `"silu"`): The activation function to use. See `~diffusers.models.activations.get_activation` for available options. norm_type (`str`, *optional*, defaults to `"group"`): The normalization type to use. Can be either `"group"` or `"spatial"`. r$r%rr.r'r(r)r*rsc 8t|||_tj||dddd|_tj g|_|dk(r|nd} t|dd|d|dk(rdn||d|| |_ tt|} | d } t|D]_\} } | }| | } | t|dz k(}t| |jdz|| d| d||| | | }|jj|| }at!||d |d |_|dk(rt%|d | |_n tj(|d |d |_tj*|_tj|d |dd|_d|_y)Nr;rr-r.rur2rvr<)r$r5r7r>r?r9r8r:rrw)rrrrArEF)rFrGr(rHrIrJrKrzr rPr{r|rMrNrrOrcondition_encoderr rRrQrSrTrUrV)rWr$r%rrr'r(r)r*rsr:r}rYrZr~rxr]rr`s rrGzMaskConditionDecoder.__init__s  0yy  r "   r*'0I'= 4 (*2. !1:g1EI91"5)'  '+84F+G&H#4Q7 ). 9 1 A}"0 8;N#&8"9A"==N#0014/+$(!//$-#1+(1 H NN ! !( +"0 + 10"6%a(%b)"   !!,-?-BM!RD !#;Ma;P]lrv!wD    "4Q"7qRST &+#rzimagerrracj|}|j|}tt|jj j }t jr|jr|j|j||}|j|}|'|%d|z |z}|j|j||}|jD]w} |`|^tt|j} t j"j%||jddd} || z| d| z zz}|j| ||}y||||ztt|jd|z zz}n|j||}|j|}||d|z |z}|j||}|jD]m} |`|^tt|j} t j"j%||jddd} || z| d| z zz}| ||}o|/|-||ztt|jd|z zz}||j'|}n|j'||}|j)|}|j+|}|S)z7The forward method of the `MaskConditionDecoder` class.Nr-nearest)sizemode)rJrrrzrrrrcrVrdrPrrrlrrrH functional interpolaterRrTrU) rWrrrrrr masked_imageim_xrsample_mask_s rrezMaskConditionDecoder.forwardsf%T$..";";"=>?EE  "t'B'B66t~~v}]FYY}-F T%5 !DE1 88** !NN \$)9"3uV\\':#;Fv&v& r)rrrrgr rirjrv)NNNrros@rrrs0*?.3 !! O,O,O,c3h O, "#s(O O,  O,O,O,O,h)-'+04 ? <<? %?u||$ ?   - ?  ?rrc TeZdZdZ ddedededededef fd Zd e jd e jfd Z d e jd e jfd Z d e jd ee je jeffdZde jdeedfd e jfdZxZS)VectorQuantizerz Improved version over VectorQuantizer, can be used as a drop-in replacement. Mostly avoids costly matrix multiplications and allows for post-hoc remapping of indices. n_e vq_embed_dimbeta unknown_indexsane_index_shapelegacyc >t|||_||_||_||_t j|j|j|_|jjjjd|jz d|jz ||_ |j|jdtjt!j"|j||j$j&d|_||_|j*dk(r%|j(|_|j(dz|_t-d|jd|j(d |j*d ||_y||_||_y) Ng?usedrextrar-z Remapping z indices to z indices. Using z for unknown indices.)rFrGrrrrrH Embedding embeddingweightdatauniform_remapregister_bufferrtensornploadrrre_embedrprintr) rWrrrrrrrr`s rrGzVectorQuantizer.__init__BsL (  dhh0A0AB ""++D488OS488^L :: !  bggdjj6I)J K  IIOOA.DM!.D !!W,%)]]" $  1 TXXJl4==/B++,,AC !1 DM 0rindsrac|j}t|dkDsJ|j|dd}|jj |}|dddddf|dk(j }|j d}|jddk}|jdk(rMtjd|j||jj |j||<n|j||<|j|S) Nr-rr;)NN.r random)r)device) rrNreshaperrlongargmaxsumrrrandintrr)rWrishapermatchnewunknowns r remap_to_usedzVectorQuantizer.remap_to_usedgs6{Q||F1Ir*yy||D!aDj!T/%::@@Bll2))A,"    ) ==DMMG @R@RSVV^a^h^hViCL--CL{{6""rc|j}t|dkDsJ|j|dd}|jj |}|j |jjdkDrd|||jjdk\<t j|dddf|jddgzddfd|}|j|S)Nr-rr;)rrNrrrrrgather)rWrrrbacks r unmap_to_allzVectorQuantizer.unmap_to_allus6{Q||F1Ir*yy||D! ==499??1- -/0D++ ,||DqM$**Q-1#*=q*@A1dK||F##rrc|jddddj}|jd|j}t j t j ||jjd}|j|j|j}d}d}|jsa|jt j|j|z dzzt j||jz dzz}n`t j|j|z dz|jt j||jz dzzz}|||z jz}|jddddj}|jB|j|jdd}|j!|}|jdd}|j"r:|j|jd|jd|jd}|||||ffS)Nrr rr-r;dim)permute contiguousviewrrargmincdistrrrrrmeandetachrrrr)rWr z_flattenedmin_encoding_indicesz_q perplexity min_encodingslosss rrezVectorQuantizer.forwards IIaAq ! , , .ffR!2!23  %||EKK T^^EZEZ,[abcnn1277@  {{99uzz3::g4@?)rr) rrchunkrlogvarclamprexpstdvar zeros_likerr)rWrrs rrGz%DiagonalGaussianDistribution.__init__s$!&Z!B 4;kk$++ud; *99S4;;./99T[[)   "'"2"2 $//"8"8@U@U# DHtx rN generatorract|jj||jj|jj }|j|j |zz}|S)N)r rr)rrrrrrr)rWr rrs rrz#DiagonalGaussianDistribution.samplesR IIOO??))//''   II6) )rotherc|jrtjdgS|Wdtjtj|j d|j zdz |jz gdzSdtjtj|j |j z d|j z |j |j z zdz |jz |jzgdzS)Nrr rr-r rr)rrrrpowrrr)rWr s rklzDiagonalGaussianDistribution.kls   <<& &}UYYIIdii+dhh6rEs "" -(-  J     4J 4  4vbiivrFbiiFRryy822992jg299gTwbiiwt565p6?"))?DH"))Hr