biKCddlmZmZddlZddlmcmZddlmZddlm Z ddl m Z m Z ddl mZdd lmZmZdd lmZmZmZmZmZdd lmZmZdd lmZdd lmZmZddl m!Z!m"Z"Gddee eZ#GddejHZ%GddejHZ&GddejHZ'GddejHZ(y))DictUnionN)nn checkpoint) ConfigMixinregister_to_config)PeftAdapterMixin)BasicTransformerBlockSkipFFTransformerBlock)ADDED_KV_ATTENTION_PROCESSORSCROSS_ATTENTION_PROCESSORSAttentionProcessorAttnAddedKVProcessor AttnProcessor)TimestepEmbeddingget_timestep_embedding) ModelMixin)GlobalResponseNormRMSNorm) Downsample2D Upsample2Dc.eZdZdZe ddededededededed ed ed ed ed ededededededededededef,fd Zd dZ e de e e ffdZdee e e e fffdZdZxZS)! UVit2DModelT hidden_sizeuse_biashidden_dropoutcond_embed_dimmicro_cond_encode_dimmicro_cond_embed_dimencoder_hidden_size vocab_size codebook_size in_channelsblock_out_channelsnum_res_blocks downsampleupsampleblock_num_headsnum_hidden_layersnum_attention_headsattention_dropoutintermediate_sizelayer_norm_epsln_elementwise_affine sample_sizect|tj||||_t ||||_t| | |||||_t||z|||_ t| | |||||||| d |_ t | |||_ tj| |||_tjt!|Dcgc]}t#||||z|||d||||||| c}|_t ||||_tj|| ||_t| | |||||||d| |_t-| | |||| |_d|_ycc}w)Nbias)sample_proj_biasFada_norm_continuous)dimr-attention_head_dimdropoutcross_attention_dimattention_bias norm_type-ada_norm_continous_conditioning_embedding_dimnorm_elementwise_affinenorm_eps ada_norm_bias ff_inner_dimff_biasattention_out_bias)r)r*)super__init__rLinear encoder_projrencoder_proj_layer_normUVit2DConvEmbedembedr cond_embed UVitBlock down_blockproject_to_hidden_normproject_to_hidden ModuleListranger transformer_layersproject_from_hidden_normproject_from_hiddenup_block ConvMlmLayer mlm_layergradient_checkpointing)selfrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2_ __class__s Y/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/models/unets/uvit_2d.pyrFzUVit2DModel.__init__*sD II&9;XV'.{NLa'b$$ +Z9NP^`h  , > 1;QY $     !        '..@.Rg&h#!#+={QY!Z"$--$01# "!&#(;'26I'I*(3#+3BM,A+"*!2$'/ # ,)0 ^Mb(c%#%99[:LS[#\ !     !       &  X7Ln^k ',#U s#E%c|j|}|j|}t|j|jj dd}|j |jddf}tj||gd}|j|j}|j|j|j}|j|}|j||||}|j\}} } } |jdd d dj || | z| }|j!|}|j#|}|j$D]8tj&r|j(rfd } n} | |||d |i }:|j+|}|j-|}|j || | | jdd dd }|j/||||}|j1|} | S)NTr)flip_sin_to_cosdownscale_freq_shiftr8)dtype)pooled_text_embencoder_hidden_statescross_attention_kwargsr rctg|SNr)argslayers r]layer_z#UVit2DModel.forward..layer_s%e3d33re)rfrgadded_cond_kwargs)rHrIrflattenconfigr!reshapeshapetorchcattordrLrKrNpermuterOrPrSis_grad_enabledrYrTrUrVrX)rZ input_idsrfre micro_condsrgmicro_cond_embeds hidden_states batch_sizechannelsheightwidthrllogitsrks @r]forwardzUVit2DModel.forwards $ 1 12G H $ < <=R S2    !4;;#D#DVZqr .55yq7I26NO))_6G$HaP),,4::,>///:==>S>Y>YZ 9-  +"7#9 ( /<.A.A+ Hfe%--aAq9AA*fW\n^fg 33MB ..}= ,, E$$&4+F+F4"&;'=#4o"F M  55mD 00? %--j&%RZZ[\^_abdef  +"7#9 &  . rmreturnci}dtdtjjdttt fffd |j D]\}}||||S)z Returns: `dict` of attention processors: A dictionary containing all attention processors used in the model with indexed by its weight name. namemodule processorsct|dr|j||d<|jD]\}}|d||||S)N get_processor .processor.)hasattrrnamed_children)rrrsub_namechildfn_recursive_add_processorss r]rz@UVit2DModel.attn_processors..fn_recursive_add_processorssdv/282F2F2H dV:./#)#8#8#: U%+tfAhZ,@%T U rm)strrsrModulerrr)rZrrrrs @r]attn_processorszUVit2DModel.attn_processorssm  c 588?? X\]`bt]tXu !//1 BLD& 'fj A Brm processorc Tt|jj}t|tr,t||k7rt dt|d|d|ddt dtjjffd |jD]\}}|||y) a4 Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.rrct|drEt|ts|j|n#|j|j |d|j D]\}}|d|||y)N set_processorrr)r isinstancedictrpopr)rrrrrfn_recursive_attn_processors r]rzCUVit2DModel.set_attn_processor..fn_recursive_attn_processorsxv/!)T2((3(($z7J)KL#)#8#8#: T%+tfAhZ,@%S TrmN) lenrkeysrr ValueErrorrrsrrr)rZrcountrrrs @r]set_attn_processorzUVit2DModel.set_attn_processorsD((--/0 i &3y>U+BPQTU^Q_P`a005w6QRWQXXkm   Tc T588?? T!//1 ALD& 'fi @ Armc jtd|jjDr t}nmtd|jjDr t }n8t dt t|jj|j|y)ze Disables custom attention processors and sets the default attention implementation. c3@K|]}|jtvywri)r\r.0procs r] z9UVit2DModel.set_default_attn_processor..si4t~~!>>ic3@K|]}|jtvywri)r\rrs r]rz9UVit2DModel.set_default_attn_processor..sh$#==hrzOCannot call `set_default_attn_processor` when attention processors are of type N) allrvaluesrrrnextiterr)rZrs r]set_default_attn_processorz&UVit2DModel.set_default_attn_processors i4K_K_KfKfKhi i,.I h$J^J^JeJeJgh h%IabfgklpmAmAmHmHmJhKcLbMN   *rm)iFiri@ i rrrFF ri gư>T@ri)__name__ __module__ __qualname__ _supports_gradient_checkpointingr intboolfloatrFrpropertyrrrrrrr __classcell__r\s@r]rr's'+$  #!%($(#&!"% !!##%#&!% $&*?j,j, j,  j,j, #j,"j,!j,j,j,j,  !j,"#j,$%j,&'j,()j,,-j,.!/j,2!3j,67j,:;j,< $=j,>?j,j,X;zc+=&=!>0 AE2Dd3PbKbFc2c,d AF +rmrc$eZdZfdZdZxZS)rJct|tj|||_t ||||_tj||d||_yNrb) kernel_sizer5) rErFr Embedding embeddingsr layer_normConv2dconv)rZr&r'r$elementwise_affineepsr5r\s r]rFzUVit2DConvEmbed.__init__"sI ,,z;?!+s4FGIIk+=1SWX rmc|j|}|j|}|jdddd}|j|}|S)Nrrrbr )rrrvr)rZrxrs r]rzUVit2DConvEmbed.forward(sH__Y/ __Z0 ''1a3 YYz* rmrrrrFrrrs@r]rJrJ!sY rmrJc2eZdZdededeffd ZdZxZS)rMr(r)r*c t|| rt|ddddd||| |_nd|_t j t |D cgc]} t||||||c} |_t j t |D cgc]} t||||z||| ||| c} |_ | rt|ddddd|||d |_ yd|_ ycc} wcc} w) NTrConv2d_0r rms_norm)use_convpaddingrrr=rrr5)r<rDrF) use_conv_transposerrrr=rrr5 interpolate) rErFrr)rrQrR ConvNextBlock res_blocksrattention_blocksrr*)rZr}r(rrr1r0rr+r.r)r*ir[r\s r]rFzUVitBlock.__init__1s#  *$"#8 DO#DO--~. ")"    !# ~. '#/%#+'/  ! " &#'$"#8! DM!DMY  s CC"c|j|j|}t|j|jD]v\}}|||}|j\}}} } |j ||| | zj ddd}||||}|j dddj ||| | }x|j|j|}|S)Nrr rb)rfrg)r)ziprrrrviewrvr*) rZxrerfrg res_blockattention_blockr|r}r~rs r]rzUVitBlock.forwards ?? &"A*-doot?T?T*U M &I!_-A23'' /J&%z8Ve^<DDQ1MA)>WmA !Q"'' HfeLA M == $ a Arm)rrrrrrFrrrs@r]rMrM0s0L!L!L!L!\rmrMc(eZdZ dfd ZdZxZS)rct|tj||dd|||_t ||||_tj|t||z||_ tj|_ tt||z|_ tjt||z|||_tj||_tj||dz||_y)Nrrb)rrgroupsr5r4r )rErFrr depthwisernormrGrchannelwise_linear_1GELUchannelwise_actrchannelwise_normchannelwise_linear_2Dropoutchannelwise_dropoutcond_embeds_mapper) rZr}r0r1rrrres_ffn_factorr\s r]rFzConvNextBlock.__init__s     Hn6KL $&IIhH~eZdZdedededededef fd ZdZxZS) rWr'r&rr1r0r%ct|tj||d||_t ||||_tj||d||_yr)rErFrrconv1rrconv2)rZr'r&rr1r0r%r\s r]rFzConvMlmLayer.__init__sO YY1;AT\] !+~?TUYY{MqxX rmc|j|}|j|jddddjdddd}|j|}|S)Nrr rrb)rrrvr)rZr{rs r]rzConvMlmLayer.forwardsW =1  (=(=aAq(IJRRSTVWYZ\]^ M* rm) rrrrrrrFrrrs@r]rWrWsN Y Y Y Y $ Y  Y YrmrW))typingrrrstorch.nn.functionalr functionalrtorch.utils.checkpointrconfiguration_utilsr r loadersr attentionr rattention_processorrrrrrrrrmodeling_utilsr normalizationrrresnetrrrrrJrMrrWrmr]rs  -B'EC'7-w+*k+;w+t bii ` `F*BII*Z299rm