biIddlmZmZmZddlZddlmcmZddlmZddl m Z m Z ddl m Z e rddlmZmZe j"eZdej(fd Zdej(fd Zdd efd ZGd dej j(ZGddej(ZGddej(ZGddej8ZGddej<Zy))OptionalTupleUnionN)nn) deprecatelogging)is_transformers_available) CLIPTextModelCLIPTextModelWithProjection text_encoderc&g}t|ttfrXt|jj j D]*\}}d|d}|j}|j||f,|Std|jj)Ntext_model.encoder.layers.z .self_attnz.do not know how to get attention modules for: ) isinstancer r enumerate text_modelencoderlayers self_attnappend ValueError __class____name__)r attn_modulesilayernamemods P/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/models/lora.pytext_encoder_attn_modulesr )sL,0K LM!,"9"9"A"A"H"HI -HAu/s*=D//C   s , - I,J`J`JiJiIjkllc&g}t|ttfrXt|jj j D]*\}}|j}d|d}|j||f,|Std|jj)Nrz.mlpz(do not know how to get mlp modules for: ) rr r rrrrmlprrrr)r mlp_modulesrrmlp_modrs rtext_encoder_mlp_modulesr&7sK,0K LM!,"9"9"A"A"H"HI 0HAuiiG/s$7D   g / 0 CLDZDZDcDcCdeffr! lora_scalect|D]d\}}t|jts!||j_||j _||j _||j_ft|D]B\}}t|jts!||j_||j_DyN) r rq_projPatchedLoraProjectionr'k_projv_projout_projr&fc1fc2)r r'_ attn_module mlp_modules radjust_lora_scale_text_encoderr4Es3LA9; k((*? @,6K   ),6K   ),6K   ).8K + 92,?3 : jnn&; <(2JNN %(2JNN %3r!cHeZdZd fd Zddddfd Zd dZdZd ZxZS) r+Ncbd}tdd|t | ddlm}||_|j j j}| |j j j}||j j|j j|||||_ ||_ y)NzsUse of `PatchedLoraProjection` is deprecated. Please switch to PEFT backend by installing PEFT: `pip install peft`.r+1.0.0r)LoRALinearLayer) network_alphadevicedtyperank) rsuper__init__ models.lorar8regular_linear_layerweightr:r; in_features out_featureslora_linear_layerr') selfr@r'r9r<r;deprecation_messager8r:rs rr>zPatchedLoraProjection.__init__TsT)74GH 1$8!**1188 =--44::E!0  % % 1 1  % % 2 2' " %r!F destinationprefix keep_varscz|j|jj||||dSt|||||dS)NrH)rDr@ state_dictr=)rErIrJrKargsrs rrMz PatchedLoraProjection.state_dictosL  ! ! )74,,77;v w!4[[deer!c 4|jy|jjjj|jjjj }}|jjjj }|jjjjj }|jjjjj }|jj0||jjz|jjz }||tj|dddf|dddfdzz}|r@tj|jjrt!d|d|j#|||jj_d|_|j%|_|j%|_||_yNraThis LoRA weight seems to be broken. Encountered NaN values when trying to fuse LoRA weights for .LoRA weights will not be fused.r:r;)rDr@rAdatar;r:floatupdownr9r<torchbmmisnananyitemrtocpuw_upw_downr' rEr' safe_fusingr;r:w_origr_r` fused_weights r _fuse_loraz PatchedLoraProjection._fuse_loraws  ! ! ) 1188==CCTE^E^EeEeEjEjEqEqv**1166<<>%%((//44::<'',,3388>>@  ! ! / / ;$00>>>AWAWA\A\\DeiiT1W vdTUg.WXY.Z!Z[ 5;;|488:??AOOSfU22  1=vUZ0[!!((-"&HHJ jjl $r!c Nt|dd t|ddy|jjj}|j|j }}|j j|j}|jj|j}|j|jtj|dddf|dddfdzz }|j|||jj_d|_d|_ yNr_r`r:rrS) getattrr@rArTr;r:r_r]rUr`r'rXrYrErdr;r:r_r`unfused_weights r _unfuse_loraz"PatchedLoraProjection._unfuse_lorasfd+7GD(TX0A0AW\0A0]!!((-  r!c|jd|_|j|j|S|j||j|j|zzS)N?)r'rDr@)rEinputs rforwardzPatchedLoraProjection.forwards[ ?? "!DO  ! ! ),,U3 3((/4??TE[E[\aEb3bccr!)NNrnF) r __module__ __qualname__r>rMrerlrp __classcell__rs@rr+r+Ss(%6-1uf%@ dr!r+ceZdZdZ d dedededeedeeeje fdeejf fd Z d ejd ejfd ZxZS) r8a A linear layer that is used with LoRA. Parameters: in_features (`int`): Number of input features. out_features (`int`): Number of output features. rank (`int`, `optional`, defaults to 4): The rank of the LoRA layer. network_alpha (`float`, `optional`, defaults to `None`): The value of the network alpha used for stable learning and preventing underflow. This value has the same meaning as the `--network_alpha` option in the kohya-ss trainer script. See https://github.com/darkstorm2150/sd-scripts/blob/main/docs/train_network_README-en.md#execute-learning device (`torch.device`, `optional`, defaults to `None`): The device to use for the layer's weights. dtype (`torch.dtype`, `optional`, defaults to `None`): The dtype to use for the layer's weights. rBrCr<r9r:r;ct|d}tdd|tj||d|||_tj||d|||_||_||_||_ ||_ tjj|j jd|z tjj|j jy)NzmUse of `LoRALinearLayer` is deprecated. Please switch to PEFT backend by installing PEFT: `pip install peft`.r8r7F)biasr:r;rqstd)r=r>rrLinearrWrVr9r<rCrBinitnormal_rAzeros_) rErBrCr<r9r:r;rFrs rr>zLoRALinearLayer.__init__s N#W.ABIIk4eFRWX ))D,U6QVW+ (&  ((a$h7 tww~~&r! hidden_statesreturnc.|j}|jjj}|j|j|}|j |}|j ||j |j z z}|j|Sr)r;rWrAr]rVr9r<rEr orig_dtyper;down_hidden_statesup_hidden_statess rrpzLoRALinearLayer.forward"((    &&!YY}'7'7'>?77#56    )  2 2TYY > > "":..r!)rrNNN)rrtru__doc__intrrUrrXr:strr;r>Tensorrprvrws@rr8r8s0)-59'+''' '   ' u||S012 ' $'4 /U\\ /ell /r!r8ceZdZdZ d dedededeeeeeffdeeeeeffdeeeeefefdee ffd Z d e jd e jfd Z xZS)LoRAConv2dLayera" A convolutional layer that is used with LoRA. Parameters: in_features (`int`): Number of input features. out_features (`int`): Number of output features. rank (`int`, `optional`, defaults to 4): The rank of the LoRA layer. kernel_size (`int` or `tuple` of two `int`, `optional`, defaults to 1): The kernel size of the convolution. stride (`int` or `tuple` of two `int`, `optional`, defaults to 1): The stride of the convolution. padding (`int` or `tuple` of two `int` or `str`, `optional`, defaults to 0): The padding of the convolution. network_alpha (`float`, `optional`, defaults to `None`): The value of the network alpha used for stable learning and preventing underflow. This value has the same meaning as the `--network_alpha` option in the kohya-ss trainer script. See https://github.com/darkstorm2150/sd-scripts/blob/main/docs/train_network_README-en.md#execute-learning rBrCr< kernel_sizestridepaddingr9ct |d}tdd|tj|||||d|_tj||ddd|_||_||_tjj|j jd|z tjj|j jy) NzmUse of `LoRAConv2dLayer` is deprecated. Please switch to PEFT backend by installing PEFT: `pip install peft`.rr7F)rrrrzrqrq)rrrzrqr{) r=r>rrConv2drWrVr9r<r~rrAr) rErBrCr<rrrr9rFrs rr>zLoRAConv2dLayer.__init__s N#W.ABIIk4[QWahotu ))D,F6X]^+   ((a$h7 tww~~&r!rrc.|j}|jjj}|j|j|}|j |}|j ||j |j z z}|j|Sr)rrs rrpzLoRAConv2dLayer.forwardrr!)rrrrrN)rrtrurrrrrrrUr>rXrrprvrws@rrrs439.445)-''' ' 3c3h/0 ' c5c?*+ 'sE#s(OS01' '8 /U\\ /ell /r!rceZdZdZdddeeffdZdeefdZddede fd Z d Z dd e jd ed e jfdZxZS)LoRACompatibleConvz; A convolutional layer that can be used with LoRA. N lora_layerrcPd}tdd|t||i|||_y)NzpUse of `LoRACompatibleConv` is deprecated. Please switch to PEFT backend by installing PEFT: `pip install peft`.rr7rr=r>rrErrNkwargsrFrs rr>zLoRACompatibleConv.__init__0s3Q&1DE $)&)$r!c0d}tdd|||_yNznUse of `set_lora_layer()` is deprecated. Please switch to PEFT backend by installing PEFT: `pip install peft`.set_lora_layerr7rrrErrFs rrz!LoRACompatibleConv.set_lora_layer7s O"G-@A$r!r'rbc<|jy|jjj|jjj}}|jjj }|jj jjj }|jjjjj }|jj0||jjz|jjz }tj|jd|jd}|j|j}|||zz} |r@tj| j!j#rt%d|d| j'|||j_d|_|j)|_|j)|_||_y)Nrq start_dimrQrRrS)rrArTr;r:rUrVrWr9r<rXmmflattenreshapeshaperZr[r\rr]r^r_r` _lora_scale) rEr'rbr;r:rcr_r`fusionrds rrezLoRACompatibleConv._fuse_lora=s ?? "  ((.. 0@0@0G0Gv!!'')!!((--335%%,,11779 ?? ( ( 4$//777$//:N:NND$,,,3V^^a^5PQ/f!45 5;;|488:??AOOSfU22  (??&?F HHJ jjl %r!ct|dd t|ddy|jj}|jj|jj}}|j j |j|_|jj |j|_tj|j jd|jjd}|j|j}|j|j|zz }|j |||j_d|_d|_y)Nr_r`rhrqrrS)rirArTr;r:r_r]rUr`rXrrrrr)rErdr;r:rrks rrlzLoRACompatibleConv._unfuse_lora_s fd+7GD(TXrrUboolrerlrXrrprvrws@rrr+spGK%(?*C%%/)B% &U &t &D$OU\\O%O%,,Or!rceZdZdZdddeeffdZdeefdZddede fd Z d Z dd e jd ed e jffd ZxZS)LoRACompatibleLinearz4 A Linear layer that can be used with LoRA. NrrcPd}tdd|t||i|||_y)NzrUse of `LoRACompatibleLinear` is deprecated. Please switch to PEFT backend by installing PEFT: `pip install peft`.rr7rrs rr>zLoRACompatibleLinear.__init__s3S('3FG $)&)$r!c0d}tdd|||_yrrrs rrz#LoRACompatibleLinear.set_lora_layers O"G-@A$r!r'rbc |jy|jjj|jjj}}|jjj }|jj jjj }|jjjjj }|jj0||jjz|jjz }||tj|dddf|dddfdzz}|r@tj|jjrtd|d|j!|||j_d|_|j#|_|j#|_||_yrP)rrArTr;r:rUrVrWr9r<rXrYrZr[r\rr]r^r_r`rras rrezLoRACompatibleLinear._fuse_loras ?? "  ((.. 0@0@0G0Gv!!'')!!((--335%%,,11779 ?? ( ( 4$//777$//:N:NNDeiiT1W vdTUg.WXY.Z!Z[ 5;;|488:??AOOSfU22  (??&?F HHJ jjl %r!c &t|dd t|ddy|jj}|j|j}}|j j |j}|jj |j}|j|jtj|dddf|dddfdzz }|j |||j_d|_d|_yrg) rirArTr;r:r_r]rUr`rrXrYrjs rrlz!LoRACompatibleLinear._unfuse_lorasfd+7GD(TXrrUrrerlrXrrprvrws@rrrsqGK%(?*C%%/)B% &U&t&@ U\\%%,,r!rr) typingrrrrXtorch.nn.functionalr functionalrutilsrr utils.import_utilsr transformersr r get_loggerrloggerModuler r&rUr4r+r8rrrr}rr!rrs0*) &:G   H % BII  299  3U 3YdEHHOOYdx9/bii9/x=/bii=/@TOTOnG299Gr!