biJddlmZddlZddlZddlZddlmZddlmZm Z ddl m Z ddl m Z mZddlmZmZddlZddlmZdd lmZdd lmZmZdd lmZmZmZmZmZdd lmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%dd l&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5ddl6m7Z7ddl8m9Z9ddl:m;Z;ddlZ>m?Z?m@Z@ddlAmBZBddlCmDZDdZEGddeZFy)) annotationsN)contextmanager)asdictreplace)Enum)partialreduce)LiteralOptional)nn)tqdm)is_bnb_4bit_availableis_bnb_available) BaseTunerBaseTunerLayercheck_target_module_exists onload_layerreplicate_layers)2TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPINGAuxiliaryTrainingWrapperModulesToSaveWrapper_freeze_adapter_get_submodulesget_peft_model_state_dictget_quantization_config) dare_linear dare_tiesmagnitude_prunetask_arithmeticties)get_pattern_key) dispatch_aqlm) dispatch_awq) LoraConfig) dispatch_eetq) dispatch_gptq) dispatch_hqq) dispatch_inc)Conv2d LoraLayer ParamWrapperdispatch_default)dispatch_torchao)dispatch_megatronc||d<||fS)N adapter_names)targetargskwargsr1s Q/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/peft/tuners/lora/model.py_adapter_names_pre_forward_hookr7=s+F? <ceZdZUdZdZded<d dZedZd!dZ dd  d"d Z d Z d#d Z ed Z d$fd Zd%d&dZd'd(dZd)dZd)dZd*dZedZfdZedZ d+ d,dZ d-dZ d. d/dZ d0dZdZd1dZ d2 d3dZd4dZd5d6dZ xZ!S)7 LoraModela Creates Low Rank Adapter (LoRA) model from a pretrained transformers model. The method is described in detail in https://huggingface.co/papers/2106.09685. Args: model ([`torch.nn.Module`]): The model to be adapted. config ([`LoraConfig`]): The configuration of the Lora model. adapter_name (`str`): The name of the adapter, defaults to `"default"`. low_cpu_mem_usage (`bool`, `optional`, defaults to `False`): Create empty adapter weights on meta device. Useful to speed up the loading process. Returns: `torch.nn.Module`: The Lora model. Example: ```py >>> from transformers import AutoModelForSeq2SeqLM >>> from peft import LoraModel, LoraConfig >>> config = LoraConfig( ... task_type="SEQ_2_SEQ_LM", ... r=8, ... lora_alpha=32, ... target_modules=["q", "v"], ... lora_dropout=0.01, ... ) >>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base") >>> lora_model = LoraModel(model, config, "default") ``` ```py >>> import torch >>> import transformers >>> from peft import LoraConfig, PeftModel, get_peft_model, prepare_model_for_kbit_training >>> rank = ... >>> target_modules = ["q_proj", "k_proj", "v_proj", "out_proj", "fc_in", "fc_out", "wte"] >>> config = LoraConfig( ... r=4, lora_alpha=16, target_modules=target_modules, lora_dropout=0.1, bias="none", task_type="CAUSAL_LM" ... ) >>> quantization_config = transformers.BitsAndBytesConfig(load_in_8bit=True) >>> tokenizer = transformers.AutoTokenizer.from_pretrained( ... "kakaobrain/kogpt", ... revision="KoGPT6B-ryan1.5b-float16", # or float32 version: revision=KoGPT6B-ryan1.5b ... bos_token="[BOS]", ... eos_token="[EOS]", ... unk_token="[UNK]", ... pad_token="[PAD]", ... mask_token="[MASK]", ... ) >>> model = transformers.GPTJForCausalLM.from_pretrained( ... "kakaobrain/kogpt", ... revision="KoGPT6B-ryan1.5b-float16", # or float32 version: revision=KoGPT6B-ryan1.5b ... pad_token_id=tokenizer.eos_token_id, ... use_cache=False, ... device_map={"": rank}, ... torch_dtype=torch.float16, ... quantization_config=quantization_config, ... ) >>> model = prepare_model_for_kbit_training(model) >>> lora_model = get_peft_model(model, config) ``` **Attributes**: - **model** ([`~transformers.PreTrainedModel`]) -- The model to be adapted. - **peft_config** ([`LoraConfig`]): The configuration of the Lora model. lora_strprefixct|jdkDr2|jdk7r"t|jj dyy)z A helper method to check the config when a new adapter is being added. Raise a ValueError if there is something wrong with the config or if it conflicts with existing adapters. r"nonezf supports only 1 adapter with bias. When using multiple adapters, set bias to 'none' for all adapters.N)len peft_configbias ValueError __class____name__)selfconfigs r6_check_new_adapter_configz#LoraModel._check_new_adapter_configsR   !A %FKK6,A>>**+,77 -B %r8ct||SN)r) lora_configkeys r6_check_target_module_existsz%LoraModel._check_target_module_existss)+s;;r8cJ|jrt||jyy)a A private method to modify the model structure before adapter is applied. Args: peft_config (`PeftConfig`): The prepared adapter config. model (`nn.Module`): The model that is going to be adapted. N)layer_replicationr)rFrAmodels r6_prepare_modelzLoraModel._prepare_models"  ( ( UK$A$A B )r8N)parameter_namec\| td|jrGtfd|jj D}|rtd|jdt |j j|} t |jj|} |j j| |j} |jj| |j} | | |j|j|j|j|j |j"|j$|j&j(|j*t-|j.ddt-|j.dd|d} t1j2d |j.| d <gd }|D]$}t7|j.| }||| |d <&ddlm}t=|t>xr|j@v}t=|tBrZt=||sN|sL|jE| | |j|j|j|j |j*yt=|t>r||jFk(r tdtI|j.dr|j.jJnd}|jL||fd|i| }|jNvr|jQd|jS||||y#t4$rYlwxYw)NzCurrent Key shouldn't be `None`c3HK|]\}}|k7s |jywrJ)target_parameters).0rLconf adapter_names r6 z0LoraModel._create_and_replace..s)2+43TW[gTg&&2s ""z-Adding a LoRA config with `target_parameters=z` but there are already other LoRA adapters on this model that use `target_parameters`. At the moment, only one LoRA adapter per model with `target_parameters` is allowed.is_loaded_in_8bitFis_loaded_in_4bit)r lora_alpha lora_dropoutfan_in_fan_outinit_lora_weights use_rslorause_dora use_qaloraqalora_group_sizeephemeral_gpu_offload lora_biasloaded_in_8bitloaded_in_4bitrRz:hf_quantizer.quantization_config.get_apply_tensor_subclassget_apply_tensor_subclass)gptqaqlmawq)method_quantization_configr) AdaLoraLayer)r]r^r`rarbrfzTrying to target the same nn.Parameter twice, this should not happen. Please open an issue on the PEFT repo: https://github.com/huggingface/peft/issues hf_device_map device_map)*rCrUanyrAitemsr! rank_patternkeys alpha_patterngetr\r]r^r_r`rarbrcrdruntime_configrerfgetattrrPoperator attrgetterAttributeErrorrpeft.tuners.adaloraro isinstancer,lora_Ar+ update_layerrRhasattrrp_create_new_moduleactive_adaptersrequires_grad__replace_module)rFrKrXr3 target_nameparent current_keyrRother_configs_use_target_paramsr_key alpha_keyr\alphar5 quant_methods quant_methodquantization_configrowrap_target_paramrq new_modules ` r6_create_and_replacezLoraModel._create_and_replaces  >? ?  ( (.128<8H8H8N8N8P2/ +/ CKDaDaCbcVV 8 8 = = ?M#K$=$=$B$B$DkR  $ $ ( ( >))--i9O9OP'44)88!,!>!>%00#,,%00!,!>!>%0%?%?%U%U$..%djj2EuM%djj2EuM, " 3(2E2EL3jj3F. / 0 ) TL"9$**\"Z ".@S,';<= T 5'v|<`,RXR_R_B_ fi (FL1QZk    (55"-"?"?&11$--%// &,/^vG\G\5\ L6=TZZ5Y11_cJ000lFt_itmstJ4#7#77))%0  j& II   s(L L+*L+cHt|||t|dr |j}tjd|j D]\}}|j |vsd|vst|dr |j}net|dr |j}nLt|dr |j}n3t|dd |j}nt|j}tfd|jDr|j|jy) N base_layermetaranknumqweightW_qweightin_proj_weightc3<K|]}|jk(ywrJ)device)rVprs r6rYz,LoraModel._replace_module.. sI188t+I)setattrrrtorchr named_modulesr=rrrryrnext parametersrrto) rFr child_namerchildnamemodulerrs @r6rzLoraModel._replace_module s J/ 5, '$$E||F#&446 -LD& t#d):5),"]]FUE*"YYFUH-"\\FU$4d;G"11F!%"2"2"45FIV5F5F5HIIIIfmm, -r8c|jD]\}}|j|vsd|_|jD]}|j|j }|dk(r"|dk(r%|jD]\}}d|vs d|_L|dk(rR|j D]>}t|tst|ds!|j .d|j _@td|dy) NFr?allrBT lora_onlyzRequested bias: z, is not implemented.) named_parametersr= requires_gradrrArBmodulesr~r+rNotImplementedError)rFrPnractive_adapterrBms r6 _mark_only_adapters_as_trainablez*LoraModel._mark_only_adapters_as_trainable#s**, (DAq{{!#"' (#22 ZN##N388Dv~u}!224/DAq{*./$4A!!Y/GAv4F166K]/3,4*,.dynamic_dispatch_funcAsu! fn5(.(=(=(?%(.%'2'B'B'H'H'JOC!"3S9%/ %O%O !!  "!r8r")dispatch_bnb_8bit)dispatch_bnb_4bitrKzTarget module z is not supported. Currently, only the following modules are supported: `torch.nn.Linear`, `torch.nn.Embedding`, `torch.nn.Conv1d`, `torch.nn.Conv2d`, `torch.nn.Conv3d`, `transformers.pytorch_utils.Conv1D`, `torch.nn.MultiheadAttention.`.)rappendrbnbrrrextendr&r#r$r'r(r)r.r/r-rC) rKrXr3r5 dispatchersrrrr dispatchers r6rzLoraModel._create_new_module8s  & & "   4 5   .   0 1 " .   0 1 !    % J#FL\k\U[\J%    )WW  r8cz t||S#t$r|dk(rt|j|cYSwxYw)z1Forward missing attributes to the wrapped module.rP)super __getattr__r|ryrP)rFrrDs r6rzLoraModel.__getattr__{sB -7&t, , -w4::t, , -s %::c i}|jjD]U\}}t|jDcic]$\}}|t|tr |j n|&}}}|sQd|d<W|<|Scc}}w)NTinference_mode)rArsrr~rvalue)rF inference config_dictrLrkvrGs r6get_peft_config_as_dictz!LoraModel.get_peft_config_as_dicts **002 0JCKQRW=K^K^K`a41aaJq$$7Q>aFa+/'( 0" C bs)A<c|jjD]*}t|ttfs|j |,yrJ)rPrr~rrenable_adapters)rFenabledrs r6_set_adapter_layerszLoraModel._set_adapter_layerss<jj((* 0F&>3K"LM&&w/ 0r8c(|jdy)zyEnable all adapters. Call this if you have previously disabled all adapters and want to re-enable them. TrN)rrFs r6enable_adapter_layerszLoraModel.enable_adapter_layerss    .r8c|jD]<}|j|j}|dk7s"d|d}tj|>|j dy)zDisable all adapters. When disabling all adapters, the model output corresponds to the output of the base model. r?z>Careful, disabling adapter layers with bias configured to be 'zL' does not produce the same output as the base model would without adaption.FrN)rrArBwarningswarnr)rFrvalmsgs r6disable_adapter_layersz LoraModel.disable_adapter_layerssr #22 #N"">277Cf}TUXTYZGG c" #    /r8c|jjD]U}t|ts|jr%t j d|j|j|W||_ y)aSet the active adapter(s). Additionally, this function will set the specified adapters to trainable (i.e., requires_grad=True). If this is not desired, use the following code. ```py >>> for name, param in model_peft.named_parameters(): ... if ...: # some check on name (ex. if 'lora' in name) ... param.requires_grad = False ``` Args: adapter_name (`str` or `list[str]`): Name of the adapter(s) to be activated. zJAdapter cannot be set when the model is merged. Unmerging the model first.N) rPrr~r+mergedrrunmerge set_adapterr)rFrXrs r6rzLoraModel.set_adapters`jj((* 1F&),==MM"noNN$""<0  1 +r8c/Kjdd}|dy|jr tdt}|j D]M}t |t s||jjz}||jjz}O|Dchc] }|dk7s | }}||z }|r&tddjt|jdd} t | txr| dkD} |dd} | rCt |ttfst!dt#|d t%fd |Dg}g} |j D]X} t | t st | t&s$t)t*| }| j-|d }| j/|Z| rt1|j2dr|j2j5j D]X} t | t st | t&s$t)t*| }| j-|d }| j/|Zd| D]}|j7ycc}ww)Nr1z?Cannot pass `adapter_names` when the model is in training mode.__base__z.Trying to infer with non-existing adapter(s): z, num_beamsr"zGot adapter names of type z, expected a list of str.c30K|] }|gdzyw)rNr2)rVrr5s r6rYz7LoraModel._enable_peft_forward_hooks..s Rq!vk':!: Rs)r1T) with_kwargs get_encoder)poptrainingrCsetrr~r+rrulora_embedding_Ajoinsortedrwintlisttuple TypeErrortypesumrrr7register_forward_pre_hookrrrPrremove)rFr4r5r1expected_adapterslayerrunique_adaptersunexpected_adaptersruses_beam_searchoriginal_adapter_names hook_handlesr pre_forwardhandles ` r6_enable_peft_forward_hooksz$LoraModel._enable_peft_forward_hookss^ ?D9    ==^_ _  E\\^ CE%+!U\\%6%6%88!!U%;%;%@%@%BB! C-:PDTZ=O4PP-0AA MdiiX^_rXsNtMuvw wJJ{D1 %i5I9q=!.q!1 mdE];"Cannot merge LORA layers when base model layers are replicated)r_check_merge_allowedryrPrCrArw)rFrDs r6rzLoraModel._check_merge_allowedsX $& 4::4d ;v EXY Y      3 4]^ ^ 5r8c|j>|dtvrtt|d|_|S|js t d|S)N model_typezFPlease specify `target_modules` or `target_parameters`in `peft_config`)target_modulesrrrUrC)rA model_configs r6_prepare_adapter_configz!LoraModel._prepare_adapter_configs\  % % -L)-__-0F|T`Gab. * !22 !ijjr8c|r|j|jjDcgc]\}}|j|vs|}}}d|rdndzdz}t || |D]} t |j|\} } } t| 5t| dr)| j|||} |j| | | | nCt| dr7|r| j|| |j| | | j| ddd|jScc}}w#t $rYwxYw#1swYxYw) Nz Unloading z and merging rP)disabledesc"unload_and_optionally_merge_module)merge safe_merger1r)r r1) rrPrr=r rr|rrr rr r) rFr  progressbarr r1rL_key_listrrr3runloaded_modules r6_unload_and_optionally_mergez&LoraModel._unload_and_optionally_mergesV   % % '&*jj&>&>&@[FCDKKWZDZC[[~B?'Ik/E _C .=djj#.N+ f% _6#GH&,&O&O# -'P'O((ovVV\2 - X((f>S>S>UW]^ _ _  _"zz'\ "   _ _s*D!D!-D'A9D6' D32D36D? c.|D]6}|tjjvs)td|d|D]*}j|jstd|dj Dcgc]}t |ts|}}|Dcgc]tfd|DdkDr}}|rtdt|dt|dk(rd n|}fd |DD cgc]H} | js | jn-t| jg| jjJ} } |d vr(tt| dk7r td | d } n@|dk(r t| } n/|jdr|xs t| } ntd||Dcgc]$}t!j|j"&} }| std|tt| dkDr td| d t$urdj'fd|D} nA| d tur$t)t*j,fd|D} nt/d| d d|| | fScc}wcc}wcc} wcc}w)z Helper function to check if the arguments to add_weighted_adapter are valid and compatible with the underlying model. Adapter  does not existzSadd_weighted_adapter does not support targeting nn.Parameter (problematic adapter 'z')c3:K|]}|jvywrJ)modules_to_save)rVadapterwrappers r6rYz8LoraModel._check_add_weighted_adapter..HsN'7g555Nr"z\Cannot add weighted adapters if they target the same module with modules_to_save, but found z such instance(s).linearc3<K|]}j|ywrJ)rArVrrFs r6rYz8LoraModel._check_add_weighted_adapter..VsM4++G4Mr)rr rrrzkAll adapters must have the same r value when using combination_type linear, ties, dare_ties or dare_linear.rcatsvdzInvalid combination_type: z'Found no adapter matching the names in zall adapter configs should follow the same target modules type. Combining adapters with `target_modules` type being a mix of list/set and string is not supported.|c3XK|]!}dj|jd#yw)()NrArrs r6rYz8LoraModel._check_add_weighted_adapter..us.)r^eAd.>.>w.G.V.V-WWX*Y)rs'*c3PK|]}j|jywrJr"rs r6rYz8LoraModel._check_add_weighted_adapter..xs"`Gt//8GG`s#&z Invalid type z found in target_modules)rrArurCrUrr~rrr@rtr\maxvaluesrendswithrrr<rr rzor_r)rFadapterscombination_typesvd_rankrrmodules_to_save_wrappersrproblematic_wrappersrGadapters_ranksnew_ranktarget_module_typesnew_target_moduless` ` r6_check_add_weighted_adapterz%LoraModel._check_add_weighted_adapter0s  FGd4#3#3#8#8#:;; 8G9O!DEE F  G(:: ijqirrtu :>#tv:V\^rKsF#t #t4 NXNNQRR   n+,--?A  (+8}'98?O NHM% #//FHHS5aFDWDWD^D^D`5a a% % ` `3~&'1, #&a(H  &>*H  & &u -63~#6H9:J9KLM M]efRYtD$4$4W$=$L$LMff"FxjQR R s&' (1 ,u  q !S (!$)riq)r!r   #s *!' `W_`" m,?,B+CC[\] ]+===s$u % 0gs?JJ J;A J :)Jc |t|jjvry|j|||\}} } t |j|d| | | ii|j|<|j |j |t|j ||j jD cgc]\} }|j| vs| }} }|D]} t|j | \}}}t|ts/||jvr3|j|j}|j|j}n.||j vr|j |}|j"|}n|j$dz|_|j$dz|_|dk(r]gg}}t'||D]\}}||jvr3|j|j}|j|j}n.||j vr|j |}|j"|}nu|j)|j$|z|j*|z|j)|j$t-|dk(r t/dt1j2|d}t1j2|d }||j$d|j4dddf<||j$ddd|j4d f<)|d vr,|j7|||| |||| | ||| \|_|_Y|d vs_|j9||||| | \|_|_ycc}} w) a This method adds a new adapter by merging the given adapters with the given weights. When using the `cat` combination_type you should be aware that rank of the resulting adapter will be equal to the sum of all adapters ranks. So it's possible that the mixed adapter may become too big and result in OOM errors. Args: adapters (`list`): List of adapter names to be merged. weights (`list`): List of weights for each adapter. adapter_name (`str`): Name of the new adapter. combination_type (`str`): The merging type can be one of [`svd`, `linear`, `cat`, `ties`, `ties_svd`, `dare_ties`, `dare_linear`, `dare_ties_svd`, `dare_linear_svd`, `magnitude_prune`, `magnitude_prune_svd`]. When using the `cat` combination_type, the rank of the resulting adapter is equal to the sum of all adapters ranks (the mixed adapter may be too big and result in OOM errors). svd_rank (`int`, *optional*): Rank of output adapter for svd. If None provided, will use max rank of merging adapters. svd_clamp (`float`, *optional*): A quantile threshold for clamping SVD decomposition output. If None is provided, do not perform clamping. Defaults to None. svd_full_matrices (`bool`, *optional*): Controls whether to compute the full or reduced SVD, and consequently, the shape of the returned tensors U and Vh. Defaults to True. svd_driver (`str`, *optional*): Name of the cuSOLVER method to be used. This keyword argument only works when merging on CUDA. Can be one of [None, `gesvd`, `gesvdj`, `gesvda`]. For more info please refer to `torch.linalg.svd` documentation. Defaults to None. density (`float`, *optional*): Value between 0 and 1. 0 means all values are pruned and 1 means no values are pruned. Should be used with [`ties`, `ties_svd`, `dare_ties`, `dare_linear`, `dare_ties_svd`, `dare_linear_svd`, `magnintude_prune`, `magnitude_prune_svd`] majority_sign_method (`str`): The method, should be one of ["total", "frequency"], to use to get the magnitude of the sign values. Should be used with [`ties`, `ties_svd`, `dare_ties`, `dare_ties_svd`] N)r(r)r*r)r\r]rrvrtgrz9No matching LoRAs found. Please raise an issue on GitHub.dimr")rties_svddare_linear_svd dare_ties_svdmagnitude_prune_svd full_matricesdriver)rr rrr)rrArur1rinject_adapterrPrrr=rr~r+rrlora_Brlora_embedding_Bdataziprscalingr@rCrrshape1_svd_generalized_task_arithmetic_weighted_adapter-_generalized_task_arithmetic_weighted_adapter)rFr(weightsrXr)r* svd_clampsvd_full_matrices svd_driverdensitymajority_sign_methodr.r0rLr rr3 target_lora_A target_lora_Bloras_Aloras_Brrcurrent_adapter_lora_Acurrent_adapter_lora_Bs r6add_weighted_adapterzLoraModel.add_weighted_adaptersvj 4 0 0 5 5 78 8 9=9Y9Y-:Z: 6($6 *1   Xa[ )- * & DJJ 5  L1&*jj&>&>&@[FCDKKWZDZC[[: C*4::s;LAvq&),6==0$*MM,$?$F$FM$*MM,$?$F$FM!V%<%<<$*$;$;L$IM$*$;$;L$IM%2%7%7#%= "%2%7%7#%= "#u,')2WG+.x+A D"fmm35;]]75K5R5R25;]]75K5R5R2$(?(??5;5L5LW5U25;5L5LW5U2$'='B'BV'Kfnn]dNe'ef'='B'BC D7|q(()dee#iiQ7G#iiQ7G@GM&&'9q)9'91'<=@GM&&q*B=s=s(  %%,!&7)>t >:M& (:&)jj=A=o=o((GVWNb>:M& (:q: \s <M"M"c g} g}tfd|D}t||D]T\}}|jvs|jvs#| j ||j |j |zVt | dk(r td| Dcgc]}j|}}tj|j|dj}|dk(r t||}n\|dk(rt|||| }nH|dk(rt|||}n5|dk(rt!|||| }n!|dk(rt#|||}ntd |t%t&}|rEj(j+d d d k(}|s|j-d }n|j/}t1dr j2s|r |j4}tj6j9|| | \}}}|ddd|f}|d|}|tj:|z}|d|ddf}| ptj<|j-|j-g}tj>|| }| }|jA||}|jA||}|rJ|jC|jDjF}|jC|jDjF}||fScc}w)Nc3:K|]}|jvywrJ)r)rVrr3s r6rYzNLoraModel._svd_generalized_task_arithmetic_weighted_adapter..sV'7f&=&==Vrrz9No matching LoRAs found. Please raise an issue on Github.rr5r6r7r8z*Invalid value passed to combination type: )r"r"r") start_dimr_r9)$rrr@rrrrAr@rCget_delta_weightrtensorrrrr rrrr~r*rsizeflattensqueezerr_Tlinalgrdiagrquantileclampreshaper?rB)rFr)r(rEr.r3rKrLrIrJr`r:r;valid_adapters valid_weights is_embeddingrr delta_weightconv2d conv2d_1x1USVhdisthi_vallow_vals ` r6rCz;LoraModel._svd_generalized_task_arithmetic_weighted_adapters VXVV "8W5 GOGV&--'7f6M6M+M%%g.$$VfnnW.E%EF G ~ ! #XY YHVWW//8W W ]366|A7M7MN u $*<GL  + mWFZ !2 2&|]GLL  0$\='K_`L !6 6*<PLIJZI[\] ]FF+ ++-a2f>L<<##L V\#]1b a(lO ixL  1    1   99aiik2::<89D^^D%0FgG(A'6*B  -,,223AM..445B1u QXsK+cg}g}g} t||D]\} } | |jvr3|j| j} |j| j} n.| |jvr|j| } |j | } nu|j tj| |j| z|j | j| j | jtj|j|dj}|| g}|dj}t!|D]~\}}|dk(rt#||||<|dk(rt%||||||<2|dk(rt'|||||<H|dk(rt)||||||<_|dk(rt+|||||<ut-d|Dcgc]}|j|}}|Scc}w)Nrrr rrrzInvalid combination type)r@rrr=rr>rmathsqrtrAr?rrXrrdtype enumeraterr rrrrC)rFr)r(rEr3rIrJrc lora_A_deltas lora_B_deltasrrrOrP lora_deltasrqi task_tensorsdeltas r6rDz7LoraModel._generalized_task_arithmetic_weighted_adapterKs   "8W5 >OGV&--')/w)?)F)F&)/w)?)F)F&F333)/)@)@)I&)/)@)@)I&  6FNN74K+K!L M  !7! ]366}Q7G7N7NO $m4 a &&(5 =OA|8+!0}!M A!V+!%lM7L`!a A!]2!,\='!R A![0!*<Qe!f A!%66!0}g!V A !;<< =5@@5uxx@ @AsG$c|t|jjvrtd|d|j|=|jj Dcgc]\}}|j |vs|}}}d}|D]P}t|j|\}}}t|ts.|j||B|jdd}R|xsg|_ |j||ycc}}w)z Deletes an existing adapter. Args: adapter_name (str): Name of the adapter to be deleted. rrN)new_active_adapters)rrArurCrPrr=rr~r+delete_adapterrr_delete_auxiliary_adapter)rFrXrLr r new_adapterr3s r6r{zLoraModel.delete_adapterws tD$4$4$9$9$;< <x ~_EF F   \ *&*jj&>&>&@[FCDKKWZDZC[[  >> from transformers import AutoModelForCausalLM >>> from peft import PeftModel >>> base_model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-40b") >>> peft_model_id = "smangrul/falcon-40B-int4-peft-lora-sfttrainer-sample" >>> model = PeftModel.from_pretrained(base_model, peft_model_id) >>> merged_model = model.merge_and_unload() ``` )r r r1r)rFr r r1s r6merge_and_unloadzLoraModel.merge_and_unloads#800# -1  r8c&|jdS)z Gets back the base model by removing all the lora modules without merging. This gives back the original base model. F)r rrs r6unloadzLoraModel.unloads 00u0==r8c |jjD]\}}|jjtj k7s.|jjtj k7sV|jjtjk7s~|jdstjdt||jdd|}i}|jD]}d|vrDt j|||dj|j!dddgd ||<Kd |vsPt j|||dj|j!ddd gd ||<|S) a! This function can calculate the updates of the PiSSA/CorDA/OLoRA by comparing the parameters of the PiSSA/CorDA/OLoRA adapter in `output_state_dict` with the initial values of PiSSA/CorDA/OLoRA in `adapter_name`, thus converting PiSSA/CorDA/OLoRA to LoRA. pissaaNote that Quant(W_res) + AB != Quant(W) + \Delta(AB); the converted LoRA, when combined with W or Quant(W), may introduce a certain gap in the fine-tuned model. Therefore, we recommend directly using the Quant(W_res) in conjunction with the PiSSA adapter. state_dictN)rrXr.r"rr3r=)rPrr?rqrfloat32float16bfloat16 startswithrrrrwrurrsplit)rFoutput_state_dictrXr5rparammutated_init_state_dict tensors_loras r6subtract_mutated_initzLoraModel.subtract_mutated_initsq  ::668 KD%   EMM1JJ$$ 5JJ$$6))'2 v #< zz,5%#   %**, D4%*YY&t,.EchhtzzZ]_`_aObFc.dekl& T"T!%*YY&t,/FsxxPTPZPZ[^P_`a`bPcGd/e.eflm& T" r8)rGr%returnNone)rAr%rP nn.Module)rRz Optional[str]rr)rPrrr)rr<)F)rbool)T)rrrr)rr)rXzstr | list[str]rr)TFFN)r rr rr1Optional[list[str]])r( list[str]r)r<r* int | Nonerztuple[str, int, str])rNNTNNtotal)r(rrEz list[float]rXr<r)r<r*rrFrrGrrHz str | NonerIz float | NonerJzLiteral['total', 'frequency']rr)NTN)rXr<rr)FFN)r rr rr1rrtorch.nn.Module)rrrJ)rzdict[str, torch.Tensor]rXr<)"rE __module__ __qualname____doc__r=__annotations__rH staticmethodrMrQrrrrrrrrrrrrrrrr1rQrCrDr{rrr __classcell__)rDs@r6r:r:CsFPFC << C,)-YJ&YJ YJv-4Z*@@D-0 / 0+.99v _! -1   + >M>!M>58M>DNM> M>h!&# $"&!% $>EFFF F  F  FF FFF<F FfBH*XV0im  59 Re  @>$$r8r:)G __future__rrorzr contextlibr dataclassesrrenumr functoolsrr typingr r rr r peft.import_utilsrrpeft.tuners.tuners_utilsrrrrr peft.utilsrrrrrrrpeft.utils.merge_utilsrrrrr peft.utils.otherr!rkr#rlr$rGr%eetqr&rjr'hqqr(incr)rr*r+r,r-torchaor.tp_layerr/r7r:r2r8r6rs# %'%$ Eba,DD%' V Vr8