bi̅&ddlZddlmZmZddlmZmZddlmZddl m Z ddl m Z ddl mZddlmZmZddlZddlmZdd lmZdd lmZdd lmZmZdd lmZdd lmZddl m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'm(Z(m)Z)e&e(e)fZ*GddZ+Gdde+Z,Gdde+Z-de.de.fdZ/dej`jbdejdfdZ3dee4ejdffdZ5de$fd Z6dejnfd!Z8d"Z9dej`jbd#edee$d$e:d%ee:d&ee:d'ee:e.e4e:fdfd(e;d)e;de.fd*Z!((*I'00#i.1Y;CTCTUVUWCX1YZL3>L/;,,Q/ 0HMZGXY!u// =YOY OOO\-D-D-F GBE_V_A`avet}aOa99_!4 4Zbs E>F)NT)__name__ __module__ __qualname____doc__strr callableboolr' staticmethodr1r2r"no_gradr>rYr(r&rr)sz7;*.    "*(!3  $(   RWR^R^  U]]_WWr(rcxeZdZdZdedeeejfffd Z ejdZ xZ S)SVDHooka* A forward hook for calculating incremental SVD on layer inputs. The hook is designed to be registered to a PyTorch module using the `register_forward_hook` method. This hook performs a step of incremental Singular Value Decomposition (SVD) on the inputs of a specified layer during the forward pass of a neural network. The hook also tracks convergence of the computed components using cosine similarity between the current and previous components. Args: name (str): Name of the layer to which this hook is attached. n_components (int): Number of principal components to compute. sim_thresh (Union[float, torch.Tensor]): Similarity threshold for convergence. prepare_layer_inputs_fn (Optional[callable]): Function to prepare layer inputs for SVD. n_components sim_threshc t|di|||_||_t |t j rit|jdkDrQ|jd|k(xs|jddk(}t|jdk(}|r|s tdt|ddd|_ d|_ t j|ft j|_y) Nrrz`if sim_thresh is a tensor with more than 0 dimensions it must have shape (n_components,) or (1,)T*)rfcopylowrank lowrank_seed)rArc)superr'rfrgr0r1r2lenrGr9r5rsvdr$zerosr` converged)r%rfrgbase_class_kwargscheck1check2 __class__s r&r'zSVDHook.__init__zs -,-($ j%,, /C 8H8H4IA4M__Q'<7R:??1;MQR;RF))*a/Fv v"%    l_EJJGr(cd}t|jdr2|jjjj }|j |}|j |}|jd|jkr'td|jd|jdy|jj|jtj|y|jj}t|j dk(r$|j#dd}|j#dd}tj$j&j)||}||j*k\|_y)N components_rzskipping SVD for z because there are less than z examplesrr/)hasattrrorwclonedetachr>rYr9rfprintr partial_fittor1float32rnrGreshapenn functionalcosine_similarityrgrq)r%modelinputoutputprevious_componentsstates componentssims r&__call__zSVDHook.__call__s2" 488] +"&(("6"6"<"<">"E"E"G **51))&1 ;;q>D-- - %dii[0MdN_N_M``ij k  VYYu}}56  & XX)) z A %#++Ar2J"5"="=a"D hh!!33J@ST/r() rZr[r\r]intr floatr1r2r'rbr __classcell__rus@r&rerejsL HH%-.H2U]]_00r(rec`eZdZdZfdZedZejdZ xZ S)HashHooka A forward hook for hashing layer inputs. The hook is designed to be registered to a PyTorch module using the `register_forward_hook` method. This hook hashes the inputs of a specified layer during the forward pass of a neural network and stores the hash values for later analysis or comparison. Args: name (str): Name of the layer to which this hook is attached. hashed_inputs (list): List of hashed inputs. prepare_layer_inputs_fn (Optional[callable]): Function to prepare layer inputs for hashing. c 2t|di|g|_yNrc)rmr' hashed_inputs)r%rrrus r&r'zHashHook.__init__s -,-r(cdtt|jdjS)Nr/)hashr3r8rJ)rFs r&hash_fnzHashHook.hash_fns#E&++b/002344r(c|j|}|j|}|jj|j |j yr!)r>rYrappendrcpu)r%rrrxs r&rzHashHook.__call__sE  % %e ,  $ $Q ' !!$,,quuw"78r() rZr[r\r]r'rarr1rbrrrs@r&rrs=  55U]]_99r(r dictionaryr)ctt}|jD]\}}||j||jDcic]\}}t |dkDs||c}}Scc}}w)a: Find keys in a dictionary that have the same value. This function takes a dictionary and returns a new dictionary containing keys that have the same value. The keys in the output dictionary are the values from the input dictionary, and the values are lists of keys that share the same value. r)rr4itemsrrn)r value_dictkvs r&find_equal_valuesrskT"J  " 11 Q '--/ >TQ3q6A:AqD >> >s A/&A/rct|jDchc](}|jjdk7s|j*c}}t |dkDrt j d|y|dScc}w)za Get the device of the model's parameters. Useful if some parameters are on meta device. metarz8Could not determine device, model has multiple devices: Nr)r4 parametersr@r6rnwarningswarn)rpdevicess r&get_device_with_meta_paramsrsje&6&6&8TAHHMMVz(move_inputs_to_device..sM1!V<Mszinput of type z) could not be moved to the correct device) rxr}r0rr6rrr3r4rr)inputsr@rrs ` r&rrsvtyy  &'"tF|V\\^\TQQ 5a @@\]] FUDM *tF|MfMMM tF|n4]^_ ]s C peft_configcht|ts td|jdt j |dj }|jjrr(ct|tjrn7t|ttfr|d}nt dt |d|d||jjS)a if not all items in the input should be used for SVD, this function can be used to get the indices of the items that should be used. Attributes: layer_input (torch.Tensor): The layer inputs. model_input (torch.Tensor): The model inputs or if `prepare_model_inputs_fn` is not None the output of this function. layer_name (str): The name of the layer. Returns: torch.Tensor: The input to the SVD. rr+r,r-) r0r1r2r3r4r5r6Tunbindr:s r&)prepare_layer_inputs_fn_language_modelingrst+u||, K% /!!n %d;&7%88^_i^jk> >  {}}++- ..r(c|di|Srrc)rrs r&forward_fn_dictrs ?6?r( dataloadertarget_module_check_fn forward_fnprepare_model_inputs_fnrrshow_progress_barc 0d} t|dk(r tdtjr|rt j dd} |j j} | | kDr;td|jD} | |jz} t| | } |j}t|}|jtt!|}| t#||}| |||}n t%|}i}i0d}|j'D]\}}|||st)|t*r|j-|d}n|}t/|||}||_|j3|}||f||<|j4j7t9|j4j;||j}t=|| z0|<||z }t)|t*r*t|dkDrtd|j;||||j?Dcic]\}}||dj@d}}}tCtE|jG}|Dcic]}|d dD]}||d } }}|D]"}!t0fd |!D}"|!D]}#|"0|#< $tC|j;D]}|j-|\}}|jI|| vr,tK0||j jL||jN| }|jQ|}|j3|}||f||<itStU|j;|j;| }$|rNtjrtjVdk(r#tYt!t[|dd }%d}&nt!t[|}%d }&|j;Dcic]}|d }'}0j]}(|%D]}| t#||}| |||}n t%|}tC|j;D]}||\}}t_j`|jbd|(|})|'|s|)r|r|jId}d|'|<Q|'|r)|)s'|jQ|}|j3|}d |'|<||_||f||<|&rdtC|'jG| jGDcgc]}|'| c}z}*|%jetg|*dt|*dta|'jGrn:|||tad|jGDs}| ||$| |0}(|j'D#+chc]7\}#}+|+jhjGD]}t)|tjs|#9},}+}#}t|,dkDrtd|,di}-|(j?D]\}}.||$|d}t_j`|jbd|.std|d|.d|jljnd|.}/|j jpr:|/|jljrd|.jujwdd z}/|/|-|<|jy||2|-j?Dcic]\}}||j{|}-}}|-Scc}}wcc}}wcc}wcc}wcc}}+}#wcc}}w)Nc r|jDcic]%\}}||djjd||'}}}t|jD cgc]\}}||D]} || fc} }}\} } t j | j d} t| d|D cgc]} | |  c} }|jDcic]}||j|d}}|jD] \}}||||}}||k\r||c||<||<"|Scc}}wcc} }}wcc} wcc}w)NrT) descending) rroexplained_variance_ratio_rQr1stackargsortrkeysr)hookslayer_hook_mapequal_inputs_map rank_budgetmax_componentsrhexp_varsrcrvaluesidxicountsk_hookrank rank_hooks r&_get_rank_distributionz3_get_eva_state_dict.._get_rank_distribution.sO[`[f[f[hiSWSTVWAqtxx99:MN13G3G3Ibb4S[\`SabaaVbVbc fkk&!))T):3| +<=a$q'=>/=/B/B/DE!!VZZ1%%EE)//1 8IAv$Qi)DD (,i %F6NF1I  8  jb=Es*D"D( , D/D4rzdataloader is emptyztorch.distributed is initialized and `gather_distributed_inputs` is True, therefore EVA initialization will gather tensors from all ranks. Ensure the model does not receive the same inputs on different ranks.ic3FK|]}t|jywr!)rLrG)rrs r&rz&_get_eva_state_dict..Ls?qc!''l?s!)rrrzaprepare_layer_inputs_fn is a mapping but the following module names were not found in the model: rc3(K|] }| ywr!rc)rnrs r&rz&_get_eva_state_dict..s9aq)9s)rfrgrrrF)positionleaveT/z layers have convergedc3NK|]}t|djdyw)rrwN)rxro)rrs r&rz&_get_eva_state_dict..sL71Q488]3Ls#%z?Found active hooks added by EVA that weren't properly removed: zH. Please report this issue at https://github.com/huggingface/peft/issueszLayer z) has not converged but was assigned rank r/)>rnr5rCrDrrrrhorLrrmintrainingrevalnextiterrr named_modulesr0rpoprr$register_forward_hook rank_patternrrrroundrrr4rrremoveretaur# get_submodulerrQget_rankr r rjr1allrqset_descriptionsum_forward_hooksrrorwwhitensingular_values_sqrtrtrainr})1rrrrrrrrrr rho_thresholdrmax_dimrho_ceilrr@rmodel_inputs_for_hooksrrrmodulefnhookhandle layer_rankrr hash_dict equal_inputsrvvrnames max_valuerrpbaruse_tqdmconvergence_dict rank_distrqlayer_convergedmremaining_hookseva_state_dictrurs1 @r&_get_eva_state_dictr!s  :!.// !:  T M  $ $C ]?E,<,<,>??kmm+#x ~~H ( /F JJL$z" #F &vv6*!8!M!)&!1 ENK++-" f%dF3  -w 7(,,T48B(BT2Yrs1--d3Vnd  --11 K4499;T BKMM  %Z#%56tz! " )73L*6GQ12G2AaDGGG*9599  *A )N1  ** UZZ\"%yy f  # # '-"--11$($A$A&?  $$T*--d3Vnd %SSuzz|<=RAQRN$"5"5"74==?a;ODz*+auEE*%&*/**,7Q577##%I*q  *66:F " .%k\r tA||}||j|_!||j|_ | |j|_|r tEjFd |d tHyy) N)r lora_dropout use_rslorause_dora lora_biaszbase_model.model.rreva)r lora_alphainit_lora_weightsTzuthe following layers were initialized with init_lora_weights=True because they were not found in the eva state_dict: z@ currently the following lora modules are not supported by EVA: rc)%rrrrrrreplacer0rrrrrrr alpha_patternr rr9rsetattrget_base_layerradjust_scaling_factorslora_Aweightr@r6 update_layercopy_rnrrtarget_modulesrrUNSUPPORTED_LORA_MODULES)rrrrupdate_layer_kwargsmissing_eva_initsnew_target_modulesother_module_namesrr#rrname_in_base_modelralphawnew_rankparentrW target_names r&_load_eva_state_dictr7s  ##L1K$#00!,,(( ** LM++- 6 f!\\*=rB&),  % %&8 9   $ $ ( (9Q9Q9V9V9XZ^)_alanan o))-- K55:: !""4(AvvayH1})8)E&; V-B-B-DEQ))@@X\)E1} l ; B B I I N NRX X###qh5TYq]pq MM, ' . . 4 4Q 7  % %&8 9 F   e! eQd e  $ $%7 8H {}} $/7L+ , K** *05M, -A 6F,.?? "EE9:LN`a5GEl#24@El#05BEl#1  55F4GH@@X?Y [ r(defaultTc :d} d} t|d} | r||j|}n | td| r"|j} t | t } nt } t | |} | 5t|||| ||||| }ddd|S#1swYSxYw) a Compute the SVD for each layer in the model. This function computes the Singular Value Decomposition (SVD) for each layer in the model. It uses the incremental PCA method to compute the SVD components. The function also checks for convergence of the computed components using cosine similarity. The rank distribution for each layer is determined based on the explained variance ratio. Args: model (torch.nn.Module): The model to compute the SVD for. Does not need to be a PeftModel. dataloader (Iterable): The dataloader to use for the forward pass. peft_config (Optional[LoraConfig]): The configuration for the LoRA layers. Only required if `model` is not a PeftModel. forward_fn (callable): The forward function to use for the forward pass. Takes two arguments: `model` and `inputs`. Default behavior is `return model(**inputs)` prepare_model_inputs_fn (Optional[callable]): This function receives the model inputs and the peft_config and passes the output to `prepare_layer_inputs_fn`. Can be used to modify the input to the SVD computation based on the original model inputs. For example for language modeling the attention mask is used to determine which indices are padding tokens and should not be used for SVD. Any function defined here expects two arguments: `model_input` and `peft_config`. `peft.tuners.lora.eva.prepare_model_inputs_fn_language_modeling` is used by default. prepare_layer_inputs_fn (Union[callable, Dict[str, callable], None]): This function receives the layer inputs, the model inputs (potentially modified by `prepare_model_inputs_fn`) and the name of the layer and returns the inputs that should be used for SVD for that particular layer. Any custom function defined here expects three arguments: `layer_input`, `model_input`, and `layer_name` and should return a 2d tensor. The default logic can be found in peft.tuners.lora.eva.prepare_layer_inputs_fn_language_modeling and works for language modeling. In this case model_inputs is the mask used to determine which indices should be used for SVD (created by `prepare_model_inputs_fn_language_modeling`). adapter_name (str): The name of the adapter to compute the SVD for. gather_distributed_inputs (bool): Whether to gather the layer inputs from all ranks. Default is True meaning in a distributed setting the layer inputs will be gathered from all ranks for the SVD computation. For non-distributed settings this argument is ignored. Set to False if you are using a non-distributed dataloader in a distributed setting. show_progress_bar (bool): Whether to show a progress bar. Default is True. Returns: eva_state_dict (dict): The state dictionary containing the SVD components for each layer. c8t|dxr t|| S)z?check if a module is an adapter module via base_layer attribute base_layer)rxr0)rrunsupported_lora_moduless r&!target_module_check_fn_peft_modelz=get_eva_state_dict..target_module_check_fn_peft_modelfsv|,aZH`5a1aar(cd}|j t||}t|tjj t fxr|S)z9check if a module is an adapter module via target_modulesT)r+rr0r1rLinearr)rrris_target_modules r&target_module_check_fn_defaultz:get_eva_state_dict..target_module_check_fn_defaultjsA  % % 19+tL &588??F";<QAQQr(rNz3peft_config is required if model is not a PeftModel)r<)r) rrrrrrrrr)rxrr5disable_adapterr r,rr)rrrrrrrrrr=rA is_peft_modelctxrrs r&get_eva_state_dictrE1sjbRE=1M,'' 5  NOO##%!( -H`" m!()GU`!a  ,!##9!$;$;&?/        s 1BBc $t|ds tdt|jdkDr td|j|j dk7r td| | tdt |||||||| }t|||y) a Initialize the weights of the LoRA layers using the EVA method. This function initializes the weights of the LoRA layers using the EVA method. It computes the SVD for each adapter layer and updates the weights accordingly. Args: model (PeftModel): The peft model to compute the SVD for. dataloader (Optional[Iterable]): The dataloader to use for the forward pass. If None, eva_state_dict needs to be provided. eva_state_dict (Optional[dict]): The state_dict to load into the model. If None, a dataloader needs to be provided and the state_dict will be computed using `get_eva_state_dict`. forward_fn (callable): The forward function to use for the forward pass. Takes two arguments: `model` and `inputs`. Default behavior is `return model(**inputs)` prepare_model_inputs_fn (Optional[callable]): This function receives the model inputs and the peft_config and passes the output to `prepare_layer_inputs_fn`. Can be used to modify the input to the SVD computation based on the original model inputs. For example for language modeling the attention mask is used to determine which indices are padding tokens and should not be used for SVD. Any function defined here expects two arguments: `model_input` and `peft_config`. `peft.tuners.lora.eva.prepare_model_inputs_fn_language_modeling` is used by default. prepare_layer_inputs_fn (Union[callable, Dict[str, callable], None]): This function receives the layer inputs, the model inputs (potentially modified by `prepare_model_inputs_fn`) and the name of the layer and returns the inputs that should be used for SVD for that particular layer. Any custom function defined here expects three arguments: `layer_input`, `model_input`, and `layer_name` and should return a 2d tensor. The default logic can be found in peft.tuners.lora.eva.prepare_layer_inputs_fn_language_modeling and works for language modeling. In this case model_inputs is the mask used to determine which indices should be used for SVD (created by `prepare_model_inputs_fn_language_modeling`). adapter_name (str): The name of the adapter to initialize the weights for. gather_distributed_inputs (bool): Whether to gather the layer inputs from all ranks. Default is True meaning in a distributed setting the layer inputs will be gathered from all ranks for the SVD computation. For non-distributed settings this argument is ignored. Set to False if you are using a non-distributed dataloader in a distributed setting. show_progress_bar (bool): Whether to show a progress bar. Default is True. Returns: model (torch.nn.Module): The model with the initialized LoRA weights. rzmodel must be a PeftModelrzO`initialize_lora_eva_weights` currently only works with a single active adapterrzM`initialize_lora_eva_weights` can only be used with `init_lora_weights='eva'`Nz8dataloader is required if eva_state_dict is not provided)rrrrrrrr)rxr5rnactive_adaptersrr!rEr7) rrrrrrrrrs r&initialize_lora_eva_weightsrHsj 5- (455 5 !A%jkk &88EAhii  WX X+!!$;$;%&?/   =r()Ar collectionsrrcollections.abcrr contextlibrrjr functoolsr itertoolsr typingr r r1torch.distributed distributedrCr transformers.pytorch_utilsrpeft.tuners.tuners_utilsrrpeft.utils.constantsrpeft.utils.incremental_pcarpeft.utils.otherrrconfigrlayerrrrrr,rrerrrrModuler@rr^rrr2rrr_r`rr7rbrErHrcr(r&rYs@,-"" -]D5=DD&'97C>>B>0e>0F9u98 ?$ ?4 ?uxx5<< %U\\0A*B    /W\WcWc/8E 88??EE*%E% E " E &h/ E#8T#x--@$#FGE $EE EPE 88??E E E P)-%42[Js!&*"^ 88??^^*%^" ^ &h/ ^ #8T#x--@$#FG ^^ $^^ ^^B&*%)%42[Js!&*"O> 88??O>"O>TNO>" O> &h/ O> #8T#x--@$#FG O>O> $O>O>O>r(