biQOddlmZddlZddlmZddlmZmZmZddl Z ddl m Z ddl m Z ddl mZddlmZd d lmZd d lmZd d lmZd d lmZmZddZddZGddee jj8Zy)) annotationsN)contextmanager)AnyOptionalUnion)remove_hook_from_submodules)nn)PushToHubMixin)DUMMY_MODEL_CONFIG) PeftConfig) PeftModel) MixedModel) _set_adapter_set_trainablect|dds|St|ddsit|dds[t|ddsMt|dr|jy t|dr#d }|jj |y y y y y ) zD Prepares the model for gradient checkpointing if necessary is_gradient_checkpointingTis_loaded_in_8bitFis_loaded_in_4bit is_quantizedenable_input_require_gradsget_input_embeddingsc&|jdy)NT)requires_grad_)moduleinputoutputs K/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/peft/mixed_model.pymake_inputs_require_gradzK_prepare_model_for_gradient_checkpointing..make_inputs_require_grad3s%%d+N)getattrhasattrrrregister_forward_hook)modelrs r)_prepare_model_for_gradient_checkpointingr%"s 55t <  *E2 5-u 5 5.% 0 56 7  , , . U2 3 ,  & & ( > >?W X 4 1 6 3r cvddlm}|j|vr%td|jjd|y)Nr )COMPATIBLE_TUNER_TYPESzThe provided `peft_type` 'zE' is not compatible with the `PeftMixedModel`. Compatible types are: ) tuners.mixedr' peft_type ValueErrorvalue) peft_configr's r_check_config_compatibler-9sK4$::()>)>)D)D(EF%%;$< >  ;r cHeZdZdZddfd ZeddZeddZeddZdZ dZ dfd Z d d Z d d Z ed Zd!d"d Zd#d Zd$dZd$dZd dZd dZdZdZed%dZd&dZd'dZd(dZ d) d*dZe d+ d,dZxZS)-PeftMixedModela, PeftMixedModel for loading mixing different types of adapters for inference. This class does not support loading/saving, and it shouldn't usually be initialized directly. Instead, use `get_peft_model` with the argument `mixed=True`. Read the [Mixed adapter types](https://huggingface.co/docs/peft/en/developer_guides/mixed_models) guide to learn more about using different adapter types. Example: ```py >>> base_model = ... # load the base model, e.g. from transformers >>> peft_model = PeftMixedModel.from_pretrained(base_model, path_to_adapter1, "adapter1").eval() >>> peft_model.load_adapter(path_to_adapter2, "adapter2") >>> peft_model.set_adapter(["adapter1", "adapter2"]) # activate both adapters >>> peft_model(data) # forward pass using both adapters ``` Args: model (`torch.nn.Module`): The model to be tuned. config (`PeftConfig`): The config of the model to be tuned. The adapter type must be compatible. adapter_name (`str`, `optional`, defaults to `"default"`): The name of the first adapter. low_cpu_mem_usage (`bool`, `optional`, defaults to `False`): Create empty adapter weights on meta device. Useful to speed up the loading process. czt|t|t|d|_t |||i||_|j||t|dt|_ t|j dr=t|j jdrd|j j_ yyy)Nconfigpretraining_tpr ) super__init__r-r%modules_to_saver base_modelset_modules_to_saver!r r1r"r2)selfr$r, adapter_name __class__s rr4zPeftMixedModel.__init__fs  -1%8#$U\;,GV   l;eX/AB 4??H -'$//:P:PRb2c45DOO " " 13d -r c.|jjSN)r6r,r8s rr,zPeftMixedModel.peft_configvs***r c.|jjSr<)r6active_adapterr=s rr?zPeftMixedModel.active_adapterzs---r c.|jjSr<)r6active_adaptersr=s rrAzPeftMixedModel.active_adapters~s...r cd}d}|jD]g\}}|j}|dk(rt|dr |j}|jj dk(r|dz}||z }|j sc||z }i||fS)zg Returns the number of trainable parameters and number of all parameters in the model. rds_numel Params4bit)named_parametersnumelr"rCr:__name__ requires_grad)r8trainable_params all_param_param num_paramss rget_nb_trainable_parametersz*PeftMixedModel.get_nb_trainable_parameterss  --/ /HAuJQ75*#="^^ ''<7'!^  #I"" J.  /  **r c d|j\}}td|dd|ddd|z|z dy)a Prints the number of trainable parameters in the model. Note: print_trainable_parameters() uses get_nb_trainable_parameters() which is different from num_parameters(only_trainable=True) from huggingface/transformers. get_nb_trainable_parameters() returns (trainable parameters, all parameters) of the Peft Model which includes modified backbone transformer model. For techniques like LoRA, the backbone transformer model is modified in place with LoRA modules. However, for prompt tuning, the backbone transformer model is unmodified. num_parameters(only_trainable=True) returns number of trainable parameters of the backbone transformer model which can be different. ztrainable params: z,dz || all params: z || trainable%: dz.4fN)rOprint)r8rJrKs rprint_trainable_parametersz)PeftMixedModel.print_trainable_parameterssU'+&F&F&H#)  !1" 56$R.)!11I=cB D r cz t||S#t$r|dk(rt|j|cYSwxYw)z1Forward missing attributes to the wrapped module.r6)r3 __getattr__AttributeErrorr!r6)r8namer:s rrUzPeftMixedModel.__getattr__sC 27&t, , 2|#4??D1 1 2s %::c&|j|i|S)z, Forward pass of the model. )r6r8argskwargss rforwardzPeftMixedModel.forwardst///r c:|jj|i|S)z" Generate output. )r6generaterYs rr^zPeftMixedModel.generates (t''888r c#K |jjd|jjy#|jjwxYww)z. Disables the adapter module. N)r6disable_adapter_layersenable_adapter_layersr=s rdisable_adapterzPeftMixedModel.disable_adapters>  4 OO 2 2 4  OO 1 1 3DOO 1 1 3sA=AAAct| ||j|<|jj||||j ||y#t$r||jvr |j|=wxYw)a Add an adapter to the model based on the passed configuration. This adapter is not trained. To load a trained adapter, check out [`PeftModel.load_adapter`]. The name for the new adapter should be unique. The new adapter is not automatically set as the active adapter. Use [`PeftModel.set_adapter`] to set the active adapter. Args: adapter_name (`str`): The name of the adapter to be added. peft_config ([`PeftConfig`]): The configuration of the adapter to be added. low_cpu_mem_usage (`bool`, `optional`, defaults to `False`): Create empty adapter weights on meta device. Useful to speed up the process when loading saved adapters. Don't use `low_cpu_mem_usage=True` when creating a new PEFT adapter for training (training is untested and discouraged for PeftMixedModel in general). )low_cpu_mem_usageN)r-r,r6inject_adapter Exceptionr7)r8r9r,rds r add_adapterzPeftMixedModel.add_adapters}6 !- -8D  \ * OO * *4Qb * c   l;  t///$$\2  s -A &A3c t|ddx}y|jt||_n|jj|t ||t|ddy)Nr5) module_names)r!r5setupdater)r8r,r9r5s rr7z"PeftMixedModel.set_modules_to_saves^&{4EtL LO U     '#&#7D  ' ' 8t\ M^`d8efr c Pt|tr|g}t|t|jj z }|r;t dt |dt |jj |jj|t||y)a Sets the active adapter(s) for the model. Note that the order in which the adapters are applied during the forward pass may not be the same as the order in which they are passed to this function. Instead, the order during the forward pass is determined by the order in which the adapters were loaded into the model. The active adapters only determine which adapters are active during the forward pass, but not the order in which they are applied. Additionally, this function will set the specified adapters to trainable (i.e., requires_grad=True). If this is not desired, use the following code. ```py >>> for name, param in model_peft.named_parameters(): ... if ...: # some check on name (ex. if 'lora' in name) ... param.requires_grad = False ``` Args: adapter_name (`str` or `List[str]`): The name of the adapter(s) to be activated. Adapter(s) not found, available adapters: N) isinstancestrrjr,keysr*sortedr6 set_adapterrr8r9 mismatcheds rrszPeftMixedModel.set_adapters, lC ((>L&T-=-=-B-B-D)EE fZ011QRXY]YiYiYnYnYpRqQrs  ##L1T<(r c 8t|tr|g}t|t|jj z }|r;t dt |dt |jj |jj|y)Nrmrn) rorprjr,rqr*rrr6delete_adapterrts rrwzPeftMixedModel.delete_adapter"s lC ((>L&T-=-=-B-B-D)EE fZ011QRXY]YiYiYnYnYpRqQrs  &&|4r c:|jj|i|S)a This method merges the adapter layers into the base model. This is needed if someone wants to use the base model as a standalone model. Args: progressbar (`bool`): whether to show a progressbar indicating the unload and merge process safe_merge (`bool`): whether to activate the safe merging check to check if there is any potential Nan in the adapter weights adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. )r6merge_and_unloadrYs rryzPeftMixedModel.merge_and_unload.s 0t//@@@r c:|jj|i|S)z Gets back the base model by removing all the adapter modules without merging. This gives back the original base model. )r6unloadrYs rr{zPeftMixedModel.unload?s &t%%t6v66r cHtd|jjd)Nz&get_layer_status is not supported for . TypeErrorr:rHr=s rget_layer_statuszPeftMixedModel.get_layer_statusF#@AXAX@YYZ[\\r cHtd|jjd)Nz&get_model_status is not supported for r}r~r=s rget_model_statuszPeftMixedModel.get_model_statusIrr c,tj|Sr<)r _split_kwargs)clsr[s rrzPeftMixedModel._split_kwargsLs&&v..r c2tj|||S)N) is_trainable)r_check_new_adapter_config)r8r,rs rrz(PeftMixedModel._check_new_adapter_configPs224S_``r crtj|||g|i|}|j|j|S)a Load a trained adapter into the model. The name for the new adapter should be unique. The new adapter is not automatically set as the active adapter. Use [`PeftModel.set_adapter`] to set the active adapter. Args: adapter_name (`str`): The name of the adapter to be added. peft_config ([`PeftConfig`]): The configuration of the adapter to be added. is_trainable (`bool`, *optional*, defaults to `False`): Whether the adapter should be trainable or not. If `False`, the adapter will be frozen and can only be used for inference. torch_device (`str`, *optional*, defaults to None): The device to load the adapter on. If `None`, the device will be inferred. autocast_adapter_dtype (`bool`, *optional*, defaults to `True`): Whether to autocast the adapter dtype. Defaults to `True`. Right now, this will only cast adapter weights using float16 and bfloat16 to float32, as this is typically required for stable training, and only affect select PEFT tuners. ephemeral_gpu_offload (`bool`, *optional*, defaults to `False`): Whether to use ephemeral GPU offloading for partially loaded modules. Defaults to `False`. low_cpu_mem_usage (`bool`, `optional`, defaults to `False`): Create empty adapter weights on meta device before loading the saved weights. Useful to speed up the process. kwargs: (`optional`): Additional arguments to modify the way the adapter is loaded, e.g. the token for Hugging Face Hub. )r load_adapterrsrA)r8model_idr9rZr[rs rrzPeftMixedModel.load_adapterSs<@''h VtVvV --. r cHtd|jjd)Nz)Model card creation is not supported for  (yet).NotImplementedErrorr:rH)r8 output_dirs rcreate_or_update_model_cardz*PeftMixedModel.create_or_update_model_cardxs#!$MdnnNeNeMffm"noor c Htd|jjd)NzSaving is not supported for rr)r8save_directorysafe_serializationselected_adaptersr[s rsave_pretrainedzPeftMixedModel.save_pretrained{s%"$@AXAX@YY`"abbr c ddlm}m}|o|tj||j dd|j dd|j dd|j ddj |fi|}n1t|tr | |_ntd |j|j|vrtd |jd t|d dKtt|jj!j#d dhdkDr t%||j&r |r td| |_||||}|j(||fd|i||S)a Instantiate a PEFT mixed model from a pretrained model and loaded PEFT weights. Note that the passed `model` may be modified inplace. Args: model (`nn.Module`): The model to be adapted. model_id (`str` or `os.PathLike`): The name of the PEFT configuration to use. Can be either: - A string, the `model id` of a PEFT configuration hosted inside a model repo on the Hugging Face Hub. - A path to a directory containing a PEFT configuration file saved using the `save_pretrained` method (`./my_peft_config_directory/`). adapter_name (`str`, *optional*, defaults to `"default"`): The name of the adapter to be loaded. This is useful for loading multiple adapters. is_trainable (`bool`, *optional*, defaults to `False`): Whether the adapter should be trainable or not. If `False`, the adapter will be frozen and use for inference config ([`~peft.PeftConfig`], *optional*): The configuration object to use instead of an automatically loaded configuration. This configuration object is mutually exclusive with `model_id` and `kwargs`. This is useful when configuration is already loaded before calling `from_pretrained`. low_cpu_mem_usage (`bool`, `optional`, defaults to `False`): Create empty adapter weights on meta device before loading the saved weights. Useful to speed up the process. kwargs: (`optional`): Additional keyword arguments passed along to the specific PEFT configuration class. r )PEFT_TYPE_TO_CONFIG_MAPPING PEFT_TYPE_TO_MIXED_MODEL_MAPPINGN subfolderrevision cache_diruse_auth_token)rrrrz+The input config must be a PeftConfig, got zAdapter of type z# is not supported for mixed models. hf_device_mapcpudiskrzRCannot set a prompt learning adapter to trainable when loading pretrained adapter.r)mappingrrr _get_peft_typegetfrom_pretrainedroinference_moder*r:r)r!lenrjrvalues intersectionris_prompt_learningr) rr$rr9rr1r[rrs rrzPeftMixedModel.from_pretrainedsP [ >0))$jjd;#ZZ D9$jjd;#)::.>#E oh2+12F +(4$4F !J6K[K[J\]^ ^   #C C/0@0@/AAdef f E?D 1 =3 ##**, - : :E6? KD D ( .  $ $qr r(4$4F !E6<08\W WPVW r )default)r$ nn.Moduler,r r9rpreturnNone)rzdict[str, PeftConfig])rrp)rz list[str])rWrp)rZrr[r)F)r9rpr,r rdboolrr)r,r r9rprr)r9zUnion[str, list[str]]rr)r[zdict[str, Any])r,r rrrr)rrpr9rprZrr[r)rrp)FN)rrprrrzOptional[list[str]]r[r)rFN) r$rrzstr | os.PathLiker9rprrr1zOptional[PeftConfig]r[r) rH __module__ __qualname____doc__r4propertyr,r?rArOrSrUr\r^rrbrgr7rsrwryr{rr classmethodrrrrrr __classcell__)r:s@rr/r/Csd D6 ++..//+2 (20 9 44%rsT# %'' 8-3!/Y. N^UXX__Nr