biaRddlmZddlZddlZddlmZmZddlmZddl m Z ddl Z ddl m Z ddl mZddlmZmZdd lmZmZmZdd lmZmZmZmZmZd d lmZmZmZm Z Gd deZ!y)) annotationsN)asdictreplace)Enum)Optional)nn)Conv1D)is_bnb_4bit_availableis_bnb_available) BaseTunerBaseTunerLayercheck_target_module_exists)6TRANSFORMERS_MODELS_TO_IA3_FEEDFORWARD_MODULES_MAPPING1TRANSFORMERS_MODELS_TO_IA3_TARGET_MODULES_MAPPINGModulesToSaveWrapper_freeze_adapter_get_submodules)Conv2dConv3dIA3LayerLinearceZdZUdZdZded<edZedZddZ dZ edd Z d Z dfd Z ddd Zdd ZddZddZddZedZ d d!dZd"d#dZd$dZd%dZd&dZ d'dZxZS)(IA3ModelaP Creates a Infused Adapter by Inhibiting and Amplifying Inner Activations ((IA)^3) model from a pretrained transformers model. The method is described in detail in https://huggingface.co/papers/2205.05638 Args: model ([`~transformers.PreTrainedModel`]): The model to be adapted. config ([`IA3Config`]): The configuration of the (IA)^3 model. adapter_name (`str`): The name of the adapter, defaults to `"default"`. low_cpu_mem_usage (`bool`, `optional`, defaults to `False`): Create empty adapter weights on meta device. Useful to speed up the loading process. Returns: `torch.nn.Module`: The (IA)^3 model. Example: ```py >>> from transformers import AutoModelForSeq2SeqLM, ia3Config >>> from peft import IA3Model, IA3Config >>> config = IA3Config( ... peft_type="IA3", ... task_type="SEQ_2_SEQ_LM", ... target_modules=["k", "v", "w0"], ... feedforward_modules=["w0"], ... ) >>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base") >>> ia3_model = IA3Model(config, model) ``` **Attributes**: - **model** ([`~transformers.PreTrainedModel`]) -- The model to be adapted. - **peft_config** ([`ia3Config`]): The configuration of the (IA)^3 model. ia3_strprefixc Ptr ddl}ddlm}t rddlm}|j dd}|j dd}|j dd} t|tr|j} n|} |rt| jjre|j} | j| jj| jj| j d ||fd| i| } | S|rt| jj re|j} | j| j"| j$j&| j$j(d ||fd| i| } | St|t*jj,rt-||fd| i|} | St|t*jj.rt/||fd| i|} | St| t*jj0r7|d r!t3j4d dx|d <|_t1||fd| i|} | St| t8r8|d s!t3j4d dx|d <|_t1||f| dd|} | St;d|d)Nrr) Linear8bitLt) Linear4bitloaded_in_8bitFloaded_in_4bitis_feedforward)has_fp16_weights thresholdindex) compute_dtypecompress_statistics quant_typefan_in_fan_outzjfan_in_fan_out is set to True but the target module is `torch.nn.Linear`. Setting fan_in_fan_out to False.zafan_in_fan_out is set to False but the target module is `Conv1D`. Setting fan_in_fan_out to True.T)r#is_target_conv_1d_layerzTarget module zd is not supported. Currently, only `torch.nn.Linear`, `torch.nn.Conv2d`, and `Conv1D` are supported.)r bitsandbytesbnbrr r pop isinstancer get_base_layerrcopyupdatestater$r%r&r'weightr(r)torchrrrwarningswarnr*r ValueError) ia3_config adapter_nametargetkwargsr-rr r!r"r#target_base_layereightbit_kwargs new_modulefourbit_kwargss P/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/peft/tuners/ia3/model.py_create_new_modulezIA3Model._create_new_moduleNs   & ) " '$4e<$4e<$4e< fn - & 5 5 7  &  j):CFFm]lmJJI +O>O P#[[]N  ! !%6%D%D+<+C+C+W+W"3":":"E"E $FLjj[ijJ65 0 ^^^W]^J21 0 ^^^W]^J.-)588?? ;&' 7HML'(:+D ^^^W]^J)6 2*+ wHLK'(:+D 5C]aekJ  )de ct||SN)r)r9keys rA_check_target_module_existsz$IA3Model._check_target_module_existss)*c::rCc`|jD]\}}|j|vsd|_y)NF)named_parametersr requires_grad)selfmodelnps rA _mark_only_adapters_as_trainablez)IA3Model._mark_only_adapters_as_trainables1**, (DAq{{!#"' (rCc |j||}|j|j|t|jddt|jddd}t |t r|j||jy|j|||fi|} ||jvr| jd|j||| |y)Nis_loaded_in_8bitFis_loaded_in_4bit)r*init_ia3_weightsr#r!r") _check_target_module_feedforwardr*rSgetattrrLr/r update_layerrBactive_adaptersrequires_grad__replace_module) rKr9r:r; target_nameparent current_keyr#r<r?s rA_create_and_replacezIA3Model._create_and_replaces>>z;W)77 * ; ;,%djj2EuM%djj2EuM   fh '   ++  100\6\U[\J4#7#77))%0  j& IrCct|jtr+tt j |j}|St fd|jD}|S)z A helper private method that checks if the target module `key` matches with a feedforward module specified in `ia3_config` c3@K|]}j|ywrE)endswith).0 target_keyrFs rA z.s kjj!9 ks)r/feedforward_modulesrboolre fullmatchany)r9rFr#s ` rArTz)IA3Model._check_target_module_feedforwardsV j44c :!",,z/M/Ms"STN! kJLjLj kkNrCct|||t|dr |j}t|ds.|j|_t|dr|j|_t |dd^t|dr|j |j_n|j |_|j|jjtjd|jD]\\}}|j|vstfd|jDr8|j|jj^y)N base_layerbiasr3metac3<K|]}|jk(ywrE)device)rarNrls rArcz+IA3Model._replace_module..sI188t+Is)setattrhasattrrjr4rkrUr3tornr5 named_modulesrrh parameters)rKr[ child_namer?childnamemodulerls @rArYzIA3Model._replace_modules J/ 5, '$$Ez<0 % J uf%"'**  5'4 ( 4z<0.3kk %%+#(;;  MM%,,-- .||F#&446 3LD&{{d"IV5F5F5HIIIIell112 3rCcz t||S#t$r|dk(rt|j|cYSwxYw)z1Forward missing attributes to the wrapped module.rL)super __getattr__AttributeErrorrUrL)rKrv __class__s rArzzIA3Model.__getattr__sB -7&t, , -w4::t, , -s %::c i}|jjD]U\}}t|jDcic]$\}}|t|tr |j n|&}}}|sQd|d<W|<|Scc}}w)NTinference_mode) peft_configitemsrr/rvalue)rK inference config_dictrFrkvconfigs rAget_peft_config_as_dictz IA3Model.get_peft_config_as_dicts **002 0JCKQRW=K^K^K`a41aaJq$$7Q>aFa+/'( 0" C bs)A<c|jjD]*}t|ttfs|j |,yrE)rLmodulesr/rrenable_adapters)rKenabledrws rA_set_adapter_layerszIA3Model._set_adapter_layerss<jj((* 0F&8-A"BC&&w/ 0rCc(|jdy)zyEnable all adapters. Call this if you have previously disabled all adapters and want to re-enable them. TrNrrKs rAenable_adapter_layerszIA3Model.enable_adapter_layerss    .rCc(|jdy)zDisable all adapters. When disabling all adapters, the model output corresponds to the output of the base model. FrNrrs rAdisable_adapter_layerszIA3Model.disable_adapter_layerss    /rCc|jjD]U}t|ts|jr%t j d|j|j|W||_ y)aSet the active adapter(s). Additionally, this function will set the specified adapters to trainable (i.e., requires_grad=True). If this is not desired, use the following code. ```py >>> for name, param in model_peft.named_parameters(): ... if ...: # some check on name (ex. if 'lora' in name) ... param.requires_grad = False ``` Args: adapter_name (`str` or `list[str]`): Name of the adapter(s) to be activated. zJAdapter cannot be set when the model is merged. Unmerging the model first.N) rLrr/rmergedr6r7unmerge set_adapteractive_adapter)rKr:rws rArzIA3Model.set_adapters`jj((* 1F&(+==MM"noNN$""<0  1 +rCc|j0|dtvr tdtt|d|_|j0|dt vr tdtt |d|_|S)N model_typez0Please specify `target_modules` in `peft_config`z5Please specify `feedforward_modules` in `peft_config`)target_modulesrr8setrdr)r model_configs rA_prepare_adapter_configz IA3Model._prepare_adapter_configs  % % -L)1bb !STT),A,|B\]*K &  * * 2L)1gg !XYY.1F|T`Gab/K +rCct|jddr tdt|jddr td|j||jj Dcgc]\}}|j |vs|}}}|D]} t |j|\}}} t|dr8|r|j|||j|| |j|bt|tss|j|j} t| dr%|r| j||| j} t!|| | |jScc}}w#t$rYwxYw)a This method merges the (IA)^3 layers into the base model. This is needed if someone wants to use the base model as a standalone model. Args: safe_merge (`bool`, `optional`, defaults to `False`): If True, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. rQFz>Cannot merge ia3 layers when the model is loaded in 8-bit moderRz>Cannot merge ia3 layers when the model is loaded in 4-bit moderj safe_merge adapter_names)rUrLr8_unloading_checksrrrrr{rpmergerYr0r/rmodules_to_saverro) rKrrrrF_key_listr[r;rZr?s rA_unload_and_optionally_mergez%IA3Model._unload_and_optionally_merge.se 4::2E :]^ ^ 4::2E :]^ ^ }-&*jj&>&>&@[FCDKKWZDZC[[ 9C .=djj#.N+ v|,LLJmLT$$V[&:O:O:QSYZF$89#33F4I4IJ :|4"((Jm(\!+!:!:!>> from transformers import AutoModelForCausalLM >>> from peft import PeftModel >>> base_model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-40b") >>> peft_model_id = "smangrul/falcon-40B-int4-peft-lora-sfttrainer-sample" >>> model = PeftModel.from_pretrained(base_model, peft_model_id) >>> merged_model = model.merge_and_unload() ``` rr)rKrrs rAmerge_and_unloadzIA3Model.merge_and_unload\s200JVc0ddrCc&|jdS)u Gets back the base model by removing all the IA³ modules without merging. This gives back the original base model. F)rrrs rAunloadzIA3Model.unloadws 00u0==rCc||jvrtd|d|j|=|jjDcgc]\}}|j|vs|}}}d}|D]P}t |j|\}}}t |ts.|j||B|jdd}R|xsg|_ |j||ycc}}w)z Deletes an existing adapter. Args: adapter_name (str): Name of the adapter to be deleted. Adapter  does not existN)new_active_adapters) rr8rLrrrrr/rdelete_adapterrWr_delete_auxiliary_adapter)rKr:rFrr new_adapterr;s rArzIA3Model.delete_adapter~s t// /x ~_EF F   \ *&*jj&>&>&@[FCDKKWZDZC[[  ..sKw7222KsrN)sum)raradapterss @rArcz7IA3Model._check_add_weighted_adapter..s& T[CK(K Ka O s!%zLCannot add weighted adapters targeting the same module with modules_to_save.rzQAll adapter configs should have the same type for target and feedforward modules.|c3XK|]!}dj|jd#yw()NrrrarrKs rArcz7IA3Model._check_add_weighted_adapter..s.)r^eAd.>.>w.G.V.V-WWX*Y)r'*c3PK|]}j|jywrErrs rArcz7IA3Model._check_add_weighted_adapter..s#,nZaT-=-=g-F-U-U,n#&c3XK|]!}dj|jd#ywrrrdrs rArcz7IA3Model._check_add_weighted_adapter..s1/IP!D$$W-AAB!D/rc3PK|]}j|jywrErrs rArcz7IA3Model._check_add_weighted_adapter..s"XG$""7+??Xr)rr8rr/rrhtyperrdlenrjoinrunion) rKrrrwmodules_to_save_wrapperstarget_module_typesfeedforward_module_typesnew_target_modulesnew_feedforward_moduless `` rA_check_add_weighted_adapterz$IA3Model._check_add_weighted_adapters   FGd... 8G9O!DEE F :>#tv:V\^rKsF#t #t  _w  kl l^ffRYtD$4$4W$=$L$LMffgo#p\cD)9)9')B)V)V$W#p #p " #a '3/G+H1+Lpq q % %!$)riq)r!r !$,nem,n!o  * *&)hh/T\/' #"#::: '*iiXxX' #"#:::7$ug#psE<E<8)F')Fc@|t|jjvry|j|\}}t |j|d|||j|<|j |j |t|j ||j jDcgc]\}}|j|vs|}}}|D]}t|j |\}} }t| ts.|| jvr| j|} nM| jj| _t!||D]F\} } | | jvr| j| } n%| xj| j| zz c_Hycc}}w)ac This method adds a new adapter by merging the given adapters with the given weights. Args: adapters (`list`): List of adapter names to be merged. weights (`list`): List of weights for each adapter. adapter_name (`str`): Name of the new adapter. N)rr)rrd)listrkeysrrinject_adapterrLrrrrrr/ria3_ldatazero_zip)rKrweightsr:rrrFrrr; target_ia3_lrr4current_adapter_ia3_ls rAadd_weighted_adapterzIA3Model.add_weighted_adapters" 4 0 0 5 5 78 8 6:6V6V7W7 33*1   Xa[ )- 7* & DJJ 5  L1&*jj&>&>&@[FCDKKWZDZC[[ MC*4::s;LAvq&(+6<</#)<< #=L$0$5$5$;$;$= !'*8W'=MOGV&,,.06 W0E-  %%)>)C)Cf)LL% M M\s 6F F)rLz nn.ModulereturnNone)rre)rvr)F)rre)T)rr)r:zstr | list[str]rr)TFN)rrerrerOptional[list[str]])FN)rrerrrtorch.nn.Module)rr)r:rrr)r list[str]rztuple[str, str])rrrz list[float]r:rrr)__name__ __module__ __qualname____doc__r__annotations__ staticmethodrBrGrOr]rTrYrzrrrrrrrrrrrr __classcell__)r|s@rArr's"HFCAAF;;( J@  36-0 /0+.   bf,,.2,K^,\e6>V.&;P1M1M1M 1M  1MrCr)" __future__rrfr6 dataclassesrrenumrtypingrr5rtransformers.pytorch_utilsr peft.import_utilsr r peft.tuners.tuners_utilsr r r peft.utilsrrrrrlayerrrrrrrCrArsM# ' -EZZ43GMyGMrC