bi-ddlmZddlZddlZddlmZddlmZddlm Z ddl m Z ddl Z ddl m Z ddlmZmZmZdd lmZmZmZd d lmZd d lmZmZGd deZy)) annotationsN)asdict)Enum)chain)Optional)tqdm) BaseTunerBaseTunerLayercheck_target_module_exists)1TRANSFORMERS_MODELS_TO_C3A_TARGET_MODULES_MAPPINGModulesToSaveWrapper_get_submodules) C3AConfig)C3ALayer C3ALinearceZdZUdZdZded<ddZedZdZ dZ dd Z ed Z dfd Z ddd Zddd ZddZddZddZedZ d ddZ d d!dZd"dZxZS)#C3AModela@ Creates C3A model from a pretrained transformers model. The method is described in detail in [TODO]. Args: model ([`torch.nn.Module`]): The model to be adapted. config ([`C3AConfig`]): The configuration of the C3A model. adapter_name (`str`): The name of the adapter, defaults to `"default"`. Returns: `torch.nn.Module`: The C3A model. **Attributes**: - **model** ([`~transformers.PreTrainedModel`]) -- The model to be adapted. - **peft_config** ([`C3AConfig`]): The configuration of the C3A model. c3a_strprefixct|jdkDr2|jdk7r"t|jj dyy)z A helper method to check the config when a new adapter is being added. Raise a ValueError if there is something wrong with the config or if it conflicts with existing adapters. rnonezf supports only 1 adapter with bias. When using multiple adapters, set bias to 'none' for all adapters.N)len peft_configbias ValueError __class____name__)selfconfigs P/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/peft/tuners/c3a/model.py_check_new_adapter_configz"C3AModel._check_new_adapter_config:sR   !A %FKK6,A>>**+,77 -B %ct||SN)r ) c3a_configkeys r"_check_target_module_existsz$C3AModel._check_target_module_existsIs)*c::r$c  tdtt|jj }t t fd|} |jj| |j} | |jd} t|tr|j|| |jy|j|||fi| } ||jk7r| jd|j!||| |y)NzCurrent Key shouldn't be `None`c8tjd|dS)Nz.*\.$)rematch)r( current_keys r"z.C3AModel._create_and_replace..[s"((d3%q>;2Wr$) block_size init_weightsF)rlistrblock_size_patternkeysnextfiltergetr1r2 isinstancer update_layer_create_new_moduleactive_adapterrequires_grad__replace_module) r r' adapter_nametarget target_nameparentr/optional_kwargs pattern_keystarget_name_keyr1kwargs new_modules ` r"_create_and_replacezC3AModel._create_and_replaceMs  >? ?E*"?"?"D"D"FGH v&WYefhst2266 H]H]^ $&33  fi (   ''  100\6\U[\Jt222))%0  j& Ir$ct|||t|dr |j}t|ds.|j|_t|dr|j|_t |dd^t|dr|j |j_n|j |_|j|jjtjd|jD]\\}}|j|vstfd|jDr8|j|jj^y)N base_layerrstatemetac3<K|]}|jk(ywr&)device).0prLs r" z+C3AModel._replace_module..sI188t+Is)setattrhasattrrJweightrgetattrrKtorNtorch named_modulesrany parameters)r rB child_namerGchildnamemodulerLs @r"r>zC3AModel._replace_moduleps J/ 5, '$$Ez<0 % J uf%"'**  5'4 ( 4z<0.3kk %%+#(;;  MM%,,-- .||F#&446 3LD&{{d"IV5F5F5HIIIIell112 3r$c|jD]\}}|j|vsd|_|jD]}|j|j }|dk(r"|dk(r%|jD]\}}d|vs d|_L|dk(rR|j D]>}t|tst|ds!|j .d|j _@td|dy) NFrallrTc3a_onlyzRequested bias: z, is not implemented.) named_parametersr requires_gradactive_adaptersrrmodulesr9rrSNotImplementedError)r modelnrPr<rms r" _mark_only_adapters_as_trainablez)C3AModel._mark_only_adapters_as_trainables**, (DAq{{!#"' (#22 ZN##N388Dv~u}!224/DAq{*./#4A!!X.71f3E!&&J\/3,4*,aFa+/'( 0" C bs)A<c|jjD]*}t|ttfs|j |,yr&)rgrer9r r enable_adapters)r enabledr^s r"_set_adapter_layerszC3AModel._set_adapter_layerss<jj((* 0F&>3G"HI&&w/ 0r$c(|jdy)zyEnable all adapters. Call this if you have previously disabled all adapters and want to re-enable them. TrN)rr s r"enable_adapter_layerszC3AModel.enable_adapter_layerss    .r$c|jD]<}|j|j}|dk7s"d|d}tj|>|j dy)zDisable all adapters. When disabling all adapters, the model output corresponds to the output of the base model. rz>Careful, disabling adapter layers with bias configured to be 'zP' does not produce the same output as the the base model would without adaption.FrN)rdrrwarningswarnr)r r<valmsgs r"disable_adapter_layerszC3AModel.disable_adapter_layerssr #22 #N"">277Cf}TUXTYZKK c" #    /r$c|jjD]U}t|ts|jr%t j d|j|j|W||_ y)zSet the active adapter(s). Args: adapter_name (`str` or `list[str]`): Name of the adapter(s) to be activated. zJAdapter cannot be set when the model is merged. Unmerging the model first.N) rgrer9rmergedrrunmerge set_adapterr<)r r?r^s r"rzC3AModel.set_adapters` jj((* 1F&(+==MM"noNN$""<0  1 +r$c~|j0|dtvr tdtt|d|_|S)N model_typez0Please specify `target_modules` in `peft_config`)target_modulesr rset)r model_configs r"_prepare_adapter_configz C3AModel._prepare_adapter_configsJ  % % -L)1bb !STT),A,|B\]*K &r$c"|jjDcgc]\}}|j|vs|}}}d|rdndzdz}t|| |D]} t |j|\} } } t | dr8|r| j|||j| | | j| bt| tsst| | | j| j|jScc}}w#t $rYwxYw)Nz Unloading z and merging rg)disabledescrJ) safe_merge adapter_names)rgrXrrrrsrSmerger>rlr9r rRmodules_to_saver<) r r progressbarrrr(_key_listrrBr@rAs r"_unload_and_optionally_mergez%C3AModel._unload_and_optionally_merges '+jj&>&>&@[FCDKKWZDZC[[~B?'Ik/E \C .=djj#.N+ v|,LLJmLT$$V[&:O:O:QSYZF$89 V-C-CFDYDY-Z[ \zz!\ "  sC<C<D D Dc*|j|||S)a This method merges the C3A layers into the base model. This is needed if someone wants to use the base model as a standalone model. Args: progressbar (`bool`): whether to show a progressbar indicating the unload and merge process safe_merge (`bool`): whether to activate the safe merging check to check if there is any potential Nan in the adapter weights adapter_names (`list[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. )rrrr)r rrrs r"merge_and_unloadzC3AModel.merge_and_unload s#"00# -1  r$c&|jdS)z Gets back the base model by removing all the C3A modules without merging. This gives back the original base model. F)rrrs r"unloadzC3AModel.unloads 00u0==r$)r!rreturnNone)rgtorch.nn.Modulerr)r]r)F)rxbool)T)rrrr)rr)r?zstr | list[str]rr)TFFN)rrrrrOptional[list[str]])FFN)rrrrrrrr)rr)r __module__ __qualname____doc__r__annotations__r# staticmethodr)rHr>rjr;rrr|rrrrrrrr __classcell__)rs@r"rr%s$FC ;;!JF38Z*  -0 / 0 +! -1   + 4im  59 Re  *>r$r) __future__rr-r dataclassesrenumr itertoolsrtypingrrWrpeft.tuners.tuners_utilsr r r peft.utilsr r rr!rlayerrrrr$r"rsH#  ZZ &~>y~>r$