biD ddlZddlmZddlmZddlmZddlZddlmZddl m Z ddl m Z m Z ddl mZmZmZmZdd lmZmZmZmZd d lmZd d lmZd d lmZd dlmZd dlm Z d dl!m"Z"d dl#m$Z$d dl%m&Z&m'Z'GddeZ(y)N)asdict)Enum)Optional)nn)tqdm)is_bnb_4bit_availableis_bnb_available) BaseTunerBaseTunerLayercheck_target_module_exists onload_layer)2TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPINGModulesToSaveWrapper_get_submodulesget_quantization_config) dispatch_aqlm) dispatch_awq) OFTConfig) dispatch_eetq) dispatch_gptq) dispatch_hqq) dispatch_inc)OFTLayerdispatch_defaultc eZdZUdZdZeed<deddfdZe dZ d Z d Z d e jddfd Ze d Zdeffd Zd defdZd!dZd"dZdZdZfdZe dZ d#dededeeefdZdeddfdZ d$dededeeedejjfdZdejjfdZ xZ!S)%OFTModelaE Creates Orthogonal Finetuning model from a pretrained model. The method is described in https://huggingface.co/papers/2306.07280 Args: model (`torch.nn.Module`): The model to which the adapter tuner layers will be attached. config ([`OFTConfig`]): The configuration of the OFT model. adapter_name (`str`): The name of the adapter, defaults to `"default"`. low_cpu_mem_usage (`bool`, `optional`, defaults to `False`): Create empty adapter weights on meta device. Useful to speed up the loading process. Returns: `torch.nn.Module`: The OFT model. Example: ```py >>> from diffusers import StableDiffusionPipeline >>> from peft import OFTModel, OFTConfig >>> config_te = OFTConfig( ... r=8, ... target_modules=["k_proj", "q_proj", "v_proj", "out_proj", "fc1", "fc2"], ... module_dropout=0.0, ... init_weights=True, ... ) >>> config_unet = OFTConfig( ... r=8, ... target_modules=[ ... "proj_in", ... "proj_out", ... "to_k", ... "to_q", ... "to_v", ... "to_out.0", ... "ff.net.0.proj", ... "ff.net.2", ... ], ... module_dropout=0.0, ... init_weights=True, ... ) >>> model = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5") >>> model.text_encoder = OFTModel(model.text_encoder, config_te, "default") >>> model.unet = OFTModel(model.unet, config_unet, "default") ``` **Attributes**: - **model** ([`~torch.nn.Module`]) -- The model to be adapted. - **peft_config** ([`OFTConfig`]): The configuration of the OFT model. oft_prefixconfigreturnNct|jdkDr2|jdk7r"t|jj dyy)z A helper method to check the config when a new adapter is being added. Raise a ValueError if there is something wrong with the config or if it conflicts with existing adapters. rnonezf supports only 1 adapter with bias. When using multiple adapters, set bias to 'none' for all adapters.N)len peft_configbias ValueError __class____name__)selfr s P/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/peft/tuners/oft/model.py_check_new_adapter_configz"OFTModel._check_new_adapter_configfsR   !A %FKK6,A>>**+,77 -B %ct||SN)r ) oft_configkeys r+_check_target_module_existsz$OFTModel._check_target_module_existsus)*c::r-c | td|j|j|j|j|j |j |j|j|j|jt|jddt|jddd }gd} | D]$} t|j| } | | || d<&t|tsyt!|jd r|jj"nd} |j$|||fd | i|} ||j&vr| j)d|j+||| |y|j-||j|j|j|j|j |j |j|j|j y) NzCurrent Key shouldn't be `None`is_loaded_in_8bitFis_loaded_in_4bit) roft_block_sizemodule_dropoutcofteps block_shareuse_cayley_neumannnum_cayley_neumann_termsfan_in_fan_out init_weightsloaded_in_8bitloaded_in_4bit)gptqaqlmawq)method_quantization_config hf_device_map device_map) r6r7r8r9r:r;r<r=r?)r'r6r7r8r9r:r;r<r=r>r?getattrmodelr isinstancerhasattrrG_create_new_moduleactive_adaptersrequires_grad__replace_module update_layer)r*r0 adapter_nametarget target_nameparent current_keyoptional_kwargskwargs quant_methods quant_methodquantization_configrH new_modules r+_create_and_replacezOFTModel._create_and_replaceys  >? ?(77(77OO>>%11","?"?(2(K(K(77&33%djj2EuM%djj2EuM  0 ) TL"9$**\"Z ".@S,';<= T &(+5z+OFTModel._replace_module..sI188t+Is)setattrrLr_torchrg named_modulesrrbrcrdrIrenext parametersanyto) r*rU child_namer\childnamemodulerdr`s @r+rPzOFTModel._replace_modules J/ 5, '$$E||F#&446 -LD& t#d):5),"]]FUE*"YYFUH-"\\FU$4d;G"11F!%"2"2"45FIV5F5F5HIIIIfmm, -r-rJc|jD]\}}|j|vsd|_|jD]}|j|j }|dk(r"|dk(r%|jD]\}}d|vs d|_L|dk(rU|j D]A\}}t|tst|ds$|j 1d|j _Ctd|dy) NFr#allr&Toft_onlyzRequested bias: z, is not implemented.) named_parametersr requires_gradrNr%r&rmrKrrLNotImplementedError)r*rJnriactive_adapterr&rtms r+ _mark_only_adapters_as_trainablez)OFTModel._mark_only_adapters_as_trainables**, (DAq{{!#"' (#22 ZN##N388Dv~u}!224/DAq{*./#$2244GD!!!X.71f3E!&&J\/3,4*,aFa+/'( 0" C bs)A<c|jjD]*}t|ttfs|j |,yr/)rJmodulesrKr renable_adapters)r*enabledrus r+_set_adapter_layerszOFTModel._set_adapter_layerss<jj((* 0F&>3G"HI&&w/ 0r-c(|jdy)zyEnable all adapters. Call this if you have previously disabled all adapters and want to re-enable them. TrN)rr*s r+enable_adapter_layerszOFTModel.enable_adapter_layers#s    .r-c|jD]<}|j|j}|dk7s"d|d}tj|>|j dy)Nr#z>Careful, disabling adapter layers with bias configured to be 'zL' does not produce the same output as the base model would without adaption.Fr)rNr%r&warningswarnr)r*r}valmsgs r+disable_adapter_layerszOFTModel.disable_adapter_layers*sp"22 #N"">277Cf}TUXTYZGG c" #    /r-c|jjD]U}t|ts|jr%t j d|j|j|W||_ y)aSet the active adapter(s). Additionally, this function will set the specified adapters to trainable (i.e., requires_grad=True). If this is not desired, use the following code. ```py >>> for name, param in model_peft.named_parameters(): ... if ...: # some check on name (ex. if 'lora' in name) ... param.requires_grad = False ``` Args: adapter_name (`str` or `list[str]`): Name of the adapter(s) to be activated. zJAdapter cannot be set when the model is merged. Unmerging the model first.N) rJrrKrmergedrrunmerge set_adapterr})r*rRrus r+rzOFTModel.set_adapter5s`jj((* 1F&(+==MM"noNN$""<0  1 +r-ct|t|jdddk(r t d|j j dr t dy)zVerify that the configuration supports merging. Currently gptq quantization and replicated layers do not support merging. quantization_methodNrBz8Cannot merge OFT layers when the model is gptq quantizedlayer_replicationz=Cannot merge OFT layers when base model layers are replicated)r_check_merge_allowedrIrJr'r%get)r*r(s r+rzOFTModel._check_merge_allowedLsX $& 4::4d ;v EWX X      3 4\] ] 5r-c~|j0|dtvr tdtt|d|_|S)N model_typez0Please specify `target_modules` in `peft_config`)target_modulesrr'set)r% model_configs r+_prepare_adapter_configz OFTModel._prepare_adapter_configWsK  % % -L)1cc !STT),B&>&@[FCDKKWZDZC[[~B?'Ik/E _C .=djj#.N+ f% _6#GH&,&O&O# -'P'O((ovVV\2 - X((f>S>S>UW]^ _ _  _"zz'\ "   _ _s*D!D!-D'A9D6' D32D36D? rRc|t|jjvrtd|d|j|=|jj Dcgc]\}}|j |vs|}}}d}|D]P}t|j|\}}}t|ts.|j||B|jdd}R|xsg|_ |j||ycc}}w)z Deletes an existing adapter. Args: adapter_name (str): Name of the adapter to be deleted. zAdapter z does not existN)new_active_adapters)listr%keysr'rJrmrrrKrdelete_adapterrNr}_delete_auxiliary_adapter)r*rRr1rr new_adapterrSs r+rzOFTModel.delete_adapters tD$4$4$9$9$;< <x ~_EF F   \ *&*jj&>&>&@[FCDKKWZDZC[[  V3V4V0im  59 RZ[_`c[dRe  ,>>r-r))r dataclassesrenumrtypingrrlrrpeft.import_utilsrr peft.tuners.tuners_utilsr r r r peft.utilsrrrrrCrrDrr reetqrrBrhqqrincrlayerrrrr-r+rs[ E  -B>yB>r-