biM:ddlZddlZddlmZmZmZddlZddlmZddl mcm Z ddl m Z mZGdde ZGddej eZy)N)AnyOptionalUnion)BaseTunerLayercheck_adapters_to_mergeceZdZdZdZdej ddfdZdede d e ddfd Z defd Z defd Z defd ZdeddfdZdddZy) BoneLayer) bone_block)bone_r base_layerreturnNc T||_i|_tji|_d|_g|_d|_||_|j}t|tjr$|j|jc|_ |_ ytdt|)NFTzUnsupported layer type )r r nn ParameterDictr _disable_adaptersmerged_adapterscast_input_dtype_enabledkwargsget_base_layer isinstanceLinear in_features out_features ValueErrortype)selfr rs Q/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/peft/tuners/bone/layer.py__init__zBoneLayer.__init__s$ **2.!&!(,% ((* j")) ,2<2H2H*JaJa /D d/6tJ7G6HIJ J adapter_namer init_weightsc |dkrtd|||j|<|j}t|tj rCt j tj||jd|j|<n!tdt|j|dk(rB|j|zdk7s|j|zdk7r td|j||n&|r|j!||n|j#||j%||j'|j(y) zInternal function to create bone adapter Args: adapter_name (`str`): Name for the adapter to add. r (`int`): Rank for the added adapter. init_weights (`bool`): Whether to initialize weights. rz?`r` should be a positive integer value but the value passed is T requires_gradz0Bone is not implemented for base layers of type batz=The weight matrix must be fully divisible into [r, r] blocks.N)rr rrrr Parametertorchzerosrr TypeErrorr__name__rreset_bat_parametersreset_bone_parametersreset_bone_parameters_random%_move_adapter_to_device_of_base_layer set_adapteractive_adapters)rr r!r"rr s r update_layerzBoneLayer.update_layer0s' 6^_`^abc c$% L!((* j")) ,,.LLQHYHY9Zjn,oDOOL )NtT^O_OhOhNijk k 5 !#q(D,=,=,AQ,F !`aa  % %lA 6   & &|Q 7  - -l ; 22<@ --.rctjtj||jd|j |<yNTr$rr'r(r)rr rr r!s rr-zBoneLayer.reset_bone_parametersXs-(* U[[DDUDU5Vfj(k %rctjtj|j|z||d|j |<yr4r5r6s rr,zBoneLayer.reset_bat_parameters[s7(* U[[ARARVWAWYZ\]5^nr(s %rctjj|j|t j dy)N)a)rinitkaiming_uniform_r mathsqrt)rr s rr.z&BoneLayer.reset_bone_parameters_random^s*   !>$))A, Orscalec|dk(ry|jD]4}||jjvr tjd6y)NzGScaling operation for Bone not supported! Automatically set scale to 1.r1r keyswarningswarnrr?active_adapters r scale_layerzBoneLayer.scale_layerasH A: "22 eNT__%9%9%;; MMc d  erc|jD]4}||jjvr tjd6y)Nz?Unscaling operation for Bone not supported! Keeping scale at 1.rBrFs r unscale_layerzBoneLayer.unscale_layerks>"22 ]NT__%9%9%;; MM[ \  ]rNr N)r+ __module__ __qualname__adapter_layer_namesother_param_namesrModulerstrintboolr2r-r,r.floatrHrJrrr r s)#K299K4K"&/&/ &/ &/  &/Pl#lttPPee4e]rr c eZdZdZ ddededeeefddffd Zdded e e eddfd Z dd Z dd ede jfd Zdd ede jfdZde jdedede jfdZdeffd ZxZS) BoneLinearz, Bone implemented in a dense layer. r r!r"r Nc t|tj||fi|||_|j|||fi|||_yrK)superrr _active_adapterr2bone_fn)rr r r!r"r __class__s rrzBoneLinear.__init__xsM 46v6+,<B6B# r safe_merge adapter_namesct||}|sy|D]}||jjvs!|j}|jj }|r|jj j}|jdk(r|j||}||z }n2|j||jjj }|}tj|jstd|d|j!||j_n|jdk(r_|j||jjj }|jxj |j!|z c_nP|j||jjj }|j!||j_|j"j%|y)ab Merge the active adapter weights into the base weights Args: safe_merge (`bool`, *optional*): If `True`, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If `None`, all active adapters will be merged. Defaults to `None`. Nr&z1NaNs detected in the merged weights. The adapter z seems to be broken)rr rCrweightdtypedatacloner\get_delta_weightget_delta_weight_boner r(isfiniteallrtorappend)rr^r_rGr orig_dtype orig_weight delta_weights rmergezBoneLinear.merges0mD  + ">"@K||u,'+'<'<^['Y #|3 '+'A'A.RVRaRaRhRhRmRm'n &2  >>+6::<(OP^O__rs.9^^J-GJ%%*||u,'+'<'<^T__McMcMhMh'i "))..,//*2MM.'+'A'A.RVRaRaRhRhRmRm'n 1=1L )).$$++N;9 :D ! AvvbzA~*()AFF2JN(:a'? EE!a%67..q$**=A#eii  0\1773B<0\PRWX0\Z[0\bd&ehl&llF m>* rc*t|}d|zS)Nzbone.)rZ__repr__)rrepr]s rrzBoneLinear.__repr__]sg }r)rT)FNrL)F)r+rMrN__doc__rRrSrrTrrlistrnrtr(Tensorrerfrrr __classcell__)r]s@rrXrXss)- $ $  $ D#I& $  $.<.rsG '' LW]W]tlIlr