bi& `ddlZddlmZddlZdZdej dedej fdZdej dededej fd Z ddej ded ed dedej f d Z ddej d ed dej fdZ dej dej dej fdZ de ej dej dej fdZde ej dej dedej fdZ dde ej dej deded dej f dZde ej dej dedej fdZ dde ej dej deded dej f dZy)N)Literalc|jd|j|jz zz}|j|}|S)aI Reshapes `weights` to match the shape of `task_tensors` by unsqeezing in the remaining dimenions. Args: task_tensors (`torch.Tensor`): The tensors that will be used to reshape `weights`. weights (`torch.Tensor`): The tensor to be reshaped. Returns: `torch.Tensor`: The reshaped tensor. ))shapedimview) task_tensorsweights new_shapes Q/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/peft/utils/merge_utils.pyreshape_weight_task_tensorsr s>  (8(8(:W[[](J KKIll9%G Ntensordensityreturnc8tj|jd}t||j z}tj |j jd|d}d||d<||j|jzS)a^ Prune the smallest values of the task tensors and retain the top-k values based on the specified fraction `density`. Args: tensor (`torch.Tensor`):The tensor to prune. density (`float`):The fraction of values to preserve. Should be in [0,1]. Returns: `torch.Tensor`: The tensor with the pruned weights. T)klargestr)torch zeros_likereshapeintnumeltopkabsr)rrmaskrtop_ks r magnitude_based_pruningr%s|   F # + +B /D Gflln $%A JJvzz|++B/1d CEDqN DLL. ..rrescalectjtj||}||z}|rtj|||S)a Prune random values based on the specified fraction `density`. Args: tensor (`torch.Tensor`):The tensor to prune. density (`float`):The fraction of values to preserve. Should be in [0,1]. rescale (`bool`):Whether to rescale the result to preserve the expected value of the original tensor. Returns: `torch.Tensor`: The pruned tensor. )input fill_value)r"other)r bernoulli full_likediv)rrr r pruned_tensors r random_pruningr)8s= ??5??GL MDTMM W5 rmethod) magnituderandomc|dk\rtjd|d|S|dkrtd||dk(r t||S|dk(rt |||Std |) a Prune the values of task tensors based on the `method`. Args: tensor (`torch.Tensor`):The tensor to prune. density (`float`):The fraction of values to preserve. Should be in [0,1]. method (`str`):The method to use to prune. Should be one of ["magnitude", "random"]. rescale (`bool`):Whether to rescale the result to preserve the expected value of the original tensor. Returns: `torch.Tensor`: The pruned tensor. rz The density z= is greater than or equal to 1, no pruning will be performed.rzDensity should be >= 0, got r+r,)r zUnknown method )warningswarn ValueErrorrr))rrr*r s r pruner1Ks~!| WI-jkl 17yABB &vw77 8 fgw???6(344r)total frequencyc|j}|dk(r|jd}n'|dk(r|jd}ntd|dtj|dk\dd}||k(S) aZ Get the mask of the majority sign across the task tensors. Task tensors are stacked on dimension 0. Args: tensor (`torch.Tensor`):The tensor to get the mask from. method (`str`):The method to use to get the mask. Should be one of ["total", "frequency"]. Returns: `torch.Tensor`: The majority sign mask. r2rrr3zUnimplemented mask method ""rr)signsum RuntimeErrorrwhere)rr*r7sign_magnitude majority_signs r calculate_majority_sign_maskr=gsu ;;=D * ; a8BCCKK! 3Q;M =  rr majority_sign_maskc||zjd}|jd}|tj|dz S)a Merge the task tensors using disjoint merge. Args: task_tensors (`torch.Tensor`):The task tensors to merge. majority_sign_mask (`torch.Tensor`):The mask of the majority sign across the task tensors. Returns: `torch.Tensor`: The merged tensor. rr5g?)min)r8rclamp)r r>mixed_task_tensorsnum_params_preserveds r disjoint_mergerDsI');;@@Q@G-11a18  ,@c J JJrr cztj|d}t||}||z}|jd}|S)a Merge the task tensors using `task arithmetic`. Args: task_tensors(`List[torch.Tensor]`):The task tensors to merge. weights (`torch.Tensor`):The weights of the task tensors. Returns: `torch.Tensor`: The merged tensor. rr5)rstackr r8)r r weighted_task_tensorsrBs r task_arithmeticrHsE;;|3L),@G(72.22q29 rc|Dcgc]}t||d}}tj|d}t||}||z}|j d}|Scc}w)aX Merge the task tensors using `task arithmetic`. Args: task_tensors(`List[torch.Tensor]`):The task tensors to merge. weights (`torch.Tensor`):The weights of the task tensors. density (`float`): The fraction of values to preserve. Should be in [0,1]. Returns: `torch.Tensor`: The merged tensor. r+r*rr5r1rrFr r8r r rrrGrBs r magnitude_prunerMsiNZZ6E&'+>ZLZ;;|3L),@G(72.22q29  [sAmajority_sign_methodc|Dcgc]}t||d}}tj|d}t||}t ||}||z}t ||}|Scc}w)a Merge the task tensors using `ties`. Args: task_tensors(`List[torch.Tensor]`):The task tensors to merge. weights (`torch.Tensor`):The weights of the task tensors. density (`float`):The fraction of values to preserve. Should be in [0,1]. majority_sign_method (`str`): The method to use to get the majority sign mask. Should be one of ["total", "frequency"]. Returns: `torch.Tensor`: The merged tensor. r+rJrr5r1rrFr=r rDr r rrNrr>rGrBs r tiesrRsr(NZZ6E&'+>ZLZ;;|3L5lK_`),@G(72'(=?QR [sAc |Dcgc]}t||dd}}tj|d}t||}||z}|j d}|Scc}w)aS Merge the task tensors using `dare linear`. Args: task_tensors(`List[torch.Tensor]`):The task tensors to merge. weights (`torch.Tensor`):The weights of the task tensors. density (`float`):The fraction of values to preserve. Should be in [0,1]. Returns: `torch.Tensor`: The merged tensor. r,Tr*r rr5rKrLs r dare_linearrUskYeefE&'(DIeLe;;|3L),@G(72.22q29  fsAc |Dcgc]}t||dd}}tj|d}t||}t ||}||z}t ||}|Scc}w)a Merge the task tensors using `dare ties`. Args: task_tensors(`List[torch.Tensor]`):The task tensors to merge. weights (`torch.Tensor`):The weights of the task tensors. density (`float`):The fraction of values to preserve. Should be in [0,1]. majority_sign_method (`str`): The method to use to get the majority sign mask. Should be one of ["total", "frequency"]. Returns: `torch.Tensor`: The merged tensor. r,TrTrr5rJrPrQs r dare_tiesrWst(YeefE&'(DIeLe;;|3L5lK_`),@G(72'(=?QR fsA )F)r2)r.typingrrr Tensorfloatrboolr)r1r=rDlistrHrMrRrUrWrr r^s<   /ELL/5/U\\/&5<<%$5<<(ch5 LL5#(529:O2P5[_5 \\5:CJ! LL!")*>"?! \\!2 K K5<< KTYT`T` K $u||"4u||PUP\P\&$u||"4u||V[`e`l`l4;B u||$ \\""67   \\ @d5<<05<<RW\a\h\h4;B u||$ \\""67   \\ r