biddlmZddlZddlmZmZddlmZmZm Z ddl m Z ddl m Z ddlmZeGd d e Zy) ) annotationsN) dataclassfield)LiteralOptionalUnion) PeftConfig)PeftType) random_maskceZdZUdZedddiZded<eddd iZd ed <ed dd iZded<ed ddiZ ded<edddiZ ded<edddiZ ded<ed ddiZ ded<fdZ xZS) ShiraConfiga This is the configuration class to store the configuration of a [`ShiraModel`]. Args: r (`int`, *optional*, defaults to `32`): For a given target module, the number of SHiRA parameters is computed as r(m+n), where the original tensor dimensions are m x n. This means the number of SHiRA parameters is the same as that for a LoRA adapter. SHiRA is a high rank adapter. Setting this r parameter does not restrict the rank to this value. mask_type (`str`, defaults to `random`): Type of mask function. Defaults to a random sparse mask. An optional user-defined mask_fn to compute the mask value can also be supplied by instantiating `config = ShiraConfig(...)` and then setting `config.mask_fn = `. For a pretrained weight with shape m x n, the custom mask function must return only one mask (shape: m x n) which must be binary 0 or 1 with num_shira_parameters = r(m + n) for linear layers. Device and dtype of mask must be same as base layer's weight's device and dtype. Please see mask_functions.py for more details and to see the default random sparse mask implementation. random_seed (`int`, *optional*, defaults to `None`): random seed for the torch generator for random_mask. target_modules (`Union[List[str], str]`): List of module names or regex expression of the module names to replace with SHiRA. For example, ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$'. Only linear layers are supported. fan_in_fan_out (`bool`): Set this to True if the layer to replace stores weight like (fan_in, fan_out). For example, gpt-2 uses `Conv1D` which stores weights like (fan_in, fan_out) and hence this should be set to `True`. init_weights (`bool`, defaults to `True`): Initialize SHiRA weight to have zero values. If set to False, SHiRA weights are initialized to randn values instead of zeros and this is used only for testing. modules_to_save (`List[str]`): List of modules apart from SHiRA layers to be set as trainable and saved in the final checkpoint. helpa3For a given target module, the number of SHiRA parameters is computed as r(m+n), where the original tensor dimensions are m x n. This means the number of SHiRA parameters is the same as that for a LoRA adapter. SHiRA is a high rank adapter. Setting this r parameter does not restrict the rank to this value.)defaultmetadataintrrandomaoType of mask function. Defaults to a random sparse mask. An optional user-defined mask_fn to compute the mask value can also be supplied by instantiating `config = ShiraConfig(...)` and then setting `config.mask_fn = `. For a pretrained weight with shape m x n, the custom mask function must return only one mask (shape: m x n) which must be binary 0 or 1 with num_shira_parameters = r(m + n) for linear layers. Device and dtype of mask must be same as base layer's weight's device and dtype. Please see mask_functions.py for more details and to see the default random sparse mask implementation.zLiteral['random'] mask_typeNz3random seed for the torch generator for random_maskz Optional[int] random_seedzList of module names or regex expression of the module names to replace with SHiRA.For example, ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$'. Only linear layers are supported.zOptional[Union[list[str], str]]target_modulesFzMSet this to True if the layer to replace stores weight like (fan_in, fan_out)boolfan_in_fan_outTzInitialize SHiRA weight to have zero values. If set to False, SHiRA weights are initialized to randn values instead of zeros and this is used only for testing. init_weightsaList of modules apart from SHiRA layers to be set as trainable and saved in the final checkpoint. For example, in Sequence Classification or Token Classification tasks, the final layer `classifier/score` are randomly initialized and as such need to be trainable and saved.zOptional[list[str]]modules_to_savecnt|tj|_t |j trt|j n |j |_|jdk(r t|_ y|js#tjd|jdd|_ y)NrzArgument self.mask_type=ze is not recognized, please supply your own masking function by calling `config.mask_fn = my_mask_fn`.)super __post_init__r SHIRA peft_type isinstancerlistsetrr mask_fninference_modewarningswarn)self __class__s S/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/peft/tuners/shira/config.pyrzShiraConfig.__post_init__ts !(243F3F(MC## $SWSfSf  >>X %&DL&& /01VW DL)__name__ __module__ __qualname____doc__rr__annotations__rrrrrrr __classcell__)r*s@r+rrs> s  As $) z  $I "'(]^"K7< 4  7N3 !ijND v L$ ,1 k  ,O(    r,r) __future__rr' dataclassesrrtypingrrr peft.configr peft.utilsr mask_functionsr rr,r+r:s=#(++"' e *e  e r,