bi"hddlmZddlmZmZddlmZmZddlm Z ddl m Z eGdde Z y) ) annotations) dataclassfield)OptionalUnion) PeftConfig)PeftTypecZeZdZUdZeddddZded<ed d d iZd ed <eddddZded<edd diZ ded<edd diZ d ed<edd diZ ded<edd diZ ded<edd diZ d ed!<edd d"iZd#ed$<fd%ZxZS)& HRAConfiga4 This is the configuration class to store the configuration of a [`HRAModel`]. Args: r (`int`): The rank of HRA across different layers. It is best to set 'r' to an even number; otherwise, the default initialization method will not work. apply_GS (`bool`): Whether to apply Gram-Schmidt orthogonalization. target_modules (`Optional[Union[List[str], str]]`): The names of the modules to apply the adapter to. If this is specified, only the modules with the specified names will be replaced. When passing a string, a regex match will be performed. When passing a list of strings, either an exact match will be performed or it is checked if the name of the module ends with any of the passed strings. If this is specified as 'all-linear', then all linear modules are chosen, excluding the output layer. If this is not specified, modules will be chosen according to the model architecture. If the architecture is not known, an error will be raised -- in this case, you should specify the target modules manually. exclude_modules (`Optional[Union[List[str], str]]`): The names of the modules to not apply the adapter. When passing a string, a regex match will be performed. When passing a list of strings, either an exact match will be performed or it is checked if the name of the module ends with any of the passed strings. init_weights (`bool`): Whether to perform initialization of HRA weights. layers_to_transform (`Union[List[int], int]`): The layer indices to transform. If a list of ints is passed, it will apply the adapter to the layer indices that are specified in this list. If a single integer is passed, it will apply the transformations on the layer at this index. layers_pattern (`Optional[Union[List[str], str]]`): The layer pattern name, used only if `layers_to_transform` is different from `None`. This should target the `nn.ModuleList` of the model, which is often called `'layers'` or `'h'`. modules_to_save (`List[str]`): List of modules apart from adapter layers to be set as trainable and saved in the final checkpoint. z(The rank of HRA across different layers.zdIt is best to set 'r' to an even number; otherwise, the default initialization method will not work.)helpnote)defaultmetadataintrFr z7Whether to apply Gram-Schmidt orthogonalization or not.boolapply_GSNzQList of module names or regex expression of the module names to replace with HRA.zPFor example, ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$' )r examplezOptional[Union[list[str], str]]target_moduleszQList of module names or regex expression of the module names to exclude from HRA.exclude_modulesTzWhether to initialize the weights of the HRA layers with their default initialization. Don't change this setting, except if you know exactly what you're doing. init_weightszThe layer indexes to transform, is this argument is specified, PEFT will transform only the layers indexes that are specified inside this list. If a single integer is passed, PEFT will transform only the layer at this index.zOptional[Union[list[int], int]]layers_to_transformzThe layer pattern name, used only if `layers_to_transform` is different to None and if the layer pattern is not in the common layers pattern. This should target the `nn.ModuleList` of the model, which is often called `'layers'` or `'h'`.layers_patternnonez5Bias type for HRA. Can be 'none', 'all' or 'hra_only'strbiasaList of modules apart from HRA layers to be set as trainable and saved in the final checkpoint. For example, in Sequence Classification or Token Classification tasks, the final layer `classifier/score` are randomly initialized and as such need to be trainable and saved.zOptional[list[str]]modules_to_savecXt|tj|_t |j trt|j n |j |_t |jtrt|jn |j|_ t |j tr|j tdt |j tr|j td|jr|js tdyy)NzD`layers_to_transform` cannot be used when `target_modules` is a str.z?`layers_pattern` cannot be used when `target_modules` is a str.zRWhen `layers_pattern` is specified, `layers_to_transform` must also be specified. )super __post_init__r HRA peft_type isinstancerlistsetrrr ValueErrorr)self __class__s Q/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/peft/tuners/hra/config.pyr!zHRAConfig.__post_init__rs !(243F3F(MC## $SWSfSf *4D4H4H$)OC$$ %UYUiUi  d))3 /D4L4L4Xcd d d))3 /D4G4G4S^_ _   t'?'?qr r(@ )__name__ __module__ __qualname____doc__rr__annotations__rrrrrrrrr! __classcell__)r)s@r*r r sC D>z AsSTHd7<gi 7N38=mn8O4 N L$r8s7#("" ls ls lsr+