bihddlmZddlmZmZddlmZmZddlm Z ddl m Z eGdde Z y) ) annotations) dataclassfield)OptionalUnion) LycorisConfig)PeftTypeceZdZUdZedddiZded<edddiZded <ed dd iZd ed <ed ddiZ d ed<edddiZ ded<edddiZ ded<edddiZ ded<edddiZ ded<edddiZded<eddd iZded!<eddd"iZd#ed$<fd%ZxZS)& LoHaConfigaO This is the configuration class to store the configuration of a [`LoHaModel`]. Args: r (`int`): LoHa rank. alpha (`int`): The alpha parameter for LoHa scaling. rank_dropout (`float`): The dropout probability for rank dimension during training. module_dropout (`float`): The dropout probability for disabling LoHa modules during training. use_effective_conv2d (`bool`): Use parameter effective decomposition for Conv2d with ksize > 1 ("Proposition 3" from FedPara paper). target_modules (`Optional[Union[List[str], str]]`): The names of the modules to apply the adapter to. If this is specified, only the modules with the specified names will be replaced. When passing a string, a regex match will be performed. When passing a list of strings, either an exact match will be performed or it is checked if the name of the module ends with any of the passed strings. If this is specified as 'all-linear', then all linear/Conv1D modules are chosen, excluding the output layer. If this is not specified, modules will be chosen according to the model architecture. If the architecture is not known, an error will be raised -- in this case, you should specify the target modules manually. exclude_modules (`Optional[Union[List[str], str]]`): The names of the modules to not apply the adapter. When passing a string, a regex match will be performed. When passing a list of strings, either an exact match will be performed or it is checked if the name of the module ends with any of the passed strings. init_weights (`bool`): Whether to perform initialization of adapter weights. This defaults to `True`, passing `False` is discouraged. layers_to_transform (`Union[List[int], int]`): The layer indices to transform. If a list of ints is passed, it will apply the adapter to the layer indices that are specified in this list. If a single integer is passed, it will apply the transformations on the layer at this index. layers_pattern (`Optional[Union[List[str], str]]`): The layer pattern name, used only if `layers_to_transform` is different from `None`. This should target the `nn.ModuleList` of the model, which is often called `'layers'` or `'h'`. rank_pattern (`dict`): The mapping from layer names or regexp expression to ranks which are different from the default rank specified by `r`. For example, `{'^model.decoder.layers.0.encoder_attn.k_proj': 16}`. alpha_pattern (`dict`): The mapping from layer names or regexp expression to alphas which are different from the default alpha specified by `alpha`. For example, `{'^model.decoder.layers.0.encoder_attn.k_proj': 16}`. modules_to_save (`Optional[List[str]]`): List of modules apart from adapter layers to be set as trainable and saved in the final checkpoint. helpz LoHa rank)defaultmetadataintrz LoHa alphaalphagz:The dropout probability for rank dimension during trainingfloat rank_dropoutzBThe dropout probability for disabling LoHa modules during trainingmodule_dropoutFzhUse parameter effective decomposition for Conv2d 3x3 with ksize > 1 ("Proposition 3" from FedPara paper)booluse_effective_conv2dNa List of module names or regex expression of the module names to replace with LoHa.For example, ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$' This can also be a wildcard 'all-linear' which matches all linear/Conv1D layers except the output layer.zOptional[Union[list[str], str]]target_moduleszRList of module names or regex expression of the module names to exclude from LoHa.exclude_modulesTzWhether to initialize the weights of the LoHa layers with their default initialization. Don't change this setting, except if you know exactly what you're doing. init_weightszThe layer indexes to transform, is this argument is specified, PEFT will transform only the layers indexes that are specified inside this list. If a single integer is passed, PEFT will transform only the layer at this index.zOptional[Union[list[int], int]]layers_to_transformzThe layer pattern name, used only if `layers_to_transform` is different to None and if the layer pattern is not in the common layers pattern. This should target the `nn.ModuleList` of the model, which is often called `'layers'` or `'h'`.layers_patternaList of modules apart from LoHA layers to be set as trainable and saved in the final checkpoint. For example, in Sequence Classification or Token Classification tasks, the final layer `classifier/score` are randomly initialized and as such need to be trainable and saved.zOptional[list[str]]modules_to_savect|tj|_t |j trt|j n |j |_t |jtrt|jn |j|_ |jr|js tdyy)NzRWhen `layers_pattern` is specified, `layers_to_transform` must also be specified. ) super __post_init__r LOHA peft_type isinstancerlistsetrrr ValueError)self __class__s R/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/peft/tuners/loha/config.pyr zLoHaConfig.__post_init__s !(243F3F(MC## $SWSfSf *4D4H4H$)OC$$ %UYUiUi    t'?'?qr r(@ )__name__ __module__ __qualname____doc__rr__annotations__rrrrrrrrrrr __classcell__)r(s@r)r r sy,\1 '< =As=qFL+ABE3Bv'cdL%"v'klNE"' ~ "$ 7< w 7N38=no8O4 N L$r7s7#("3 ssss ssr*