biL)hddlmZddlmZmZddlmZmZddlm Z ddl m Z eGdde Z y) ) annotations) dataclassfield)OptionalUnion) PeftConfig)PeftTypec"eZdZUdZedddiZded<eddd iZded <eddd iZded <ed ddiZ ded<edddiZ ded<edddiZ ded<edddiZ ded<edddiZ ded<edddiZded<ed dd!iZd"ed#<eddd$iZd%ed&<ed'dd(iZded)<ed*dd+iZded,<eddd-iZd.ed/<eddd0iZded1<fd2ZxZS)3 VBLoRAConfiga This is the configuration class to store the configuration of a [`VBLoRAConfig`]. Paper: https://huggingface.co/papers/2405.15179 Args: r (`int`): The rank of incremental matrices. num_vectors (`int`): Number of vectors in the vector bank. Use higher values when the model size increases. vector_length (`int`): The length of the vectors in the vector bank. The length of the vectors should be divisible by the hidden dimension of the model. topk (`int`): The K value for top-K selection. A larger value of K increases the size of the saved model. In practice, setting K=2 typically provides the best performance and parameter efficiency. For more details, refer to the discussion in the paper. target_modules (`Union[List[str], str]`): The names of the modules to apply the adapter to. If this is specified, only the modules with the specified names will be replaced. When passing a string, a regex match will be performed. When passing a list of strings, either an exact match will be performed or it is checked if the name of the module ends with any of the passed strings. If this is specified as 'all-linear', then all linear/Conv1D modules are chosen, excluding the output layer. If this is not specified, modules will be chosen according to the model architecture. If the architecture is not known, an error will be raised -- in this case, you should specify the target modules manually. exclude_modules (`Optional[Union[List[str], str]]`): The names of the modules to not apply the adapter. When passing a string, a regex match will be performed. When passing a list of strings, either an exact match will be performed or it is checked if the name of the module ends with any of the passed strings. save_only_topk_weights (`bool`): Whether to only save the topk weights. Setting `save_only_topk_weights = True` significantly reduces storage space. However, models saved in this mode can be used for merging or inference only, not for resuming training. vblora_dropout (`float`): The dropout probability for VBLoRA layers. fan_in_fan_out (`bool`): Set this to True if the layer to replace stores weight like (fan_in, fan_out). For example, gpt-2 uses `Conv1D` which stores weights like (fan_in, fan_out) and hence this should be set to `True`. bias (`str`): Bias type for VBLoRA. Can be 'none', 'all' or 'vblora_only'. If 'all' or 'vblora_only', the corresponding biases will be updated during training. Be aware that this means that, even when disabling the adapters, the model will not produce the same output as the base model would have without adaptation. modules_to_save (`List[str]`): List of modules apart from VBLoRA layers to be set as trainable and saved in the final checkpoint. init_vector_bank_bound (`float`): The vector bank is initialized with a uniform distribution between -init_vector_bank_bound and init_vector_bank_bound. Avoid initializing the vector bank with all zeros to prevent zero gradients. A small value, such as 0.02, is typically effective. Initializing with a large value may cause training instability. init_logits_std (`float`): The logits are initialized with a normal distribution with a standard deviation of init_logits_std. Default is 0.1. layers_to_transform (`Union[List[int],int]`): The layer indices to transform. If a list of ints is passed, it will apply the adapter to the layer indices that are specified in this list. If a single integer is passed, it will apply the transformations on the layer at this index. layers_pattern (`Optional[Union[List[str], str]]`): The layer pattern name, used only if `layers_to_transform` is different from `None`. This should target the `nn.ModuleList` of the model, which is often called `'layers'` or `'h'`. helpz!The rank of incremental matrices.)defaultmetadataintrzVNumber of vectors in the vector bank. Use higher values when the model size increases. num_vectorszThe length of the vectors in the vector bank. The length of the vectors should be divisible by the hidden dimension of the model. vector_lengthzThe K value for top-K selection. A larger value of K increases the size of the saved model. In practice, setting K=2 typically provides the best performance and parameter efficiency. For more details, refer to the discussion in the paper.topkNaList of module names or regex expression of the module names to replace with LoRA.For example, ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$'.This can also be a wildcard 'all-linear' which matches all linear/Conv1D layers except the output layer.If not specified, modules will be chosen according to the model architecture, If the architecture is not known, an error will be raised -- in this case, you should specify the target modules manually.zOptional[Union[list[str], str]]target_moduleszTList of module names or regex expression of the module names to exclude from VBLoRA.exclude_modulesFzWhether to only save the topk weights. Setting `save_only_topk_weights = True` significantly reduces storage space. However, models saved in this mode can be used for merging or inference only, not for resuming training.boolsave_only_topk_weightsgzVBLoRA dropoutfloatvblora_dropoutzMSet this to True if the layer to replace stores weight like (fan_in, fan_out)fan_in_fan_outnonez;Bias type for VBLoRA. Can be 'none', 'all' or 'vblora_only'strbiasaList of modules apart from VBLoRA layers to be set as trainable and saved in the final checkpoint. For example, in Sequence Classification or Token Classification tasks, the final layer `classifier/score` are randomly initialized and as such need to be trainable and saved.zOptional[list[str]]modules_to_saveg{Gz?a8The vector bank is initialized with a uniform distribution between -init_vector_bank_bound and init_vector_bank_bound. Avoid initializing the vector bank with all zeros to prevent zero gradients. A small value, such as 0.02, is typically effective. Initializing with a large value may cause training instability.init_vector_bank_boundg?zThe logits are initialized with a normal distribution with a standard deviation of init_logits_std. Default value 0.1 typically works well.init_logits_stdavThe layer indexes to transform, is this argument is specified, PEFT will transform only the layers indexes that are specified inside this list. If a single integer is passed, PEFT will transform only the layer at this index. This only works when target_modules is a list of str. This should target the `nn.ModuleList` of the model, which is often called `'layers'` or `'h'`.zOptional[Union[list[int], int]]layers_to_transformzThe layer pattern name, used only if `layers_to_transform` is different to None and if the layer pattern is not in the common layers pattern.This only works when target_modules is a list of str.layers_patternct|tj|_t |j trt|j n |j |_t |jtrt|jn |j|_ |jr|js tdyy)NzRWhen `layers_pattern` is specified, `layers_to_transform` must also be specified. ) super __post_init__r VBLORA peft_type isinstancerlistsetrr%r$ ValueError)self __class__s T/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/peft/tuners/vblora/config.pyr(zVBLoRAConfig.__post_init__s !(243F3F(MC## $SWSfSf *4D4H4H$)OC$$ %UYUiUi    t'?'?qr r(@ )__name__ __module__ __qualname____doc__rr__annotations__rrrrrrrrr r!r"r#r$r%r( __classcell__)r0s@r1r r s;z10S'T UAsUrsK 1 M3 F D#7< v  7N3 8=pq8O4$) %  $D "#AQ8RSNES ijNDf8u/vwD#w+0 k  ,O( %* )  %E # : OUr?s7#("" ks:ks ksr2