bi hddlmZddlmZmZddlmZmZddlm Z ddl m Z eGdde Z y) ) annotations) dataclassfield)OptionalUnion) PeftConfig)PeftTypeceZdZUdZeddddZded<ed d ddZded <ed d ddZded<eddddZ ded<edddiZ ded<edddiZ ded<edddiZ ded <ed!dd"iZ d#ed$<edd%d&dZd'ed(<ed)dd*iZded+<eddd,iZd-ed.<eddd/iZded0<fd1ZxZS)2 BOFTConfigaf This is the configuration class to store the configuration of a [`BOFTModel`]. Args: boft_block_size (`int`): BOFT block size across different layers. boft_block_num (`int`): Number of BOFT blocks per injected layer. boft_n_butterfly_factor (`int`): Number of butterfly factors across different layers. target_modules (`Union[List[str],str]`): The names of the modules to apply the adapter to. exclude_modules (`Optional[Union[List[str], str]]`): The names of the modules to not apply the adapter. When passing a string, a regex match will be performed. When passing a list of strings, either an exact match will be performed or it is checked if the name of the module ends with any of the passed strings. boft_dropout (`float`): The multiplicative dropout probability, by setting OFT blocks to identity during training, similar to the dropout layer in LoRA. fan_in_fan_out (`bool`): Set this to True if the layer to replace stores weight like (fan_in, fan_out). For example, gpt-2 uses `Conv1D` which stores weights like (fan_in, fan_out) and hence this should be set to `True`. bias (`str`): Bias type for BOFT. Can be 'none', 'all' or 'boft_only'. If 'all' or 'boft_only', the corresponding biases will be updated during training. Be aware that this means that, even when disabling the adapters, the model will not produce the same output as the base model would have without adaptation. modules_to_save (`List[str]`):List of modules apart from BOFT layers to be set as trainable and saved in the final checkpoint. layers_to_transform (`Union[List[int],int]`): The layer indexes to transform, if this argument is specified, it will apply the BOFT transformations on the layer indexes that are specified in this list. If a single integer is passed, it will apply the BOFT transformations on the layer at this index. layers_pattern (`Optional[Union[List[str], str]]`): The layer pattern name, used only if `layers_to_transform` is different from `None` and if the layer pattern is not in the common layers pattern. This should target the `nn.ModuleList` of the model, which is often called `'layers'` or `'h'`. z(BOFT block size across different layers.zYou can only specify either boft_block_size or boft_block_num, but not both simultaneously, because boft_block_size x boft_block_num = layer dimension.)helpnote)defaultmetadataintboft_block_sizerz)Number of BOFT blocks per injected layer.boft_block_numzNumber of butterfly factors.)zfor example, boft_n_butterfly_factor=2, the effective block size of OFT becomes twice as big and the number of blocks become half.zEnote: for boft_n_butterfly_factor=1, BOFT is the same as vanilla OFT.boft_n_butterfly_factorNzRList of module names or regex expression of the module names to replace with BOFT.zPFor example, ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$' )r examplezOptional[Union[list[str], str]]target_modulesr zRList of module names or regex expression of the module names to exclude from BOFT.exclude_modulesgzxBOFT multiplicative dropout, randomly setting blocks of OFT to be identity matrix, similar to the dropout layer in LoRA.float boft_dropoutFzMSet this to True if the layer to replace stores weight like (fan_in, fan_out)boolfan_in_fan_outnonez7Bias type for BOFT. Can be 'none', 'all' or 'boft_only'strbiaszaList of modules apart from BOFT layers to be set as trainable and saved in the final checkpoint. )zGFor example, in Sequence Classification or Token Classification tasks, zgthe final layer `classifier/score` are randomly initialized and as such need to be trainable and saved.zOptional[list[str]]modules_to_saveT)zeWhether to initialize the weights of the BOFT layers with their default initialization. Don't change z;this setting, except if you know exactly what you're doing. init_weightszThe layer indexes to transform, is this argument is specified, PEFT will transform only the layers indexes that are specified inside this list. If a single integer is passed, PEFT will transform only the layer at this index.zOptional[Union[list[int], int]]layers_to_transformzThe layer pattern name, used only if `layers_to_transform` is different to None and if the layer pattern is not in the common layers pattern. This should target the `nn.ModuleList` of the model, which is often called `'layers'` or `'h'`.layers_patternct|tj|_t |j trt|j n |j |_t |jtrt|jn |j|_ |jr|js td|jdk(r5|jdk(r&td|jd|jd|jdk7|jdk7z s&td|jd|jdy) NzRWhen `layers_pattern` is specified, `layers_to_transform` must also be specified. rz\Either `boft_block_size` or `boft_block_num` must be non-zero. Currently, boft_block_size = z and boft_block_num = .z-You can only specify either boft_block_size (z) or boft_block_num (zX), but not both simultaneously, because boft_block_size x boft_block_num == in_features.)super __post_init__r BOFT peft_type isinstancerlistsetrr#r" ValueErrorrr)self __class__s R/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/peft/tuners/boft/config.pyr'zBOFTConfig.__post_init__sf !(243F3F(MC## $SWSfSf *4D4H4H$)OC$$ %UYUiUi    t'?'?qr r   1 $)<)<)AnospDpDoEE[\`\o\o[ppqr $$)d.A.AQ.FG?@T@T?UUjkok~k~j@XY H)__name__ __module__ __qualname____doc__rr__annotations__rrrrrrrr r!r"r#r' __classcell__)r/s@r0r r sB!>n OS ?n NC$)2  $S 7<hi 7N38=no8O4  O L% !ijNDf8q/rsD#s+0w  ,O(   L$r>s4$#("" DD Dr1