bihddlmZddlmZmZddlmZmZddlm Z ddl m Z eGdde Z y) ) annotations) dataclassfield)OptionalUnion) PeftConfig)PeftTypec~eZdZUdZeeddiZded<edddi Zd ed <ed dd i Z ded<dZ y)TrainableTokensConfiga  Configuration for the `TrainableTokens` method. Allows for training new tokens (and re-training existing ones) without training the full embedding matrix. By marking a few select tokens (identified by their indices) trainable and leaving the rest untouched, this method can be used to add new tokens or changing the embedding of existing tokens while saving on memory. Both storage as well as working memory usage are reduced in contrast to training the embedding matrix fully. Note that training with FSDP/DeepSpeed might not yet be fully supported. Args: token_indices (`list[int]`): List of integers, signifying the indices of the tokens you want to be trainable. To find the index of a token with a tokenizer, you can tokenize the string and look at the returned `input_ids`. The closer the amount of indices is to the total amount of tokens, the less efficient this method gets. target_modules (`Optional[Union[list[str], str]]`): List of module names or regex expression of the module names to replace with our `TrainableTokensLayer`. If not defined, it will attempt to get the model's input embedding layer if the model has a `get_input_embeddings` method (transformer models usually do), if that fails the default is 'embed_tokens'. Other example targets are `embedding`, `encoder.embeddings` or `decoder.embeddings`. init_weights (`bool`): By default the new token weights are initialized to be the same as the respective token embeddings. This makes TrainableTokens a no-op when not trained. If set to `False` the weights will be random values. Do not change this setting unless you know exactly what you're doing. helpa)List of integers, signifying the indices of the tokens you want to be trainable. To find the index of a token with a tokenizer, you can tokenize the string and look at the returned `input_ids`. The closer the amount of indices is to the total amount of tokens, the less efficient this method gets.)default_factorymetadataz list[int] token_indicesNaList of module names or regex expression of the module names to replace with our `TrainableTokensLayer`. If not defined, it will default to the model's input embedding layer if the model has a `get_input_embeddings` method (transformer models usually do), if that fails the default is 'embed_tokens'. Other example targets could be `embedding`, `encoder.embeddings` or `decoder.embeddings`.)defaultrzOptional[Union[list[str], str]]target_modulesTaBy default the new token weights are initialized to be the same as the respective token embeddings. This makes TrainableTokens a no-op when not trained. If set to `False` the weights will be random values. Do not change this setting unless you know exactly what you're doing. bool init_weightsc.tj|_y)N)r TRAINABLE_TOKENS peft_type)selfs ^/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/peft/tuners/trainable_tokens/config.py __post_init__z#TrainableTokensConfig.__post_init__Ws!22) __name__ __module__ __qualname____doc__rlistr__annotations__rrrrrr r sw4 % ?  M9 7< (  7N3  a  L$ 3rr N) __future__r dataclassesrrtypingrr peft.configr peft.utilsr r r!rrr's1#("" ?3J?3 ?3r