bi"LddlmZmZddlmZddlmZeGddeZy)) dataclassfield)Optional)TrainingArgumentscveZdZUdZedddiZeed<edddiZeed <ed dd iZ e e ed <ed dd iZ e e ed<ed ddiZe e ed<edddiZeed<edddiZeed<edddiZeed<ed ddiZe eed<edddiZeeed<ed d!d d"gd#Ze ed$<ed dd%iZe eed&<ed'dd(iZe ed)<ed*dd+iZe ed,<ed-dd.iZe eed/<ed'dd0iZe ed1<fd2ZxZS)3OnlineDPOConfigu{ Configuration class for the [`OnlineDPOTrainer`]. This class includes only the parameters that are specific to Online DPO training. For a full list of training arguments, please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may differ from those in [`~transformers.TrainingArguments`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the command line. Parameters: reward_model_path (`str` or `None`, *optional*, defaults to `None`): Path to the reward model. Either `judge` or `reward_model_path` must be set, but not both. judge (`str` or `None`, *optional*, defaults to `None`): Name of the judge to use. Either `judge` or `reward_model_path` must be set, but not both. max_new_tokens (`int`, *optional*, defaults to `64`): Maximum number of tokens to generate per completion. max_length (`int`, *optional*, defaults to `256`): Maximum total length of the sequence (prompt + completion) used to compute log probabilities. If the sequence exceeds this limit, the leftmost tokens will be truncated to preserve as much of the completion as possible. temperature (`float`, *optional*, defaults to `0.9`): Temperature for sampling. The higher the temperature, the more random the completions. missing_eos_penalty (`float` or `None`, *optional*, defaults to `None`): Penalty applied to the score when the model fails to generate an EOS token. This is useful to encourage to generate completions shorter than the maximum length (`max_new_tokens`). The penalty must be a positive value. beta (`float` or `list[float]`, *optional*, defaults to `0.1`): Parameter controlling the deviation from the reference model. Higher β means less deviation from the reference model. For the IPO loss (`loss_type="ipo"`), β is the regularization parameter denoted by τ in the [paper](https://huggingface.co/papers/2310.12036). If a list of floats is provided then the β is selected for each new epoch and the last β is used for the rest of the epochs. loss_type (`str`, *optional*, defaults to `"sigmoid"`): Type of loss to use. Possible values are: - `"sigmoid"`: sigmoid loss from the original [DPO](https://huggingface.co/papers/2305.18290) paper. - `"ipo"`: IPO loss from the [IPO](https://huggingface.co/papers/2310.12036) paper. dataset_num_proc (`int` or `None`, *optional*, defaults to `None`): Number of processes to use for processing the dataset. disable_dropout (`bool`, *optional*, defaults to `True`): Whether to disable dropout in the model and reference model. use_vllm (`bool`, *optional*, defaults to `False`): Whether to use vLLM for generating completions. Requires vLLM to be installed (`pip install vllm`). gpu_memory_utilization (`float`, *optional*, defaults to `0.55`): The vLLM memory utilization. The default value is 0.55. ds3_gather_for_generation (`bool`, *optional*, defaults to `True`): This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation, improving generation speed. However, disabling this option allows training models that exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation. gƠ>helpz$The initial learning rate for AdamW.)defaultmetadata learning_rate zLog every X updates steps. Should be an integer or a float in range `[0,1)`. If smaller than 1, will be interpreted as ratio of total training steps. logging_stepsNzWhether to use bf16 (mixed) precision instead of 32-bit. Requires Ampere or higher NVIDIA architecture or Intel XPU or using CPU (use_cpu) or Ascend NPU. If not set, it defaults to `True` if `fp16` is not set.bf16zZPath to the reward model. Either `judge` or `reward_model_path` must be set, but not both.reward_model_pathzZName of the judge to use. Either `judge` or `reward_model_path` must be set, but not both.judge@z4Maximum number of tokens to generate per completion.max_new_tokensizMaximum total length of the sequence (prompt + completion) used to compute log probabilities. If the sequence exceeds this limit, the leftmost tokens will be truncated to preserve as much of the completion as possible. max_lengthg?zVTemperature for sampling. The higher the temperature, the more random the completions. temperaturezPenalty applied to the score when the model fails to generate an EOS token. This is useful to encourage to generate completions shorter than the maximum length (`max_new_tokens`). The penalty must be a positive value.missing_eos_penaltycdgS)Ng?rX/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/trl/trainer/online_dpo_config.pyzOnlineDPOConfig.sruParameter controlling the deviation from the reference model. Higher β means less deviation from the reference model. For the IPO loss (`loss_type='ipo'`), β is the regularization parameter denoted by τ in the [paper](https://huggingface.co/papers/2310.12036). If a list of floats is provided then the β is selected for each new epoch and the last β is used for the rest of the epochs.)default_factoryr betasigmoidzType of loss to use.ipo)r choices loss_typez6Number of processes to use for processing the dataset.dataset_num_procTz(Whether to disable dropout in the model.disable_dropoutFzcWhether to use vLLM for generating completions. Requires vLLM to be installed (`pip install vllm`).use_vllmg?z7The vLLM memory utilization. The default value is 0.55.gpu_memory_utilizationaThis setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation, improving generation speed. However, disabling this option allows training models that exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation.ds3_gather_for_generationc|j |j n |j|_t| t |j dr.t |j dk(r|j d|_yyy)N__len__r)rfp16super __post_init__hasattrrlen)self __class__s rr,zOnlineDPOConfig.__post_init__s_'+yy'8Odii   499i (S^q-@ ! DI.A (r)__name__ __module__ __qualname____doc__rr float__annotations__rrrboolrstrrrintrrrrlistr!r"r#r$r%r&r, __classcell__)r0s@rrrsP3l!@AM5! D M5! ! D(4.(- p (x} ! p E8C=  PQNC & JrsK,1   ,%% a D$u+*!5) Is',RS'hsm"DEOT $ Hd/4 M /HUO ', a 't%%rrN) dataclassesrrtypingr transformersrrrrrr?s/)* c%'c% c%r