bi~dddlmZmZddlmZmZddlZddlmZddlm Z eGdde Z y)) dataclassfield)OptionalUnionN)version)TrainingArgumentsc$eZdZUdZej ejej dk\rejdgzZ e dddiZ e e d<e d dd iZe e d <e d dd iZeee d<e d ddiZeeeefe d<e dddiZee d<e dddiZeee d<e dddiZeee d<e dddiZeee d<e dddiZeee d<e dddiZee d <e ddd!iZeee d"<e d dd#iZeee d$<e d dd%iZeee d&<e d'dd(iZ e e d)<e d'dd*iZ!e e d+<e d dd,iZ"eee d-<e d dd.iZ#ee e d/<e d dd0iZ$eee d1<e d'dd2iZ%e e d3<e d dd4iZ&eee d5<e ddd6iZ'ee d7<e d dd8iZ(eee d9<e d:dd;iZ)ee d<<e d dd=iZ*eee d><e d?dd@iZ+ee dA<e dBddCiZ,ee dD<e dEddFiZ-e e dG<e dHddIiZ.e e dJ<e dKddLiZ/ee dM<e dNddOiZ0e e dP<e dKddQiZ1ee dR<e dSddTiZ2e e dU<e d ddViZ3ee e dW<e d ddXiZ4ee e dY<e d ddZiZ5ee6e e d[<e ddd\iZ7ee d]<e d^dd_iZ8ee d`<e dddaiZ9ee db<e dddciZ:ee dd<e deddfiZ;e e dg<e dddhiZee dm<e d ddniZ?eee do<e dddpiZ@eee dq<fdrZAxZBS)s GRPOConfigu4 Configuration class for the [`GRPOTrainer`]. This class includes only the parameters that are specific to GRPO training. For a full list of training arguments, please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may differ from those in [`~transformers.TrainingArguments`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the command line. Parameters: > Parameters that control the model and reference model model_init_kwargs (`str`, `dict[str, Any]` or `None`, *optional*, defaults to `None`): Keyword arguments for [`~transformers.AutoModelForCausalLM.from_pretrained`], used when the `model` argument of the [`GRPOTrainer`] is provided as a string. disable_dropout (`bool`, *optional*, defaults to `False`): Whether to disable dropout in the model. This is useful for training with a reference model, as it prevents the model from generating different logprobs for the same input. > Parameters that control the data preprocessing remove_unused_columns (`bool`, *optional*, defaults to `False`): Whether to only keep the column `"prompt"` in the dataset. If you use a custom reward function that requires any column other than `"prompts"` and `"completions"`, you should keep this to `False`. max_prompt_length (`int` or `None`, *optional*, defaults to `512`): Maximum length of the prompt. If the prompt is longer than this value, it will be truncated left. num_generations (`int` or `None`, *optional*, defaults to `8`): Number of generations per prompt to sample. The effective batch size (num_processes * per_device_batch_size * gradient_accumulation_steps) must be evenly divisible by this value. max_completion_length (`int` or `None`, *optional*, defaults to `256`): Maximum length of the generated completion. ds3_gather_for_generation (`bool`, *optional*, defaults to `True`): This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation, improving generation speed. However, disabling this option allows training models that exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation. Disabling this option is not compatible with vLLM generation. shuffle_dataset (`bool`, *optional*, defaults to `True`): Whether to shuffle the training dataset. > Parameters that control generation generation_batch_size: (`int` or `None`, *optional*, defaults to `None`): Batch size to use for generation. If `None`, it defaults to the effective training batch size: `per_device_train_batch_size * num_processes * gradient_accumulation_steps`. steps_per_generations: (`int` or `None`, *optional*, defaults to `None`): Number of optimization steps per generation. If `None`, it defaults to gradient_accumulation_steps. temperature (`float`, defaults to `1.0`): Temperature for sampling. The higher the temperature, the more random the completions. top_p (`float`, *optional*, defaults to `1.0`): Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to `1.0` to consider all tokens. top_k (`int` or `None`, *optional*, defaults to `None`): Number of highest probability vocabulary tokens to keep for top-k-filtering. If `None`, top-k-filtering is disabled and all tokens are considered. min_p (`float` or `None`, *optional*, defaults to `None`): Minimum token probability, which will be scaled by the probability of the most likely token. It must be a value between `0.0` and `1.0`. Typical values are in the `0.01-0.2` range. repetition_penalty (`float`, *optional*, defaults to `1.0`): Float that penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > `1.0` encourage the model to use new tokens, while values < `1.0` encourage the model to repeat tokens. cache_implementation (`str` or `None`, *optional*, defaults to `None`): Implementation of the cache method for faster generation when use_vllm is set to False. generation_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`): Additional keyword arguments to pass to `GenerationConfig` (if using transformers) or `SamplingParams` (if using vLLM) when sampling completions. This can be used to further customize the generation behavior, such as setting `supress_tokens`, `num_beams`, etc. If it contains keys that conflict with the other generation parameters (like `min_p`, `top_p`, etc.), they will override them. > Parameters that control generation acceleration powered by vLLM use_vllm (`bool`, *optional*, defaults to `False`): Whether to use vLLM for generating completions. If set to `True`, the trainer will use vLLM for generation instead of the default model.generate(). Requires `vllm` to be installed. vllm_mode (`str`, *optional*, defaults to `"server"`): Mode to use for vLLM integration when `use_vllm` is set to `True`. Must be one of `"server"` or `"colocate"`. - `"server"`: The trainer will send generation requests to a separate vLLM server. Make sure a TRL vLLM server is running (start with `trl vllm-serve`). - `"colocate"`: vLLM will run in the same process and share the training GPUs. This avoids the need for a separate server but may cause resource contention with training. vllm_guided_decoding_regex (`str` or `None`, *optional*, defaults to `None`): Regex for vLLM guided decoding. If `None` (default), guided decoding is disabled. > Parameters that control the vLLM server (only used when `vllm_mode` is `"server"`) vllm_server_base_url (`str` or `None`, *optional*, defaults to `None`): Base URL for the vLLM server (e.g., `"http://localhost:8000"`). If provided, `vllm_server_host` and `vllm_server_port` are ignored. vllm_server_host (`str`, *optional*, defaults to `"0.0.0.0"`): Host of the vLLM server to connect to. Ignored if `vllm_server_base_url` is provided. vllm_server_port (`int`, *optional*, defaults to `8000`): Port of the vLLM server to connect to. Ignored if `vllm_server_base_url` is provided. vllm_server_timeout (`float`, *optional*, defaults to `240.0`): Total timeout duration in seconds to wait for the vLLM server to be up. If the server is not up after the timeout, a `ConnectionError` is raised. > Parameters that control colocated vLLM execution (only used when `vllm_mode` is `"colocate"`) vllm_gpu_memory_utilization (`float`, *optional*, defaults to `0.3`): Control the GPU memory utilization for vLLM. This setting only applies when `vllm_mode` is set to `"colocate"`. If you are using `vllm_mode="server"`, this parameter must be passed separately when launching the vLLM server via the `--vllm_gpu_memory_utilization` flag. vllm_tensor_parallel_size (`int`, *optional*, defaults to `1`): Control the tensor parallel size for vLLM. This setting only applies when `vllm_mode` is set to `"colocate"`. If you are using `vllm_mode="server"`, this parameter must be passed separately when launching the vLLM server via the `--vllm_tensor_parallel_size` flag. > Parameters that control the training beta (`float`, *optional*, defaults to `0.0`): KL coefficient. If `0.0` (default), the reference model is not loaded, reducing memory usage and improving training speed. num_iterations (`int`, *optional*, defaults to `1`): Number of iterations per batch (denoted as μ in the algorithm). epsilon (`float`, *optional*, defaults to `0.2`): Epsilon value for clipping. delta: (`float` or `None`, *optional*, defaults to `None`): Enables the upper clipping bound in two-sided GRPO loss when set to a float. If `None` (default), standard GRPO clipping is used. Recommended to be greater than `1 + ε` when enabled. This method is introduced in the [INTELLECT-2 tech report](https://huggingface.co/papers/2505.07291). epsilon_high (`float` or `None`, *optional*, defaults to `None`): Upper-bound epsilon value for clipping. If not specified, it defaults to the same value as the lower-bound specified in argument `epsilon`. Paper [DAPO](https://huggingface.co/papers/2503.14476) recommends `0.28`. reward_weights (`list[float]` or `None`, *optional*, defaults to `None`): Weights for each reward function. Must match the number of reward functions. If `None`, all rewards are weighted equally with weight `1.0`. scale_rewards (`bool`, *optional*, defaults to `True`): Whether to scale the rewards by dividing them by their standard deviation. If `True` (default), the rewards are normalized by the standard deviation, ensuring they have unit variance. If `False`, no scaling is applied. The [Dr. GRPO paper](https://huggingface.co/papers/2503.20783) recommends not scaling the rewards, as scaling by the standard deviation introduces a question-level difficulty bias. loss_type (`str`, *optional*, defaults to `"bnpo"`): Specifies the loss formulation to use. Supported values are: - `"grpo"`: Aggregates token-level losses by normalizing over sequence length. Not recommended due to length bias—this approach tends to prefer shorter completions with positive advantages and longer ones with negative advantages. - `"bnpo"`: Aggregates token-level losses by normalizing number of active token in the local batch. Note that normalization is performed over the local batch only, so results may slightly vary depending on the local batch size, despite a constant effective batch size. When using `per_device_train_batch_size==1`, the loss is equivalent to the GRPO loss. - `"dr_grpo"`: Aggregates token-level losses by normalizing with a global constant. This method was introduced in the [Dr. GRPO paper](https://huggingface.co/papers/2503.20783) to eliminate length bias. The value of the constant corresponds to `max_completion_length`. mask_truncated_completions (`bool`, *optional*, defaults to `False`): When enabled, truncated completions are excluded from the loss calculation, preventing them from being incorrectly penalized and introducing noise during training. According to the [DAPO](https://huggingface.co/papers/2503.14476) paper, this is a good practice for training stability. sync_ref_model (`bool`, *optional*, defaults to `False`): Whether to synchronize the reference model with the active model every `ref_model_sync_steps` steps, using the `ref_model_mixup_alpha` parameter. This synchronization originates from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper. ref_model_mixup_alpha (`float`, *optional*, defaults to `0.6`): α parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which controls the mix between the current policy and the previous reference policy during updates. The reference policy is updated according to the equation: `π_ref = α * π_θ + (1 - α) * π_ref_prev`. To use this parameter, you must set `sync_ref_model=True`. ref_model_sync_steps (`int`, *optional*, defaults to `512`): τ parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which determines how frequently the current policy is synchronized with the reference policy. To use this parameter, you must set `sync_ref_model=True`. use_liger_loss (`bool`, *optional*, defaults to `False`): Whether to use the Liger GRPO loss. > Parameters that control the logging log_completions (`bool`, *optional*, defaults to `False`): Whether to log a sample of (prompt, completion) pairs every `logging_steps` steps. If `rich` is installed, it prints the sample. If `wandb` logging is enabled, it logs it to `wandb`. num_completions_to_print (`int` or `None`, *optional*, defaults to `None`): Number of completions to print with `rich`. If `None`, all completions are logged. wandb_log_unique_prompts (`bool`, *optional*, defaults to `False`): Whether to log unique prompts in wandb. If `True`, only unique prompts are logged. If `False`, all prompts are logged. z4.51.0model_init_kwargsgư>helpz$The initial learning rate for AdamW.)defaultmetadata learning_rate zLog every X updates steps. Should be an integer or a float in range `[0,1)`. If smaller than 1, will be interpreted as ratio of total training steps. logging_stepsNzWhether to use bf16 (mixed) precision instead of 32-bit. Requires Ampere or higher NVIDIA architecture or Intel XPU or using CPU (use_cpu) or Ascend NPU. If not set, it defaults to `True` if `fp16` is not set.bf16zKeyword arguments for `transformers.AutoModelForCausalLM.from_pretrained`, used when the `model` argument of the `GRPOTrainer` is provided as a string.FzWhether to disable dropout in the model. This is useful for training with a reference model, as it prevents the model from generating different logprobs for the same input.disable_dropoutzWhether to only keep the column 'prompt' in the dataset. If you use a custom reward function that requires any column other than 'prompts' and 'completions', you should keep this to `False`.remove_unused_columnsizaMaximum length of the prompt. If the prompt is longer than this value, it will be truncated left.max_prompt_lengthzNumber of generations to sample. The effective batch size (num_processes * per_device_batch_size * gradient_accumulation_steps) must be evenly divisible by this value.num_generationsz+Maximum length of the generated completion.max_completion_lengthTaSThis setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation, improving generation speed. However, disabling this option allows training models that exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation. Disabling this option is not compatible with vLLM generation.ds3_gather_for_generationz(Whether to shuffle the training dataset.shuffle_datasetzBatch size to use for generation. If `None`, it defaults to the effective training batch size: `per_device_train_batch_size * num_processes * gradient_accumulation_steps`.generation_batch_sizezcNumber of optimization steps per generation. If `None`, it defaults to gradient_accumulation_steps.steps_per_generationg?zVTemperature for sampling. The higher the temperature, the more random the completions. temperaturezFloat that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1.0 to consider all tokens.top_pzNumber of highest probability vocabulary tokens to keep for top-k-filtering. If `None`, top-k-filtering is disabled and all tokens are considered.top_kzMinimum token probability, which will be scaled by the probability of the most likely token. It must be a value between 0.0 and 1.0. Typical values are in the 0.01-0.2 range.min_paAdditional keyword arguments to pass to `GenerationConfig` (if using transformers) or `SamplingParams` (if using vLLM) when sampling completions. This can be used to further customize the generation behavior, such as setting `supress_tokens`, `num_beams`, etc. If it contains keys that conflict with the other generation parameters (like `min_p`, `top_p`, etc.), they will override them.generation_kwargszFloat that penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > 1.0 encourage the model to use new tokens, while values < 1.0 encourage the model to repeat tokens.repetition_penaltyzWImplementation of the cache method for faster generation when use_vllm is set to False.cache_implementationzWhether to use vLLM for generating completions. If set to `True`, the trainer will use vLLM for generation instead of the default model.generate(). Requires `vllm` to be installed.use_vllmzBase URL for the vLLM server (e.g., 'http://localhost:8000'). If provided, `vllm_server_host` and `vllm_server_port` are ignored.vllm_server_base_urlserveraMode to use for vLLM integration when `use_vllm` is set to `True`. Must be one of `server` or `'colocate'`. `'server'`: The trainer will send generation requests to a separate vLLM server. Make sure a TRL vLLM server is running (start with `trl vllm-serve`). `'colocate'`: vLLM will run in the same process and share the training GPUs. This avoids the need for a separate server but may cause resource contention with training. vllm_modezQRegex for vLLM guided decoding. If `None` (default), guided decoding is disabled.vllm_guided_decoding_regexz0.0.0.0zSHost of the vLLM server to connect to. Ignored if vllm_server_base_url is provided.vllm_server_hosti@zSPort of the vLLM server to connect to. Ignored if vllm_server_base_url is provided.vllm_server_portgn@zTotal timeout duration in seconds to wait for the vLLM server to be up. If the server is not up after the timeout, a `ConnectionError` is raised.vllm_server_timeoutg333333?a Control the GPU memory utilization for vLLM. This setting only applies when `vllm_mode` is set to `'colocate'`. If you are using `vllm_mode='server'`, this parameter must be passed separately when launching the vLLM server via the `--vllm_gpu_memory_utilization` flag.vllm_gpu_memory_utilizationaControl the tensor parallel size for vLLM. This setting only applies when `vllm_mode` is set to `'colocate'`. If you are using `vllm_mode='server'`, this parameter must be passed separately when launching the vLLM server via the `--vllm_tensor_parallel_size` flag.vllm_tensor_parallel_sizegzzKL coefficient. If `0.0` (default), the reference model is not loaded, reducing memory usage and improving training speed.betau@Number of iterations per batch (denoted as μ in the algorithm).num_iterationsg?zEpsilon value for clipping.epsilonuEnables the upper clipping bound in two-sided GRPO loss when set to a float. If `None` (default), standard GRPO clipping is used. Recommended to be greater than `1 + ε` when enabled. This method is introduced in the [INTELLECT-2 tech report](https://huggingface.co/papers/2505.07291).deltazUpper-bound epsilon value for clipping. If not specified, it defaults to the same value as the lower-bound specified in argument `epsilon`. Paper DAPO recommends `0.28`. epsilon_highzWeights for each reward function. Must match the number of reward functions. If `None`, all rewards are weighted equally with weight `1.0`.reward_weightsacWhether to scale the rewards by dividing them by their standard deviation. If `True` (default), the rewards are normalized by the standard deviation, ensuring they have unit variance. If `False`, no scaling is applied. The Dr. GRPO paper recommends not scaling the rewards, as scaling by the standard deviation introduces a question-level difficulty bias. scale_rewardsbnpouSpecifies the loss formulation to use. Supported values are `grpo`, `bnpo`, and `dr_grpo`. `'grpo'`: Aggregates token-level losses by normalizing over sequence length. Not recommended due to length bias—this approach tends to prefer shorter completions with positive advantages and longer ones with negative advantages. `'bnpo'`: Aggregates token-level losses by normalizing number of active token in the local batch. Note that normalization is performed over the local batch only, so results may slightly vary depending on the local batch size, despite a constant effective batch size. When using `per_device_train_batch_size==1`, the loss is equivalent to the GRPO loss. `'dr_grpo'`: Aggregates token-level losses by normalizing with a global constant. This method was introduced in the Dr. GRPO paper to eliminate length bias. The value of the constant corresponds to `max_completion_length`. loss_typezWhen enabled, truncated completions are excluded from the loss calculation, preventing them from being incorrectly penalized and introducing noise during training. According to the DAPO paper, this is a good practice for training stability.mask_truncated_completionszWhether to synchronize the reference model with the active model every `ref_model_sync_steps` steps, using the `ref_model_mixup_alpha` parameter.sync_ref_modelg333333?u-α parameter from the TR-DPO paper, which controls the mix between the current policy and the previous reference policy during updates. The reference policy is updated according to the equation: `π_ref = α * π_θ + (1 - α) * π_ref_prev`. To use this parameter, you must set `sync_ref_model=True`.ref_model_mixup_alphauτ parameter from the TR-DPO paper, which determines how frequently the current policy is synchronized with the reference policy. To use this parameter, you must set `sync_ref_model=True`.ref_model_sync_stepsz#Whether to use the Liger GRPO loss.use_liger_losszWhether to log a sample of (prompt, completion) pairs every `logging_steps` steps. If `rich` is installed, it prints the sample. If `wandb` logging is enabled, it logs it to `wandb`.log_completionszRNumber of completions to print with `rich`. If `None`, all completions are logged.num_completions_to_printzvWhether to log unique prompts in wandb. If `True`, only unique prompts are logged. If `False`, all prompts are logged.wandb_log_unique_promptsc |j |j n |j|_t| |j}|j |j td|j |j|_|j !|j|z|j z|_|j |j|zzdk7r)td|j d|j|zd|j |j|zz|_|jdkrtd|jdtd|j d zDcgc]}|j |zdk(s|}}|j|vr9td |d |jd |j d |jd |d |jdk7rn|j|z}td|d zDcgc] }||zdk(s |}}|j|vr,td|d |jd |jd|d |j|jr tdyycc}wcc}w)Nz^'generation_batch_size' and 'steps_per_generation' can not be both configured at the same timerzgeneration_batch_size (z.) must be divisible by the global batch size (z).zZGRPO requires at least 2 generations per prompt to calculate the advantages. You provided z*, which is less than the minimum required.r.z The effective train batch size (z x zD) must be evenly divisible by the number of generations per prompt (ze). Given the current effective train batch size, the valid values for the number of generations are: .nozThe global eval batch size (za). Given the current global eval batch size, the valid values for the number of generations are: z4Liger loss does not support two-sided GRPO loss yet.)rfp16super __post_init__ world_sizerr ValueErrorgradient_accumulation_stepsper_device_train_batch_sizerrange eval_strategyper_device_eval_batch_sizer3r=)self num_processesn_genpossible_valuesglobal_eval_batch_size __class__s R/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/trl/trainer/grpo_config.pyrGzGRPOConfig.__post_init__s'+yy'8Odii    % % 1d6O6O6[p   $ $ ,(,(H(HD %  % % -)-)I)IM)Y\`\u\u)uD &  % %)I)IM)Y Z^_ _)$*D*D)EF44}DERI  %)$>$>4CcCcfsCs$t!   ! #l''((RT  %Q(B(BQ(FG DLfLfjoKostKtE      62=/TEeEeDffi,,-.//0122A1B!E     %%)%D%D}%T "#(,BQ,F#GLbfkKkopKpO##?: 2=/TEdEdDefQQUQeQePfgk&'q* :: !d&9&9ST T': !/ sI&7I&3 I+I+)C__name__ __module__ __qualname____doc__rparse transformers __version__r_VALID_DICT_FIELDSrrfloat__annotations__rrrboolr rdictstrrrrintrrrrrrrrr r!r"r#r$r%r&r(r)r*r+r,r-r/r0r1r2r3r4r5listr6r8r9r:r;r<r=r>r?r@rG __classcell__)rTs@rUr r sqfw}}\--.-'--2II.AAEXDYY!@AM5! D M5! ! D(4.5: E 5xdCi 01" [ OT-2 p -8D>(- w (x} &+ U &OXc],1GH,8C=', 6 't',DE'OXd^ ,1 [ ,8C=+0 y +(3- rsK 1 E5! I E8C=# ] E8E?). t )x~!&   !+0st+(3-  c Hd+0 2 +(3- (  Is 16mn1  "opc"opc"' @ "*/ V *&+ T &s ( D% \]NC78GU# o E8E?%* Y %L(5/-2 > -NHT%[)  E M4  '  Is (- 6 (! B ND$) y $5!& q !#!?@ND " e OT/4no/hsm05 & 0htn8U8Ur ) dataclassesrrtypingrrr[ packagingrrr rfrUrks8)"* qU"qU qUrf