biHddlZddlmZmZddlmZeGddeZy)N) dataclassfield)OnPolicyConfigceZdZUdZeej jeddddiZ e e d<eddd iZ e e d <ed dd iZ ee d <edddiZee d<edddiZee d<edddiZee d<edddiZee d<edddiZee d<edddiZee d<edddiZee d <eddd!iZee d"<ed#dd$iZee d%<y)& RLOOConfiga Configuration class for the [`RLOOTrainer`]. This class includes only the parameters that are specific to RLOO training. For a full list of training arguments, please refer to the [`~transformers.TrainingArguments`] and [`OnPolicyConfig`] documentation. Note that default values in this class may differ from those in [`~transformers.TrainingArguments`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the command line. Parameters: exp_name (`str`, *optional*, defaults to `os.path.basename(__file__)[: -len(".py")]`): Name of this experiment. reward_model_path (`str`, *optional*, defaults to `"EleutherAI/pythia-160m"`): Path to the reward model. num_ppo_epochs (`int`, *optional*, defaults to `4`): Number of epochs to train. whiten_rewards (`bool`, *optional*, defaults to `False`): Whether to whiten the rewards. kl_coef (`float`, *optional*, defaults to `0.05`): KL coefficient. cliprange (`float`, *optional*, defaults to `0.2`): Clip range. rloo_k (`int`, *optional*, defaults to `2`): REINFORCE Leave-One-Out (RLOO) number of online samples per prompt. normalize_reward (`bool`, *optional*, defaults to `False`): Whether to normalize rewards. reward_clip_range (`float`, *optional*, defaults to `10.0`): Clip range for rewards. normalize_advantage (`bool`, *optional*, defaults to `False`): Whether to normalize advantages. token_level_kl (`bool`, *optional*, defaults to `True`): Whether to use token-level KL penalty or sequence-level KL penalty. ds3_gather_for_generation (`bool`, *optional*, defaults to `True`): This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation, improving generation speed. However, disabling this option allows training models that exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation. NhelpzName of this experiment.)defaultmetadataexp_namezEleutherAI/pythia-160mzPath to the reward model.reward_model_pathzNumber of epochs to train.num_ppo_epochsFzWhether to whiten the rewards.whiten_rewardsg?zKL coefficient.kl_coefg?z Clip range. cliprangerzCREINFORCE Leave-One-Out (RLOO) number of online samples per prompt.rloo_kzWhether to normalize rewardsnormalize_rewardg$@zClip range for rewardsreward_clip_rangezWhether to normalize advantagesnormalize_advantagezBWhether to use token-level KL penalty or sequence-level KL penaltytoken_level_klTaThis setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation, improving generation speed. However, disabling this option allows training models that exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation.ds3_gather_for_generation)__name__ __module__ __qualname____doc__rospathbasename__file__r str__annotations__rrintrboolrfloatrrrrrrrR/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/trl/trainer/rloo_config.pyrrs&P  *3B/45Hc#(56s 67NC!:;ND+,GU-(Iu_`FC#89d %23 u!&;<!!^_ND', a 'tr(r)r dataclassesrr trainer.utilsrrr'r(r)r,s. (* \\ \r(