bi,ddlZddlZddlZddlZddlZddlZddlZddlmZddl m Z m Z ddl m Z mZddlZddlmZddlmZmZddlmZej.eZe Gdd Zd ZGd d eZd Zy)N)Iterable) dataclassfield)OptionalUnion)HfArgumentParser) DataClass DataClassType)is_rich_availablec eZdZUdZedddiZeeed<edddiZ eeed<ed dd iZ eed <ed dd iZ eed<edddiZ e ed<edddiZe ed<edddiZe ed<y)ScriptArgumentsa Arguments common to all scripts. Args: dataset_name (`str`): Dataset name. dataset_config (`str` or `None`, *optional*, defaults to `None`): Dataset configuration name. Corresponds to the `name` argument of the [`~datasets.load_dataset`] function. dataset_train_split (`str`, *optional*, defaults to `"train"`): Dataset split to use for training. dataset_test_split (`str`, *optional*, defaults to `"test"`): Dataset split to use for evaluation. dataset_streaming (`bool`, *optional*, defaults to `False`): Whether to stream the dataset. If True, the dataset will be loaded in streaming mode. gradient_checkpointing_use_reentrant (`bool`, *optional*, defaults to `False`): Whether to apply `use_reentrant` for gradient checkpointing. ignore_bias_buffers (`bool`, *optional*, defaults to `False`): Debug argument for distributed training. Fix for DDP issues with LM bias/mask buffers - invalid scalar type, inplace operation. See https://github.com/huggingface/transformers/issues/22482#issuecomment-1595790992. Nhelpz Dataset name.)defaultmetadata dataset_namezgDataset configuration name. Corresponds to the `name` argument of the `datasets.load_dataset` function.dataset_configtrainz"Dataset split to use for training.dataset_train_splittestz$Dataset split to use for evaluation.dataset_test_splitFzUWhether to stream the dataset. If True, the dataset will be loaded in streaming mode.dataset_streamingz.warning_handlerks/8*AfXR0A0A/B"WINOr")NN) r3warningsr rich.loggingr% StreamHandler basicConfigERROR showwarning)r5FORMATr%handlerr4r3s @r#init_zero_verboser=Ws_  F,-'''')Gvv QXQ^Q^_P+Hr"c eZdZdZ d deeeeefffd Z d deee de de de e dffd Z dee fd ZxZS) TrlParsera A subclass of [`transformers.HfArgumentParser`] designed for parsing command-line arguments with dataclass-backed configurations, while also supporting configuration file loading and environment variable management. Args: dataclass_types (`Union[DataClassType, Iterable[DataClassType]]` or `None`, *optional*, defaults to `None`): Dataclass types to use for argument parsing. **kwargs: Additional keyword arguments passed to the [`transformers.HfArgumentParser`] constructor. Examples: ```yaml # config.yaml env: VAR1: value1 arg1: 23 ``` ```python # main.py import os from dataclasses import dataclass from trl import TrlParser @dataclass class MyArguments: arg1: int arg2: str = "alpha" parser = TrlParser(dataclass_types=[MyArguments]) training_args = parser.parse_args_and_config() print(training_args, os.environ.get("VAR1")) ``` ```bash $ python main.py --config config.yaml (MyArguments(arg1=23, arg2='alpha'),) value1 $ python main.py --arg1 5 --arg2 beta (MyArguments(arg1=5, arg2='beta'),) None ``` dataclass_typesc |g}nt|ts|g}|D])}d|jvstd|jdt |dd|i|y)Nconfigz Dataclass zu has a field named 'config'. This field is reserved for the config file path and should not be used in the dataclass.r@r!) isinstancer__dataclass_fields__ ValueErrorrsuper__init__)selfr@kwargsdataclass_type __class__s r#rGzTrlParser.__init__s  " OOX6./O. N>>>>  !8!8 9:PQ  CCFCr"argsreturn_remaining_stringsfail_with_unknown_argsreturn.c| t|ntjdd}d|vr|jd}|j ||j |}t |5}t j|}ddddvra|j di}t|ts td|jD]!\} } t| tj| <#|jd i|} ng} |j!||} |r| d} | dd| | zfzS|r| rtd| d | S#1swYxYw) a0 Parse command-line args and config file into instances of the specified dataclass types. This method wraps [`transformers.HfArgumentParser.parse_args_into_dataclasses`] and also parses the config file specified with the `--config` flag. The config file (in YAML format) provides argument values that replace the default values in the dataclasses. Command line arguments can override values set by the config file. The method also sets any environment variables specified in the `env` field of the config file. Nz--configenvz.`env` field should be a dict in the YAML file.)rLrMz$Unknown arguments from config file: zW. Please remove them, add them to the dataclass, or set `fail_with_unknown_args=False`.r!)listsysargvindexpopopenyaml safe_loadrCdictrEitemsrosenvironset_defaults_with_configparse_args_into_dataclasses)rHrLrMrN config_index config_path yaml_filerBenv_varskeyvalueconfig_remaining_stringsoutputargs_remaining_stringss r#parse_args_and_configzTrlParser.parse_args_and_configsf"-tDz388AB<  ::j1L HH\ "((<0Kk" 3i 2 3!::eR0!(D1$%UVV"*.."21JC&)%jBJJsO1(Et'D'D'Nv'N $') $11tVn1o $%+BZ "#2;":=S"S!UU U #(@67O6PQDD  M; 3 3s #D66D?c fd||}|jDcgc]!\}}||vs d|t|fD]}|#}}}}|Scc}}}w)a6 Overrides the parser's default values with those provided via keyword arguments, including for subparsers. Any argument with an updated default will also be marked as not required if it was previously required. Returns a list of strings that were not consumed by the parser. clt}|jD]}t|tjr8|j j D]}|j||U|j|vsd||j|_ d|_ |j|j|S)NF) set_actionsrCargparse_SubParsersActionchoicesvaluesupdatedestrrequiredadd)parserkw used_keysaction subparserapply_defaultss r#r}z:TrlParser.set_defaults_with_config..apply_defaultssI // /fh&@&@A%+^^%:%:%<H !(( 2)FGH[[B&%' _FN&+FOMM&++. / r"z--)r]r)rHrIrzrfrgitem remainingr}s @r#r`z"TrlParser.set_defaults_with_configs} #40 $*<<>  S%S 5IY[\_[`WacfglcmVn NRD    s AA)N)NFT)rrrrrrr rrGrr tupler rkrTr` __classcell__)rKs@r#r?r?rs-bTXD!% x 7N(N"OPD.)-).'+ 1x}%1#'1!% 1 y#~  1fDIr"r?cD tj|}tjj t j |}tjjtjj|d}tjj|d}tjj|r8tjgd|jjd}|Sy#t$r}dt|cYd}~Sd}~wwxYw)Nz..z.git)gitz rev-parseHEAD)cwdzutf-8zError: ) importlib import_moduler^pathdirnameinspectgetfileabspathjoinisdir subprocess check_outputstripdecode Exceptionr) package_namepackage package_path git_repo_pathgit_dir commit_hashes r#get_git_commit_hashrs")),7wwww'?@  \4(HI '',,}f5 77== !''(D-X^^`gghop   "Q!!"sC:C>> D DDD)rprrr3r^rrUcollections.abcr dataclassesrrtypingrrrZ transformersrtransformers.hf_argparserr r transformers.utilsr getLoggerrloggerr r=r?rr!r"r#rsy  $(" )>0   8 $ 00 0f+6S Sl"r"