biXddlZddlZddlmZddlmZmZmZddlZddl m Z ddl m Z ddl mZmZmZmZmZmZmZmZmZmZmZmZmZddlmZddlmZd d lm Z d d l!m"Z"d d l#m$Z$m%Z%erddl&m'Z'erddl(Z(GddeZ)y)N)Path)CallableOptionalUnion)Dataset) DataLoader) AutoModelForCausalLM AutoTokenizerBaseImageProcessor DataCollatorDataCollatorForLanguageModelingDataCollatorForSeq2SeqFeatureExtractionMixinPreTrainedModelPreTrainedTokenizerBaseProcessorMixinTrainerTrainingArgumentsis_wandb_available)EvalLoopOutput)is_peft_available) PPODecorators)IterativeSFTConfig)generate_model_cardget_comet_experiment_url) PeftModelceZdZdZddgZ d"deeefdeee e fdee deee e ee ffd eeeeeefd eej(j*ej(j,j.fd eeej2ej2gej2fd eeege fd eedeedeeffd Zdede defdZdej2dej2dej2fdZe de!ejDde!ejDde!ejDde!ede!ef dZ#e$jJ d#dee!ejDdee!ejDdee!ejDdee!edee!ef dZ&dZ'fdZ( d$deedeed eee!edffd!Z)xZ*S)%IterativeSFTTrainera The IterativeSFTTrainer can be used to finetune models with methods that requires some steps between optimization. Args: model (`Union[str, PreTrainedModel]`): Model to be trained. Can be either: - A string, being the *model id* of a pretrained model hosted inside a model repo on huggingface.co, or a path to a *directory* containing model weights saved using [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded using [`~transformers.AutoModelForCausalLM.from_pretrained`] with the keyword arguments in `args.model_init_kwargs`. - A [`~transformers.PreTrainedModel`] object. Only causal language models are supported. args ([`IterativeSFTConfig`], *optional*, defaults to `None`): Configuration for this trainer. If `None`, a default configuration is used. data_collator (`DataCollator`, *optional*): Function to use to form a batch from a list of elements of the processed `train_dataset` or `eval_dataset`. Will default to [`~transformers.default_data_collator`] if no `processing_class` is provided, an instance of [`~transformers.DataCollatorWithPadding`] otherwise if the processing_class is a feature extractor or tokenizer. eval_dataset (`datasets.Dataset`): The dataset to use for evaluation. processing_class ([`~transformers.PreTrainedTokenizerBase`], *optional*, defaults to `None`): Processing class used to process the data. If `None`, the processing class is loaded from the model's name with [`~transformers.AutoTokenizer.from_pretrained`]. optimizers (`tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`): The optimizer and scheduler to use for training. preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`): The function to use to preprocess the logits before computing the metrics. compute_metrics (`Callable[[EvalPrediction], dict]`, *optional*): The function to use to compute the metrics. Must take a `EvalPrediction` and return a dictionary string to metric values. max_length (`int`, *optional*, deprecated): Maximum length of the tokenized sequence. Use `args.max_length` instead. truncation_mode (`str`, *optional*, deprecated): The truncation mode to use. Use `args.truncation_mode` instead. optimize_device_cache (`bool`, *optional*, deprecated): Whether to optimize accelerator cache. Use `args.optimize_device_cache` instead. trlz iterative-sftNmodelargs data_collator eval_datasetprocessing_class optimizerspreprocess_logits_for_metricscompute_metrics max_lengthtruncation_modeoptimize_device_cachec i} | | | d<tjdt| | | d<tjdt| | | d<tjdtt|tr|n|j j } |#| jdd}t|d }n[t|trKt|ts;|j}|j|d <|jd tdi|}| r%| jD]\}}t||||tj | }|j"%t|tstjd t|tr|j%||}t'rt|t(rd |_nd|_||_t/|j dd|_|=|j0rt3|dd|_n$t7|j,d|_n||_|j8|_|j:|_|j<|_t>||||j4|||||tC|jDdr%|jDjG|jH|jK|jLjN|jPjS|jD|jT|jV\|_"|_*|_+|j:dk(rdnd|j,_,tC|ds t[d|j<t\_y)Nr*zThe `max_length` parameter is deprecated and will be removed in version 0.20. Pass it through the `args` parameter using `IterativeSFTConfig(max_length=...)` instead.r+zThe `truncation_mode` parameter is deprecated and will be removed in version 0.20. Pass it through the `args` parameter using `IterativeSFTConfig(truncation_mode=...)` instead.r,zThe `optimize_device_cache` parameter is deprecated and will be removed in version 0.20 Pass it through the `args` parameter using `IterativeSFTConfig(optimize_device_cache=...)` instead./z -IterativeSFT hub_tokenpush_to_hub_tokenzYou passed model_init_kwargs to the `IterativeSFTConfig`, but your model is already instantiated. The `model_init_kwargs` will be ignored.TFis_encoder_decoder)label_pad_token_idpad_to_multiple_of)mlm)r"r#r$r%r&r)r'r(add_model_tagskeep_endleftright acceleratorzXYour `Trainer` does not have an `accelerator` object. Consider upgrading `transformers`.)/warningswarnDeprecationWarning isinstancestrconfig _name_or_pathsplitrrto_dictr0popitemssetattrr from_pretrainedmodel_init_kwargs_create_model_from_pathrr is_peft_modelr&getattrr2rr$r r*r+r,super__init__hasattrr"r8 _tag_namescreate_optimizer_and_schedulerr# max_stepsr<prepare optimizer lr_schedulertruncation_sideAttributeErrorr)selfr"r#r$r%r&r'r(r)r*r+r,deprecated_paramsmodel_id model_name dict_argskeyvalue __class__s \/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/trl/trainer/iterative_sft_trainer.pyrPzIterativeSFTTrainer.__init__as<*  !.8 l + MMk"   &3B / 0 MMp"  ! ,9N 5 6 MMv" 'uc25 8R8R <!,R0J% M&BCD / 0DJ\9] I%)^^Ik " MM- .%2 2D /557 * Uc5) *  #,<GHseooc2HNH  " "++*-Y)O%S##&L b""#  NN. 5_cJx H!59N9N9[9[![ \++KNyZhKijxsCsc:jb""#  ?? &##|3BLBRBRBTU$!Qa#4T__!55U U  %%3CMCSCSCUV41aaDOO#3#5!66V V!#[a?b*m%D+&k48(D61RSWXcSdRe)fgg%k!nellC(<v=TUYZefgZhUiTj)kll meT* #J4PU;-!XYYeAh, #LTRWXYRZ^L\!]^^'!,5$'UVZ[gVhUi%jkk!,q/37$'WX\]ijk]lXmWn%opp.&%EErccjjjjdk(rXt j dj jj_ jj_ | | td||tjdt||jr td||ddnd}||ddnd}||ddnd}||ddnd}||ddnd}j!|||||\}}}}}|*j#|j$ddd }|d |d }}|#j#|j$ddd d }||}j'|||}t)|j+}i}|j-|fd } t/j0|} | j3d t5| jj6d| } t9| D]\} } j:j=j5|Dcic]}|| | }}j?j|}jj@dkDr|jC}|jE}j:jG|j:jHr^jjJHj:jMjjOjjJjPjSjPjUjVjVjSjxjdz c_xj|z c_ jYdddycc}w#1swYxYw)a Run an optimisation step given a list of input_ids, attention_mask, and labels or a list of text and text_labels. Args: input_ids (list[`torch.LongTensor`]): List of tensors containing the input_ids (if not provided, text will be used) attention_mask (list[`torch.LongTensor`], , *optional*): List of tensors containing the attention_mask labels (list[`torch.FloatTensor`], *optional*): List of tensors containing the labels (if set to None, will default to input_ids) texts (list[`str`], *optional*): List of strings containing the text input (if not provided, input_ids will directly be used) texts_labels (list[`str`], *optional*): List of strings containing the text labels (if set to None, will default to text) Returns: `dict[str, Any]`: A summary of the training statistics rgNz@Step should include `input_ids` or `texts` as keyword arguments.ztBoth `input_ids` and `texts` argument are provided. `input_ids` will be ignored. Please provide only one of the two.zNo 'labels' or 'text_labels' are provided. When using an encoder-decoder architecture, 'labels' or 'text_labels' must be passed.Tpt)r* truncationpaddingreturn_tensorsrgrhct}|dD]S}|dvstj|Dcgc]}|| c}jjj ||<U|Scc}w)Nrrk)dictrnstackrqr"rr)data return_dictr_drZs rbcollatorz*IterativeSFTTrainer.step..collatorsm&KAw aCC',{{D3IqAcF3I'J'M'MdjjN_N_'`K$ a 4Js A) rn) batch_sizeshuffle collate_fnr)-r"trainstate global_steprntensorrqr#rrtr_loss_globalstep_last_loggedrtr>r? UserWarningr2rr&r*r{rkeysupdater from_dict set_formatrper_device_train_batch_size enumerater< accumulate compute_lossn_gpumeandetachbackwardsync_gradients max_grad_normclip_grad_norm_ parametersrVstep zero_gradrW_maybe_log_save_evaluate)rZrgrhrir|r} model_inputsmodel_inputs_names batch_dictr batch_datastep_dataloader_batchryloss tr_loss_steps` rbrzIterativeSFTTrainer.step;s8  :: ! !Q & <<,// 0@0@ADL+/::+A+AD (  _` `  "u'8 MM6  >l2t7N7NS  %.$9IaLt .<.H*d$0d!-a4*6*B|A AEAZAZ ~vulB > >65,  00$//dDae1L)5[(AF00NFS !,"3"3"56 ,' &&z2 g&$ yy<<  "/2 0HAu!!,,TZZ8 05GH58 H H((\B99??Q&99;D#{{}   ))$/##22tyy7N7N7Z$$44 --/ // ##%((*$$0%%**, &&!+& , --/7 0 0 0H 0 0s6O; OFOOO' cR|jjd|jj|jjzdk(r4|jjdk7r|j |j |jj |jj|jj zdk(r|jjdk7ri}|j|jjj}|xj|jzc_t||jj|jz z d|d<|j|d<|jj|_ |j|yyyy)Nrr learning_rate)r# eval_stepsrrevaluater% logging_steps_nested_gatherrritemroundr_get_learning_ratelog)rZlogstr_loss_scalars rbrz,IterativeSFTTrainer._maybe_log_save_evaluatesF 99   +zz%% (<(<<AdjjF\F\`aFa d//0 99 " " .zz%% (?(??1DI_I_cdId)+!%!4!4T\\!B!G!G!I!N!N!P  , $^tzz7M7MPTPlPl7l%mopqV (,(?(?(A_%/3zz/E/E,JeD /rcc|jj*t|jjj}n(|jjj dd}|j |t|!||y)Nr.r/)r]) r# hub_model_idr output_dirrrEcreate_model_cardrO_save_checkpoint)rZr"trialr]ras rbrz$IterativeSFTTrainer._save_checkpointsl 99 ! ! )dii22388J//55c:2>J *5  .rcr] dataset_nametagsc V|jsyt|jjdr^tj j |jjjs!|jjj}nd}| t}nt|tr|h}n t|}t|jjdr|jd|j|jt|||j||t!r.t"j$t"j$j'ndt)d}|j+tj j-|j.j0dy)a Creates a draft of a model card using the information available to the `Trainer`. Args: model_name (`str` or `None`, *optional*, defaults to `None`): Name of the model. dataset_name (`str` or `None`, *optional*, defaults to `None`): Name of the dataset used for training. tags (`str`, `list[str]` or `None`, *optional*, defaults to `None`): Tags to be associated with the model card. NrDunsloth_versionunslothz Iterative SFT) base_modelr]rrr wandb_url comet_url trainer_namez README.md)is_world_process_zerorQr"rCospathisdirrDsetrArBaddrrRrrrwandbrunget_urlrsavejoinr#r)rZr]rrr model_cards rbrz%IterativeSFTTrainer.create_model_cards)"))+  4::$$o 6rww}}TZZM^M^MlMl?m**88JJ <5D c "6Dt9D 4::$$&7 8 HHY  DOO$(!!**%-?-AeiiF[eii'')ae.0(    TYY%9%9;GHrc) NNNN)NNNNNNN)NNNNN)NNN)+__name__ __module__ __qualname____doc__rRrrBrrrrr rrrr rrtuplernoptim OptimizerrWLambdaLRrrrintboolrPrLr{ staticmethodr LongTensorrrempty_device_cacherrrr __classcell__)ras@rbr r 6s:&P)J HL04EI W imFJ$()-04%}IS/)*}Iu/1BBCD}I - }I uWd3<.@%@AB }I # )+=?UWee f }I%++//1I1I1R1RRS}I(0%,, 9UW\WcWc9c0d'e}I"(N+;T+A"BC}I SM!}I""##}I$ (~%}I~U#U=OUTcU ellELLbgbnbn@3F(()3FU--.3FU%%&3FCy 3F 3i 3F3Fj&]%%'7;;?37%),0 }0D!1!123}0!e&6&6!78}0e../0 }0 S " }0 tCy) }0(}0~0/%)&*,0 1ISM1Ism1ICcD() 1Ircr )*rr>pathlibrtypingrrrrndatasetsrtorch.utils.datar transformersr r r r r rrrrrrrrtransformers.trainer_utilsrtransformers.utilsrcoreriterative_sft_configrutilsrrpeftrrr r=rcrbrsj ,, '60 4@VI'VIrc