bi;HddlZddlZddlZddlmZmZmZmZddlZddl m Z ddl m cm Z ddlmZddlmZmZmZmZmZmZmZmZmZddlmZddlmZddlmZdd l m!Z!dd l"m#Z#d d l$m%Z%d d l&m'Z'd dl(m)Z)m*Z*m+Z+m,Z,m-Z-erddl.m/Z/erddl0Z0Gdde'Z1y)N)AnyCallableOptionalUnion)Dataset) AutoModelForCausalLMBaseImageProcessor DataCollatorFeatureExtractionMixinGenerationConfigPreTrainedModelPreTrainedTokenizerBaseProcessorMixinis_wandb_available)TrainerCallback)EvalPrediction)is_peft_available)prepare_deepspeed)unwrap_model_for_generation) GKDConfig) SFTTrainer)DataCollatorForChatMLdisable_dropout_in_model empty_cachegenerate_model_cardget_comet_experiment_url) PeftConfigceZdZddgZ ddeeeeje fdeeeje fdee dee dee d eee e e e ffd eeeeeefd eeege fd eeed eej0j2ej0j4j6fdeeej8ej8gej8fdeddeeffd Ze ddZd dZ ed!dZ! d!dejde e eej8e"ffdee#dej8ffd Z$ d"dee dee dee ee dffdZ%xZ&S)# GKDTrainertrlgkdNmodel teacher_modelargs data_collator train_dataset eval_datasetprocessing_classcompute_metrics callbacks optimizerspreprocess_logits_for_metrics peft_configrformatting_funccLd|_t||j}t||||||||| | | | |  |j i}nIt |ts td|j }|ddvr|dntt|d|d<t |trtj|fi|}|jrt|j|j rt#||j$|_n"|j$j)|d|_|j*|_|j,|_|j.|_|j0|_t3|j4|j.dd |j6rdnd|j8j: |_t?|jj<d rQ|jj<j@0|jj<j@|j<_ yyy) NF) tokenizer max_length) r&r'r(r)r*r+r,r-r.r/r0zfYou passed teacher_model_init_kwargs to the GKDConfig, but your teacher_model is already instantiated. torch_dtype)autoNT)evaluation_moder)max_new_tokens temperature do_sampletop_k use_cache pad_token_id eos_token_id)!remove_unused_columnsrr3super__init__teacher_model_init_kwargs isinstancestr ValueErrorgetattrtorchrfrom_pretraineddisable_dropoutrr$is_deepspeed_enabledr acceleratorr% prepare_modellmbdabetar8seq_kdr r7gradient_checkpointingr*r<generation_confighasattrr=)selfr$r%r&r'r(r)r*r+r,r-r.r/r0rA __class__s R/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/trl/trainer/gkd_trainer.pyr@zGKDTrainer.__init__>s&&+"-8HUYUdUde   ''%-+!*G#+    ) ) 1(* %M3/x )-(F(F %-];~M*-8U$=m$LM &m 4 mS )0@@lRklM    $TZZ 0  $ $!2=$BRBR!SD !%!1!1!?!? _c!?!dD ZZ II ++kk !1..((#::e..;; "  DJJ00. A ,,99E26**2N2N2[2[D " " /F Bc ||z }||z }tj|d}tj|d}|dk(rtj||dd}n|dk(rtj||dd}ntj||j }tj tj|tjd|z z|tj|zgd} tj| |dd} tj| |dd} || zd|z | zz}| |d k7} || }|d k(rW|!|j jz S|j|jd|jdzz S|d k(r|jS|d k(r|jS|S) a Compute the generalized Jensen-Shannon Divergence loss for knowledge distillation using F.kl_div. See Eq. (1) of https://huggingface.co/papers/2306.13649 for the definition. Args: student_logits: Tensor of shape (batch_size, sequence_length, vocab_size) teacher_logits: Tensor of shape (batch_size, sequence_length, vocab_size) labels: Tensor of shape (batch_size, sequence_length) with -100 for padding tokens to ignore when computing loss beta: Interpolation coefficient between 0 and 1 (default: 0.5) temperature: Softmax temperature (default: 1.0) reduction: Specifies the reduction to apply to the output (default: 'batchmean') Returns: loss: Scalar tensor with the generalized JSD loss )dimrnoneT) reduction log_targetr)dtype batchmeansummean) F log_softmaxkl_divrFtensorr\ logsumexpstacklogr_sizer`) student_logitsteacher_logitslabelsrMr8rZstudent_log_probsteacher_log_probsjsdmixture_log_probs kl_teacher kl_studentmasks rTgeneralized_jsd_losszGKDTrainer.generalized_jsd_losss8(+5'+5MM.bAMM.bA 19((,.?6^bcC QY((,.?6^bcC<<,=,C,CDD % .1t81DDFWZ_ZcZcdhZiFijk! "35FRXeijJ"35FRXeijJ#q4x:&==C  T>Dd)C  #-3-?3779txxz) lSWWYRURZRZ[\R]`c`h`hij`kRkEl l % 779  & 88: JrUc||d|d}|jjtj5|j|d|d}ddd|djd}|j dd|dz dddf}j dd|dz dddf} |ddd|df} |j || | |j} t|r| |fS| S#1swYxYw) N input_idsattention_mask)rurvpromptsrrWrk)rirjrkrM) r%evalrFno_gradshapelogitsrsrMr) rRr$inputsreturn_outputsnum_items_in_batchoutputs_studentoutputs_teacherprompt_lengthsshifted_student_logitsshifted_teacher_logitsshifted_labelslosss rT compute_losszGKDTrainer.compute_losss,[)!"23  ! ]]_ "00 -%&671O  *003!0!7!7>A;MPR;RTU8U!V!0!7!7>A;MPR;RTU8U!V)!^_*<=((11! )   +9o&BdB1  s C((C1c|j|d|jdd|d}|j}tj|}|j }|d|||k(<d|||k(<|||fS)Nrwprompt_attention_maskT)rurvrPreturn_dict_in_generater]r)generateget sequencesrF ones_likeclone)r$r|rPr<generated_outputsgenerated_tokensnew_attention_mask new_labelss rTgenerate_on_policy_outputsz%GKDTrainer.generate_on_policy_outputss"NNY'!::&=tD/$( + -66"__-=>%++-   #59Jz\1 2CD /<? @!3Z??rUr|r~returncX|jrnt|j|j5}|j |||j |j j\}}}ddd|d<|d<|d<tj|jkrdt||j5}|j |||j |j j\}}}ddd|d<|d<|d<t |-|||}|S#1swYxYw#1swY7xYw)aa Perform a training step for the Generalized Knowledge Distillation (GKD) model. This method implements the on-policy learning approach described in the GKD paper. With probability `self.lmbda`, it generates new responses using the student model, which are then used for training instead of the original inputs. Nrurvrk) rNrr%rJrrPr*r<randomrLr? training_step) rRr$r|r~unwrapped_model new_input_idsrrrrSs rTrzGKDTrainer.training_steps6 ;;,T-?-?AQAQR Ve@D@_@_#VT-C-CTEZEZEgEgA= 1: #0F; '9F# $)F8  ==?djj (,UD4D4DE @D@_@_#VT-C-CTEZEZEgEgA= 1: #0F; '9F# $)F8 w$UF4FG #    s7D37D D D) model_name dataset_nametagsc |jsyt|jjdr^tj j |jjjs!|jjj}nd}| t}nt|tr|h}n t|}t|jjdr|jd|j|jtjd}t!|||j"||t%r.t&j(t&j(j+ndt-d|dd }|j/tj j1|j2j4d y) a Creates a draft of a model card using the information available to the `Trainer`. Args: model_name (`str` or `None`, *optional*, defaults to `None`): Name of the model. dataset_name (`str` or `None`, *optional*, defaults to `None`): Name of the dataset used for training. tags (`str`, `list[str]` or `None`, *optional*, defaults to `None`): Tags to be associated with the model card. N _name_or_pathunsloth_versionunslothan @inproceedings{agarwal2024on-policy, title = {{On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes}}, author = {Rishabh Agarwal and Nino Vieillard and Yongchao Zhou and Piotr Stanczyk and Sabela Ramos Garea and Matthieu Geist and Olivier Bachem}, year = 2024, booktitle = {The Twelfth International Conference on Learning Representations, {ICLR} 2024, Vienna, Austria, May 7-11, 2024}, publisher = {OpenReview.net}, url = {https://openreview.net/forum?id=3zKtaqxLhW}, }GKDzPOn-Policy Distillation of Language Models: Learning from Self-Generated Mistakesz 2306.13649) base_modelr hub_model_idrr wandb_url comet_url trainer_nametrainer_citation paper_titlepaper_idz README.md)is_world_process_zerorQr$configospathisdirrsetrBrCaddupdate _tag_namestextwrapdedentrrrwandbrunget_urlrsavejoinr& output_dir)rRrrrrcitation model_cards rTcreate_model_cardzGKDTrainer.create_model_card3sG"))+  4::$$o 6rww}}TZZM^M^MlMl?m**88JJ <5D c "6Dt9D 4::$$&7 8 HHY  DOO$??$ )!!**%-?-AeiiF[eii'')ae.0%j!    TYY%9%9;GHrU) NNNNNNNNN)NNNNN)Ng?g?r^)FN)N)NNN)'__name__ __module__ __qualname__rrrr nnModulerCrr rdictrr r rrrlistrtuplerFoptim Optimizer lr_schedulerLambdaLRTensorr@ staticmethodrsrrrintrr __classcell__)rSs@rTr!r!;sJCG@D$(04+/EI FJ59Vbhl.2.2!T\oryy#=>?T\_bii<=T\y! T\  - T\  ( T\uWd3<.@%@ABT\# )+=?UWee f T\"(N+;T+A"BCT\D12T\%++//1I1I1R1RRST\(0%,, 9UW\WcWc9c0d'eT\l+T\ "(+!T\lZeDDL!CF@@.rvYY(,S% c8I2J-J(Kaijman @%)&*,0 >ISM>Ism>ICcD() >IrUr!)2rrrtypingrrrrrFtorch.nnrtorch.nn.functional functionalradatasetsr transformersrr r r r r rrrtransformers.trainer_callbackrtransformers.trainer_utilsrtransformers.utilsrmodelsr models.utilsr gkd_configr sft_trainerrutilsrrrrrpeftrrr!rUrTrsy 11    :50&6!#vIvIrU