bie5 ddlZddlmZddlZddlZddlZddlmZddlm Z m Z ddl m Z ddl mZmZmZmZddlZddlZddlZddlZddlmcmZddlZddlmZmZddl m!Z!ddl"m#Z#m$Z$dd l%m&Z&dd lm'Z'dd l(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0dd l1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7d dl8m9Z9e4rddl:m;Z;ddlm?Z?ddl@mAZAe0rddlBZBe3rddlCmDZDmEZEGdde*ZFe GddZGe GddZH ddeIejdeKdeLdeeKdejf dZMe Gd d!ZNGd"d#e'ZOe Gd$d%ZPej dd&ejdeRejejeKffd'ZSd(e+deTeLeUffd)ZVdd*ejd+eKd,eeKeUfd-eKdejf d.ZWd/ej2jddfd0ZYdd1ZZGd2d3Z[d4Z\d5e9dee)fd6Z]deeTeLeKffd7Z^d5e9dd8fd9Z_dd:Z`dd;ZadZdd?Zee Gd@dAe.Zfe GdBdCe/ZgejfdDejdejfdEZid/ej2jdFejdGeKdHeKdeRejejejff dIZjd/ej2jdFejdGeKde2fdJZk dd/ej2jdKeKdLeldMeldej2jf dNZmdOeKdGeKdPejdejfdQZndRej2jdSejdGeKdTe,deRejejff dUZoejd/ej2jdSejdVeKdGeKdTe,f dWZpdXeeKdYeKdZeTeLeIeKfd[eKd\eTeLeIeKfd]eKd^eTeLeIeKffd_Zqd`eKd\eTeLeIeKfd^eTeLeIeKffdaZrdbejdOeKdGeKdeRejejffdcZsdddZtdeejdfe-deIeLfdgZu ddheeLdieLdjeLdkeeLdleIeLdmeeLdneLdoeeLdpeeLdqeeLdreeLde#fdsZvdeeLfdtZwdueLdvejddfdwZxdxejdejdeejeRejdyfffdzZydxejdejdeejeRejdyfffd{Zzdejfd|Z{ dd}eIeLd~eIeLdeTeLeIeUfdeIeUdeKdeKddfdZ|y)N)deque) dataclassfield)version)AnyLiteralOptionalUnion) Accelerator PartialState)AcceleratorState) ModelCard ModelCardData) pad_sequence)IterableDataset)BitsAndBytesConfigDataCollatorForLanguageModelingEvalPredictionGenerationConfigPreTrainedTokenizerBase TrainerStateTrainingArgumentsis_comet_available) ModelOutputis_peft_availableis_rich_availableis_torch_mlu_availableis_torch_npu_availableis_torch_xpu_available) ModelConfig)Console)Panel)Table)Text) LoraConfig PeftConfigc eZdZdZ ddddddeeeefdeeeeefde ded e f fd Z d eeeee e ee ffd e ee fffd Z xZS)DataCollatorForCompletionOnlyLMa Data collator used for completion tasks. It ensures that all the tokens of the labels are set to an 'ignore_index' when they do not come from the assistant. This ensure that the loss is only calculated on the completion made by the assistant. Args: response_template (`Union[str, list[int]]`): the template form that indicates the start of the response, typically something like '### Response: '. It can also be passed as tokenized ids, which can be useful when using a tokenizer that encodes the response differently if it does not have proper context. instruction_template (`Union[str, list[int]]`): the template form that indicates the start of the human instruction, typically something like '### Human: '. Useful for assistant-style conversation datasets. It can also be passed as tokenized ids. mlm (`bool`, *optional*, defaults to `False`): Whether to use masked language modeling in the underlying `DataCollatorForLanguageModeling` class. Note that this option currently has no effect but is present for flexibility and backwards-compatibility. ignore_index (`int`, *optional*, defaults to `-100`): The index to use to ignore the initial tokens with F)mlm ignore_index padding_freeresponse_templateinstruction_templater+r,r-cbt||d|i|tjdt||_t |tr-|jj|j d|_ n||_ ||_ t |tr-|jj|jd|_ n||_ |jsS|j rG|jj|jjk(rtjdt ||_||_y)Nr+zThis class is deprecated and will be removed in version 0.20.0. To train on completion only, please use the parameter `completion_only_loss` of `SFTConfig` instead.F)add_special_tokensa The pad_token_id and eos_token_id values of this tokenizer are identical. If you are planning for multi-turn training, it can result in the model continuously generating questions and answers without eos token. To avoid this, set the pad_token_id to a different value.)super__init__warningswarnDeprecationWarningr/ isinstancestr tokenizerencodeinstruction_token_idsr.response_token_idsr+ pad_token_id eos_token_id UserWarningr,r-) selfr.r/r+r,r-argskwargs __class__s L/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/trl/trainer/utils.pyr3z(DataCollatorForCompletionOnlyLM.__init__\s  $2C262  K  %9! *C 0)-)>)>t?X?Xmr)>)sD &*>D &!2 ' -&*nn&;&;Dt |dk(r`tjd|jd|jj|d|dt|j|d|ddf<t |dkDrt |dkDr|d|dkDrdg|z}t%t'||D]:\}\} } |dk7r|j|d|| | f<%|j|d|d| f<<t |t |ks|j|d||ddf<|j(r|j+d }|d|j-j/d|d<|j1d |j-j/dd z |d <|d|j-j/d|d<|j|d|d dk(<|d j3}t5j6|j9d|j:t4j< }t5j>||dk(t5j@|j9|j:t4j< fj/d|d <|d |d<t5j@|jCjEd zg|d<|d|d<|S)NlabelsrzCould not find response key `z` in the following instance: input_idszw. This instance will be ignored in loss calculation. Note, if this happens often, consider increasing the `max_length`.z Could not find instruction key `attention_mask position_ids)devicedtype cu_seq_lens_q cu_seq_lens_k max_length_k max_length_q)#r2 torch_callr/rangelennpwherer<tolistr4r5r.r9decoder?r,appendr; enumeratezipr-popbool unsqueezecumsumflattentorcharangesizerOint32cattensormaxitem)r@rFbatchiresponse_token_ids_start_idxidxresponse_token_ids_end_idxresponse_token_ids_idxshuman_token_ids_idxs assistant_idxhuman_token_ids human_idxstartend attn_maskflattened_position_ids indices_qrCs rDrUz*DataCollatorForCompletionOnlyLM.torch_calls"8,  $ $ ,3x=) X/3,88E(OA$6$:Q:QRS:T$TUVWX;C// ?1-cC#d>U>U:V4VW^^`a8;4 ;07MM78N8N7OOl>>00{1CA1FGHIjj$  -1,=,=E(OAqD)1MPSTXTkTkPl1l.GKFWFWE(OA'B(B'B$BC/ X43x=)3 W*,'')$%'XXeHoa.@DD[D[\]D^.^%_`a%beM// ?1-mmcRVRiRiNj>jkrrtu066}s4KbKbGc7cd e./14MM78N8N7OOl>>00{1CA1FGHIjj$  -1,=,=E(OAqD)"&"<"<!#%/!*<PQ@R*R!STU!V?I&%/!*? +,1MM:4;T;T:UUr>>00{1CA1FGHIjj$  -1,=,=E(OAqD),-134q8,Q/2I!2LL,-31E+E()237KMd3e)fE%C%ax8<8I8Ih59 5373D3Dh4C40 E./#6J2KKEIEVEVE(OA';B'?'A$ABg3 Wj    "23I!&{!3INN4D!E!O!OPQ!RE+ $-$4$4Q$7 8H$I$S$STU$VYZ$ZE. !#Hoinn.>?II!LE(O:>:K:KE(OE.1Q6 7&+>%:%B%B%D " &++A.7M7T7T\a\g\gI&+YY49:LL.335>T>[>[chcncn&il / "&+?%;E/ "%*LL2H2L2L2N2S2S2UXY2Y1Z$[E. !$).$9E. ! rEN)__name__ __module__ __qualname____doc__r r8listintr r`r3rdictrU __classcell__rCs@rDr)r)Gs.AE+)  "+) d3i0+)'uS$s)^'<=+)  +)  +)+)Zn4d3id38n.L(M#NnSWX[]`X`SannrEr)ceZdZUdZeed<dZeed<dZeed<dZ e ed<d Z e ed <d Z d e ee efd ee ej"ffdZy)DataCollatorForChatMLz3 Data collator for ChatML format datasets. r9r*r,N max_lengthprompt prompt_keymessages messages_keyc|jj td|j&t |jj d|_yy)NzTThe tokenizer does not have a pad token. Please set `pad_token_id` in the tokenizer.)r9r= ValueErrorrminmodel_max_lengthr@s rD __post_init__z#DataCollatorForChatML.__post_init__sE >> & & .st t ?? "!$.."A"A4HDO #rErFrGc g}g}g}g}g}|D]}|j|jd}|0||jdd} |jj | dd}d|vr||j} |jj | dd} |j| d|j ddd} |j | dd|vr|j | dnp|j dgt| dznN|j |dd|vr|j |dn!|j dgt|dz|j|dt|dddd} |j | d|j | d|jgt|dz}t| d}|d|d||d|j ||Dcgc]'}tj|tj )}}|Dcgc]'}tj|tj )}}|Dcgc]'}tj|tj )}}t|d |jj }t|d d }t|d |j }|Dcgc]'}tj|tj )}}|Dcgc]'}tj|tj )}}t|d |jj }t|d d }|||||d Scc}wcc}wcc}wcc}wcc}w)NrKFT)tokenizeadd_generation_promptrJ) truncationrpaddingreturn_tensorsr1rLrMrPleft) padding_side padding_valuer)rJrLrIpromptsprompt_attention_mask)getrrr9apply_chat_templaterr\rWr,rdrilongpadr=)r@rFrJrLprompts_input_idsrrIexampleformatted_promptrmessageformatted_messagetokenized_messagetokenized_promptlabelcompletion_start_idxidsmasks rD__call__zDataCollatorForChatML.__call__ s  "1 !G&{{4??DA ' !2!23CR8#'>>#E#EU$$F$ ')!$"3"34$(NN$F$Fe5%G%!%)NN%##!#'', %3%!  !2;!?@#w."))*;ZZEKLE%,,uEJJ7LL dnnFaFab ^&PQRV&@Q@QRL]^SU\\#UZZ@^^Rg h$d%**!E h h 1VZVdVdVqVqr #$9^_ `#,(%:   OZL _ hs&,M,M! ,M&,M+6,M0)r|r}r~rr__annotations__r,rrrr8rrrrrrdTensorrrErDrrsn'&L#JJ"L#"IM d38n!5M $sELL?P:QM rErceZdZUdZeed<dZeee fed<dZ e e ed<dZ e ed<d eee efd ee effd Zy) RewardDataCollatorWithPaddinga\ Reward DataCollator class that pads the inputs to the maximum length of the batch. Args: tokenizer (`PreTrainedTokenizerBase`): The tokenizer used for encoding the data. padding (`Union[bool, str, `PaddingStrategy`]`, `optional`, defaults to `True`): padding_strategy to pass to the tokenizer. pad_to_multiple_of (`int` or `None`, `optional`, defaults to `None`): If set will pad the sequence to a multiple of the provided value. return_tensors (`str`, `optional`, defaults to `"pt"`): The tensor type to use. r9TrNpad_to_multiple_ofptrfeaturesrGcng}g}g}d|dv}|D]h}d|vs d|vsd|vsd|vr td|j|d|dd|j|d|dd|sU|j|dj|jj||j|j |j }|jj||j|j |j }|d |d |d |d d d } |r*tj|tj}|| d<| S)Nmarginrinput_ids_choseninput_ids_rejectedattention_mask_chosenattention_mask_rejectedz{The features should include `input_ids_chosen`, `attention_mask_chosen`, `input_ids_rejected` and `attention_mask_rejected`)rJrL)rrrrJrLT)rrrr return_lossr) rr\r9rrrrrdrifloat) r@rfeatures_chosenfeatures_rejectedr has_marginfeature batch_chosenbatch_rejectedrls rDrz&RewardDataCollatorWithPadding.__call__rs!,  1G#'1'w6*'9,G; R  " "!();!<&-.E&F   $ $!()=!>&-.G&H   gh/03 14~~)) LL#66.. * ++ LL#66.. , !-[ 9%12B%C"0"='56F'G   \\& >> import torch >>> pad([torch.tensor([1, 2, 3]), torch.tensor([4, 5])]) tensor([[1, 2, 3], [4, 5, 0]]) >>> pad([torch.tensor([[1, 2], [3, 4]]), torch.tensor([[5, 6]])]) tensor([[[1, 2], [3, 4]], [[5, 6], [0, 0]]]) ``` rNrPrOrrightz&padding_side must be 'left' or 'right'c34K|]}td|yw)rN)slice).0ss rD zpad..s%GaeAqk%GsrM) rXrjshaperZrdfullrWrPrOr]rrtuple) rrrrt output_shape remainderoutputrm seq_start seq_sliceslicess rDrrs>N66G4q1774a8??AL% O&88 > O1I= =OZZW5 5}GTUJL\L\elmneoevev wF'" 1 6 !$Q!''!*4I W $IEF F)Y%;< %G17712;%G GGq &  M15sD&cleZdZUdZdZeed<dZeed<dZe e ed<de e e efd e e effd Zy ) DPODataCollatorWithPaddinga DPO DataCollator class that pads the tokenized inputs to the maximum length of the batch. Args: pad_token_id (`int` defaults to 0): The tokenizer's pad_token_id. label_pad_token_id (`int`, defaults to -100): The label used for masking. is_encoder_decoder (`bool` or `None`, `optional`, defaults to `None`): Whether you model has an encoder_decoder architecture. rr=r*label_pad_token_idFis_encoder_decoderrrGci}|djD]4}|jdr|jr|Dcgc]}tj||}}|j dr5|jdr$|j td|j }nE|jdrd}n1|j dsd|vr |j}ntd |d t|d | ||<|jd r$|j td|j }nU|jdr |j}n7|jdrd}n#|jdrd}ntd |d |dvrd}nd}|jdrtj}ntj}|Dcgc]}tj|||}}t|||||<|jdr+tj|Dcgc]}|| c}||<|Dcgc]}|| c}||<7|Scc}wcc}wcc}wcc}w)Nr) _input_ids_attention_mask_labels _pixel_valuesrrJzPadding is enabled, but the tokenizer is not configured with a padding token. Explicitly set `tokenizer.pad_token` (e.g. `tokenizer.pad_token = tokenizer.eos_token`) before calling the trainer.r)chosenrejected completiondecoderzUnexpected key in batch ''T) batch_firstrrrr)prompt_input_idsrrrrrr_logps)keysendswithrrd LongTensor startswithr=rrrfloat32int64rir) r@r padded_batchkexto_padrrrPs rDrz#DPODataCollatorWithPadding.__call__sT !!!#: =AzzWX**@HI"e..r!u5IFI X.QZZ 5L,,4",!?# )-(9(9 $56() &JKPY]^P^(,(?(? (+DQCq)IJJ&26t[h&iLOzz,/,,4",!?# )-(9(9 I.(,(?(? $56() O4() (+DQCq)IJJII'- '. zz/2 %  % JRR2ell2a5>RFR&)& \h&iLOH%"',,/I"1/I"J Q3;"z0ConstantLengthDataset.__init__..s Q/A-BrEzDEither `dataset_text_field` or `formatting_func` should be provided.FrJT)r4r5r6r9r>concat_token_iddataset seq_lengthinfinite current_sizemax_buffer_sizeshuffleappend_concat_tokenr1r?formatting_funcr pretokenizedr7datasetsDatasetr column_names)r@r9rrrrrnum_of_sequenceschars_per_tokenr>rrr1rs ` rDr3zConstantLengthDataset.__init__ds-   0  #9B9O9Oy55Ua $  )O;>NN #6 "4  )o.I MMM   D  +#BD cd d!$.w9I9I8KcKc8d$eG ko   # |(C $D #-0@#@D )D #rEc,t|jSr{)rWrrs rD__len__zConstantLengthDataset.__len__s4<<  rEc#Kt|j}d}|rgd}} ||jk\rn< |j|j t ||t |dz }L|jrtj||jr|}n!|j||jdd}g}|D]/}|jr||jgz}|j!|1g}t#dt ||j$D]>} || | |j$z} t | |j$k(s.|j| @|jrtj||D]D} |xj&dz c_t)j*| t)j*| dF|ryy#t$r+|jrt|j}nd}YYwxYww) NTrrKF)r1rrJrM)rJrI)iterrrr\rnextrW StopIterationrrrandomrr9r1rrextendrVrrrdr) r@iterator more_examplesbuffer buffer_lentokenized_inputs all_token_idstokenized_inputrFrmrJrs rD__iter__zConstantLengthDataset.__iter__s % !#QJF!5!55MM$"6"6tH~"FG#fRj/1J ||v&  #) #'>>t/F/FSX$2$$ M#3 6++&59M9M8N&NO$$_5 6H1c-0$//B /)!a$//.AB y>T__4OOI. /||x(# !!Q&!!&!1!1'!:#..w7 G%}}#' #5(- !s60G?:G-CG?A?G?G?-G<5G?;G<<G?) NNFrrg @rTTT)r|r}r~rr3r rrrErDrrAs9 L  4Al!+rErceZdZUdZeed<dZeed<dZeed<dZ eed<dZ eed <e jd e jd eeeffd Zd efdZeded efdZy)RunningMomentsz Calculates the running mean and standard deviation of a data stream. Reference: https://github.com/OpenLMLab/MOSS-RLHF/blob/40b91eb2f2b71b16919addede0341d2bef70825d/utils.py#L75 acceleratorrmeanrMstdvargW:countxsrGc>|jjrt|j|\}}}n*|j}t j |d\}}|j |j }}||jz }|j|z}||z}|j|jz|dz|jz|z|z z}||z} |xj||z|z jz c_| |z } | |z|dz z j jj|_ | j|_ ||_|j||z|dz z j jjfS)zT Updates running moments from batch's moments computed across ranks F)unbiasedr rM) ruse_distributedget_global_statisticsnumelrdvar_meanrrr rrksqrtr) r@r!xs_meanxs_varxs_countdelta tot_countnew_sumold_sumtot_sumnew_vars rDupdatezRunningMoments.updatesh    + +(=d>N>NPR(S %GVXxxzH#nnR%@OFG!--/6<<>$))#JJ) 8#((TZZ'%(TZZ*?(*JY*VVG# eh&288:: I%i'9q=9@@BGGINNP<<> ||~ 1X\ BIIKPPRWWYYYrE json_pathc|jjr\tj|d}t j |dddz}t |dd 5}|j|d d d y y #1swYy xYw) zDSave the content of this instance in JSON format inside `json_path`.cF|Dcic]\}}|dk7s ||c}}Scc}}w)Nrr)rrvs rDrz-RunningMoments.save_to_json..s)\]HtRXSTVWabfsasAHtHts ) dict_factoryr T)indent sort_keys wutf-8encodingN)ris_main_process dataclassesasdictjsondumpsopenwrite)r@r3 save_dict json_stringfs rD save_to_jsonzRunningMoments.save_to_jsonsv    + +#**4>tuI**YqDIDPKiw7 %1 $ % % , % %s A44A=ct|d5}|j}ddd|dd|itjS#1swY&xYw)z3Create an instance from the content of `json_path`.r<r=Nrr)rDreadrBloads)clsrr3rHtexts rDload_from_jsonzRunningMoments.load_from_jsonsL)g . !668D ?{?djj.>??  s AA N)r|r}r~rr rrrrrr rdno_gradrrr2r8rI classmethodrOrrErDrrs D%OCNCNE5U]]_ZZ%u *=ZZ6%c%@@@@rErr!c|j|j}tj|j ||j n|j g|j}|j |}|\}}||z }tj||z dzj|dn|}|j |}||z } |j|| j||jfS)z Computes element-wise mean and variance of the tensor across processes. Reference: https://github.com/OpenLMLab/MOSS-RLHF/blob/40b91eb2f2b71b16919addede0341d2bef70825d/utils.py#L57C1-L73C75 rOr rM) torOrdrisumr&reducemulrk) rr!rrO sum_and_count global_sumr  global_meansum_var global_vars rDr%r%s {!! "BLL"&&(4>& !:==#8%**, FFrE eval_predc |\}}|jdk(rtj|d}tjt ||Dcgc]!\}}t ||D] \}}|dk7s |#c}}}}}tj|Dcgc]}|D] }|dk7s | c}}}n|dddf|dddfk(}t |j }|dkDr1tjd|dt|dddfd t||}||}tj|d}tj||k(t jj} d | iScc}}}}wcc}}w) Nr )axisr*rrMz There are z out of zu instances where the predictions for both options are equal. These instances are ignored in the accuracy computation.raccuracy)ndimrXargmaxarrayr^rrUr4r5rWr?rrrk) r] predictionsrI predictionrplbl equal_maskequal_predictions_countras rDcompute_accuracyrks#K1ii !4 hh(+K(@ w w$:uQTU_afQg wXaknrvkvQ wQ w FS5eSssd{3S3ST !A&+ad*;; "%jnn&6"7 "Q & MM45Xc+aQRdBS>T=UVgg ":+.  $ii !4 xx v-U;@@BGGIH  !!3 xSs E2,E2E: E: rilength pad_valuedimc |j||k\r|St|j}||j|z ||<tj||tj ||j |jdzg|S)Nrrn)rfrrrdrhonesrPrO)rirlrmrnpad_sizes rD pad_to_lengthrs@sx {{36!  %S!11 yyEJJ V]][[    rEmodelc|jD].}t|tjjs(d|_0y)Nr)modulesr7rdnnDropoutrg)rtmodules rDdisable_dropout_in_modelrzOs2--/ fehh.. /FHrEc R||z}|||zk7rt|d|d|d||z |S)Nz, inexact division: z / z = )r)abcustom_error_messageqs rD exact_divrUsH QAAEz011EaSA3cRSVWRWQXYZZ HrEc"eZdZdZdZdZdZy)PerPromptStatTrackeraI Class for tracking statistics per prompt. Mainly used to calculate advantage for the DPPO algorithm Args: buffer_size (`int`): Size of the buffer to keep for each prompt. min_count (`int`): Minimum number of samples to keep in the buffer before calculating the mean and std. c.||_||_i|_yr{) buffer_size min_countstats)r@rrs rDr3zPerPromptStatTracker.__init__hs&" rEctj|}tj|}tj|}tj|}|D]}|||k(}||jvr#t |j |j|<|j|j|t|j||jkr.tj|}tj|dz}nGtj|j|}tj|j|dz}||z |z |||k(<|S)N)maxlengư>) rXrdunique empty_likerrrrrWrrr) r@rrewardsr advantagesrprompt_rewardsrrs rDr2zPerPromptStatTracker.updatems$((7#((7#7#]]7+  JF$W%67NTZZ'%*$2B2B%C 6" JJv  % %n 54::f%&7www'ffWo,wwtzz&12ffTZZ/047-;d-Bc,IJw&( ) JrEc |jjDcic];\}}|tj|tj|t |d=c}}Scc}}w)N)rrr )ritemsrXrrrW)r@rr6s rD get_statszPerPromptStatTracker.get_statssJW[WaWaWgWgWijtqRSBGGAJrvvay3q6JJjjjsAA"N)r|r}r~rr3r2rrrErDrr]s *krErc|jD]\}t|tjjsdvr |j tj }NtfddDsct|dsp|jjtj k(s|j tj}y)Nnormc3&K|]}|v ywr{r)rrnames rDrz.peft_module_casting_to_bf16..sNqdNs)lm_head embed_tokenswtewpeweight) named_modulesr7rdrw LayerNormrTranyhasattrrrPbfloat16)rtryrs @rDpeft_module_casting_to_bf16rs++-7 f fehh00 1Vt^YYu}}-F N$MN Nvx(==&&%--7#YYu~~6F 7rE model_argsc|jr:td|j|j|j|j}|S|j rtd}|Sd}|S)NT) load_in_4bitbnb_4bit_compute_dtypebnb_4bit_quant_typebnb_4bit_use_double_quantbnb_4bit_quant_storage) load_in_8bit)rr torch_dtyperuse_bnb_nested_quantr)rquantization_configs rDget_quantization_configrst0#-#9#9 * > >&0&E&E#-#9#9     0  # rEctjjs trdt j iSy)N)rdcuda is_availablerr local_process_indexrrErDget_kbit_device_maprs. zz $:$<LN6677rEzOptional[PeftConfig]c |jduryts tdt|j|j |j |j|jd|j|j|j }|S)NFzYou need to have PEFT library installed in your environment, make sure to install `peft`. Make sure to run `pip install -U peft`.none) task_typertarget_modules lora_alpha lora_dropoutbias use_rslorause_doramodules_to_save) use_peftrrr&lora_task_typelora_rlora_target_modulesrrrrlora_modules_to_save)r peft_configs rDget_peft_configrse#   6  ++   !55((,, (($$"77 K rEc`tjdgj|jtj|jj z}tj |j|j}|dkDr!tj|d|zzd|zz S|S)a1 Get the exponent cap of a value. This is used to cap the exponent of a value to avoid overflow. The formula is : log(value.dtype.max) E.g. For float32 data type, the maximum exponent value is 88.7228 to 4 decimal points. Args: value (`torch.Tensor`): The input tensor to obtain the data type decimal (`int`): The number of decimal points of the output exponent cap. eg: direct calling exp(log(torch.float32.max)) will result in inf so we cap the exponent to 88.7228 to avoid overflow. rMr ) rdzerosrTrPfinforjlogrOfloor)valuedecimal vdtype_maxvdtype_log_maxs rD get_exp_caprsaS!$$U[[1EKK 4L4P4PPJYYz*--ell;NFMPQk5;;~G 3 4r7{ BeWeerEcz|dkr t|n|}tjtj||S)Nr)rj)rrdexpclamp)rcaps rDcap_exprs. #a+e SC 99U[[C0 11rEdfcPts tdt}td}|jD]}|j ||j D]5\}}|j|jtj7|j|y)NzgThe function `print_rich_table` requires the `rich` library. Please install it with `pip install rich`.T) show_lines) r ImportErrorr"r$columns add_columniterrowsadd_rowastyper8rZprint)rconsoletablecolumn_rows rDprint_rich_tablers   u  iG T "E**!  !++-13 szz#--/01 MM%rEzT{% for message in messages %}{{' ' + message['content']}}{% endfor %}{{ eos_token }}z{% for message in messages %}{{message['role'].capitalize() + ': ' + message['content'] + ' '}}{% endfor %}{% if add_generation_prompt %}{{ 'Assistant:' }}{% endif %}ceZdZUdZeed<y)OnlineTrainerStaterepisodeN)r|r}r~rrrrrErDrrs GSrErcjeZdZUdZedddiZeed<edddiZe e ed <eddd iZ e e ed <eddd iZ e eed <edddiZeed<edddiZe eed<edddiZeed<edddiZeed<edddiZeed<edddiZe eded<edddiZe eed<ed dd!iZeed"<eddd#iZe eed$<ed%dd&iZe ed'<eddd(iZe eed)<eddd*iZe eed+<eddd,iZe eed-<eddd.iZe eed/<eddd0iZe eed1<eddd2iZe eed3<eddd4iZ e eed5<ed6dd7iZ!e ed8<fd9Z"xZ#S):OnPolicyConfiga Base configuration class for on-policy trainers. This class includes only the parameters that are specific to some on-policy training. For a full list of training arguments, please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may differ from those in [`~transformers.TrainingArguments`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the command line. Parameters: run_name (`str` or `None`, *optional*, defaults to `None`): Name of the run. dataset_num_proc (`int` or `None`, *optional*, defaults to `None`): Number of processes to use for processing the dataset. num_mini_batches (`int`, *optional*, defaults to `1`): Number of minibatches to split a batch into. total_episodes (`int` or `None`, *optional*, defaults to `None`): Total number of episodes in the dataset. local_rollout_forward_batch_size (`int`, *optional*, defaults to `64`): Per rank no grad forward pass in the rollout phase. num_sample_generations (`int`, *optional*, defaults to `10`): Number of debugging samples generations (i.e., `generate_completions` calls) throughout training. response_length (`int`, *optional*, defaults to `53`): Length of the response. stop_token (`str` or `None`, *optional*, defaults to `None`): Specifies the stop token to use for text generation. This parameter is mutually exclusive with `stop_token_id`. - `None`: No stop token is applied, unless `stop_token_id` is specified. - `'eos'`: Uses the tokenizer's `eos_token`. stop_token_id (`int` or `None`, *optional*, defaults to `None`): Specifies the ID of the stop token to use for text generation. If `None`, no stop token ID is applied, unless `stop_token` is specified. This parameter is mutually exclusive with `stop_token`. temperature (`float`, *optional*, defaults to `0.7`): Sampling temperature. missing_eos_penalty (`float` or `None`, *optional*, defaults to `None`): Penalty applied to the score when the model fails to generate an EOS token. This is useful to encourage to generate completions shorter than the maximum length (`max_new_tokens`). The penalty must be a positive value. sft_model_path (`str`, *optional*, defaults to `"EleutherAI/pythia-160m"`): Path to the SFT model. world_size (`int` or `None`, *optional*, defaults to `None`): Number of processes (GPUs) to use for the training. num_total_batches (`int` or `None`, *optional*, defaults to `None`): Number of total batches to train. micro_batch_size (`int` or `None`, *optional*, defaults to `None`): Micro batch size across devices (HF's `per_device_train_batch_size` * `world_size`). local_batch_size (`int` or `None`, *optional*, defaults to `None`): Batch size per GPU (HF's `per_device_train_batch_size` * `gradient_accumulation_steps`). batch_size (`int` or `None`, *optional*, defaults to `None`): Batch size across devices (HF's `per_device_train_batch_size` * `world_size` * `gradient_accumulation_steps`). local_mini_batch_size (`int` or `None`, *optional*, defaults to `None`): Mini batch size per GPU. mini_batch_size (`int` or `None`, *optional*, defaults to `None`): Mini batch size across GPUs. push_to_hub (`bool`, *optional*, defaults to `False`): Whether to push the model to the Hub after training. rhelpzLog every X updates steps. Should be an integer or a float in range `[0,1)`. If smaller than 1, will be interpreted as ratio of total training steps.)defaultmetadata logging_stepsNzWhether to use bf16 (mixed) precision instead of 32-bit. Requires Ampere or higher NVIDIA architecture or Intel XPU or using CPU (use_cpu) or Ascend NPU. If not set, it defaults to `True` if `fp16` is not set.bf16zName of the run.run_namez6Number of processes to use for processing the dataset.dataset_num_procrMz,Number of minibatches to split a batch into.num_mini_batchesz(Total number of episodes in the dataset.total_episodes@z3Per rank no grad forward pass in the rollout phase. local_rollout_forward_batch_sizezaNumber of debugging samples generations (i.e., `generate_completions` calls) throughout training.num_sample_generations5zLength of the response.response_lengthzoSpecifies the stop token to use for text generation. This parameter is mutually exclusive with `stop_token_id`.eos stop_tokenzSpecifies the ID of the stop token to use for text generation. If `None`, no stop token ID is applied, unless `stop_token` is specified. This parameter is mutually exclusive with `stop_token`. stop_token_idgffffff?zSampling temperature. temperaturezPenalty applied to the score when the model fails to generate an EOS token. This is useful to encourage to generate completions shorter than the maximum length (`max_new_tokens`). The penalty must be a positive value.missing_eos_penaltyzEleutherAI/pythia-160mzPath to the SFT model.sft_model_pathz3Number of processes (GPUs) to use for the training. world_sizez!Number of total batches to train.num_total_batcheszTMicro batch size across devices (HF's `per_device_train_batch_size` * `world_size`).micro_batch_sizezXBatch size per GPU (HF's `per_device_train_batch_size` * `gradient_accumulation_steps`).local_batch_sizeznBatch size across devices (HF's `per_device_train_batch_size` * `world_size` * `gradient_accumulation_steps`). batch_sizezMini batch size per GPU.local_mini_batch_sizezMini batch size across GPUs.mini_batch_sizeFz4Whether to push the model to the Hub after training. push_to_hubcv|j |j n |j|_t| yr{)rfp16r2r)r@rCs rDrzOnPolicyConfig.__post_init__s*'+yy'8Odii  rE)$r|r}r~rrrrrrr r`rr8rrrrrrrrrrrrrrrrrrrrrrrrs@rDrrs=@! D M5! ! D(4.$,-Hhsm',RS'hsm"HIc%*DE%NHSM-2OP-$c#( w #C !34OS,1  ,J($) q $M8C=12K,1   ,% (23NC!&OP!J (-=>(x}',pq'hsm',tu'hsm!& . !J ,145,8C=&+89&OXc]PQK   rErboolsc|jd}||j|ztj|||jz}tj |dj S)a Takes an N-dimensional bool tensor and returns an (N-1)-dimensional tensor of integers giving the position of the first True in each "row". Returns the length of the rows (bools.size(-1)) if no element is True in a given row. Args: bools (`torch.Tensor`): An N-dimensional boolean tensor. dtype (`torch.dtype`, optional): The desired data type of the output tensor. Defaults to `torch.long`. Returns: `torch.Tensor`: An (N-1)-dimensional tensor of integers indicating the position of the first True in each row. If no True value is found in a row, returns the length of the row. rKrrp)rftyperdrerOrvalues)rrProw_len zero_or_indexs rDfirst_true_indicesr sW$jjnGvmmE22U\\'QV_d_k_k5llM 99] + 2 22rEquery_responsesr=context_lengthc||k7}|jd|jz }t||j}t j ||d}||||ddd}|j |jd} t|dd|df|k(dz |z} | | t j| jd| j| fjd| fS) a5 Computes the reward logits and the rewards for a given model and query responses. Args: model (`torch.nn.Module`): The model used to compute the reward logits. query_responses (`torch.Tensor`): The tensor containing the query responses. pad_token_id (`int`): The token ID representing the pad token. context_length (`int`): The length of the context in the query responses. Returns: tuple: - `reward_logits` (`torch.Tensor`): The logits for the reward model. - `final_rewards` (`torch.Tensor`): The final rewards for each query response. - `sequence_lengths` (`torch.Tensor`): The lengths of the sequences in the query responses. rMrTF)rJrLrN return_dictoutput_hidden_states use_cacherKNrS) rbrgetattrbase_model_prefixrd masked_fillscore hidden_statesr rerfrOsqueeze) rtrr=rrLrN lm_backbonerJr reward_logitssequence_lengthss rD get_rewardrs2% 4N!((+n.A.A.CCL%!8!89K!!/N?AFI %!! FKK 4 4R 89M)/!^_:L*MQ]*]^abbess  LL++A.}7K7K L    '"+  rEc||k7}|jd|jz }tj||d}||||ddS)a Performs a forward pass through the model with the given query responses and pad token ID. Args: model (`torch.nn.Module`): The model to perform the forward pass. query_responses (`torch.Tensor`): The tensor containing the query responses. pad_token_id (`int`): The token ID representing the pad token. Returns: `ModelOutput`: The output of the model, including hidden states. rMrT)rJrLrNrr)rbrrdr)rtrr=rLrNrJs rDforwardrs^(% 4N!((+n.A.A.CCL!!/N?AFI %!!  rEper_device_train_batch_sizerrcddl}tj}|j}|dddk7r"||d<|dddd}|rd d i|d <n|rd d i|d <nt |d rut |j ddrt|j jnt |j dd}|&|dddk(r|j||zd|zdd|j||^}}|j|S)a Prepares the model for training with DeepSpeed (both for stage 2 and 3), configuring the appropriate settings based on the model and batch size. Args: model (`torch.nn.Module`): The model to be prepared for DeepSpeed training. per_device_train_batch_size (`int`): The training batch size per device. Returns: `torch.nn.Module`: The model initialized and configured with DeepSpeed for training. rNzero_optimizationstager_train_micro_batch_size_per_gpuF)r$prescale_gradientswall_clock_breakdownenabledTrrconfig hidden_sizes hidden_sizer)z$zero_optimization.reduce_bucket_sizez4zero_optimization.stage3_param_persistence_thresholdz-zero_optimization.stage3_prefetch_bucket_size)rtr() deepspeedr deepspeed_plugindeepspeed_configrrr(rjr)r2 initializeeval) rtr rrr+r, config_kwargsr*rs rDprepare_deepspeedr1s-"')::$55M()'2a7:U 67.;<\.]"'$) %.$5M& ! %.$5M& ! 5( #5<<>ELL--.U\\=$?  &=9L+Mg+VZ[+[$$@Kk@YPRU`P`IJ$$5$GIEA JJL LrEr responsescLt||k(jd}dgt|jdz z|jdgz}t j |jd|jj|}t j|||kD|}|S)aA Truncates the responses at the first occurrence of the stop token, filling the rest with pad tokens. Args: stop_token_id (`int`): The token ID representing the stop token where truncation occurs. pad_token_id (`int`): The token ID representing the pad token used to fill the truncated responses. responses (`torch.Tensor`): The tensor containing the responses to be truncated. Returns: `torch.Tensor`: The truncated responses tensor with pad tokens filled after the stop token. rKrMrS) r rarWrfrrdrerOviewr)rr=r2 trunc_idxsnew_sizeidxspostprocessed_responsess rDtruncate_responser9Ks $I$>?II"MJsc)..*+a/0IOOA4F3GGH I5<< *93C3C D I I8 TD#// 4*;Ll[ ""rErqueriesgeneration_configc"|jd}||k7}tj||d}|j|||dd}tj|j d}tj ||jdd|dffd|fS)a Generates sequences from the language model backbone in a way that does not affect padding tokens. Args: lm_backbone (`torch.nn.Module`): The language model backbone used for generation. queries (`torch.Tensor`): The tensor containing the input queries. pad_token_id (`int`): The token ID representing the pad token. generation_config (`GenerationConfig`): The configuration for the generation process. Returns: tuple: - `generated_sequences` (`torch.Tensor`): The concatenated tensor of input queries and generated sequences. - `logits` (`torch.Tensor`): The logits output from the generation process. rMrT)rJrLr;return_dict_in_generate output_scoresNrp)rrdrgeneratestackscoresrh sequences) rr:r=r;rrLrJrlogitss rDr?r?bs.]]1%N ,N!!'N?A>I  ! !%, $"F[[ *F 99gv//>?0BCD! Lf TTrErcg}g}|jd}td||D]=}||||z} t|| ||\} } |j| |j| ?t ||d} t |dd} | j d| jdd|} | j dg| jddd|} | | fS)NrrrrKr )rrVr?r\rr4)rtr:rr=r;rlogitssrrmqueryquery_responserCpadded_query_responsespadded_logitsss rDbatch_generationrJsOGq!J 1j"B C A @@A!)     "  ~.v ! [bcHN488=S=Y=YZ\=]^_j`jk(^((Gn.B.B12.FG TN !> 11rE bos_token_idprompt_len_input_ids prompt_tokenschosen_prompt_len_input_ids chosen_tokensrejected_prompt_len_input_idsrejected_tokensc|x|dk(s ||ddk7r|g|dz|d<dg|dz|d<|dk(s ||ddk7r|g|dz|d<dg|dz|d<|dk(s ||ddk7r|g|dz|d<dg|dz|d<|||fS)NrrrMrr)rKrLrMrNrOrPrQs rDadd_bos_token_if_neededrSs 1 $ FX8YZ[8\(\1=OaAb0bM, -67S=I`;a5aM1 2 &! +|}M_?`ab?c/c1=OaAb0bM, -67S=I`;a5aM1 2 (A -QcAdefAg1g3?.?SeCf2fO. /89s_Md=e7eO3 4 - 88rEr>ct|ddk(s ||ddk7r(|dj||djdt|ddk(s ||ddk7r(|dj||djd||fS)NrJrrKrLrM)rWr\)r>rOrQs rDadd_eos_token_if_neededrUs = %&!+|}[?YZ\?]/]k")),7&'..q1 ?; '(A-Q\A]^`Aa1a $++L9()003 / ))rErJct||k(jd}dgt|jdz z|jdgz}t j |jd|jj|}t j|||kD|}t jt j|||kDd}||fS)a Truncates the input tensor from the right side after the first occurrence of the stop token. Args: input_ids (`torch.Tensor`): The tensor containing the responses to be truncated stop_token_id (`int`): The token ID representing the stop token where truncation occurs pad_token_id (`int`): The token ID representing the pad token used to fill the truncated responses Returns: tuple: - `output_ids` (`torch.Tensor`): The truncated responses tensor with pad tokens filled after the stop token - `mask` (`torch.Tensor`): The mask tensor to indicate the padding tokens rKrMrSr) r rarWrfrrdrerOr4r ones_like)rJrr=r5r6r7 output_idsrs rDtruncate_rightrYs*$I$>?II"MJsc)..*+a/0IOOA4F3GGH I5<< *93C3C D I I8 TD""9dZ.?NJ   U__Y7 9JA ND t rEc6trtjjyt rtj jyt rtjjytjjy)a3Empties the cache of the available torch device. This function checks for the availability of different torch devices (XPU, MLU, NPU, CUDA) and empties the cache of the first available device it finds. If none of the specific devices are available, it defaults to emptying the CUDA cache. N) rrdxpu empty_cachermlurnpurrrErDr\r\sV   !   !   rEinputsr9c|j|d}|Dcgc]}|j|jd c}Scc}w)ay Decodes the input tensor and strips the padding tokens. Args: inputs (`torch.Tensor`): The input tensor to be decoded. tokenizer (`transformers.PreTrainedTokenizerBase`): The tokenizer used to decode the input tensor. Returns: `list[str]`: The list of decoded strings with padding tokens stripped. F)skip_special_tokensr) batch_decodereplace pad_token)r_r9decodedds rDdecode_and_strip_paddingrgs>$$V$GG8? @1AIIi))2 . @@ @s#> base_model model_name hub_model_id dataset_nametags wandb_url trainer_nametrainer_citation paper_titlepaper_id comet_urlc ft||dd|dg|} tj| fidtt j dj dd|d |d |d |d |d | d|d|d|d| dtddtddtddtddtd} | S)a Generate a `ModelCard` from a template. Args: base_model (`str` or `None`): Base model name. model_name (`str`): Model name. hub_model_id (`str`): Hub model ID as `username/model_id`. dataset_name (`str` or `None`): Dataset name. tags (`list[str]`): Tags. wandb_url (`str` or `None`): Weights & Biases run URL. comet_url (`str` or `None`): Comet experiment URL. trainer_name (`str`): Trainer name. trainer_citation (`str` or `None`, defaults to `None`): Trainer citation as a BibTeX entry. paper_title (`str` or `None`, defaults to `None`): Paper title. paper_id (`str` or `None`, defaults to `None`): ArXiv paper ID as `YYMM.NNNNN`. Returns: `ModelCard`: A ModelCard object. transformerslicensegenerated_from_trainer)rhr library_namelicencerirl template_pathtrlztemplates/lm_model_card.mdrhrirjrkrmrrrnrorprq trl_versiontransformers_versionpytorch_versionrddatasets_versionrtokenizers_version tokenizers)rr from_templater8 pkg_resourcesfilesjoinpathr) rhrirjrkrlrmrnrorprqrr card_datacards rDgenerate_model_cardr sX#& . . I  " " ---e4==>Z[\     "   "    " *    EN %^4  (  !,! "#<0# D& KrEc~tsytjtjjSy)zt If Comet integration is enabled, return the URL of the current Comet experiment; otherwise, return `None`. N)rcomet_mlget_running_experimenturlrrErDget_comet_experiment_urlrWs4  &&(4..0444 rErrcts tdtj}||j ||yy)a If Comet integration is enabled logs a table to the Comet experiment if it is currently running. Args: name (`str`): Table name. table (`pd.DataFrame`): The Pandas DataFrame containing the table to log. zLThe comet-ml is not installed. Please install it first: pip install comet-mlN) tabular_datafilename)rModuleNotFoundErrorrr log_table)rr experiments rDlog_table_to_comet_experimentrdsB  !"pqq002J%$?rEr.c|j\}}|j}|Dcgc]}|j}}|jd}tj||j j d}||j dz|z}|jd|} |Dcgc]}|jd|} }| jd} | dk(} | jr6t| jtjjn|} | ddd| f}| Dcgc] }|ddd| f}}|s|S|g|Scc}wcc}wcc}w)a Shift non-zero elements in the mask and corresponding tensors to the left. This function operates on a binary mask and any number of additional tensors with the same dimensions as the mask. For each row, non-zero values are shifted to the leftmost positions. Then, columns that contain only zeros across all rows are truncated from the mask and tensors. Visually, this operation can be represented as follows: ``` [[0, 0, x, x, x, x], -> [[x, x, x, x], [0, x, x, x, 0, 0]] [x, x, x, 0]] ``` Args: mask (`torch.Tensor`): 2D tensor (binary mask) with shape `(N, M)`. *tensors (`torch.Tensor`) One or more 2D tensors with the same shape as `mask`. These tensors will be processed alongside `mask`, with non-zero values shifted and excess zero columns truncated in the same manner. Returns: `torch.Tensor`: Updated binary mask with non-zero values flushed to the left and trailing zero columns removed. `*torch.Tensor` Updated tensors, processed in the same way as the mask. Example: ```python >>> mask = torch.tensor([[0, 0, 1, 1, 1], [0, 1, 1, 0, 0]]) >>> tensor = torch.tensor([[9, 9, 2, 3, 4], [9, 5, 6, 9, 9]]) >>> new_mask, new_tensor = flush_left(mask, tensor) >>> print(new_mask) tensor([[1, 1, 1], [1, 1, 0]]) >>> print(new_tensor) tensor([[2, 3, 4], [5, 6, 0]]) ``` rMrprSrN) rclonercrdrerOragatherrUrrrTint8)rrrM mask_copyrfirst_non_zeroposidx_roll mask_rollrolled_tensorscol_sums empty_colsfirst_empty_col flushed_maskflushed_tensorss rD flush_leftrvs^P ::DAq I")*Qqwwy*G*%%!%,N ,,q!1!1 2 < GsE,E5Ec|j\}}|j}|Dcgc]}|j}}tj|}|j d}tj ||j jd}||jdz |z} |jd| } |Dcgc]}|jd| } }| jd} | dk7} | jr6t| jtjj n|}| dd|df}| Dcgc] }|dd|df}}|s|S|g|Scc}wcc}wcc}w)zs Shift non-zero elements in the mask and corresponding tensors to the right. See `flush_left` for details. rMrprSrN)rrrdfliplrrcrerOrarrUrrrTr)rrrrrr flipped_maskrrrrrrnon_empty_colsfirst_non_empty_colrrs rD flush_rightrsl ::DAq I")*Qqwwy*G*<< *L!((Q(/N ,,q!1!1 2 < KsE(E- E2c Z|jtjtjfvrutj|d|j dj d}tj|Dcgc]}tj|dc}}||z }|Sg}t||D]^\}}tj|d}|j d|j dj d} |j| `tj|}|Scc}w)aw A memory-efficient implementation of the common `log_softmax -> gather` operation. This function is equivalent to the following naive implementation: ```python logps = torch.gather(logits.log_softmax(-1), dim=-1, index=index.unsqueeze(-1)).squeeze(-1) ``` Args: logits (`torch.Tensor`): Logits tensor of shape `(..., num_classes)`. index (`torch.Tensor`): Index tensor of shape `(...)`, specifying the positions to gather from the log-softmax output. Returns: `torch.Tensor`: Gathered log probabilities with the same shape as `index`. rK)rnindexrp) rPrdrfloat64rrarr@ logsumexpr^F log_softmaxr\) rCrselected_logitslglogsumexp_valuesper_token_logps row_logits row_labels row_logpsrow_per_token_logpss rDselective_log_softmaxrs&|| u}}55,,v2U__R=PQYYZ\] ;;f'U(C'UV),<<  &)&%&8 8 "J  jb9I"+"2"2rAUAUVXAY"2"Z"b"bce"f   " "#6 7 8 ++o6 (Vs6D(r completionsrrstep num_samplesc  ts tdt}tddd}|j dd|j dd |j D]}|j |d d |j d dd ||t |k\rd}n|dkry|tjtt ||} | D cgc]} ||  }} | D cgc]} ||  }} |jD cic]\} } | | D cgc]} | |  c} }} } } | D cgc]} ||  }} tt |D]j} |j D cgc] } || | d} } |jt|| t|| g| || d|jlt|dd|d}|j|ycc} wcc} wcc} wcc} } } wcc} wcc} w)u Print out a sample of model completions to the console with multiple reward metrics. This function creates a nicely formatted table showing prompt-completion pairs, useful for monitoring model outputs during training. It requires the `rich` library to be installed. Args: prompts (`list[str]`): List of prompts. completions (`list[str]`): List of completions corresponding to the prompts. rewards (`dict[str, list[float]]`): Dictionary where keys are reward names and values are lists of rewards. advantages (`list[float]`): List of advantages corresponding to the prompts and completions. step (`int`): Current training step number, used in the output title. num_samples (`int` or `None`, *optional*, defaults to `None`): Number of random samples to display. If `None` (default), all items will be displayed. Example: ```python >>> from trl.trainer.utils import print_prompt_completions_sample >>> prompts = ["The sky is", "The sun is"] >>> completions = [" blue.", " in the sky."] >>> rewards = {"Correctness": [0.123, 0.456], "Format": [0.789, 0.101]} >>> advantages = [0.987, 0.654] >>> print_prompt_completions_sample(prompts, completions, rewards, advantages, 42) ╭──────────────────────────── Step 42 ─────────────────────────────╮ │ ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┓ │ │ ┃ Prompt ┃ Completion ┃ Correctness ┃ Format ┃ Advantage ┃ │ │ ┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━┩ │ │ │ The sky is │ blue. │ 0.12 │ 0.79 │ 0.99 │ │ │ ├────────────┼──────────────┼─────────────┼────────┼───────────┤ │ │ │ The sun is │ in the sky. │ 0.46 │ 0.10 │ 0.65 │ │ │ └────────────┴──────────────┴─────────────┴────────┴───────────┘ │ ╰──────────────────────────────────────────────────────────────────╯ ``` zvThe function `print_prompt_completions_sample` requires the `rich` library. Please install it with `pip install rich`.Tz bold white) show_header header_styleexpandPrompt bright_yellow)style Completion bright_greenz bold cyanr)rjustify Advantagez bold magentaNrz.2fFzStep )rtitle border_style)rrr"r$rrrWrsamplerVrrr% add_sectionr#r)rrrrrrrr reward_nameindicesrmkeyval reward_valuespanels rDprint_prompt_completions_samplers`   "  iG dd KE X_5 \8||~J  KIJ [H #g, &K A  --c'l 3[A'./!71:///67!{1~7 7BI--/RRhc331AQ11RR-45jm5 5 3w< =D\\^LcGCLOC01L L d71:&[^(<e}eQ[\]Q^_bPce  %uTF^, WE MM%071R5Ms0 G%* G* G4 G/%G42 G;)H/G4)rrN)Ncpu)rK)r))FF)rGN)NNNNr{)}r@importlib.resources resourcesrrBrr4 collectionsrrrimportlib.metadatartypingrrr r rnumpyrXpandaspdrdtorch.nn.functionalrw functionalrtorch.utils.data accelerater r accelerate.stater huggingface_hubrrtorch.nn.utils.rnnrrrtrrrrrrrrtransformers.utilsrrrrrrtrainer.model_configr! rich.consoler" rich.panelr# rich.tabler$ rich.textr%rpeftr&r'r)rrrrrr8rrrrrPrr%rrrkrsModulerzrrrrrrrr DataFramerSIMPLE_SFT_CHAT_TEMPLATESIMPLE_CHAT_TEMPLATErrrr rrr`r1r9r?rJrSrUrYr\rgrrrrrrrrrErDrs+ (&00 0-4+,   /$  +p&Epf _ _  _ D JJ J^(, ? %,, ???! ?  \\ ?D PP PfGOGT 7@7@ 7@t5:G\\G 5<<s *+GG(""""4U 3C""J  %,,    c5j@Q  X[  ejeqeq  EHHOO  &k&kR7 AS8T&Xd38n5 0F2f$2  $ rC   t &t  t n38**3ell33./ 88??/-2\\/IL/^a/ 5<<u|| 34/d 88??\\ B`e3 88??39<3DH3X\3 XX__3l#S## #Y^YeYe#.$U$U+0<<$UGJ$U_o$U 5<< %&$UN2 88??2 \\2'*2 2 ( 22@93-99T#Y'9"% 9 T#Y' 9 $' 9#tCy.)9, * *&*3S >&: *MQRUW[\_W`R`Ma *||,/?B 5<< %&:!$AU\\A>UAZ^_bZcA4'+!%"#G GGG3- G s) G } GGsmG#GsmG}GGT (3- @@BLL@T@$>*U\\>*U\\>*eELLRWX]XdXdfiXiRjDj>k>*B*ell*ell*uU\\SXY^YeYegjYjSkEk?l*< ELL R T #YTcT#tE{" #TU T  T  T TrE