bi ddlZddlmZmZmZmZddlZddlmZmZddl m Z ddl m Z ddl mZe jej e jdk\r8dd lmZeed d ed d d ed d ed d d dZn,ed d ed d d ged d ed d d dZ dde deddeefdZde fdZ ddeeefde deedeefdZy)N)CallableLiteralOptionalUnion)DatasetValue)version) AutoTokenizer)ConstantLengthDatasetz4.0.0)Liststring)dtypeid)contentrole) completionprompt)chatml instruction tokenizermessages_field)messages conversationstoolscfd}|S)z return a callable function that takes in a "messages" dataset and returns a formatted dataset, based on the tokenizer apply chat template to the dataset along with the schema of the list of functions in the tools list. ct|dtrIg}tt|D]+}|j j ||d-|Sj |dS)NrF)tokenizer isinstancelistrangelenappendapply_chat_template)examples output_textsirrrs X/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/trl/extras/dataset_formatting.pyformat_datasetz9conversations_formatting_function..format_dataset0s h~.q14 8L3x789 ##11(>2J12MX]ej1k  00.1ITYaf0g g)rrrr*s``` r)!conversations_formatting_functionr-(s h r+cfd}|S)z return a callable function that takes in an "instructions" dataset and returns a formatted dataset, based on the tokenizer apply chat template to the dataset c t|dtrXg}tt|dD]:}d|d|dd|d|dg}|j j |d<|Sd|ddd|ddg}j |dS)Nruser)rr assistantrF)rr)r&r'r(converted_samplers r)r*z8instructions_formatting_function..format_datasetDs hx($ /L3x123 e#0B10EF(Xl5KA5NO$ ##I$A$ABR]b$A$cd  e   HX,>?$,1GH  001AE0R Rr+r,)rr*s` r) instructions_formatting_functionr3>s S" r+datasetreturnct|trd|jvr;|jdtdk(r"t j dt |d|Sd|jvr<|jdtdk(r"t j dt |d|Sy|jtdk(r t j dt|Sy)a Finds the correct formatting function based on the dataset structure. Currently supported datasets are: - `ChatML` with [{"role": str, "content": str}] - `instruction` with [{"prompt": str, "completion": str}] Args: dataset (Dataset): User dataset tokenizer (AutoTokenizer): Tokenizer used for formatting Returns: Callable: Formatting function if the dataset format is supported else None rrz%Formatting dataset with chatml formatrrz*Formatting dataset with instruction formatN)r rfeaturesFORMAT_MAPPINGlogginginfor-r3)r4rrs r) get_formatting_func_from_datasetr;Xs'7# )) ) +~h/GG DE8JPUVV g.. .0N84LL DE8OUZ[[     !> > LLE F3I> > r+)N)r9typingrrrrdatasetsrr packagingr transformersr trainer.utilsr parse __version__r r8r!r-r3r;r,r+r)rCsB55#&17==%%&-'--*@@5xD#A5W_dhKijk&+(t&DPU\dimPnoN %8=uS[`dGefg&+(t&DPU\dimPnoNmq.56Q.R[cdh[i, 6gk 711 2?LU]^bUc hr+