biw VddlZddlZddlZddlmZddlmZddlZddlm Z ddl m Z ddl m Z ddlmZmZmZmZddlmZddlmZmZmZmZdd lmZerdd lmZmZm Z m!Z!m"Z"m#Z#m$Z$dd l%m&Z&gd Z'Gd de jPZ) dde)dee*dee+de)fdZ,GddeZ-y)N)deepcopy)Optional) PartialState)hf_hub_download)EntryNotFoundErrorHFValidationErrorLocalEntryNotFoundErrorRepositoryNotFoundError) load_file)GenerationMixinPreTrainedModelis_torch_npu_availableis_torch_xpu_available)is_peft_available) PeftConfig PeftModelPeftModelForCausalLMPeftModelForSeq2SeqLMPromptLearningConfigget_peft_modelprepare_model_for_kbit_training)is_deepspeed_zero3_enabled)ztransformer.h.{layer}zmodel.decoder.layers.{layer}zgpt_neox.layers.{layer}zmodel.layers.{layer}ceZdZdZdZdZdZdZese ne e e fZ dfd Z edZe ddZedZed Ze dd Zd Zd Zd ZdZddZxZS)PreTrainedModelWrappera. A wrapper class around a (`transformers.PreTrainedModel`) to be compatible with the (`~transformers.PreTrained`) class in order to keep some attributes and methods of the (`~transformers.PreTrainedModel`) class. Attributes: pretrained_model (`transformers.PreTrainedModel`): The model to be wrapped. parent_class (`transformers.PreTrainedModel`): The parent class of the model to be wrapped. supported_args (`list`): The list of arguments that are supported by the wrapper class. N)v_head)scorec t|||_|j|_|j|_t |dd|_t |dd|_d|_t|dr|j|_ t|dr|j|_ t|dr|j|_ ||_ ||_d|_|||_yy)Nis_loaded_in_8bitFis_loaded_in_4bitgradient_checkpointing_disablegradient_checkpointing_enableenable_input_require_gradsdefault)super__init__pretrained_modelconfigprepare_inputs_for_generationgetattrrris_sequential_parallelhasattrr r!r"supports_rm_adapterrm_adapter_namepolicy_adapter_namer)selfr& score_moduler,r-kwargs __class__s S/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/trl/models/modeling_base.pyr%zPreTrainedModelWrapper.__init__Ss  0&-- -=-[-[*!()9;NPU!V!()9;NPU!V&+# #%E F2B2a2aD / #%D E1A1_1_D . #%A B.>.Y.YD +#6 .#,  #%DJ $cZ |p|jdd}|jdd}|jdd}|jdd}|j|\}} } | jdd} n d}d}i}i} i} d} t|ts t dd} |j } t|trd | vr| d nd}d | vr| d nd}nt|d d}t|d d}|s|r d | vrtjdd| i| d <tr|t|ts t dt|trztr t|d| }nd}t"j$j't"j$j)|d}|s|tr|tjd||rtj*|}n4t"j$j-|}tj*|}|j.j*|j0g|i| }t3j*|||| }tj4dn|j.j*|g|i| }||s|r t7|fi| }t9||}tj4dnst||j:rF|}|Yt|t<rI|s|r t7|fi| }t9||}tj4dnt dt?|trCt|t2r3d} tA|dr%t|jBtDr t d| s | t d| r||jG||| }|d|d}nddi}||fi||}d}t|trt"j$j)|d}t"j$j)|d}t"j$j)|d }t"j$j)|d!}d}t"j$j'|}t"j$j'|s^t"j$j'|s?|jI|||| \}}}}|!||jI|||| dd!"\}}}}d}nd}|rtJntLjN} |rind#dd$}!|rP|r0i}"D](}#t||#| }|"jQ| |fi|!*n| |s|n|fi|!}"n|jS}"| |_*| |_+|r|jY"%|S#tttt f$rd}YBwxYw)&ay Instantiates a new model from a pretrained model from `transformers`. The pretrained model is loaded using the `from_pretrained` method of the `transformers.PreTrainedModel` class. The arguments that are specific to the `transformers.PreTrainedModel` class are passed along this method and filtered out from the `kwargs` argument. Args: pretrained_model_name_or_path (`str` or `transformers.PreTrainedModel`): The path to the pretrained model or its name. *model_args (`list`, *optional*)): Additional positional arguments passed along to the underlying model's `from_pretrained` method. **kwargs (`dict`, *optional*): Additional keyword arguments passed along to the underlying model's `from_pretrained` method. We also pre-process the kwargs to extract the arguments that are specific to the `transformers.PreTrainedModel` class and the arguments that are specific to trl models. The kwargs also support `prepare_model_for_kbit_training` arguments from `peft` library. N peft_configreward_adapterreward_adapter_name is_trainableFtokenzThe `reward_adapter` argument should be a string representing the name of local path or the Hub id to the Reward Modeling adapter. load_in_8bit load_in_4bitrr device_mapzThe `device_map` argument is not provided. We will override the device_map argument. to set the entire model on the current device. If you want to set the model on multiple devices, please provide a custom `device_map` argument.zLThe `peft_config` argument should be an instance of `peft.PeftConfig` class.zadapter_config.jsonr:zE`peft_config` argument ignored since a peft config file was found in )r9r:zTrained peft adapter loadedzpeft adapter initialisedzNpretrained_model_name_or_path should be a string or a PreTrainedModel, but is Tactive_peft_configz7PromptLearningConfig is not supported for PPO training.z2reward_adapter can only be used with a PeftModel. )r0r,r-r,zmodel.safetensorspytorch_model.binpytorch_model.bin.index.jsonzmodel.safetensors.index.json)r: model_namemodel_index_namecpu map_location weights_only) state_dict)-pop _split_kwargsget isinstancestr ValueError_get_current_devicer)loggingwarningrrrrr rr ospathexistsjoinfrom_pretraineddirnametransformers_parent_classbase_model_name_or_pathrinforr(supported_pretrained_model_architecturesr typer+r@r$add_and_load_reward_modeling_adapter_get_checkpoint_from_hubsafe_load_filetorchloadupdaterI is_peft_modelcurrent_device post_init)$clspretrained_model_name_or_path model_argsr1r6r7r8r9trl_model_argspretrained_kwargspeft_quantization_kwargsr:rdrerrremote_adapter_configlocal_adapter_presenttrained_adapter_configremote_adapter_dirr&r0multi_adapter_argsmodelis_resuming_training safe_filenamefilenamesharded_index_filenamesafe_sharded_index_filename is_shardeduse_safefiles_to_download loading_func load_kwargsrI shard_files$ r3rWz&PreTrainedModelWrapper.from_pretrainednsJ$   **]D9K#ZZ(8$?N"(**-BDT"U !::ne.B l +  ;#::k[eCfkl l 3S 9 "1,;5-#-))-%$&GGNN277<<@]_t3u$v !%)>)JPaPc*OO_89; )-7-G-GHe-f*)+9N)O&-7-G-GHZ-[*$Q3#@#@#P#P*BB$EO$Sd$ $-#<#<$&CR^fk$  :;#P3#@#@#P#P1$4>$BS$ *(,=+J,,6,((66F 'T$LL!;< 5s7c7c d< &:6F+X$(9'F((2($$22BK#P  78<=>@   *I6 $ +-ABz$779MH%%^__!;QR R ~9CC .2EUDL!-'+#6"  #8!? $M(:MnM $ 3S 9GGLL)FH[\Mww||$ACVWH%'WW\\2OQo%p "*,'',,7TVt*u 'Jww~~m4HGGNN8,}0MPSPlPl$1* QmQM+Z9M#(9(AY\YuYu(53##6)G ZvZVM#4jBV $H$H-5>5::L ("uVZ.[K#!#J&7Q #29&"'$ #)),x*O;*OP Q".hhM!i]h!iJ7AACJ+-  OOzO 2 q+,CEVXop1,0)1sV V*)V*c d}d}d} d} t|||}||| | fS#ttttf$r t j j|r|} nV t|||} nF#ttttf$r*d} tjdt|d|dYnwxYw| rt 5} tj| } dddn #1swYnxYwt} djD]5\}t!fd|j"Ds%|j%|7d} Y*wxYw) NTFr?zA z model is loaded from 'zX', and no v_head weight is found. This IS expected if you are not resuming PPO training. weight_mapc3&K|]}|v ywN).0moduleks r3 zBPreTrainedModelWrapper._get_checkpoint_from_hub..~sK66Q;K)rrr rr rSrTrUrQrRr]openjsonrbsetitemsanysupported_modulesadd)rgr&rhindex_filenamer:rCrDrzrursrxindex_file_namefindexvrs @r3r_z/PreTrainedModelWrapper._get_checkpoint_from_hubPsj!# ! "&-HD*J8LLL9#$;=NPgh "ww~~n-"0 &55(#'O +,CEVXop+0(OOT"2344KLiKjkpq$/*)a IIaLE)))%(E!!,/5571DAqKS5J5JKK)--a01" 5 "sSC EC& "AE+EEct}tjjs t r |j St rd|j Sy)a< Get the current device. For GPU & XPU, we return the local process index using the `accelerate.PartialState` object to handle corner cases when running scripts in distributed environments. Returns: current_device (`Union[int, str]`): The current device. znpu:rE)rracuda is_availablerlocal_process_indexr)rgstates r3rPz*PreTrainedModelWrapper._get_current_devicesJ :: " " $(>(@,, , # %%3345 5r4cd}trddlm}d}i}i}i}|jD]U\}}||jvr|||<n|||<|s"|j j vs;|||<||vsE|j|W|||fS)z| Separate the kwargs from the arguments that we support inside `supported_args` and the ones that we don't. Fr)rT)rpeftrrsupported_args__code__ co_varnamesrJ) rgr1check_peft_kwargsrsupported_kwargsunsupported_kwargs peft_kwargskeyvalues r3rKz$PreTrainedModelWrapper._split_kwargss "   < $   ,,. 4JCc((((- %*/"3' 9BBNNN',K$00*..s3 4 !3[@@r4c|j||d|jtjj |d}d}tjj |s t |d|}n|}|rtntj} |rind dd } | |fi| } |jD](tfd | jDs&} ni} | jD]O\}} |vs d j |j!d d d}|j#|j%| |<Q| dj&\}}td| jD}t)j*|||j#|j%|j,}|j/| |j1D] }d|_ |S#t$rttjj |d}d}tjj |s- t |d|}n#t$r}td|d}~wwxYw|}YwxYw)a] Add and load a reward modeling adapter. This method can only be used if the model is a `PeftModel` and if you have initialized the model with the `reward_modeling_adapter_id` argument, pointing to the id of the reward modeling adapter. The latest needs also to contain the score head in order to produce the reward. F)r9zadapter_model.binr?zadapter_model.safetensorsTzYCould not find adapter model in the Hub, make sure you have the correct adapter model id.NrErFc3&K|]}|v ywrr)rnamescore_name_candidates r3rzNPreTrainedModelWrapper.add_and_load_reward_modeling_adapter..sVD'4/Vr.weightc3$K|]}d|v yw)biasNr)rrs r3rzNPreTrainedModelWrapper.add_and_load_reward_modeling_adapter..sL$v~Ls)r)devicedtype) load_adaptertrainrSrTrVrUr ExceptionrOr`rarbsupported_rm_modulesrkeysrsplittorPshapennLinearrload_state_dict parameters requires_grad)rgr&adapter_model_id adapter_namer:ru safe_loadinglocal_filenameexcr{r|adapter_state_dict score_name score_dictrparamkey_name num_labels hidden_dimhas_biasrrs @r3r^z;PreTrainedModelWrapper.add_and_load_reward_modeling_adapters\ %%&6 SX%Y 77<< 02EF ww~~h' .!0$'",&N)5~5:: (buVZ.[ ).HKH$'$<$<  V*>? ""$ ]]_ 4 5 5 5!#-%) !  ! "3!@!@!D ZZ 23F 4 ))$*B*BC ""$  4 4s +8C,,C5)NNFN)NrArB)reward_model_adapterNr)__name__ __module__ __qualname____doc__rYrrrrr rrr\r% classmethodrWr_rPrKr^rrrIrfr __classcell__r2s@r3rr;s !%N#%!" 35J K-dh&6__B &71M1Mf"AA:\`DDL "F6" "r4rrrnum_shared_layerspatternreturnc>tr td|jDcgc]\}}| }}}t|}|/|D]}|j |}d|_|j S||j|}n5tD],j|tfd|Ds*}n| tdg} g} d} |jD]0\} } || vrd} | r| j|  | j| 2| D]+}|j |}d|_|j |}-| D]}|j |}d|_|#t| dk(rtjd|j Scc}}w) a3 Creates a static reference copy of a model. Note that model will be in `.eval()` mode. Args: model (`PreTrainedModelWrapper`): The model to be copied. num_shared_layers (`int`, *optional*): The number of initial layers that are shared between both models and kept frozen. pattern (`str`, *optional*): The shared layers are selected with a string pattern (e.g. "transformer.h.{layer}" for GPT2) and if a custom pattern is necessary it can be passed here. Returns: `PreTrainedModelWrapper` zDeepSpeed ZeRO-3 is enabled and is not compatible with `create_reference_model()`. Please instantiate your reference model directly with `AutoModelForCausalLM.from_pretrained()`.F)layerc3&K|]}|v ywrr)rrpattern_candidates r3rz)create_reference_model..usI$,Irz#Layer pattern could not be matched.TrzNPattern passed or found, but no layers matched in the model. Check for a typo.)rrOnamed_parametersr get_parameterrrformatLAYER_PATTERNSrappendlenrQrR)rrrrnrparameter_names ref_model param_namershared_param_listunshared_param_listshared_parameterr_param _ref_paramrs @r3create_reference_modelrPs "# A  &+%;%;%=>TQq>O>I ) (J++J7E"'E  (~~..'8.9!/   1 8 8?P 8 Q III+   >??..0- f d?$    $ $T *  & &t , -(9 ##J/#,,Z8 9*$ '' 3#$s#671<hi >> g?s FcveZdZdZdZdZdZd fd ZdZe jdZ dZ dZ d ZxZS) GeometricMixtureWrappera Geometric Mixture generation wrapper that samples from the logits of two model's geometric mixture. Args: model (`PreTrainedModel`): The model to be wrapped. ref_model (`PreTrainedModel`): The reference model. generation_config (`GenerationConfig`): The generation config. mixture_coef (`float`, *optional* - default: 0.5): The mixture coefficient. rFct|||_|j|_||_||_||_||_yr)r$r%rrr'rgeneration_config mixture_coefr)r/rrrrrrr2s r3r%z GeometricMixtureWrapper.__init__s>  ll "!2( r4c&|j|i|Sr)forwardrs r3__call__z GeometricMixtureWrapper.__call__st||T,V,,r4c|j|i|}|j}|j|i|j}tjj j |j|zd|jz |zzd|_|S)Nr)dim)rrlogitsrrar functional log_softmaxr)r/rr1 model_outputs model_logitsref_model_logitss r3rzGeometricMixtureWrapper.forwards" D3F3 $++ )4>>4:6:AA$xx22>>    0 0A8I8I4I\3Y Y_a ? r4cd|d<|jj|i|}|jj|i|}|S)NF use_cache)rrr(r)r/rr1 model_inputsrs r3r(z5GeometricMixtureWrapper.prepare_inputs_for_generationsH#{?tzz??PP 8DNN 8 8$ I& Ir4c8|jjyr)rr_validate_model_class)r/s r3rz-GeometricMixtureWrapper._validate_model_classs ((*r4c8|jj|Sr)rr_validate_model_kwargs)r/ model_kwargss r3rz.GeometricMixtureWrapper._validate_model_kwargsszz00>>r4)g?N)rrrrmain_input_name_supports_cache_class_supports_static_cacher%r rainference_moderr(rrrrs@r3rrsS"O!"-U  +?r4r)NN).rrQrScopyrtypingrratorch.nnr acceleraterhuggingface_hubrhuggingface_hub.utilsrrr r safetensors.torchr r` transformersr r rrtransformers.utilsrrrrrrrrr#transformers.integrations.deepspeedrrModulerintrNrrrr4r3r+s   #+ :ii0KRRYYRlfjH !H6>smHU]^aUbHHV4?o4?r4