biIddlZddlmZddlmZmZmZmZddlm Z GddejZ Gdde Z Gd d e Z y) N)AutoModelForCausalLMAutoModelForSeq2SeqLMis_torch_npu_availableis_torch_xpu_available)PreTrainedModelWrapperc(eZdZdZfdZdZxZS) ValueHeadze The ValueHead class implements a head for GPT2 that returns a scalar for each output token. c Bt|t|ds|jdd}n |j}|rt j |nt j|_t|dr |j}t|dr |j}nPt|drD|jr8t|dr,t|jdr|jj}t jd|_t j|_y)Nsummary_dropout_probg? hidden_sizeword_embed_proj_dimis_encoder_decoderdecoderr)super__init__hasattrpopr nnDropoutIdentitydropoutr rrrLinearsummaryFlattenflatten)selfconfigkwargsr r __class__s Y/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/trl/models/modeling_value_head.pyrzValueHead.__init__s v56#)::.Dc#J #)#>#> ;Orzz"67UWU`U`Ub  6= ) ,,K 60 1 44K V1 2((WVY-G6>>=9"(.."<"v>>r"c|js|jj|i|}ni}|jj|i|}|j D] \}}||d|<|Sz Returns the state dictionary of the model. We add the state dictionary of the value head to the state dictionary of the wrapped model by prepending the key with `v_head.`. v_head.rUr9 state_dictr:itemsrrirpretrained_model_state_dictv_head_state_dictkvs r!rnz,AutoModelForCausalLMWithValueHead.state_dict !!*J$*?*?*J*JD*[TZ*[ '+- '2DKK22DCFC%++- ;DAq9: ''! 6 ;**r"cp|j|j_|jj|i|Sr$r:r9 push_to_hubrhs r!rxz-AutoModelForCausalLMWithValueHead.push_to_hub2'+{{$0t$$00$A&AAr"ct|jD]+}d|vs|j|||jdd<-|jj |d~t |jdrd|jjjvs&d|jjjvr tdtt|jjjd ttr%trd ntrd nd |jj!|_fd }|j#|d|_yy) We add the state dictionary of the value head to the state dictionary of the wrapped model by prepending the key with `v_head.`. This function removes the `v_head.` prefix from the keys of the value head state dictionary. rlFstrict hf_device_mapcpudiskdThe model is offloaded on CPU or disk - CPU & disk offloading is not supported for ValueHead models.rznpu:zxpu:zcuda:cd}|D]8}t|tjr||jfz }3||fz }:|S)Nr7 isinstancer\Tensorr')moduleinputoutputs new_outputr) first_devices r!set_device_hookzDAutoModelForCausalLMWithValueHead.post_init..set_device_hooksO %0F!&%,,7"vyy'>&@@ "vi/ 0 "!r"TN)listkeysrreplacer:load_state_dictrr9rvalues ValueErrorsetrintrrr'register_forward_hookis_sequential_parallel)rrnrsrrs @r! post_initz+AutoModelForCausalLMWithValueHead.post_initsY joo'( IAA~7A~~a7H 199Y34 I ##Ju#=  4((/ :..<<CCEET22@@GGII z D$9$9$G$G$N$N$P QRSTUL,,)+%),#8L+-%),#8L%*<.#9L++..6DK "  & & 7*.D '= ;r"NNNF)r+r,r-r.rtransformers_parent_classsupported_argsrr;r*rgrnrxrr/r0s@r!r2r2>sJ,!5N , 20$ 5,n ?+ B */r"r2cfeZdZdZeZgdZdZfdZdZ dZ dZ dZ d Z d d Zd ZxZS) "AutoModelForSeq2SeqLMWithValueHeada A seq2seq model with a value head in addition to the language model head. This class inherits from `~trl.PreTrainedModelWrapper` and wraps a `transformers.PreTrainedModel` class. The wrapper class supports classic functions such as `from_pretrained` and `push_to_hub` and also provides some additional functionalities such as `generate`. Args: pretrained_model (`transformers.PreTrainedModel`): The model to wrap. It should be a causal language model such as GPT2. or any model mapped inside the `AutoModelForSeq2SeqLM` class. kwargs: Additional keyword arguments passed along to the `ValueHead` class. )lm_head embed_outoutput_projectionr3c t||fi||j|\}}}d|_|j s t dt |jjfi||_ |jdi|y)NTzOThe model does not have a language model head, please use a model that has one.r7) rrr8r _has_lm_headrr r9rr:r;r<s r!rz+AutoModelForSeq2SeqLMWithValueHead.__init__ s} )4V4"008 q!"&  "no o 5 5 < <N N +]+r"c|jjD]%\}tfd|jDs%yy)Nc3&K|]}|v ywr$r7.0 attributenames r! zBAutoModelForSeq2SeqLMWithValueHead._has_lm_head../sK9$KTF)r9 named_modulesanylm_head_namings)r_modulers @r!rz/AutoModelForSeq2SeqLMWithValueHead._has_lm_head,sA!22@@B MD'Kd6J6JKK r"ct|jD]+}d|vs|j|||jdd<-|jj |d~t |jdrd|jjjvs&d|jjjvr td|jjD];\}tfd |jDs%|jjn|jj!|_fd }|j#|d |_y y ) r{rlr|Fr}rrrrc3&K|]}|v ywr$r7rs r!rz?AutoModelForSeq2SeqLMWithValueHead.post_init..JsOYyD(Orcd}|D]8}t|tjr||jfz }3||fz }:|S)a A hook that sets the device of the output of the model to the device of the first parameter of the model. Args: module (`nn.Module`): The module to which the hook is attached. input (`tuple`): The input to the module. outputs (`tuple`): The output of the module. r7r)rrrrr)lm_head_devices r!rzEAutoModelForSeq2SeqLMWithValueHead.post_init..set_device_hookQsQ %0F!&%,,7"vyy'@&BB "vi/ 0 "!r"TN)rrrrr:rrr9rrrrrrr&rZr'rr)rrnrsrrrrs @@r!rz,AutoModelForSeq2SeqLMWithValueHead.post_init3s@ joo'( IAA~7A~~a7H 199Y34 I ##Ju#=  4((/ :..<<CCEET22@@GGII z !% 5 5 C C E  fO$:N:NOO%+]]%9%9N  ++..8DK "*  & & 7*.D 'Q ;r"c|js|jj|i|}ni}|jj|i|}|j D] \}}||d|<|Srkrmrps r!rnz-AutoModelForSeq2SeqLMWithValueHead.state_dictirur"cp|j|j_|jj|i|Sr$rwrhs r!rxz.AutoModelForSeq2SeqLMWithValueHead.push_to_hubyryr"c D|jdd}|jdd}|y|dk(rt|jjjjj d||jjj jjyy)z> We initialize the weights of the value head. r4r@r5NrArBrCrFrKs r!r;z0AutoModelForSeq2SeqLMWithValueHead._init_weights~s#JJ'A3G #94@   h & KK   & & + + 3 3BS 3 T KK   $ $ ) ) / / 1'r"c ||d<|jr4|jjjdk(r|j d|jd||dd|}|j d}|j }|j} |j|jd} |jtjk7r|j}|r|| | |jfS|| | fS)NrPrQT)rRrSrOrTr7)rUr9rVrWrdecoder_hidden_statesrXrYr:r[r%r\r]r^rPr_s r!r*z*AutoModelForSeq2SeqLMWithValueHead.forwards%4 !   $"7"7"J"J"T"TXg"g JJ( )1D11 )!%   .CCBG%,,  %% -.66r: ??emm +!)I !tU,=,M,MN NtU+ +r"c:|jj|i|S)z: We call `generate` on the wrapped model. rfrhs r!rgz+AutoModelForSeq2SeqLMWithValueHead.generates".t$$--t>v>>r"r)r+r,r-r.rrrrrrrrnrxr;r*rgr/r0s@r!rr sU !6CON ,4/l+ B 2 $ ,D?r"r)r\torch.nnr transformersrrrr modeling_baserModuler r2rr7r"r!rsF tt1% %PH/(>H/Vi?)?i?r"