bi2h ddlZddlZddlZddlmZddlmZddlmZm Z ddl m Z ddl m Z mZddlmZddlmZddlZdd lmZdd lmZmZmZmZmZerdd lmZerdd lmZerddl Z er3dd l!m"Z"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-erddl.m/Z%ej`e1Z2dejfd<GddZ4eGddZ5de5de6de6deddf dZ7d e8d!e6de8e8fd"Z9de5fd#Z:d'd$ejvfd%Znamedtypeshapec|j tdtt|j dd}tj |||j }|jj||j|jjj|jjj||fgy)a Receives updated weights from the client process and updates the named parameter in the model. Args: name (`str`): Name of the weight tensor being updated. dtype (`str`): Data type of the weight tensor as a string (e.g., `"torch.float32"`). shape (`Sequence[int]`): Shape of the weight tensor. Nz=Communicator not initialized. Call `init_communicator` first..)r2r&)src)weights)r(r)getattrtorchsplitemptyr& broadcastr+groupbarrier model_runnermodel load_weights)r,r1r2r3weights r.update_named_paramz,WeightSyncWorkerExtension.update_named_paramqs    #^_ _u{{3/34U% D ""6t/?/?"@ &&( ,,tVn5E,Fr0c>|j|`d|_d|_yy)z Closes the communicator when weight synchronization is no longer needed. This method deletes the NCCL communicator to release associated resources. N)r(r+)r,s r.close_communicatorz,WeightSyncWorkerExtension.close_communicators+    ' #D #D  (r0)r#N) __name__ __module__ __qualname____doc__r(r+strintr/rrDrFr0r.rrFsdKK*c**#*$*<GsG3Gx}GQUG4 $r0rceZdZUdZeddiZeed<edddiZe eed <ed dd iZ e ed <ed dd iZ e ed<edddiZ eed<edddiZe ed<edddiZeed<edddiZeed<edddiZe e ed<edddiZe eed<eddd iZe eed!<eddd"iZeed#<eddd$iZeed%<ed&dd'iZeed(<y))ScriptArgumentsa Arguments for the script. Args: model (`str`): Model name or path to load the model from. revision (`str` or `None`, *optional*, defaults to `None`): Revision to use for the model. If not specified, the default branch will be used. tensor_parallel_size (`int`, *optional*, defaults to `1`): Number of tensor parallel workers to use. data_parallel_size (`int`, *optional*, defaults to `1`): Number of data parallel workers to use. host (`str`, *optional*, defaults to `"0.0.0.0"`): Host address to run the server on. port (`int`, *optional*, defaults to `8000`): Port to run the server on. gpu_memory_utilization (`float`, *optional*, defaults to `0.9`): Ratio (between 0 and 1) of GPU memory to reserve for the model weights, activations, and KV cache on the device dedicated to generation powered by vLLM. Higher values will increase the KV cache size and thus improve the model's throughput. However, if the value is too high, it may cause out-of-memory (OOM) errors during initialization. dtype (`str`, *optional*, defaults to `"auto"`): Data type to use for vLLM generation. If set to `"auto"`, the data type will be automatically determined based on the model configuration. Find the supported values in the vLLM documentation. max_model_len (`int` or `None`, *optional*, defaults to `None`): If set, the `max_model_len` to use for vLLM. This can be useful when running with reduced `vllm_gpu_memory_utilization`, leading to a reduced KV cache size. If not set, vLLM will use the model context size, which might be much larger than the KV cache, leading to inefficiencies. enable_prefix_caching (`bool` or `None`, *optional*, defaults to `None`): Whether to enable prefix caching in vLLM. If set to `True`, ensure that the model and the hardware support this feature. enforce_eager (`bool`, *optional*, defaults to `False`): Whether to enforce eager execution. If set to `True`, we will disable CUDA graph and always execute the model in eager mode. If `False` (default behavior), we will use CUDA graph and eager execution in hybrid. kv_cache_dtype (`str`, *optional*, defaults to `"auto"`): Data type to use for KV cache. If set to `"auto"`, the dtype will default to the model data type. trust_remote_code (`bool`, *optional*, defaults to `False`): Whether to trust remote code when loading models. Set to `True` to allow executing code from model repositories. This is required for some custom models but introduces security risks. log_level (`str`, *optional*, defaults to `"info"`): Log level for uvicorn. Possible choices: `"critical"`, `"error"`, `"warning"`, `"info"`, `"debug"`, `"trace"`. helpz*Model name or path to load the model from.)metadatarANzQRevision to use for the model. If not specified, the default branch will be used.)defaultrQrevisionr'z)Number of tensor parallel workers to use.tensor_parallel_sizez'Number of data parallel workers to use.data_parallel_sizez0.0.0.0z"Host address to run the server on.r i@zPort to run the server on.r!g?aQRatio (between 0 and 1) of GPU memory to reserve for the model weights, activations, and KV cache on the device dedicated to generation powered by vLLM. Higher values will increase the KV cache size and thus improve the model's throughput. However, if the value is too high, it may cause out-of-memory (OOM) errors during initialization.gpu_memory_utilizationautozData type to use for vLLM generation. If set to 'auto', the data type will be automatically determined based on the model configuration. Find the supported values in the vLLM documentation.r2aIf set, the `max_model_len` to use for vLLM. This can be useful when running with reduced `vllm_gpu_memory_utilization`, leading to a reduced KV cache size. If not set, vLLM will use the model context size, which might be much larger than the KV cache, leading to inefficiencies. max_model_lenzxWhether to enable prefix caching in vLLM. If set to `True`, ensure that the model and the hardware support this feature.enable_prefix_cachingFzWhether to enforce eager execution. If set to `True`, we will disable CUDA graph and always execute the model in eager mode. If `False` (default behavior), we will use CUDA graph and eager execution in hybrid. enforce_eagerz_Data type to use for KV cache. If set to 'auto', the dtype will default to the model data type.kv_cache_dtypezWhether to trust remote code when loading models. Set to True to allow executing code from model repositories. This is required for some custom models but introduces security risks.trust_remote_codeinfozbLog level for uvicorn. Possible choices: 'critical', 'error', 'warning', 'info', 'debug', 'trace'. log_level)rGrHrIrJrrArK__annotations__rSr rTrLrUr r!rVfloatr2rXrYboolrZr[r\r^rMr0r.rOrOs*XFGE3$mnHhsm!&EF!#$CD>?D#67D#%* @ %E p E3$) e $M8C=-2 - -8D>%* # %M8D>  u NC $ c t  Isr0rO script_argsdata_parallel_rank master_port connectionr#c Ft|tjd<t|tjd<t|jtjd<t|tjd<t |j |j |j|j|j|j|j|j|jd|j }|jddi |j!}|d d vrW|d }|j'dd|j'di}}t)||} | |i|} |d dk(r|j| n |d dk(ryy#t"$r|j%d YywxYw)N VLLM_DP_RANKVLLM_DP_RANK_LOCAL VLLM_DP_SIZEVLLM_DP_MASTER_PORTz0trl.scripts.vllm_serve.WeightSyncWorkerExtension) rArSrTrVrZr2rYr[rXworker_extension_clsr\statusreadyrF)methodtype)callfire_and_forgetrnargsrMkwargsrpshutdown)rKosenvironrUrrArSrTrVrZr2rYr[rXr\sendrecvKeyboardInterruptcollective_rpcgetr9) rbrcrdrellmcommand method_namerrrsrnresults r. llm_workerrs"%%7!8BJJ~'*+='>BJJ#$!$[%C%C!DBJJ~(+K(8BJJ$% %%(==*AA!//*??"11!//O%77 C$OOXw'(   oo'G 6?9 9!(+K";;vr2GKK"4M&DS+.FT,V,Fv&(' V_ * # !    &:  ;  s FF F lstnc tt||\}}t|Dcgc]-}|||zt||z|dz|zt|dz|z/c}Scc}w)aI Split list `lst` into `n` evenly distributed sublists. Example: ```python >>> chunk_list([1, 2, 3, 4, 5, 6], 2) [[1, 2, 3], [4, 5, 6]] >>> chunk_list([1, 2, 3, 4, 5, 6], 4) [[1, 2], [3, 4], [5], [6]] >>> chunk_list([1, 2, 3, 4, 5, 6], 8) [[1], [2], [3], [4], [5], [6], [], []] ``` r')divmodlenrangemin)rrkris r. chunk_listrKs^ #c(A DAqJOPQ( SQCAAq !QUaK#a!eQ-$? @ SS Ss2AcJts tdts tdts tdt s tdt }ggt jD]V}t\}}tt|||f}|jj|j|Xtdtffd }t|}|jd d }|jd fd } Gd dt } Gddt } |j#d| d| ffd } Gddt } |j#dd| ffd }Gddt }|j#dd|ffd }|j#dfd}|j#dfd }t%j&|j(j*j,!y)"Nz`FastAPI is required to run the vLLM serve script. Please install it using `pip install fastapi`.zbPydantic is required to run the vLLM serve script. Please install it using `pip install pydantic`.z`Uvicorn is required to run the vLLM serve script. Please install it using `pip install uvicorn`.zZvLLM is required to run the vLLM serve script. Please install it using `pip install vllm`.)targetrrappcKt}t|jkrgD]I}|j}t |t s$|j ddk(s9|j|Kt|jkrgdD]^}|jd|js&tjd|d|j|j`yw)Nrlrm )timeoutzProcess z< is still alive after 10 seconds, attempting to terminate...) setrrUrx isinstancedictr{addjoinis_aliveloggerwarning terminate)rready_connectionsremsgprocess connections processesrbs r.lifespanzmain..lifespan}s E#${'E'EE) 6  oo'c4(SWWX->'-I%))*5 6#${'E'EE ! G LLL $!'2nop!!#   sAC6 C6!+C6 ,C6:.healths $sz/get_world_size/cBKdjjziSw)a" Retrieves the world size of the LLM engine, which is `tensor_parallel_size * data_parallel_size`. Returns: `dict`: A dictionary containing the world size. Example response: ```json {"world_size": 8} ``` r")rTrU)rbsr.get_world_sizezmain..get_world_sizes%k>>A_A__``sceZdZUeeed<dZeed<dZe ed<dZ e ed<dZ e ed<dZ eed <d Z e ed <d Zeed <dZeeed<eeZeed<y)main..GenerateRequestpromptsr'rg?repetition_penalty temperaturetop_pr6top_kgmin_p max_tokensNguided_decoding_regex)default_factorygeneration_kwargs)rGrHrIlistrKr_rrLrr`rrrrrrr rrrrMr0r.GenerateRequestrspc3 $'E' U usu C/3x}3"'"=4=r0rc$eZdZUeeeed<y)main..GenerateResponsecompletion_idsN)rGrHrIrrLr_rMr0r.GenerateResponsersT#Y'r0rz /generate/)response_modelrequestc 8K|jtd|j}nd}|j|j|j|j |j |j|j|d}|j|jtd i|}t|j j}t |D]$\}}|sdg}||d}|j!dd|d & Dcgc]}|j#}}t||D cgc] \} }|s |  }} }t%t'j(|}|D cgc](} | j*D]} t%| j,*} } } d | iScc}wcc}} wcc} } ww) a Generates completions for the provided prompts. Args: request (`GenerateRequest`): - `prompts` (list of `str`): A list of prompts (text strings) for the model to generate completions. - `n` (`int`, *optional*, defaults to `1`): Number of completions to generate for each prompt. - `repetition_penalty` (`float`, *optional*, defaults to `1.0`): Repetition penalty to apply during generation. - `temperature` (`float`, *optional*, defaults to `1.0`): Temperature for sampling. Higher values lead to more random outputs. - `top_p` (`float`, *optional*, defaults to `1.0`): Top-p (nucleus) sampling parameter. It controls the diversity of the generated text. - `top_k` (`int`, *optional*, defaults to `-1`): Top-k sampling parameter. If set to `-1`, it disables top-k sampling. - `min_p` (`float`, *optional*, defaults to `0.0`): Minimum probability threshold for sampling. - `max_tokens` (`int`, *optional*, defaults to `16`): Maximum number of tokens to generate for each completion. - `guided_decoding_regex` (`str`, *optional*): A regex pattern for guided decoding. If provided, the model will only generate tokens that match this regex pattern. - `generation_kwargs` (`dict`, *optional*): Additional generation parameters to pass to the vLLM `SamplingParams`. This can include parameters like `seed`, `frequency_penalty`, etc. If it contains keys that conflict with the other parameters, they will override them. Returns: `GenerateResponse`: - `completion_ids` (list of list of `int`): A list of lists of token IDs for each generated completion. Example request: ```json {"prompts": ["Hello world", "What is AI?"]} ``` Example response: ```json {"completion_ids": [[101, 102, 103], [201, 202, 203]]} ``` Noutlines)backendregex)rrrrrrrguided_decodingz )rsampling_paramsrpgeneraterornrsrrM)rrrrrrrrrupdaterrrrrUziprwrxrr from_iterableoutputs token_ids)rrrrchunked_promptsrerrs all_outputsoutputrrrrbs r.rzmain..generatesD  ( ( 42:WMjMjkO"O")"<"<"..]]]]]]!,,.     !:!:;(=+<=%W__k6T6TU$'{O#D V J*+!(_MF OOVzVT U V.InitCommunicatorRequestr r!r"N)rGrHrIrKr_rLrMr0r.InitCommunicatorRequestrs  r0rz/init_communicator/cKjjzdz}d|j|j|fd}D]}|j dd|dddiSw) a Initializes the communicator for synchronizing model weights between a client and multiple server workers. Args: request (`InitCommunicatorRequest`): - `host` (`str`): Hostname or IP address of the master node. - `port` (`int`): Port number to be used for communication. - `world_size` (`int`): Total number of participating processes in the group. r'r/rnrrrqrzrmessagez+Request received, initializing communicator)rTrUr r!rw)rr"rsrerrbs r.r/zmain..init_communicator sv!55 8V8VVYZZ 0',, V`9ab% gJ OO%6BR^de f gHIIsAAc2eZdZUeed<eed<eeed<y)"main..UpdateWeightsRequestr1r2r3N)rGrHrIrKr_rrLrMr0r.UpdateWeightsRequestr!s  Cyr0rz/update_named_param/cKd|j|jt|jfd}D]}|j dd|dddiSw)a Updates the model weights with the provided tensor. Once this endpoint is called, the client process should broadcast the updated weights to all server workers. Args: request (`UpdateWeightsRequest`): - `name` (`str`): Name of the weight tensor being updated. - `dtype` (`str`): Data type of the weight tensor (e.g., `"torch.float32"`). - `shape` (list of `int`): Shape of the weight rDrrqrzrrz*Request received, updating named parameter)r1r2tupler3rw)rrsrers r.rDz main..update_named_param&sd"17<<X]^e^k^kXl:mn% gJ OO%6BR^de f gGHHsAAz/reset_prefix_cache/cKD]}|jdddDcgc]}|j}}td|D}ddt|ziScc}ww)z8 Resets the prefix cache for the model. rpreset_prefix_cache)rornc3 K|]}|ywNrM).0rs r. z3main..reset_prefix_cache..Fs7f7s rz1Request received, resetting prefix cache status: )rwrxallrK)rersuccessrs r.rz main..reset_prefix_cache=sw & NJ OOV7KL M N.close_communicatorIsG 01% gJ OO%6BR^de f gCDDs%()r r!r^)r ImportErrorrrrrrrUrr rstartappendrrr{rpostuvicornrunr r!r^)rbrdrcparent_connectionchild_connectionrrrrrrrrrr/rrDrrFrrs` @@r.mainr_sP  ! n   ! " p   ! n    vww /KKI#K$B$BC".2f++;@RT_aq2rs ,-! "G& 8 $C WWZ   WW   a! a >) >(9( XXl+;X<I2I2=I2V)  XX#$J)@J%J*y  XX$%I*>I&I, XX$% _& _ XX$%E&E KK+**1A1A[MbMbcr0 subparsersc\||jddt}|Stt}|S)Nz vllm-servezRun the vLLM serve script)rPdataclass_types) add_parserrOr )rparsers r. make_parserrWs9&&|:Ugv&w M?+ Mr0__main__r)@argparseloggingrucollections.abcr contextlibr dataclassesrr itertoolsrmultiprocessingrr multiprocessing.connectionr typingr r:trlr trl.import_utilsr rrrrfastapirpydanticrrvllmrr,vllm.distributed.device_communicators.pyncclrvllm.distributed.parallel_statervllm.distributed.utilsrvllm.sampling_paramsr vllm.utilsr3vllm_ascend.distributed.device_communicators.pyhcclr getLoggerrGrrvrrOrLrrrr_SubParsersActionrrparse_args_and_configrbrMr0r.rsi $*()1 "(O?<9(!p   8 $ .5 )*O$O$d ~~ ~B. .69.HK.Yc. .bTDTSTT$ZT(udoudpH66 z ]F113N[r0