bi6ddlZddlZddlZddlZddlmZddlmZddlZddlm Z ddl m Z m Z m Z e r ddlZddlmZe rddlmZdd lmZe rdd lmZej,eZGd d Zed k(r{ddlmZeZej;ej=ddgddeZe deddl!m"Z"e"jFdjIdZ%ejMe%yy)N)Optional)urlparse)nn)is_requests_availableis_vllm_ascend_availableis_vllm_available)ConnectionError)PyNcclCommunicator)StatelessProcessGroup)PyHcclCommunicatorc eZdZdZ d deededededef dZd!d ed efd Z d"d e ed edededededededeedee de e efdZ dZ dedejfdZdej$fdZdZdZy)# VLLMClientau A client class to interact with a vLLM server. This class provides methods to generate completions, initialize and manage weight update groups, and update model weights in a distributed setting. Before using it, start the vLLM server with `trl vllm-serve`. Args: base_url (`str` or `None`, *optional*, defaults to `None`): Base URL for the vLLM server (e.g., `"http://localhost:8000"`). If provided, `host` and `server_port` are ignored. host (`str`, *optional*, defaults to `"0.0.0.0"`): IP address of the vLLM server. Ignored if `base_url` is provided. server_port (`int`, *optional*, defaults to `8000`): Port number of the vLLM server. Ignored if `base_url` is provided. group_port (`int`, *optional*, defaults to `51216`): Port number for the weight update group. connection_timeout (`float`, *optional*, defaults to `0.0`): Total timeout duration in seconds to wait for the server to be up. If the server is not up after the timeout, a `ConnectionError` is raised. Examples: Run the vLLM server with the model `Qwen/Qwen2.5-7B`: ``` $ trl vllm-serve --model Qwen/Qwen2.5-7B ... INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) ``` Use the client to generate completions and update model weights: ```python >>> from trl.extras.vllm_client import VLLMClient >>> client = VLLMClient() >>> client.generate(["Hello, AI!", "Tell me a joke"]) [[2980, 498, 1492, 752, 448, 264, 13027, 8645, 30, 358, 2776, 4460, 311, 3270, 264, 2025], [911, 7988, 1251, 382, 3838, 653, 498, 1618, 4325, 879, 2581, 20027, 264, 21428, 30, 362]] >>> from transformers import AutoModelForCausalLM >>> model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B", device_map="cuda") >>> client.init_communicator() >>> client.update_model_params(model) ``` There are several ways to initialize the client: ```python VLLMClient(base_url="http://localhost:8000") VLLMClient(base_url="http://192.168.1.100:8000") VLLMClient(host="localhost", server_port=8000) VLLMClient(host="192.168.1.100", server_port=8000) ``` Nbase_urlhost server_port group_portconnection_timeoutcts tdts tdtj|_|bt |}tj|j|_ |jxsd}|d|j|j|_n/||_ ||_d|jd|j|_||_|j#|y)NzIrequests is not installed. Please install it with `pip install requests`.zAvLLM is not installed. Please install it with `pip install vllm`.httpz://zhttp://:)r ImportErrorr requestsSessionsessionrsocket gethostbynamehostnamerschemenetlocpathrrr check_server)selfrrrrr parsed_urlrs Q/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/trl/extras/vllm_client.py__init__zVLLMClient.__init__fs%&ij j "ab b'')  !(+J,,Z-@-@ADI&&0&F%hc**;*;)t}}oWUbTcdRR1 sB11D9D  Dpromptsnrepetition_penalty temperaturetop_ptop_kmin_p max_tokensguided_decoding_regexgeneration_kwargsreturnc |jd} |jj| ||||||||| | xsid } | jdk(r| j dSt d| jd| j )a Generates model completions for the provided prompts. Args: prompts (`list[str]`): List of text prompts for which the model will generate completions. n (`int`, *optional*, defaults to `1`): Number of completions to generate for each prompt. repetition_penalty (`float`, *optional*, defaults to `1.0`): Parameter for repetition penalty. 1.0 means no penalty. temperature (`float`, *optional*, defaults to `1.0`): Temperature parameter for sampling. Higher values increase diversity. top_p (`float`, *optional*, defaults to `1.0`): Top-p sampling parameter.`1.0` means no truncation. top_k (`int`, *optional*, defaults to `-1`): Top-k sampling parameter. `-1` means no truncation. min_p (`float`, *optional*, defaults to `0.0`): Minimum probability for sampling. max_tokens (`int`, *optional*, defaults to `16`): Maximum number of tokens to generate for each prompt. guided_decoding_regex (`str` or `None`, *optional*, defaults to `None`): Regular expression to guide the decoding process. generation_kwargs (`dict` or `None`, *optional*, defaults to `None`): Additional generation parameters to pass to the vLLM `SamplingParams`. This can include parameters like `seed`, `frequency_penalty`, etc. If it contains keys that conflict with the other parameters, they will override them. Returns: `list[list[int]]`: List of lists of token IDs representing the model-generated completions for each prompt. z /generate/) r:r;r<r=r>r?r@rArBrCjsonr+completion_idsRequest failed: , )rrpostr.rG Exceptiontext) r#r:r;r<r=r>r?r@rArBrCr5r7s r%generatezVLLMClient.generatesXz*<<$$ "&8*()>%6%<" %    3 &==?#34 4.x/C/C.DBx}}oVW Wr'c|jd}tj|}|jdk(r|j d}n%t d|jd|j |dz}||_|jd}|jj|d|j|d  }|jdk7r%t d|jd|j tjd tj|j|j|j| }t!|d |_t%j&|j(y)zg Initializes the weight update group in a distributed setup for model synchronization. z/get_world_size/r+ world_sizerIrJz/init_communicator/0.0.0.0)rportrPrFg?)rrSrankrPr)deviceN)rrr-r.rGrLrMrTrrKrr,r4r createrr pynccl_commatexitregisterclose_communicator)r#r5r7vllm_world_sizerPpgs r%init_communicatorzVLLMClient.init_communicators8 /0<<$   3 &&mmol;O.x/C/C.DBx}}oVW W$q( # 23<<$$S 4??jt/u$v   3 &.x/C/C.DBx}}oVW W 3# ) )tyytUYU^U^ku v-b; //0r'nameweightsct|jt|j}}|jd}|j j ||||d}|jdk7r%td|jd|j|jj||j|jjjy) a0 Updates a specific named parameter in the model and broadcasts it to other processes. Args: name (`str`): Name of the layer whose weights are being updated. weights (`torch.Tensor`): Tensor containing the updated weights. z/update_named_param/)r^dtypeshaperFr+rIrJ)srcN)strratuplerbrrrKr.rLrMrW broadcastrTgroupbarrier)r#r^r_rarbr5r7s r%update_named_paramzVLLMClient.update_named_param s7==)5+?u34<<$$SuW\/]$^   3 &.x/C/C.DBx}}oVW W ""7 ": &&(r'modelcl|jD]!\}}|j||j#y)z Updates all parameters of the given model by calling `update_named_param` for each parameter in the model. Args: model (`nn.Module`): Model whose parameters (weights/biases) are to be updated. N)named_parametersridata)r#rjr^params r%update_model_paramszVLLMClient.update_model_paramss4!113 6KD%  # #D%** 5 6r'c|jd}|jj|}|jdk7r%t d|jd|j y)z8 Resets the prefix cache for the model. z/reset_prefix_cache/r+rIrJN)rrrKr.rLrMr#r5r7s r%reset_prefix_cachezVLLMClient.reset_prefix_cache)s`34<<$$S)   3 &.x/C/C.DBx}}oVW W 'r'c|jd} |jj|}|jdk7r%t d|jd|j y#t $rYywxYw)zW Closes the weight update group and cleans up the communication group. z/close_communicator/r+rIrJN)rrrKr.rLrMr rqs r%rZzVLLMClient.close_communicator2s|34 \||((-H ##s*"283G3G2H8==/ Z[[+    sA!! A-,A-)NrRi@i)rtg@) rQ?rururtNN)__name__ __module__ __qualname____doc__rrdintfloatr&r"listdictrNr]torchTensorrirModulerorrrZr'r%rr,sN7v#'$' .3-.. .  . " .8"'%"'u"'N$' /3,0?Xc?X ?X" ?X  ?X  ?X?X?X?X (}?X$D>?X d3i?XB 1D)s)U\\)( 6 6X \r'r__main__)SamplingParamsz Hello, AI!zTell me a joke )r;rAsampling_paramsz Responses:)AutoModelForCausalLMzQwen/Qwen2.5-7Bcuda)'rXloggingrr,typingr urllib.parserrr import_utilsrrr rr ,vllm.distributed.device_communicators.pyncclr vllm.distributed.utilsr 3vllm_ascend.distributed.device_communicators.pyhcclr getLoggerrxr0rvllmrclientr]rN responsesprint transformersrfrom_pretrainedtorjrorr'r%rs ! ]](O<!p   8 $S\S\n z# \F /? @ARTftfvwI , "2 0 0 01B C F Fv NE u%r'