birL P d dl Zd dlZd dlmZmZ d dlmZmZ d dl Z
d dl m Z d dl
mZ d dlmZ ddlmZ e rd dlZ e rd d lmZ d
Z G d d e Z G d
de Z G d de Z G d de Z G d de Z G d de Z G d de Z G d de Z y) N)ABCabstractmethod)OptionalUnion) Accelerator)InferenceClient)is_openai_available )is_llm_blender_available)OpenAIa I require a leaderboard for various large language models. I'll provide you with prompts given to these models and their corresponding outputs. Your task is to assess these responses, and select the model that produces the best output from a human perspective.
## Instruction
{{
"instruction": """{prompt}""",
}}
## Model Outputs
Here are the unordered outputs from the models. Each output is associated with a specific model, identified by a unique model identifier.
{{
{{
"model_identifier": "0",
"output": """{response0}"""
}},
{{
"model_identifier": "1",
"output": """{response1}"""
}}
}}
## Task
Evaluate the models on the basis of the quality and relevance of their results, and select the model that generated the best result. Reply with the identifier of the best model. Our evaluation will only take into account the first character of your answer, so make sure it contains only one of the identifiers and nothing else (no quotation marks, no spaces, no new lines, ...).
c
@ e Zd ZdZeddee dee dedefd Zy) BaseJudgezb
Base class for judges. The subclasses of this class should implement the `judge` method.
prompts completions
shuffle_orderreturnc t d )N3Judge subclasses must implement the `judge` method.NotImplementedErrorselfr r r s M/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/trl/trainer/judges.pyjudgezBaseJudge.judgeE s !"WXX NT) __name__
__module__ __qualname____doc__r liststrboolr r r r r @ sF YT#Y YT#Y Yt Y_c Y Yr r c R e Zd ZdZeddee deee dedeee fd Z y)
BaseRankJudgea
Base class for LLM ranking judges.
**Example**:
```python
class MyRankJudge(BaseRankJudge):
def judge(self, prompts, completions, shuffle_order=True):
return ... # Your ranking logic here
judge = MyRankJudge()
judge.judge(
prompts=["The capital of France is", "The capital of Germany is"],
completions=[[" Paris", " Marseille", "Lyon"], [" Munich", " Berlin"]],
) # [[0, 1, 2], [1, 0]]
```
r r r r c t d )a8
Judge the completion for the given prompts and return the ranks of each completion.
Args:
prompts (`list[str]`):
List of prompts.
completions (`list[list[str]]`):
List of completions list, where each element is a list of completions for the corresponding prompt.
shuffle_order (`bool`, *optional*, defaults to `True`):
Whether to shuffle the order of the completions to avoid positional bias.
Returns:
`list[list[int]]`:
List of lists of idxs, where each list contains the ranks of the completions for the corresponding
prompt. E.g., `[1, 2, 0]` means that the second completion (`idx=1`) is the best, followed by the
third, and then the first.
r r r s r r zBaseRankJudge.judge] s & ""WXXr Nr
r r r r r r! r" r# intr r$ r r r& r& J sW $ YT#Y YT$s)_ YUY Yeijnorjset Y Yr r& c L e Zd ZdZeddee deee dedee fd Z y) BasePairwiseJudgez)
Base class for pairwise judges.
r r r r c t d )a]
Judge the completion pairs for the given prompts.
Args:
prompts (`list[str]`):
List of prompts.
completions (`list[list[str]]`):
List of completions pairs, where each element is a pair of completions for the corresponding prompt.
shuffle_order (`bool`, *optional*, defaults to `True`):
Whether to shuffle the order of the completions to avoid positional bias.
Returns:
`list[int]`:
List of idxs, where each idx is the rank of the best completion for the corresponding prompt. E.g., `1`
means that the second completion (`idx=1`) is the best.
Note:
If the judge returns `-1` for any prompt, it indicates that the inner process used to compute the
preference has failed. For instance, this could occur if the underlying language model returned an invalid
answer. In such cases, the caller should handle these invalid indices appropriately, possibly by
implementing fallback logic or error handling.
r r r s r r zBasePairwiseJudge.judgex s 0 ""WXXr Nr r( r$ r r r+ r+ s sQ YT#Y YT$s)_ YUY Yeijmen Y Yr r+ c
Z e Zd ZdZe d dee dee deee dedee f
d Z
y)
BaseBinaryJudgez'
Base class for binary judges.
Nr r gold_completionsr r c t d )ar
Judge the completion for a given prompt. Used to assess if a completion satisfies a constraint.
This base class should be used to implement binary evaluations as done in section 4.1.4 of the [CGPO
paper](https://huggingface.co/papers/2409.20370). It is relevant for assessing whether a prompt completion pair
satisfies a specific contraint.
Args:
prompts (`list[str]`): List of prompts.
completions (`list[str]`): List of completions.
gold_completions (`list[str]`, `optional`): List of gold completions if it exists.
shuffle_order (`bool`): Whether to shuffle the order of the completions to avoid positional bias.
Returns:
list[int]: A list of binary labels:
- 1 indicates that the completion satisfies the evaluated constraint.
- 0 indicates that the completion does not satisfy the evaluated constraint.
Note:
If the judge returns -1 for any prompt, it indicates that the inner process used to compute the preference
has failed. For instance, this could occur if the underlying language model or rule based contraint
returned an invalid answer. In such cases, the caller should handle these invalid indices appropriately,
possibly by implementing fallback logic or error handling.
r r )r r r r/ r s r r zBaseBinaryJudge.judge s @ ""WXXr NT) r r r r r r! r" r r# r) r r$ r r r. r. so
15 " Yc Y #Y Y #49- Y
Y
c
Y Yr r. c ` e Zd ZdZd Z d dee deee dedededee e
ef f d Z y
) PairRMJudgea
LLM judge based on the PairRM model from AllenAI.
This judge uses the PairRM model to rank pairs of completions for given prompts. It's designed for pairwise
comparison of language model outputs. The PairRM model is loaded using the llm-blender library and runs on the
default Accelerator device.
**Attributes**:
blender (`llm_blender.Blender`):
An instance of the Blender class from llm-blender.
**Example**:
```python
>>> pairrm_judge = PairRMJudge()
>>> prompts = ["Translate 'hello' to French", "What's the capital of Japan?"]
>>> completions = [["Bonjour", "Salut"], ["Kyoto", "Tokyo"]]
>>> results = pairrm_judge.judge(prompts, completions)
>>> print(results) # [0, 1] (indicating the first completion is preferred for the first prompt and the second)
```
This class requires the llm-blender library to be installed. Install it with: `pip install llm-blender`.
c t s t d t j | _ | j j dt
j y )NzOllm-blender is not installed. Please install it with `pip install llm-blender`.zllm-blender/PairRM)device)r
ValueError llm_blenderBlenderblender
loadrankerr r5 )r s r __init__zPairRMJudge.__init__ sD ') no o"**, 4[]=Q=Q Rr r r r
return_scores temperaturer c h t |d dk7 r t d |rTt j j ddgt | }t || D cg c] \ }}|r|ddd n| }}}| j j |||d } |s| d
z } n| |z } |r| dddddf | |< |rbt j | dd }
t j | |
z
} | t j | dd z } | dddf j S | dddf j S c c}}w ) a'
Judge the completion pairs for the given prompts using the PairRM model.
Args:
prompts (`list[str]`):
List of prompts to judge.
completions (`list[list[str]]`):
List of completion pairs for each prompt.
shuffle_order (`bool`, *optional*, defaults to `True`):
Whether to shuffle the order of the completions to avoid positional bias.
return_scores (`bool`, *optional*, defaults to `False`):
If `True`, return probability scores of the first completion instead of ranks (i.e. a *soft-judge*).
temperature (`float`, *optional*, defaults to `1.0`):
Temperature for scaling logits if `return_scores` is True.
Returns:
`Union[list[int, float]]`:
If `return_scores` is `False`, returns a list of ranks (`0` or `1`) for each prompt, indicating which
completion is preferred. If `return_scores` is `True`, returns softmax probabilities for the first
completion.
Raises:
`ValueError`:
If the number of completions per prompt is not exactly 2.
Note:
Unlike llm-blender, ranks are 0-indexed (`0` means the first completion is preferred).
r r
z7PairRM judge requires exactly 2 completions per prompt.TFsizeN)r< disable_tqdm )axiskeepdims) lenr6 nprandomchoicezipr9 rankamaxexpsumtolist)
r r r r r< r= flip_maskflippairranks logit_maxexp_logit_shiftedprobss
r r zPairRMJudge.judge sG J {1~ ! # VW W (($ S\(JIJMiYdJefJD$4":47fKf !!';mbf!g QJE
[ E $Y /4R4 8E) B>I "uy'8 9 %/@rTX(YYEA;%%' 'A;%%' '+ gs D.N)TFg ?) r r r r r; r! r" r# floatr r) r r$ r r r3 r3 sx 8S # #
@(c@( $s)_@( @(
@(
@(
eCJ @(r r3 c
f e Zd ZdZ d dee dee fdZd dee deee ded ee fd
Z
y)
HfPairwiseJudgea
Pairwise judge based on the Hugging Face API with chat completion.
This judge is relevant for assessing the quality chat models, where the completion is a response to a given prompt.
Args:
model (`str`, *optional*, defaults to `"meta-llama/Meta-Llama-3-70B-Instruct"`):
Model to use for the judge.
token (`str`, *optional*):
Hugging Face API token to use for the [`huggingface_hub.InferenceClient`].
system_prompt (`str` or `None`, *optional*, defaults to `None`):
The system prompt to be used for the judge. If not provided, a default prompt is used. Note that the system
prompt should contain the following placeholders: `{prompt}`, `{response0}`, and `{response1}`. Also, the
inference is called with `max_tokens=1`, consequently the system prompt should ask for a single token
response.
Ntoken
system_promptc F t || | _ |xs t | _ y )N)modelrZ )r client DEFAULT_PAIRWISE_SYSTEM_PROMPTr[ )r r] rZ r[ s r r; zHfPairwiseJudge.__init__3 s &E? * L.Lr r r r r c |rTt j j ddgt | }t || D cg c] \ }}|r|d d d n| }}} fd}t
j j 5 }t |j ||| } d d d |r*t D
cg c] \ }
}|s |
nd |
z
} }
} S c c}}w # 1 sw Y =xY wc c}}
w )NTFr? rA c & j j | |d |d }j j d|dgd }|j d j
j }|dv r t | S t j d| d y
) Nr rC prompt response0 response1userrolecontent)messages
max_tokens01(Invalid response from the judge model: ''. Returning -1.rA )
r[ formatr^ chat_completionchoicesmessageri r) loggingdebug)rc
candidatesri
completionresponser s r get_rankz'HfPairwiseJudge.judge..get_rankC s ((//vTU blmnbo/pG 44[b?c>dqr4sJ!))!,44<.get_rank s ((//vTU blmnbo/pG!'G<=H ))55<<4::X`mn<oJ!))!,44<->$BSBS-S<<>t?P?P>Q Ro o $ 4#g,& & (($ S\(JIJMiYdJefJD$4":47fKf
2
2
4 G hEFE G KTU^K_`4TU1Xq58|;`E`
S\) 5 g G G
as )E% E E( E%)zgpt-4-turbo-previewNi r ) r r r r r r" r r) r; r! r# r r$ r r r r Y sh $ rw
:B3-
^cdgimdm^n
(T#Y (T$s)_ (UY (eijmen (r r c b e Zd ZdZdee fdZ d dee dee deee de d ee
f
d
Z y) AllTrueJudgea
Unify the decision of multiple [`BaseBinaryJudge`] instances.
Returns `1` only if all inner binary judges return `1`. If any judge returns `0`, it returns `0`. If any judge
returns `-1`, indicating a failure in its process, this judge will also return `-1`.
Implements the Mixture of Judges as described in the [CGPO paper](https://huggingface.co/papers/2409.20370).
Args:
judges (`list[BaseBinaryJudge]`): A list of [`BaseBinaryJudge`] instances whose decisions will be unified.
judgesc || _ y )N)r )r r s r r; zAllTrueJudge.__init__ s r Nr r r/ r r c \ | j D cg c] }|j |||| }}g }t | D ]n }t d |D rt d| d d|v r|j d :t
d |D r|j d ^|j d p |S c c}w )Nc 3 $ K | ] }|d v
yw)> r rC rA Nr$ .0binary_judgments r z%AllTrueJudge.judge.. s ] ?*4]s zInvalid binary judgment: z(, expected list of values in {0, 1, -1}.rA c 3 & K | ] }|d k( yw)rC Nr$ r s r r z%AllTrueJudge.judge.. s Ro_)Rs rC r )r r rJ anyr6 appendall) r r r r/ r r all_binary_judgmentsoutputbinary_judgmentss r r zAllTrueJudge.judge s ]a\g\g
SXEKK.>
N
#%9 :
! ]L\]] /0@/AAkl
%%
b!RAQRR
a
a
!
%
s B)r1 ) r r r r r! r. r; r" r r# r) r r$ r r r r sj
tO 4 15 " c #Y #49-
c
r r ) concurrent.futuresr{ ru abcr r typingr r numpyrG
accelerater huggingface_hubr transformers.utilsr import_utilsr r7 openair r_ r r& r+ r. r3 rY r r r$ r r r s # " " + 2 3 " :Y Y&YC &YR Y Y@%Yi %YPc(# c(L5' 5pF+ FR)? )r