bi/fddlZddlZddlmZmZddlmZddlmZddl m Z eGddZ y) N) dataclassfield)Optional)is_bitsandbytes_available) flatten_dictceZdZUdZeej jejdde d ddiZ e e d<ed dd iZe e d <eddd iZee d <eddddgdZee e d<eeddiZee d<eeddiZee d<eeddiZee d<edddiZe e d<edddiZe e d<ed dd!iZee d"<ed#dd$iZee d%<ed&dd'iZee d(<ed)dd*iZe e d+<ed,dd-iZee d.<ed dd/iZe e d0<ed1dd2iZ ee d3<ed4dd5iZ!e"e d6<ed7dd8iZ#e"e d9<ed#dd:iZ$ee d;<ed<ed#dd?iZ&ee d@<edAddBiZ'ee dC<edDddEiZ(e"e dF<edGddHiZ)e"e dI<edJddKiZ*e"e dL<edMddNiZ+e"e dO<edPddQiZ,e"e dR<ed#ddSiZ-ee dT<ed4ddUiZ.e"e dV<ed#ddWiZ/ee dX<ed,ddYiZ0ee dZ<ed7dd[iZ1e"e d\<edMdd]iZ2e"e d^<ed4dd_iZ3e"e d`<edAddaiZ4ee db<edcdddiZ5ee de<edcddfiZ6ee dg<edAddhiZ7ee di<edz Adam epsilon.train_adam_epsilonz&Number of gradient accumulation steps.!train_gradient_accumulation_stepsz,Maximum gradient norm for gradient clipping.train_max_grad_normz'Number of inner epochs per outer epoch.train_num_inner_epochsz8Whether to use classifier-free guidance during training. train_cfgzClip advantages to the range.train_adv_clip_maxzPPO clip range.train_clip_rangez"Fraction of timesteps to train on.train_timestep_fractionz7Whether to track statistics for each prompt separately.per_prompt_stat_trackingz?Number of reward values to store in the buffer for each prompt.$per_prompt_stat_tracking_buffer_sizez7Minimum number of reward values to store in the buffer."per_prompt_stat_tracking_min_countz*Whether to compute rewards asynchronously.async_reward_computationz>Maximum number of workers to use for async reward computation. max_workerszr?r@rArBrCrLrPrMrKr r sN[z  !-mU <mnHc-.Hc.)D#$?/ Hhsm!STND %BC !fgND!&@A!#NOFC67JNOIs!&VW!#!56OS89J>?K">?c?@J$)<=$5#EFs).BC) #"EFc!&FG!"'*+"$-(e$-(e&+./&U!&/*!.3BC.%s"'HI"#(CD#CTUIt!&9:!$+,e&+>?&U&+ST&d16[\1(#/4ST/&&+FG&dZ[K"XYcRSK ) rMr ) rUrX dataclassesrrtypingr transformersrcorerr rarMrKrfs5 (2 QQ QrM