bi9ddlZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZd Zd ZGd dZGddZGddeZGdde ZGddZGddZy)N)deepcopy)optim)AcceleratedOptimizer)AcceleratedScheduler)DistributedType)is_bnb_available)compare_versionsc |jjDcic] \}}|dvs ||}}}ddlm}|}t dddrmd|d<t |t j}tr=|s;dd l m}t ||j|jfr |jd k(}nd}|rd |d<t ddd rht |t j}tr:|s8dd l m}t ||j|jfr |jd k(}|rdd lm} | }tdr=t dddr0ddl m} m} t || | fr |jd k(} | rddlm} | }||j,fi|Scc}}w#t$r|jjd k(}YwxYw#t$r|jjd k(}YwxYw#t$r|jjd k(} YwxYw)z Args: optimizer: torch.optim.Optimizer Returns the DeepSeedCPUOptimizer (deepspeed.ops) version of the optimizer. )lr weight_decayr)DeepSpeedCPUAdam deepspeedz>=z0.3.1F adamw_modeN Tz0.5.5)DeepSpeedCPUAdagradz0.38.0) min_versionz0.11.0)Lion Lion32bit)DeepSpeedCPULion)defaultsitemsdeepspeed.ops.adamrr isinstancerAdamWr bitsandbytes.optim AdamW32bit optim_bitsAttributeErrorargsAdagrad Adagrad32bitdeepspeed.ops.adagradrrrdeepspeed.ops.lionr param_groups) optimizerkvrroptimizer_classis_adawbnb_optis_adarrr is_bnb_32bitsrs U/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/accelerate/utils/deepspeed.pymap_pytorch_optim_to_deepspeedr0s"+!3!3!9!9!;[AqDZ?Z1[H[4&O T73!&Y 4  g 0)gmmW5G5G%HI>'22b8G  %)H\ " T73Iu}}5  f 0)goow7K7K%LM=&11R7F  A1OH-2B;PTV^2_6 i$ !2 3 @ ) 4 4 : ?"2 911 >X >>y\&&>'nn772=G>(&=&^^66"z.get_active_deepspeed_plugin..tsY6Ys) distributed_typer DEEPSPEED ValueErrorrdeepspeed_pluginsdictnextvalues)states r/get_active_deepspeed_pluginr@dsh !:!:: ,  e--t 4&&& YU%<%<%C%C%EY YYcPeZdZdZdZdZdZd dZddZdZ d Z d Z d Z d Z y)HfDeepSpeedConfigaJ This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage. A `weakref` of this object is stored in the module's globals to be able to access the config from areas where things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore it's important that this object remains alive while the program is still running. [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic the DeepSpeed configuration is not modified in any way. Args: config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict. c6t|tr t|}nbtjj |r,t |d5}tj|}dddn tj|}|_|j%y#1swY!xYw#tj$r<tj|jd}tj|}YrwxYw#ttt f$rt!d|wxYw)Nzutf-8)encodingzoExpected a string path to an existing deepspeed config, or a dictionary, or a base64 encoded string. Received: )rr<rospathexistsopenjsonloadloadsJSONDecodeErrorbase64urlsafe_b64decodedecodeUnicodeDecodeErrorr r:configset_stage_and_offload)selfconfig_file_or_dictrRfconfig_decodeds r/__init__zHfDeepSpeedConfig.__init__s  )4 012F WW^^/ 0)G< &1 & & 8!ZZ(;Q%R%Y%YZa%bN!ZZ7F8' C  FGZF[\ s1 B*B#B #A C2/C51C22C55#Dc$|jdd|_d|_|js|j rRt ddg}t |jd|jdg}t ||zdkDrd |_yyy) Nzzero_optimization.stageFcpunvmez*zero_optimization.offload_optimizer.devicez&zero_optimization.offload_param.devicerT) get_value_stage_offloadis_zero2is_zero3setlen)rToffload_devices_validoffload_devicess r/rSz'HfDeepSpeedConfig.set_stage_and_offloadsnn%>C  ==?dmmo$'$8 !!NN#OPNN#KLO ?%::;a? $ @.rAc|j}|jd}|j}|D]}|j|}|d|fcS||fS)N.)rRsplitpopget)rT ds_key_longrRnodesds_keynodes r/find_config_nodez"HfDeepSpeedConfig.find_config_nodes`!!#& $DZZ%F~V|# $ v~rANcV|j|\}}||S|j||S)zG Returns the set value or `default` if no value is set )rorj)rTrkdefaultrRrms r/r]zHfDeepSpeedConfig.get_values3..{; >Nzz&'**rAc|j}|jd}|D]6}|}|j|}||rtd|d|jy|j yy)z Deletes a sub-section of the config file if it's found. Unless `must_exist` is `True` the section doesn't have to exist. rgNz Can't find z entry in the config: )rRrhrjr:ri)rTrk must_existrRrlrn parent_configs r/del_config_sub_treez%HfDeepSpeedConfig.del_config_sub_trees !!#& D"MZZ%F~${;-?UVZVaVaUb%cdd   $   d # %rAcB|j|}|dSt|S)z Returns `True`/``False` only if the value is set, always `False` otherwise. So use this method to ask the very specific question of whether the value is set to `True` (and it's not set to `False`` or isn't set). Fr]boolrTrkvalues r/is_truezHfDeepSpeedConfig.is_trues% {+ u64;6rAcD|j|}|dSt| S)z Returns `True`/``False` only if the value is set, always `False` otherwise. So use this method to ask the very specific question of whether the value is set to `False` (and it's not set to `True`` or isn't set). Frwrys r/is_falsezHfDeepSpeedConfig.is_falses' {+ u:tE{?:rAc |jdk(S)Nrr^rTs r/r`zHfDeepSpeedConfig.is_zero2{{arAc |jdk(S)Nrrs r/razHfDeepSpeedConfig.is_zero3rrAc|jSr3)r_rs r/ is_offloadzHfDeepSpeedConfig.is_offloads }}rAr3)F)__name__ __module__ __qualname____doc__rXrSror]rur{r}r`rarrAr/rCrCws9 %4%& +$.7;  rArCc$eZdZdZdZddZdZy)DeepSpeedEngineWrapperz Internal wrapper for deepspeed.runtime.engine.DeepSpeedEngine. This is used to follow conventional training loop. Args: engine (deepspeed.runtime.engine.DeepSpeedEngine): deepspeed engine to wrap c||_yr3)engine)rTrs r/rXzDeepSpeedEngineWrapper.__init__s  rAc |jj||jj|fi||r|jjyy)N) is_boundary)r"set_gradient_accumulation_boundarybackwardstep)rTlosssync_gradientskwargss r/rzDeepSpeedEngineWrapper.backwardsP 66>6R  T,V,  KK    rAcr|jj}t|dr|jS|S)z3Get the global gradient norm from DeepSpeed engine.item)rget_global_grad_normhasattrr)rT grad_norms r/rz+DeepSpeedEngineWrapper.get_global_grad_norms1KK446 9f %>># #rAN)T)rrrrrXrrrrAr/rrs,rArc@eZdZdZfdZddZdZedZxZ S)DeepSpeedOptimizerWrapperz Internal wrapper around a deepspeed optimizer. Args: optimizer (`torch.optim.optimizer.Optimizer`): The optimizer to wrap. c`t||ddt|jd|_y)NF)device_placementscaleroverflow)superrXrr'__has_overflow__)rTr' __class__s r/rXz"DeepSpeedOptimizerWrapper.__init__0s* U4H ' CrAcyr3r)rT set_to_nones r/ zero_gradz#DeepSpeedOptimizerWrapper.zero_grad4 rAcyr3rrs r/rzDeepSpeedOptimizerWrapper.step7rrAcH|jr|jjSy)zTWhether or not the optimizer step was done, or skipped because of gradient overflow.F)rr'rrs r/step_was_skippedz*DeepSpeedOptimizerWrapper.step_was_skipped:s   >>** *rAr3) rrrrrXrrpropertyr __classcell__rs@r/rr's,D  rArc(eZdZdZfdZdZxZS)DeepSpeedSchedulerWrapperz Internal wrapper around a deepspeed scheduler. Args: scheduler (`torch.optim.lr_scheduler.LambdaLR`): The scheduler to wrap. optimizers (one or a list of `torch.optim.Optimizer`): c&t|||yr3)rrX)rT scheduler optimizersrs r/rXz"DeepSpeedSchedulerWrapper.__init__Ls J/rAcyr3rrs r/rzDeepSpeedSchedulerWrapper.stepOrrA)rrrrrXrrrs@r/rrBs0 rArceZdZdZddZy) DummyOptima Dummy optimizer presents model parameters or param groups, this is primarily used to follow conventional training loop when optimizer config is specified in the deepspeed config file. Args: lr (float): Learning rate. params (iterable): iterable of parameters to optimize or dicts defining parameter groups weight_decay (float): Weight decay. **kwargs (additional keyword arguments, *optional*): Other arguments. c <||_||_||_||_yr3)paramsr rr)rTrr rrs r/rXzDummyOptim.__init__cs ( rAN)gMbP?rrrrrrXrrAr/rrSs  rArceZdZdZddZy)DummySchedulera Dummy scheduler presents model parameters or param groups, this is primarily used to follow conventional training loop when scheduler config is specified in the deepspeed config file. Args: optimizer (`torch.optim.optimizer.Optimizer`): The optimizer to wrap. total_num_steps (int, *optional*): Total number of steps. warmup_num_steps (int, *optional*): Number of steps for warmup. lr_scheduler_callable (callable, *optional*): A callable function that creates an LR Scheduler. It accepts only one argument `optimizer`. **kwargs (additional keyword arguments, *optional*): Other arguments. Nc J||_||_||_||_||_yr3)r'total_num_stepswarmup_num_stepslr_scheduler_callabler)rTr'rrrrs r/rXzDummyScheduler.__init__|s(". 0%:" rA)NrNrrrAr/rrjs "rAr)rNrJrFcopyrtorchrr'rrr dataclassesr importsr versionsr r0r@rCrrrrrrrAr/rsx ,,(%&D?NZ&CCL''T 46 4 ".rA