biL$|ddlZddlmZddlmZGddeZGddeZGdd eZGd d eZy) N) Optimizer)Optimizer1Statec2eZdZ dfd ZxZS)LARSc `|dk(r tdt | d||||fd|||| | | d y)aG Base LARS optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`): The learning rate. momentum (`float`, defaults to 0): The momentum value speeds up the optimizer by taking bigger steps. dampening (`float`, defaults to 0): The dampening value reduces the momentum of the optimizer. weight_decay (`float`, defaults to 1e-2): The weight decay value for the optimizer. nesterov (`bool`, defaults to `False`): Whether to use Nesterov momentum. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. max_unorm (`float`, defaults to 0.02): The maximum gradient norm. r'LARS without momentum is not supported!larsF max_unorm block_wiseNNotImplementedErrorsuper__init__) selfparamslrmomentum dampening weight_decaynesterov optim_bitsargs min_8bit_sizepercentile_clippingr __class__s R/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/bitsandbytes/optim/lars.pyrz LARS.__init__ sTR q=%&OP P     y !        ) rrrF Nd{Gz?__name__ __module__ __qualname__r __classcell__rs@rrr s*  8 8 rrc0eZdZ dfd ZxZS)LARS8bitc `|dk(r tdt | d||||fd|d||| | d y) a 8-bit LARS optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`): The learning rate. momentum (`float`, defaults to 0): The momentum value speeds up the optimizer by taking bigger steps. dampening (`float`, defaults to 0): The dampening value reduces the momentum of the optimizer. weight_decay (`float`, defaults to 1e-2): The weight decay value for the optimizer. nesterov (`bool`, defaults to `False`): Whether to use Nesterov momentum. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. max_unorm (`float`, defaults to 0.02): The maximum gradient norm. rrr r Fr Nr rrrrrrrrrrr rs rrzLARS8bit.__init__HsTL q=%&OP P     y !       rrrrFNr!r"r#r$r)s@rr+r+G'  5 5 rr+c0eZdZ dfd ZxZS) LARS32bitc `|dk(r tdt | d||||fd|d||| | d y) a 32-bit LARS optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`): The learning rate. momentum (`float`, defaults to 0): The momentum value speeds up the optimizer by taking bigger steps. dampening (`float`, defaults to 0): The dampening value reduces the momentum of the optimizer. weight_decay (`float`, defaults to 1e-2): The weight decay value for the optimizer. nesterov (`bool`, defaults to `False`): Whether to use Nesterov momentum. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. max_unorm (`float`, defaults to 0.02): The maximum gradient norm. rrr r r Fr Nrr.s rrzLARS32bit.__init__sTL q=%&OP P     y !        rr/r$r)s@rr2r2r0rr2cfeZdZ dfd ZfdZej ddZxZS) PytorchLARSc|dkrtd||dkrtd||dkrtd|t||||||}|r|dks|dk7r tdt | ||y)Nr zInvalid learning rate: zInvalid momentum value: zInvalid weight_decay value: )rrrrrr rz8Nesterov momentum requires a momentum and zero dampening) ValueErrordictrr) rrrrrrrr defaultsrs rrzPytorchLARS.__init__s 86rd;< < c>7zBC C # ;L>JK K%   Q)q.WX X *rcjt|||jD]}|jddy)NrF)r __setstate__ param_groups setdefault)rstategrouprs rr;zPytorchLARS.__setstate__s5 U#&& 0E   Z / 0rc\d}|$tj5|}ddd|jD]f}|d}|d}|d}|d}|d}|d} |dD]<} | j|j| } | j} |d k7r| j | | } |d k7ro| j d d} | )tj| j} | | d <n%| j|j| d |z |r | | |zz}n| }d }|dkDrg| jtjk(sJtj| j}tj}|||zkDr||z|z }| j| |z ?i|S#1swYxYw)zPerforms a single optimization step. Args: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nrrrrr rrr)alphamomentum_bufferg?r )torch enable_gradr<gradr>addgetclonedetachmul_add_dtypefloat32norm)rclosurelossr?rrrrr rpr>d_pbufupdate update_scalepnormunorms rstepzPytorchLARS.steps  ""$ !y !&&' 9E 0LZ(Hk*IZ(Hk*ItB8_ 966> 1 ff1$''!<'8Cq=))$5tr^sE  !89 ?9 x6 6 r6 6 rY)Yr