bi]-ddlmZGddeZGddeZGddeZGdd eZGd d eZGd d eZy))Optimizer1Statec2eZdZ dfd ZxZS)Lionc <t |d|||d|||||| |  y)aj Base Lion optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-4): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0): The weight decay value for the optimizer. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. is_paged (`bool`, defaults to `False`): Whether the optimizer is a paged optimizer or not. lionis_pagedNsuper__init__) selfparamslrbetas weight_decay optim_bitsargs min_8bit_sizepercentile_clipping block_wiser __class__s R/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/bitsandbytes/optim/lion.pyr z Lion.__init__ s<L              ) -C6?g?gGz?r NdTF__name__ __module__ __qualname__r __classcell__rs@rrrs*  3 3 rrc0eZdZ dfd ZxZS)Lion8bitc <t |d|||d|d|||||  y)a 8-bit Lion optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-4): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0): The weight decay value for the optimizer. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. is_paged (`bool`, defaults to `False`): Whether the optimizer is a paged optimizer or not. rrr Nr rrrrrrrrrr rs rr zLion8bit.__init__@s<F             rrrrNrrTFr r%s@rr'r'?'  0 0 rr'c0eZdZ dfd ZxZS) Lion32bitc <t |d|||d|d|||||  y)a 32-bit Lion optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-4): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0): The weight decay value for the optimizer. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. is_paged (`bool`, defaults to `False`): Whether the optimizer is a paged optimizer or not. rrrr Nr r*s rr zLion32bit.__init__ts<F              rr+r r%s@rr.r.sr,rr.c0eZdZ dfd ZxZS) PagedLionc <t |d|||d|||||| d y)a Paged Lion optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-4): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0): The weight decay value for the optimizer. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. rrTr Nr ) rrrrrrrrrrrs rr zPagedLion.__init__s<F              r)rrrrNrrTr r%s@rr1r1s'  0 0 rr1c.eZdZ dfd ZxZS) PagedLion8bitc <t |d|||d|d||||d y)a Paged 8-bit Lion optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-4): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0): The weight decay value for the optimizer. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. rrr)Tr Nr rrrrrrrrrrs rr zPagedLion8bit.__init__s<D             rrrrNrrTr r%s@rr4r4$  / / rr4c.eZdZ dfd ZxZS)PagedLion32bitc <t |d|||d|d||||d y)a Paged 32-bit Lion optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-4): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0): The weight decay value for the optimizer. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. rrrTr Nr r6s rr zPagedLion32bit.__init__s<D              rr7r r%s@rr:r:r8rr:N)bitsandbytes.optim.optimizerrrr'r.r1r4r:rrr>sW 94 ?4 n1 1 h1 1 h1 1 h0 O0 f0 _0 r