bi9jdZddlZddlmZddlmZmZddlmZddl m Z ddl m Z e jeZGd d eZdd ed ed e fdZdd eded ed e fdZdd eded ed e fdZ dd ededed ed e f dZ dd edededed ed e f dZ dd edededed ed e f dZ d d ededededed ed e fdZej6eej8eej:eej<eej>eej@eejBeiZ" d!deeefd edeedeedeededed ed e fdZ#y)"z*PyTorch optimization for diffusion models.N)Enum)OptionalUnion) OptimizerLambdaLR)loggingc(eZdZdZdZdZdZdZdZdZ y) SchedulerTypelinearcosinecosine_with_restarts polynomialconstantconstant_with_warmuppiecewise_constantN) __name__ __module__ __qualname__LINEARCOSINECOSINE_WITH_RESTARTS POLYNOMIALCONSTANTCONSTANT_WITH_WARMUPPIECEWISE_CONSTANTQ/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/optimization.pyr r s( F F1JH1-rr optimizer last_epochreturnc t|d|S)a Create a schedule with a constant learning rate, using the learning rate set in optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. cyNr r)_s r z'get_constant_schedule..5srr"r)r!r"s r get_constant_scheduler*(s I{z BBrnum_warmup_stepsc8dtffd }t|||S)ad Create a schedule with a constant learning rate preceded by a warmup period during which the learning rate increases linearly between 0 and the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. current_stepcR|kr!t|ttdz Sy)N?floatmax)r-r+s r lr_lambdaz4get_constant_schedule_with_warmup..lr_lambdaIs, * *&s38H/I)JJ Jrr)intr)r!r+r"r3s ` r !get_constant_schedule_with_warmupr68s " IyZ @@r step_rulesci}|jd}|ddD]1}|jd\}}t|}t|} | ||<3t|d} d} | || } t|| |S)a Create a schedule with a constant learning rate, using the learning rate set in optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. step_rules (`string`): The rules for the learning rate. ex: rule_steps="1:10,0.1:20,0.01:30,0.005" it means that the learning rate if multiple 1 for the first 10 steps, multiple 0.1 for the next 20 steps, multiple 0.01 for the next 30 steps and multiple 0.005 for the other steps. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. ,N:c0dtdtffd }|S)Nstepsr#c~tj}t|D]\}}||ks ||cSS)N)sortedkeys enumerate)r= sorted_stepsi sorted_steplast_lr_multiple rules_dicts r rule_funczQget_piecewise_constant_schedule..create_rules_function..rule_funcmsK!*//"34L"+L"9 7;;&%l1o66 7$ #r)r5r1)rFrErGs`` r create_rules_functionz>get_piecewise_constant_schedule..create_rules_functionls $S $U $rr))splitr5r1r) r!r7r"rF rule_listrule_str value_str steps_strr=valuerErH rules_funcs r get_piecewise_constant_schedulerPQs$J  %IcrN"'~~c2 9Ii ! 5 " Yr]+'z3CDJ Izj AArnum_training_stepsc:dtffd }t|||S)a Create a schedule with a learning rate that decreases linearly from the initial lr set in the optimizer to 0, after a warmup period during which it increases linearly from 0 to the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_training_steps (`int`): The total number of training steps. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. r-c |kr!t|ttdz Stdt|z ttdz z S)Nr r0)r-rQr+s r r3z2get_linear_schedule_with_warmup..lr_lambdas^ * *&s16F/G)HH H )L89E#aI[^nInBo.lr_lambdas * *&s16F/G)HH H(889E#aI[^nInBo.lr_lambdas * *&s16F/G)HH H(889E#aI[^nInBo.lr_lambdas * *&s16F/G)HH H . .G# #'H,/??K0@!@K OOM}e33f> }///Y:*UUD6!\]^^ }111Y9IV`aa!D6!^_`` }111 -1!!    }''' -1!    $4I[hr r)r:)rYr:)r r:)gHz>r/r:)NNNr r/r:)$__doc__r[enumrtypingrr torch.optimrtorch.optim.lr_schedulerrutilsr get_loggerrloggerr r5r*r6strrPrUr1rardrrrrrrrrrrurwrrr rs1 "!-   H %.D. CY CC C C AAcA_bAltA2'By'Bc'BWZ'Bdl'BV]_66,/6EH6VY6 6@vx66,/6EH6V[6or6 6Frt 6 6,/ 6EH 6VY 6kn 6  6N 46464646  46  46  4646p99&&(ZG1&&(I$$&E!%&*(,H ]" #HH Hsm H ! H  H HHHr