bisddlZddlZddlZddlZddlZddlZddlZddlmZddlm Z m Z m Z m Z m Z mZmZddlZddlZddlmZddlmZddlmZddlmZmZmZmZmZmZm Z e r,ddl!Z!e!jDjFjIrddl#Z#erdd l%m&Z&erdd l'm(Z(erddl)Z)d e*fd Z+d Z,de-fdZ. d6dededej^dej^dej^dej^dej^de0dee ej^e ej^ffdZ1dede e-ej^ffdZ2ejffdeejhjje ejhjjffdZ6de e-ej^fde-dejhjjfd Z7d!e e-ejhjjfde e-e ffd"Z8 d7d#e-d$e*d%e0d&e0d'e0d(eejre-fd)e ejtfd*Z;d8d#e-fd+Zfd1Z?d2Z@d3ZAGd4d5ZBy)9N)contextmanager)AnyDictIterableListOptionalTupleUnion)UNet2DConditionModel)DiffusionPipeline)SchedulerMixin)convert_state_dict_to_diffusersconvert_state_dict_to_peft deprecateis_peft_availableis_torch_npu_availableis_torchvision_availableis_transformers_available)set_peft_model_state_dict) transformsseedc(tj|tjj|tj|t r tj j|ytjj|y)z Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch`. Args: seed (`int`): The seed to set. Returns: `None` N) randomrnptorch manual_seedrnpumanual_seed_allcuda)rs S/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/training_utils.pyset_seedr",sX KKIINN4 d !!$' ""4(c|j}|dz}d|z dz}|j|j|j}t |j t |j kr1|d}t |j t |j kr1|j |j }|j|j|j}t |j t |j kr1|d}t |j t |j kr1|j |j }||z dz}|S)a Computes SNR as per https://github.com/TiankaiHang/Min-SNR-Diffusion-Training/blob/521b624bd70c67cee4bdf49225915f5945a872e3/guided_diffusion/gaussian_diffusion.py#L847-L849 for the given timesteps using the provided noise scheduler. Args: noise_scheduler (`NoiseScheduler`): An object containing the noise schedule parameters, specifically `alphas_cumprod`, which is used to compute the SNR values. timesteps (`torch.Tensor`): A tensor of timesteps for which the SNR is computed. Returns: `torch.Tensor`: A tensor containing the computed SNR values for each timestep. ??device).N)alphas_cumprodtor(floatlenshapeexpand)noise_scheduler timestepsr*sqrt_alphas_cumprodsqrt_one_minus_alphas_cumprodalphasigmasnrs r! compute_snrr7@sH %33N(#-%(>%9c$A!.00 8H8H0I)TZZ\ !'' (3y+? ?1)< !'' (3y+? ?  & &y 7E$A$D$DIL\L\$D$]^g$h$n$n$p! +11 2S5I I(Ei(P% +11 2S5I I ) 0 0 AE 5=Q C Jr#interpolation_typects td|dk(rtjj}|S|dk(rtjj }|S|dk(rtjj }|S|dk(rtjj}|S|dk(rtjj}|S|dk(rtjj}|S|dk(rtjj}|Std |d ) a Maps a string describing an interpolation function to the corresponding torchvision `InterpolationMode` enum. The full list of supported enums is documented at https://pytorch.org/vision/0.9/transforms.html#torchvision.transforms.functional.InterpolationMode. Args: interpolation_type (`str`): A string describing an interpolation method. Currently, `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos` are supported, corresponding to the supported interpolation modes in torchvision. Returns: `torchvision.transforms.InterpolationMode`: an `InterpolationMode` enum used by torchvision's `resize` transform. zhPlease make sure to install `torchvision` to be able to use the `resolve_interpolation_mode()` function.bilinearbicubicboxnearest nearest_exacthamminglanczoszThe given interpolation mode z is not supported. Currently supported interpolation modes are `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos`.) r ImportErrorrInterpolationModeBILINEARBICUBICBOXNEAREST NEAREST_EXACTHAMMINGLANCZOS ValueError)r8interpolation_modes r!resolve_interpolation_moderLes; $ % v  Z''99BB& % y ('99AA" ! u $'99==  y ('99AA   .'99GG  y ('99AA  y ('99AA  +,>+?@m n  r#unetr0r1noise noisy_latentstargetencoder_hidden_statesdream_detail_preservationreturncX|jj|j|dddf}d|z dz} | |z} d} tj5||||j } dddd\} } |j jdk(rO| }||z j}|j| |j| |z} |j|} | | fS|j jdk(r tdtd|j j#1swYxYw) a Implements "DREAM (Diffusion Rectification and Estimation-Adaptive Models)" from https://huggingface.co/papers/2312.00210. DREAM helps align training with sampling to help training be more efficient and accurate at the cost of an extra forward step without gradients. Args: `unet`: The state unet to use to make a prediction. `noise_scheduler`: The noise scheduler used to add noise for the given timestep. `timesteps`: The timesteps for the noise_scheduler to user. `noise`: A tensor of noise in the shape of noisy_latents. `noisy_latents`: Previously noise latents from the training loop. `target`: The ground-truth tensor to predict after eps is removed. `encoder_hidden_states`: Text embeddings from the text model. `dream_detail_preservation`: A float value that indicates detail preservation level. See reference. Returns: `tuple[torch.Tensor, torch.Tensor]`: Adjusted noisy_latents and target. Nr&r%)NNepsilon v_predictionz/DREAM has not been implemented for v-predictionzUnknown prediction type ) r*r+r(rno_gradsampleconfigprediction_typedetachmul_addNotImplementedErrorrJ)rMr0r1rNrOrPrQrRr*r3 dream_lambdapred_noisy_latents_targetpredicted_noise delta_noises r! compute_dream_and_update_latentsresC:%3366y7G7GHTXZ^`dIdeN%(>%9c$A!12KKL D LM9.CDKKL+NG--:.668 &&**+H;+VW**[) 7 ""    / /> A!"STT3O4J4J4Z4Z3[\]]LLs D  D)ci}|jD]S\}}t|dst|d}|"|j}|j D]\}}|||d|<U|S)zL Returns: A state dict containing just the LoRA parameters. set_lora_layer lora_layerz.lora.) named_moduleshasattrgetattr state_dictitems)rMlora_state_dictnamemodulerhcurrent_lora_layer_sdlora_layer_matrix_name lora_params r!unet_lora_state_dictrts O**,Z f 6+ , 6J%(2(=(=(?%:O:U:U:WZ6*JOYOtfF3I2J$KLZ Z r#modelct|ts|g}|D]:}|jD]%}|js|j ||_'<y)z Casts the training parameters of the model to the specified data type. Args: model: The PyTorch model whose parameters will be cast. dtype: The data type to which the model parameters will be cast. N) isinstancelist parameters requires_gradr+data)rudtypemparams r!cast_training_paramsrsQ eT " -\\^ -E"""XXe_  --r#rnprefix text_encoderc|jDcic]+\}}|j|s|j|d|-}}}tt |}t ||dycc}}w)aD Sets the `lora_state_dict` into `text_encoder` coming from `transformers`. Args: lora_state_dict: The state dictionary to be set. prefix: String identifier to retrieve the portion of the state dict that belongs to `text_encoder`. text_encoder: Where the `lora_state_dict` is to be set. default) adapter_nameN)rm startswithreplacerrr)rnrrkvtext_encoder_state_dicts r!!_set_state_dict_into_text_encoderrss3B2G2G2I*.!QQ\\Z`Ma199VR !A%99XYp9qrl,CR[\ s A*A*modules_to_saveci}|jD]+\}}| |jdj||d<-|S)Nr_lora_adapter_metadata)rm peft_configto_dict)r metadatas module_namerps r!_collate_lora_metadatars[I.446h V  @F@R@RS\@]@e@e@gI %;< =h r#weighting_scheme batch_size logit_mean logit_std mode_scaler( generatorc|dk(rFtj|||f||}tjjj |}|S|dk(rVtj |f||}d|z |tj tj|zdz dzdz |zzz }|Stj |f||}|S)a Compute the density for sampling the timesteps when doing SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://huggingface.co/papers/2403.03206v1. logit_normal)meanstdsizer(rmode)rr(rr r)) rnormalnn functionalsigmoidrandcosmathpi)rrrrrr(rus r!%compute_density_for_timestep_samplingrs >) LLjizmTZfo p HH   ' ' * H V # JJZM&I N EJ%))DGGaK!O"<"AA"E"IJ J H JJZM&I N Hr#c|dk(r|dzj}|S|dk(r)dd|zz d|dzzz}dtj|zz }|Stj|}|S)z Computes loss weighting scheme for SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://huggingface.co/papers/2403.03206v1. sigma_sqrtgcosmapr r))r,rrr ones_like)rsigmas weightingbots r!compute_loss_weighting_for_sd3r!sz<'T\((*  X %!f*nq619},3'  OOF+ r#c"tjtjj rtjj ytj jj rtjj ytrtjj yttdr>tjj rtjj yyy)zV Runs garbage collection. Then clears the cache of the available accelerator. xpuN) gccollectrr is_available empty_cachebackendsmpsr torch_npurrjrr#r! free_memoryr3sJJL zz      ( ( *   ! !!#  599#9#9#; $< r#T)offloadmodulesrc'K|r~td|D }|r1|Dcgc]%}t|jj'}}n t |dk(sJ|djg}|D]}|j | d|r&t |D]\}}|j |yycc}w#|r&t |D]\}}|j |wwxYww)a Context manager that, if offload=True, moves each module to `device` on enter, then moves it back to its original device on exit. Args: device (`str` or `torch.Device`): Device to move the `modules` to. offload (`bool`): Flag to enable offloading. c3<K|]}t|tywN)rwr ).0r}s r! z!offload_models..PsM:a):;Msr rN)anynextryr(r-r+zip)r(rris_modelr}original_devicesorig_devs r!offload_modelsrCsMWMMM ELMQ\\^ 4 ; ;M Mw<1$ $$ ' 1 12  A DDL  "7,<=  8X   N "7,<=  8X  s(C#*B1;C#B6.C#6*C  C#c0|s td|jjd}g}|D]}tjd|}|std|d t |j d}t |j d}|dks|dkr td |d zdk7s|d zdk7rtjd |d |d |j||f|s td|S#t$r}td|d||d}~wwxYw)zGParses a string defining buckets into a list of (height, width) tuples.zBucket string cannot be empty.;z^\s*(\d+)\s*,\s*(\d+)\s*$zInvalid bucket format: 'z'. Expected 'height,width'.r r)rz,Bucket dimensions must be positive integers.zBucket dimension (,z.) not divisible by 8. This might cause issues.z Invalid integer in bucket pair 'z': Nz.No valid buckets found in the provided string.) rJstripsplitrematchintgroupwarningswarnappend) buckets_str bucket_pairsparsed_bucketspair_strrheightwidthes r!parse_buckets_stringres8 9::$$&,,S1LN Y5x@7zA\]^ ^ YQ(F A'E{eqj !OPPzQ%!)q. 26(!E7Bpqr  ! !65/ 2 Y IJJ   Y?zQCPQWX X YsBC55 D>DDctd}d}t|D]&\}\}}t||z||zz }||ks#|}|}(|S)z6Finds the closes bucket to the given height and width.infN)r, enumerateabs) hwbucket_options min_metricbest_bucket_idx bucket_idxbucket_hbucket_wmetrics r!find_nearest_bucketrs_uJO,5n,E)( (XxQ\AL01 Z J(O ) r#ceZdZdZ ddeej jdedede de de ee fd e ee fd e d e e d eee ffd ZedddZdZde defdZej*deej jfdZdeej jddfdZddZd ddZdefdZdeej jddfdZdeej jddfdZdeddfdZy)!EMAModelz6 Exponential Moving Average of models weights Nrydecay min_decayupdate_after_stepuse_ema_warmup inv_gammapowerforeach model_cls model_configc  t|tjjr#d} t dd| d|j }d}| j ddd } t dd| d| d}| j d dd } t d d| d| d }t|}|D cgc] } | jj"c} |_ | j d d&d } t d d| d|j| d d|_ ||_ ||_||_||_||_||_d|_d|_||_| |_| |_ycc} w)a{ Args: parameters (Iterable[torch.nn.Parameter]): The parameters to track. decay (float): The decay factor for the exponential moving average. min_decay (float): The minimum decay factor for the exponential moving average. update_after_step (int): The number of steps to wait before starting to update the EMA weights. use_ema_warmup (bool): Whether to use EMA warmup. inv_gamma (float): Inverse multiplicative factor of EMA warmup. Default: 1. Only used if `use_ema_warmup` is True. power (float): Exponential factor of EMA warmup. Default: 2/3. Only used if `use_ema_warmup` is True. foreach (bool): Use torch._foreach functions for updating shadow parameters. Should be faster. device (Optional[Union[str, torch.device]]): The device to store the EMA weights on. If None, the EMA weights will be stored on CPU. @crowsonkb's notes on EMA Warmup: If gamma=1 and power=1, implements a simple average. gamma=1, power=2/3 are good values for models you plan to train for a million or more steps (reaches decay factor 0.999 at 31.6K steps, 0.9999 at 1M steps), gamma=1, power=3/4 for models you plan to train for less (reaches decay factor 0.999 at 10K steps, 0.9999 at 215.4k steps). zzPassing a `torch.nn.Module` to `ExponentialMovingAverage` is deprecated. Please pass the parameters of the module instead.z9passing a `torch.nn.Module` to `ExponentialMovingAverage`1.0.0F standard_warnT max_valueNzCThe `max_value` argument is deprecated. Please use `decay` instead. min_valuezGThe `min_value` argument is deprecated. Please use `min_decay` instead.r(z=The `device` argument is deprecated. Please use `to` instead.r'r)rwrrModulerrygetrxcloner[ shadow_paramsr+temp_stored_paramsrrrrrroptimization_stepcur_decay_valuerrr)selfryrrrrrrrrrkwargsdeprecation_messageps r!__init__zEMAModel.__init__stF j%((// 2D  K##   $..0J"N ::k4 ( 4"g  k7,?u U;'E ::k4 ( 4"k  k7,?u U{+I*% :DEQaggi..0E ::h % 1"a  h)> !ij j    $lm m**4+<+<=__& -   .:. U%%'( d#r#rctd||jz dz }|dkry|jr$dd||jz z|j zz }n d|zd|zz }t ||j }t||j}|S)zN Compute the decay factor for the exponential moving average. rr )maxrrrrminrr)rrsteprs r! get_decayzEMAModel.get_decays1'$*@*@@1DE 19   1tdnn'<#<$**"LLO 4xBI6Ootzz:ot~~>r#c t|tjjr!d}t dd|d|j }t |}|xjdz c_|j|j}||_ d|z }tj}|jrZtrItjj j#r!t j$j'|d}|5|Dcgc]}|j(s|}}t+|j,|Dcgc]\}}|j(s|} }}t/|t/|krgtj0t+|j,|Dcgc]\}}|j(r|c}}|Dcgc]}|j(r|c}d tj2| tj4| || dddyt+|j,|D]\}}trItjj j#r!t j$j'|d}|5|j(r|j7|||z zn|j9|dddycc}wcc}}wcc}}wcc}w#1swYyxYw#1swYxYw) NzPassing a `torch.nn.Module` to `ExponentialMovingAverage.step` is deprecated. Please pass the parameters of the module instead.z>passing a `torch.nn.Module` to `ExponentialMovingAverage.step`rFrr ) modifier_rankT) non_blocking)r4)rwrrrrryrxrr r contextlib nullcontextrr transformers integrations deepspeedis_deepspeed_zero3_enabledzeroGatheredParametersrzrrr-_foreach_copy_ _foreach_sub_ _foreach_subsub_copy_) rryrrone_minus_decaycontext_managerr~ params_grads_param s_params_grads r!rz EMAModel.stepsz j%((// 2D  P##   $..0J*%  !#t556$e)$002 <<(*|/H/H/R/R/m/m/o"+.."C"CJ^b"C"c  2<T@S@SuT T25d6H6H*2U! .Y^YlYlG! !{#c*o5((7:4;M;Mz7Zv^Webgbubuv,6R5e>Q>QR%) ##!5#5#5m[#QYh  $#&d&8&8*"E -,.<3L3L3V3V3q3q3s&/nn&G&G]a&G&bO$-** _%%HI e, --  -#U! wR  ,--sgKJ:J:K9J?J?AKK )K -K4K K 5K96K:KKK% c t|}|jrytj|Dcgc]}|jc}t |j |Dcgc]*\}}|j|jj,c}}yt |j |D]C\}}|jj|j|jjEycc}wcc}}w)aa Copy current averaged parameters into given collection of parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored moving averages. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. N) rxrrr,r{rrr+r(r0)rryr~r4s r!rzEMAModel.copy_toMs*% <<  )344EHI[I[]gEhi>7EELL)..i  #&d&8&8*"E @   ELL!9!>!>? @ 5is C./C3 ch|jDcgc]}|jc}|_ycc}w)z Move internal buffers of the ExponentialMovingAverage to pinned memory. Useful for non-blocking transfers for offloading EMA params to the host. N)r pin_memory)rr s r!r8zEMAModel.pin_memory`s& 7;6H6HIallnIIs/c|jDcgc]9}|jr|j|||n|j||;c}|_ycc}w)z Move internal buffers of the ExponentialMovingAverage to `device`. Args: device: like `device` argument to `torch.Tensor.to` )r(r|r#)r(r#N)ris_floating_pointr+)rr(r|r#r s r!r+z EMAModel.tohs_''  ""$ DDe,D GV,? @  s>Ac |j|j|j|j|j|j |j |jdS)z Returns the state of the ExponentialMovingAverage as a dict. This method is used by accelerate during checkpointing to save the ema state dict. rrrrrrrrr<)rs r!rlzEMAModel.state_dictwsNZZ!%!7!7!%!7!7"11ZZ!//  r#c|Dcgc].}|jjj0c}|_ycc}w)z Saves the current parameters for restoring later. Args: parameters: Iterable of `torch.nn.Parameter`. The parameters to be temporarily stored. N)r[cpurr)rryr~s r!storezEMAModel.stores4NX"XE5<<>#5#5#7#=#=#?"X"Xs3Ac|j td|jrXtj|Dcgc]}|j c}|jDcgc]}|j c}d|_yt |j|D]*\}}|j j|j ,d|_ycc}wcc}w)aG Restore the parameters stored with the `store` method. Useful to validate the model with EMA parameters without: affecting the original optimization process. Store the parameters before the `copy_to()` method. After validation (or model saving), use this to restore the former parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored parameters. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. NzGThis ExponentialMovingAverage has no `store()`ed weights to `restore()`)r RuntimeErrorrrr,r{rr0)rryr~c_params r!restorezEMAModel.restores  " " *hi i <<  )344SWSjSj6kw||6k #' #&d&=&=z"J /   . /#'56ks CC rlctj|}|jd|j|_|jdks|jdkDr t d|jd|j |_t |j ts t d|jd|j|_t |jts t d|jd |j|_ t |jts t d |jd |j|_ t |jts t d |jd |j|_ t |jttfs t d|jd|j|_t |jttfs t d|jdd}|T||_t |jt s t dt#d|jDs t dyy)a Loads the ExponentialMovingAverage state. This method is used by accelerate during checkpointing to save the ema state dict. Args: state_dict (dict): EMA state. Should be an object returned from a call to :meth:`state_dict`. rrr&zDecay must be between 0 and 1rzInvalid min_decayrzInvalid optimization_steprzInvalid update_after_steprzInvalid use_ema_warmuprzInvalid inv_gammarz Invalid powerrNzshadow_params must be a listc3PK|]}t|tj ywr)rwrTensor)rr s r!rz+EMAModel.load_state_dict..sOqz!U\\2Os$&z!shadow_params must all be Tensors)copydeepcopyrrrJrrwr,rrrrboolrrrrxall)rrlrs r!rzEMAModel.load_state_dicts]]:. ^^GTZZ8 :: tzzC/<= =# T^^D$..%001 1!+0CTE[E[!\$00#689 9!+0CTE[E[!\$00#689 9(nn-=t?R?RS$--t456 6# T^^D$..5#,701 1^^GTZZ8 $**ucl3_- -"=  $!.D d00$7 !?@@OD@4@&J   D &Y);); <YY'(588+=+=">'4'2.F$.F4.Fr#r)r&)NNNr>Nr)Cr$rGrrrrrrtypingrrrrrr r numpyrrmodelsr pipelinesr schedulersrutilsrrrrrrrr&r'r(r)peftr torchvisionrrrr"r7rPrLrFr,rertfloat32rrrrrr( GeneratorrrrrIrrrrrr#r!r]s %DDD ((&  **EEG.&)3)("J)3)h(+3# 3##3#||3# << 3# << 3# LL 3#!<<3# %3# 8ELL !8ELL#9 9:3#l3S%,,=N8O&V[UbUb-ehhootEHHOO7L&L M-"]#u||+,]69]IN]&Dehhoo1E,F4PSUXPX>',+/        %,,# $  ( 6S$  koEHHOO%667AFsELLGXAYdhB8 LFLFr#