bi72ddlZddlmZmZmZmZmZmZmZddl Z ddl m Z ddl m Z mZddlmZddlmZmZerdd lmZGd d eZy) N) TYPE_CHECKINGAnyDictListOptionalTupleUnion)register_to_config) HookRegistryLayerSkipConfig)_apply_layer_skip_hook) BaseGuidancerescale_noise_cfg) BlockStateceZdZdZgdZe ddededededeee e e fd ee e e e e effd ed ed ed effd Zdej$j&ddfdZdej$j&ddfdZ d dddee e ee ee e fffde dfdZ d!dej0deej0deej0dej0fdZedefdZede fdZdefdZdefdZxZS)"SkipLayerGuidancea Skip Layer Guidance (SLG): https://github.com/Stability-AI/sd3.5 Spatio-Temporal Guidance (STG): https://huggingface.co/papers/2411.18664 SLG was introduced by StabilityAI for improving structure and anotomy coherence in generated images. It works by skipping the forward pass of specified transformer blocks during the denoising process on an additional conditional batch of data, apart from the conditional and unconditional batches already used in CFG ([~guiders.classifier_free_guidance.ClassifierFreeGuidance]), and then scaling and shifting the CFG predictions based on the difference between conditional without skipping and conditional with skipping predictions. The intution behind SLG can be thought of as moving the CFG predicted distribution estimates further away from worse versions of the conditional distribution estimates (because skipping layers is equivalent to using a worse version of the model for the conditional prediction). STG is an improvement and follow-up work combining ideas from SLG, PAG and similar techniques for improving generation quality in video diffusion models. Additional reading: - [Guiding a Diffusion Model with a Bad Version of Itself](https://huggingface.co/papers/2406.02507) The values for `skip_layer_guidance_scale`, `skip_layer_guidance_start`, and `skip_layer_guidance_stop` are defaulted to the recommendations by StabilityAI for Stable Diffusion 3.5 Medium. Args: guidance_scale (`float`, defaults to `7.5`): The scale parameter for classifier-free guidance. Higher values result in stronger conditioning on the text prompt, while lower values allow for more freedom in generation. Higher values may lead to saturation and deterioration of image quality. skip_layer_guidance_scale (`float`, defaults to `2.8`): The scale parameter for skip layer guidance. Anatomy and structure coherence may improve with higher values, but it may also lead to overexposure and saturation. skip_layer_guidance_start (`float`, defaults to `0.01`): The fraction of the total number of denoising steps after which skip layer guidance starts. skip_layer_guidance_stop (`float`, defaults to `0.2`): The fraction of the total number of denoising steps after which skip layer guidance stops. skip_layer_guidance_layers (`int` or `List[int]`, *optional*): The layer indices to apply skip layer guidance to. Can be a single integer or a list of integers. If not provided, `skip_layer_config` must be provided. The recommended values are `[7, 8, 9]` for Stable Diffusion 3.5 Medium. skip_layer_config (`LayerSkipConfig` or `List[LayerSkipConfig]`, *optional*): The configuration for the skip layer guidance. Can be a single `LayerSkipConfig` or a list of `LayerSkipConfig`. If not provided, `skip_layer_guidance_layers` must be provided. guidance_rescale (`float`, defaults to `0.0`): The rescale factor applied to the noise predictions. This is used to improve image quality and fix overexposure. Based on Section 3.4 from [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://huggingface.co/papers/2305.08891). use_original_formulation (`bool`, defaults to `False`): Whether to use the original formulation of classifier-free guidance as proposed in the paper. By default, we use the diffusers-native implementation that has been in the codebase for a long time. See [~guiders.classifier_free_guidance.ClassifierFreeGuidance] for more details. start (`float`, defaults to `0.01`): The fraction of the total number of denoising steps after which guidance starts. stop (`float`, defaults to `0.2`): The fraction of the total number of denoising steps after which guidance stops.  pred_cond pred_uncondpred_cond_skipNguidance_scaleskip_layer_guidance_scaleskip_layer_guidance_startskip_layer_guidance_stopskip_layer_guidance_layersskip_layer_configguidance_rescaleuse_original_formulationstartstopc t|| | ||_||_||_||_||_||_d|cxkrdksntd|d||cxkrdksntd|d| | td| | td|Ut|tr|g}t|tstdt|d|D cgc]} t| d }} t|trtj|}t|tr|g}t|tstd t|dtt!t#|dtr"|D cgc]} tj| }} ||_t't)|j$D cgc]} d |  c} |_ycc} wcc} wcc} w) N?zHExpected `skip_layer_guidance_start` to be between 0.0 and 1.0, but got .zGExpected `skip_layer_guidance_stop` to be between 0.0 and 1.0, but got zjEither `skip_layer_guidance_layers` or `skip_layer_config` must be provided to enable Skip Layer Guidance.zPOnly one of `skip_layer_guidance_layers` or `skip_layer_config` can be provided.zNExpected `skip_layer_guidance_layers` to be an int or a list of ints, but got auto)fqnz[Expected `skip_layer_config` to be a LayerSkipConfig or a list of LayerSkipConfig, but got SkipLayerGuidance_)super__init__rrrrrr ValueError isinstanceintlisttyper dict from_dictnextiterrrangelen_skip_layer_hook_names)selfrrrrrrrr r!r"layerconfigi __class__s `/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/guiders/skip_layer_guidance.pyr+zSkipLayerGuidance.__init__Zs: %,)B&)B&(@% 0(@%0636Z[tZuuvw *-ELLYZrYsstu  & -2C2K|  & 16G6Sop p % 14c:.H-I*8$? deikEfFeGGHIRl lF!C l  l ' . / 9 9:K L  ' 9!2 3 +T2mnrtEoFnGGHI T"34d;T BQb cv!:!:6!B c  c!2INsSWSiSiOjIk&lA);A3'?&l#!!m!d'msG 0G5 Gdenoiserreturnc|xjdz c_|jrT|jrG|jdkDr7t|j|j D]\}}t |||yyyy)Nr)name)_count_prepared_is_slg_enabledis_conditionalzipr7rr)r8r>rAr:s r=prepare_modelsz SkipLayerGuidance.prepare_modelssu !    !d&9&9d>R>RUV>V #D$?$?AWAW X D f&xdC D?W&9 !c|jrW|jrJ|jdkDr:tj|}|j D]}|j |dyyyy)NrT)recurse)rCrDrBr check_if_exists_or_initializer7 remove_hook)r8r>registry hook_names r=cleanup_modelsz SkipLayerGuidance.cleanup_modelssh    !d&9&9d>R>RUV>V#AA(KH!88 > $$Y$= >?W&9 !rGdatar input_fieldscD| |j}|jdk(rdg}dg}n4|jdk(rddg}|jrddgnddg}ngd}gd}g}t|jD]-}|j ||||||}|j |/|S) Nrrrr rr)rrrr) _input_fieldsnum_conditions_is_cfg_enabledr5_prepare_batchappend)r8rOrP tuple_indicesinput_predictions data_batchesr; data_batchs r=prepare_inputsz SkipLayerGuidance.prepare_inputss  --L   ! #CM!,   A %FM040D0D0Fm,[ZjLk &M N  t**+ ,A,,\4qAQSdefSghJ    + ,rGrrrcd}|js|js|}n|js(||z }|jr|n|}||j|zz}nt|js(||z }|jr|n|}||j|zz}n<||z }||z }|jr|n|}||j|zz|j|zz}|j dkDrt |||j }|ifS)Nr$)rTrCr rrrr)r8rrrpredshift shift_skips r=forwardzSkipLayerGuidance.forwards  ##%d.B.B.DD%%'.E $ = =9>D$885@@D%%' +E $ = =9;D$--55D +E"^3J $ = =9;D$--558V8VYc8ccD  3 &$T9d6K6KLDRxrGcB|jdk(xs|jdk(S)Nr)rB)r8s r=rDz SkipLayerGuidance.is_conditionals###q(ED,@,@A,EErGc^d}|jr|dz }|jr|dz }|S)Nr)rTrC)r8rSs r=rSz SkipLayerGuidance.num_conditionss9    ! a N    ! a NrGc|jsyd}|j^t|j|jz}t|j|jz}||j cxkxr|knc}d}|j r!tj|jd}n tj|jd}|xr| S)NFTr$r%) _enabled_num_inference_stepsr._start_stop_stepr mathiscloser)r8is_within_rangeskip_start_stepskip_stop_stepis_closes r=rTz!SkipLayerGuidance._is_cfg_enableds}}  $ $ 0!$++0I0I"IJO d.G.G!GHN-LnLO  ( (||D$7$7=H||D$7$7=H/x</rGcB|jsyd}|j^t|j|jz}t|j|jz}||j cxkxr|knc}t j|jd}|xr| S)NFTr$) rerfr.rrrirjrkr)r8rlrmrnis_zeros r=rCz!SkipLayerGuidance._is_slg_enableds}}  $ $ 0!$"@"@4C\C\"\]O !>!>AZAZ!Z[N- K^KO,,t==sC.w;.rG) g@gffffff@g{Gz?g?NNr$Fr$r%)N)NN) __name__ __module__ __qualname____doc___input_predictionsr floatrr r.rr rstrrboolr+torchnnModulerFrNrr[Tensorr`propertyrDrSrTrC __classcell__)r<s@r=rrs7rH!$+.+/*-FJ[_"%).=m=m$)=m$) =m #( =m %-U3S >-B$C =m!$2GcSVh!WX=m =m#'=m=m=m=m~DuxxD4D >uxx>4>dh 08c5eTWY\T\oI]C^>^9_0` l 4/315 <<ell+!.   :FFF00$ / /rGr)rjtypingrrrrrrr rzconfiguration_utilsr hooksr r hooks.layer_skipr guider_utilsrr"modular_pipelines.modular_pipelinerrrGr=rs8 III 4159?h/ h/rG