bi'ddlZddlmZmZmZmZddlZddlmZddl m Z m Z m Z m Z ddlmZej eZGddZy)N)DictListTupleUnion) AttentionAttentionProcessor"PAGCFGIdentitySelfAttnProcessor2_0PAGIdentitySelfAttnProcessor2_0)loggingceZdZdZdZdZ ddZdZee ffde e e e fde eeffdZed efd Zed efd Zed efd Zed efd Zed ee effdZy)PAGMixinzZMixin class for [Pertubed Attention Guidance](https://huggingface.co/papers/2403.17377v1).c|j}| td|r|dn|d}t|dr |j}n |j}dt j dtfd}d }|D]}g} |jD]X\} } || stj|| &||| r0tjd | | j| Zt| dk(rtd || D] } || _ y) zA Set the attention processor for the PAG layers. NzNo PAG attention processors have been set. Set the attention processors by calling `set_pag_applied_layers` and passing the relevant parameters.runetmodulereturnc@t|txr |j S)zY Check if the module is self-attention module based on its name. ) isinstanceris_cross_attention)rs \/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/pipelines/pag/pag_utils.py is_self_attnz6PAGMixin._set_pag_attn_processor..is_self_attn5sfi0R9R9R5R Rc|jdd}|jdd}|jxr|jxr||k(S)N.)split isnumeric)layer_idnames ris_fake_integral_matchz@PAGMixin._set_pag_attn_processor..is_fake_integral_match;sL~~c*2.H::c?2&D%%'QDNN,<QTAQ QrzApplying PAG to layer: z6Cannot find PAG layer to set attention processor for: )_pag_attn_processors ValueErrorhasattrr transformernnModulebool named_modulesresearchloggerdebugappendlen processor) selfpag_applied_layersdo_classifier_free_guidancepag_attn_processors pag_attn_procmodelrr!rtarget_modulesr rs r_set_pag_attn_processorz PAGMixin._set_pag_attn_processor$s4#77  &c 3N+A.SfghSi 4 #yyE#//E S St S  R + 1HN % 3 3 5 2 f!( (D1=28TBLL#:4&!AB"))&1 2>"a' #YZbYc!dee( 1#0  1+ 1rc|jr(|j|jd|z zz }|dkrd}|S|jS)z\ Get the scale factor for the perturbed attention guidance at timestep `t`. ir)do_pag_adaptive_scaling pag_scalepag_adaptive_scale)r1t signal_scales r_get_pag_scalezPAGMixin._get_pag_scaleXsG  ' '>>D,C,Ctax,PPLa  >> !rc|j|}|r*|jd\}}} ||||z zz||| z zz}n|jd\}} |||| z zz}|r||fS|S)a Apply perturbed attention guidance to the noise prediction. Args: noise_pred (torch.Tensor): The noise prediction tensor. do_classifier_free_guidance (bool): Whether to apply classifier-free guidance. guidance_scale (float): The scale factor for the guidance term. t (int): The current time step. return_pred_text (bool): Whether to return the text noise prediction. Returns: Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]: The updated noise prediction tensor after applying perturbed attention guidance and the text noise prediction. r)r?chunk) r1 noise_predr3guidance_scaler=return_pred_textr;noise_pred_uncondnoise_pred_textnoise_pred_perturbs r#_apply_perturbed_attention_guidancez,PAGMixin._apply_perturbed_attention_guidancees"''* &EOEUEUVWEX B 0B! O6G$GHI1CCDE  3=2B2B12E /O/(9J\8\+]]J . .rcrtj|gdzd}|rtj||gd}|S)a Prepares the perturbed attention guidance for the PAG model. Args: cond (torch.Tensor): The conditional input tensor. uncond (torch.Tensor): The unconditional input tensor. do_classifier_free_guidance (bool): Flag indicating whether to perform classifier-free guidance. Returns: torch.Tensor: The prepared perturbed attention guidance tensor. rAr)dim)torchcat)r1conduncondr3s r%_prepare_perturbed_attention_guidancez.PAGMixin._prepare_perturbed_attention_guidances6yy$!+ &99fd^3D rr2r4cNt|dsd|_t|ts|g}|)t|trt |dk7r t dtt |D]/}t||trt dt||||_ ||_y)a Set the self-attention layers to apply PAG. Raise ValueError if the input is invalid. Args: pag_applied_layers (`str` or `List[str]`): One or more strings identifying the layer names, or a simple regex for matching multiple layers, where PAG is to be applied. A few ways of expected usage are as follows: - Single layers specified as - "blocks.{layer_index}" - Multiple layers as a list - ["blocks.{layers_index_1}", "blocks.{layer_index_2}", ...] - Multiple layers as a block name - "mid" - Multiple layers as regex - "blocks.({layer_index_1}|{layer_index_2})" pag_attn_processors: (`Tuple[AttentionProcessor, AttentionProcessor]`, defaults to `(PAGCFGIdentitySelfAttnProcessor2_0(), PAGIdentitySelfAttnProcessor2_0())`): A tuple of two attention processors. The first attention processor is for PAG with Classifier-free guidance enabled (conditional and unconditional). The second attention processor is for PAG with CFG disabled (unconditional only). r"NrAz,Expected a tuple of two attention processorsz:Expected either a string or a list of string but got type ) r$r"rlisttupler/r#rangestrtyper2)r1r2r4is rset_pag_applied_layerszPAGMixin.set_pag_applied_layerss4t34(,D %,d3"4!5   *159SAT=UYZ=Z !OPPs-./ A03S9 PQUVhijVkQlPmn  #5$7!rrc|jS)z:Get the scale factor for the perturbed attention guidance.) _pag_scaler1s rr;zPAGMixin.pag_scalesrc|jS)zCGet the adaptive scale factor for the perturbed attention guidance.)_pag_adaptive_scaler[s rr<zPAGMixin.pag_adaptive_scales'''rcv|jdkDxr)|jdkDxrt|jdkDS)zNCheck if the adaptive scaling is enabled for the perturbed attention guidance.r)r]rZr/r2r[s rr:z PAGMixin.do_pag_adaptive_scalings9''!+h!0ChDLcLcHdghHhhrcT|jdkDxrt|jdkDS)z5Check if the perturbed attention guidance is enabled.r)rZr/r2r[s rdo_perturbed_attention_guidancez(PAGMixin.do_perturbed_attention_guidances("Gs4+B+B'Ca'GGrc\|jiS|jDchc]}|j}}i}t|dr |j}n$t|dr |j}n t d|j jD]\}}|j|vs|||<|Scc}w)z Returns: `dict` of PAG attention processors: A dictionary contains all PAG attention processors used in the model with the key as the name of the layer. rr%zNo denoiser module found.)r" __class__r$rr%r#attn_processorsitems)r1xvalid_attn_processors processorsdenoiser_moduler procs rr4zPAGMixin.pag_attn_processorss  $ $ ,I6:6O6O P P P  4 "iiO T= )"..O89 9)99??A (JD$~~!66#' 4  (!!QsB)N)F)__name__ __module__ __qualname____doc__r8r?rIrPr r rrUrrr rXpropertyfloatr;r<r(r:r`rr4rrrr!se21h "\a@. / 0 + -N *8!#tCy.1*8##57I#IJ*8X5(E((iiiHHHT#/A*A%Brr)r*typingrrrrrLtorch.nnr&models.attention_processorrr r r utilsr get_loggerrjr,rrprrrvsE ++     H %RRr