bi*w"ddlZddlmZmZmZmZmZddlZddlZddl Z ddl m Z m Z ddl mZddlmZddlmZmZddlmZdd lmZmZmZmZdd lmZd d lmZmZer ddl m!cm"Z#d Z$ndZ$ejJe&Z'dZ(GddeeZ)y)N)CallableDictListOptionalUnion)T5EncoderModel T5Tokenizer)VaeImageProcessor)StableDiffusionLoraLoaderMixin)Kandinsky3UNetVQModel) DDPMScheduler) deprecateis_torch_xla_availableloggingreplace_example_docstring) randn_tensor)DiffusionPipelineImagePipelineOutputTFa? Examples: ```py >>> from diffusers import AutoPipelineForImage2Image >>> from diffusers.utils import load_image >>> import torch >>> pipe = AutoPipelineForImage2Image.from_pretrained( ... "kandinsky-community/kandinsky-3", variant="fp16", torch_dtype=torch.float16 ... ) >>> pipe.enable_model_cpu_offload() >>> prompt = "A painting of the inside of a subway train with tiny raccoons." >>> image = load_image( ... "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky3/t2i.png" ... ) >>> generator = torch.Generator(device="cpu").manual_seed(0) >>> image = pipe(prompt, image=image, strength=0.75, num_inference_steps=25, generator=generator).images[0] ``` c%eZdZdZgdZdedededede f fd Z d Z d Z e j d+dee j dee j dee j dee j fdZd,dZdZ d-dZedZedZedZe jeed d dddd d d d d d d dd d dgfdeeeefdee j ej>j>ee j eej>j>fd e d!e!d"e d#eeeeefd$ee!d%eee jDee jDfdee j dee j dee j dee j d&eed'e#d(ee$e!e!e%gd fd)eef d*Z&xZ'S).Kandinsky3Img2ImgPipelineztext_encoder->movq->unet->movq)latents prompt_embedsnegative_prompt_embedsnegative_attention_maskattention_mask tokenizer text_encoderunet schedulermovqcPt||j|||||t|ddr/dt |j j jdz znd}t|ddr |j j jnd}t||dd|_ y) N)rr r!r"r#r#rbicubic)vae_scale_factorvae_latent_channelsresample reducing_gap) super__init__register_modulesgetattrlenr#configblock_out_channelslatent_channelsr image_processor) selfrr r!r"r#movq_scale_factormovq_latent_channels __class__s u/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/pipelines/kandinsky3/pipeline_kandinsky3_img2img.pyr.z"Kandinsky3Img2ImgPipeline.__init__Bs  lQZae  T[[_agimSnA#dii&6&6&I&I"JQ"NOtuCJ4QWY]C^tyy//??de0. 4  ctt||z|}t||z d}|jj|d}|||z fS)Nr)minintmaxr" timesteps)r6num_inference_stepsstrengthdevice init_timestept_startr@s r: get_timestepsz'Kandinsky3Img2ImgPipeline.get_timestepsXsSC 3h >?ATU )M91=NN,,WX6 -777r;c|rYtj||dk(||dk(<|jdjdz}|ddd|f}|ddd|f}||fS)Nrr%)torch zeros_likesumr?)r6 embeddingsr cut_contextmax_seq_lengths r:_process_embedsz)Kandinsky3Img2ImgPipeline._process_embedsasy .3.>.>z.\]J]?^._J~* ++//3779A=N#A$67J+A,>?N>))r;Tr%Nrrrrc |<|:t|t|ur$tdt|dt|d| |j}|t|trd} n-|t|t r t |} n|jd} d} ||j|d| d d } | jj|}| jj|} |j|| }|d}|j|| |\}} || jd z}|j|jj}nd}|j||}|j\}}}|j!d|d}|j#||z|d}| j!|d} |r'|$|dg| z}nEt|tr|g}n1| t |k7r!t%d|dt |d|d| d |}||j|ddd d d }|jj|}|jj|} |j|| }|d}|ddd|jdf}| ddd|jdf} || jd z}n*t'j(|}t'j(| } |rw|jd}|j||}|j|jk7r@|j!d|d}|j#| |z|d}| j!|d} nd}d} ||| | fS)aY Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*): prompt to be encoded device: (`torch.device`, *optional*): torch device to place the resulting embeddings on num_images_per_prompt (`int`, *optional*, defaults to 1): number of images that should be generated per prompt do_classifier_free_guidance (`bool`, *optional*, defaults to `True`): whether to use classifier free guidance or not negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds`. instead. If not defined, one has to pass `negative_prompt_embeds`. instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. attention_mask (`torch.Tensor`, *optional*): Pre-generated attention mask. Must provide if passing `prompt_embeds` directly. negative_attention_mask (`torch.Tensor`, *optional*): Pre-generated negative attention mask. Must provide if passing `negative_prompt_embeds` directly. Nz?`negative_prompt` should be the same type to `prompt`, but got z != .r%r max_lengthTpt)paddingrS truncationreturn_tensors)rrdtyperCrHz`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.)rUrSrVreturn_attention_maskrW)type TypeError_execution_device isinstancestrlistr1shaper input_idstorr rO unsqueezerYrepeatview ValueErrorrIrJ)r6promptdo_classifier_free_guidancenum_images_per_promptrCnegative_promptrr _cut_contextrr batch_sizerS text_inputstext_input_idsrYbs_embedseq_len_ uncond_tokens uncond_inputs r: encode_promptz'Kandinsky3Img2ImgPipeline.encode_promptjs T  /"=F|4#88UVZ[jVkUlmV ~Q( >++F  *VS"9J  Jvt$<VJ&,,Q/J  ..$%# )K)2255f=N(77::6BN ---.M*!,M,0,@,@P^`l,m )M>)N,D,DQ,GGM    (%%++EE%((uV(D ,22'1%,,Q0EqI %**86K+KWVXY '../DaH &+A+I&!#z 1 OS1!0 1 s?33 )/)::J3K_J`ax/ |<33 !0 *#~~!("#*.#' .  ".!7!7!:!:6!B*6*E*E*H*H*P')-):):"#:*;*&*@)B&)?C[]EXEXYZE[C[@[)\&*A!E]}GZGZ[\G]E]B]*^')?BYBcBcdeBf)f&*/)9)9-)H&*/*:*:>*J' &,2215G%;%>%>USY%>%Z "%++}/B/BB)?)F)FqJ_ab)c&)?)D)DZRgEgiprt)u&*A*H*HI^`a*b' &* "&* #4nF]]]r;c t|tjtjjt fst dt||j||}||z}|jddk(r|}n t|t r)t||k7rt dt|d|dt|t rkt|D cgc]?} |jj|| | dzjj|| A}} tj |d }n4|jj|jj|}|jj"j$|z}tj |gd }|j} t'| ||| } |j(j+|| |}|} | Scc} w) NzK`image` has to be of type `torch.Tensor`, `PIL.Image.Image` or list but is )rCrYr%r'z/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.rdim) generatorrCrY)r_rITensorPILImagerarhr\rdrbr1ranger#encode latent_distsamplecatr2scaling_factorrr" add_noise) r6imagetimesteprnrkrYrCrz init_latentsirbnoisers r:prepare_latentsz)Kandinsky3Img2ImgPipeline.prepare_latentss%%,, !FG]^bch^i]jk e4"77 ;;q>Q  L)T*s9~/K Ec)nEUV *|+km It,afgqar \]DII$$U1q1u%56BBII)TU,W    %yy1= #yy//6BBII)T 99++::\ILyy,Q7 ""UieT~~// eXN ' sAGcVdttj|jjj j v}i}|r||d<dttj|jjj j v}|r||d<|S)Netarz)setinspect signaturer"step parameterskeys)r6rzr accepts_etaextra_step_kwargsaccepts_generators r:prepare_extra_step_kwargsz3Kandinsky3Img2ImgPipeline.prepare_extra_step_kwargs+s s7#4#4T^^5H5H#I#T#T#Y#Y#[\\  '* e $(3w/@/@ATAT/U/`/`/e/e/g+hh -6 k *  r;c |0t|tr|dkrtd|dt|d|Lt fd|Ds8tdj d|D cgc]} | j vs| c} ||td|d |d | | td |7t|t s't|tstd t|||td |d|d |A|?|j|jk7r&td|jd|jd| | td|G|E|jdd|jk7r)td|jddd|jd| | td|I|F|jdd|jk7r)td|jddd|jdyyycc} w)Nrz5`callback_steps` has to be a positive integer but is z of type rQc3:K|]}|jvywN_callback_tensor_inputs.0kr6s r: z9Kandinsky3Img2ImgPipeline.check_inputs..M#F 23A-- -F 2`callback_on_step_end_tensor_inputs` has to be in , but found zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is z'Cannot forward both `negative_prompt`: z and `negative_prompt_embeds`: zu`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but got: `prompt_embeds` z != `negative_prompt_embeds` zLPlease provide `negative_attention_mask` along with `negative_prompt_embeds`rz`negative_prompt_embeds` and `negative_attention_mask` must have the same batch_size and token length when passed directly, but got: `negative_prompt_embeds` z != `negative_attention_mask` z:Please provide `attention_mask` along with `prompt_embeds`z`prompt_embeds` and `attention_mask` must have the same batch_size and token length when passed directly, but got: `prompt_embeds` z != `attention_mask` ) r_r>rhr\allrr`rarb) r6ricallback_stepsrlrr"callback_on_step_end_tensor_inputsrrrs ` r: check_inputsz&Kandinsky3Img2ImgPipeline.check_inputs<s  %z.#/NR`deReGGWX(),  . 9#F 7YF C DTEaEaDbbn|^pHvwbckokGkGbGpqpHoIJ   -";08N}o^00 ^ 5w  FC)@TZ\`IaQRVW]R^Q_`a a  &+A+M9/9JK*++]_   $)?)K""&<&B&BB --:-@-@,AB.445Q8 " -2I2Qkl l ! -2I2U%++BQ/3J3P3PP 66L6R6RSUTU6V5WX/556a9  $)?YZ Z  $)C""2A&.*>*>> --:-@-@!-D,EF&,,-Q0?*D $WpHs !G,5G,c|jSr_guidance_scaler6s r:guidance_scalez(Kandinsky3Img2ImgPipeline.guidance_scales###r;c |jdkDS)Nr%rrs r:rjz5Kandinsky3Img2ImgPipeline.do_classifier_free_guidances##a''r;c|jSr)_num_timestepsrs r: num_timestepsz'Kandinsky3Img2ImgPipeline.num_timestepss"""r;g333333?g@pilrrirrBrArrlrkrz output_type return_dictcallback_on_step_endrc |jdd}|jdd}| tddd| tddd|Ltfd|Ds8tdjd |Dcgc]}|jvs|c}d }j |||| | || | |_|t|trd }n-|t|tr t|}n| jd }j}j|j|||| | || | \} } } } jr|jW|,||zd k(r$|tYj.dd z}%||%||tZst]j^| d k(s=j4ja|d !d"}j&jc|| }n|}je|s |fcdddStg|#cdddScc}wcc}wcc}w#1swYyxYw)$a/ Function invoked when calling the pipeline for generation. Args: prompt (`str` or `List[str]`, *optional*): The prompt or prompts to guide the image generation. If not defined, one has to pass `prompt_embeds`. instead. image (`torch.Tensor`, `PIL.Image.Image`, `np.ndarray`, `List[torch.Tensor]`, `List[PIL.Image.Image]`, or `List[np.ndarray]`): `Image`, or tensor representing an image batch, that will be used as the starting point for the process. strength (`float`, *optional*, defaults to 0.8): Indicates extent to transform the reference `image`. Must be between 0 and 1. `image` is used as a starting point and more noise is added the higher the `strength`. The number of denoising steps depends on the amount of noise initially added. When `strength` is 1, added noise is maximum and the denoising process runs for the full number of iterations specified in `num_inference_steps`. A value of 1 essentially ignores `image`. num_inference_steps (`int`, *optional*, defaults to 50): The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. guidance_scale (`float`, *optional*, defaults to 3.0): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://huggingface.co/papers/2207.12598). `guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://huggingface.co/papers/2205.11487). Guidance scale is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality. negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. attention_mask (`torch.Tensor`, *optional*): Pre-generated attention mask. Must provide if passing `prompt_embeds` directly. negative_attention_mask (`torch.Tensor`, *optional*): Pre-generated negative attention mask. Must provide if passing `negative_prompt_embeds` directly. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generate image. Choose between [PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.stable_diffusion.IFPipelineOutput`] instead of a plain tuple. callback_on_step_end (`Callable`, *optional*): A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by `callback_on_step_end_tensor_inputs`. callback_on_step_end_tensor_inputs (`List`, *optional*): The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. Examples: Returns: [`~pipelines.ImagePipelineOutput`] or `tuple` callbackNrz1.0.0zhPassing `callback` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`znPassing `callback_steps` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`c3:K|]}|jvywrrrs r:rz5Kandinsky3Img2ImgPipeline.__call__..rrrrTr%r)rkrCrlrrrmrrc3K|]8}t|tjjtjf:ywr)r_r|r}rIr{)rrs r:rz5Kandinsky3Img2ImgPipeline.__call__..*s(Qa:a#))//5<AzInput is in incorrect format: z:. Currently, we only support PIL image and pytorch tensorrxrX)rCrtext_encoder_offload_hook)totalr)encoder_hidden_statesencoder_attention_maskg?)rzrrrrorderlatent)force_not_quantizer)images)4poprrrhrrrr_r`rar1rbr^rvrjrIrboolr\r5 preprocessrdrYr" set_timestepsrFr#rrepeat_interleaverfrhasattrroffloadrr progress_bar enumerater!chunkr prev_samplelocalsupdater0 XLA_AVAILABLExm mark_stepdecode postprocessmaybe_free_model_hooksr)&r6rirrBrArrlrkrzrrrrrrrrkwargsrrrrMrnrCrr@rlatent_timestepnum_warmup_stepsrtlatent_model_input noise_prednoise_pred_uncondnoise_pred_textcallback_kwargscallback_outputsstep_idxs&` r:__call__z"Kandinsky3Img2ImgPipeline.__call__s+p::j$/$4d;   z   %  A  . 9#F 7YF C DTEaEaDbbn|^pHvwbckokGkGbGpqpHoIJ        " .  #  .  *VS"9J  Jvt$<VJ&,,Q/J''Z^YkYk   , ,"7+'#9$)$;Zl Z V -~?V  + +!II'=}&MNM"YY(?'PQVVXN%&GEQ5QQ051Ia$q'1I0JKEF  uM!4//::1=MSTU}226B $$%8$H)-););,?$..BVBV,VV!)n   %8  99 5\!),* #1AEAaAaUYYy1}%=gn""YY&*7+9 '   339C9I9I!9L6%"03"6/!IN]nLn!nJ..--' . + (3&(O?9-3Xa[*9';D!Q'X$.229gFG$4$8$8-$XM-=-A-ABZ\r-s*%5%9%9:JN%[N.>.B.BC\^u.v+I**A9I/IqSTuX\XfXfXlXlNlpqNq '')+N0Ba0G#$(K#K 1g6 LLNU* #Z(* ((T(J8T,,88 L  ' ' )xo9 59 5r'e4s9 59 5KpH`2JN$9 59 5s21U#U#"U( "U-.F%U2A/U2 U22U;) Tr%NNNNFNNr)NNNNNN)(__name__ __module__ __qualname__model_cpu_offload_seqrr rr rrr.rFrOrIno_gradrr{rvrrrpropertyrrjrrEXAMPLE_DOC_STRINGrr`rr|r}floatr> Generatorrrrr __classcell__)r9s@r:rr8s<  %   !    ,8*U]]_%)049=15:>S^  -S^!) 6S^!.S^"*%,,!7S^S^j(V!*#+/ $FP$$((##U]]_12)-ae#% #;?/0MQ049=15:>%* KO9B #h5c49n%h5U\\399??D4FSYY__H]]^h5 h5 ! h5  h5"%T#Y"78h5 (}h5E%//43H"HIJh5  -h5!) 6h5!.h5"*%,,!7h5c]h5h5 'xc40@$0F'GH!h5"-1I#h53h5r;r)*rtypingrrrrrr| PIL.ImagerI transformersrr r5r loadersr modelsr r schedulersrutilsrrrrutils.torch_utilsrpipeline_utilsrrtorch_xla.core.xla_modelcore xla_modelrr get_loggerrloggerrrr;r:rs|88  405-' .C))MM   H %.B 5 13QB 5r;