biEddlmZmZmZmZddlZddlmZddlm Z m Z ddl m Z m Z ddlmZmZmZddlmZd d lmZmZd d lmZer ddlmcmZd ZndZej<eZ dZ!ddZ"GddeZ#y))CallableListOptionalUnionN)XLMRobertaTokenizer)UNet2DConditionModelVQModel) DDIMScheduler DDPMScheduler)is_torch_xla_availableloggingreplace_example_docstring) randn_tensor)DiffusionPipelineImagePipelineOutput)MultilingualCLIPTFav Examples: ```py >>> from diffusers import KandinskyPipeline, KandinskyPriorPipeline >>> import torch >>> pipe_prior = KandinskyPriorPipeline.from_pretrained("kandinsky-community/Kandinsky-2-1-prior") >>> pipe_prior.to("cuda") >>> prompt = "red cat, 4k photo" >>> out = pipe_prior(prompt) >>> image_emb = out.image_embeds >>> negative_image_emb = out.negative_image_embeds >>> pipe = KandinskyPipeline.from_pretrained("kandinsky-community/kandinsky-2-1") >>> pipe.to("cuda") >>> image = pipe( ... prompt, ... image_embeds=image_emb, ... negative_image_embeds=negative_image_emb, ... height=768, ... width=768, ... num_inference_steps=100, ... ).images >>> image[0].save("cat.png") ``` cv||dzz}||dzzdk7r|dz }||dzz}||dzzdk7r|dz }||z||zfS)Nrrr)hw scale_factornew_hnew_ws k/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/pipelines/kandinsky/pipeline_kandinsky.py get_new_h_wrKsg q E<?a   q E<?a   < !5 55c!eZdZdZdZdedededee e fde f fd Z d Z dd Zej ee dd eeeefd eej*eej*fdeej*eej*fdeeeeefdedededededeeej2eej2fdeej*deedeeeeej*gd fdedefdZxZS)KandinskyPipelinea1 Pipeline for text-to-image generation using Kandinsky This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) Args: text_encoder ([`MultilingualCLIP`]): Frozen text-encoder. tokenizer ([`XLMRobertaTokenizer`]): Tokenizer of class scheduler (Union[`DDIMScheduler`,`DDPMScheduler`]): A scheduler to be used in combination with `unet` to generate image latents. unet ([`UNet2DConditionModel`]): Conditional U-Net architecture to denoise the image embedding. movq ([`VQModel`]): MoVQ Decoder to generate the image from the latents. ztext_encoder->unet->movq text_encoder tokenizerunet schedulermovqct||j|||||dt|jj j dz z|_y)N)r"r#r$r%r&rr)super__init__register_moduleslenr&configblock_out_channelsmovq_scale_factor)selfr"r#r$r%r& __class__s rr)zKandinskyPipeline.__init__ks\  %  "#s499+;+;+N+N'ORS'S!Trc|t||||}n;|j|k7rtd|jd||j|}||jz}|S)N) generatordevicedtypezUnexpected latents shape, got z , expected )rshape ValueErrortoinit_noise_sigma)r/r5r4r3r2latentsr%s rprepare_latentsz!KandinskyPipeline.prepare_latentssg ?"5IfTYZG}}% #A'--P[\a[b!cddjj(GI666rNc ht|tr t|nd}|j|dddddd}|j}|j|ddj} | j d |j d k\rt j|| sj|jj| dd|jjdz d f} tjd |jjd | |j|}|jj|} |j|| \} } | j|d } | j|d } | j|d } |r|dg|z}nt!|t!|ur$t#dt!|dt!|dt|t$r|g}n1|t|k7r!t'd|dt|d|d|d |}|j|dddddd}|jj|}|jj|}|j|| \}}|j d}|j)d|}|j+||z|}|j d}|j)d|d}|j+||z|d }|j|d }t j,|| g} t j,|| g} t j,|| g} | | | fS)Nr max_lengthTMpt)padding truncationr<return_attention_maskadd_special_tokensreturn_tensorslongest)r?rCz\The following part of your input was truncated because CLIP can only handle sequences up to z tokens: ) input_idsattention_maskrdimz?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.)r?r<r@rArBrC) isinstancelistr+r#rFr5torchequal batch_decodemodel_max_lengthloggerwarningr7rGr"repeat_interleavetype TypeErrorstrr6repeatviewcat)r/promptr3num_images_per_promptdo_classifier_free_guidancenegative_prompt batch_size text_inputstext_input_idsuntruncated_ids removed_text text_mask prompt_embedstext_encoder_hidden_states uncond_tokens uncond_inputuncond_text_input_idsuncond_text_masknegative_prompt_embeds!uncond_text_encoder_hidden_statesseq_lens r_encode_promptz KandinskyPipeline._encode_prompts%/vt$>66q$..JiJilmJmprJrGr7stL NNNN334Il^M  (**62..11&9 484E4E$Y5F5 1 1&778MST7U %?%Q%QRgmn%Q%o"//0E1/M &&!#z 1 fT/%::UVZ[jVkUlmV ~Q(OS1!0 1 s?33 )/)::J3K_J`ax/ |<33 !0 >>$&*#'#*L%1$:$:$=$=f$E !+::==fE HLHYHY/@PIZI E "$E -2215G%;%B%B1F[%\ "%;%@%@NcAcel%m "7==a@G0Q0X0XYZ\qst0u -0Q0V0V22GR1 - 0AABW]^A_ "II'=}&MNM).4UWq3r)s & #3Y"?@I8)CCrr[ image_embedsnegative_image_embedsr^heightwidthnum_inference_stepsguidance_scaler\r2r9 output_typecallbackcallback_steps return_dictcPt|trd}n3t|tr t|}nt dt ||j }|| z}|dkD}|j||| ||\}}}t|trtj|d}t|trtj|d}|rZ|j| d}|j| d}tj||gdj|j|}|jj|||jj}|j j"j$}t'|||j(\}}|j+||||f|j|| | |j} t-|j/|D]\}}|rtj| gdzn| }||d }|j!||||d d}|ro|j1| j2dd\}}|j5d\}}|j5d\}} ||||z zz}tj|| gd}t7|jj"d r"|jj"j8d vs#|j1| j2dd\}}|jj;||| | j<} | ,||zdk(r$|t?|jddz}!| |!|| t@swtCjD|jFjI| dd}"|jK| dvrt d| | dvrX|"dzdz}"|"jMdd}"|"jOjQddddjSjU}"| dk(r|jW|"}"|s|"fStY|"S)a Function invoked when calling the pipeline for generation. Args: prompt (`str` or `List[str]`): The prompt or prompts to guide the image generation. image_embeds (`torch.Tensor` or `List[torch.Tensor]`): The clip image embeddings for text prompt, that will be used to condition the image generation. negative_image_embeds (`torch.Tensor` or `List[torch.Tensor]`): The clip image embeddings for negative text prompt, will be used to condition the image generation. negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). height (`int`, *optional*, defaults to 512): The height in pixels of the generated image. width (`int`, *optional*, defaults to 512): The width in pixels of the generated image. num_inference_steps (`int`, *optional*, defaults to 100): The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. guidance_scale (`float`, *optional*, defaults to 4.0): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://huggingface.co/papers/2207.12598). `guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://huggingface.co/papers/2205.11487). Guidance scale is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality. num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will ge generated by sampling using the supplied random `generator`. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generate image. Choose between: `"pil"` (`PIL.Image.Image`), `"np"` (`np.array`) or `"pt"` (`torch.Tensor`). callback (`Callable`, *optional*): A function that calls every `callback_steps` steps during inference. The function is called with the following arguments: `callback(step: int, timestep: int, latents: torch.Tensor)`. callback_steps (`int`, *optional*, defaults to 1): The frequency at which the `callback` function is called. If not specified, the callback is called at every step. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.ImagePipelineOutput`] instead of a plain tuple. Examples: Returns: [`~pipelines.ImagePipelineOutput`] or `tuple` rz2`prompt` has to be of type `str` or `list` but is g?rrH)r4r3)r3r) text_embedsroF)sampletimestepencoder_hidden_statesadded_cond_kwargsrx variance_type)learned learned_range)r2orderT)force_not_quantizer{)r>nppilzIOnly the output types `pt`, `pil` and `np` are supported not output_type=)rrg?rr)images)-rLrWrMr+r6rU_execution_devicernrNrZrTr7r4r% set_timesteps timestepsr$r, in_channelsrr.r: enumerate progress_barsplitr5chunkhasattrrstep prev_samplegetattr XLA_AVAILABLExm mark_stepr&decodemaybe_free_model_hooksclampcpupermutefloatnumpy numpy_to_pilr)#r/r[rorpr^rqrrrsrtr\r2r9rurvrwrxr_r3r]rerf_timesteps_tensornum_channels_latentsitlatent_model_inputr~ noise_pred variance_prednoise_pred_uncondnoise_pred_textvariance_pred_textstep_idximages# r__call__zKandinskyPipeline.__call__s@R fc "J  %VJQRVW]R^Q_`a a''"77 &4s&:#7;7J7J F13NP_8 4 11 lD ) 99\q9L +T 2$)II.C$K ! &'99:OUV9WL$9$K$KLagh$K$i ! 99&;\%JPQRUU#))&VL $$%8$H>>33#yy//;;#FE43I3IJ && -vu = & , ,    NN  d//0@AB' DAq=XG9q=!9^e 0=| \ )&@"3! #  J+,6,<,%%.?UfCf1gg "YY 4F'GQO --?NN))77;WW * 0 0q1Aq 0 I Ann))# * k #N(:a(? CC1g. O' T   T B8L ##% 1 1hithuvw w - 'CK#%EKK1%EIIK''1a399;AACE % %%e,E8O"%00r)N) Nrdg@rNNrNrT)__name__ __module__ __qualname____doc__model_cpu_offload_seqrrr rr r r r)r:rnrNno_gradrEXAMPLE_DOC_STRINGrWrTensorrintr Generatorrboolr __classcell__)r0s@rr!r!Us&7U&U'U# U  56 U  U( " dDLU]]_12 <@#& #%&MQ*.%*GK !q1c49n%q1ELL$u||*<<=q1 %U\\4 3E%EF q1 "%T#Y"78 q1  q1q1!q1q1 #q1E%//43H"HIJq1%,,'q1c]q18S#u||$rsx32 46 .C*))MM   H %>6N1)N1r