biWddlZddlmZmZmZmZddlZddlmZ ddl m Z m Z ddl mZddlmZmZmZddlmZdd lmZmZdd lmZd d lmZmZmZd dlmZer ddl m!cm"Z#dZ$ndZ$ejJe&Z'GddeeZ(y)N)ListOptionalTupleUnion) functional)CLIPTextModelWithProjection CLIPTokenizer)CLIPTextModelOutput)PriorTransformerUNet2DConditionModel UNet2DModel)UnCLIPScheduler)is_torch_xla_availablelogging) randn_tensor)DeprecatedPipelineMixinDiffusionPipelineImagePipelineOutput)UnCLIPTextProjModelTFc deZdZUdZdZdgZeed<eed<e ed<e ed<e ed<e ed<e ed <e ed <e ed <e ed <d Zdedede de de de d e d e d e d e ffd ZdZ d!deeeefdeej,fdZej0 d"deeeeefdededededeeej8eej8fdeej,deej,deej,deeeefdeej,dededeedefd ZxZ S)#UnCLIPPipelineaE Pipeline for text-to-image generation using unCLIP. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, running on a particular device, etc.). Args: text_encoder ([`~transformers.CLIPTextModelWithProjection`]): Frozen text-encoder. tokenizer ([`~transformers.CLIPTokenizer`]): A `CLIPTokenizer` to tokenize text. prior ([`PriorTransformer`]): The canonical unCLIP prior to approximate the image embedding from the text embedding. text_proj ([`UnCLIPTextProjModel`]): Utility class to prepare and combine the embeddings before they are passed to the decoder. decoder ([`UNet2DConditionModel`]): The decoder to invert the image embedding into an image. super_res_first ([`UNet2DModel`]): Super resolution UNet. Used in all but the last step of the super resolution diffusion process. super_res_last ([`UNet2DModel`]): Super resolution UNet. Used in the last step of the super resolution diffusion process. prior_scheduler ([`UnCLIPScheduler`]): Scheduler used in the prior denoising process (a modified [`DDPMScheduler`]). decoder_scheduler ([`UnCLIPScheduler`]): Scheduler used in the decoder denoising process (a modified [`DDPMScheduler`]). super_res_scheduler ([`UnCLIPScheduler`]): Scheduler used in the super resolution denoising process (a modified [`DDPMScheduler`]). z0.33.1priordecoder text_proj text_encoder tokenizersuper_res_firstsuper_res_lastprior_schedulerdecoder_schedulersuper_res_schedulerzAtext_encoder->text_proj->decoder->super_res_first->super_res_lastc Xt ||j||||||||| |  y)N) rrrrrr r!r"r#r$)super__init__register_modules) selfrrrrrr r!r"r#r$ __class__s e/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/pipelines/unclip/pipeline_unclip.pyr'zUnCLIPPipeline.__init__YsA  %+)+/ 3  c|t||||}n;|j|k7rtd|jd||j|}||jz}|S)N) generatordevicedtypezUnexpected latents shape, got z , expected )rshape ValueErrortoinit_noise_sigma)r)r1r0r/r.latents schedulers r+prepare_latentszUnCLIPPipeline.prepare_latentsusg ?"5IfTYZG}}% #A'--P[\a[b!cddjj(GI666r,text_model_outputtext_attention_maskc|t|tr t|nd}|j|d|jjdd}|j } |j jj|} |j|ddj } | jd| jdk\rtj| | s|jj| dd|jjdz df} tjd |jjd | | ddd|jjf} |j| j|} | j }| j"}n|d jd }|d |d}}|} |j%|d }|j%|d }| j%|d } |rad g|z}|j|d|jjdd}|j jj|}|j|j j|}|j }|j"}|jd}|j'd|}|j)||z|}|jd}|j'd|d}|j)||z|d}|j%|d }tj*||g}tj*||g}tj*|| g} ||| fS)Nr max_lengthTpt)paddingr; truncationreturn_tensorslongest)r=r?z\The following part of your input was truncated because CLIP can only handle sequences up to z tokens: rdim) isinstancelistlenrmodel_max_length input_idsattention_maskboolr3r1torchequal batch_decodeloggerwarningr text_embedslast_hidden_staterepeat_interleaverepeatviewcat)r)promptr/num_images_per_promptdo_classifier_free_guidancer8r9 batch_size text_inputstext_input_ids text_maskuntruncated_ids removed_texttext_encoder_output prompt_embedstext_enc_hid_states uncond_tokens uncond_inputuncond_text_mask*negative_prompt_embeds_text_encoder_outputnegative_prompt_embedsuncond_text_enc_hid_statesseq_lens r+_encode_promptzUnCLIPPipeline._encode_promptsv  $(264(@VaJ..$>>::# )K)22N#22779<>$>>::# *L ,::??ADDVL 9=9J9J}!n.t9j:||j4j<dfd }!|j@j|||j@j}"|jBjjD}#|jBjjF}$|jBjjF}%|j||#|$|%f|j ||||j@}t#|j%|"D] \}}|rt'j(|gdzn|}|jC|||| |!jH}&|r~|&j-d\}'}(|'jK|j dd\}'})|(jK|j dd\}(}*|'| |(|'z zz}&t'j(|&|*gd}&|dz|"j dk(rd}n|"|dz}|j@j/|&||||j0}|jMdd}|}+|jNj|||jNj},|jPjjDdz}-|jPjjF}$|jPjjF}%|j||-|$|%f|+j ||| |jN} |j d k(rt9jR|+|$|%g}.nRi}/dtUjVt8jRjXvrd |/d<t9jR|+f|$|%gddd|/}.t#|j%|,D]\}}||,j ddz k(r |jZ}0n |jP}0t'j(| |.gd}|0||jH}&|dz|,j dk(rd}n|,|dz}|jNj/|&|| ||j0} t\st_j`| }1|jc|1dzdz}1|1jMdd}1|1jejgddddjijk}1|dk(r|jm|1}1|s|1fSto|1S)a The call function to the pipeline for generation. Args: prompt (`str` or `List[str]`): The prompt or prompts to guide image generation. This can only be left undefined if `text_model_output` and `text_attention_mask` is passed. num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. prior_num_inference_steps (`int`, *optional*, defaults to 25): The number of denoising steps for the prior. More denoising steps usually lead to a higher quality image at the expense of slower inference. decoder_num_inference_steps (`int`, *optional*, defaults to 25): The number of denoising steps for the decoder. More denoising steps usually lead to a higher quality image at the expense of slower inference. super_res_num_inference_steps (`int`, *optional*, defaults to 7): The number of denoising steps for super resolution. More denoising steps usually lead to a higher quality image at the expense of slower inference. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. prior_latents (`torch.Tensor` of shape (batch size, embeddings dimension), *optional*): Pre-generated noisy latents to be used as inputs for the prior. decoder_latents (`torch.Tensor` of shape (batch size, channels, height, width), *optional*): Pre-generated noisy latents to be used as inputs for the decoder. super_res_latents (`torch.Tensor` of shape (batch size, channels, super res height, super res width), *optional*): Pre-generated noisy latents to be used as inputs for the decoder. prior_guidance_scale (`float`, *optional*, defaults to 4.0): A higher guidance scale value encourages the model to generate images closely linked to the text `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`. decoder_guidance_scale (`float`, *optional*, defaults to 4.0): A higher guidance scale value encourages the model to generate images closely linked to the text `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`. text_model_output (`CLIPTextModelOutput`, *optional*): Pre-defined [`CLIPTextModel`] outputs that can be derived from the text encoder. Pre-defined text outputs can be passed for tasks like text embedding interpolations. Make sure to also pass `text_attention_mask` in this case. `prompt` can the be left `None`. text_attention_mask (`torch.Tensor`, *optional*): Pre-defined CLIP text attention mask that can be derived from the tokenizer. Pre-defined text attention masks are necessary when passing `text_model_output`. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generated image. Choose between `PIL.Image` or `np.array`. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.ImagePipelineOutput`] instead of a plain tuple. Returns: [`~pipelines.ImagePipelineOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.ImagePipelineOutput`] is returned, otherwise a `tuple` is returned where the first element is a list with the generated images. Nrz2`prompt` has to be of type `str` or `list` but is rg?)r/r)timestepproj_embeddingencoder_hidden_statesrJ)rvsampler. prev_timestep)image_embeddingsratext_encoder_hidden_statesrYmps)valueT)ryrvrx class_labelsrJrB)rzr.rA)size antialiasbicubicF)rmode align_corners)ryrvg?r pil)images)8rEstrrFrGr2typer1_execution_devicerjr" set_timesteps timestepsrconfig embedding_dimr7r0 enumerate progress_barrLrVpredicted_image_embeddingchunkstep prev_samplepost_process_latentsrintFpadclip_extra_context_tokensrKr#r in_channels sample_sizerysplitclampr$r interpolateinspect signature parametersr! XLA_AVAILABLExm mark_stepmaybe_free_model_hookscpupermutefloatnumpy numpy_to_pilr)2r)rWrXrkrlrmr.rnrorpr8r9rqrrrsrtrZr/rYrarbr]prior_timesteps_tensorritlatent_model_inputr predicted_image_embedding_uncondpredicted_image_embedding_textrzr{additive_clip_time_embeddingsdecoder_text_maskdecoder_timesteps_tensornum_channels_latentsheightwidth noise_prednoise_pred_uncondnoise_pred_text_predicted_variance image_smallsuper_res_timesteps_tensorchannelsimage_upscaledinterpolate_antialiasunetimages2 r+__call__zUnCLIPPipeline.__call__sJ  &#& FD) [  #UVZ[aVbUc!dee*1-33A6J''"77 &:S&@&`DZ]`D`#8<8K8K F13NPacv9 5 *I **+DV*T!%!5!5!?!? ))77 ,,  '         d//0FGH DAqC^M?Q+>!?dq (, ",&9( )3) (' &+SlSrSrstSuP02P,LOc25UUP-)1u.44Q77 $ 6q1u =  0055)$#+ 6 k / > 77 F ( >B^^-'':(C >L> :: ;;% "uyy1I !i$..2Z2Z\]1^fg h  1 6 6uzz B  !i$..2Z2Z\]1^fj k  ,,-HQW,X#'#9#9#C#C #||22>>$$00 ##//.. -vu =  % %     " "  d//0HIJ DAqE`O+'>?Q?W?WXY?Z`a'>'b$!16E6K6KL^LdLdefLgmn6K6o3!3.1G?]nKn1oo "YY 4F'GQO 1u066q99 $ 8Q ? #4499AmW`:k 3 :*//A6%   ../LU[.\%)%=%=%G%G"''..::a?%%,,88$$++77 00 65 1        $ $   ;;% ]];fe_MN$& !g// >III59%k2]]#)5/ QVZoNd//0JKL DAq.44Q7!;;**++!&,=~+NTU!V )f  1u288;; $ :1q5 A !% 8 8 = =A0 Yb!>!k  5 8" ##% c! Aq! ##Aq!Q/557==? % %%e,E8O"%00r,)NN)NrrNNNNNNg@g @rT)!__name__ __module__ __qualname____doc___last_supported_version_exclude_from_cpu_offloadr __annotations__r rrr rrmodel_cpu_offload_seqr'r7rrr rrLTensorrjno_gradrrr GeneratorrrKr __classcell__)r*s@r+rr)s\<'!(   !!""--  $$&&((_  & 2  !  '  % $ ) + - 8 "JN6:Y= $E*=u*D$EF Y=&ell3Y=vU]]_37%&)++--.MQ042648IM6:&)(+%* !\1sDI~./\1 #\1$' \1 &) \1 (+ \1E%//43H"HIJ\1  -\1"%,,/\1$ELL1\1$E*=u*D$EF\1&ell3\1$\1!&\1c]\1 !\1\1r,r))rtypingrrrrrLtorch.nnrr transformersrr &transformers.models.clip.modeling_clipr modelsr r r schedulersrutilsrrutils.torch_utilsrpipeline_utilsrrrrrtorch_xla.core.xla_modelcore xla_modelrr get_loggerrrOrr,r+rsp// $CFII)4-\\*))MM   H %O1,.?O1r,