bi€ ddlZddlmZmZmZmZmZddlZddl Z ddl m Z m Z ddl mZmZddlmZmZddlmZddlmZmZmZmZdd lmZdd lmZd d lmZd dl m!Z!erddl"m#Z#n GddZ#er ddl$m%cm&Z'dZ(ndZ(ejRe*Z+dZ, ddee-deee.e j^fdeee-deee0fdZ1GddeZ2y)N)CallableDictListOptionalUnion)T5EncoderModelT5TokenizerFast)MultiPipelineCallbacksPipelineCallback)AutoencoderKLWanCosmosTransformer3DModel)FlowMatchEulerDiscreteScheduler)is_cosmos_guardrail_availableis_torch_xla_availableloggingreplace_example_docstring) randn_tensor)VideoProcessor)DiffusionPipeline)CosmosImagePipelineOutput)CosmosSafetyCheckerceZdZdZy)rctd)Nz|`cosmos_guardrail` is not installed. Please install it to use the safety checker for Cosmos: `pip install cosmos_guardrail`.) ImportError)selfargskwargss q/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/pipelines/cosmos/pipeline_cosmos2_text2image.py__init__zCosmosSafetyChecker.__init__%sO N)__name__ __module__ __qualname__r"r#r!rr$s r#rTFa8 Examples: ```python >>> import torch >>> from diffusers import Cosmos2TextToImagePipeline >>> # Available checkpoints: nvidia/Cosmos-Predict2-2B-Text2Image, nvidia/Cosmos-Predict2-14B-Text2Image >>> model_id = "nvidia/Cosmos-Predict2-2B-Text2Image" >>> pipe = Cosmos2TextToImagePipeline.from_pretrained(model_id, torch_dtype=torch.bfloat16) >>> pipe.to("cuda") >>> prompt = "A close-up shot captures a vibrant yellow scrubber vigorously working on a grimy plate, its bristles moving in circular motions to lift stubborn grease and food residue. The dish, once covered in remnants of a hearty meal, gradually reveals its original glossy surface. Suds form and bubble around the scrubber, creating a satisfying visual of cleanliness in progress. The sound of scrubbing fills the air, accompanied by the gentle clinking of the dish against the sink. As the scrubber continues its task, the dish transforms, gleaming under the bright kitchen lights, symbolizing the triumph of cleanliness over mess." >>> negative_prompt = "The video captures a series of frames showing ugly scenes, static with no motion, motion blur, over-saturation, shaky footage, low resolution, grainy texture, pixelated images, poorly lit areas, underexposed and overexposed scenes, poor color balance, washed out colors, choppy sequences, jerky movements, low frame rate, artifacting, color banding, unnatural transitions, outdated special effects, fake elements, unconvincing visuals, poorly edited content, jump cuts, visual noise, and flickering. Overall, the video is of poor quality." >>> output = pipe( ... prompt=prompt, negative_prompt=negative_prompt, generator=torch.Generator().manual_seed(1) ... ).images[0] >>> output.save("output.png") ``` num_inference_stepsdevice timestepssigmasc | | td|dttj|jj j v}|std|jd|jd ||d||j}t|}||fS|dttj|jj j v}|std|jd|jd ||d||j}t|}||fS|j|fd |i||j}||fS) a Calls the scheduler's `set_timesteps` method and retrieves timesteps from the scheduler after the call. Handles custom timesteps. Any kwargs will be supplied to `scheduler.set_timesteps`. Args: scheduler (`SchedulerMixin`): The scheduler to get timesteps from. num_inference_steps (`int`): The number of diffusion steps used when generating samples with a pre-trained model. If used, `timesteps` must be `None`. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. timesteps (`List[int]`, *optional*): Custom timesteps used to override the timestep spacing strategy of the scheduler. If `timesteps` is passed, `num_inference_steps` and `sigmas` must be `None`. sigmas (`List[float]`, *optional*): Custom sigmas used to override the timestep spacing strategy of the scheduler. If `sigmas` is passed, `num_inference_steps` and `timesteps` must be `None`. Returns: `Tuple[torch.Tensor, int]`: A tuple where the first element is the timestep schedule from the scheduler and the second element is the number of inference steps. zYOnly one of `timesteps` or `sigmas` can be passed. Please choose one to set custom valuesr*zThe current scheduler class zx's `set_timesteps` does not support custom timestep schedules. Please check whether you are using the correct scheduler.)r*r)r+zv's `set_timesteps` does not support custom sigmas schedules. Please check whether you are using the correct scheduler.)r+r)r)r') ValueErrorsetinspect signature set_timesteps parameterskeys __class__r*len) schedulerr(r)r*r+r accepts_timesteps accept_sigmass r!retrieve_timestepsr9Ls>!3tuu'3w/@/@AXAX/Y/d/d/i/i/k+ll .y/B/B.CDab  M)FMfM'' !)n ) ))   C(9(9):Q:Q(R(](](b(b(d$ee .y/B/B.CD_`  GvfGG'' !)n ) ))  3MFMfM'' ) ))r#c$PeZdZdZdZgdZdgZ d6dedede d e d e de f fd Z d7d eeeefdedeej(deej*fdZ d8d eeeefdeeeeefdededeej0deej0dedeej(deej*fdZ d9deddded ed!edeej*deej(d"eeej4eej4fd#eej0d$ej0fd%Z d:d&Zed'Zed(Zed)Z ed*Z!ed+Z"ejFe$e%ddddd,d-dddddd.ddd#gd fd eeeefdeeeeefded ed/ed0e&deed"eeej4eej4fd#eej0deej0deej0d1eed2ed3eee'eee(gdfe)e*fd4eedef d5Z+xZ,S);Cosmos2TextToImagePipelinea Pipeline for text-to-image generation using [Cosmos Predict2](https://github.com/nvidia-cosmos/cosmos-predict2). This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, running on a particular device, etc.). Args: text_encoder ([`T5EncoderModel`]): Frozen text-encoder. Cosmos uses [T5](https://huggingface.co/docs/transformers/model_doc/t5#transformers.T5EncoderModel); specifically the [t5-11b](https://huggingface.co/google-t5/t5-11b) variant. tokenizer (`T5TokenizerFast`): Tokenizer of class [T5Tokenizer](https://huggingface.co/docs/transformers/model_doc/t5#transformers.T5Tokenizer). transformer ([`CosmosTransformer3DModel`]): Conditional Transformer to denoise the encoded image latents. scheduler ([`FlowMatchEulerDiscreteScheduler`]): A scheduler to be used in combination with `transformer` to denoise the encoded image latents. vae ([`AutoencoderKLWan`]): Variational Auto-Encoder (VAE) Model to encode and decode videos to and from latent representations. ztext_encoder->transformer->vae)latents prompt_embedsnegative_prompt_embedssafety_checkerN text_encoder tokenizer transformervaer6cVt|| t}|j||||||t |ddr"dt |j jznd|_t |ddr"dt|j jznd|_ t|j|_ d|_ d|_d |_d |_|j"H|j"j%|j|j|j|j yy) N)rCr@rArBr6r?rCr)vae_scale_factorgT@gMb`?? sigma_min) sigma_maxrI sigma_datafinal_sigmas_type)superr"rregister_modulesgetattrsumrCtemperal_downsamplevae_scale_factor_temporalr5vae_scale_factor_spatialrvideo_processorrJrIrKrLr6register_to_config)rr@rArBrCr6r?r4s r!r"z#Cosmos2TextToImagePipeline.__init__s   !02N %#)  T[[_afhlSmc$((2N2N.O)Ost&RYZ^`egkRlS1M1M-N(Nrs%-t?\?\]!, >> % NN - -....??"&"8"8 .  &r#promptmax_sequence_lengthr)dtypec >|xs |j}|xs|jj}t|tr|gn|}|j |d|dddd}|j }|jjj|}|j |ddj }|jd|jdk\rXtj||sB|j j|dd|d z df} tjd |d | |j|j|| j } | j|| } |j#d j%} t'| D]\} } d| | | df<| S)N max_lengthTptF)paddingr[ truncationreturn_tensors return_lengthreturn_offsets_mappinglongest)r]r_rzXThe following part of your input was truncated because `max_sequence_length` is set to z tokens: )attention_mask)rYr))dimr)_execution_devicer@rY isinstancestrrA input_idsrdbooltoshapetorchequal batch_decodeloggerwarninglast_hidden_staterPcpu enumerate)rrWrXr)rY text_inputstext_input_idsprompt_attention_maskuntruncated_ids removed_textr=lengthsilengths r!_get_t5_prompt_embedsz0Cosmos2TextToImagePipeline._get_t5_prompt_embedss14110**00'4&&nn  *#(% %.. + : : ? ? A D DV L..SW.Xbb   $(<(>66qJ]`aJadfJfGf7ghL NN'( ,A  ))   f %6K*   &((uV(D '+++2668"7+ *IAv()M!VW* % *r#Trnegative_promptdo_classifier_free_guidancenum_images_per_promptr=r>c |xs |j}t|tr|gn|}| t|} n|jd} |N|j |||| }|j\} } } |j d|d}|j| |z| d}|r||xsd}t|tr| |gzn|}|:t|t|ur$tdt|dt|d| t|k7r!td |d t|d |d | d |j |||| }|j\} } } |j d|d}|j| |z| d}||fS) a" Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*): prompt to be encoded negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). do_classifier_free_guidance (`bool`, *optional*, defaults to `True`): Whether to use classifier free guidance or not. num_images_per_prompt (`int`, *optional*, defaults to 1): Number of videos that should be generated per prompt. torch device to place the resulting embeddings on prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. device: (`torch.device`, *optional*): torch device dtype: (`torch.dtype`, *optional*): torch dtype r)rWrXr)rYrrcz?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.) rfrgrhr5rlr}repeatviewtype TypeErrorr-) rrWr~rrr=r>rXr)rY batch_size_seq_lens r! encode_promptz(Cosmos2TextToImagePipeline.encode_promptsL1411'4&&  VJ&,,Q/J   663Fv]b7M *//MAw)004I1MM)..zZ+GA#i.AQR&<'gi  u &PUV..8888r#c |dzdk7s|dzdk7rtd|d|d|Ltfd|Ds8tdjd|Dcgc]}|jvs|c}||td |d |d | | td |9t|ts(t|t std t |yyycc}w)Nrrz8`height` and `width` have to be divisible by 16 but are z and rc3:K|]}|jvywN)_callback_tensor_inputs).0krs r! z:Cosmos2TextToImagePipeline.check_inputs..rs#F 23A-- -F sz2`callback_on_step_end_tensor_inputs` has to be in z , but found zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is )r-allrrgrhrr)rrWrrr="callback_on_step_end_tensor_inputsrs` r! check_inputsz'Cosmos2TextToImagePipeline.check_inputsgs_ B;! urzQWX^W__dejdkklmn n - 9#F 7YF C DTEaEaDbbn|^pHvwbckokGkGbGpqpHoIJ   -";08N}o^00 ^ 5w  FC)@TZ\`IaQRVW]R^Q_`a aJb)@ pHs C%Cc|jSr_guidance_scalers r!guidance_scalez)Cosmos2TextToImagePipeline.guidance_scales###r#c |jdkDS)NrHrrs r!rz6Cosmos2TextToImagePipeline.do_classifier_free_guidances##c))r#c|jSr)_num_timestepsrs r! num_timestepsz(Cosmos2TextToImagePipeline.num_timestepss"""r#c|jSr)_current_timesteprs r!current_timestepz+Cosmos2TextToImagePipeline.current_timesteps%%%r#c|jSr) _interruptrs r! interruptz$Cosmos2TextToImagePipeline.interrupts r##g@pilr(r output_type return_dictcallback_on_step_endrc |jtd|jdt|tt fr |j }d}|j|||| |||_d|_ d|_ |j}|j~|jj||Ft|tr|gn|}|D],}|jj|rtd|d|jjd|t|trd}n-|t|tr t!|}n| j"d }|j%|||j&|| | || \} } t(j*j,j/rt(j0nt(j2}t)j4d d|| }t7|j8|| \}}|j8j:j=d ddk(r0|j8j>d|j8j>d<|j@jB}|j@j:jD}|jG||z||||t(j0||| } | jIdd||| }t!|||j8jJzz }t!||_&|jO|5}tQ|D]\}}|jRr||_ |j8j>|} | | dzz }!d|!z }"d|!z }#|! }$|!jU| j"d j|}%| |"z}&|&j|}&|jA|&|%| |dd }'|#| z|$|'jWzzj|}'|j&rV|jA|&|%| |dd }(|#| z|$|(jWzzj|}(|'|jX|'|(z zz}'| |'z | z }'|j8j[|'|| dd } |Zi})|D]}*t]|*|)|*<|||||)}+|+j_d| } |+j_d| } |+j_d| } |t!|dz k(s'|dz|kDr/|dz|j8jJzd k(r|jatbstejf dddd|_ | dk(sFt)jh|jjj:jljod|jjj:jpdddj| jr| jB},dt)jh|jjj:jtjod|jjj:jpdddj| jr| jBz }-| |-z |j8j:jvz |,z} |jjjy| j|jjjBdd }.|j+|jj||jzj}|.d}.|.dzjtj}.g}/|.D].}0|jj|0}0|/j|00tj|/jtj0dz dzdz }.t)j|.jd dddd }.|jzj}|.| }.|jjdn|jzj}|.| }.|.D1cgc]}1|1d  }2}1t|.t(jrt)j|2}2n 1`. num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor is generated by sampling using the supplied random `generator`. prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.FloatTensor`, *optional*): Pre-generated negative text embeddings. For PixArt-Sigma this negative prompt should be "". If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generated image. Choose between `PIL.Image` or `np.array`. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`CosmosImagePipelineOutput`] instead of a plain tuple. callback_on_step_end (`Callable`, `PipelineCallback`, `MultiPipelineCallbacks`, *optional*): A function or a subclass of `PipelineCallback` or `MultiPipelineCallbacks` that is called at the end of each denoising step during the inference. with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by `callback_on_step_end_tensor_inputs`. callback_on_step_end_tensor_inputs (`List`, *optional*): The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. Examples: Returns: [`~CosmosImagePipelineOutput`] or `tuple`: If `return_dict` is `True`, [`CosmosImagePipelineOutput`] is returned, otherwise a `tuple` is returned where the first element is a list with the generated images and the second element is a list of `bool`s indicating whether the corresponding generated image contains "not-safe-for-work" (nsfw) content. Nz)You have disabled the safety checker for z. This is in violation of the [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license). Please ensure that you are compliant with the license agreement.rFz5Cosmos Guardrail detected unsafe text in the prompt: zR. Please ensure that the prompt abides by the NVIDIA Open Model License Agreement.rsr)rWr~rrr=r>r)rX)rY)r)r+rLzerorIrc)total) hidden_statestimestepencoder_hidden_states padding_maskr)rr<r=r>latentrHnp)rgo@rrEr )images)Kr?r-r4rgr r tensor_inputsrrrrrfrkrhcheck_text_safetyrr5rlrrrmbackendsmps is_availablefloat32float64linspacer9r6rgetr+rBrY in_channelsr new_zerosorderr progress_barrtrexpandfloatrsteplocalspopupdate XLA_AVAILABLExm mark_steptensorrC latents_meanrz_dimr) latents_stdrKdecoderTpostprocess_videoastyperuint8check_video_safetyappendstack from_numpypermuteTensorndarraymaybe_free_model_hooksr)3rrWr~rrr(rrrr<r=r>rrrrrXrr) prompt_listpr sigmas_dtyper+r*transformer_dtyperrnum_warmup_stepsrr{t current_sigma current_tc_inc_skipc_outrlatent_model_input noise_prednoise_pred_uncondcallback_kwargsrcallback_outputsrrvideo video_batchvidbatchimages3 r!__call__z#Cosmos2TextToImagePipeline.__call__sb`    &;DNN;KLST  *-=?U,V W1E1S1S .  &&%@bc-!%''    *    " "6 *!*4VS*Avhv $A..@@C(STUSVWXY    " "5 )  *VS"9J  Jvt$<VJ&,,Q/J   +(,(H(H"7'#9 3    ").(:(:(G(G(Iu}}u}} 1&9N);DNNSYbh)i& & >> $ $%8& A[ P(,(=(=b(ADNN ! !" %!,,22#//66BB&& . .    MM     ((AvuDU(V y>,?$..BVBV,VV!)n   %8  97 #\!),6 #1>>)*& $ 5 5a 8 )]Q->? 9}Y" $++GMM!,<=@@ARS%,t^"%7%:%:;L%M"!--"4%*7!- % .   %w.9I9I9K1KKOOPab 33(,(8(8&8!).D%1$) )9)  )%*0')9EDUD[D[D]<])](a(abs(t%!+d.A.AZRcEc.d!dJ% 2mC ..--j!WRW-XYZ['3&(O?9-3Xa[*9';D!Q'X$.229gFG$4$8$8-$XM-=-A-ABZ\r-s*I**A9I/IqSTuX\XfXfXlXlNlpqNq '') LLNm6 #7 #r"&h& TXX__99:a..1a8GNNGMM2   TXX__-H-H I N NqRVRZRZRaRaRgRgijlmop q t t !K +dnn.C.C.N.NNQ]]GHHOOGJJtxx~~$>EORSTUE"".##&&v.,,>>uRV>W,,RXX6  ,C--@@EC&&s+,-44RZZ@5H1LqP((/771aAF,,>>uR]>^##&&u-,,>>uR]>^+01%U1X1E1%. E*E2::.Aq!G$E ##%8O(66G7 #7 #j2sHb5b5 c5b?r)NrVNN)NTrNNrVNN)rrrNNNN)NN)-r$r%r&__doc__model_cpu_offload_seqr_optional_componentsrr rr rrr"rrhrintrrmr)rYr}rjrr Generatorrrpropertyrrrrrno_gradrEXAMPLE_DOC_STRINGrrrr r r __classcell__)r4s@r!r;r;sH,=T,-/3%$%#%. %  % 3 %,%T)-#&)-'+ (c49n%(!(& (  $ (\<@,0%&049=#&)-'+Q5c49n%Q5"%T#Y"78Q5&* Q5 # Q5   - Q5!) 6Q5!Q5&Q5 $Q5n'+)-MQ*.99!9 9  9  9 $9&9E%//43H"HIJ9%,,'9 9F+/ b<$$**##&&U]]_12)-;?#% #/0MQ*.049=%*  9B #&'F7c49n%F7"%T#Y"78F7 F7  F7 ! F7F7 (}F7E%//43H"HIJF7%,,'F7  -F7!) 6F7c]F7F7' (Cd+T124DF\\ ] F7$-1I%F7&!'F73F7r#r;)NNNN)3r/typingrrrrrnumpyrrm transformersrr callbacksr r modelsr r schedulersrutilsrrrrutils.torch_utilsrrTrpipeline_utilsrpipeline_outputrcosmos_guardrailrtorch_xla.core.xla_modelcore xla_modelrr get_loggerr$rpr rrhr)rr9r;r'r#r!rs88 8A@9nn--.6!"4))MM   H %2*.15%)$( 8*!#8* U3 ,- .8*S " 8* T%[ ! 8*vZ7!2Z7r#