biM ddlZddlmZmZmZmZmZmZmZddl Z ddl Z ddl m Z mZddlmZmZddlmZddlmZddlmZmZdd lmZdd lmZdd lmZmZm Z dd l!m"Z"d dl#m$Z$er ddl%m&cm'Z(dZ)ndZ)ejTe+Z,dZ- dde.de/de/de/fdZ0 ddee.deee1e jdfdeee.deee/fdZ3GddeeZ4y) N)AnyCallableDictListOptionalTupleUnion) AutoTokenizerGlmModel)MultiPipelineCallbacksPipelineCallback)VaeImageProcessor)CogView4LoraLoaderMixin) AutoencoderKLCogView4Transformer2DModel)DiffusionPipeline)FlowMatchEulerDiscreteScheduler)is_torch_xla_availableloggingreplace_example_docstring) randn_tensor)CogView4PipelineOutputTFa Examples: ```python >>> import torch >>> from diffusers import CogView4Pipeline >>> pipe = CogView4Pipeline.from_pretrained("THUDM/CogView4-6B", torch_dtype=torch.bfloat16) >>> pipe.to("cuda") >>> prompt = "A photo of an astronaut riding a horse on mars" >>> image = pipe(prompt).images[0] >>> image.save("output.png") ``` base_seq_len base_shift max_shiftreturnc&||z dz}||z|z}|S)Ng?) image_seq_lenrrrmmus i/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/pipelines/cogview4/pipeline_cogview4.pycalculate_shiftr%;s&  %#-A Y #B Inum_inference_stepsdevice timestepssigmasc 4dttj|jjj v}dttj|jjj v}|P|N|s|st d|jd|jd |||d||j}t|}||fS|M|K|st d|jd|jd ||d||j}t|}||fS|M|K|st d|jd|jd ||d ||j}t|}||fS|j|fd |i||j}||fS) a Calls the scheduler's `set_timesteps` method and retrieves timesteps from the scheduler after the call. Handles custom timesteps. Any kwargs will be supplied to `scheduler.set_timesteps`. Args: scheduler (`SchedulerMixin`): The scheduler to get timesteps from. num_inference_steps (`int`): The number of diffusion steps used when generating samples with a pre-trained model. If used, `timesteps` must be `None`. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. timesteps (`List[int]`, *optional*): Custom timesteps used to override the timestep spacing strategy of the scheduler. If `timesteps` is passed, `num_inference_steps` and `sigmas` must be `None`. sigmas (`List[float]`, *optional*): Custom sigmas used to override the timestep spacing strategy of the scheduler. If `sigmas` is passed, `num_inference_steps` and `timesteps` must be `None`. Returns: `Tuple[torch.Tensor, int]`: A tuple where the first element is the timestep schedule from the scheduler and the second element is the number of inference steps. r)r*zThe current scheduler class z's `set_timesteps` does not support custom timestep or sigma schedules. Please check whether you are using the correct scheduler.)r)r*r(zx's `set_timesteps` does not support custom timestep schedules. Please check whether you are using the correct scheduler.)r)r(zv's `set_timesteps` does not support custom sigmas schedules. Please check whether you are using the correct scheduler.)r*r(r(r ) setinspect signature set_timesteps parameterskeys ValueError __class__r)len) schedulerr'r(r)r*kwargsaccepts_timestepsaccepts_sigmass r$retrieve_timestepsr9Fs>$s7+<+ .y/B/B.CDab  M)FMfM'' !)n ) ))  v1.y/B/B.CD_`  GvfGG'' !)n ) ))  3MFMfM'' ) ))r&c1eZdZdZgZdZgdZdedede de de f fd Z d6d e eeefd edeej&deej(fdZ d7d e eeefdee eeefdededeej.deej.deej&deej(d efdZd8dZ d9dZedZedZedZedZedZ ed Z!ejDe#e$d d d d d!d d d"dd d d d d d#d$dd d d%gd fd ee eeefdee eeefd&eed'eed(ed)eeed*eee%d+e%ded,ee ejLeejLfd%eejNdeejNdeejNd-ee(eefd.e(eefd/ed0ed1ee)ee*fd2ee e+eee)gd fe,e-fd3eed ed4e e.e(ff,d5Z/xZ0S):CogView4PipelineaR Pipeline for text-to-image generation using CogView4. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) Args: vae ([`AutoencoderKL`]): Variational Auto-Encoder (VAE) Model to encode and decode images to and from latent representations. text_encoder ([`GLMModel`]): Frozen text-encoder. CogView4 uses [glm-4-9b-hf](https://huggingface.co/THUDM/glm-4-9b-hf). tokenizer (`PreTrainedTokenizer`): Tokenizer of class [PreTrainedTokenizer](https://huggingface.co/docs/transformers/main/en/main_classes/tokenizer#transformers.PreTrainedTokenizer). transformer ([`CogView4Transformer2DModel`]): A text conditioned `CogView4Transformer2DModel` to denoise the encoded image latents. scheduler ([`SchedulerMixin`]): A scheduler to be used in combination with `transformer` to denoise the encoded image latents. ztext_encoder->transformer->vae)latents prompt_embedsnegative_prompt_embeds tokenizer text_encodervae transformerr5c t||j|||||t|ddr/dt |j j jdz znd|_t|j|_ y)N)r?r@rArBr5rAr)vae_scale_factor) super__init__register_modulesgetattrr4rAconfigblock_out_channelsrFrimage_processor)selfr?r@rArBr5r3s r$rHzCogView4Pipeline.__init__s~  lQ\hq  W^^bdikoVpc$((//*L*L&MPQ&Q Rvw0$BWBWXr&Npromptmax_sequence_lengthr(dtypec|xs |j}|xs|jj}t|tr|gn|}|j |d|ddd}|j }|j |ddj }|jd|jdk\rXtj||sB|j j|dd|dz df}tjd|d ||jd} d | d zz d z} | d kDritj|jd | f|j j|j|j } tj | |gd }|j|j#|dj$d} | j#||} | S)NlongestTpt)padding max_length truncationadd_special_tokensreturn_tensors)rVrZrzXThe following part of your input was truncated because `max_sequence_length` is set to z tokens: r) fill_valuerRr()dim)output_hidden_statesrRr()_execution_devicer@rR isinstancestrr? input_idsshapetorchequal batch_decodeloggerwarningfull pad_token_idr(catto hidden_states) rNrPrQr(rR text_inputstext_input_idsuntruncated_ids removed_textcurrent_length pad_lengthpad_idsr=s r$_get_glm_embedsz CogView4Pipeline._get_glm_embedss14110**00'4&&nn *# % %....SW.Xbb   $(<(>66qJ]`aJadfJfGf7ghL NN'( ,A (--a0NR/0B6 >jj%%a(*5>>66$**%,, G #YY'@aHN)).*;*;F*CZ^)_mmnpq %((uV(D r&Trnegative_promptdo_classifier_free_guidancenum_images_per_promptr=r>c |xs |j}t|tr|gn|}| t|} n|jd} ||j || ||}|j d} |jd|d}|j| |z| d}|r||xsd}t|tr| |gzn|}|:t|t|ur$tdt|dt|d| t|k7r!td|d t|d |d | d |j || ||}|j d} |jd|d}|j| |z| d}||fS) a Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*): prompt to be encoded negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). do_classifier_free_guidance (`bool`, *optional*, defaults to `True`): Whether to use classifier free guidance or not. num_images_per_prompt (`int`, *optional*, defaults to 1): Number of images that should be generated per prompt. torch device to place the resulting embeddings on prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. device: (`torch.device`, *optional*): torch device dtype: (`torch.dtype`, *optional*): torch dtype max_sequence_length (`int`, defaults to `1024`): Maximum sequence length in encoded prompt. Can be set to other values but may lead to poorer results. rrr[z?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.) rbrcrdr4rfrxsizerepeatviewtype TypeErrorr2) rNrPryrzr{r=r>r(rRrQ batch_sizeseq_lens r$ encode_promptzCogView4Pipeline.encode_promptsP1411'4&&  VJ&,,Q/J   009LfV[\M$$Q'%,,Q0EqI %**:8M+MwXZ[ &+A+I-3O@J?\_@`jO+<Z+GA#i.AQR&<'gi u &PUVr&c $|dzdk7s|dzdk7rtd|d|d|Ltfd|Ds8tdjd|Dcgc]}|jvs|c}||td |d |d | | td |7t|ts't|t std t |||td |d|d ||td|d|d |||jd|jdk7r&td|jd|jd|jd|jdk7r&td|jd|jdyyycc}w)Nr\rz8`height` and `width` have to be divisible by 16 but are z and r~c3:K|]}|jvywN)_callback_tensor_inputs).0krNs r$ z0CogView4Pipeline.check_inputs..Ks#F 23A-- -F sz2`callback_on_step_end_tensor_inputs` has to be in z , but found zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is z and `negative_prompt_embeds`: z'Cannot forward both `negative_prompt`: zz`prompt_embeds` and `negative_prompt_embeds` must have the same batch size when passed directly, but got: `prompt_embeds` z and `negative_prompt_embeds` r[zy`prompt_embeds` and `negative_prompt_embeds` must have the same dimension when passed directly, but got: `prompt_embeds` )r2allrrcrdrrrf) rNrPrrry"callback_on_step_end_tensor_inputsr=r>rs ` r$ check_inputszCogView4Pipeline.check_inputs>s B;! urzQWX^W__dejdkklmn n - 9#F 7YF C DTEaEaDbbn|^pHvwbckokGkGbGpqpHoIJ   -";08N}o^00 ^ 5w  FC)@TZ\`IaQRVW]R^Q_`a a  "8"D09*++]_   &+A+M9/9JK*++]_   $)?)K""1%)?)E)Ea)HH --:-@-@,AB.445Q8 ""2&*@*F*Fr*JJ --:-@-@,AB.445Q8K*L $5pHs F %F c|jSr_guidance_scalerNs r$guidance_scalezCogView4Pipeline.guidance_scalews###r&c |jdkDS)Nrrrs r$rzz,CogView4Pipeline.do_classifier_free_guidance~s##a''r&c|jSr)_num_timestepsrs r$ num_timestepszCogView4Pipeline.num_timestepss"""r&c|jSr)_attention_kwargsrs r$attention_kwargsz!CogView4Pipeline.attention_kwargs%%%r&c|jSr)_current_timesteprs r$current_timestepz!CogView4Pipeline.current_timesteprr&c|jSr) _interruptrs r$ interruptzCogView4Pipeline.interrupts r&2g@)rrpilr<rrr'r)r*rr original_sizecrops_coords_top_left output_type return_dictrcallback_on_step_endrrc  t|ttfr |j}|xs-|jj j |jz}|xs-|jj j |jz}|xs||f}||f}|j|||||| | ||_ ||_ d|_ d|_ |t|trd}n-|t|tr t|}n| j d}|j"}|j%|||j&| | | ||\} } |jj j(}|j+|| z|||t,j.|| | } t-j0|g| j2|}t-j0|g| j2|}t-j0|g| j2|}|j5|| zd}|j5|| zd}|j5|| zd}||jz||jzz|jj j6dzz}|5t9j:|j<j j>d|nt9j@|}|jCt8jDjCt8j.}|#||j<j j>z n|}tG||j<j jId d |j<j jId d |j<j jId d}tK|j<|||||\}}t||_&|jj2}tOt|||j<jPzz d}|jS|5}tU|D]\}} |jVr| |_ | jY|}!| j[| j d}"|jj]d5|j |!| |"||||dd}#ddd|j&rV|jj]d5|j |!| |"||||dd}$ddd$|j^#|$z zz}%n#}%|j<ja|%| | dd} |qi}&|D]}'tc|'|&|'<||||j<jd||&}(|(jgd| } |(jgd| } |(jgd| } |t|dz k(s'|dz|kDr/|dz|j<jPzdk(r|jitjstmjn dddd|_ |dk(sh| jY|jpj2|jpj jrz } |jpju| d| d})n| })|jvjy|)|})|j{|s|)fSt}|)S#1swY'xYw#1swYxYw#1swYxYw)aL Function invoked when calling the pipeline for generation. Args: prompt (`str` or `List[str]`, *optional*): The prompt or prompts to guide the image generation. If not defined, one has to pass `prompt_embeds`. negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). height (`int`, *optional*, defaults to self.transformer.config.sample_size * self.vae_scale_factor): The height in pixels of the generated image. If not provided, it is set to 1024. width (`int`, *optional*, defaults to self.transformer.config.sample_size * self.vae_scale_factor): The width in pixels of the generated image. If not provided it is set to 1024. num_inference_steps (`int`, *optional*, defaults to `50`): The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. timesteps (`List[int]`, *optional*): Custom timesteps to use for the denoising process with schedulers which support a `timesteps` argument in their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is passed will be used. Must be in descending order. sigmas (`List[float]`, *optional*): Custom sigmas to use for the denoising process with schedulers which support a `sigmas` argument in their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is passed will be used. guidance_scale (`float`, *optional*, defaults to `5.0`): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://huggingface.co/papers/2207.12598). `guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://huggingface.co/papers/2205.11487). Guidance scale is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality. num_images_per_prompt (`int`, *optional*, defaults to `1`): The number of images to generate per prompt. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.FloatTensor`, *optional*): Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will ge generated by sampling using the supplied random `generator`. prompt_embeds (`torch.FloatTensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.FloatTensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. original_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)): If `original_size` is not the same as `target_size` the image will appear to be down- or upsampled. `original_size` defaults to `(height, width)` if not specified. Part of SDXL's micro-conditioning as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). crops_coords_top_left (`Tuple[int]`, *optional*, defaults to (0, 0)): `crops_coords_top_left` can be used to generate an image that appears to be "cropped" from the position `crops_coords_top_left` downwards. Favorable, well-centered images are usually achieved by setting `crops_coords_top_left` to (0, 0). Part of SDXL's micro-conditioning as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generate image. Choose between [PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.stable_diffusion_xl.StableDiffusionXLPipelineOutput`] instead of a plain tuple. attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). callback_on_step_end (`Callable`, *optional*): A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by `callback_on_step_end_tensor_inputs`. callback_on_step_end_tensor_inputs (`List`, *optional*): The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. max_sequence_length (`int`, defaults to `224`): Maximum sequence length in encoded prompt. Can be set to other values but may lead to poorer results. Examples: Returns: [`~pipelines.cogview4.pipeline_CogView4.CogView4PipelineOutput`] or `tuple`: [`~pipelines.cogview4.pipeline_CogView4.CogView4PipelineOutput`] if `return_dict` is True, otherwise a `tuple`. When returning a tuple, the first element is a list with the generated images. NFrr)r{r=r>rQr(rarDg?base_image_seq_lenr?r?)r#)totalcond)rpencoder_hidden_statestimestepr target_size crop_coordsrruncond)rr<r=r>latent)rr)r)images)?rcrr tensor_inputsrBrK sample_sizerFrrrrrrdrr4rfrbrrz in_channelsrrgfloat32tensorrRr patch_sizenplinspacer5num_train_timestepsarrayastypeint64r%getr9rmaxorder progress_bar enumeraterroexpand cache_contextrsteplocalsr*popupdate XLA_AVAILABLExm mark_steprAscaling_factordecoderM postprocessmaybe_free_model_hooksr)*rNrPryrrr'r)r*rr{rr<r=r>rrrrrrrrQrrr(latent_channelsr!r#transformer_dtypenum_warmup_stepsritlatent_model_inputrnoise_pred_condnoise_pred_uncond noise_predcallback_kwargsrcallback_outputsimages* r$__call__zCogView4Pipeline.__call__sf *-=?U,V W1E1S1S .V4++22>>AVAVVT))00<\]    # # . . 1   KK--AA3H[ \)$  $$RXX.55bjjA JP.T^^22FFF^d   NN ! ! % %&:C @ NN ! ! % %lD 9 NN ! ! % %k4 8   *< NN/Fr* & &")n!,,22s9~0CdnnFZFZ0ZZ\]^   %8  98 #\!),7 #1>>)*&%,ZZ0A%B"88GMM!$45%%33F; &*&6&6&8.;!)&3$/$9)9$)'7 ' 'O 33))77A ,0,<,<*<2H%-*7(3(=-=(--= - -) "3T5H5HO^oLo5p!pJ!0J..--j!WRW-XYZ[(3&(O?9-3Xa[*9';D!T^^EZEZ[\E]_n'o$.229gFG$4$8$8-$XM-=-A-ABZ\r-s*I**A9I/IqSTuX\XfXfXlXlNlpqNq '') LLNo7 #8 #t"&h&jj0488??3Q3QQGHHOOG)OTUVWEE$$00K0P ##%8O%U33    38 #8 #sDA1[' [ (/['[4D['9[' [ ['[$ [''[0)NrONN)NTrNNNNrOr)NN)1__name__ __module__ __qualname____doc___optional_componentsmodel_cpu_offload_seqrr r rrrrHr rdrrrrgr(rRrxboolTensorrrrpropertyrrzrrrrno_gradrEXAMPLE_DOC_STRINGfloat Generator FloatTensorrrrrrr rr __classcell__)r3s@r$r;r;s5(<TY YY Y 0 Y 3 Y$)-#')-'+ )c49n%)!)& )  $ )\<@,0%&049=)-'+#'M5c49n%M5"%T#Y"78M5&* M5 # M5   - M5!) 6M5&M5 $M5!M5^2#7r$$ ((##&&&&U]]_1237;? $##%)-(, #%&MQ/359>B3717  59 9B #'1Y4sDI~./Y4"%T#Y"78Y4 Y4 } Y4 ! Y4DI&Y4e%Y4Y4 #Y4E%//43H"HIJY4%++,Y4  1 12Y4!)):): ;Y4 c3h0Y4 %S#X!Y4"#Y4$%Y4&#4S>2'Y4(' (Cd+T124DF\\ ] )Y4.-1I/Y40!1Y42 %u, -3Y43Y4r&r;)rrr)NNNN)5r-typingrrrrrrr numpyrrg transformersr r callbacksr rrMrloadersrmodelsrrpipelines.pipeline_utilsr schedulersrutilsrrrutils.torch_utilsrpipeline_outputrtorch_xla.core.xla_modelcore xla_modelrr get_loggerrrjrrrr%rdr(r9r;r r&r$r s DDD 0A0.?99OO-3))MM   H % $    *.15%)$( @*!#@* U3 ,- .@*S " @* T%[ ! @*Fd4(*Ad4r&