bi ddlZddlmZmZmZmZmZmZmZddl Z ddl Z ddl m Z mZddlmZmZddlmZmZddlmZmZddlmZdd lmZdd lmZmZmZdd l m!Z!d d l"m#Z#er ddl$m%cm&Z'dZ(ndZ(ejRe*Z+dZ, dde-de.de.de.fdZ/ ddee-deee0e jbfdeee-deee.fdZ2GddeZ3y)N)AnyCallableDictListOptionalTupleUnion) AutoTokenizerGlmModel)MultiPipelineCallbacksPipelineCallback)PipelineImageInputVaeImageProcessor) AutoencoderKLCogView4Transformer2DModel)DiffusionPipeline)FlowMatchEulerDiscreteScheduler)is_torch_xla_availableloggingreplace_example_docstring) randn_tensor)CogView4PipelineOutputTFaw Examples: ```python >>> import torch >>> from diffusers import CogView4ControlPipeline >>> pipe = CogView4ControlPipeline.from_pretrained("THUDM/CogView4-6B-Control", torch_dtype=torch.bfloat16) >>> control_image = load_image( ... "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/sd_controlnet/bird_canny.png" ... ) >>> prompt = "A bird in space" >>> image = pipe(prompt, control_image=control_image, height=1024, width=1024, guidance_scale=3.5).images[0] >>> image.save("cogview4-control.png") ``` base_seq_len base_shift max_shiftreturnc&||z dz}||z|z}|S)Ng?) image_seq_lenrrrmmus q/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/pipelines/cogview4/pipeline_cogview4_control.pycalculate_shiftr%<s&  %#-A Y #B Inum_inference_stepsdevice timestepssigmasc 4dttj|jjj v}dttj|jjj v}|P|N|s|st d|jd|jd |||d||j}t|}||fS|M|K|st d|jd|jd ||d||j}t|}||fS|M|K|st d|jd|jd ||d ||j}t|}||fS|j|fd |i||j}||fS) a Calls the scheduler's `set_timesteps` method and retrieves timesteps from the scheduler after the call. Handles custom timesteps. Any kwargs will be supplied to `scheduler.set_timesteps`. Args: scheduler (`SchedulerMixin`): The scheduler to get timesteps from. num_inference_steps (`int`): The number of diffusion steps used when generating samples with a pre-trained model. If used, `timesteps` must be `None`. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. timesteps (`List[int]`, *optional*): Custom timesteps used to override the timestep spacing strategy of the scheduler. If `timesteps` is passed, `num_inference_steps` and `sigmas` must be `None`. sigmas (`List[float]`, *optional*): Custom sigmas used to override the timestep spacing strategy of the scheduler. If `sigmas` is passed, `num_inference_steps` and `timesteps` must be `None`. Returns: `Tuple[torch.Tensor, int]`: A tuple where the first element is the timestep schedule from the scheduler and the second element is the number of inference steps. r)r*zThe current scheduler class z's `set_timesteps` does not support custom timestep or sigma schedules. Please check whether you are using the correct scheduler.)r)r*r(zx's `set_timesteps` does not support custom timestep schedules. Please check whether you are using the correct scheduler.)r)r(zv's `set_timesteps` does not support custom sigmas schedules. Please check whether you are using the correct scheduler.)r*r(r(r ) setinspect signature set_timesteps parameterskeys ValueError __class__r)len) schedulerr'r(r)r*kwargsaccepts_timestepsaccepts_sigmass r$retrieve_timestepsr9Hs>$s7+<+ .y/B/B.CDab  M)FMfM'' !)n ) ))  v1.y/B/B.CD_`  GvfGG'' !)n ) ))  3MFMfM'' ) ))r&c3eZdZdZgZdZgdZdedede de de f fd Z d8d e eeefd edeej&deej(fdZ d9d e eeefdee eeefdededeej.deej.deej&deej(d efdZd:dZ d;dZ dtransformer->vae)latents prompt_embedsnegative_prompt_embeds tokenizer text_encodervae transformerr5c t||j|||||t|ddr/dt |j j jdz znd|_t|j|_ y)N)r?r@rArBr5rAr)vae_scale_factor) super__init__register_modulesgetattrr4rAconfigblock_out_channelsrFrimage_processor)selfr?r@rArBr5r3s r$rHz CogView4ControlPipeline.__init__s~  lQ\hq  W^^bdikoVpc$((//*L*L&MPQ&Q Rvw0$BWBWXr&Npromptmax_sequence_lengthr(dtypec|xs |j}|xs|jj}t|tr|gn|}|j |d|ddd}|j }|j |ddj }|jd|jdk\rXtj||sB|j j|dd|dz df}tjd|d ||jd} d | d zz d z} | d kDritj|jd | f|j j|j|j } tj | |gd }|j|j#|dj$d} | j#||} | S)NlongestTpt)padding max_length truncationadd_special_tokensreturn_tensors)rVrZrzXThe following part of your input was truncated because `max_sequence_length` is set to z tokens: r) fill_valuerRr(dim)output_hidden_statesrRr()_execution_devicer@rR isinstancestrr? input_idsshapetorchequal batch_decodeloggerwarningfull pad_token_idr(catto hidden_states) rNrPrQr(rR text_inputstext_input_idsuntruncated_ids removed_textcurrent_length pad_lengthpad_idsr=s r$_get_glm_embedsz'CogView4ControlPipeline._get_glm_embedss14110**00'4&&nn *# % %....SW.Xbb   $(<(>66qJ]`aJadfJfGf7ghL NN'( ,A (--a0NR/0B6 >jj%%a(*5>>66$**%,, G #YY'@aHN)).*;*;F*CZ^)_mmnpq %((uV(D r&Trnegative_promptdo_classifier_free_guidancenum_images_per_promptr=r>c |xs |j}t|tr|gn|}| t|} n|jd} ||j || ||}|j d} |jd|d}|j| |z| d}|r||xsd}t|tr| |gzn|}|:t|t|ur$tdt|dt|d| t|k7r!td|d t|d |d | d |j || ||}|j d} |jd|d}|j| |z| d}||fS) a Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*): prompt to be encoded negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). do_classifier_free_guidance (`bool`, *optional*, defaults to `True`): Whether to use classifier free guidance or not. num_images_per_prompt (`int`, *optional*, defaults to 1): Number of images that should be generated per prompt. torch device to place the resulting embeddings on prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. device: (`torch.device`, *optional*): torch device dtype: (`torch.dtype`, *optional*): torch dtype max_sequence_length (`int`, defaults to `1024`): Maximum sequence length in encoded prompt. Can be set to other values but may lead to poorer results. rrr[z?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.) rcrdrer4rgrysizerepeatviewtype TypeErrorr2) rNrPrzr{r|r=r>r(rRrQ batch_sizeseq_lens r$ encode_promptz%CogView4ControlPipeline.encode_promptsP1411'4&&  VJ&,,Q/J   009LfV[\M$$Q'%,,Q0EqI %**:8M+MwXZ[ &+A+I-3O@J?\_@`jO+<Z+GA#i.AQR&<'gi u &PUVr&c Rt|tjrn|jj |||}|j d} | dk(r|} n|} |j | d|j d| z}|j||}|r| stj|gdz}|S)N)rrrr)r_ output_size)r(rRrD) rdrhTensorrM preprocessrgrepeat_interleaverpro) rNimagerrrr|r(rRr{ guess_modeimage_batch_size repeat_bys r$ prepare_imagez%CogView4ControlPipeline.prepare_imageBs eU\\ * ((33E&PU3VE ;;q> q "I.I'' qekkRSnW`F`'ae4 &zIIugk*E r&c |dzdk7s|dzdk7rtd|d|d|Ltfd|Ds8tdjd|Dcgc]}|jvs|c}||td |d |d | | td |7t|ts't|t std t |||td |d|d ||td|d|d |C|@|j|jk7r&td|jd|jdyyycc}w)Nr\rz8`height` and `width` have to be divisible by 16 but are z and rc3:K|]}|jvywN)_callback_tensor_inputs).0krNs r$ z7CogView4ControlPipeline.check_inputs..qs#F 23A-- -F sz2`callback_on_step_end_tensor_inputs` has to be in z , but found zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is z and `negative_prompt_embeds`: z'Cannot forward both `negative_prompt`: zu`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but got: `prompt_embeds` z != `negative_prompt_embeds` )r2allrrdrerrrg) rNrPrrrz"callback_on_step_end_tensor_inputsr=r>rs ` r$ check_inputsz$CogView4ControlPipeline.check_inputsds' B;! urzQWX^W__dejdkklmn n - 9#F 7YF C DTEaEaDbbn|^pHvwbckokGkGbGpqpHoIJ   -";08N}o^00 ^ 5w  FC)@TZ\`IaQRVW]R^Q_`a a  "8"D09*++]_   &+A+M9/9JK*++]_   $)?)K""&<&B&BB --:-@-@,AB.445Q8C*L $5pHs E%Ec|jSr_guidance_scalerNs r$guidance_scalez&CogView4ControlPipeline.guidance_scales###r&c |jdkDS)Nrrrs r$r{z3CogView4ControlPipeline.do_classifier_free_guidances##a''r&c|jSr)_num_timestepsrs r$ num_timestepsz%CogView4ControlPipeline.num_timestepss"""r&c|jSr)_attention_kwargsrs r$attention_kwargsz(CogView4ControlPipeline.attention_kwargs%%%r&c|jSr)_current_timesteprs r$current_timestepz(CogView4ControlPipeline.current_timesteprr&c|jSr) _interruptrs r$ interruptz!CogView4ControlPipeline.interrupts r&2g@)rrpilr< control_imagerrr'r)r*rr original_sizecrops_coords_top_left output_type return_dictrcallback_on_step_endrrc t|ttfr |j}|xs-|jj j |jz}|xs-|jj j |jz}|xs||f}||f}|j|||||| || |_ ||_ d|_ d|_ |t|trd}n-|t|tr t|}n| j d}|j"}|j%|||j&| | |||\} }|jj j(dz}|j+||||| z| ||j,j.}|j dd\}}d}|j,j1|j2j5}||z |j,j j6z}|j9|| z|||t:j<|| | } t;j>|g| j.| }t;j>|g| j.| }t;j>|g| j.| }|jA|| zd}|jA|| zd}|jA|| zd}||jz||jzz|jj jBdzz}|5tEjF|jHj jJd |ntEjL|}|jOtDjPjOtDj<}|#||jHj jJz n|}tS||jHj jUd d |jHj jUd d|jHj jUdd}tW|jH|||||\}}t||_,|jj.}t[t|||jHj\zz d}|j_|5} ta|D]\}!}"|jbr|"|_ t;jd| |gdjg|}#|"ji| j d}$|j |#| |$||||dd}%|j&r2|j |#||$||||dd}&|&|jj|%|&z zz}'n|%}'|jHjm|'|"| dd} |qi}(|D]})to|)|(|)<|||!|jHjp|!|(}*|*jsd| } |*jsd| } |*jsd|}|!t|dz k(s'|!dz|kDr/|!dz|jHj\zdk(r| jutvstyjz dddd|_ |dk(sh| jg|j,j.|j,j j6z } |j,j}| d| d}+n| }+|j~j|+|}+|j|s|+fSt|+S#1swYxYw)a@ Function invoked when calling the pipeline for generation. Args: prompt (`str` or `List[str]`, *optional*): The prompt or prompts to guide the image generation. If not defined, one has to pass `prompt_embeds`. negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). height (`int`, *optional*, defaults to self.transformer.config.sample_size * self.vae_scale_factor): The height in pixels of the generated image. If not provided, it is set to 1024. width (`int`, *optional*, defaults to self.transformer.config.sample_size * self.vae_scale_factor): The width in pixels of the generated image. If not provided it is set to 1024. num_inference_steps (`int`, *optional*, defaults to `50`): The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. timesteps (`List[int]`, *optional*): Custom timesteps to use for the denoising process with schedulers which support a `timesteps` argument in their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is passed will be used. Must be in descending order. sigmas (`List[float]`, *optional*): Custom sigmas to use for the denoising process with schedulers which support a `sigmas` argument in their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is passed will be used. guidance_scale (`float`, *optional*, defaults to `5.0`): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://huggingface.co/papers/2207.12598). `guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://huggingface.co/papers/2205.11487). Guidance scale is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality. num_images_per_prompt (`int`, *optional*, defaults to `1`): The number of images to generate per prompt. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.FloatTensor`, *optional*): Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will ge generated by sampling using the supplied random `generator`. prompt_embeds (`torch.FloatTensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.FloatTensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. original_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)): If `original_size` is not the same as `target_size` the image will appear to be down- or upsampled. `original_size` defaults to `(height, width)` if not specified. Part of SDXL's micro-conditioning as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). crops_coords_top_left (`Tuple[int]`, *optional*, defaults to (0, 0)): `crops_coords_top_left` can be used to generate an image that appears to be "cropped" from the position `crops_coords_top_left` downwards. Favorable, well-centered images are usually achieved by setting `crops_coords_top_left` to (0, 0). Part of SDXL's micro-conditioning as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generate image. Choose between [PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.pipeline_CogView4.CogView4PipelineOutput`] instead of a plain tuple. attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). callback_on_step_end (`Callable`, *optional*): A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by `callback_on_step_end_tensor_inputs`. callback_on_step_end_tensor_inputs (`List`, *optional*): The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. max_sequence_length (`int`, defaults to `224`): Maximum sequence length in encoded prompt. Can be set to other values but may lead to poorer results. Examples: Returns: [`~pipelines.cogview4.pipeline_CogView4.CogView4PipelineOutput`] or `tuple`: [`~pipelines.cogview4.pipeline_CogView4.CogView4PipelineOutput`] if `return_dict` is True, otherwise a `tuple`. When returning a tuple, the first element is a list with the generated images. NFrr)r|r=r>rQr(rD)rrrrr|r(rRrarbg?base_image_seq_lenr?r?)r#)totalr^)rqencoder_hidden_statestimestepr target_size crop_coordsrr)rr<r=r>latent)rr)r)images)Crdrr tensor_inputsrBrK sample_sizerFrrrrrrerr4rgrcrr{ in_channelsrrArRencode latent_distsamplescaling_factorrrhfloat32tensorr patch_sizenplinspacer5num_train_timestepsarrayastypeint64r%getr9rmaxorder progress_bar enumeraterrorpexpandrsteplocalsr*popupdate XLA_AVAILABLExm mark_stepdecoderM postprocessmaybe_free_model_hooksr),rNrPrzrrrr'r)r*rr|rr<r=r>rrrrrrrrQrrr(latent_channelsvae_shift_factorr!r#transformer_dtypenum_warmup_stepsritlatent_model_inputrnoise_pred_condnoise_pred_uncond noise_predcallback_kwargsrcallback_outputsrs, r$__call__z CogView4ControlPipeline.__call__s+f *-=?U,V W1E1S1S .V4++22>>AVAVVT))00<\]    # # . . 1   KK--AA3H[ \)$  $$RXX.55bjjA JP.T^^22FFF^d   NN ! ! % %&:C @ NN ! ! % %lD 9 NN ! ! % %k4 8   *< NN/Fr* & &")n ,,22s9~0CdnnFZFZ0ZZ\]^   %8  96 #\!),5 #1>>)*&%*YY/GQ%O%R%RSd%e"88GMM!$45"&"2"2"4*7%"/ + 5%5 %#3 # #33(,(8(8&8.D!)&3$/$9)9$))9 ) )%"3T5H5HO^oLo5p!pJ!0J..--j!WRW-XYZ[(3&(O?9-3Xa[*9';D!T^^EZEZ[\E]_n'o$.229gFG$4$8$8-$XM-=-A-ABZ\r-s*I**A9I/IqSTuX\XfXfXlXlNlpqNq '') LLNk5 #6 #p"&h&jj0488??3Q3QQGHHOOG)OTUVWEE$$00K0P ##%8O%U33Q6 #6 #s9F,\;'\;;])NrONN)NTrNNNNrOr)FF)NN)3__name__ __module__ __qualname____doc___optional_componentsmodel_cpu_offload_seqrr r rrrrHr rerrrrhr(rRryboolrrrrrpropertyrr{rrrrno_gradrEXAMPLE_DOC_STRINGrfloat Generator FloatTensorrrrrrr rr __classcell__)r3s@r$r;r;sQ(<TY YY Y 0 Y 3 Y&)-#')-'+ )c49n%)!)& )  $ )^<@,0%&049=)-'+#'M5c49n%M5"%T#Y"78M5&* M5 # M5   - M5!) 6M5&M5 $M5!M5^6%* R#1f$$ ((##&&&&U]]_1237;?,0 $##%)-(, #%&MQ/359>B3717  59 9B #'3h4sDI~./h4"%T#Y"78h4* h4  h4 } h4!h4DI&h4e%h4h4 #h4E%//43H"HIJh4%++,h4  1 12h4!)):): ;h4  c3h0!h4" %S#X#h4$%h4&'h4(#4S>2)h4*' (Cd+T124DF\\ ] +h40-1I1h42!3h44 %u, -5h43h4r&r;)rrr)NNNN)4r-typingrrrrrrr numpyrrh transformersr r callbacksr rrMrrmodelsrrpipelines.pipeline_utilsr schedulersrutilsrrrutils.torch_utilsrpipeline_outputrtorch_xla.core.xla_modelcore xla_modelrr get_loggerrrkrrrr%rer(r9r;r r&r$rs DDD 0AD?99OO-3))MM   H %(    *.15%)$( @*!#@* U3 ,- .@*S " @* T%[ ! @*FQ 4/Q 4r&