bi ddlZddlmZmZmZmZmZmZmZddl Z ddl Z ddl Z ddlmcmZddlmZmZmZmZddlmZmZddlmZmZddlmZmZm Z m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)dd l*m+Z+dd l,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3dd l4m5Z5m6Z6m7Z7m8Z8d dl9m:Z:m;Z;d dlm?Z?e/r ddl@mAcmBZCdZDndZDe0jeFZGdZH ddeeIdeeeJe jfdeeeIdeeeLfdZMGdde:e;e!e eeZNy)N)AnyCallableDictListOptionalTupleUnion)CLIPImageProcessor CLIPTextModel CLIPTokenizerCLIPVisionModelWithProjection)MultiPipelineCallbacksPipelineCallback)PipelineImageInputVaeImageProcessor)FromSingleFileMixinIPAdapterMixinStableDiffusionLoraLoaderMixinTextualInversionLoaderMixin) AutoencoderKLControlNetModelImageProjectionMultiControlNetModelUNet2DConditionModel)adjust_lora_scale_text_encoder)KarrasDiffusionSchedulers)USE_PEFT_BACKEND deprecateis_torch_xla_availableloggingreplace_example_docstringscale_lora_layersunscale_lora_layers)empty_device_cacheis_compiled_moduleis_torch_version randn_tensor)DiffusionPipelineStableDiffusionMixin)StableDiffusionPipelineOutput)StableDiffusionSafetyCheckerTFa Examples: ```py >>> # !pip install opencv-python transformers accelerate >>> from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, UniPCMultistepScheduler >>> from diffusers.utils import load_image >>> import numpy as np >>> import torch >>> import cv2 >>> from PIL import Image >>> # download an image >>> image = load_image( ... "https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png" ... ) >>> image = np.array(image) >>> # get canny image >>> image = cv2.Canny(image, 100, 200) >>> image = image[:, :, None] >>> image = np.concatenate([image, image, image], axis=2) >>> canny_image = Image.fromarray(image) >>> # load control net and stable diffusion v1-5 >>> controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16) >>> pipe = StableDiffusionControlNetPipeline.from_pretrained( ... "stable-diffusion-v1-5/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16 ... ) >>> # speed up diffusion process with faster scheduler and memory optimization >>> pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config) >>> # remove following line if xformers is not installed >>> pipe.enable_xformers_memory_efficient_attention() >>> pipe.enable_model_cpu_offload() >>> # generate image >>> generator = torch.manual_seed(0) >>> image = pipe( ... "futuristic-looking woman", num_inference_steps=20, generator=generator, image=canny_image ... ).images[0] ``` num_inference_stepsdevice timestepssigmasc | | td|dttj|jj j v}|std|jd|jd ||d||j}t|}||fS|dttj|jj j v}|std|jd|jd ||d||j}t|}||fS|j|fd |i||j}||fS) a Calls the scheduler's `set_timesteps` method and retrieves timesteps from the scheduler after the call. Handles custom timesteps. Any kwargs will be supplied to `scheduler.set_timesteps`. Args: scheduler (`SchedulerMixin`): The scheduler to get timesteps from. num_inference_steps (`int`): The number of diffusion steps used when generating samples with a pre-trained model. If used, `timesteps` must be `None`. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. timesteps (`List[int]`, *optional*): Custom timesteps used to override the timestep spacing strategy of the scheduler. If `timesteps` is passed, `num_inference_steps` and `sigmas` must be `None`. sigmas (`List[float]`, *optional*): Custom sigmas used to override the timestep spacing strategy of the scheduler. If `sigmas` is passed, `num_inference_steps` and `timesteps` must be `None`. Returns: `Tuple[torch.Tensor, int]`: A tuple where the first element is the timestep schedule from the scheduler and the second element is the number of inference steps. zYOnly one of `timesteps` or `sigmas` can be passed. Please choose one to set custom valuesr0zThe current scheduler class zx's `set_timesteps` does not support custom timestep schedules. Please check whether you are using the correct scheduler.)r0r/r1zv's `set_timesteps` does not support custom sigmas schedules. Please check whether you are using the correct scheduler.)r1r/r/) ValueErrorsetinspect signature set_timesteps parameterskeys __class__r0len) schedulerr.r/r0r1kwargsaccepts_timesteps accept_sigmass m/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/pipelines/controlnet/pipeline_controlnet.pyretrieve_timestepsrBgs>!3tuu'3w/@/@AXAX/Y/d/d/i/i/k+ll .y/B/B.CDab  M)FMfM'' !)n ) ))   C(9(9):Q:Q(R(](](b(b(d$ee .y/B/B.CD_`  GvfGG'' !)n ) ))  3MFMfM'' ) ))c<eZdZdZdZgdZdgZgdZ dMded e d e d e d e e ee ee efd ededededeffd Z dNdeej2deej2deefdZ dOdeej2deej2deedeefdZdPdZdZdZ dZ!dZ" dQdZ#d Z$ dRd"Z%dPd#Z&d$ejNfd%ej2d&ed'ejPd(ej2fd)Z)e*d*Z+e*d+Z,e*d,Z-e*d-Z.e*d.Z/e*d/Z0ejbe2e3ddddd0ddd1dd2dddddddd3dddd!ddddd4gfd5e e4ee4fd6e5d7eed8eed9ed:eed;eedeed?ed@ee ejleejlfd4eej2deej2deej2dAee5dBeeej2dCee4dDedEee7e4e8fdFe eeefdGedHe eeefdIe eeefdeedJee e9eee7gdfe:e;fdKee4f6dLZ<xZ=S)S!StableDiffusionControlNetPipelinea Pipeline for text-to-image generation using Stable Diffusion with ControlNet guidance. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, running on a particular device, etc.). The pipeline also inherits the following loading methods: - [`~loaders.TextualInversionLoaderMixin.load_textual_inversion`] for loading textual inversion embeddings - [`~loaders.StableDiffusionLoraLoaderMixin.load_lora_weights`] for loading LoRA weights - [`~loaders.StableDiffusionLoraLoaderMixin.save_lora_weights`] for saving LoRA weights - [`~loaders.FromSingleFileMixin.from_single_file`] for loading `.ckpt` files - [`~loaders.IPAdapterMixin.load_ip_adapter`] for loading IP Adapters Args: vae ([`AutoencoderKL`]): Variational Auto-Encoder (VAE) model to encode and decode images to and from latent representations. text_encoder ([`~transformers.CLIPTextModel`]): Frozen text-encoder ([clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14)). tokenizer ([`~transformers.CLIPTokenizer`]): A `CLIPTokenizer` to tokenize text. unet ([`UNet2DConditionModel`]): A `UNet2DConditionModel` to denoise the encoded image latents. controlnet ([`ControlNetModel`] or `List[ControlNetModel]`): Provides additional conditioning to the `unet` during the denoising process. If you set multiple ControlNets as a list, the outputs from each ControlNet are added together to create one combined additional conditioning. scheduler ([`SchedulerMixin`]): A scheduler to be used in combination with `unet` to denoise the encoded image latents. Can be one of [`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`]. safety_checker ([`StableDiffusionSafetyChecker`]): Classification module that estimates whether generated images could be considered offensive or harmful. Please refer to the [model card](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) for more details about a model's potential harms. feature_extractor ([`~transformers.CLIPImageProcessor`]): A `CLIPImageProcessor` to extract features from generated images; used as inputs to the `safety_checker`. z&text_encoder->image_encoder->unet->vae)safety_checkerfeature_extractor image_encoderrF)latents prompt_embedsnegative_prompt_embedsimageNTvae text_encoder tokenizerunet controlnetr=rGrHrequires_safety_checkerc  t ||%| r#tjd|jd| | t dt |ttfr t|}|j|||||||||  t|ddr/dt|jjjdz znd|_t#|j d |_t#|j d d |_|j)| y)Nz)You have disabled the safety checker for a by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .zMake sure to define a feature extractor when loading {self.__class__} if you want to use the safety checker. If you do not want to use the safety checker, you can pass `'safety_checker=None'` instead.) rMrNrOrPrQr=rFrGrHrMr)T)vae_scale_factordo_convert_rgbF)rVrW do_normalize)rR)super__init__loggerwarningr;r4 isinstancelisttuplerregister_modulesgetattrr<rMconfigblock_out_channelsrVrimage_processorcontrol_image_processorregister_to_config) selfrMrNrOrPrQr=rFrGrHrRr;s rArZz*StableDiffusionControlNetPipeline.__init__s'   !&= NN;DNN;KLjj   %*;*Cx  j4- 0-j9J %!)/'  W^^bdikoVpc$((//*L*L&MPQ&Q Rvw0$BWBWhlm'8!224V[( $ 8OPrCrJrK lora_scalec d} tdd| d|jd ||||||||d| } tj| d| dg}|S) Nz`_encode_prompt()` is deprecated and it will be removed in a future version. Use `encode_prompt()` instead. Also, be aware that the output format changed from a concatenated tensor to a tuple.z_encode_prompt()1.0.0F standard_warn)promptr/num_images_per_promptdo_classifier_free_guidancenegative_promptrJrKrhrTrr3)r encode_prompttorchcat) rgrmr/rnrorprJrKrhr>deprecation_messageprompt_embeds_tuples rA_encode_promptz0StableDiffusionControlNetPipeline._encode_prompt s}a$g/BRWX0d00  "7(C+'#9!     #6q#9;Nq;Q"RS rC clip_skipc H |Jt|tr:||_tst |j |nt |j ||t|trd} n-|t|tr t|} n|jd} |t|tr|j||j}|j|d|jjdd} | j} |j|dd j} | jd | jd k\rt!j"| | sj|jj%| dd|jjdz d f}t&j)d |jjd |t+|j j,d r<|j j,j.r| j0j3|}nd}| (|j | j3||}|d}nT|j | j3||d}|d | dz }|j j4j7|}|j |j j8}n/|j:|j:j8}n |j8}|j3||}|j\}}}|j=d|d}|j?||z|d }|rm|j|dg| z}n|:tA|tA|ur$tCdtA|dtA|dt|tr|g}n1| t|k7r!tEd|dt|d|d| d |}t|tr|j||j}|jd}|j|d|dd}t+|j j,d r<|j j,j.r|j0j3|}nd}|j |jj3||}|d}|rK|jd}|j3||}|j=d|d}|j?| |z|d }|j ,t|trtrtG|j |||fS)a Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*): prompt to be encoded device: (`torch.device`): torch device num_images_per_prompt (`int`): number of images that should be generated per prompt do_classifier_free_guidance (`bool`): whether to use classifier free guidance or not negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. lora_scale (`float`, *optional*): A LoRA scale that will be applied to all LoRA layers of the text encoder if LoRA layers are loaded. clip_skip (`int`, *optional*): Number of layers to be skipped from CLIP while computing the prompt embeddings. A value of 1 means that the output of the pre-final layer will be used for computing the prompt embeddings. NrTr max_lengthTpt)paddingry truncationreturn_tensorslongest)r{r}z\The following part of your input was truncated because CLIP can only handle sequences up to z tokens: use_attention_mask)attention_mask)routput_hidden_states)dtyper/z?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.)$r]r _lora_scalerrrNr#strr^r<shapermaybe_convert_promptrOmodel_max_length input_idsrrequal batch_decoder[r\hasattrrbrrto text_modelfinal_layer_normrrPrepeatviewtype TypeErrorr4r$)rgrmr/rnrorprJrKrhrw batch_size text_inputstext_input_idsuntruncated_ids removed_textrprompt_embeds_dtypebs_embedseq_len_ uncond_tokensry uncond_inputs rArqz/StableDiffusionControlNetPipeline.encode_prompt*s6V  !j7U&V)D $.t/@/@*M!$"3"3Z@  *VS"9J  Jvt$<VJ&,,Q/J  $ ;<2264>>J..$>>::# )K)22N"nnVYW[n\ffO$$R(N,@,@,DDU[[N $~~::#At~~'F'F'JR'O$OP  778 ,Q t((//1EF4K\K\KcKcKvKv!,!;!;!>!>v!F!% $ 1 1.2C2CF2K\j 1 k -a 0 $ 1 1"%%f-ncg!2! !.b 1IM2B C !% 1 1 < < M Mm \    ("&"3"3"9"9  YY ""&))// "/"5"5 %((/B6(R ,22'1%,,Q0EqI %**86K+KWVXY  '+A+I&!#z 1 #V D>$%# *Lt((//1EF4K\K\KcKcKvKv!-! " &,2215G%;%>%>EXag%>%h "%;%B%B1F[]^%_ "%;%@%@NcAcelnp%q "    ($ >?DT#D$5$5zB444rCc|t|jjj}t |t j s|j|dj}|j||}|r}|j|djd}|j|d}|jt j|djd}|j|d}||fS|j|j}|j|d}t j|} || fS) Nrzr}r/rT)rrdim)nextrHr9rr]rrTensorrG pixel_valuesr hidden_statesrepeat_interleave zeros_like image_embeds) rgrLr/rnrrimage_enc_hidden_statesuncond_image_enc_hidden_statesruncond_image_embedss rA encode_imagez.StableDiffusionControlNetPipeline.encode_imagesDT''2245;;%.**5*FSSEe4 &*&8&8UY&8&Z&h&hik&l #&=&O&OPekl&O&m #-1-?-?  'd.@.mB. *.L-]-]%1.^. *+,JJ J--e4AAL'99:OUV9WL"'"2"2<"@ !44 4rCcg}|rg}|t|ts|g}t|t|jjj k7rBt dt|dt|jjj dt||jjj D]`\}} t| t } |j||d| \} } |j| dddf|sIj| dddfbn?|D]:} |r%| jd\} } j| |j| <g}t|D]|\} } tj| g|zd} |r7tj| g|zd} tj| | gd} | j|} |j| ~|S) NzK`ip_adapter_image` must have same length as the number of IP Adapters. Got images and z IP Adapters.rTr)rr)r/)r]r^r<rPencoder_hid_projimage_projection_layersr4ziprrappendchunk enumeraterrrsr)rgip_adapter_imageip_adapter_image_embedsr/rnrornegative_image_embedssingle_ip_adapter_imageimage_proj_layeroutput_hidden_statesingle_image_embedssingle_negative_image_embedsis rAprepare_ip_adapter_image_embedszAStableDiffusionControlNetPipeline.prepare_ip_adapter_image_embedssY &$& ! " *.5$4#5 #$DII,F,F,^,^(__ abefvbwaxyEFIJNJSJSJdJdJ|J|F}E~~KL>A $))"<"<"T"T> X9')9+55E*W&W#DHDUDU+VQ8KEA#%A##$7a$@A.)001MdTUg1VW X(? 9#.H[HaHabcHdE02E)001MN##$78  9 #%&/ &= @ "A""'))-@,ADY,Y_`"a */4yy:OPQ:R9SVk9kqr/s,&+ii1MOb0cij&k#"5"8"8"8"G  # * *+> ? @'&rCcl|jd}||fStj|r|jj |d}n|jj |}|j |dj|}|j||jj|\}}||fS)Npil) output_typerzr)images clip_input) rFrr is_tensorrd postprocess numpy_to_pilrGrr)rgrLr/rhas_nsfw_conceptfeature_extractor_inputsafety_checker_inputs rArun_safety_checkerz4StableDiffusionControlNetPipeline.run_safety_checker(s    &# &&&u%*.*>*>*J*J5^c*J*d'*.*>*>*K*KE*R'#'#9#9:Qbf#9#g#j#jkq#r &*&9&9)=)J)J)M)Me)T':' #E#&&&rCc`d}tdd|dd|jjjz |z}|jj |dd}|d z d zj dd}|j jdd d djj}|S) Nz{The decode_latents method is deprecated and will be removed in 1.0.0. Please use VaeImageProcessor.postprocess(...) insteaddecode_latentsrjFrkrT) return_dictrr)g?r) rrMrbscaling_factordecodeclampcpupermutefloatnumpy)rgrIrtrLs rArz0StableDiffusionControlNetPipeline.decode_latents7s\"G-@PUVdhhoo444w>U;A>S''1- ##Aq!Q/557==? rCcVdttj|jjj j v}i}|r||d<dttj|jjj j v}|r||d<|S)Neta generator)r5r6r7r=stepr9r:)rgrr accepts_etaextra_step_kwargsaccepts_generators rAprepare_extra_step_kwargsz;StableDiffusionControlNetPipeline.prepare_extra_step_kwargsCs s7#4#4T^^5H5H#I#T#T#Y#Y#[\\  '* e $(3w/@/@ATAT/U/`/`/e/e/g+hh -6 k *  rC?c  |0t|tr|dkrtd|dt|d| Lt fd| Ds8tdj d| D cgc]} | j vs| c} ||td|d |d | | td |7t|t s't|tstd t|||td |d|d |A|?|j|jk7r&td|jd|jdttdxr8tjtjjj}tjt s&|r9tjj"t rj%|||ntjt&s(|rltjj"t&rGt|ts t)dt+d|Drt-|Dcgc] }t|}}t/|t/jj0k7r8tdt/|dt/jj0d|D]}j%|||nt/|t/jj0k7r8tdt/|dt/jj0d|D]}j%|||nJtjt s&|r?tjj"t rt| t2st)dtjt&s&|rtjj"t&rst| trt+d| DrStdt| tr8t/| t/jj0k7r tdJt| t4tfs| g} t| t4tfs| g} t/| t/| k7r$tdt/| dt/| dtjt&rt/| t/jj0k7r[td| d t/| d!t/jj0d"t/jj0d t-| | D]D\}}||k\rtd#|d$|d|d%krtd#|d&|d'kDs7td(|d)| | td*|Ut|tstd+t||dj6d,vrtd-|dj6d.yycc} wcc}w)/Nrz5`callback_steps` has to be a positive integer but is z of type rc3:K|]}|jvywN)_callback_tensor_inputs).0krgs rA zAStableDiffusionControlNetPipeline.check_inputs..is#F 23A-- -F sz2`callback_on_step_end_tensor_inputs` has to be in z , but found zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is z'Cannot forward both `negative_prompt`: z and `negative_prompt_embeds`: zu`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but got: `prompt_embeds` z != `negative_prompt_embeds` scaled_dot_product_attentionz5For multiple controlnets: `image` must be type `list`c3<K|]}t|tywrr]r^rrs rArzAStableDiffusionControlNetPipeline.check_inputs..s8QZ4(8zFor multiple controlnets: if you pass`image` as a list of list, each sublist must have the same length as the number of controlnets, but the sublists in `image` got rz ControlNets.zbFor multiple controlnets: `image` must have the same length as the number of controlnets, but got zLFor single controlnet: `controlnet_conditioning_scale` must be type `float`.c3<K|]}t|tywrrrs rArzAStableDiffusionControlNetPipeline.check_inputs..sRqz!T*RrzA single batch of varying conditioning scale settings (e.g. [[1.0, 0.5], [0.2, 0.8]]) is not supported at the moment. The conditioning scale must be fixed across the batch.zFor multiple controlnets: When `controlnet_conditioning_scale` is specified as `list`, it must have the same length as the number of controlnetsz`control_guidance_start` has z* elements, but `control_guidance_end` has zI elements. Make sure to provide the same number of elements to each list.z`control_guidance_start`: z has z elements but there are z- controlnets available. Make sure to provide zcontrol guidance start: z4 cannot be larger or equal to control guidance end: rz can't be smaller than 0.rzcontrol guidance end: z can't be larger than 1.0.zProvide either `ip_adapter_image` or `ip_adapter_image_embeds`. Cannot leave both `ip_adapter_image` and `ip_adapter_image_embeds` defined.z:`ip_adapter_image_embeds` has to be of type `list` but is )rzF`ip_adapter_image_embeds` has to be a list of 3D or 4D tensors but is D)r]intr4rallrrr^rrFrQrr_dynamo eval_frameOptimizedModuler _orig_mod check_imagerranyrr<netsrr_ndim)rgrmrLcallback_stepsrprJrKrrcontrolnet_conditioning_scalecontrol_guidance_startcontrol_guidance_end"callback_on_step_end_tensor_inputsr is_compiledttransposed_imageimage_startends` rA check_inputsz.StableDiffusionControlNetPipeline.check_inputsTs  %z.#/NR`deReGGWX(),  . 9#F 7YF C DTEaEaDbbn|^pHvwbckokGkGbGpqpHoIJ   -";08N}o^00 ^ 5w  FC)@TZ\`IaQRVW]R^Q_`a a  &+A+M9/9JK*++]_   $)?)K""&<&B&BB --:-@-@,AB.445Q8a!?@ Z OOU]]55EEF  t 84??44oF   UFM : t(< =4??446JKeT* WXX8%8858%[#ADG#A #A'(C0D0D,EE$@ADEUAV@WWcdghlhwhwh|h|d}c~~KL/DF$$VV]CDUs4??#7#788 xy|~CzDyEEQRUVZVeVeVjVjRkQllyz$DF$$VV]CD 5 t 84??44oF;UC noo t(< =4??446JK7>R4QRR$Q94@SIfEgkn$$lF!D 505$-@&<%= ". >$8#9 % &#.B*C C/4J0K/LLvwz|PxQwRR[\  doo'; <)*c$//2F2F.GG 01G0HcRhNiMjkCDGHLHWHWH\H\D]C^^KLOPTP_P_PdPdLeKffgh46JK [JE3| .ug5ijminnops{ #;E7B[!\]]Sy #9#>X!YZZ [  ',C,O^  # .5t< PQUVmQnPop)+00> \]tuv]w]|]|\}}~? /ApHd$Bs!X:5X:X?ct|tjj}t|tj}t|t j }t|txr't|dtjj}t|txrt|dtj}t|txrt|dt j } |s!|s|s|s|s| stdt||rd} n t|} |t|trd} n/|t|tr t|} n||jd} | dk7r|  k7rtd| d| yy)Nrzimage must be passed and be one of PIL image, numpy array, torch tensor, list of PIL images, list of numpy arrays or list of torch tensors, but is rTzdIf image batch size is not 1, image batch size must be same as prompt batch size. image batch size: z, prompt batch size: )r]PILImagerrrnpndarrayr^rrr<rrr4) rgrLrmrJ image_is_pilimage_is_tensor image_is_npimage_is_pil_listimage_is_tensor_listimage_is_np_listimage_batch_sizeprompt_batch_sizes rArz-StableDiffusionControlNetPipeline.check_images!%9 $UELL9  3 &ud3] 58SYY__8])%6]:eAhPUP\P\;]%eT2Wz%(BJJ7W#%($fgklqgrfst   "5z   *VS"9 !   Jvt$< #F   & - 3 3A 6  q %59J%JvxHwII^_p^qr &K rCFc 6|jj|||jtj}|j d} | dk(r|} n|} |j | d}|j||}|r| stj|gdz}|S)N)heightwidthrrrTrrr))re preprocessrrrfloat32rrrs) rgrLrrrrnr/rro guess_moder repeat_bys rA prepare_imagez/StableDiffusionControlNetPipeline.prepare_images,,77fTY7Z]]didqdq]r ;;q> q "I.I'' q'9e4 &zIIugk*E rCc T||t||jzt||jzf} t|tr)t ||k7rt dt |d|d|t | |||}n|j|}||jjz}|S)Nz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.)rr/r) rrVr]r^r<r4r(rr=init_noise_sigma) rgrnum_channels_latentsrrrr/rrIrs rAprepare_latentsz1StableDiffusionControlNetPipeline.prepare_latents;s  K400 0 J$// /   i &3y>Z+GA#i.AQR&<'gi  ?"5IfTYZGjj(GDNN;;;rCiw embedding_dimrreturnclt|jdk(sJ|dz}|dz}tjtjd|dz z }tj tj ||| z}|j|dddf|dddfz}tjtj|tj|gd}|dzdk(r*tjjj|d}|j|jd |fk(sJ|S) a See https://github.com/google-research/vdm/blob/dc27b98a554f65cdc654b800da5aa1846545d41b/model_vdm.py#L298 Args: w (`torch.Tensor`): Generate embedding vectors with a specified guidance scale to subsequently enrich timestep embeddings. embedding_dim (`int`, *optional*, defaults to 512): Dimension of the embeddings to generate. dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): Data type of the generated embeddings. Returns: `torch.Tensor`: Embedding vectors with shape `(len(w), embedding_dim)`. rTg@@r)g@rNr)rrTr)r<rrrlogtensorexparangerrssincosnn functionalpad)rgr$r%rhalf_dimembs rAget_guidance_scale_embeddingz>StableDiffusionControlNetPipeline.get_guidance_scale_embeddingRs"177|q   J A%ii W-.(Q,?ii XU;sdBCdd5k!T'"Sq\1ii338a@ 1  !((%%))#v6CyyQWWQZ7777 rCc|jSr)_guidance_scalergs rAguidance_scalez0StableDiffusionControlNetPipeline.guidance_scaleps###rCc|jSr) _clip_skipr6s rArwz+StableDiffusionControlNetPipeline.clip_skipt rCch|jdkDxr"|jjjduS)NrT)r5rPrbtime_cond_proj_dimr6s rAroz=StableDiffusionControlNetPipeline.do_classifier_free_guidance{s.##a'WDII,<,<,O,OSW,WWrCc|jSr)_cross_attention_kwargsr6s rAcross_attention_kwargsz8StableDiffusionControlNetPipeline.cross_attention_kwargss+++rCc|jSr)_num_timestepsr6s rA num_timestepsz/StableDiffusionControlNetPipeline.num_timestepss"""rCc|jSr) _interruptr6s rA interruptz+StableDiffusionControlNetPipeline.interruptr:rC2g@rTrrIrmrLrrr.r0r1r7rprnrrrrrrr?rrrrcallback_on_step_endrc j|jdd}|jdd}| tddd| tdddt|ttfr |j }t |jr|jjn |j}t|ts t|trt||gz}nt|ts t|trt||gz}nSt|tsCt|ts3t|trt|jnd} | |gz| |gz}}|j|||| |||||||| ||_||_||_d|_|t|t$rd}!n-|t|tr t|}!n|j&d }!|j(}"t|tr)t|t*r|gt|jz}t|t,r|j.j0n"|jd j.j0}#|xs|#}|j2|j2j5d dnd}$|j7||"| |j8| |||$|j: \}}|j8rt=j>||g}||"|jA|||"|!| z|j8}%t|t,rD|jC||||!| z| |"|jD|j8| }|j&d d\}}nt|trg}&t|d trtG|D'cgc] }'t|'}}'|D]D}(|jC|(|||!| z| |"|jD|j8| }(|&jI|(F|&}|d j&d d\}}nJtK|jL||"||\}}t||_'|jPj.jR})|jU|!| z|)|||jD|"| | } d}*|jPj.jVt=jX|jZdz j]|!| z}+|j_|+|jPj.jVja|"| jD}*|jc| | },||d%ind}-g}.tet|D]w}/tG||D01cgc]8\}0}1dt+|/t|z |0kxs|/dzt|z |1kDz :}2}0}1|.jIt|t,r|2d n|2yt|||jLjfzz }3t |jP}4t |j}5tidd}6|jk|5}7tm|D]"\}/}'|jnr|4r"|5r |6rt<jpjs|j8rt=j>| gdzn| }8|jLju|8|'}8|r?|j8r3| }9|jLju|9|'}9|jwdd}:n|8}9|}:t|.|/tr%tG||.|/D;0cgc] \};}0|;|0z }<};}0n|}=t|=tr|=d }=|=|.|/z}<|j|9|'|:||<|d\}>}?|rm|j8ra|>D@cgc],}@t=j>t=jx|@|@g.}>}@t=j>t=jx|?|?g}?|jQ|8|'||*|j2|>|?|-d d }A|j8r)Ajwd\}B}C|B|jZ|C|Bz zz}A|jLjzA|'| fi|,ddid } |li}D|D]}Et}|ED|E<|||/|'D}F|Fjd| } |Fjd|}|Fjd|}|Fjd|}|/t|dz k(s'|/dz|3kDr]|/dz|jLjfzd k(r>|7j|,|/|zd k(r$|/t|jLddz}G||G|'| tstj% dddt|drL|j@|jPjad|jjadt|d k(sc|jj| |jj.jz d| !d }|j||"|jD\}}Hn| }d}HHd"g|j&d z}InHDJcgc]}J|J }I}J|jj||I#}|j|s|HfSt|H$Scc}'wcc}1}0wcc}0};wcc}@w#1swYKxYwcc}Jw)%uO$ The call function to the pipeline for generation. Args: prompt (`str` or `List[str]`, *optional*): The prompt or prompts to guide image generation. If not defined, you need to pass `prompt_embeds`. image (`torch.Tensor`, `PIL.Image.Image`, `np.ndarray`, `List[torch.Tensor]`, `List[PIL.Image.Image]`, `List[np.ndarray]`,: `List[List[torch.Tensor]]`, `List[List[np.ndarray]]` or `List[List[PIL.Image.Image]]`): The ControlNet input condition to provide guidance to the `unet` for generation. If the type is specified as `torch.Tensor`, it is passed to ControlNet as is. `PIL.Image.Image` can also be accepted as an image. The dimensions of the output image defaults to `image`'s dimensions. If height and/or width are passed, `image` is resized accordingly. If multiple ControlNets are specified in `init`, images must be passed as a list such that each element of the list can be correctly batched for input to a single ControlNet. When `prompt` is a list, and if a list of images is passed for a single ControlNet, each will be paired with each prompt in the `prompt` list. This also applies to multiple ControlNets, where a list of image lists can be passed to batch for each prompt and each ControlNet. height (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`): The height in pixels of the generated image. width (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`): The width in pixels of the generated image. num_inference_steps (`int`, *optional*, defaults to 50): The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. timesteps (`List[int]`, *optional*): Custom timesteps to use for the denoising process with schedulers which support a `timesteps` argument in their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is passed will be used. Must be in descending order. sigmas (`List[float]`, *optional*): Custom sigmas to use for the denoising process with schedulers which support a `sigmas` argument in their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is passed will be used. guidance_scale (`float`, *optional*, defaults to 7.5): A higher guidance scale value encourages the model to generate images closely linked to the text `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`. negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts to guide what to not include in image generation. If not defined, you need to pass `negative_prompt_embeds` instead. Ignored when not using guidance (`guidance_scale < 1`). num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. eta (`float`, *optional*, defaults to 0.0): Corresponds to parameter eta (η) from the [DDIM](https://huggingface.co/papers/2010.02502) paper. Only applies to the [`~schedulers.DDIMScheduler`], and is ignored in other schedulers. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor is generated by sampling using the supplied random `generator`. prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not provided, text embeddings are generated from the `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not provided, `negative_prompt_embeds` are generated from the `negative_prompt` input argument. ip_adapter_image: (`PipelineImageInput`, *optional*): Optional image input to work with IP Adapters. ip_adapter_image_embeds (`List[torch.Tensor]`, *optional*): Pre-generated image embeddings for IP-Adapter. It should be a list of length same as number of IP-adapters. Each element should be a tensor of shape `(batch_size, num_images, emb_dim)`. It should contain the negative image embedding if `do_classifier_free_guidance` is set to `True`. If not provided, embeddings are computed from the `ip_adapter_image` input argument. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generated image. Choose between `PIL.Image` or `np.array`. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] instead of a plain tuple. callback (`Callable`, *optional*): A function that calls every `callback_steps` steps during inference. The function is called with the following arguments: `callback(step: int, timestep: int, latents: torch.Tensor)`. callback_steps (`int`, *optional*, defaults to 1): The frequency at which the `callback` function is called. If not specified, the callback is called at every step. cross_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the [`AttentionProcessor`] as defined in [`self.processor`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). controlnet_conditioning_scale (`float` or `List[float]`, *optional*, defaults to 1.0): The outputs of the ControlNet are multiplied by `controlnet_conditioning_scale` before they are added to the residual in the original `unet`. If multiple ControlNets are specified in `init`, you can set the corresponding scale as a list. guess_mode (`bool`, *optional*, defaults to `False`): The ControlNet encoder tries to recognize the content of the input image even if you remove all prompts. A `guidance_scale` value between 3.0 and 5.0 is recommended. control_guidance_start (`float` or `List[float]`, *optional*, defaults to 0.0): The percentage of total steps at which the ControlNet starts applying. control_guidance_end (`float` or `List[float]`, *optional*, defaults to 1.0): The percentage of total steps at which the ControlNet stops applying. clip_skip (`int`, *optional*): Number of layers to be skipped from CLIP while computing the prompt embeddings. A value of 1 means that the output of the pre-final layer will be used for computing the prompt embeddings. callback_on_step_end (`Callable`, `PipelineCallback`, `MultiPipelineCallbacks`, *optional*): A function or a subclass of `PipelineCallback` or `MultiPipelineCallbacks` that is called at the end of each denoising step during the inference. with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by `callback_on_step_end_tensor_inputs`. callback_on_step_end_tensor_inputs (`List`, *optional*): The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. Examples: Returns: [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] is returned, otherwise a `tuple` is returned where the first element is a list with the generated images and the second element is a list of `bool`s indicating whether the corresponding generated image contains "not-safe-for-work" (nsfw) content. callbackNrrjzjPassing `callback` as an input argument to `__call__` is deprecated, consider using `callback_on_step_end`zpPassing `callback_steps` as an input argument to `__call__` is deprecated, consider using `callback_on_step_end`rTFrscale)rJrKrhrw) rLrrrrnr/rrorr)r%rrrz>=z2.1)totalr))encoder_hidden_statescontrolnet_condconditioning_scalerr)rL timestep_condr?down_block_additional_residualsmid_block_additional_residualadded_cond_kwargsrrrIrJrKrLorderfinal_offload_hookrlatent)rrT)rdo_denormalize)rnsfw_content_detected)Opoprr]rr tensor_inputsr&rQrr^r<rrr r5r9r>rDrr_execution_devicerrrbglobal_pool_conditionsr?getrqrorwrrrsrrrrrrBr=rArP in_channelsr#r<r)r7rr3rrrangerSr' progress_barrrE _inductorcudagraph_mark_step_beginscale_model_inputrrrlocalsupdatera XLA_AVAILABLExm mark_steprrTr%rMrrrrdrmaybe_free_model_hooksr,)KrgrmrLrrr.r0r1r7rprnrrrIrJrKrrrrr?rrrrrwrGrr>rIrrQmultrr/r[text_encoder_lora_scalerrrrr"rOguidance_scale_tensorrrRcontrolnet_keeprsekeepsnum_warmup_stepsis_unet_compiledis_controlnet_compiledis_torch_higher_equal_2_1r_latent_model_inputcontrol_model_inputcontrolnet_prompt_embedsc cond_scalecontrolnet_cond_scaledown_block_res_samplesmid_block_res_sampled noise_prednoise_pred_uncondnoise_pred_textcallback_kwargsrcallback_outputsstep_idxrrVhas_nsfwsK rA__call__z*StableDiffusionControlNetPipeline.__call__sJ `::j$/$4d;   |   %  C  *-=?U,V W1E1S1S .2DT__2UT__..[_[j[j 0$7JG[]aBXAY%Y "0$7JG]_cBVAW#W 2D9*MacgBh+5jBV+W3z']^D.//,--%9 "       "  # ) " .  .#'=$  *VS"9J  Jvt$<VJ&,,Q/J'' j"6 7JGdfkY>Y>eD ' ' + +GT :ko 150B0B   !  , , '#9.nn1C 1 - -  + +!II'=}&MNM  '+B+N?? '2200 L j/ 2&&%(==&; &&,0,L,L%' E"KK,MFE  $8 9F%(D)*-u+6Qa66 &++ !),AA*?!$**040P0P),  f% &E!!HNN23/MFE 5*< NN/F* & &")n $yy//;;&& . .           99   . . :$)LL1D1Dq1H$I$P$PQ[^sQs$t ! ==%TYY5E5E5X5X>bgmmb4  !::9cJ  +/F/R\ * s9~& cA 68LMAqeAI.2Rq1uI6NQR6RSSE  " "z*o/V58\a b  cy>,?$..BVBV,VV-dii8!3DOO!D$4T5$A!   %8  9X #\!),W #1>>%)?E^OO==?AEAaAaUYYy1}%=gn"%)^^%E%EFXZ[%\"$"B"B*1'*...*J*JK^`a*b'/0Ea0H-!69K!KJ?C'*B$)'1) %@O@<&(<$"B"B\r-rVWeii9I9I!9La8P.Q-r*-r+099e6F6FG[6\^r5s+t("YY&*7"/+/+F+F4J2F&7 %'   339C9I9I!9L6%!2T5H5HO^oLo5p!pJ.$..--j!WmHYmglmnop'3&(O?9-3Xa[*9';D!Q'X$.229gFG$4$8$8-$XM-=-A-ABZ\r-s*,00%@EI**A9I/IqSTuX\XfXfXlXlNlpqNq '')+N0Ba0G#$(K#K 1g6 LLNoW #X #v 4- .43J3J3V IILL  OO  u %  h&HHOOGdhhoo.L.L$LZ_ktOuE'+&=&=eV]M`M`&a #E#E#   #"Vekk!n4N;KLx(lLNL$$00K`n0o ##%+, ,,EQabb7|F"l*.s[X #X #XMsEj =j'C1j#j 'Aj#71j(Fj#j# j0 j##j-)NTNNNN)NNNNNr) NNNNNrrrN)FF)>__name__ __module__ __qualname____doc__model_cpu_offload_seq_optional_components_exclude_from_cpu_offloadrrr r rr rrrrrr-r r boolrZrrrrrrvrrqrrrrrr rrr#rrr3propertyr7rwror?rBrEno_gradr"EXAMPLE_DOC_STRINGrr Generatorrrrrrr __classcell__)r;s@rArErEs#JES!1 2]8<(,2Q 2Q$2Q! 2Q # 2Q /4+@%BXZnno 2Q-2Q52Q.2Q52Q"&2Qv049=&*  -!) 6UON049=&*#'t5  -t5!) 6t5UOt5C=t5n52+'\ ' !,# $&)" +/aF#\%*>058emm.1@E  <$$ XX,,##U]]_12)-$( $##%#" #;?/0MQ*.049=9=@D%* ;?CF rs DDD hhADwwqq93hgDLJ))MM   H %+b*.15%)$( 8*!#8* U3 ,- .8*S " 8* T%[ ! 8*vtc" tcrC