bi ddlZddlmZmZmZmZmZmZmZddl Z ddl Z ddl Z ddlmcmZddlmZmZmZmZddlmZmZddlmZmZddlmZmZm Z m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)dd l*m+Z+dd l,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3dd l4m5Z5m6Z6m7Z7d dl8m9Z9m:Z:d dl;mZ>e/r ddl?m@cmAZBdZCndZCe0jeEZFdZG dde jdee jdeJfdZKdZLGdde9e:e!e eeZMy)N)AnyCallableDictListOptionalTupleUnion)CLIPImageProcessor CLIPTextModel CLIPTokenizerCLIPVisionModelWithProjection)MultiPipelineCallbacksPipelineCallback)PipelineImageInputVaeImageProcessor)FromSingleFileMixinIPAdapterMixinStableDiffusionLoraLoaderMixinTextualInversionLoaderMixin) AutoencoderKLControlNetModelImageProjectionMultiControlNetModelUNet2DConditionModel)adjust_lora_scale_text_encoder)KarrasDiffusionSchedulers)USE_PEFT_BACKEND deprecateis_torch_xla_availableloggingreplace_example_docstringscale_lora_layersunscale_lora_layers)empty_device_cacheis_compiled_module randn_tensor)DiffusionPipelineStableDiffusionMixin)StableDiffusionPipelineOutput)StableDiffusionSafetyCheckerTFa  Examples: ```py >>> # !pip install opencv-python transformers accelerate >>> from diffusers import StableDiffusionControlNetImg2ImgPipeline, ControlNetModel, UniPCMultistepScheduler >>> from diffusers.utils import load_image >>> import numpy as np >>> import torch >>> import cv2 >>> from PIL import Image >>> # download an image >>> image = load_image( ... "https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png" ... ) >>> np_image = np.array(image) >>> # get canny image >>> np_image = cv2.Canny(np_image, 100, 200) >>> np_image = np_image[:, :, None] >>> np_image = np.concatenate([np_image, np_image, np_image], axis=2) >>> canny_image = Image.fromarray(np_image) >>> # load control net and stable diffusion v1-5 >>> controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16) >>> pipe = StableDiffusionControlNetImg2ImgPipeline.from_pretrained( ... "stable-diffusion-v1-5/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16 ... ) >>> # speed up diffusion process with faster scheduler and memory optimization >>> pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config) >>> pipe.enable_model_cpu_offload() >>> # generate image >>> generator = torch.manual_seed(0) >>> image = pipe( ... "futuristic-looking woman", ... num_inference_steps=20, ... generator=generator, ... image=image, ... control_image=canny_image, ... ).images[0] ``` encoder_output generator sample_modect|dr |dk(r|jj|St|dr|dk(r|jjSt|dr |jSt d)N latent_distsampleargmaxlatentsz3Could not access latents of provided encoder_output)hasattrr1r2moder4AttributeError)r-r.r/s u/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/pipelines/controlnet/pipeline_controlnet_img2img.pyretrieve_latentsr9gst~}-+2I))00;;  /K84K))..00  +%%%RSSc t|tjrB|jdk(r|j d}|j tj }|St|tjjtjfr|g}t|trwt|dtjjrP|Dcgc]-}tj|jddddf/}}tj|d}nYt|trIt|dtjr,tj|Dcgc] }|dddf c}d}|jdddd}tj |j tj dz d z }|Scc}wcc}w) NrrdtypeRGB)axisr(g_@?) isinstancetorchTensorndim unsqueezetofloat32PILImagenpndarraylistarrayconvert concatenate transpose from_numpy)imageis r8 prepare_imagerUtsS%& ::?OOA&Eu}}- L eciioorzz: ;GE eT "z%(CIIOO'LBGHQRXXaii./a8HEHNN5q1E t $E!Hbjj)INN#>1AdAgJ#>QGE1a+  '***?%G#M LI$?s 2GGc<FeZdZdZdZgdZdgZgdZ dIded e d e d e d e e ee ee efd ededededeffd Z dJdeej2deej2deefdZ dKdeej2deej2deedeefdZdLdZdZdZ dZ!dZ" dMdZ#d Z$ dNd"Z%d#Z&dLd$Z'e(d%Z)e(d&Z*e(d'Z+e(d(Z,e(d)Z-e(d*Z.ej^e0e1dddddd+d,d-dd.dddddddd/ddd+d!ddddd0gfd1e e2ee2fd2e3d3e3d4eed5eed6ed7ed8ed9ee e2ee2fd:eed;edeeej2d?ee2d@edAee5e2e6fdBe eeefdCedDe eeefdEe eeefdeedFee e7eee5gdfe8e9fdGee2f6dHZ:xZ;S)O(StableDiffusionControlNetImg2ImgPipelinea Pipeline for image-to-image generation using Stable Diffusion with ControlNet guidance. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, running on a particular device, etc.). The pipeline also inherits the following loading methods: - [`~loaders.TextualInversionLoaderMixin.load_textual_inversion`] for loading textual inversion embeddings - [`~loaders.StableDiffusionLoraLoaderMixin.load_lora_weights`] for loading LoRA weights - [`~loaders.StableDiffusionLoraLoaderMixin.save_lora_weights`] for saving LoRA weights - [`~loaders.FromSingleFileMixin.from_single_file`] for loading `.ckpt` files - [`~loaders.IPAdapterMixin.load_ip_adapter`] for loading IP Adapters Args: vae ([`AutoencoderKL`]): Variational Auto-Encoder (VAE) model to encode and decode images to and from latent representations. text_encoder ([`~transformers.CLIPTextModel`]): Frozen text-encoder ([clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14)). tokenizer ([`~transformers.CLIPTokenizer`]): A `CLIPTokenizer` to tokenize text. unet ([`UNet2DConditionModel`]): A `UNet2DConditionModel` to denoise the encoded image latents. controlnet ([`ControlNetModel`] or `List[ControlNetModel]`): Provides additional conditioning to the `unet` during the denoising process. If you set multiple ControlNets as a list, the outputs from each ControlNet are added together to create one combined additional conditioning. scheduler ([`SchedulerMixin`]): A scheduler to be used in combination with `unet` to denoise the encoded image latents. Can be one of [`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`]. safety_checker ([`StableDiffusionSafetyChecker`]): Classification module that estimates whether generated images could be considered offensive or harmful. Please refer to the [model card](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) for more details about a model's potential harms. feature_extractor ([`~transformers.CLIPImageProcessor`]): A `CLIPImageProcessor` to extract features from generated images; used as inputs to the `safety_checker`. ztext_encoder->unet->vae)safety_checkerfeature_extractor image_encoderrX)r4 prompt_embedsnegative_prompt_embeds control_imageNTvae text_encoder tokenizerunet controlnet schedulerrYrZrequires_safety_checkerc  t ||%| r#tjd|jd| | t dt |ttfr t|}|j|||||||||  t|ddr/dt|jjjdz znd|_t#|j d |_t#|j d d |_|j)| y)Nz)You have disabled the safety checker for a by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .zMake sure to define a feature extractor when loading {self.__class__} if you want to use the safety checker. If you do not want to use the safety checker, you can pass `'safety_checker=None'` instead.) r^r_r`rarbrcrXrYrZr^r(r@T)vae_scale_factordo_convert_rgbF)rgrh do_normalize)rd)super__init__loggerwarning __class__ ValueErrorrBrMtuplerregister_modulesgetattrlenr^configblock_out_channelsrgrimage_processorcontrol_image_processorregister_to_config) selfr^r_r`rarbrcrXrYrZrdrns r8rkz1StableDiffusionControlNetImg2ImgPipeline.__init__s'   !&= NN;DNN;KLjj   %*;*Cx  j4- 0-j9J %!)/'  W^^bdikoVpc$((//*L*L&MPQ&Q Rvw0$BWBWhlm'8!224V[( $ 8OPr:r[r\ lora_scalec d} tdd| d|jd ||||||||d| } tj| d| dg}|S) Nz`_encode_prompt()` is deprecated and it will be removed in a future version. Use `encode_prompt()` instead. Also, be aware that the output format changed from a concatenated tensor to a tuple.z_encode_prompt()1.0.0F standard_warn)promptdevicenum_images_per_promptdo_classifier_free_guidancenegative_promptr[r\rzr@r)r encode_promptrCcat) ryrrrrrr[r\rzkwargsdeprecation_messageprompt_embeds_tuples r8_encode_promptz7StableDiffusionControlNetImg2ImgPipeline._encode_prompts}a$g/BRWX0d00  "7(C+'#9!     #6q#9;Nq;Q"RS r: clip_skipc H |Jt|tr:||_tst |j |nt |j ||t|trd} n-|t|tr t|} n|jd} |t|tr|j||j}|j|d|jjdd} | j} |j|dd j} | jd | jd k\rt!j"| | sj|jj%| dd|jjdz d f}t&j)d |jjd |t+|j j,d r<|j j,j.r| j0j3|}nd}| (|j | j3||}|d}nT|j | j3||d}|d | dz }|j j4j7|}|j |j j8}n/|j:|j:j8}n |j8}|j3||}|j\}}}|j=d|d}|j?||z|d }|rm|j|dg| z}n|:tA|tA|ur$tCdtA|dtA|dt|tr|g}n1| t|k7r!tEd|dt|d|d| d |}t|tr|j||j}|jd}|j|d|dd}t+|j j,d r<|j j,j.r|j0j3|}nd}|j |jj3||}|d}|rK|jd}|j3||}|j=d|d}|j?| |z|d }|j ,t|trtrtG|j |||fS)a Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*): prompt to be encoded device: (`torch.device`): torch device num_images_per_prompt (`int`): number of images that should be generated per prompt do_classifier_free_guidance (`bool`): whether to use classifier free guidance or not negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. lora_scale (`float`, *optional*): A LoRA scale that will be applied to all LoRA layers of the text encoder if LoRA layers are loaded. clip_skip (`int`, *optional*): Number of layers to be skipped from CLIP while computing the prompt embeddings. A value of 1 means that the output of the pre-final layer will be used for computing the prompt embeddings. Nr@r max_lengthTpt)paddingr truncationreturn_tensorslongest)rrz\The following part of your input was truncated because CLIP can only handle sequences up to z tokens: use_attention_mask)attention_mask)routput_hidden_states)r=rz?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.)$rBr _lora_scalerrr_r#strrMrsshapermaybe_convert_promptr`model_max_length input_idsrCequal batch_decoderlrmr5rtrrrG text_modelfinal_layer_normr=rarepeatviewtype TypeErrorror$)ryrrrrrr[r\rzr batch_size text_inputstext_input_idsuntruncated_ids removed_textrprompt_embeds_dtypebs_embedseq_len_ uncond_tokensr uncond_inputs r8rz6StableDiffusionControlNetImg2ImgPipeline.encode_prompts6V  !j7U&V)D $.t/@/@*M!$"3"3Z@  *VS"9J  Jvt$<VJ&,,Q/J  $ ;<2264>>J..$>>::# )K)22N"nnVYW[n\ffO$$R(N,@,@,DDU[[N $~~::#At~~'F'F'JR'O$OP  778 ,Q t((//1EF4K\K\KcKcKvKv!,!;!;!>!>v!F!% $ 1 1.2C2CF2K\j 1 k -a 0 $ 1 1"%%f-ncg!2! !.b 1IM2B C !% 1 1 < < M Mm \    ("&"3"3"9"9  YY ""&))// "/"5"5 %((/B6(R ,22'1%,,Q0EqI %**86K+KWVXY  '+A+I&!#z 1 #V D>$%# *Lt((//1EF4K\K\KcKcKvKv!-! " &,2215G%;%>%>EXag%>%h "%;%B%B1F[]^%_ "%;%@%@NcAcelnp%q "    ($ >?DT#D$5$5zB444r:c|t|jjj}t |t j s|j|dj}|j||}|r}|j|djd}|j|d}|jt j|djd}|j|d}||fS|j|j}|j|d}t j|} || fS) Nrrrr=T)rrdim)nextrZ parametersr=rBrCrDrY pixel_valuesrG hidden_statesrepeat_interleave zeros_like image_embeds) ryrSrrrr=image_enc_hidden_statesuncond_image_enc_hidden_statesruncond_image_embedss r8 encode_imagez5StableDiffusionControlNetImg2ImgPipeline.encode_imagesDT''2245;;%.**5*FSSEe4 &*&8&8UY&8&Z&h&hik&l #&=&O&OPekl&O&m #-1-?-?  'd.@.mB. *.L-]-]%1.^. *+,JJ J--e4AAL'99:OUV9WL"'"2"2<"@ !44 4r:cg}|rg}|t|ts|g}t|t|jjj k7rBt dt|dt|jjj dt||jjj D]`\}} t| t } |j||d| \} } |j| dddf|sIj| dddfbn?|D]:} |r%| jd\} } j| |j| <g}t|D]|\} } tj| g|zd} |r7tj| g|zd} tj| | gd} | j|} |j| ~|S) NzK`ip_adapter_image` must have same length as the number of IP Adapters. Got images and z IP Adapters.r@r(rrr)rBrMrsraencoder_hid_projimage_projection_layersroziprrappendchunk enumeraterCrrG)ryip_adapter_imageip_adapter_image_embedsrrrrnegative_image_embedssingle_ip_adapter_imageimage_proj_layeroutput_hidden_statesingle_image_embedssingle_negative_image_embedsrTs r8prepare_ip_adapter_image_embedszHStableDiffusionControlNetImg2ImgPipeline.prepare_ip_adapter_image_embedssY &$& ! " *.5$4#5 #$DII,F,F,^,^(__ abefvbwaxyEFIJNJSJSJdJdJ|J|F}E~~KL>A $))"<"<"T"T> X9')9+55E*W&W#DHDUDU+VQ8KEA#%A##$7a$@A.)001MdTUg1VW X(? 9#.H[HaHabcHdE02E)001MN##$78  9 #%&/ &= @ "A""'))-@,ADY,Y_`"a */4yy:OPQ:R9SVk9kqr/s,&+ii1MOb0cij&k#"5"8"8"8"G  # * *+> ? @'&r:cl|jd}||fStj|r|jj |d}n|jj |}|j |dj|}|j||jj|\}}||fS)Npil) output_typerr)images clip_input) rXrC is_tensorrv postprocess numpy_to_pilrYrGr)ryrSrr=has_nsfw_conceptfeature_extractor_inputsafety_checker_inputs r8run_safety_checkerz;StableDiffusionControlNetImg2ImgPipeline.run_safety_checkers    &# &&&u%*.*>*>*J*J5^c*J*d'*.*>*>*K*KE*R'#'#9#9:Qbf#9#g#j#jkq#r &*&9&9)=)J)J)M)Me)T':' #E#&&&r:c`d}tdd|dd|jjjz |z}|jj |dd}|d z d zj dd}|j jdd d djj}|S) Nz{The decode_latents method is deprecated and will be removed in 1.0.0. Please use VaeImageProcessor.postprocess(...) insteaddecode_latentsr|Fr}r@) return_dictrr(g?r) rr^rtscaling_factordecodeclampcpupermutefloatnumpy)ryr4rrSs r8rz7StableDiffusionControlNetImg2ImgPipeline.decode_latents!s\"G-@PUVdhhoo444w>U;A>S''1- ##Aq!Q/557==? r:cVdttj|jjj j v}i}|r||d<dttj|jjj j v}|r||d<|S)Netar.)setinspect signaturercsteprkeys)ryr.r accepts_etaextra_step_kwargsaccepts_generators r8prepare_extra_step_kwargszBStableDiffusionControlNetImg2ImgPipeline.prepare_extra_step_kwargs-s s7#4#4T^^5H5H#I#T#T#Y#Y#[\\  '* e $(3w/@/@ATAT/U/`/`/e/e/g+hh -6 k *  r:rAc  |0t|tr|dkrtd|dt|d| Lt fd| Ds8tdj d| D cgc]} | j vs| c} ||td|d |d | | td |7t|t s't|tstd t|||td |d|d |A|?|j|jk7r&td|jd|jdtjtrRt|trBtjdtjjdt|dtt dxr8tjt"j$j&j(}tjt*s&|r8tjj,t*rj/|||ntjts&|rtjj,trt|ts t1dt3d|Dr tdt|tjjk7r8tdt|dtjjd|D]}j/|||nJtjt*s&|r?tjj,t*rt| t4st1dtjts&|rtjj,trst| trt3d| DrStdt| tr8t| tjjk7r tdJt| t| k7r$tdt| dt| d tjtrt| tjjk7r[td!| d"t| d#tjjd$tjjd t7| | D]D\}}||k\rtd%|d&|d|d'krtd%|d(|d)kDs7td*|d+| | td,|Ut|tstd-t||dj8d.vrtd/|dj8d0yycc} w)1Nrz5`callback_steps` has to be a positive integer but is z of type rc3:K|]}|jvywN)_callback_tensor_inputs).0krys r8 zHStableDiffusionControlNetImg2ImgPipeline.check_inputs..Ss#F 23A-- -F sz2`callback_on_step_end_tensor_inputs` has to be in z , but found zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is z'Cannot forward both `negative_prompt`: z and `negative_prompt_embeds`: zu`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but got: `prompt_embeds` z != `negative_prompt_embeds` z You have z! ControlNets and you have passed z= prompts. The conditionings will be fixed across the prompts.scaled_dot_product_attentionz5For multiple controlnets: `image` must be type `list`c3<K|]}t|tywrrBrMrrTs r8rzHStableDiffusionControlNetImg2ImgPipeline.check_inputs..s8QZ4(8zEA single batch of multiple conditionings are supported at the moment.zbFor multiple controlnets: `image` must have the same length as the number of controlnets, but got rz ControlNets.zLFor single controlnet: `controlnet_conditioning_scale` must be type `float`.c3<K|]}t|tywrrrs r8rzHStableDiffusionControlNetImg2ImgPipeline.check_inputs..sRqz!T*RrzFor multiple controlnets: When `controlnet_conditioning_scale` is specified as `list`, it must have the same length as the number of controlnetsz`control_guidance_start` has z* elements, but `control_guidance_end` has zI elements. Make sure to provide the same number of elements to each list.z`control_guidance_start`: z has z elements but there are z- controlnets available. Make sure to provide zcontrol guidance start: z4 cannot be larger or equal to control guidance end: rz can't be smaller than 0.rAzcontrol guidance end: z can't be larger than 1.0.zProvide either `ip_adapter_image` or `ip_adapter_image_embeds`. Cannot leave both `ip_adapter_image` and `ip_adapter_image_embeds` defined.z:`ip_adapter_image_embeds` has to be of type `list` but is )rzF`ip_adapter_image_embeds` has to be a list of 3D or 4D tensors but is D)rBintrorallrrrMrrbrrlrmrsnetsr5FrC_dynamo eval_frameOptimizedModuler _orig_mod check_imageranyrrrE)ryrrScallback_stepsrr[r\rrcontrolnet_conditioning_scalecontrol_guidance_startcontrol_guidance_end"callback_on_step_end_tensor_inputsr is_compiledimage_startends` r8 check_inputsz5StableDiffusionControlNetImg2ImgPipeline.check_inputs>s  %z.#/NR`deReGGWX(),  . 9#F 7YF C DTEaEaDbbn|^pHvwbckokGkGbGpqpHoIJ   -";08N}o^00 ^ 5w  FC)@TZ\`IaQRVW]R^Q_`a a  &+A+M9/9JK*++]_   $)?)K""&<&B&BB --:-@-@,AB.445Q8 doo'; <&$'DOO$8$8 9::[\_`f\g[hST a!?@ Z OOU]]55EEF  t 84??44oF   UFM : t(< =4??446JKeT* WXX8%88 !hiiUs4??#7#788 xy|~CzDyEEQRUVZVeVeVjVjRkQllyz  @  ? @ 5 t 84??44oF;UC noo t(< =4??446JK7>R4QRR$%lmm94@SIfEgkn$$lF!D 5 % &#.B*C C/4J0K/LLvwz|PxQwRR[\  doo'; <)*c$//2F2F.GG 01G0HcRhNiMjkCDGHLHWHWH\H\D]C^^KLOPTP_P_PdPdLeKffgh46JK [JE3| .ug5ijminnops{ #;E7B[!\]]Sy #9#>X!YZZ [  ',C,O^  # .5t< PQUVmQnPop)+00> \]tuv]w]|]|\}}~? /upHs !W 5W ct|tjj}t|tj}t|t j }t|txr't|dtjj}t|txrt|dtj}t|txrt|dt j } |s!|s|s|s|s| stdt||rd} n t|} |t|trd} n/|t|tr t|} n||jd} | dk7r|  k7rtd| d| yy)Nrzimage must be passed and be one of PIL image, numpy array, torch tensor, list of PIL images, list of numpy arrays or list of torch tensors, but is r@zdIf image batch size is not 1, image batch size must be same as prompt batch size. image batch size: z, prompt batch size: )rBrIrJrCrDrKrLrMrrrsrrro) ryrSrr[ image_is_pilimage_is_tensor image_is_npimage_is_pil_listimage_is_tensor_listimage_is_np_listimage_batch_sizeprompt_batch_sizes r8rz4StableDiffusionControlNetImg2ImgPipeline.check_images!%9 $UELL9  3 &ud3] 58SYY__8])%6]:eAhPUP\P\;]%eT2Wz%(BJJ7W#%($fgklqgrfst   "5z   *VS"9 !   Jvt$< #F   & - 3 3A 6  q %59J%JvxHwII^_p^qr &K r:Fc 6|jj|||jtj}|j d} | dk(r|} n|} |j | d}|j||}|r| stj|gdz}|S)Nheightwidthr<rr@rrr()rw preprocessrGrCrHrrr) ryrSr&r%rrrr=r guess_moder! repeat_bys r8prepare_control_imagez>StableDiffusionControlNetImg2ImgPipeline.prepare_control_images,,77fTY7Z]]didqdq]r ;;q> q "I.I'' q'9e4 &zIIugk*E r:cNtt||z|}t||z d}|jj||jj zd}t |jdr2|jj||jj z|||z fS)Nrset_begin_index)minrmaxrc timestepsorderr5r,)rynum_inference_stepsstrengthr init_timestept_startr/s r8 get_timestepsz6StableDiffusionControlNetImg2ImgPipeline.get_timesteps!sC 3h >?ATU )M91=NN,,Wt~~7K7K-K-MN 4>>#4 5 NN * *7T^^5I5I+I J-777r:c @t|tjtjjt fst dt||j||}||z}|jddk(r|}nt|t r)t||k7rt dt|d|dt|t r|jd|krA||jdzdk(r,tj|g||jdzzd }nF|jd|kr4||jdzdk7rt d |jdd |d t|D cgc]1} t|jj|| | dz||  3}} tj|d }n&t|jj|| }|jj j"|z}||jdkDrh||jdzdk(rSd|d|jdd} t%dd| d||jdz} tj|g| zd }n^||jdkDr4||jdzdk7rt d |jdd|dtj|gd }|j} t'| |||} |j(j+|| |}|}|Scc} w)NzK`image` has to be of type `torch.Tensor`, `PIL.Image.Image` or list but is rr@rz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.rrz'Cannot duplicate `image` of batch size z to effective batch_size  )r.zYou have passed z# text prompts (`prompt`), but only a  initial images (`image`). Initial images are now duplicating to match the number of text prompts. Note that this behavior is deprecated and will be removed in a version 1.0.0. Please make sure to update your script to pass as many initial images as text prompts to suppress this warning.zlen(prompt) != len(image)r|Fr}z to z text prompts.)r.rr=)rBrCrDrIrJrMrorrGrrsrranger9r^encodertrrr'rc add_noise)ryrStimesteprrr=rr. init_latentsrTradditional_image_per_promptrnoiser4s r8prepare_latentsz8StableDiffusionControlNetImg2ImgPipeline.prepare_latents-s=%%,, !FG]^bch^i]jk e4"77 ;;q>Q  L)T*s9~/K Ec)nEUV *|+km It,;;q>J.: A3NRS3S!IIugu{{1~1M&NTUVE[[^j0Z%++a.5PTU5U$A%++a.AQQjkujvvwx #:. %TXX__U1q1u5E%FR[\]R^_    %yy1= /0FR[\ 88??99LHL  **1- -*|?Q?QRS?T2TXY2Y#:,.QR^RdRdefRgQhihh 17||g}||"|jA|||"|!| z|j8}%|jBjE||| jGt<jH }t|t,r2|jK||||!| z| |"|jL|j8| }n`t|trNg}&|D]D}'|jK|'|||!| z| |"|jL|j8| }'|&jO|'F|&}nJ|jPjS||"|jU|||"\}(}|(ddjW|!| z})t|(|_,| !|j[||)|!| |jL|"| } |j]| | }*||d%ind}+g},t_t|(D]w}-ta||D./cgc]8\}.}/dt+|-t|(z |.kxs|-dzt|(z |/kDz :}0}.}/|,jOt|t,r|0d n|0yt|(||jPjbzz }1|je|5}2tg|(D]\}-}3|jhr|j8rt=j>| gdzn| }4|jPjk|4|3}4|r?|j8r3| }5|jPjk|5|3}5|jmdd}6n|4}5|}6t|,|-tr%ta||,|-D7.cgc] \}7}.|7|.z }8}7}.n|}9t|9tr|9d }9|9|,|-z}8|j|5|3|6||8|d\}:};|rm|j8ra|:Dt=jn|<|t=jn|;|;g};|jq|4|3||j2|:|;|+dd }=|j8r|=jmd\}>}?|>||?|>z zz}=|jPjr|=|3| fi|*ddid } |li}@|D]}Atu|A@|A<|||-|3@}B|Bjd| } |Bjd|}|Bjd|}|Bjd|}|-t|(dz k(s'|-dz|1kDr]|-dz|jPjbzd k(r>|2jw|,|-|zd k(r$|-ty|jPddz}C||C|3| tzst}j~ dddt|drL|j@|jpjGd|jjGdt|dk(sc|jj| |jj.jz d| d }|j||"|jL\}}Dn| }d}DDd g|j&d z}EnDDFcgc]}F|F }E}F|jBj||E!}|j|s|DfSt|D"Scc}/}.wcc}.}7wcc} 1`. negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts to guide what to not include in image generation. If not defined, you need to pass `negative_prompt_embeds` instead. Ignored when not using guidance (`guidance_scale < 1`). num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. eta (`float`, *optional*, defaults to 0.0): Corresponds to parameter eta (η) from the [DDIM](https://huggingface.co/papers/2010.02502) paper. Only applies to the [`~schedulers.DDIMScheduler`], and is ignored in other schedulers. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor is generated by sampling using the supplied random `generator`. prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not provided, text embeddings are generated from the `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not provided, `negative_prompt_embeds` are generated from the `negative_prompt` input argument. ip_adapter_image: (`PipelineImageInput`, *optional*): Optional image input to work with IP Adapters. ip_adapter_image_embeds (`List[torch.Tensor]`, *optional*): Pre-generated image embeddings for IP-Adapter. It should be a list of length same as number of IP-adapters. Each element should be a tensor of shape `(batch_size, num_images, emb_dim)`. It should contain the negative image embedding if `do_classifier_free_guidance` is set to `True`. If not provided, embeddings are computed from the `ip_adapter_image` input argument. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generated image. Choose between `PIL.Image` or `np.array`. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] instead of a plain tuple. cross_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the [`AttentionProcessor`] as defined in [`self.processor`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). controlnet_conditioning_scale (`float` or `List[float]`, *optional*, defaults to 1.0): The outputs of the ControlNet are multiplied by `controlnet_conditioning_scale` before they are added to the residual in the original `unet`. If multiple ControlNets are specified in `init`, you can set the corresponding scale as a list. guess_mode (`bool`, *optional*, defaults to `False`): The ControlNet encoder tries to recognize the content of the input image even if you remove all prompts. A `guidance_scale` value between 3.0 and 5.0 is recommended. control_guidance_start (`float` or `List[float]`, *optional*, defaults to 0.0): The percentage of total steps at which the ControlNet starts applying. control_guidance_end (`float` or `List[float]`, *optional*, defaults to 1.0): The percentage of total steps at which the ControlNet stops applying. clip_skip (`int`, *optional*): Number of layers to be skipped from CLIP while computing the prompt embeddings. A value of 1 means that the output of the pre-final layer will be used for computing the prompt embeddings. callback_on_step_end (`Callable`, `PipelineCallback`, `MultiPipelineCallbacks`, *optional*): A function or a subclass of `PipelineCallback` or `MultiPipelineCallbacks` that is called at the end of each denoising step during the inference. with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by `callback_on_step_end_tensor_inputs`. callback_on_step_end_tensor_inputs (`List`, *optional*): The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. Examples: Returns: [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] is returned, otherwise a `tuple` is returned where the first element is a list with the generated images and the second element is a list of `bool`s indicating whether the corresponding generated image contains "not-safe-for-work" (nsfw) content. callbackNrr|zjPassing `callback` as an input argument to `__call__` is deprecated, consider using `callback_on_step_end`zpPassing `callback_steps` as an input argument to `__call__` is deprecated, consider using `callback_on_step_end`r@Frscale)r[r\rzrr$r<) rSr&r%rrrr=rr(rrrA)totalr()encoder_hidden_statescontrolnet_condconditioning_scaler(r)rXrKdown_block_additional_residualsmid_block_additional_residualadded_cond_kwargsrrr4r[r\r]r0final_offload_hookrlatent)rr.T)rdo_denormalize)rnsfw_content_detected)JpoprrBrr tensor_inputsr&rbr rMrsrrrrBrFrJrPrr_execution_devicerrrtglobal_pool_conditionsrKgetrrrrCrrrvr'rGrHr*r=rrc set_timestepsr5rrMr?rr8rr0 progress_barrrQscale_model_inputrrrarlocalsupdaterr XLA_AVAILABLExm mark_stepr5r^r%r^rrrrmaybe_free_model_hooksr+)GryrrSr]r%r&r2r1rDrrrr.r4r[r\rrrrrKrr(rrrrSrrrUrrbmultrrretext_encoder_lora_scalercontrol_imagescontrol_image_r/latent_timesteprr]controlnet_keeprTsekeepsnum_warmup_stepsrhtlatent_model_inputcontrol_model_inputcontrolnet_prompt_embedsc cond_scalecontrolnet_cond_scaledown_block_res_samplesmid_block_res_sampled noise_prednoise_pred_uncondnoise_pred_textcallback_kwargsrcallback_outputsstep_idxrr`has_nsfwsG r8__call__z1StableDiffusionControlNetImg2ImgPipeline.__call__sd T::j$/$4d;   |   %  C  *-=?U,V W1E1S1S .2DT__2UT__..[_[j[j 0$7JG[]aBXAY%Y "0$7JG]_cBVAW#W 2D9*MacgBh+5jBV+W3z']^D.//,--%9 "       "  # ) " .  .#'=$  *VS"9J  Jvt$<VJ&,,Q/J'' j"6 7JGdfkY>Y>eD ' ' + +GT :ko 150B0B   !  , , '#9.nn1C 1 - -  + +!II'=}&MNM  '+B+N?? '2200 L$$//fE/RUU\a\i\iUj j/ 2 66#%(==&; &&,0,L,L%7 M $8 9N"/ 6!%!;!;(!),AA*?!$**040P0P)"< "%%n5 6+M 5 $$%8$H)-););,?$..BVBV,VV   %8  9S #\!),R #1>>BFAaAaUYYy1}%=gn"%)^^%E%EFXZ[%\"$"B"B*1'*...*J*JK^`a*b'/0Ea0H-!69K!KJ?C'*B$1'1) %@O@<&(<$"B"B\r-rVWeii9I9I!9La8P.Q-r*-r+099e6F6FG[6\^r5s+t("YY&*7+/+F+F4J2F&7 %'   339C9I9I!9L6%!2^YjGj5k!kJ.$..--j!WmHYmglmnop'3&(O?9-3Xa[*9';D!Q'X$.229gFG$4$8$8-$XM-=-A-ABZ\r-s*$4$8$8-$XMI**A9I/IqSTuX\XfXfXlXlNlpqNq '')+N0Ba0G#$(K#K 1g6 LLNeR #S #n 4- .43J3J3V IILL  OO  u %  h&HHOOGdhhoo.L.L$LZ_ktOuE'+&=&=eV]M`M`&a #E#E#   #"Vekk!n4N;KLx(lLNL$$00K`n0o ##%+, ,,EQabbu8"l*.sSS #S #PMs? =e6?C f e< Af+1fF f+f) f< ff)NT)NNNN)NNNNNr) NNNNNrArrAN)FF)<__name__ __module__ __qualname____doc__model_cpu_offload_seq_optional_components_exclude_from_cpu_offloadrrr r rr rrrrrr,r r boolrkrrCrDrrrrrrrrrrrr*r5r?propertyrDrrrKrNrQno_gradr"EXAMPLE_DOC_STRINGrr Generatorrrrrrr __classcell__)rns@r8rWrWs#J6S!1 2e8<(,2Q 2Q$2Q! 2Q # 2Q /4+@%BXZnno 2Q-2Q52Q.2Q52Q"&2Qv049=&*  -!) 6UON049=&*#'t5  -t5!) 6t5UOt5C=t5n52+'\ ' !,# $&)" +/[|#^%*> 8?B$$ ((,,##U]]_12)-$(,0 $##% #;?/0MQ*.049=9=@D%* ;?CF rsDDD hhADwwqq93VUD<J))MM   H %,bck TLL T-5eoo-F T\_ T0nc" ncr: