bipddlmZddlmZddlmZmZmZmZm Z m Z ddl Z ddl Z ddlmZddlmZddlmZmZdd lmZdd lmZmZdd lmZmZdd lmZmZm Z m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)er ddl*m+cm,Z-dZ.ndZ.e j^e0Z1dZ2eGddeZ3Gdde'Z4y)) dataclass)partial)AnyDictListOptionalTupleUnionN)Image)tqdm) CLIPTextModel CLIPTokenizer)PipelineImageInput) AutoencoderKLUNet2DConditionModel) DDIMScheduler LCMScheduler) BaseOutputis_torch_xla_availableloggingreplace_example_docstring)is_scipy_available) randn_tensor)DiffusionPipeline)MarigoldImageProcessorTFaE Examples: ```py >>> import diffusers >>> import torch >>> pipe = diffusers.MarigoldDepthPipeline.from_pretrained( ... "prs-eth/marigold-depth-v1-1", variant="fp16", torch_dtype=torch.float16 ... ).to("cuda") >>> image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg") >>> depth = pipe(image) >>> vis = pipe.image_processor.visualize_depth(depth.prediction) >>> vis[0].save("einstein_depth.png") >>> depth_16bit = pipe.image_processor.export_depth_to_16bit_png(depth.prediction) >>> depth_16bit[0].save("einstein_depth_16bit.png") ``` ceZdZUdZeej ejfe d<edej ejfe d<edejfe d<y)MarigoldDepthOutputa Output class for Marigold monocular depth prediction pipeline. Args: prediction (`np.ndarray`, `torch.Tensor`): Predicted depth maps with values in the range [0, 1]. The shape is $numimages imes 1 imes height imes width$ for `torch.Tensor` or $numimages imes height imes width imes 1$ for `np.ndarray`. uncertainty (`None`, `np.ndarray`, `torch.Tensor`): Uncertainty maps computed from the ensemble, with values in the range [0, 1]. The shape is $numimages imes 1 imes height imes width$ for `torch.Tensor` or $numimages imes height imes width imes 1$ for `np.ndarray`. latent (`None`, `torch.Tensor`): Latent features corresponding to the predictions, compatible with the `latents` argument of the pipeline. The shape is $numimages * numensemble imes 4 imes latentheight imes latentwidth$. predictionN uncertaintylatent) __name__ __module__ __qualname____doc__r npndarraytorchTensor__annotations__o/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/pipelines/marigold/pipeline_marigold_depth.pyr r RsO bjj%,,.//tRZZ566 $ $ %%r.r c!eZdZdZdZdZ d,dededee e fde de d e ed e ed e ed e ed e effd Zdededededededede eeefde ej,de eej.eej.fdededefdZej4j6d-dZej:ee d.dede edede ededededede eeefde eej,eej,fde eej.eej.fdededed efd!Z dej,de ej,de ej.dedede!ej,ej,ff d"Z"d#ej,dej,fd$Z#e$ d/d%ej,d ed eded&ed'e%d(ed)e%d*ede!ej,e ej,ffd+Z&xZ'S)0MarigoldDepthPipelinea7 Pipeline for monocular depth estimation using the Marigold method: https://marigoldmonodepth.github.io. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) Args: unet (`UNet2DConditionModel`): Conditional U-Net to denoise the depth latent, conditioned on image latent. vae (`AutoencoderKL`): Variational Auto-Encoder (VAE) Model to encode and decode images and predictions to and from latent representations. scheduler (`DDIMScheduler` or `LCMScheduler`): A scheduler to be used in combination with `unet` to denoise the encoded image latents. text_encoder (`CLIPTextModel`): Text-encoder, for empty text embedding. tokenizer (`CLIPTokenizer`): CLIP tokenizer. prediction_type (`str`, *optional*): Type of predictions made by the model. scale_invariant (`bool`, *optional*): A model property specifying whether the predicted depth maps are scale-invariant. This value must be set in the model config. When used together with the `shift_invariant=True` flag, the model is also called "affine-invariant". NB: overriding this value is not supported. shift_invariant (`bool`, *optional*): A model property specifying whether the predicted depth maps are shift-invariant. This value must be set in the model config. When used together with the `scale_invariant=True` flag, the model is also called "affine-invariant". NB: overriding this value is not supported. default_denoising_steps (`int`, *optional*): The minimum number of denoising diffusion steps that are required to produce a prediction of reasonable quality with the given model. This value must be set in the model config. When the pipeline is called without explicitly setting `num_inference_steps`, the default value is used. This is required to ensure reasonable results with various model flavors compatible with the pipeline, such as those relying on very short denoising schedules (`LCMScheduler`) and those with full diffusion schedules (`DDIMScheduler`). default_processing_resolution (`int`, *optional*): The recommended value of the `processing_resolution` parameter of the pipeline. This value must be set in the model config. When the pipeline is called without explicitly setting `processing_resolution`, the default value is used. This is required to ensure reasonable results with various model flavors trained with varying optimal processing resolution values. ztext_encoder->unet->vae)depth disparityunetvae scheduler text_encoder tokenizerprediction_typescale_invariantshift_invariantdefault_denoising_stepsdefault_processing_resolutionc t |||jvr&tj d|d|jd|j ||||||j |||| | t|ddr/dt|jjjdz znd |_ ||_ ||_| |_| |_d|_t%|j |_y) Nz*Potentially unsupported `prediction_type='z&'`; values supported by the pipeline: .)r4r5r6r7r8)r9r:r;r<r=r5rr)vae_scale_factor)super__init__supported_prediction_typesloggerwarningregister_modulesregister_to_configgetattrlenr5configblock_out_channelsrAr:r;r<r=empty_text_embeddingrimage_processor) selfr4r5r6r7r8r9r:r;r<r= __class__s r/rCzMarigoldDepthPipeline.__init__s  $"A"A A NN<_$-J*$(!5tG\G\]r.imagenum_inference_steps ensemble_sizeprocessing_resolutionresample_method_inputresample_method_output batch_sizeensembling_kwargslatents generator output_typeoutput_uncertaintyreturnc  dt|jjjdz z} | |jk7rt d|jd| d| t d|dkr t d|dkr t d|dk(rt jd |dkDr-|js |jrts td |dk(r | r t d | t d |d kr t d||jzd k7rt d|jd|dvr t d|dvr t d|dkr t d| dvr t d| t d|1t|ts t dd|vr|ddvr t dd }d\}}t|ts|g}t|D]\}}t|t j"st%j&|r]|j(dvrt d|d|j*d|j*d d\}}d}|j(d!k(rW|j*d }nGt|t,j,r|j.\}}d}nt d"|d#t1|d|||}}n!||f||fk7rt d$|d%||fd&||f||z } | t%j&| s t d'| j3d!k7rt d(| j*d|d kDrz5MarigoldDepthPipeline.check_inputs..Bs.X188==IaL,?,?,D,DDXs:=z;`generator` device placement is not consistent in the list.zUnsupported generator type: )rJr5rKrLrA ValueErrorrErFr:r;r ImportError isinstancedictlist enumerater(r)r* is_tensorndimshaper sizerndimmaxlatent_channelsall Generator)rOrQrRrSrTrUrVrWrXrYrZr[r\actual_vae_scale_factor num_imagesWHiimgH_iW_iN_imax_orignew_Hnew_Wwhshape_expecteds ` r/ check_inputsz"MarigoldDepthPipeline.check_inputssJ#$DHHOO,N,N(ORS(S"T "d&;&; ;A$BWBWAXXwyPxQQST   &vw w  "FG G 1 @A A A  NN?  1 $"6"6$:N:NXjXlYZ Z A "4"  ! (k  !1 $*  !4#8#8 8A =MdNcNcMddefg g (c cC  ")d dC  >=> > l *IJ J  9#8QR R  (/6 !LMM//4Ek4RZl4l !ijj 1%&GE& FAs#rzz*eooc.B889,$wqc1YZ]ZcZcYdde%fgg99RS>S88q=))A,CC-88S #6qc49+Q!OPPyC1QC:% #A3&EsCj\Qkmnpqlrkst # J' ,  ??7+ !DEE{{}! #RSZS`S`Raab!cdd$q(q!911X=11X=A:!$'QRSQTTWXYWZZ[%\]]e1T***Q.43H3HHAT***Q.43H3HHA(=8$((//:Y:Y[\^_`N}}. #B7==/Q[\j[kkl!mnn  )T*y>Z-%??$xXiXX$%bcc 5??; #?Y?PPQ!RSSr.cpt|dsi|_n%>?&9&=&=fd&KF#':'>'>w'NG$  8$78 8  ;e;':; ;NO Or.match_input_resolution output_latent return_dictch|j}|j}| |j}| |j}|j |||||||| | | | | }|j `d}|j |d|j jdd}|jj|}|j|d|_|jj|||||\}}}|j|| | ||\}}~|j j||j|d d }g}|jt!d||z|dd D]}||||z}||||z}|j"d}|d|} |j$j'|| |j|j$j(d d D]w}!t+j,||gd }"|j/|"|!| d d}#|j$j1|#|!|| j2}t4sdt7j8y|j;|t+j,|d}~~~~~~ ~"~#t+j,t!d|j"d|Dcgc]}|j=||||zc}d}$|sd}|jj?|$|}$d}%|d kDr|$j@||g|$j"d d}$t!|Dcgc]3}|jB|$||jD|jF| fi| xsi5}$}tI|$\}$}%t+j,|$d}$| rt+j,|%d}%nd}%|rB|jjK|$||d }$|%!| r|jjK|%||d }%| dk(r:|jjM|$}$|%| r|jjM|%}%|jO|s|$|%|fStQ|$|%|Scc}wcc}w)aA Function invoked when calling the pipeline. Args: image (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`), `List[torch.Tensor]`: An input image or images used as an input for the depth estimation task. For arrays and tensors, the expected value range is between `[0, 1]`. Passing a batch of images is possible by providing a four-dimensional array or a tensor. Additionally, a list of images of two- or three-dimensional arrays or tensors can be passed. In the latter case, all list elements must have the same width and height. num_inference_steps (`int`, *optional*, defaults to `None`): Number of denoising diffusion steps during inference. The default value `None` results in automatic selection. ensemble_size (`int`, defaults to `1`): Number of ensemble predictions. Higher values result in measurable improvements and visual degradation. processing_resolution (`int`, *optional*, defaults to `None`): Effective processing resolution. When set to `0`, matches the larger input image dimension. This produces crisper predictions, but may also lead to the overall loss of global context. The default value `None` resolves to the optimal value from the model config. match_input_resolution (`bool`, *optional*, defaults to `True`): When enabled, the output prediction is resized to match the input dimensions. When disabled, the longer side of the output will equal to `processing_resolution`. resample_method_input (`str`, *optional*, defaults to `"bilinear"`): Resampling method used to resize input images to `processing_resolution`. The accepted values are: `"nearest"`, `"nearest-exact"`, `"bilinear"`, `"bicubic"`, or `"area"`. resample_method_output (`str`, *optional*, defaults to `"bilinear"`): Resampling method used to resize output predictions to match the input resolution. The accepted values are `"nearest"`, `"nearest-exact"`, `"bilinear"`, `"bicubic"`, or `"area"`. batch_size (`int`, *optional*, defaults to `1`): Batch size; only matters when setting `ensemble_size` or passing a tensor of images. ensembling_kwargs (`dict`, *optional*, defaults to `None`) Extra dictionary with arguments for precise ensembling control. The following options are available: - reduction (`str`, *optional*, defaults to `"median"`): Defines the ensembling function applied in every pixel location, can be either `"median"` or `"mean"`. - regularizer_strength (`float`, *optional*, defaults to `0.02`): Strength of the regularizer that pulls the aligned predictions to the unit range from 0 to 1. - max_iter (`int`, *optional*, defaults to `2`): Maximum number of the alignment solver steps. Refer to `scipy.optimize.minimize` function, `options` argument. - tol (`float`, *optional*, defaults to `1e-3`): Alignment solver tolerance. The solver stops when the tolerance is reached. - max_res (`int`, *optional*, defaults to `None`): Resolution at which the alignment is performed; `None` matches the `processing_resolution`. latents (`torch.Tensor`, or `List[torch.Tensor]`, *optional*, defaults to `None`): Latent noise tensors to replace the random initialization. These can be taken from the previous function call's output. generator (`torch.Generator`, or `List[torch.Generator]`, *optional*, defaults to `None`): Random number generator object to ensure reproducibility. output_type (`str`, *optional*, defaults to `"np"`): Preferred format of the output's `prediction` and the optional `uncertainty` fields. The accepted values are: `"np"` (numpy array) or `"pt"` (torch tensor). output_uncertainty (`bool`, *optional*, defaults to `False`): When enabled, the output's `uncertainty` field contains the predictive uncertainty map, provided that the `ensemble_size` argument is set to a value above 2. output_latent (`bool`, *optional*, defaults to `False`): When enabled, the output's `latent` field contains the latent codes corresponding to the predictions within the ensemble. These codes can be saved, modified, and used for subsequent calls with the `latents` argument. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.marigold.MarigoldDepthOutput`] instead of a plain tuple. Examples: Returns: [`~pipelines.marigold.MarigoldDepthOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.marigold.MarigoldDepthOutput`] is returned, otherwise a `tuple` is returned where the first element is the prediction, the second element is the uncertainty (or `None`), and the third is the latent (or `None`). N do_not_padTrd)padding max_length truncationreturn_tensorsr)rmdtyperzMarigold predictions...)rr)rmFzDiffusion steps...r|)encoder_hidden_statesr)rZ)is_aar()r!r"r#))_execution_devicerr<r=rrMr8model_max_length input_idstor7rN preprocessprepare_latentsrepeatrrangerzr6 set_timesteps timestepsr*catr4step prev_sample XLA_AVAILABLExm mark_stepappenddecode_prediction unpad_imagereshapeensemble_depthr:r;zipresize_antialias pt_to_numpymaybe_free_model_hooksr )&rOrQrRrSrTrrUrVrWrXrYrZr[r\rrrmrrprompt text_inputstext_input_idsroriginal_resolution image_latent pred_latentbatch_empty_text_embedding pred_latentsrbatch_image_latentbatch_pred_latenteffective_batch_sizetextt batch_latentnoiser!r"s& r/__call__zMarigoldDepthPipeline.__call__\st''   &"&">">  ($($F$F !&&    ! ! "        "  $ $ ,F..$>>::# )K)2255f=N(,(9(9.(I!(LD %/3.B.B.M.M (*?/ +w+%)$8$8 7I}j% ! k %)%>%>%A%AW\%A%]%d%d 1& " "" !Z-/ -.C/CDD NN ( ()O)PVWX  ,[` abcd$(NN$7$71/9%8%+"!LLN #    1 2' 3*ii !4    &      YYq+"3"3A"6 C &&{1q:~'FG   K))55j'J  1 +++J ] HXHXYZY[H\]Jz* $##qM((((&  ).B  J '*:&6 #J :15J!#ii ; " "-->>/1Gu?J&+="22CC!46LTYD $ --99*EJ&+="22>>{K  ##% [9 9"!#   * s P* 8P/c d}tjtd|jd|Dcgc])}||jj ||||z+c}d}||jj jz}|j|d}|} | -t|j||j|j} || fScc}w)Nct|dr|jjSt|dr |jSt d)N latent_distrYz3Could not access latents of provided encoder_output)rrmoderYAttributeError)encoder_outputs r/retrieve_latentsz?MarigoldDepthPipeline.prepare_latents..retrieve_latentsvsC~}5%1166883%---$%Z[[r.rr)rZrmr) r*rrrzr5encoderKscaling_factorrepeat_interleaverrmr) rOrQrYrZrSrWrrrrs r/rz%MarigoldDepthPipeline.prepare_latentsns \yyq%++a.*= !q1z>1J!KL   $dhhoo&D&DD #55m5K   &""##**"(( K[((% s.Crc|jdk7s0|jd|jjjk7r:t d|jjjd|jd|jj ||jjjz dd}|jdd }tj|d d }|d zd z }|S)Nrirz Expecting 4D tensor of shape [B,z ,H,W]; got r?F)rrTr|keepdimg?g@) r|rzr5rKr~rrdecoderrgr*clip)rOrr!s r/rz'MarigoldDepthPipeline.decode_predictions ??  ![%6%6q%9TXX__=\=\%\2488??3R3R2SS^_j_p_p^qqrs XX__[488??3Q3Q%Q_d_efgh __D_9 ZZ D#6  3&#- r.r2reregularizer_strengthmax_itertolmax_resc |jdk7s|jddk7rtd|jddvrtdds r tddtjffd dtjd t j d tjffd  dd tjdtd ttjttjfffd d t j dtjd tffd dtjffd } xs} |jd| r| |} || }||\}} |j} rr|j}nrd}n td| |z jd}||z |z }|r| |z} || fS)a Ensembles the depth maps represented by the `depth` tensor with expected shape `(B, 1, H, W)`, where B is the number of ensemble members for a given prediction of size `(H x W)`. Even though the function is designed for depth maps, it can also be used with disparity maps as long as the input tensor values are non-negative. The alignment happens when the predictions have one or more degrees of freedom, that is when they are either affine-invariant (`scale_invariant=True` and `shift_invariant=True`), or just scale-invariant (only `scale_invariant=True`). For absolute predictions (`scale_invariant=False` and `shift_invariant=False`) alignment is skipped and only ensembling is performed. Args: depth (`torch.Tensor`): Input ensemble depth maps. scale_invariant (`bool`, *optional*, defaults to `True`): Whether to treat predictions as scale-invariant. shift_invariant (`bool`, *optional*, defaults to `True`): Whether to treat predictions as shift-invariant. output_uncertainty (`bool`, *optional*, defaults to `False`): Whether to output uncertainty map. reduction (`str`, *optional*, defaults to `"median"`): Reduction method used to ensemble aligned predictions. The accepted values are: `"mean"` and `"median"`. regularizer_strength (`float`, *optional*, defaults to `0.02`): Strength of the regularizer that pulls the aligned predictions to the unit range from 0 to 1. max_iter (`int`, *optional*, defaults to `2`): Maximum number of the alignment solver steps. Refer to `scipy.optimize.minimize` function, `options` argument. tol (`float`, *optional*, defaults to `1e-3`): Alignment solver tolerance. The solver stops when the tolerance is reached. max_res (`int`, *optional*, defaults to `1024`): Resolution at which the alignment is performed; `None` matches the `processing_resolution`. Returns: A tensor of aligned and ensembled depth maps and optionally a tensor of uncertainties of the same shape: `(1, 1, H, W)`. rirz,Expecting 4D tensor of shape [B,1,H,W]; got r?rfUnrecognized reduction method: z1Pure shift-invariant ensembling is not supported.r2c$|jdjdj}|jdjdj}rTrRd||z j dz }| |z}t j ||fjj}nAr4d|j dz }|jj}n td|jtj}|S)Nrrrư>minUnrecognized alignment.) rrvaluesr}clampr*rcpunumpyrrastyper(float64) r2init_mininit_maxinit_sinit_tparamrSr:r;s r/ init_paramz8MarigoldDepthPipeline.ensemble_depth..init_params}}]B7;;;BIIH}}]B7;;;BIIH?8 3::t:DD 8+ 66"23779??A x~~$~77 **, !:;;LL,ELr.rr]crrtj|d\}}tj|j |j ddd}tj|j |j ddd}||z|z}|Sr=tj|j |j ddd}||z}|St d)Nrrr)r(splitr* from_numpyrviewrr)r2rsroutrSr:r;s r/alignz3MarigoldDepthPipeline.ensemble_depth..aligns?xxq)1$$Q'**5166}aAN$$Q'**5166}aANai!m J !$$U+..u5::=!QPQRaiJ!!:;;r. depth_alignedreturn_uncertaintycfd}dk(r6tj|dd}|rtj|dd}||fSdk(r`tj|ddj}|r8tjtj ||z ddj}||fSt dd)NrgrTrrhrr?)r*rgstdrhrabsrr)rrr"r!res r/ensemblez6MarigoldDepthPipeline.ensemble_depth..ensemblesKF""ZZ 1dK %"'))Mq$"OK{* * h&"\\-QMTT %"',,uyy9S/TZ[ei"j"q"qK{* *!#B9+Q!OPPr.cd} ||}tjtj D]B\}}||||z }||dzjj j z }DdkDrs |d\}}|j jj } d|jz jj } || | zzz }|S)NgrrFrr) r* combinationsarangergsqrtitemrrr})rr2costrrjdiffr!_err_nearerr_farrrrSrs r/cost_fnz5MarigoldDepthPipeline.ensemble_depth..cost_fnsD!%/M**5<< +FG 71$Q'-*::q(--/4466 7$a' (5 Q A%>>+//1668!11668==?G+/CCCKr.c 6ddl}|jtj}2t |j ddkDrt j|d}|}|jjt||d dd}|jS) Nrrr`)r2BFGSF)maxiterdisp)methodroptions) scipyrr*float32r}rzrresize_to_max_edgeoptimizeminimizerx) r2rdepth_to_alignrresrrrrrs r/ compute_paramz;MarigoldDepthPipeline.ensemble_depth..compute_params "XXemm4N"s>+?+?+C'Dw'N!7!J!J>[bds!t~.E..))~6$,e< *C55Lr.rrrrr)F)r|rzrrr*r+r(r)boolr rfloatr}rr)r2r:r;r\rerrrrr"requires_aligningrr" depth_max depth_min depth_rangerrrrSrs `` ````` @@@@@r/rz$MarigoldDepthPipeline.ensemble_depths\ 99;! u{{1~2KEKK=XYZ[ [ . .>ykKL L?PQ Q ell "  bjj U\\ EJ + << +=A + 5<<%,,!77 8 +  2:: ell u      &,> A !%(E%'E%e@RS{IIK  I I67 7 9,333= "k1  ; &Kk!!r.)NTTNN)NNNT)NrNTrararNNNr(FFT)TTFrhg{Gz?rgMbP?i)(r$r%r&r'model_cpu_offload_seqrDrrr rrr rrstrr#intrCrrrr*r+rrrcompilerdisablerno_gradrEXAMPLE_DOC_STRINGrr rr staticmethodr$r __classcell__)rPs@r/r1r1is 'R6!7*.*.*.157;-^"-^-^ 45 -^ $ -^ ! -^"#-^"$-^"$-^"*#-^(0}-^^B!B!B B # B # B!$BB$DcN3B%,,'BE%//43H"HIJBB!B BH ^^PP$U]]_12.2/3'+%/&06:EIMQ#(# !N !N &c]N  N (} N !% N  #N !$N N $DcN3N % d5<<.@ @ABN E%//43H"HIJN N !N N !N 3N `#)||#)%,,'#)EOO, #)  #)  #) u||U\\) *#)J U\\ ell !% $#(!&*Z"||Z"Z"Z"! Z"  Z" $ Z"Z"Z"Z" u||Xell33 4Z"Z"r.r1)5 dataclassesr functoolsrtypingrrrrr r rr(r*PILr tqdm.autor transformersr rrNrmodelsrr schedulersrrutilsrrrrutils.import_utilsrutils.torch_utilsrpipeline_utilsrmarigold_image_processingrtorch_xla.core.xla_modelcore xla_modelrr get_loggerr$rEr/r r1r-r.r/rCs&":: 51 5-.=))MM   H %, &*& &,S "-S "r.