biXddlmZddlmZmZmZmZmZmZddl Z ddl Z ddl m Z ddlmZddlmZmZddlmZdd lmZmZdd lmZmZdd lmZmZmZmZdd l m!Z!d dl"m#Z#ddl$m%Z%er ddl&m'cm(Z)dZ*ndZ*ejVe,Z-dZ.eGddeZ/Gdde#Z0y)) dataclass)AnyDictListOptionalTupleUnionN)Image)tqdm) CLIPTextModel CLIPTokenizer)PipelineImageInput) AutoencoderKLUNet2DConditionModel) DDIMScheduler LCMScheduler) BaseOutputis_torch_xla_availableloggingreplace_example_docstring) randn_tensor)DiffusionPipeline)MarigoldImageProcessorTFa Examples: ```py >>> import diffusers >>> import torch >>> pipe = diffusers.MarigoldIntrinsicsPipeline.from_pretrained( ... "prs-eth/marigold-iid-appearance-v1-1", variant="fp16", torch_dtype=torch.float16 ... ).to("cuda") >>> image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg") >>> intrinsics = pipe(image) >>> vis = pipe.image_processor.visualize_intrinsics(intrinsics.prediction, pipe.target_properties) >>> vis[0]["albedo"].save("einstein_albedo.png") >>> vis[0]["roughness"].save("einstein_roughness.png") >>> vis[0]["metallicity"].save("einstein_metallicity.png") ``` ```py >>> import diffusers >>> import torch >>> pipe = diffusers.MarigoldIntrinsicsPipeline.from_pretrained( ... "prs-eth/marigold-iid-lighting-v1-1", variant="fp16", torch_dtype=torch.float16 ... ).to("cuda") >>> image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg") >>> intrinsics = pipe(image) >>> vis = pipe.image_processor.visualize_intrinsics(intrinsics.prediction, pipe.target_properties) >>> vis[0]["albedo"].save("einstein_albedo.png") >>> vis[0]["shading"].save("einstein_shading.png") >>> vis[0]["residual"].save("einstein_residual.png") ``` ceZdZUdZeej ejfe d<edej ejfe d<edejfe d<y)MarigoldIntrinsicsOutputa Output class for Marigold Intrinsic Image Decomposition pipeline. Args: prediction (`np.ndarray`, `torch.Tensor`): Predicted image intrinsics with values in the range [0, 1]. The shape is $(numimages * numtargets) imes 3 imes height imes width$ for `torch.Tensor` or $(numimages * numtargets) imes height imes width imes 3$ for `np.ndarray`, where `numtargets` corresponds to the number of predicted target modalities of the intrinsic image decomposition. uncertainty (`None`, `np.ndarray`, `torch.Tensor`): Uncertainty maps computed from the ensemble, with values in the range [0, 1]. The shape is $(numimages * numtargets) imes 3 imes height imes width$ for `torch.Tensor` or $(numimages * numtargets) imes height imes width imes 3$ for `np.ndarray`. latent (`None`, `torch.Tensor`): Latent features corresponding to the predictions, compatible with the `latents` argument of the pipeline. The shape is $(numimages * numensemble) imes (numtargets * 4) imes latentheight imes latentwidth$. predictionN uncertaintylatent) __name__ __module__ __qualname____doc__r npndarraytorchTensor__annotations__t/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/pipelines/marigold/pipeline_marigold_intrinsics.pyrr_sO$bjj%,,.//tRZZ566 $ $ %%r,rc!eZdZdZdZdZ d(dededee e fde de d e ed e eeefd e ed e effd ZedZdededededededede eeefde ej.de eej0eej0fdededefdZej8j:d)dZej>e e! d*dede edede ededededede eeefde eej.eej.fde eej0eej0fdededed efd!Z"dej.de ej.de ej0dedede#ej.ej.ff d"Z$d#ej.dej.fd$Z%e& d+d%ej.ded&ede#ej.e ej.ffd'Z'xZ(S),MarigoldIntrinsicsPipelinea Pipeline for Intrinsic Image Decomposition (IID) using the Marigold method: https://marigoldcomputervision.github.io. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) Args: unet (`UNet2DConditionModel`): Conditional U-Net to denoise the targets latent, conditioned on image latent. vae (`AutoencoderKL`): Variational Auto-Encoder (VAE) Model to encode and decode images and predictions to and from latent representations. scheduler (`DDIMScheduler` or `LCMScheduler`): A scheduler to be used in combination with `unet` to denoise the encoded image latents. text_encoder (`CLIPTextModel`): Text-encoder, for empty text embedding. tokenizer (`CLIPTokenizer`): CLIP tokenizer. prediction_type (`str`, *optional*): Type of predictions made by the model. target_properties (`Dict[str, Any]`, *optional*): Properties of the predicted modalities, such as `target_names`, a `List[str]` used to define the number, order and names of the predicted modalities, and any other metadata that may be required to interpret the predictions. default_denoising_steps (`int`, *optional*): The minimum number of denoising diffusion steps that are required to produce a prediction of reasonable quality with the given model. This value must be set in the model config. When the pipeline is called without explicitly setting `num_inference_steps`, the default value is used. This is required to ensure reasonable results with various model flavors compatible with the pipeline, such as those relying on very short denoising schedules (`LCMScheduler`) and those with full diffusion schedules (`DDIMScheduler`). default_processing_resolution (`int`, *optional*): The recommended value of the `processing_resolution` parameter of the pipeline. This value must be set in the model config. When the pipeline is called without explicitly setting `processing_resolution`, the default value is used. This is required to ensure reasonable results with various model flavors trained with varying optimal processing resolution values. ztext_encoder->unet->vae) intrinsicsunetvae scheduler text_encoder tokenizerprediction_typetarget_propertiesdefault_denoising_stepsdefault_processing_resolutionc t |||jvr&tj d|d|jd|j ||||||j |||| t|ddr/dt|jjjdz znd |_ ||_ ||_| |_d|_t#|j |_y) Nz*Potentially unsupported `prediction_type='z&'`; values supported by the pipeline: .)r1r2r3r4r5)r6r7r8r9r2rr)vae_scale_factor)super__init__supported_prediction_typesloggerwarningregister_modulesregister_to_configgetattrlenr2configblock_out_channelsr=r7r8r9empty_text_embeddingrimage_processor) selfr1r2r3r4r5r6r7r8r9 __class__s r-r?z#MarigoldIntrinsicsPipeline.__init__s  $"A"A A NN<_$-J*$(!5tG\G\]r,c|jjj|jjjzS)N)r1rG out_channelsr2latent_channels)rKs r- n_targetsz$MarigoldIntrinsicsPipeline.n_targetss+yy,,0O0OOOr,imagenum_inference_steps ensemble_sizeprocessing_resolutionresample_method_inputresample_method_output batch_sizeensembling_kwargslatents generator output_typeoutput_uncertaintyreturnc  dt|jjjdz z} | |jk7rt d|jd| d| t d|dkr t d|dkr t d|dk(rt jd |dk(r | r t d | t d |d kr t d ||jzd k7rt d|jd|dvr t d|dvr t d|dkr t d| dvr t d| t d|1t|ts t dd|vr|ddvr t dd }d\}}t|ts|g}t|D]\}}t|tjstj|r]|j dvrt d|d|j"d|j"dd\}}d}|j d k(rW|j"d }nGt|t$j$r|j&\}}d}nt d!|d"t)|d|||}}n!||f||fk7rt d#|d$||fd%||f||z } | tj| s t d&| j+d k7rt d'| j"d|d kDrz:MarigoldIntrinsicsPipeline.check_inputs..Ms.X188==IaL,?,?,D,DDXs:=z;`generator` device placement is not consistent in the list.zUnsupported generator type: )rFr2rGrHr= ValueErrorrArB isinstancedictlist enumerater&r'r( is_tensorndimshaper sizermdimmaxr1rNall Generator)rKrQrRrSrTrUrVrWrXrYrZr[r\actual_vae_scale_factor num_imagesWHiimgH_iW_iN_imax_orignew_Hnew_Wwhshape_expecteds ` r- check_inputsz'MarigoldIntrinsicsPipeline.check_inputss"#$DHHOO,N,N(ORS(S"T "d&;&; ;A$BWBWAXXwyPxQQST   &vw w  "FG G 1 @A A A  NN?  A "4"  ! (k  !1 $*  !4#8#8 8A =MdNcNcMddefg g (c cC  ")d dC  >=> > l *IJ J  9#8QR R  (/6 !LMM//4Ek4RZl4l !ijj 1%&GE& FAs#rzz*eooc.B889,$wqc1YZ]ZcZcYdde%fgg99RS>S88q=))A,CC-88S #6qc49+Q!OPPyC1QC:% #A3&EsCj\Qkmnpqlrkst # J' ,  ??7+ !DEE{{}! #RSZS`S`Raab!cdd$q(q!911X=11X=A:!$'QRSQTTWXYWZZ[%\]]e1T***Q.43H3HHAT***Q.43H3HHA(=8$)):J:J:W:WYZ\]^N}}. #B7==/Q[\j[kkl!mnn  )T*y>Z-%??$xXiXX$%bcc 5??; #?Y?PPQ!RSSr,cpt|dsi|_n%>?&9&=&=fd&KF#':'>'>w'NG$  8$78 8  ;e;':; ;NO Or,match_input_resolution output_latent return_dictc|j}|j}| |j}| |j}|j |||||||| | | | | }|j `d}|j |d|j jdd}|jj|}|j|d|_|jj|||||\}}}|j|| | ||\}}~|j j||j|d d }g}|jt!d||z|dd D]}||||z}||||z}|j"d}|d|} |j$j'|| |j|j$j(d d D]w}!t+j,||gd }"|j/|"|!| d d}#|j$j1|#|!|| j2}t4sdt7j8y|j;|t+j,|d}~~~~~~ ~"~#|j<||z|j>z|j@jBjDg|j"dd}$t+j,t!d|$j"d|Dcgc]}|jG|$|||zc}d}%~$|sd}|jjI|%|}%d}&|d kDr|%j<|||j>g|%j"d d}%t!|Dcgc]}|jJ|%|| fi| xsi}%}tM|%\}%}&t+j,|%d}%| rt+j,|&d}&nd}&|rB|jjO|%||d }%|&!| r|jjO|&||d }&| dk(r:|jjQ|%}%|&| r|jjQ|&}&|jS|s|%|&|fStU|%|&|Scc}wcc}w)aj Function invoked when calling the pipeline. Args: image (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`), `List[torch.Tensor]`: An input image or images used as an input for the intrinsic decomposition task. For arrays and tensors, the expected value range is between `[0, 1]`. Passing a batch of images is possible by providing a four-dimensional array or a tensor. Additionally, a list of images of two- or three-dimensional arrays or tensors can be passed. In the latter case, all list elements must have the same width and height. num_inference_steps (`int`, *optional*, defaults to `None`): Number of denoising diffusion steps during inference. The default value `None` results in automatic selection. ensemble_size (`int`, defaults to `1`): Number of ensemble predictions. Higher values result in measurable improvements and visual degradation. processing_resolution (`int`, *optional*, defaults to `None`): Effective processing resolution. When set to `0`, matches the larger input image dimension. This produces crisper predictions, but may also lead to the overall loss of global context. The default value `None` resolves to the optimal value from the model config. match_input_resolution (`bool`, *optional*, defaults to `True`): When enabled, the output prediction is resized to match the input dimensions. When disabled, the longer side of the output will equal to `processing_resolution`. resample_method_input (`str`, *optional*, defaults to `"bilinear"`): Resampling method used to resize input images to `processing_resolution`. The accepted values are: `"nearest"`, `"nearest-exact"`, `"bilinear"`, `"bicubic"`, or `"area"`. resample_method_output (`str`, *optional*, defaults to `"bilinear"`): Resampling method used to resize output predictions to match the input resolution. The accepted values are `"nearest"`, `"nearest-exact"`, `"bilinear"`, `"bicubic"`, or `"area"`. batch_size (`int`, *optional*, defaults to `1`): Batch size; only matters when setting `ensemble_size` or passing a tensor of images. ensembling_kwargs (`dict`, *optional*, defaults to `None`) Extra dictionary with arguments for precise ensembling control. The following options are available: - reduction (`str`, *optional*, defaults to `"median"`): Defines the ensembling function applied in every pixel location, can be either `"median"` or `"mean"`. latents (`torch.Tensor`, *optional*, defaults to `None`): Latent noise tensors to replace the random initialization. These can be taken from the previous function call's output. generator (`torch.Generator`, or `List[torch.Generator]`, *optional*, defaults to `None`): Random number generator object to ensure reproducibility. output_type (`str`, *optional*, defaults to `"np"`): Preferred format of the output's `prediction` and the optional `uncertainty` fields. The accepted values are: `"np"` (numpy array) or `"pt"` (torch tensor). output_uncertainty (`bool`, *optional*, defaults to `False`): When enabled, the output's `uncertainty` field contains the predictive uncertainty map, provided that the `ensemble_size` argument is set to a value above 2. output_latent (`bool`, *optional*, defaults to `False`): When enabled, the output's `latent` field contains the latent codes corresponding to the predictions within the ensemble. These codes can be saved, modified, and used for subsequent calls with the `latents` argument. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.marigold.MarigoldIntrinsicsOutput`] instead of a plain tuple. Examples: Returns: [`~pipelines.marigold.MarigoldIntrinsicsOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.marigold.MarigoldIntrinsicsOutput`] is returned, otherwise a `tuple` is returned where the first element is the prediction, the second element is the uncertainty (or `None`), and the third is the latent (or `None`). N do_not_padTrc)padding max_length truncationreturn_tensorsr)rldtyperzMarigold predictions...)rr)rlFzDiffusion steps...rz)encoder_hidden_statesr)rZr)is_aar&)rr r!)+_execution_devicerr8r9rrIr5model_max_length input_idstor4rJ preprocessprepare_latentsrepeatrrangerxr3 set_timesteps timestepsr(catr1step prev_sample XLA_AVAILABLExm mark_stepappendreshaperPr2rGrOdecode_prediction unpad_imageensemble_intrinsicszipresize_antialias pt_to_numpymaybe_free_model_hooksr)'rKrQrRrSrTrrUrVrWrXrYrZr[r\rrrlrrprompt text_inputstext_input_idsroriginal_resolution image_latent pred_latentbatch_empty_text_embedding pred_latentsrbatch_image_latentbatch_pred_latenteffective_batch_sizetextt batch_latentnoisepred_latent_for_decodingrr s' r-__call__z#MarigoldIntrinsicsPipeline.__call__gs<d''   &"&">">  ($($F$F !&&    ! ! "        "  $ $ ,F..$>>::# )K)2255f=N(,(9(9.(I!(LD %/3.B.B.M.M (*?/ +w+%)$8$8 7I}j% ! k %)%>%>%A%AW\%A%]%d%d 1& " "" !Z-/ -.C/CDD NN ( ()O)PVWX  ,[` abcd$(NN$7$71/9%8%+"!LLN #    1 2' 3*ii !4    &      $7;#6#6  & 79X9X$ [f[l[lmnmo[p$  YYq":"@"@"CZP &&'?A N'ST    %K))55j'J  1 +++M4>>>/1Gu?J&+="22CC!46LTYD $ --99*EJ&+="22>>{K  ##% [9 9'!#  y 0s Q/%"Q4c d}tjtd|jd|Dcgc])}||jj ||||z+c}d}||jj jz}|j|d}|j\} } } } |} | 4t| |j| z| | f||j|j} || fScc}w)Nct|dr|jjSt|dr |jSt d)N latent_distrYz3Could not access latents of provided encoder_output)rrmoderYAttributeError)encoder_outputs r-retrieve_latentszDMarigoldIntrinsicsPipeline.prepare_latents..retrieve_latents{sC~}5%1166883%---$%Z[[r,rr)rZrlr) r(rrrxr2encoderGscaling_factorrepeat_interleaverrPrlr)rKrQrYrZrSrWrrrN_ECrrrs r-rz*MarigoldIntrinsicsPipeline.prepare_latentsss \yyq%++a.*= !q1z>1J!KL   $dhhoo&D&DD #55m5K #)) Q1  &dnnq(!Q/##**"(( K[((' s.C)rc|jdk7s0|jd|jjjk7r:t d|jjjd|jd|jj ||jjjz dd}tj|d d }|d zd z }|S) Nrhrz Expecting 4D tensor of shape [B,z ,H,W]; got r;F)rrgg?g@) rzrxr2rGrOrqdecoderr(clip)rKrrs r-rz,MarigoldIntrinsicsPipeline.decode_predictions ??  ![%6%6q%9TXX__=\=\%\2488??3R3R2SS^_j_p_p^qqrs XX__[488??3Q3Q%Q_d_efgh ZZ D#6  3&#- r,targetsrdcJ|jdk7s|jddk7rtd|jd|dvrtd|d|j\}}}}}d}|d k(r4tj|d } |rtj |d }| |fS|d k(rrtj |d d j} |r9tj|| z }tj |d j}| jd } | |fStd|d)a Ensembles the intrinsic decomposition represented by the `targets` tensor with expected shape `(B, T, 3, H, W)`, where B is the number of ensemble members for a given prediction of size `(H x W)`, and T is the number of predicted targets. Args: targets (`torch.Tensor`): Input ensemble of intrinsic image decomposition maps. output_uncertainty (`bool`, *optional*, defaults to `False`): Whether to output uncertainty map. reduction (`str`, *optional*, defaults to `"mean"`): Reduction method used to ensemble aligned predictions. The accepted values are: `"median"` and `"mean"`. Returns: A tensor of aligned and ensembled intrinsic decomposition maps with shape `(T, 3, H, W)` and optionally a tensor of uncertainties of shape `(T, 3, H, W)`. rrz.Expecting 4D tensor of shape [B,T,3,H,W]; got r;rezUnrecognized reduction method: NrgrrrfT)rzkeepdim) rzrxrqr(rgstdrfvaluesabssqueeze) rr\rdBT_rrr rs r-rz.MarigoldIntrinsicsPipeline.ensemble_intrinsicss,0 ;;=A q!1Q!6Mgmm_\]^_ _ . .>ykKL L 1aA  G3J!#iiQ7 ;&&( "g1dCJJJ!#ii*(<= #ll;A>EE #++A.J;&&>ykKL Lr,)NNNN)NNNT)NrNTr`r`rNNNr&FFT)Frf))r"r#r$r%model_cpu_offload_seqr@rrr rrr r rstrrrintr?propertyrPrr(r)r}rboolrcompilerdisablerno_gradrEXAMPLE_DOC_STRINGrrrr staticmethodr __classcell__)rLs@r-r/r/xs$L6!0*.6:157;*^"*^*^ 45 *^ $ *^ ! *^"#*^$DcN3*^"*#*^(0}*^XPP@!@!@ @ # @ # @!$@@$DcN3@%,,'@E%//43H"HIJ@@!@ @D ^^PP$U]]_12.2/3'+%/&06:EIMQ#(# !H !H &c]H  H (} H !% H  #H !$H H $DcN3H % d5<<.@ @ABH E%//43H"HIJH H !H H !H 3H T$)||$)%,,'$)EOO, $)  $)  $) u||U\\) *$)L U\\ ell $)!*'*' *'*' u||Xell33 4 *'*'r,r/)1 dataclassesrtypingrrrrrr numpyr&r(PILr tqdm.autor transformersr r rJrmodelsrr schedulersrrutilsrrrrutils.torch_utilsrpipeline_utilsrmarigold_image_processingrtorch_xla.core.xla_modelcore xla_modelrr get_loggerr"rArrr/r+r,r-rs&":: 51 ..=))MM   H %"J &z& &0Y '!2Y 'r,