bi_ddlZddlmZmZmZmZddlZddlmZddl m Z m Z ddl m Z ddlmZe j eZGdd eZGd d ee ZGd d ej*ZGddej*ZGddej*ZGddej*ZGddej*ZGddej*ZGddej*Zy)N)CallableListOptionalUnion) ConfigMixinregister_to_config)logging) ModelMixinc eZdZdZdedffd Zddejdeee deejfdZ dd e e e jfd ed ed ed ee f dZedee e e jffdZxZS) MultiAdaptera MultiAdapter is a wrapper model that contains multiple adapter models and merges their outputs according to user-assigned weighting. This model inherits from [`ModelMixin`]. Check the superclass documentation for common methods such as downloading or saving. Args: adapters (`List[T2IAdapter]`, *optional*, defaults to None): A list of `T2IAdapter` model instances. adapters T2IAdapterc$tt| t||_t j ||_t|dk(r tdt|dk(r td|dj}|dj}tdt|D]]}||j|k7s||j|k7s(td|d|d|d||jd|d ||j ||_ ||_ y) NrzExpecting at least one adapterr zQFor a single adapter, please use the `T2IAdapter` class instead of `MultiAdapter`zjExpecting all adapters to have the same downscaling behavior, but got: adapters[0].total_downscale_factor=z adapters[0].downscale_factor=z adapter[`z`].total_downscale_factor=z`].downscale_factor=) superr__init__len num_adapternn ModuleListr ValueErrortotal_downscale_factordownscale_factorrange)selfr$first_adapter_total_downscale_factorfirst_adapter_downscale_factoridx __class__s S/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/models/adapter.pyrzMultiAdapter.__init__)s5 lD*,x= h/ x=A => > x=A pq q08{/Q/Q,)1!)E)E&CM* C 448\\C=115SS ::^9_`44R3ST #u$>x}?c?c>de #u$8#9W9W8X Z 'K# >xsadapter_weightsreturnc|1tjd|jz g|jz}ntj|}d}t|||jD]c\}}}||}|'|}t t |D] }|||z||<8t t |D]}||xx|||zz cc<e|S)az Args: xs (`torch.Tensor`): A tensor of shape (batch, channel, height, width) representing input images for multiple adapter models, concatenated along dimension 1(channel dimension). The `channel` dimension should be equal to `num_adapter` * number of channel per image. adapter_weights (`List[float]`, *optional*, defaults to None): A list of floats representing the weights which will be multiplied by each adapter's output before summing them together. If `None`, equal weights will be used for all adapters. Nr )torchtensorrziprrr) rr#r$ accume_statexwadapterfeaturesis r!forwardzMultiAdapter.forwardLs  "#llA0@0@,@+ADDTDT+TUO#ll?;O  _dmmD 7MAq'qzH#' s<01:A&',q/&9LO:s8}-7A Oq8A;6O7 7r"save_directoryis_main_process save_functionsafe_serializationvariantctd}|}|jD]%}|j||||||dz }|d|z}'y)a6 Save a model and its configuration file to a specified directory, allowing it to be re-loaded with the `[`~models.adapter.MultiAdapter.from_pretrained`]` class method. Args: save_directory (`str` or `os.PathLike`): The directory where the model will be saved. If the directory does not exist, it will be created. is_main_process (`bool`, optional, defaults=True): Indicates whether current process is the main process or not. Useful for distributed training (e.g., TPUs) and need to call this function on all processes. In this case, set `is_main_process=True` only for the main process to avoid race conditions. save_function (`Callable`): Function used to save the state dictionary. Useful for distributed training (e.g., TPUs) to replace `torch.save` with another method. Can also be configured using`DIFFUSERS_SAVE_MODE` environment variable. safe_serialization (`bool`, optional, defaults=True): If `True`, save the model using `safetensors`. If `False`, save the model with `pickle`. variant (`str`, *optional*): If specified, weights are saved in the format `pytorch_model..bin`. r)r2r3r4r5r _N)rsave_pretrained) rr1r2r3r4r5rmodel_path_to_saver-s r!r8zMultiAdapter.save_pretrainedisb8+}} @G  # #" /+#5 $  1HC!3#i!?  @r"pretrained_model_pathc d}g}|}tjj|rTtj|fi|}|j ||dz }|d|z}tjj|rTt jt|d|dt|dk(r2tdtjj|d|dzd||S) a Instantiate a pretrained `MultiAdapter` model from multiple pre-trained adapter models. The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train the model, set it back to training mode using `model.train()`. Warnings: *Weights from XXX not initialized from pretrained model* means that the weights of XXX are not pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning. *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, so those weights are discarded. Args: pretrained_model_path (`os.PathLike`): A path to a *directory* containing model weights saved using [`~diffusers.models.adapter.MultiAdapter.save_pretrained`], e.g., `./my_model_directory/adapter`. torch_dtype (`torch.dtype`, *optional*): Override the default `torch.dtype` and load the model under this dtype. output_loading_info(`bool`, *optional*, defaults to `False`): Whether or not to also return a dictionary containing missing keys, unexpected keys and error messages. device_map (`str` or `Dict[str, Union[int, str, torch.device]]`, *optional*): A map that specifies where each submodule should go. It doesn't need to be refined to each parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the same device. To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For more information about each option see [designing a device map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map). max_memory (`Dict`, *optional*): A dictionary mapping device identifiers to their maximum memory. Default to the maximum memory available for each GPU and the available CPU RAM if unset. low_cpu_mem_usage (`bool`, *optional*, defaults to `True` if torch version >= 1.9.0 else `False`): Speed up model loading by not initializing the weights and only loading the pre-trained weights. This also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the model. This is only supported when torch version >= 1.9.0. If you are using an older version of torch, setting this argument to `True` will raise an error. variant (`str`, *optional*): If specified, load weights from a `variant` file (*e.g.* pytorch_model..bin). `variant` will be ignored when using `from_flax`. use_safetensors (`bool`, *optional*, defaults to `None`): If `None`, the `safetensors` weights will be downloaded if available **and** if`safetensors` library is installed. If `True`, the model will be forcibly loaded from`safetensors` weights. If `False`, `safetensors` is not used. rr r7z adapters loaded from .zNo T2IAdapters found under z. Expected at least _0) ospathisdirrfrom_pretrainedappendloggerinforrdirname)clsr:kwargsrrmodel_path_to_loadr-s r!rAzMultiAdapter.from_pretrainedsZ 3ggmm./ 001CNvNG OOG $ 1HC!61SE!B  ggmm./  s8}o%;S.T-UUijCGkGjHHIJ 8}r"N)TNTN)__name__ __module__ __qualname____doc__rrr'Tensorrfloatr0rstrr>PathLikeboolrr8 classmethodrA __classcell__r s@r!rrs !?l!3!?F%,,$u+9NZ^_d_k_kZl@!%"&#'!% (@c2;;./(@(@ (@ ! (@ # (@TAHU3 CS=T4UAAr"rc eZdZdZedgddddfdedeed ed ed ef fd Zd e jdee jfdZ e dZ e dZxZS)ra\ A simple ResNet-like model that accepts images containing control signals such as keyposes and depth. The model generates multiple feature maps that are used as additional conditioning in [`UNet2DConditionModel`]. The model's architecture follows the original implementation of [Adapter](https://github.com/TencentARC/T2I-Adapter/blob/686de4681515662c0ac2ffa07bf5dda83af1038a/ldm/modules/encoders/adapter.py#L97) and [AdapterLight](https://github.com/TencentARC/T2I-Adapter/blob/686de4681515662c0ac2ffa07bf5dda83af1038a/ldm/modules/encoders/adapter.py#L235). This model inherits from [`ModelMixin`]. Check the superclass documentation for the common methods, such as downloading or saving. Args: in_channels (`int`, *optional*, defaults to `3`): The number of channels in the adapter's input (*control image*). Set it to 1 if you're using a gray scale image. channels (`List[int]`, *optional*, defaults to `(320, 640, 1280, 1280)`): The number of channels in each downsample block's output hidden state. The `len(block_out_channels)` determines the number of downsample blocks in the adapter. num_res_blocks (`int`, *optional*, defaults to `2`): Number of ResNet blocks in each downsample block. downscale_factor (`int`, *optional*, defaults to `8`): A factor that determines the total downscale factor of the Adapter. adapter_type (`str`, *optional*, defaults to `full_adapter`): Adapter type (`full_adapter` or `full_adapter_xl` or `light_adapter`) to use. @r[r full_adapter in_channelschannelsnum_res_blocksr adapter_typect||dk(rt|||||_y|dk(rt |||||_y|dk(rt |||||_yt d|d)Nr]full_adapter_xl light_adapterzUnsupported adapter_type: 'zH'. Choose either 'full_adapter' or 'full_adapter_xl' or 'light_adapter'.)rr FullAdapterr- FullAdapterXL LightAdapterr)rr^r_r`rrar s r!rzT2IAdapter.__init__s  > )&{HnN^_DL . .(hP`aDL _ ,' X~O_`DL-l^<88 r"r+r%c$|j|S)a This function processes the input tensor `x` through the adapter model and returns a list of feature tensors, each representing information extracted at a different scale from the input. The length of the list is determined by the number of downsample blocks in the Adapter, as specified by the `channels` and `num_res_blocks` parameters during initialization. )r-rr+s r!r0zT2IAdapter.forward s||Ar"c.|jjSrI)r-rrs r!rz!T2IAdapter.total_downscale_factors||222r"cB|jjjS)zThe downscale factor applied in the T2I-Adapter's initial pixel unshuffle operation. If an input image's dimensions are not evenly divisible by the downscale_factor then an exception will be raised. )r- unshufflerrks r!rzT2IAdapter.downscale_factors ||%%666r")rJrKrLrMr intrrPrr'rNr0propertyrrrTrUs@r!rrs44 !* s)     ,$u||*<3377r"rc eZdZdZdgdddfdedeeded effd Zd ejd eejfd Z xZ S)re2 See [`T2IAdapter`] for more information. rWrXrr\r^r_r`rc t|||dzz}tj||_tj ||ddd|_tjt|d|d|gtdt|Dcgc]}t||dz |||dc}|_ |dt|dz zz|_ ycc}wNrrrWr  kernel_sizepaddingTdown) rrrPixelUnshufflermConv2dconv_inr AdapterBlockrrbodyrrr^r_r`rr/r s r!rzFullAdapter.__init__'s !$4a$77 **+;<yyhqkqRST MMXa[(1+~F #1c(m4!!a%(1+~TXY   '7s8}q?P9Q&Q#s C r+r%c|j|}|j|}g}|jD]}||}|j||S)a This method processes the input tensor `x` through the FullAdapter model and performs operations including pixel unshuffling, convolution, and a stack of AdapterBlocks. It returns a list of feature tensors, each capturing information at a different stage of processing within the FullAdapter model. The number of feature tensors in the list is determined by the number of downsample blocks specified during initialization. rmr{r}rBrr+r.blocks r!r0zFullAdapter.forwardAsS NN1  LLOYY EaA OOA  r" rJrKrLrMrnrrr'rNr0rTrUs@r!rere"sj 4 ! RRs)R R  R4$u||*<r"rec eZdZdZdgdddfdedeeded effd Zd ejd eejfd Z xZ S)rfrqrWrXrr^r_r`rc tt|||dzz}tj||_tj ||ddd|_g|_tt|D]}|dk(r0|jjt||dz |||8|dk(r2|jjt||dz |||do|jjt|||||tj|j|_|dz|_ yrs)rrrryrmrzr{r}rrrBr|rrr~s r!rzFullAdapterXL.__init__Ys !$4a$77 **+;<yyhqkqRST  s8}% YAAv   hq1uox{N![\a   hq1uox{Nae!fg   hqk8A;!WX  YMM$)), &6&:#r"r+r%c|j|}|j|}g}|jD]}||}|j||S)z This method takes the tensor x as input and processes it through FullAdapterXL model. It consists of operations including unshuffling pixels, applying convolution layer and appending each block into list of feature tensors. rrs r!r0zFullAdapterXL.forwardusS NN1  LLOYY EaA OOA  r"rrUs@r!rfrfTse 4 " ;;s); ;  ;8$u||*<r"rfc neZdZdZd dedededeffd Zdejdejfd Z xZ S) r|a# An AdapterBlock is a helper model that contains multiple ResNet-like blocks. It is used in the `FullAdapter` and `FullAdapterXL` models. Args: in_channels (`int`): Number of channels of AdapterBlock's input. out_channels (`int`): Number of channels of AdapterBlock's output. num_res_blocks (`int`): Number of ResNet blocks in the AdapterBlock. down (`bool`, *optional*, defaults to `False`): If `True`, perform downsampling on AdapterBlock's input. r^ out_channelsr`rxc8t|d|_|rtjddd|_d|_||k7rtj ||d|_tjt|Dcgc] }t|c}|_ ycc}w)NrTrustride ceil_moder ru) rr downsampler AvgPool2din_convrz SequentialrAdapterResnetBlockresnets)rr^rr`rxr7r s r!rzAdapterBlock.__init__s{   llqdSDO , &99[,ANDL}}8=n8M N1 . N  Ns;Br+r%c|j|j|}|j|j|}|j|}|S)a This method takes tensor x as input and performs operations downsampling and convolutional layers if the self.downsample and self.in_conv properties of AdapterBlock model are specified. Then it applies a series of residual blocks to the input tensor. )rrrris r!r0zAdapterBlock.forwardsE ?? &"A << # QA LLOr"F rJrKrLrMrnrRrr'rNr0rTrUs@r!r|r|sE   C  s  C  W[  %,,r"r|c`eZdZdZdeffd ZdejdejfdZxZ S)rz An `AdapterResnetBlock` is a helper model that implements a ResNet-like block. Args: channels (`int`): Number of channels of AdapterResnetBlock's input and output. r_ct|tj||dd|_tj |_tj||d|_y)NrWr rtrrrrrzblock1ReLUactblock2rr_r s r!rzAdapterResnetBlock.__init__sE ii(1M 779ii(B r"r+r%cn|j|j|}|j|}||zS)z This method takes input tensor x and applies a convolutional layer, ReLU activation, and another convolutional layer on the input tensor. It returns addition with the input tensor. rrrrr+hs r!r0zAdapterResnetBlock.forward0 HHT[[^ $ KKN1u r" rJrKrLrMrnrr'rNr0rTrUs@r!rrs1CC  %,, r"rc eZdZdZdgdddfdedeeded effd Zd ejd eejfd Z xZ S)rgrqrW)rYrZr[r\r^r_r`rc t|||dzz}tj||_tj t ||d|gtt|dz Dcgc]}t ||||dz|dc}t |d|d|d|_ |dt|zz|_ ycc}w)Nrrr Trw) rrrryrmrLightAdapterBlockrrr}rr~s r!rzLightAdapter.__init__s !$4a$77 **+;<MM!+x{NK #3x=1#45&hqk8AE?NY]^  "(2, nSWX     '7!s8}:L&M#s*C r+r%c|j|}g}|jD]}||}|j||S)z This method takes the input tensor x and performs downscaling and appends it in list of feature tensors. Each feature tensor corresponds to a different level of processing within the LightAdapter. )rmr}rBrs r!r0zLightAdapter.forwardsF NN1 YY EaA OOA  r"rrUs@r!rgrgsj . ! NNs)N N  N4  $u||*< r"rgc neZdZdZd dedededeffd Zdejdejfd Z xZ S) ra< A `LightAdapterBlock` is a helper model that contains multiple `LightAdapterResnetBlocks`. It is used in the `LightAdapter` model. Args: in_channels (`int`): Number of channels of LightAdapterBlock's input. out_channels (`int`): Number of channels of LightAdapterBlock's output. num_res_blocks (`int`): Number of LightAdapterResnetBlocks in the LightAdapterBlock. down (`bool`, *optional*, defaults to `False`): If `True`, perform downsampling on LightAdapterBlock's input. r^rr`rxcdt||dz}d|_|rtjddd|_tj ||d|_tjt|Dcgc] }t|c}|_ tj ||d|_ ycc}w)NrrTrr r) rrrrrrzrrrLightAdapterResnetBlockrout_conv)rr^rr`rx mid_channelsr7r s r!rzLightAdapterBlock.__init__s #q(   llqdSDOyylJ }}V[\jVk&lQR'>|'L&lm  , !L 'ms4B-r+r%c|j|j|}|j|}|j|}|j|}|S)z This method takes tensor x as input and performs downsampling if required. Then it applies in convolution layer, a sequence of residual blocks, and out convolutional layer. )rrrrris r!r0zLightAdapterBlock.forwardsI ?? &"A LLO LLO MM! r"rrrUs@r!rrsJ  MC Ms MC MW[ M  %,, r"rc`eZdZdZdeffd ZdejdejfdZxZ S)ra A `LightAdapterResnetBlock` is a helper model that implements a ResNet-like block with a slightly different architecture than `AdapterResnetBlock`. Args: channels (`int`): Number of channels of LightAdapterResnetBlock's input and output. r_ct|tj||dd|_tj |_tj||dd|_y)NrWr rtrrs r!rz LightAdapterResnetBlock.__init__8sG ii(1M 779ii(1M r"r+r%cn|j|j|}|j|}||zS)z This function takes input tensor x and processes it through one convolutional layer, ReLU activation, and another convolutional layer and adds it to input tensor. rrs r!r0zLightAdapterResnetBlock.forward>rr"rrUs@r!rr.s1NN  %,, r"r)r>typingrrrrr'torch.nnrconfiguration_utilsrr utilsr modeling_utilsr get_loggerrJrCrrModulererfr|rrgrrr"r!rs 22 A&   H %y:yxD7[D7T/"))/d/BII/d-299-`<,299,^( (Vbiir"