bi!ddlmZddlmZddlmZddlmZddlm Z ddl m Z ddl m Z mZmZdd lmZej$eZGd d Zy ) ) nullcontext)Dict)!SD3IPAdapterJointAttnProcessor2_0)IPAdapterTimeImageProjection)load_model_dict_into_meta)_LOW_CPU_MEM_USAGE_DEFAULT)is_accelerate_availableis_torch_versionlogging)empty_device_cachecXeZdZdZefdededefdZefdededefdZ efdededdfdZ y) SD3Transformer2DLoadersMixinzBLoad IP-Adapters and LoRA layers into a `[SD3Transformer2DModel]`. state_dictlow_cpu_mem_usagereturnc|r(trddlm}nd}tj d|durt dds t d|jj|jjz}|jj|jjz}|d jd }tt|jDcic]}|i}}|jD],\} } | jd d \}} | |t!|| <.i} |rnt"} t%|jj'D]\}} | 5t)|||jj| | | <ddd|s| | j+||dXd|j,i}t/| | ||||j0t3| Scc}w#1swYexYw)Nrinit_empty_weightsF,Cannot initialize model with low cpu memory usage because `accelerate` was not found in the environment. Defaulting to `low_cpu_mem_usage=False`. It is strongly recommended to install `accelerate` for faster and less memory-intense model loading. You can do so with: ``` pip install accelerate ``` .T>=1.9.0~Low memory initialization requires torch >= 1.9.0. Please either update your PyTorch version or set `low_cpu_mem_usage=False`.z0.norm_ip.linear.weight.)maxsplit) hidden_sizeip_hidden_states_dimhead_dimtimesteps_emb_dimstrict device_mapdtype)r acceleraterloggerwarningr NotImplementedErrorconfigattention_head_dimnum_attention_headsshaperangelenattn_processorsitemssplitintr enumeratekeysrload_state_dictdevicerr&r )selfrrrrrr idxlayer_state_dictkeyweightsname attn_procs init_contextr%s \/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/loaders/transformer_sd3.py%_convert_ip_adapter_attn_to_diffuserszBSD3Transformer2DLoadersMixin._convert_ip_adapter_attn_to_diffuserss &(9%*!2  $-=dG-L%.  kk44t{{7V7VV #{{== @_@__&'@AGGJ05S9M9M5N/OPCGPP&,,. 7LC # 2IC/6 SX &t , 7  ->)K "4#7#7#<#<#>? IC #D +)=![[;;&7 $ 4  %4 001A#1Ft0T $++. )t$&6s&; Z^ZdZd " 7Q  s G'G$$G- c |r(trddlm}nd}tj d|durt dds t d|rnt}i}|jD]\}}|jd r|jd d }|jd |d d |d }|jd |dd |d}|jd |dd |d}|jd |dd |d}|jd |dd |d}|jd |dd |d}|jd |dd |d}|jd |dd |d}|jd |dd |d}|||<|djd } |djd} |djd} |d jdd!z} |d"jd } |d#jd }|5t| | | | | |$}ddd|sj|d%|Sd&|ji}t!|||j"'t%|S#1swYSxYw)(NrrFrTrrrzlayers.rrz.0.norm1z.ln0z.0.norm2z.ln1z.0.to_qz .attn.to_qz.0.to_kvz .attn.to_kvz .0.to_outz.attn.to_out.0z.1.0z .adaln_normz.1.1z.ff.net.0.projz.1.3z .ff.net.2z.2.1z .adaln_projzproj_in.weightzproj_out.weightzlayers.0.attn.to_q.weight@latentsztime_embedding.linear_1.weight) embed_dim output_dim hidden_dimheads num_queriestimestep_in_dimr!r#r$)r r'rr(r)r r*rr2 startswithr3replacer.rr7r8rr&r )r9rrrr@updated_state_dictr<valuer:rFrGrHrIrJrK image_projr%s rA+_convert_ip_adapter_image_proj_to_diffuserszHSD3Transformer2DLoadersMixin._convert_ip_adapter_image_proj_to_diffusersYs &(9%*!2  $-=dG-L%.  .?)K  $**, ,JC~~i(iinQ'kkGC5"9WSE;NOkkGC5"9WSE;NOkkGC5"8GC5 :STkkGC5"9WSE;UVkkGC5 ":gcU.>qA '(9:@@C '(89??B "#>?EEaHBN(399!< ,-MNTTUVW^ 5#%%' / J !  & &'9$ & G  dkk*J %j2DQ[cgcmcm n  #  s 6II#Nc|j|d|}|j||j|d||_y)aISets IP-Adapter attention processors, image projection, and loads state_dict. Args: state_dict (`Dict`): State dict with keys "ip_adapter", which contains parameters for attention processors, and "image_proj", which contains parameters for image projection net. low_cpu_mem_usage (`bool`, *optional*, defaults to `True` if torch version >= 1.9.0 else `False`): Speed up model loading only loading the pretrained weights and not initializing the weights. This also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the model. Only supported for PyTorch >= 1.9.0. If you are using an older version of PyTorch, setting this argument to `True` will raise an error. ip_adapterrPN)rBset_attn_processorrQrP)r9rrr?s rA_load_ip_adapter_weightsz5SD3Transformer2DLoadersMixin._load_ip_adapter_weightssE?? <@XZkl   +JJ:VbKcevw) __name__ __module__ __qualname____doc__r rboolrBrrQrUrVrArrswL;U88378 8v;UCC37C %CJTnx4xDxrvxrVrN) contextlibrtypingrmodels.attention_processorrmodels.embeddingsrmodels.model_loading_utilsrmodels.modeling_utilsr utilsr r r utils.torch_utilsr get_loggerrWr(rr\rVrArfsD#J<B>FF2   H %SxSxrV