biOhddlZddlZddlmZddlZddlmZdde fdZ ejjjddfde de fdZGdd ejZGd d ejZGd d ejZGddejZGddejZy)Nkey_chunk_sizecjdd\}jdt|tjz t j t jdfdfd}t jj|tjd| \}}} tj| dd } tj| | z } |tj| d z}|| z}|jd } tj|djd } | | z S) zBMulti-head dot product attention with a limited number of queries.NF) prevent_csec\tjd||}tj|dd}tjj |}tj ||z }tjd||}tjd|}||jd|fS) Nz...qhd,...khd->...qhk) precisionrTaxiskeepdimsz...vhf,...qhv->...qhfz ...qhk->...qhr )jnpeinsummaxjaxlax stop_gradientexpsum)querykeyvalue attn_weights max_score exp_weights exp_valuesr s Z/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/models/attention_flax.pysummarize_chunkz/_query_chunk_attention..summarize_chunkszz"95#QZ[ GGLrDA GG)))4 gglY67 ZZ 7 W`a JJ : KOOO4i@@c ltjjdgjdz z|ddgzt j ddgz}tjj dg jdz z|ddgzt j dd gz}||S)Nrroperand start_indices slice_sizes)rr dynamic_slicendimlistshape) chunk_idx key_chunk value_chunk k_featuresrr num_headsrr v_featuresrs r chunk_scannerz-_query_chunk_attention..chunk_scanner+sGG))#A.)Q1BBSYYs^, :/VV* gg++#a0Iq!3DDU[["-..)Z1XX, ui==r r)fxsTr r)r*minrsqrt functoolspartialr checkpointrmaparangerr expand_dimsr)rrrr rnum_kvr1 chunk_values chunk_weights chunk_max global_max max_diffs all_values all_weightsr.r/rr0s````` @@@@r_query_chunk_attentionrDs$$'IIbcN!FIzRJ0N CHHZ( (Es~~59 A: A>>".1WW[[=SZZXY[acqMr[-s*L-T:J J./ICOOIB77LYM!!q!)J//-488a8@K  ##r iquery_chunk_sizec  jdd\   f d}tjj|ddt j z \}}t j|dS)a Flax Memory-efficient multi-head dot product attention. https://huggingface.co/papers/2112.05682v2 https://github.com/AminRezaei0x443/memory-efficient-attention Args: query (`jnp.ndarray`): (batch..., query_length, head, query_key_depth_per_head) key (`jnp.ndarray`): (batch..., key_value_length, head, query_key_depth_per_head) value (`jnp.ndarray`): (batch..., key_value_length, head, value_depth_per_head) precision (`jax.lax.Precision`, *optional*, defaults to `jax.lax.Precision.HIGHEST`): numerical precision for computation query_chunk_size (`int`, *optional*, defaults to 1024): chunk size to divide query array value must divide query_length equally without remainder key_chunk_size (`int`, *optional*, defaults to 4096): chunk size to divide key and value array value must divide key_value_length equally without remainder Returns: (`jnp.ndarray`) with shape of (batch..., query_length, head, value_depth_per_head) rNc  tjj dg jdz z|ddgzt j ddt gz}| zt| fS)Nrr"rr#)rrrr r)rrr'r(r)r*r4rD) r+_ query_chunkrrr/num_qr q_featuresrrErs rr1z5jax_memory_efficient_attention..chunk_scannerasgg++3%**q.1iA5FFU[["-.#6F2NPY[e1ff,  ( ( "!s%9]k   r r)r2initr3lengthr)r*rrscanmathceilr concatenate) rrrr rErr1rHresr/rJrKs `````` @@@rjax_memory_efficient_attentionrSJsr*$);;rs#3 E9j   WW\\   yy!112 FAs ??3R ((r ceZdZUdZeed<dZeed<dZeed<dZe ed<d Z e ed <d Z e ed <e jZe jed <d ZdZdZddZy) FlaxAttentiona A Flax multi-head attention module as described in: https://huggingface.co/papers/1706.03762 Parameters: query_dim (:obj:`int`): Input hidden states dimension heads (:obj:`int`, *optional*, defaults to 8): Number of heads dim_head (:obj:`int`, *optional*, defaults to 64): Hidden states dimension inside each head dropout (:obj:`float`, *optional*, defaults to 0.0): Dropout rate use_memory_efficient_attention (`bool`, *optional*, defaults to `False`): enable memory efficient attention https://huggingface.co/papers/2112.05682 split_head_dim (`bool`, *optional*, defaults to `False`): Whether to split the head dimension into a new axis for the self-attention computation. In most cases, enabling this flag should speed up the computation for Stable Diffusion 2.x and Stable Diffusion XL. dtype (:obj:`jnp.dtype`, *optional*, defaults to jnp.float32): Parameters `dtype` query_dimheads@dim_headdropoutFuse_memory_efficient_attentionsplit_head_dimdtypec|j|jz}|jdz|_tj|d|j d|_tj|d|j d|_tj|d|j d|_tj|j|j d|_ tj|j |_ y) NgFto_q)use_biasr_nameto_kto_vto_out_0)r_rcrate)rZrXscalennDenser_rrrrV proj_attnDropoutr\ dropout_layerself inner_dims rsetupzFlaxAttention.setupsMMDJJ. ]]D( XXi%tzzPVW 88ITZZfUXXi%tzzPVW $.. TZZT\\:r c|j\}}}|j}|j|||||z}tj|d}|j||z|||z}|SN)rr"r*rXreshaper transposerptensor batch_sizeseq_lendim head_sizes rreshape_heads_to_batch_dimz(FlaxAttention.reshape_heads_to_batch_dimse#)<< GSJJ  GYy@PQv|4 Y 6 AQR r c|j\}}}|j}|j||z|||}tj|d}|j||z|||z}|Srtrwrzs rreshape_batch_dim_to_headsz(FlaxAttention.reshape_batch_dim_to_headssd#)<< GSJJ  i 7GSQv|4 i 7# /R r Nc||n|}|j|}|j|}|j|}|jr|jd}t j ||d|j|jf}t j ||d|j|jf} t j ||d|j|jf} n3|j|}|j|} |j|} |jr|jddd}| jddd} | jddd} |jd} | dzdk(rt| dz } n9| dzdk(rt| dz } n"| dzdk(rt| dz } n t| } t|| | | d }|jddd}|j|}n|jrt jd | |} nt jd || } | |j z} t#j$| |jrdnd }|jrWt jd|| }|jd}t j ||d|j|jzf}n(t jd|| }|j|}|j'|}|j)||S)NrrrvrurrYi@)rErzb t n h, b f n h -> b n f tzb i d, b j d->b i jrzb n f t, b t n h -> b f n hzb i j, b j d -> b i d deterministic)rrrr^r*rrxrXrZrr]ryintrSrrrirjsoftmaxrlrn)rp hidden_statescontextr query_projkey_proj value_projb query_states key_states value_statesflatten_latent_dimrEattention_scoresattention_probss r__call__zFlaxAttention.__call__s#*?-ZZ . 88G$ZZ(   ##A&A;;zAr4::t}}3UVLX2tzz4==/QRJ;;zAr4::t}}3UVL:::FL88BJ:::FL  . .'11!Q:L#--aA6J'11!Q:L ".!3!3B!7 !B&!+#&'9B'>#? #b(A-#&'9B'>#? #a'1,#&'9A'=#> #&'9#: :j,IYjrM*33Aq!