bi7~ddlmZddlZddlZddlmZddlmZmZddl m Z ddl m Z m Z ddlmZGd d ee e Zy) )OptionalN)nn) GPT2ConfigGPT2LMHeadModel)ModuleUtilsMixin) ConfigMixinregister_to_config) ModelMixinc(eZdZdZddgZe d(dededeededed ed ed ed eed ede de de de de de de de de f&fd Z d)de jde jdee jdee jfdZdede jde jfd Zd!Ze j$d"Ze j$ d*d#ed$ed%e d&eefd'ZxZS)+UniDiffuserTextDecodera Text decoder model for a image-text [UniDiffuser](https://huggingface.co/papers/2303.06555) model. This is used to generate text from the UniDiffuser image-text embedding. Parameters: prefix_length (`int`): Max number of prefix tokens that will be supplied to the model. prefix_inner_dim (`int`): The hidden size of the incoming prefix embeddings. For UniDiffuser, this would be the hidden dim of the CLIP text encoder. prefix_hidden_dim (`int`, *optional*): Hidden dim of the MLP if we encode the prefix. vocab_size (`int`, *optional*, defaults to 50257): Vocabulary size of the GPT-2 model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`GPT2Model`] or [`TFGPT2Model`]. n_positions (`int`, *optional*, defaults to 1024): The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). n_embd (`int`, *optional*, defaults to 768): Dimensionality of the embeddings and hidden states. n_layer (`int`, *optional*, defaults to 12): Number of hidden layers in the Transformer encoder. n_head (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer encoder. n_inner (`int`, *optional*, defaults to None): Dimensionality of the inner feed-forward layers. `None` will set it to 4 times n_embd activation_function (`str`, *optional*, defaults to `"gelu"`): Activation function, to be selected in the list `["relu", "silu", "gelu", "tanh", "gelu_new"]`. resid_pdrop (`float`, *optional*, defaults to 0.1): The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. embd_pdrop (`float`, *optional*, defaults to 0.1): The dropout ratio for the embeddings. attn_pdrop (`float`, *optional*, defaults to 0.1): The dropout ratio for the attention. layer_norm_epsilon (`float`, *optional*, defaults to 1e-5): The epsilon to use in the layer normalization layers. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. scale_attn_weights (`bool`, *optional*, defaults to `True`): Scale attention weights by dividing by sqrt(hidden_size).. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). scale_attn_by_inverse_layer_idx (`bool`, *optional*, defaults to `False`): Whether to additionally scale attention weights by `1 / layer_idx + 1`. reorder_and_upcast_attn (`bool`, *optional*, defaults to `False`): Whether to scale keys (K) prior to computing attention (dot-product) and upcast attention dot-product/softmax to float() when training with mixed precision. zh\.\d+\.attn\.biaszh\.\d+\.attn\.masked_bias prefix_lengthprefix_inner_dimprefix_hidden_dim vocab_size n_positionsn_embdn_layern_headn_inneractivation_function resid_pdrop embd_pdrop attn_pdroplayer_norm_epsiloninitializer_rangescale_attn_weights use_cachescale_attn_by_inverse_layer_idxreorder_and_upcast_attnc8t|||_||k7r|td|d|d||_||_|j *t j|j|j nt j|_ |j t j|j |nt j|_ tdid|d|d|d|d|d | d | d | d | d | d|d|d|d|d|d|}t||_ y)Nz>`prefix_hidden_dim` cannot be `None` when `prefix_inner_dim`: z and `n_embd`: z are not equal.rrrrrrrrrrrrrrrr )super__init__r ValueErrorrrrLinearIdentity encode_prefix decode_prefixrr transformer)selfrrrrrrrrrrrrrrrrrrr gpt_config __class__s p/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/pipelines/unidiffuser/modeling_text_decoder.pyr$zUniDiffuserTextDecoder.__init__Bs. * v %*;*CPQbPcd$X_6  !1!2%%1 IId++T-C-C D  :>9O9O9[BIId,,f 5acalalan   ! #        !4 $ " "  2 0  2   -L %returncdtj||jtj|S)N)dtyper>)r:zerosrint64)r+rEr>s r.r<z&UniDiffuserTextDecoder.get_dummy_tokens"{{:t'9'9U[\\r/c$|j|S)N)r()r+prefixs r.encodezUniDiffuserTextDecoder.encodes!!&))r/cbtj|dd}g}g}|D]a}|j|j|}|j |||\}}|j |d|j |dctj |}tj |}||fS)a Generate captions given text embedding features. Returns list[L]. Args: features (`torch.Tensor` of shape `(B, L, D)`): Text embedding features to generate captions from. eos_token_id (`int`): The token ID of the EOS token for the text decoder model. device: Device to perform text generation on. Returns: `List[str]`: A list of strings generated from the decoder model. r5rr6) input_embedsr> eos_token_id)r:splitr)to generate_beamappendstack) r+featuresrPr>generated_tokensgenerated_seq_lengthsfeature output_tokens seq_lengthss r.generate_captionsz(UniDiffuserTextDecoder.generate_captionss";;x2 " 9G((F);rH)r8r?r5r6T) descending)rH)r:onesintrIboolr*r9rangelogitssoftmaxlogtopkexpandr=permutesqueezer;floatnpinfview unsqueezeeqallargsortrUtensorrH)r+r0rOr>r]r^r_rPstop_token_indextokensscoresr[ is_stopped generatedioutputsrh next_tokens scores_sumscores_sum_averagenext_tokens_sourcenext_token_embedorder output_textss r.rSz$UniDiffuserTextDecoder.generate_beamsmP(jj6K [[6L  #$I((4488CI|$$ A&&Y&?G^^FAr1H% aSQF^^B'++-F~&,kk)R&@# ,I,,YM9LM &1&9&9!Q&?PQARV >(F*V]]9Hv||AB7GHF"YY '"?QGI#knn5E&F&N&N&PPJ~~I$ L+%$/+01aq 1 1{{g{Gz?TTFF)NN)NNNCrbN)__name__ __module__ __qualname____doc__"_keys_to_ignore_on_load_unexpectedr rerstrrorfr$r:TensorrDr>r<rMno_gradr\rS __classcell__)r-s@r.r r s/b+@A])^& ,0!%#- $(#'#'05(-)>7>7>7$C= >7  >7  >7>7>7>7#>7!>7>7>7>7">7 !!>7"!#>7$%>7&*.'>7("&)>7>7H26)- <<||!.  & @]#]u||] ]*U]]_77>U]]_ &*^)  ^)  ^)^)sm^)^)r/r )typingrnumpyrpr:r transformersrrtransformers.modeling_utilsrconfiguration_utilsr r modelsr r r"r/r.rs0 48B Z)Z6FZ)r/