bi+i"ddlZddlmZddlmZmZmZddlZddlm Z ddl Z ddl mZmZddlmZmZddlmZmZddlmZmZddlmZmZmZmZdd lmZd d l m!Z!m"Z"d d l#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)d dl*m+Z+e)jXe-Z.Gdde(Z/y)N)UnpicklingError)AnyDictUnion) FrozenDictunfreeze) from_bytesto_bytes) flatten_dictunflatten_dict) create_repohf_hub_download)EntryNotFoundErrorRepositoryNotFoundErrorRevisionNotFoundErrorvalidate_hf_hub_args) HTTPError) __version__is_torch_available) CONFIG_NAMEFLAX_WEIGHTS_NAMEHUGGINGFACE_CO_RESOLVE_ENDPOINT WEIGHTS_NAMEPushToHubMixinlogging)"convert_pytorch_state_dict_to_flaxc eZdZdZeZgdZgdZedZ dde e e fde jded efd Zdde e e fdefd Zdde e e fdefd Zdde e e fdefd Zdej*d e fdZeee j0fde eej6fde jfdZ dde eej6fde e e fdedefdZy)FlaxModelMixina* Base class for all Flax models. [`FlaxModelMixin`] takes care of storing the model configuration and provides methods for loading, downloading and saving models. - **config_name** ([`str`]) -- Filename to save a model to when calling [`~FlaxModelMixin.save_pretrained`]. )_diffusers_version _class_name _name_or_path)nameparentdtypec ||fi|S)zZ All context managers that the model should be initialized under go here. )clsconfigkwargss _/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/models/modeling_flax_utils.py _from_configzFlaxModelMixin._from_config@s 6$V$$Nparamsr&maskreturncfd}|tj||St|}tj|\}}t ||j D]\}} |s || } || || <t |S)zk Helper method to cast floating-point values of given parameter `PyTree` to given `dtype`. ct|tjr?tj|jtj r|j }|SN) isinstancejnpndarray issubdtyper&floatingastype)paramr&s r,conditional_castz:FlaxModelMixin._cast_floating_to..conditional_castMs9%-#..cll2[ U+Lr.)jaxtree_mapr tree_flattenzipkeysr ) selfr/r&r0r< flat_params flat_mask_maskedkeyr;s ` r,_cast_floating_toz FlaxModelMixin._cast_floating_toGs   <<< 0&9 9"6* ''- 1y+*:*:*<= ;KFC#C(#3E#: C  ; k**r.cD|j|tj|S)a Cast the floating-point `params` to `jax.numpy.bfloat16`. This returns a new `params` tree and does not cast the `params` in place. This method can be used on a TPU to explicitly convert the model parameters to bfloat16 precision to do full half-precision training or to save weights in bfloat16 for inference in order to save memory and improve speed. Arguments: params (`Union[Dict, FrozenDict]`): A `PyTree` of model parameters. mask (`Union[Dict, FrozenDict]`): A `PyTree` with same structure as the `params` tree. The leaves should be booleans. It should be `True` for params you want to cast, and `False` for those you want to skip. Examples: ```python >>> from diffusers import FlaxUNet2DConditionModel >>> # load model >>> model, params = FlaxUNet2DConditionModel.from_pretrained("runwayml/stable-diffusion-v1-5") >>> # By default, the model parameters will be in fp32 precision, to cast these to bfloat16 precision >>> params = model.to_bf16(params) >>> # If you don't want to cast certain parameters (for example layer norm bias and scale) >>> # then pass the mask as follows >>> from flax import traverse_util >>> model, params = FlaxUNet2DConditionModel.from_pretrained("runwayml/stable-diffusion-v1-5") >>> flat_params = traverse_util.flatten_dict(params) >>> mask = { ... path: (path[-2] != ("LayerNorm", "bias") and path[-2:] != ("LayerNorm", "scale")) ... for path in flat_params ... } >>> mask = traverse_util.unflatten_dict(mask) >>> params = model.to_bf16(params, mask) ```)rHr6bfloat16rBr/r0s r,to_bf16zFlaxModelMixin.to_bf16_sJ%%fcllDAAr.cD|j|tj|S)a Cast the floating-point `params` to `jax.numpy.float32`. This method can be used to explicitly convert the model parameters to fp32 precision. This returns a new `params` tree and does not cast the `params` in place. Arguments: params (`Union[Dict, FrozenDict]`): A `PyTree` of model parameters. mask (`Union[Dict, FrozenDict]`): A `PyTree` with same structure as the `params` tree. The leaves should be booleans. It should be `True` for params you want to cast, and `False` for those you want to skip. Examples: ```python >>> from diffusers import FlaxUNet2DConditionModel >>> # Download model and configuration from huggingface.co >>> model, params = FlaxUNet2DConditionModel.from_pretrained("runwayml/stable-diffusion-v1-5") >>> # By default, the model params will be in fp32, to illustrate the use of this method, >>> # we'll first cast to fp16 and back to fp32 >>> params = model.to_f16(params) >>> # now cast back to fp32 >>> params = model.to_fp32(params) ```)rHr6float32rKs r,to_fp32zFlaxModelMixin.to_fp32s2%%fckk4@@r.cD|j|tj|S)a Cast the floating-point `params` to `jax.numpy.float16`. This returns a new `params` tree and does not cast the `params` in place. This method can be used on a GPU to explicitly convert the model parameters to float16 precision to do full half-precision training or to save weights in float16 for inference in order to save memory and improve speed. Arguments: params (`Union[Dict, FrozenDict]`): A `PyTree` of model parameters. mask (`Union[Dict, FrozenDict]`): A `PyTree` with same structure as the `params` tree. The leaves should be booleans. It should be `True` for params you want to cast, and `False` for those you want to skip. Examples: ```python >>> from diffusers import FlaxUNet2DConditionModel >>> # load model >>> model, params = FlaxUNet2DConditionModel.from_pretrained("runwayml/stable-diffusion-v1-5") >>> # By default, the model params will be in fp32, to cast these to float16 >>> params = model.to_fp16(params) >>> # If you want don't want to cast certain parameters (for example layer norm bias and scale) >>> # then pass the mask as follows >>> from flax import traverse_util >>> model, params = FlaxUNet2DConditionModel.from_pretrained("runwayml/stable-diffusion-v1-5") >>> flat_params = traverse_util.flatten_dict(params) >>> mask = { ... path: (path[-2] != ("LayerNorm", "bias") and path[-2:] != ("LayerNorm", "scale")) ... for path in flat_params ... } >>> mask = traverse_util.unflatten_dict(mask) >>> params = model.to_fp16(params, mask) ```)rHr6float16rKs r,to_fp16zFlaxModelMixin.to_fp16sJ%%fckk4@@r.rngctd|)Nz.init_weights method has to be implemented for )NotImplementedError)rBrSs r, init_weightszFlaxModelMixin.init_weightss!$RSWRX"YZZr.pretrained_model_name_or_pathc |jdd}|jdd}|jdd}|jdd}|jdd} |jdd} |jd d} |jd d} |jd d} td d d}||j|f|d|| | | | | d|\}}|j|f|dd\}}| |ntj j || }tj j|rW|r|tj jtj j |tstdtd|dtj j |t}ntj jtj j |tr%tj j |t}ntj jtj j |trttd|dtdtdtd|d t||stnt||| | | || |  }|r1t'rd'd(lm}n td)||}t-||}n0 t/|d*5}t1||j3}dddtBjDjGd/}tI|}tCjJ|jLtBjNjQd01}tStItU|jW}tItU|}|tS|jWz }tS|jW|z }|r#tXj[d2|d3|d4||_.|jWD]U} | |vs|| j^|| j^k7s(t#d5| d6|| j^d7|| j^d8|D]}!||!=ta|d0kDrKtXj[d9|d:|jbjdd;|d<|jbjdd= n-tXjgd>|jbjdd?ta|d0kDr4tXj[d@|jbjddA|dB|dCnGtXjgdD|jbjddE|dF|jbjddG|ti|fS#t$rt|dt$rt| d|dt$rt|dtdt $r}td|d|d}~wt"$r%td t$d!|d"tdtd# t$r!td$|d%|d&tdtd wxYw#1swY`xYw#t4t6j8j:f$rx} t/|5}|j3j=d+r t?d,t"|#1swYnxYwn"#t@t"f$rtd-|d.wxYwYd}~d}~wwxYw)Ha Instantiate a pretrained Flax model from a pretrained model configuration. Parameters: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* (for example `runwayml/stable-diffusion-v1-5`) of a pretrained model hosted on the Hub. - A path to a *directory* (for example `./my_model_directory`) containing the model weights saved using [`~FlaxModelMixin.save_pretrained`]. dtype (`jax.numpy.dtype`, *optional*, defaults to `jax.numpy.float32`): The data type of the computation. Can be one of `jax.numpy.float32`, `jax.numpy.float16` (on GPUs) and `jax.numpy.bfloat16` (on TPUs). This can be used to enable mixed-precision training or half-precision inference on GPUs or TPUs. If specified, all the computation will be performed with the given `dtype`. This only specifies the dtype of the *computation* and does not influence the dtype of model parameters. If you wish to change the dtype of the model parameters, see [`~FlaxModelMixin.to_fp16`] and [`~FlaxModelMixin.to_bf16`]. model_args (sequence of positional arguments, *optional*): All remaining positional arguments are passed to the underlying model's `__init__` method. cache_dir (`Union[str, os.PathLike]`, *optional*): Path to a directory where a downloaded pretrained model configuration is cached if the standard cache is not used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, for example, `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. local_files_only(`bool`, *optional*, defaults to `False`): Whether to only load local model weights and configuration files or not. If set to `True`, the model won't be downloaded from the Hub. revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, a commit id, or any identifier allowed by Git. from_pt (`bool`, *optional*, defaults to `False`): Load the model weights from a PyTorch checkpoint save file. kwargs (remaining dictionary of keyword arguments, *optional*): Can be used to update the configuration object (after it is loaded) and initiate the model (for example, `output_attentions=True`). Behaves differently depending on whether a `config` is provided or automatically loaded: - If a configuration is provided with `config`, `kwargs` are directly passed to the underlying model's `__init__` method (we assume all relevant updates to the configuration have already been done). - If a configuration is not provided, `kwargs` are first passed to the configuration class initialization function [`~ConfigMixin.from_config`]. Each key of the `kwargs` that corresponds to a configuration attribute is used to override said attribute with the supplied `kwargs` value. Remaining keys that do not correspond to any configuration attribute are passed to the underlying model's `__init__` function. Examples: ```python >>> from diffusers import FlaxUNet2DConditionModel >>> # Download model and configuration from huggingface.co and cache. >>> model, params = FlaxUNet2DConditionModel.from_pretrained("runwayml/stable-diffusion-v1-5") >>> # Model was saved using *save_pretrained('./test/saved_model/')* (for example purposes, not runnable). >>> model, params = FlaxUNet2DConditionModel.from_pretrained("./test/saved_model/") ``` If you get the error message below, you need to finetune the weights for your downstream task: ```bash Some weights of UNet2DConditionModel were not initialized from the model checkpoint at runwayml/stable-diffusion-v1-5 and are newly initialized because the shapes did not match: - conv_in.weight: found shape torch.Size([320, 4, 3, 3]) in the checkpoint and torch.Size([320, 9, 3, 3]) in the model instantiated You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. ``` r*N cache_dirforce_downloadFfrom_ptproxieslocal_files_onlytokenrevision subfoldermodelflax) diffusers file_type frameworkT)rYreturn_unused_kwargsrZr\r]r^r_r`)r&rfzError no file named z found in directory  z file found in directory z-. Please load the model using `from_pt=True`.z or .) filenamerYrZr\r]r^ user_agentr`r_z is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models' If this is a private repository, make sure to pass a token having permission to this repo with `token` or log in with `hf auth login`.z is not a valid git identifier (branch name, tag name or commit id) that exists for this model name. Check the model page at 'https://huggingface.co/z' for available revisions.z& does not appear to have a file named z:There was a specific connection error when trying to load z: zWe couldn't connect to 'zM' to load this model, couldn't find it in the cached files and it looks like z8 is not the path to a directory containing a file named z. Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.zCan't load the model for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z=' is the correct path to a directory containing a file named r)load_state_dictz|Can't load the model in PyTorch format because PyTorch is not installed. Please, install PyTorch or use native Flax weights.rbversionzYou seem to have cloned a repository without having git-lfs installed. Please install git-lfs and run `git lfs install` followed by `git lfs pull` in the folder you cloned.zUnable to convert z to Flax deserializable object. c\tj|tjddS)Ncpu)backendr)r= device_put local_devices)xs r,z0FlaxModelMixin.from_pretrained..s#3CTCT]bCcdeCf1gr.r)rSzThe checkpoint z is missing required keys: zI. Make sure to call model.init_weights to initialize the missing weights.z)Trying to load the pretrained weight for z failed: checkpoint has shape z, which is incompatible with the model shape z. z(Some weights of the model checkpoint at z! were not used when initializing z: z, - This IS expected if you are initializing zU from the checkpoint of a model trained on another task or with another architecture.z9All model checkpoint weights were used when initializing z. zSome weights of z3 were not initialized from the model checkpoint at z and are newly initialized: zo You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.zAll the weights of z/ were initialized from the model checkpoint at zf. If your task is similar to the task the model of the checkpoint was trained on, you can already use z* for predictions without further training.)5popr load_config from_configospathjoinisdirisfilerEnvironmentErrorrrrrrr ValueErrorrrmodeling_utilsrkropenr readrmsgpack exceptions ExtraData startswithOSErrorUnicodeDecodeErrorr= tree_utilr>r eval_shaperVrandomPRNGKeysetrrAloggerwarning _missing_keysshapelen __class____name__infor )"r)rWr& model_argsr+r*rYrZr[r\r]r^r_r`rj unused_kwargsra model_kwargspretrained_path_with_subfolder model_fileerrrkpytorch_model_filestatestate_fefparams_shape_treerequired_params shape_state missing_keysunexpected_keysrGunexpected_keys" r,from_pretrainedzFlaxModelMixin.from_pretrainedstHd+JJ{D1 $4e<**Y.**Y-!::&8%@ 7D)::j$/JJ{D1 %   >$3COO- %#%)-!1!# % % !FM.coofnEX\n`mn|   *;YG ' 77==7 8ww~~bggll3QS_&`a*.|n\=]^-- '*+<*=T,Oc56a9 1 ,16=.<'#1#%5)'%  d !#;&J "1!< 77I5QE n*d+? &UZZ\):: ejjl+o=  NN!"?!@@[\h[ijZZ !-C ::< Ck!eCj&6&6+c:J:P:P&P ?uDbSz''((TU`adUeUkUkTllnp . &Nn% &  ! # NN:;X:YZ!!&!9!9 :"_&>%?@12388=8P8P7QR nU+++}+ &456hh ) &j!//L.MMgi & &455[\m[nnop &PQnPoore &./N.OP>>[=\]::K9LDQ]P^_]]$ &01N0OP--J,KL//@.Al^STV 0<<#W%7%7%A%AB n nj)4Q668..y9")!6# #-!3444+J7n*-? |Kk+lmmn4 nsn!V Y*$Y?Y*AY0XAYY'"Y**#\ [2[  [ [\[77\\save_directoryis_main_process push_to_hubc 0tjj|rtj d|dytj |d|r|j dd}|j dd}|j dd }|j d d} |j d |jtjjd } t| d|| j} |} |r| j|tjj|t} t| d5} t|}| j!|dddtj#d| |r|j%|  yy#1swY;xYw)a Save a model and its configuration file to a directory so that it can be reloaded using the [`~FlaxModelMixin.from_pretrained`] class method. Arguments: save_directory (`str` or `os.PathLike`): Directory to save a model and its configuration file to. Will be created if it doesn't exist. params (`Union[Dict, FrozenDict]`): A `PyTree` of model parameters. is_main_process (`bool`, *optional*, defaults to `True`): Whether the process calling this is the main process or not. Useful during distributed training and you need to call this function on all processes. In this case, set `is_main_process=True` only on the main process to avoid race conditions. push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs (`Dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. zProvided path (z#) should be a directory, not a fileNT)exist_okcommit_messageprivate create_prFr^repo_id)rrr^wbzModel weights saved in )r^rr)rxryr|rerrormakedirsrusplitsepr r save_configrzrrr writer_upload_folder)rBrr/rrr+rrrr^r model_to_saveoutput_model_filer model_bytess r,save_pretrainedzFlaxModelMixin.save_pretrainedsi8 77>>. ) LL?>*::]^ _  NT2 #ZZ(8$?NjjD1G ;6IJJw-EjjN,@,@,Mb,QRG!'D'QVW__G    % %n 5GGLL9JK #T * !a"6*K GGK  !  -.?-@AB    -#     ! !s 5F  Fr4)TF)r __module__ __qualname____doc__r config_name_automatically_saved_args_flax_internal_args classmethodr-rrrr6r&rrHrLrOrRr=ArrayrVrrNstrrxPathLikerboolrr(r.r,r r 2swK V5%% +dJ.>(?+ +Y\+hk+0%BeD*$45%BS%BNAeD*$45ASA6%AeD*$45%AS%AN[ [d[;;a,',S"++-='>a,yya,a,N !%! @c2;;./@dJ&'@ @  @r.r )0rxpicklertypingrrrr= jax.numpynumpyr6msgpack.exceptionsrflax.core.frozen_dictrrflax.serializationr r flax.traverse_utilr r huggingface_hubr rhuggingface_hub.utilsrrrrrequestsrrrutilsrrrrrrmodeling_flax_pytorch_utilsr get_loggerrrr r(r.r,rsl  "## 63;8 .L   H %~^~r.