biddlZddlZddlZddlmZmZddlmZddlmZddl m Z m Z m Z erddl mZmZe rddl mZmZe rddlZdd d ed eeefd d fd Zddd ed eeefd dfdZy)N) TYPE_CHECKINGDict) DDUFEntry)tqdm)is_safetensors_availableis_transformers_availableis_transformers_version)PreTrainedModelPreTrainedTokenizerclsr name dduf_entriesreturnc tj5}|jD]\}}|j|dzst j j |g|jd}t jtj j|dt|d5}|j5} |j| dddddd|jtj jfi|cdddS#1swYLxYw#1swYxYw#1swYyxYw)a Load a tokenizer from a DDUF archive. In practice, `transformers` do not provide a way to load a tokenizer from a DDUF archive. This function is a workaround by extracting the tokenizer files from the DDUF archive and loading the tokenizer from the extracted files. There is an extra cost of extracting the files, but of limited impact as the tokenizer files are usually small-ish. /T)exist_okwbN)tempfileTemporaryDirectoryitems startswithospathjoinsplitmakedirsdirnameopenas_mmapwritefrom_pretrained) r rrkwargstmp_dir entry_nameentrytmp_entry_pathfmms i/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/pipelines/transformers_loading_utils.py_load_tokenizer_from_ddufr+$s  $ $ & N'!-!3!3!5 $ J$$TCZ0!#g!N 8H8H8M!N BGGOON;dK.$/$1$B $$$  $#s""277??>#BMfM N N$$$$ N NsB+D7A/D70D*D D*9D7D' #D**D4 /D77Er c X|j|d}| td|d|jd|j|dd}|jDcgc],\}}|j |dr|j dr|.}}}|s td |d|jdt s td td d r td tj5} ddl m } m } tjj!| d} t#| d5} | j%|j'ddd| j)| }|etjj!| d}t#|d5} | j%|j'ddd| j)|}i}t+j,5}t/|dD]Q}|j1|j3}t4j6j9|}|j;|S|j(dd|||d|cdddcdddScc}}w#1swY)xYw#1swYxYw#1swYnxYw dddy#1swYyxYw)a Load a transformers model from a DDUF archive. In practice, `transformers` do not provide a way to load a model from a DDUF archive. This function is a workaround by instantiating a model from the config file and loading the weights from the DDUF archive directly. z /config.jsonNz0Could not find a config.json file for component z in DDUF file (contains z).z/generation_config.jsonrz .safetensorsz-Could not find any weight file for component z\Safetensors is not available, cannot load model from DDUF. Please `pip install safetensors`.4.47.0` in order to load a transformers model from a DDUF file. You can install it with: `pip install --upgrade transformers`r) AutoConfigGenerationConfigz config.jsonwzgeneration_config.jsonzLoading state_dict)desc)pretrained_model_name_or_pathconfiggeneration_config state_dict)getEnvironmentErrorkeysrrendswithrr ImportErrorrr transformersr.r/rrrrr! read_textr" contextlib ExitStackr enter_contextr safetensorstorchloadupdate)r rrr# config_filer4r%r& weight_filesr$r.r/tmp_config_filer(r3tmp_generation_config_filer5stackmmaptensorss r*"_load_transformers_model_from_ddufrL;s""dV<#89K>tfD\]i]n]n]p\qqs t  %((D61H)I4P".!3!3!5 J  D6 ,1D1D^1T L ;D6AYZfZkZkZmYnnp q   $ % j  sH- L   $ $ &'='',,w > /3 ' -1 GGK))+ , -++O<  ()+g?W)X &0#6 7!)3356 7 0 @ @A[ \   ! ! # ul1EF +**5==?;%++006!!'*  +'3&&.2"3%     '. - -  7 7   s\1I+55J * I1 AJ  I>1/J  A8J  J 1I; 6J >J J  J J  J))r>rrtypingrrhuggingface_hubrrutilsrr r r<r r safetensors.torchrAstrr+rLr6r*rSs &%``AAN N&)N9=c9n9MNN.> >"%>59#y.5I>>rR