bi;OddlZddlZddlZddlZddlmZddlmZddlmZm Z ddl m Z ddl Z ddlZddlmZddlmZdd lmZdd lmZdd lmZd d lmZd dlmZd dlmZm Z m!Z!m"Z"m#Z#d dl$m%Z%d dl&m'Z'd dl(m)Z)ee*Z+e"r ddl,m-cm.Z/dej`jbde2fdZ3dej`jbde2fdZ4dej`jbde2fdZ5dej`jbde2fdZ6dej`jbdej`jbfdZ7dej`jbfdZ8dej`jbde2fdZ9 d5de2de2de2fd Z:d!Z;d"ee d'r e j~n e jZZ@e@jje jee jgZEe d(r%eEje jjd7d)ZId*ZJd+ZKd7d,eLde2fd-ZMdeLfd.ZNd/ZOd0ZPd1eQfd2ZRd8dej`jbd3e2deSej`jbfd4ZTy)9N)encode) OrderedDict)partialreduce) MethodType)Version) save_file)write_basic_config) get_logger) PartialState)FSDP_PYTORCH_VERSION)DistributedType)is_deepspeed_availableis_numpy_availableis_torch_distributed_availableis_torch_xla_availableis_weights_only_available)id_tensor_storage) convert_model)is_torch_versionmodulereturncttdsyt|tjjj S)zD Check whether the module was compiled with torch.compile() _dynamoF)hasattrtorch isinstancer eval_frameOptimizedModulers Q/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/accelerate/utils/other.pyis_compiled_moduler$5s. 5) $ femm66FF GGcttdsy|jrE|jD]2}t |tj j js2yy)z\ Check whether the module has submodules that were compiled with `torch.compile()`. rFT)rr_modulesmodulesrrr r!r submodules r#has_compiled_regionsr+?sP 5) $ ) I)U]]%=%=%M%MN  r%cxttjjxrt fdDS)z Check whether the module is a repeated block, i.e. `torch.nn.ModuleList` with all children of the same class. This is useful to determine whether we should apply regional compilation to the module. c3PK|]}t|djyw)rN)r __class__).0mrs r# z%is_repeated_blocks..Ts$:nbc:aPQI\I\;]:ns#&)rrnn ModuleListallr"s`r#is_repeated_blocksr5Ns- fehh11 2 ns:ngm:n7nnr%c`|jr"|jD]}t|syy)z Check whether the module has repeated blocks, i.e. `torch.nn.ModuleList` with all children of the same class, at any level of the module hierarchy. This is useful to determine whether we should apply regional compilation to the module. TF)r'r(r5r)s r#has_repeated_blocksr7Ws1 ) I!),  r%c dtjjdtjjffd |fi|}d|jvr||jd<|S)a_ Performs regional compilation where we target repeated blocks of the same class and compile them sequentially to hit the compiler's cache. For example, in `GPT2LMHeadModel`, the repeated block/class is `GPT2Block`, and can be accessed as `model.transformer.h[0]`. The rest of the model (e.g. model.lm_head) is compiled separately. This allows us to speed up the compilation overhead / cold start of models like LLMs and Transformers in general. See https://pytorch.org/tutorials/recipes/regional_compilation.html for more details. Args: module (`torch.nn.Module`): The model to compile. **compile_kwargs: Additional keyword arguments to pass to `torch.compile()`. Returns: `torch.nn.Module`: A new instance of the model with some compiled regions. Example: ```python >>> from accelerate.utils import compile_regions >>> from transformers import AutoModelForCausalLM >>> model = AutoModelForCausalLM.from_pretrained("gpt2") >>> compiled_model = compile_regions(model, mode="reduce-overhead") >>> compiled_model.transformer.h[0] OptimizedModule( (_orig_mod): GPT2Block( (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D(nf=2304, nx=768) (c_proj): Conv1D(nf=768, nx=768) (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D(nf=3072, nx=768) (c_proj): Conv1D(nf=768, nx=3072) (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) ) ``` rrc t|rLtjj}|D]'}|j tj |fi|)|St |r|jj|j}|jj|ji|_ |jD]\}}|j||fi| |Stj |fi|}|SN)r5rr2r3appendcompiler7r.__new____dict__updater'named_children add_module)rcompile_kwargs new_moduler*name_compile_regionss r#rEz)compile_regions.._compile_regionss f %,,.J# N !!%-- "L^"LM N! ())11&2B2BCJ    & &v 7"$J #)#8#8#: [i%%d,)rrBrCrEs @r#compile_regionsrHes\^uxx "&;N;J*---+1 K( r%c t|r|D]}|jdi|yt|r"|jD]}t |fi|y|jdi|y)a Performs regional compilation the same way as `compile_regions`, but specifically for `DeepSpeedEngine.module`. Since the model is wrapped in a `DeepSpeedEngine` and has many added hooks, offloaded parameters, etc that `torch.compile(...)` interferes with, version of trgional compilation uses the inplace `module.compile()` method instead. Args: module (`torch.nn.Module`): The model to compile. **compile_kwargs: Additional keyword arguments to pass to `module.compile()`. N)r5r<r7childrencompile_regions_deepspeed)rrBr*childs r#rLrLsl&! 0I I   / / 0 V $__& ?E %e >~ > ? ((r%modelcztddrddlmnddlmt fd|j DS)z Check if the model has DTensor parameters. Args: model (`torch.nn.Module`): The model to check. Returns: `bool`: Whether the model has DTensor parameters. >=z2.5.0r)DTensorc36K|]}t|ywr:)r)r/prQs r#r1z$model_has_dtensor..sB!z!W%Bs)rtorch.distributed.tensorrQtorch.distributed._tensorany parameters)rNrQs @r#model_has_dtensorrXs1g&4 6 Bu/?/?/AB BBr%keep_fp32_wrapperkeep_torch_compile recursivec  tjjjtjjf}t |}t |}|r|}|j}n|r|}|jd}tr ddl m }||fz }tdtrtr ddlm} || fz }t#||r|j$}t#||r|r fd  |}|s|j&} |jj)dd} | ._recursive_unwraps[vx(#4V]]#C #) />>@ J e($0A%0HI J# #r%_original_forwardN __wrapped__ _converted_to_transformer_engineF)to_transformer_engine)rr2parallelDistributedDataParallel DataParallelr$r+rFr>r deepspeedr]rrr2torch.distributed.fsdp.fully_sharded_data_parallelr^rrforwardpoprrdrgetattrr) rNrYrZr[options is_compiled has_compiledcompiled_modelr]FSDPrloriginal_forwardrbs @r#extract_model_from_parallelrus(xx  88%((:O:OPG$U+K'.L {+-O%%238V8XgD7 UG $  UG $ $"%( -- >>--.A4H  ''=1!--..'=1'w6EM 5 Make sure all processes will reach this instruction otherwise one of your processes will hang forever. N)r wait_for_everyonerJr%r#rwrw.sN$$&r% state_dictc tjt}|jD]3\}}t |t r|t |j|5|jDcic]\}}t|dkDs||}}}t}|jD]7}|Dcgc] }||vs| }}|j|dd|ddD]}||=9t|dkDrtjd|d|jD cic]2\} } | t | tjr| j!n| 4}} } |Scc}}wcc}wcc} } w)z Cleans the state dictionary from a model and removes tensor aliasing if present. Args: state_dict (`dict`): The state dictionary from a model rNrzRemoved shared tensor zk while saving. This should be OK, but check by verifying that you don't receive any warning while reloading) collections defaultdictlistitemsrstrrr;lensetvaluesr?loggerwarningrTensor contiguous) rxptrsrDtensorptrnames shared_ptrs warn_names found_nameskvs r# clean_state_dict_for_safetensorsr;sp  " "4 (D"((*9 f&#& "6* + 2 24 89 15 O*#uE Q3:OKOJ##% ! ).D1CtD D+ab/*O !D4  ! ! :$ZL1\ ] WaVfVfVhidaQR!z!U\\'BQ\\^IiJi #PEjs3E E3 E"=E" 7E'save_on_each_nodesafe_serializationc|tjtjk(rt j |}|r/t tddi}t|trt|}ntj}tjr |s |||ytjr |r |||yyy)a Save the data to disk. Use in place of `torch.save()`. Args: obj: The data to save f: The file (or file-like object) to use to save the data save_on_each_node (`bool`, *optional*, defaults to `False`): Whether to only save on the global main process safe_serialization (`bool`, *optional*, defaults to `False`): Whether to save `obj` using `safetensors` or the traditional PyTorch way (that uses `pickle`). formatpt)metadataN)r distributed_typerXLAxm_maybe_convert_to_cpursafe_save_filerrrrsaveis_main_processis_local_main_process)objfrr save_funcs r#rr^s$~&&/*=*==&&s+Nh5EF c; '237CJJ ~%%.?#q  - -2C#q3D -r%z2.0.0z1.25.0c 8 trKtjj}d|vrd|d<tjj t n|j ddtj|fd|i|}tr?tjjrtjj ||S#trAtjjr tjj |wwwxYw)a Compatible drop-in replacement of `torch.load()` which allows for `weights_only` to be used if `torch` version is 2.4.0 or higher. Otherwise will ignore the kwarg. Will also add (and then remove) an exception for numpy arrays Args: f: The file (or file-like object) to use to load the data map_location: a function, `torch.device`, string or a dict specifying how to remap storage locations **kwargs: Additional keyword arguments to pass to `torch.load()`. weights_onlyTN map_location) rr serializationget_safe_globalsadd_safe_globalsTORCH_SAFE_GLOBALSrmloadclear_safe_globals)rrkwargsold_safe_globals loaded_objs r#rrs G $ &$22CCE V+)-~&    0 01C D JJ~t ,ZZG GG $ &    2 2 4##445EF  % &    2 2 4##445EF  's A?C A Dct|dst|ds t|d|}t|dr |jSt|dr |jSt |S)z( Gets a pretty name from `obj`. __qualname____name__r.)rrnrrr~)rs r#get_pretty_namersX 3 'Z0Hc;,sN#sJ|| s8Or%c|jD]9\}}t|tr|j|i}t ||5|||<;|S)z Recursively merges two dictionaries. Args: source (`dict`): The dictionary to merge into `destination`. destination (`dict`): The dictionary to merge `source` into. )r}rdict setdefault merge_dicts)source destinationkeyvaluenodes r#rrsUlln% U eT "))#r2D t $$K  % r%portc|d}tjtjtj5}|jd|fdk(cdddS#1swYyxYw)z Checks if a port is in use on `localhost`. Useful for checking if multiple `accelerate launch` commands have been run and need to see if the port is already in use. Ni> D [[F XX0$,,?NAwKw'+..&wi/[\g[his s  sd3 /r%attrcFd}t||g|jdzS)z Recursive `getattr`. Args: obj: A class instance holding the attribute. attr (`str`): The attribute that is to be retrieved, e.g. 'attribute1.attribute2'. ct||Sr:)rn)rrs r#_getattrz#recursive_getattr.._getattrssD!!r%.)rr)rrrs r#recursive_getattrrs%" (SEDJJsO3 44r% return_fqnsc|s|nd|f}|g}g}|r|j}|r|\}}|jD]]\}}t|tjj s+|r r|dz|zn|} |j | |fM|j |_|r|j |fn|j ||r|dddS)aATraverse the model in bottom-up order and return the children modules in that order. Args: model (`torch.nn.Module`): the model to get the children of Returns: `list[torch.nn.Module]`: a list of children modules of `model` in bottom-up order. The last element is the `model` itself. rrN)rmr@rrr2rGr;) rNrtopstackordered_modulescurrent_modulecurrent_module_namerDr child_names r#get_module_children_bottom_uprs#%U C EEO  2@ / (779 'JD$$0EX!4s!:T!A^bJLL*d!34LL&  '   " "$7#H I  " "> 2  4R4  r%)TTF)FFr:)F)Urzrrrcodecsrr functoolsrrtypesrnumpynprpackaging.versionrsafetensors.torchr rcommands.config.defaultr loggingr stater constantsr dataclassesrimportsrrrrrmodelingrtransformer_enginerversionsrrrtorch_xla.core.xla_modelcore xla_modelrr2rGboolr$r+r5r7rHrLrXrurwrrr_corenp_core multiarray _reconstructndarraydtyperr;dtypes UInt32DTyperrrintrrrrr~rr|rrJr%r#rsa #% %98 +((-& H ))HuxxH4H  T ouxxo4o  D EEHHOOE%((//EP)ehhoo)0CUXX__CC*_dR"R?CRW[Rj '  FDdH)1"((rww ##JJ HH hbii334@ $666 "s ""4$55"!!t!X\]b]e]e]l]lXm!r%