bi"zdZddlZddlZddlmZddlmZmZddlm Z m Z ddl m Z ddl Z ddlmZmZd d lmZd d lmZmZd d lmZmZmZer ddlmcmZerdd lmZdZ dZ!dZ"dZ#dZ$e dddZ%dDdZ&dZ'dZ(dZ)dZ*dZ+dZ,dZ-dZ.Gdd e/Z0d!Z1d"Z2e1d#Z3d$e fd%Z4d$e fd&Z5dEd'Z6dFd(Z7e jpd e jrde jtd)e jvd*e jxd+e jzd,e j|d-e j~d.e jd/e jd0i ZBeBjDcic]\}}|| c}}ZDd1ZEdGd2e jfd3ZGe1dEd4eHfd5ZIdEd4eHfd6ZJdHd7ZKdEd8ZLGd9d:eMZNe2dId;ZOdEd<ZPe1dJd=ZQd>ZRGd?d@ZSdAZTdBZUedKdCZVycc}}w)LzB A set of basic tensor ops compatible with tpu, gpu, and multigpu N)Mapping)contextmanager nullcontext)update_wrapperwraps)Any)AcceleratorState PartialState)!TORCH_DISTRIBUTED_OPERATION_TYPES)DistributedTypeTensorInformation)is_npu_availableis_torch_distributed_availableis_torch_xla_available)ReduceOpc6t|tjSN) isinstancetorchTensortensors V/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/accelerate/utils/operations.pyis_torch_tensorr,s fell ++c vt|tjjtjjtjj tjj tjjtjjtjjSr) rrxpu FloatTensor ByteTensor IntTensor LongTensor HalfTensor DoubleTensorBFloat16Tensorrs ris_torch_xpu_tensorr'0sm            rc"t|tSr)rr tensor_infos ris_tensor_informationr+=s k#4 55rcZt|txrt|dxr t|dS)z Checks if `data` is a `namedtuple` or not. Can have false positives, but only if a user is trying to mimic a `namedtuple` perfectly. _asdict_fields)rtuplehasattrdatas r is_namedtupler3As* dE " \wtY'? \GDR[D\\rcht|rt|t|St||S)zO Cast a generator to the same type as obj (list, tuple, or namedtuple) )r3typelist)obj generators r honor_typer9Is2 StCy$y/**tCy##rF test_typeerror_on_other_typec t|ttfrt|fd|DSt|trDt ||j Dcic]\}}|t|gdc}}S|r |giSr2tdt |djdjd|Scc}}w)ad Recursively apply a function on a data structure that is a nested list/tuple/dictionary of a given base type. Args: func (`callable`): The function to recursively apply. data (nested list/tuple/dictionary of `main_type`): The data on which to apply `func` *args: Positional arguments that will be passed to `func` when applied on the unpacked data. main_type (`type`, *optional*, defaults to `torch.Tensor`): The base type of the objects to which apply `func`. error_on_other_type (`bool`, *optional*, defaults to `False`): Whether to return an error or not if after unpacking `data`, we get on an object that is not of type `main_type`. If `False`, the function will leave objects of types different than `main_type` unchanged. **kwargs (additional keyword arguments, *optional*): Keyword arguments that will be passed to `func` when applied on the unpacked data. Returns: The same data structure as `data` with `func` applied to every object of type `main_type`. c3DK|]}t|gdyw)r:Nrecursively_apply).0oargsr<funckwargsr;s r z$recursively_apply..ms@ "!".7M`dj s r:zUnsupported types (z ) passed to `z?`. Only nested list/tuple/dicts of objects that are valid for `z` should be passed.) rr/r6r9rr5itemsr@ TypeError__name__)rDr2r;r<rCrEkvs` ```` rr@r@Ts ,$ &       D' "tDz !JJL  Aq$!".7M`dj    4D*4*6** !$t*]4==/J++4+=+=*>>Q S   K s#C c t|s t|drdk(rd |jSt |ttfrt|fd|DSt |tr^t trgngt||jDcic]\}}||vr|nt|c}}S|S#t$r|jcYSt$r,}t rt trdn|Yd}~nd}~wwxYw |jS#t$r|jcYSwxYwcc}}w) a Recursively sends the elements in a nested list/tuple/dictionary of tensors to a given device. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to send to a given device. device (`torch.device`): The device to send the data to. Returns: The same data structure as `tensor` with all tensors sent to the proper device. tonpuznpu:0) non_blockingznpu:Nc3<K|]}t|yw)rO skip_keysN)send_to_device)rAtdevicerOrRs rrFz!send_to_device..s!ocd^AvLT]^^osrQ)rr0rMrHAssertionErrorrrintr/r6r9rstrr5rGrS)rrUrOrRerrorrJrTs ``` rrSrSsgv'&$"7 U?F 99V,9? ? FUDM * ohno   FG $ i %" I  ItF|#LLN AqY1N1fS_kt,uu    = %99V$ $  !fc*#F8_F   %99V,9? ? %99V$ $ % s5C) E D#/D#7"DD#'D::EEc d}t||S)aK Recursively gathers the information needed to rebuild a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with [`~utils.TensorInformation`] instead of tensors. cDt|j|jS)N)shapedtype)rr\r]rs r_get_data_structurez/get_data_structure.._get_data_structures v||6<<HHrr?)r2r^s rget_data_structurer_sI 0$ 77rc d}t||S)a: Recursively gathers the shape of a nested list/tuple/dictionary of tensors as a list. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with lists of tensor shapes instead of tensors. c,t|jSr)r6r\rs r _get_shapezget_shape.._get_shapesFLL!!rr?)r2rbs r get_shapercs" Z ..rc,d}t||tS)z Recursively initializes tensors from a nested list/tuple/dictionary of [`~utils.TensorInformation`]. Returns: The same data structure as `data` with tensors instead of [`~utils.TensorInformation`]. cTtj|jd|jiSNr])remptyr\r]r)s r_initialize_tensorz.initialize_tensors.._initialize_tensors"{{K--G[5F5FGGrr;)r@r+)data_structurerhs rinitialize_tensorsrksH /K` aarct|tttfr&t |dk(rt dt |dt|ttfrt|dSt|tr%|jD]}t||cSn2t|tjstdt |d|jdS)a Recursively finds the batch size in a nested list/tuple/dictionary of lists of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to find the batch size. Returns: `int`: The batch size. rz&Cannot find the batch size from empty .z0Can only find the batch size of tensors but got ) rr/r6rlen ValueErrorr5find_batch_sizekeysrrrHr\)r2rJs rrprps$g./SY!^A$t*QOPP$ &tAw'' D' " ,A"47+ + , ell +J4PT:,VWXYY ::a=rcD t|S#ttf$rYywxYw)a Same as [`utils.operations.find_batch_size`] except will ignore if `ValueError` and `TypeErrors` are raised Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to find the batch size. Returns: `int`: The batch size. N)rprorHr1s rignorant_find_batch_sizerss, t$$  "   s c d}t||S)aS Recursively finds tensors in a nested list/tuple/dictionary and converts them to a list of numbers. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to convert to regular numbers. Returns: The same data structure as `data` with lists of numbers instead of `torch.Tensor`. c|jj}|jtjk(r|j tj }|jSr)detachcpur]rbfloat16rMfloat32tolistrs r_convert_to_listz!listify.._convert_to_list sF$$& <<5>> )YYu}}-F}}rr?)r2r{s rlistifyr|s -t 44rcPd}t||d}tj|S)Nc|jdk(r|jd}|js|j}t j |S)Nr)ndimclone is_contiguous contiguousxm all_gatherrs r_tpu_gather_onez$_tpu_gather.._tpu_gather_one-sI ;;! \\^D)F##%&&(F}}V$$rTr<)r@r mark_step)rrress r _tpu_gatherr,s%% OV NCLLN Jrcttjjjj dk(rtj jfd}t||dS)Nrc|jdk(r|jd}|js|j}jjdk7rtt j j|jz|jj}|||jdg|jddStjDcgc]}t j|}}t jj!||t j"|dScc}w)Nrgloor]rUr dim)rrrrbackendrrg num_processesnumelr]rUviewsizerange empty_like distributedrcat)routput_tensors_ gather_opstates r_gpu_gather_onez$_gpu_gather.._gpu_gather_oneCs  ;;! \\^D)F##%&&(F == $&)@ #[[##flln4ll||N nf -&>&&r>FKKM!",=> > AFeFYFY@Z[1e..v6[N[    ( ( @99^3 3\s*D?Tr) r rrall_gather_into_tensorrUr5r synchronizer@)rrrrs @@r _gpu_gatherr;sU NE!!88I ||E! 48 _f$ OOrceZdZdZy)DistributedOperationExceptionz An exception class for distributed operations. Raised if the operation cannot be performed due to the shape of the tensors. N)rI __module__ __qualname____doc__rrrrbs   rrc.tfd}|S)zv Verifies that `tensor` is the same shape across all processes. Only ran if `PartialState().debug` is `True`. c 0tjtjk(stjs |i|S j d j }d|vr|d}n|d}tjjt|jk7rgtd|d|jjdtjjdtjjd|d t|}t|g}|df|j|dt|k(}|sDd jt!|Dcgc] \}}d |d |c}}} td |d |  |i|Scc}}w)Nrmrrz%One or more of the tensors passed to z were not on the z+ while the `Accelerator` is configured for z. Please move it to the z before calling z - zProcess z: znCannot apply desired operation due to shape mismatches. All shapes across devices must be valid. Operation: `z` Input shapes: - )r distributed_typerNOdebugrrIrUr5 find_devicerrc gather_objectcountrnjoin enumerate) rCrE operationrshapesoutputare_sameir\process_shape_strfunctions rwrapperz!verify_operation..wrapperps > * *o.@.@ @ H\H\T,V, ,**+1X->->,?@ v H%F!WF > % %V)<)A)A A/7 {BSTZTaTaTfTfSghST`TbTiTiTnTnSoo))5)>)>)C)C(DDTU^T__`b 6"x( !9 ||F1I.#f+=H$,MM[dek[l2mxqRWXaS5'3J2m$n!3''0k1GHYGZ\ ((( 3nsF rrrs` rverify_operationrks"  8_))4 Nrc.tfd}|S)z Checks that `verify_operation` failed and if so reports a more helpful error chaining the existing `DistributedOperationException`. c |i|S#t$r0}jdj}td|d|d}~wwxYw)NrmzError found while calling `z1`. Please see the earlier error for more details.)rrrI)rCrEerrs rrz"chained_operation..wrappersc T,V, ,, #../q1B1B0CDI/-i[8ij  s A+?Arrs` rchained_operationrs"  8_ Nrctjtjk(r t |Stjt vr t |S|S)a4 Recursively gather tensor in a nested list/tuple/dictionary of tensors from all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. Returns: The same data structure as `tensor` with all tensors sent to the proper device. )r rrXLArr rrs rgatherrsF~&&/*=*==6""  ( (,M M6"" robjectcttjDcgc]}d}}tjj |||Dcgc] }|D]}| c}}Scc}wcc}}wr)rr rrrall_gather_object)rroutput_objectsyxs r_gpu_gather_objectrs`$),.*F*F$GHqdHNH ''?% 1!q 1!A 1A 11I 2s A$A)ctjtjk(r t dtjt vr t |S|S)a5 Recursively gather object in a nested list/tuple/dictionary of objects from all devices. Args: object (nested list/tuple/dictionary of picklable object): The data to gather. Returns: The same data structure as `object` with all the objects sent to every device. z&gather objects in TPU is not supported)r rrrNotImplementedErrorr r)rs rrrsG~&&/*=*==!"JKK  ( (,M M!&)) rc(dd}t||d|S)NcHtjj|||S)Nsrc)rr broadcast)rrs r_gpu_broadcast_onez*_gpu_broadcast.._gpu_broadcast_ones  ##F#4 rT)r<rrr?)r2rrs r_gpu_broadcastrs /4UX YYrc Tt|ttfrt|fdt |DSt|t rCt ||jDcic]\}}|t|d|c}}Stj|fdScc}}w)Nc3HK|]\}}t|d|yw)rnameN)_tpu_broadcast)rArrTrs rrFz!_tpu_broadcast..s*"gTQPQ>!TF!A3-#H#H"gs"rrc|Srr)rrs rz _tpu_broadcast..s !C&r) rr6r/r9rrr5rGrr mesh_reduce)rrrrJrKs `` rrrs&4-(&"gU^_eUf"ghh FG $tF|RXR^R^R`a$!QQq$q} EEabb >>$(8 99bs%B$  cd}t}tj|tj|j}|V|j }t |j}tjt||gzt|dt|dzt|d}||j}t|ddd }|dd}||fS) ze Grabs the shape of `tensor` only available on one process and returns a tensor of its shape irNr]r sum reductionrr) r rrgrWrUr\TENSOR_TYPE_TO_INTr]rr6rnreducenonzero)rmax_tensor_dimensionr base_tensorr\ tensor_dtyper]s rgather_tensor_shapers ! NE++2%))ELLYK  )&,,7 (- T%[L>5QY\(] $c%j1n%6Kk1134K  BC # $Ecr"K  rreturnct}t|\}}|7tj|t|j |j }t|dS)a Copys a tensor that only exists on a single device and broadcasts it to other devices. Differs from `broadcast` as each worker doesn't need to know its shape when used (and tensor can be `None`) Args: tensor (`torch.tensor`): The tensor that should be sent to all devices. Must only have it be defined on a single device, the rest should be `None`. rrr)r rrzerosTENSOR_INT_TO_DTYPErMrUr)rrr\r]s rcopy_tensor_to_devicesrsP NE&v.LE5 ~U*=e*DEHHV &E **r from_processctjtjk(rt ||dStjt vr t ||S|S)a Recursively broadcast tensor in a nested list/tuple/dictionary of tensors to all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. from_process (`int`, *optional*, defaults to 0): The process from which to send the data Returns: The same data structure as `tensor` with all tensors broadcasted to the proper device. zaccelerate.utils.broadcast)rrr)r rrrrr r)rrs rrrsM~&&/*=*==f,=YZZ  ( (,M Mf,77 rc,tjtjk(r2t |D]"\}}t j d|fd||<$|Stjtvr!tjj||S)a Broadcast a list of picklable objects form one process to the others. Args: object_list (list of picklable objects): The list of objects to broadcast. This list will be modified inplace. from_process (`int`, *optional*, defaults to 0): The process from which to send the data. Returns: The same list containing the objects from process 0. z&accelerate.utils.broadcast_object_listc|Srr)rrs rrz'broadcast_object_list..>sefgsetrr) r rrrrrrr rrbroadcast_object_list) object_listrrr7s ` rrr/s~&&/*=*== , vFAs^^,TVY[tuKN v   ( (,M M // /N rc"d}t|||S)aN Recursively takes a slice in a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to slice. tensor_slice (`slice`): The slice to take. Returns: The same data structure as `data` with all the tensors slices. c ||Srr)r tensor_slices r _slice_tensorz$slice_tensors.._slice_tensorRs l##rr?)r2r process_indexrrs r slice_tensorsrDs$ ]D, ??rctdttfr.tdfdt t dDStdt rRtddjDcic]!}|tDcgc]}|| c}#c}}StdtjstdtdtjScc}wcc}}w)a Recursively concatenate the tensors in a nested list/tuple/dictionary of lists of tensors with the same shape. Args: data (nested list/tuple/dictionary of lists of tensors `torch.Tensor`): The data to concatenate. dim (`int`, *optional*, defaults to 0): The dimension on which to concatenate. Returns: The same data structure as `data` with all the tensors concatenated. rc3dK|]"}tDcgc]}|| c}$ycc}ww)rN) concatenate)rArdr2rs rrFzconcatenate..fs.#lSTKt0D!10D#$N$N#l0Ds 0 + 0rz%Can only concatenate tensors but got )rr/r6r9rrnrr5rqrrrrHr)r2rrJrs`` rrrXs$q'E4=)$q'#lX]^abfghbi^jXk#lmm DGW %tDG}UYZ[U\UaUaUcdPQaD-Aqad-As!KKdee Q .?T!W OPP 99Ts ##.BdsD  C= D =D c eZdZy)CannotPadNestedTensorWarningN)rIrrrrrrrnsrrc,dd}t||d|||S)a3 Recursively pad the tensors in a nested list/tuple/dictionary of tensors from all devices to the same size so they can safely be gathered. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. dim (`int`, *optional*, defaults to 0): The dimension on which to pad. pad_index (`int`, *optional*, defaults to 0): The value with which to pad. pad_first (`bool`, *optional*, defaults to `False`): Whether to pad at the beginning or the end. c t|ddrtjdt|St |j k\st |j  kr|Sdkrt |j z t j|j |jd}t|j}tfd|D |j k(r|S|j t } |<|jt||z}|r)t fdtt |D}n't fdtt |D}|||<|S) N is_nestedFzHCannot pad nested tensors without more information. Leaving unprocessed.r)rUc3(K|] }| ywrr)rAsrs rrFzFpad_across_processes.._pad_across_processes..s-!qv-sc3bK|]&}|k(rtz n td(ywrslice)rArrmax_sizeold_sizes rrFzFpad_across_processes.._pad_across_processes..s6[\Q#Xh#.95QU;Vs,/c3\K|]#}|k(rtdn td%ywrNrrArrrs rrFzFpad_across_processes.._pad_across_processes..s,oUVqCxE!Xc]3U4[Po),)getattrwarningswarnrrnr\rrrUrrwmaxr6 new_zerosr/r) rr pad_index pad_firstrsizesnew_size new_tensorindicesrrs ` @@r_pad_across_processesz3pad_across_processes.._pad_across_processessK 6; . MMZ, M #fll# #sc&,,.?-?'?M 7 3v||$ $C||FLL?Et   "-u-- v||C( (M<<>  %%eHo6B `efijrfs`tGoZ_`cdl`mZnooG$ 7rT)r<rrrrrFr?)rrrrr!s rpad_across_processesr#rs&" D v4ST]ir rc,dd}t||d|||S)z Takes a `tensor` of arbitrary size and pads it so that it can work given `num_processes` needed dimensions. New tensors are just the last input repeated. E.g.: Tensor: ([3,4,4]) Num processes: 4 Expected result shape: ([4,4,4]) cF ||z}|||zz }||zdk(r||z }n|||zz }|||zcxkDrdkrnn||z }|j t }||z|d<|jt|}t fdt t |D} ||| <|S)Nrr c3\K|]#}|k(rtdn td%ywrrrs rrFz@pad_input_tensors.._pad_input_tensors..s,kQR18a#/tLkr)r\r6rr/rrn) r batch_sizerr remainder last_inputsto_padrrr rs ` @r_pad_input_tensorsz-pad_input_tensors.._pad_input_tensorss-/  I $=>  &! +"Z/F"jM&ABF & ,1 , 6)F<<> 6) %%eHo6 kV[\_`h\iVjkk$ 7rT)r<r'rrrr?)rr'rrr+s rpad_input_tensorsr,s(&  #   rc*dd}t||d||S)aX Recursively reduce the tensors in a nested list/tuple/dictionary of lists of tensors across all processes by the mean of a given operation. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to reduce. reduction (`str`, *optional*, defaults to `"mean"`): A reduction method. Can be of "mean", "sum", or "none" scale (`float`, *optional*): A default scaling value to be applied after the reduce, only valied on XLA. Returns: The same data structure as `data` with all the tensors reduced. c t}|j}|jtjk(r|S|jtj k(rOt jt jt j|g|t jnJ|jjtvr.tjj|tj|dk(r||j z}|S)Nmean)r rrrrrrr all_reduce REDUCE_SUMvaluer rrrSUMr)rrscaler cloned_tensors r_reduce_across_processesz(reduce.._reduce_across_processess   ! !_%7%7 7  ! !_%8%8 8 LLN MM"---% @ LLN  # # ) )-N N    ( ( E   U00 0MrT)r<rr4r/g?r?)rrr4r6s rrrs"$&  &di_d rc*d}d}t|||S)av Recursively converts the elements nested list/tuple/dictionary of tensors in FP16/BF16 precision to FP32. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to convert from FP16/BF16 to FP32. Returns: The same data structure as `tensor` with all tensors that were in FP16/BF16 precision converted to FP32. c"|jSr)floatrs r_convert_to_fp32z)convert_to_fp32.._convert_to_fp32s||~rct|xs t|dxr,|jtjtj fvSrf)rr0r]rfloat16rxrs r_is_fp16_bf16_tensorz-convert_to_fp32.._is_fp16_bf16_tensor s@'C767+C  MM NNZ J  rrir?)rr;r>s rconvert_to_fp32r?s -vAU VVrc"eZdZdZdZdZdZy)ConvertOutputsToFp32ad Decorator to apply to a function outputing tensors (like a model forward pass) that ensures the outputs in FP16 precision will be convert back to FP32. Args: model_forward (`Callable`): The function which outputs we want to treat. Returns: The same function as `model_forward` but with converted outputs. c*||_t||yr) model_forwardr)selfrCs r__init__zConvertOutputsToFp32.__init__!s*t]+rc8t|j|i|Sr)r?rC)rDrCrEs r__call__zConvertOutputsToFp32.__call__%s 1t114B6BCCrc,tjd)NzCannot pickle a prepared model with automatic mixed precision, please unwrap the model with `Accelerator.unwrap_model(model)` before pickling it.)pickle PicklingError)rDs r __getstate__z!ConvertOutputsToFp32.__getstate__(s"" `  rN)rIrrrrErGrKrrrrArAs ,D rrAc6tfd}|_|S)Nc|i|Srr)rCrErCs rforwardz(convert_outputs_to_fp32..forward1sd-f--r)rA __wrapped__)rCrNs` rconvert_outputs_to_fp32rP.s!(7M.(G Nrct|tr'|jD]}t|}||cSyt|tt fr|D]}t|}||cSyt|t jr |jSy)z Finds the device on which a nested dict/list/tuple of tensors lies (assuming they are all on the same device). Args: (nested list/tuple/dictionary of `torch.Tensor`): The data we want to know the device of. N) rrvaluesrr/r6rrrU)r2r7rUs rrr:s$ ;;= C %F!   D5$- ( C %F!   D%,, '{{ (rc#TKtjtjk7s6tj-tjj s t }n#ddl}|jj||||}|5ddddy#1swYyxYww)z Wrapper around `deepspeed.runtime.zero.GatheredParameters`, but if Zero-3 is not enabled, will be a no-op context manager. Nr) modifier_rank fwd_moduleenabled) r rr DEEPSPEEDdeepspeed_pluginis_zero3_init_enabledr deepspeedzeroGatheredParameters)paramsrTrUrVgather_param_contextrZs rr\r\Os**o.G.GG++7 "33IIK*}(~~@@ -JPW A   sB B(B B(B%!B()FNr)rzbroadcast tensorr)NNr"r7)NNT)WrrIrcollections.abcr contextlibrr functoolsrrtypingrrrr r constantsr dataclassesrrimportsrrrtorch_xla.core.xla_modelcore xla_modelrtorch.distributedrrr'r+r3r9r@rSr_rcrkrprsr|rr Exceptionrrrrrrrrr:doublehalfrxuint8int8int16int32int64boolrrGrrrrrWrrrr UserWarningrr#r,rr?rArPrr\)rJrKs00rrts3#2+ 28;))!#*, 6]$4CX]0f1h8$/$ b."5. $PN I  F&&2s2#&Z: KK LL! JJ NNA KK JJ KK KK KK JJ );(@(@(BC1q!tC2+5<<+"C*S*@($, ; 44n%P&&RW0  4 *I Ds G