bi3~ddlZddlZddlmZddlmZmZmZddlm Z m Z ddl m Z erddl mZeGddZy) N) dataclass) TYPE_CHECKINGOptionalUnion)TorchContextParallelConfigTorchTensorParallelConfig)is_torch_version) AcceleratorceZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ e de fed<dZ e defed<dZd Zd Zed Zed Zed ZedZedZedZedZedZedZedZedZedZedZde fdZ!d#de"e fdZ#de$e$edfe$e dfffdZ%dZ&de d efd!Z'd$d"Z(y)%ParallelismConfiga A dataclass to configure parallelisms applied to the model. Inspired by torchtitan's `ParallelDims` https://github.com/pytorch/torchtitan/blob/main/torchtitan/distributed/parallel_dims.py Args: dp_replicate_size (`int`, defaults to `1`): The size of the data parallel group. If `dp_replicate_size` is set to 1, the data parallel replication group will not be used. dp_shard_size (`int`, defaults to `1`): The size of the model shard group. If `dp_replicate_size > 1` and `tp_size > 1`, `dp_shard_size` must also be greater than 1, as composing DDP + TP is currently not supported. tp_size (`int`, defaults to `1`): The size of the tensor parallel group. If `tp_size` is set to `1`, the tensor parallel group will not be used. cp_size (`int`, defaults to `1`): The size of the context parallel group. Currently not supported, but reserved for future use and enabled for downstream libraries. tp_handler (`~utils.TorchTensorParallelConfig`, defaults to `None`): The handler for the tensor parallel group. You may obtain different distributed data parallel paradigms by configuring `dp_replicate_size` and `dp_shard_size` together: - `dp_replicate_size == 1` and `dp_shard_size > 1`, we obtain Fully Sharded Data Parallel (FSDP). - `dp_replicate_size > 1` and `dp_shard_size > 1`, we obtain Hybrid Sharded Data Parallel (HSDP). - `dp_replicate_size > 1` and `dp_shard_size == 1` is an invalid configuration, to use pure DP, use `DistributedDataParallelKwargs` instead. Ndp_replicate_size dp_shard_sizetp_sizecp_size tp_handler cp_handlercd|jd|jd|jd|jd|jd|j d|j dS) Nz'ParallelismConfig( dp_replicate_size=z, dp_shard_size=z , tp_size=z , cp_size=z, total_size=z tp_handler=z, cp_handler=z) )r rrr total_sizerrselfs X/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/accelerate/parallelism_config.py__repr__zParallelismConfig.__repr__Fsy ##'#9#9":;#1123'' OO,- OO,- OO,C 1 c ddl}dg}|j|jjDcic]3\}}||vr*|t |dr|j|jn|5c}}ycc}}w)Nr device_mesh__dict__)copydeepcopyritemshasattr)rr_non_serializable_fieldskvs rto_jsonzParallelismConfig.to_jsonRsm$1?  !MM//1 Aq44:0F4==,AM   s8A1 cRg}|jr|dgz }|jr|dgz }|S)zENames of enabled dimensions across which data parallelism is applied. dp_replicatedp_shard)dp_replicate_enableddp_shard_enabledrdimss r dp_dim_nameszParallelismConfig.dp_dim_names_s9  $ $ ^$ $D  ZL D rcRg}|jr|dgz }|jr|dgz }|S)z]Names of enabled dimensions which will receive the same batch (non-data parallel dimensions).tpcp) tp_enabled cp_enabledr*s rnon_dp_dim_namesz"ParallelismConfig.non_dp_dim_namesis2 ?? TFND ?? TFND rcRg}|jr|dgz }|jr|dgz }|S)zlNames of enabled dimensions which will be flattened into a joint mesh across which is model sharded in FSDP.r'r/)r)r1r*s rdp_shard_cp_dim_namesz'ParallelismConfig.dp_shard_cp_dim_namesss5  ZL D ?? TFND rcvg}|jr|dgz }|jr|dgz }|jr|dgz }|S)z@Names of enabled dimensions across which loss should be averagedr&r'r/)r(r)r1r*s rdp_cp_dim_namesz!ParallelismConfig.dp_cp_dim_names}sK  $ $ ^$ $D  ZL D ?? TFND rc:g}|jr|dgz }|dgz }|S)z^Names of enabled dimensions across which FSDP is applied, including data parallel replication.r& dp_shard_cp)r(r*s rfsdp_dim_namesz ParallelismConfig.fsdp_dim_namess0  $ $ ^$ $D  rch|j|jz|jz|jzS)zSThe total size of the parallelism configuration, which is the product of all sizes.)r rrrrs rrzParallelismConfig.total_sizes-%%(:(::T\\IDLLXXrc4|j|jzS)zhThe size of the non-data parallel dimensions, which is the product of tensor and context parallel sizes.)rrrs rnon_data_parallel_sizez(ParallelismConfig.non_data_parallel_sizes||dll**rc4|j|jzS)z_The size of the data parallel dimensions, which is the product of data parallel replication and)r rrs rdata_parallel_sizez$ParallelismConfig.data_parallel_sizes%%(:(:::rc |jdkDS)zKTrue if data parallel replication is enabled, i.e. `dp_replicate_size > 1`.)r rs rr(z&ParallelismConfig.dp_replicate_enableds%%))rc |jdkDS)zDTrue if data parallel sharding is enabled, i.e. `dp_shard_size > 1`.r@)rrs rr)z"ParallelismConfig.dp_shard_enableds!!A%%rc |jdkDS)z:True if tensor parallelism is enabled, i.e. `tp_size > 1`.r@)rrs rr0zParallelismConfig.tp_enabled||arc |jdkDS)z;True if context parallelism is enabled, i.e. `cp_size > 1`.r@)rrs rr1zParallelismConfig.cp_enabledrCrc4|j|jzS)z$Names of all active mesh dimensions.)r,r2rs ractive_mesh_dimsz"ParallelismConfig.active_mesh_dimss  4#8#888r device_typectddrddlm}n td|j }t |dk(ry|\}}||||}|j r||j jd|jr||jjd |jr||jjd |S) a!Builds a device mesh for the given device type based on the parallelism configuration. This method will also create required joint meshes (e.g. `dp_shard_cp`, `dp_cp`, `dp`). Args: device_type (`str`): The type of device for which to build the mesh, e z>=z2.2.0r)init_device_meshz4Building a device_mesh requires to have torch>=2.2.0N)mesh_dim_namesdpr8dp_cp) r torch.distributed.device_meshrI RuntimeError _get_meshlenr,_flattenr4r6)rrGrImeshrJ mesh_shapers rbuild_device_meshz#ParallelismConfig.build_device_meshs D' * FUV V~~ t9>%)" &  )    )) * 3 3D 9  % % 22 3 < <] K    ,, - 6 6w ?rc|j&|"|j||_|jSd|?|jj|k7r&td|jjd|d|jS)Nz@You need to pass a device_type e.g cuda to build the device meshz4The device_mesh is already created with device type z@. However, you are trying to get a device mesh with device_type z<. Please check if you correctly initialized your device_mesh)rrTrG ValueError)rrGs rget_device_meshz!ParallelismConfig.get_device_meshs    #&#'#9#9+#F ZZ&##//;>$NtO_O_OkOkNlmmnymzzvwrreturn.c|jDcic]}||j|}}gdt|jfd}t t |Scc}w)zQGenerate mesh shape and dimension names for torch.distributed.init_device_mesh().)r&r'r/r.c,j|dS)Nr)index)x mesh_orders rz-ParallelismConfig._get_mesh..s:++AaD1r)key)rF_sizessortedrtuplezip)r parallelism mesh_dims sorted_itemsr]s @rrOzParallelismConfig._get_meshsdOSNcNcd{[$++k"::d d>  OO 2 S,'((esAc|j.ttjj dd|_|j .ttjj dd|_|j .ttjj dd|_|j.ttjj dd|_|j dkDr|jt|_|jdkDr|jt|_ |jdkrtd|j|j dkrtd|j |j dkrtd |j |jdkrtd |j|j dkDs|jdkDr)|jdkDr|j dk(r td |j|j |j |jd |_ y) N$PARALLELISM_CONFIG_DP_REPLICATE_SIZE1 PARALLELISM_CONFIG_DP_SHARD_SIZEPARALLELISM_CONFIG_TP_SIZEPARALLELISM_CONFIG_CP_SIZEr@z.dp_replicate_size must be at least 1, but got z*dp_shard_size must be at least 1, but got z$tp_size must be at least 1, but got z$cp_size must be at least 1, but got aCTensor/Context parallelism (tp/cp_size > 1) cannot be used with pure data parallelism (dp_replicate_size > 1 and dp_shard_size == 1). Please set dp_shard_size > 1 and dp_replicate_size == 1 to compose FSDP + TP/CP for 2D parallel, or set dp_replicate_size == 1 and dp_shard_size > 1 to compose HSDP + TP/CP for 3D parallel.)r&r'r.r/)r intosenvirongetrrrrrrrrVr`rs r __post_init__zParallelismConfig.__post_init__s  ! ! )%(8^`c)d%eD "    %!$RZZ^^4VX[%\!]D  << rzz~~.JCPQDL << rzz~~.JCPQDL <  ! !A %MdNdNdMefg g    !I$J\J\I]^_ _ <RS S <RS S LL1  q 0d6L6Lq6PUYUgUgklUlo  !22**,,,,   rrdsizec||jjvs"Jd|jj||j|<t||d|y)NzParallelism must be one of _size)r`keyssetattr)rrdrrs r _set_sizezParallelismConfig._set_sizes\dkk..00d4OPTP[P[P`P`PbOc2dd0#' K  U+T2rct}|js|jdk(ry|jdk(r|jd|j|j|jk7r&t d|jd|jd|jdkDr1|j s%|jst d|jd|jjD]4\}}|dk(s t||dd|jd |d |d 6|r:|jr-tjd d j|zt yyy)Nr@r&zParallelismConfig total_size (z ) does not match num_processes (zB). Please adjust dp_replicate_size/ dp_shard_size/tp_size/cp_size.zpParallelismConfig is only compatible DistributedType.FSDP (version 2) or DistributedType.Multi{Device}, but got ._handlerzParallelismConfig.z_handler is set, but z0_size is set to 1. This handler will be ignored.z.ParallelismConfig has the following warnings:  )set multi_devicerrw num_processesrVis_fsdp2distributed_typer`rgetattraddis_main_processwarningswarnjoin UserWarning)r accelerator _warningsrdrrs r_validate_acceleratorz'ParallelismConfig._validate_accelerator"s}E ''DOOq,@  ??a  NN>;+D+D E ??k77 700AB""-";";!<=12  ??Q  (<(< @X@XEFQFbFbEccde "&!2!2!4  KqyWTk](+CTJV ( 5J;-XHI  44 MMADIIiDXX 59r)N)rr ))__name__ __module__ __qualname____doc__r rm__annotations__rrrrrrrrrrr$propertyr,r2r4r6r9rr<r>r(r)r0r1rFstrrTrrWrbrOrqrwrrrr r s:"s!M3GSGS:>Jd556=:>Jd667>K      YY++;;**&&    99S<  8C=   )5sCx%S/!AB )' R3S33 rr )rnr dataclassesrtypingrrraccelerate.utils.dataclassesrraccelerate.utils.versionsr accelerater r rrrrs@ !11^6& ee er