uki!ddlmZddlmZddlZddlZddlmZddlZ ddl m Z ddl m Z ddl mZee j cZZej$Zej&Zej(ZeZej,Zej.ZeeefZej2ZddZd Zee_ee_eeee eeefd ffZ! dd Z" dd Z# d dd Z$y)) annotations)SequenceN)Union)config)util)pmap_libc t|t|jk(sJ||jfg}g}t|jD]O\}}||}t|tr|j t dg8t|trQ||jk(sJ|d|j|j t||j |t|trtj|j}t||\}} | r J|||f|j t|D cgc]} t | |z| dz|zc} |j|j;t!j"|Rt%j&tj|gt$j(} tt+j,|D] \} } | | | < | j/|} t|} |j0Dcgc]}t|t2s|j4!}}t7|t7t| k(rt|| k(sJt9d|j0D}t+j:dt=|}}|j0Dcgc]4}t|t>r tA|nt|tA|z6}}t%jB| || jDzjG|Scc} wcc}wcc}w)aReturns NumPy-style indices corresponding to a sharding spec. Args: shape: The shape of the logical array being sharded. Returns: An ndarray with the same shape as the logical mesh (as derived form `mesh_mapping`). Each entry is a NumPy-style index selecting the subset of the data array to be placed on a corresponding device. The indices can be ints, slice objects with step=1, or tuples of those. Nz != )dtypec3VK|]!}t|ts|j#ywN) isinstance Replicatedreplicas).0as R/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/jax/_src/sharding_specs.py z)_sharding_spec_indices..ms[qAzAZ [s))r)$lensharding enumerater NoShardingappendslice UnstackedsizerangeChunkedmathprodchunksdivmodextendrassert_unreachablenpemptyobject_ itertoolsproductreshape mesh_mapping ShardedAxisaxissettuplecountiterrnext broadcast_toshape transpose)selfr4 axis_indicesshard_indices_shapedimr axis_size total_chunks shard_sizeraggedi shard_indicesidxsnum_sharded_dimrsharded_dim_perm replica_sizes replica_dim sharded_dimperms r_sharding_spec_indicesrG:s Us4==) )AE4==+AA )(*, /(mc8c I(J'5;-( Hi ( (-- 'JI;d8==/)JJ '% *+  + Hg &YYx/l!)\:j&7)\377Z$),$79 !Z!a%:1EF9:  1 h'%(0((DII&9:;2::N-9,,l;<gaM!''(;<-+,/&*&7&7V:a;UaffVV  3u_'=#> >  ? 24 3[D,=,=[[-&__Q/6F1G{+$$ & *!Z8${  m tK0 0 1 &$ & //-9L9L)L M Yt_59"W  &s L9 ,L>L> 9Mc<d|jd|jdS)Nz ShardingSpec(z, )rr+)r6s r_sharding_spec_reprrKus" r$*;*;)>.cJt|j|jS)aReturns numpy-style indices corresponding to a sharding spec. Each index describes a shard of the array. The order of the indices is the same as the device_buffers of a Array sharded using PmapSharding (i.e. the data is laid out row-major). Args: shape: The shape of the logical array being sharded. spec: Describes how the array is sharded and how the shards are assigned to the logical mesh. Returns: A tuple of length equal to the size of the mesh (inferred as the product of sharded dimension sizes and all replication factors). Each element is an int, a slice object with step=1, or a tuple thereof, to be treated as an index into the full logical array. )r/indicesflat)r4specs rspec_to_indicesrQs& t||E"'' ((rLc  t||\}}|rJttgt|zd}|dk(rdn t |f}|t d|j d|D d fd }tjjr,tj|j |t|g} n*tj|j |t|} t| tj t# g|t%||j&St|j t |f|z|j&zS)azSharding spec for arguments or results of a pmap. Args: nrep: number of local XLA replicas (product of local axis sizes) axis_size: local axis size for outer pmap sharded_aval: the aval of the value inside the outer pmap, an instance of a ShapedArray. map_axis: the axis along which the value is mapped in the outer pmap Returns: A ShardingSpec. rJr Nc3>K|]}t|t ywr )rr)rss rrz%pmap_sharding_spec..s[AjJ77[scvt|tr'|jk\rt|jdzS|SNr )rr,r-)rsharded_in_axiss rshift_sharded_axisz.pmap_sharding_spec..shift_sharded_axiss0 A{ #/(A166A:&& hrL)rMeshDimAssignment)r" ShardingSpec_UNSHARDED_INSTANCErrsumrrpmap_no_rank_reductionvaluer tuple_updater tuple_insertrr(chainr,mapr+) nrepr: sharded_shapemap_axisreplication_factorr=pspecmaybe_replicaterYrrXs @rpmap_sharding_specrjs= &dI6f  !4 5M8J J$& (%,1B CU8V7X/ [PYQYAZ[[O $$**"" ..(GYK$8:h"" ..(Ii$8:h ??  ' (/  %"4"4 57 88 ~~y)+o=@R@RR TTrLc|Ptjjrtj||d}ntj ||}| ||}n|J|}t ||||SrW)rr^r_rr` tuple_deleterj)r4rEsharded_dim_sizeres rcreate_pmap_sharding_specrnsw $$**''{A>m''{;m{+  '' 'M ,.> ' ))rL)r4tuple[int, ...]returnz np.ndarray)r4 Sequence[int]rPr[rpztuple[Index, ...])rerqrf int | Nonerpr[)rN)r4rorEintrmrr)% __future__rcollections.abcrr(rtypingrnumpyr%jax._srcrr jax._src.librrcsafe_map unsafe_maprrrr\r,rrZr[rGrKrN__repr__rsrr/IndexrQrjrnrSrLrr~s<#$ !t}} C         l""  +z12$$ 9v?. +  c5%c5j 13 6778)&)+<),'T!+'T0<'TTJK=A)0:)rL