bi̱rddlmZddlZddlZddlZddlZddlZddlZ ddl Z ddlm Z ddl mZddlmZmZddlmZddlmZmZmZmZmZmZmZmZmZddlmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2dd l3m4Z4m5Z5dd l6m7Z7m8Z8dd l9m:Z:m;Z;dd lZ>m?Z?m@Z@mAZAmBZBmCZCmDZDmEZEdd lFmGZGmHZHmIZImJZJmKZKmLZLGddeZMGddeMZNdZOGddeMZPGddeeNZQGddeQZRGddeQZS d9dZTGddeQZUdZVdZWd ZXd!ZYGd"d#eZZGd$d%eZ[Gd&d'e[Z\Gd(d)e[Z]Gd*d+e\Z^Gd,d-eZ_Gd.d/eZ`Gd0d1eZaGd2d3eZbGd4d5eZce7d6Zd d: d;d7Ze d: d;d8Zfy)<) annotationsN)CategoricalDtype)compute)TaskTaskRef)_concat) Assign BlockwiseExprFilterPartitionsFiltered Projection ToSeriesIndexdetermine_column_projectionis_filter_pushdown_available)AllAnyCountDropDuplicatesLenMaxMean MemoryUsageMinModeNBytesNFirstNLargestNLast NSmallestProdSizeSumUnique ValueCounts) RepartitionRepartitionToFewer)LRU_convert_to_list)is_categorical_dtype make_meta) barriercollectensure_cleanup_on_exceptionmaybe_buffered_partdpartitioning_indexset_partitions_pre shuffle_groupshuffle_group_2shuffle_group_get)Mdigitget_default_shuffle_methodinsert is_index_likeis_series_likeceZdZgdZddddddZdZdZdZdZe jdZ e jd Z d Z d Ze jd Zd Zy) ShuffleBase)framer0npartitions_out ignore_indexmethodoptions index_shuffleoriginal_partitioning_indexFN)r?r@rArBrCTc(d|jdddS)NzShuffle(i)_nameselfs \/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/dask/dataframe/dask_expr/_shuffle.py__str__zShuffleBase.__str__^s$**RS/*!,,c2|j|jgSN)r=r0rHs rJ_node_label_argszShuffleBase._node_label_argsas D3344rLcP|j}t|ttfr|g}|SrN)r0 isinstancestrint)rIr0s rJ_partitioning_indexzShuffleBase._partitioning_indexds,!44 (3* 5"4!5 !!rLc|jxs |j}t|tr t |hSt SrN)rCrTrQlisttupleset)rIidxs rJ-unique_partition_mapping_columns_from_shufflez9ShuffleBase.unique_partition_mapping_columns_from_shuffleks4..J$2J2J)#t4c |?#%?rLct|tr#|j||r|j|St|trt t |||}|j}|j}|jDcgc] }||vs||vr|}}t|t|jkr4t|||g|jdd|jdSt|ttt t"t$t&t(t*t,t.t0t2t4t6t8t:t<t>fr)t||jg|jddSycc}w)Ncolumns) rQr _filter_passthrough_available_filter_simplificationrr)rrTr=r]rXtypeoperandsoperandr$rr#r!rrrrrr"rrrrrr r%r)rIparent dependents projectionr0targetcolnew_projections rJ _simplify_upzShuffleBase._simplify_uppsg ff %$*L*L J+ ..v6 6 fj )*+D&*EJ"&!9!9 ZZF">>-- 1BN >"S%88!tDz&"8M4==;LMNN9-  %  . 4< AV__QR-@A A/ sE8ct|d)NzC is abstract! Please call `simplify`before generating a task graph.)NotImplementedErrorrHs rJ_layerzShuffleBase._layers!f. /  rLc|jj}|jr!|jdk(r|j d}|S)NtasksTdrop)r=_metar?r@ reset_indexrImetas rJrqzShuffleBase._metas=zz   !7###.D rLc&d|jdzzSNrNr\)r>rHs rJ _divisionszShuffleBase._divisionss$..233rL)__name__ __module__ __qualname__ _parameters _defaults_is_length_preserving_filter_passthroughrKrO functoolscached_propertyrTrZrirlrqrwrLrJr<r<Is K'+ I!-5"" @@/Bb  4rLr<ceZdZdZdZy)ShuffleauAbstract shuffle class Parameters ---------- frame: Expr The DataFrame-like expression to shuffle. partitioning_index: str, list Column and/or index names to hash and partition by. npartitions: int Number of output partitions. ignore_index: bool Whether to ignore the index during this shuffle operation. method: str or Callable Label or callback function to convert a shuffle operation to its necessary components. options: dict Algorithm-specific options. index_shuffle : bool Whether to perform the shuffle on the index. c|j}|j}|jxs t}||jkr|dk7r t ||}|j |j|j|j|jg}|dk(r t|g|S|dk(r t|g|S|dk(r t|g|S|dk(r t|g|St|d)Np2p)new_partitionsdisksimplernz not supported)r=r>r@r7 npartitionsr&r0r?rArC P2PShuffle DiskShuffle SimpleShuffle TaskShuffle ValueError)rIr=r>r@opss rJ_lowerzShuffle._lowers  ..< : < U.. .6U?oFE  # #     LL  , ,   U?e*c* * v u+s+ + x  -- - w u+s+ +x~67 7rLN)rxryrz__doc__rrrLrJrrs *8rLrctjjj|xsEt |t xr3tjjj|j SrN)pdapitypesis_numeric_dtyperQr categories)dtypes rJ_is_numeric_cast_typersN  %%e, < e- . < FFLL ) )%*:*: ;rLceZdZdZy)RearrangeByColumnc .|j}|j}|j}|j}|j}|j }t |tr|g}|rn-t |ttfstt|dt |tsn|sl|Dcgc]}||jvs|}}t|dk(r@t|d|j}|j }d||j|d<t#||d||j$|} t'| d|||j(||j*} |j,dk(r| | jdS| | jD cgc] } | dvs|  c} Scc}wcc} w)Nz, not a supported type for partitioning_indexr\ _partitions_0r _partitions)rC)rr)r=r0r>r?rArBrQrRrVr rr`r]lenr indexcopyAssignPartitioningIndexrqrr@rTndim) rIr=r0r>r?rArBrgcs index_addedshuffledcs rJrzRearrangeByColumn._lowers !44..(( ,,**  (# ."4!5   .t =*+,,XY ,d3M!3P#s%--7O#PBP2w!|uou{{C%<%7%<%<%>"FU"#;#5#;#;BqE#BC.     KK        KK (,(@(@  ::?K//23 3 (( V1A5U,UQ V  ?Q@ Ws"F 6F : FFN)rxryrzrrrLrJrrs5 rLrcdeZdZgdZddddddZej dZedZ dZ d Z y) r)r=r0r>r?rArCrNrF)rrCr0r?rAc.|jjSrNr=rqrHs rJrqzSimpleShuffle._meta;zzrLc| t|g|St|g|jDcic] \}}||vs ||c}}Scc}}w)z$Filter the output of `shuffle_group`)r2items)df_filterargskvs rJ_shuffle_groupzSimpleShuffle._shuffle_group?sN ? +d+ +!.r!9D!9!?!?!ARAQ'\1RRRs AAc Zd|jz}d|jz}|j}i}|jr |jnd}t |jD]\}}t |j jDcgc]}|||f } }t| |jf||j|f<| D]l\} } } tj|| f| f||| | f<|| f|vs)|j|j j| f||jd|||j|f ||| f<n|Scc}w)z/Construct graph for a simple shuffle operation.group-split-Nr)rGr> _filteredr enumerateranger=rrr?operatorgetitemrr0) rIshuffle_group_name split_namerdskr global_partpart_outpart_in _concat_list_ _part_out_part_ins rJrlzSimpleShuffle._layerFsd% 2 * ** &*nn$""$%.t/?/?%@  !K %TZZ%;%;<Xw/L !!.C[) * +7 &9h$$'2:ZH56 '1<++))84//##))# ;C+X67  : 9s D(cyrNrrHs rJrzSimpleShuffle._lowermsrL) rxryrzr{r|rrrq staticmethodrrlrrrLrJrr(s\K'++ I  SS %NrLrcFeZdZdZej dZfdZxZS)rz(Staged task-based shuffle implementationcn|jj}|jr|jd}|S)NTro)r=rqr?rrrss rJrqzTaskShuffle._metats2zz   ###.D rLc |jxsijddxsd}|jj}t |j |ks||krt |S|j}ttjtj|tj|z }|dkDr%ttj|d|z zn|t|zDcgc] tfdt|D"}}t|Dcic]\}}|| }}}tt |} i} |jj } t#|jj$} t|D]?} | }| |dz k(r.||k(r)|j } |j } |j&r| nd}nd| d|j } d}d| z}d| z}t| D]\}}||}g}tD](t)|| }|| }|j+|||f*t,||j.f| | |f<|D]y\}}}t0j2||f|f| |||f<||f| vs)||}| d k(r||kr||f}n||d f}| | |<n||f}|j4|||j6| ||j.|f | ||f<{B||k7rd | z}t|Dcic]&}||ft8| |f|j6|j.|f(}}t|j D]"\}t:|||zf|f||j f<$| j=|| Scc}wcc}}wcc}w) N max_branch r\c38K|]}t|ywrN)r6).0jinsplitss rJ z%TaskShuffle._layer..s>1%1g&>szstage--rrremptyzrepartition-group-)rAgetr=rrrsuperrlr>rSmathceillogrrWrrGr+rqrr8appendrr?rrrr0r3r4update) rIrnpartitions_inputrstagesrinputsinp inp_part_map parts_outrname meta_inputstage name_inputrrrrpartoutr_inp_idxr_part input_keyrepartition_group_namedsk2pr __class__s ` @rJrlzTaskShuffle._layer{sll(b--lDAGR  JJ22 t J .2Cz2Q 7># #** TYYtxx(9:TXXj=QQRS A:$))$5!f*$EFGG'G 7F?+  >f > >  .7v->?61cQ? ?#f+& zztzz//0 6]? EJ!$8I)Izz ,, '+~~)4wa |4!)D !DJ%.y%91 ! TTl! wBA!#ua0Du:D ''T4(@A B %%,T;'( &2 MAtT ((+T25CT401 +D1< ,T 2 A:$'88-7,? .@w,O 1;I)3U(;I!//%# 33!#- --' ;/67- #1 ? B + +%9D%@ "01 (+#1I++%% . D "$"2"23 1%+Q1B-BC)djj!_%  JJt  E @V s9%M(. M-.+M3) rxryrzrrrrqrl __classcell__)rs@rJrrqs*2 uurLrc&eZdZdZedZdZy)rz!Disk-based shuffle implementationct|5|j|}|jDcic]}||vs||j|}}|j |ddddycc}w#1swYyxYw)NT)fsync)r.groupbygroups get_groupr)rrgrrgrds rJrzDiskShuffle._shuffle_groupsn ( + $ 3A,-HHEqW AKKN"EAE HHQdH # $ $E $ $s! A- A(A( A-(A--A6c hddlm}|j}|j}t j j }d|zf}||j}|t|fi}d|z}t|jD cic]#\} } || f|j| ||j|f%} } } d|zf} | tt| fi} t|jDcic]'\}}|j|ft |||j| f)}}}t#j$|| | |Scc} } wcc}}w)Nr)partd_encode_dispatchzzpartd-) encode_clszshuffle-partition-zbarrier-)dask.dataframe.dispatchrr0r=uuiduuid1hexrqr/r __dask_keys__rrr,rVrGr-toolzmerge)rIrcolumnralways_new_tokenrrdsk1rrkeyr barrier_tokendsk3rrdsk4s rJrlzDiskShuffle._layersFA(( ZZ::<++ ) ) +*2884 (J?AB$&66$B$4$4$67 31I++S&$:J:JAN N  $&668 d45 "$"2"23 1ZZOgq!RXX}E E  {{4tT22  s (D(!,D.N)rxryrzrrrrlrrLrJrrs+$$ 3rLrc"ddlm}||||S)Nr)shuffle_transfer)distributed.shuffle._shuffler )inputidinput_partitionr s rJ_shuffle_transferrs >    rLc:eZdZdZej dZdZy)rz'P2P worker-based shuffle implementationcb|jjj|jS)N)r])r=rqrpr0rHs rJrqzP2PShuffle._meta.s&zz$$T-D-D$EErLc ddlm}m}m}m}ddlm}m}i}|jjdd}||} || } d|z} |jr |jntt|j} |jrt|jn |j} t}t|j j"D]d}t%| |ft&t)|j j|f||}|||j*<|j-|j/f|| ||g|d|| |j|j0|j j2| dd i}|||j*<|j} t5| D]4\}}t%| |f||||j.}|||j*<6|S) Nr)P2PBarrierTask ShuffleId barrier_key p2p_barrier)DataFrameShuffleSpecshuffle_unpackrzshuffle-transfer-specT)rrrrtrr drop_column)distributed.shuffle._corerrrrr rrrGsplitrrrVrr>rWr=rrrrrrrefr0rqr)rIrrrrrrrtoken shuffle_id _barrier_keyrr parts_out_arg transfer_keysrtr,rs rJrlzP2PShuffle._layer2s  V   %b)u% ":. "U*!%D  Dt?S?S9T4U  (,~~E$"" #4;O;O  tzz--. *Aq !))1-. ACJ   ) *!      & 00..ZZ%%'   #GKKzz$Y/ KAxq  ACJ  rLN)rxryrzrrrrqrlrrLrJrr+s$1FF?rLrct|tr|n|g}|Dcgc]}t||s|}}||}t||r|j |j }|Scc}w)z Make a column selection that may include the index Parameters ---------- columns_or_index Column or index name, or a list of these )_index)rQrV_is_column_label_reference_contains_index_nameassignr)rcolumns_or_indexn column_names selected_dfs rJ_select_columns_or_indexr/ysu''7>EUDV 0U!3MbRS3TAULU\"KB 01!(((9 Vs A!A!ct|t xrbtj|xs;tj j j|xst|txr||jvS)z Test whether a key is a column label reference To be considered a column label reference, `key` must match the name of at least one column. ) rQr npisscalarrrr is_scalarrWr])rrs rJr(r(s] sD !!  [[  V!7!7!< V 3PU@V  2:: rLcdt|trtfd|DSt|S)zL Test whether the input contains a reference to the index of the df c36K|]}t|ywrN)_is_index_level_reference)rr,rs rJrz'_contains_index_name..sN,R3N)rQrVanyr6)rr+s` rJr)r)s0"D)N=MNNN(-=>>rLct|tr |jjjn|jj}|duxr}t|t xrjt j |xs;tjjj|xst|txr||k(xr|t|ddvS)z Test whether a key is an index level reference To be considered an index level reference, `key` must match the index name and must NOT match the name of any column. Nr]r) rQr rrqrr1r2rrrr3rWgetattr)rr index_names rJr6r6s)32t(<$$"((--J$ 23% % 2 [[  V!7!7!< V 3PU@V 2 :  2 wr9b1 1 rLc<eZdZdZgdZdddddZdZed dZy) raUAssign a partitioning index This class is used to construct a hash-based partitioning index for shuffling. Parameters ---------- frame: Expr Frame-like expression being partitioned. partitioning_index: Expr or list Index-like expression or list of columns to construct the partitioning-index from. index_name: str New column name to assign. npartitions_out: int Number of partitions after repartitioning is finished. index_shuffle : bool, default False Whether we are using solely the index for the shuffle )r=r0r;r>rtrBNFr) cast_dtyperBr0r;Tc`fd}|||}|||}i}|jjD]$\} } t| stj|| <&|r|j |d}t ||}|jdk(r|j}|jdi||iS)z)Construct a hash-based partitioning indexct|dr!|jdk(r|j}|Sr*|dgk(rddini}|jjdi|}|St ||}|S)Nrr\r'rr)hasattrrto_framerr/)rYobjrrBs rJ _get_indexz5AssignPartitioningIndex.operation.._get_indexswsF#88q=,,.CJ -0XJ->)B(cii((040J/sC8JrLignore)errorsr\r) dtypesrrr1float64astyper0rrAr*) rrrrrtrBrC meta_indexrFrgrs ` rJ operationz!AssignPartitioningIndex.operations  t, 5"%$++113 )JC$U+ jjs  ) LLL9E"5+6 77a<Bryy)D%=))rL)rrRrrSrBbool) rxryrzrr{r|#_preserves_partitioning_informationrrJrrLrJrrs=(K+# I +/'**rLrc6eZdZdZdZedZedZy)BaseSetIndexSortValuesTcRd|jvr|j |jS|jdk(ryt|jj rY|jj rC|jj|jjk(r|jjSt|j|j|j|j|j\}}}}|r/t||jk(r|j|dgz}|S)Nuser_divisionsr\NNupsampler)r{rP_npartitions_inputr9_divisions_columnrqknown_divisionsrr=other divisions_get_divisions ascendingrSrrrIrXminsmaxes presorteds rJrwz!BaseSetIndexSortValues._divisionss t// /D4G4G4S&& &  " "a ' $0066 7&&66&&22djj6L6LL::'' ',: JJ  " "  # # NN]] - ) 4  Td&=&== uRyk1IrLcT|jdxs|jjS)Nr)rbr=rrHs rJrTz)BaseSetIndexSortValues._npartitions_inputs ||M*Ddjj.D.DDrLc`|jdxst|jdz S)Nrr\)rbrrwrHs rJrz"BaseSetIndexSortValues.npartitions!s'||M*Hc$//2C.Dq.HHrLN)rxryrzr}rwpropertyrTrrrLrJrNrNs9 0EEIIrLrNc eZdZdZgdZdddddddddd ZdZed Ze jd Z ed Z ed Z d ZdZdZy)SetIndexaAbstract ``set_index`` class. Simplifies (later lowers) either to Blockwise ops if we are already sorted or to ``SetPartition`` which handles shuffling. Parameters ---------- frame: Expr Frame-like expression where the index is set. _other: Expr | Scalar Either a Series-like expression to use as Index or a scalar defining the column. drop: bool Whether we drop the old column. sorted: str No need for shuffling if we are already sorted. user_divisions: int Divisions as passed by the user. upsample: float Used to increase the number of samples for quantiles. ) r=_otherrprPpartition_sizerZrrSshuffle_methodrrATNA?F) rprPrerZrrSrfrArcr|jt|jts|jgzSgzSrN)r]rQrdr rHs rJ_projection_columnszSetIndex._projection_columnsVs6||!+DKK!>T[[M  DF  rLct|jtr|jj}n |j}|jjj ||j SNro)rQrdr rqr= set_indexrp)rIrWs rJrqzSetIndex._meta\sL dkk4 (KK%%EKKEzz))%dii)@@rLc|jSrN)rWrHs rJrUzSetIndex._divisions_columnds zzrLct|jtr |jS|j|jSrN)rQrdr r=rHs rJrWzSetIndex.otherhs- dkk4 (;; zz$++&&rLc ~|jddk(sN|jjdk(r|jt |jdk(ro|jd^|j}|jjdkDr t |d}t ||j|jd}t|S|j|j}t|jjr|jj|k(rd}nEt!|j|j|j"|j$|j&d}|re|j|jjk(rBt |j|j|j||j(}t|St+|j|j|j|j |j"nd|j$|j&|j|j,|j. S)Nrr\TrR)rbr=rrPrr'SetIndexBlockwiserdrpSortIndexBlockwiserwr9rUrqrWrXrYrTrZrSr SetPartitionrfrA)rIexpr index_setrXr^s rJrzSetIndex._lowerns LL '1 ,zz%%*$$,D4G4G0HA0M ]+3::Dzz%%))$2)$ TYYMI%i0 0    &)Id44::;JJ((I5 *JJJJ++NN!]]    T--1G1GG-JJ TYY 4;; *)44 JJ KK II'+':':'BD # # NN MM       LL  rLcddlm}m}m}t ||rt |j t tfrf|j |jjvrDt|j|j|j d}t||j St ||rt |j t tfrf|j |jjvrDt|j|j|j d}t||j St |trt |j ts |j gng}t!||||} t#| } |jjD cgc] } | | vs|  } } |jj| k(ryt%|t%||j| g|j&dd|j)dSt ||r$|j+||r|j-|Syycc} w) Nrr HeadTailTr,_columnsrZ)rdadditional_columnsr\r])dask.dataframe.dask_expr._exprr rzr{rQrdrSrRr=r]rr,rcrrr rr)r`rarbr^r_) rIrcrdr rzr{headtailaddition_columnsr]rs rJrizSetIndex._simplify_upsEE vt $4;;c 3 tzz111$**4;;RVWDD5 5 vt $4;;c 3 tzz111vxx$++QUVDD5 5 fj )%/ T%B  2fj=MG'w/G"&**"4"4EQW qEGEzz!!W,4<T 4::g.Cqr1BCy)  ff %$*L*L J+ ..v6 6+ %Fs  I$Ict|||r5ddlm|j}t fd|j D Sy)Nr)Indexc36K|]}t|ywrN)rQ)rxrs rJrz9SetIndex._filter_passthrough_available..sBA:a/Br7F)rrr predicater8walk)rIrcrdrrs @rJr^z&SetIndex._filter_passthrough_availables: 'fj A <  ABBBB BrL)rxryrzrr{r|r~rarjrrrqrUrWrrir^rrLrJrcrc&s* K I   AA'' . `&7PrLrcc eZdZgdZdddddddddd ZdZd Zedd Zed Z ed Z e jd Z e jdZdZdZy) SortValues) r=byrZ na_positionrre sort_functionsort_function_kwargsrSr?rfrArgTNlastrhF) rerZrrrrrSr?rfc0|jjdk(ryt|j|j|jd|j|j |j \}}}}|r|jjSdt|zS)Nr\rQrrRrN) r=rrYrrT_divisions_ascendingrSrXrr[s rJrwzSortValues._divisionss :: ! !Q &,: JJ JJtwwqz "  # #  % %]] - ) 4  ::'' 'Y''rLcl|j}t|ts|d}t|tsJ|SNr)rZrQrK)rIdivisions_ascendings rJrzSortValues._divisions_ascendings9"nn-t4"5a"8 -t444""rLcf|jd|jdStjS)Nr)rbr5 sort_valuesrHs rJrzSortValues.sort_functions* << ( 4<<0 0}}rLc|j|j|j|jd}|j d |j |j d|S)N)rrZrr?r)rrZrr?rbr)rI sort_kwargss rJrzSortValues.sort_function_kwargs s[''++ --  <<. / ;   t||,BC DrLc.|jjSrNrrHs rJrqzSortValues._metarrLc|jj|j}t|r|jd}|Sr)rqrFrr:iloc)rIrs rJ_meta_by_dtypezSortValues._meta_by_dtypes4 !!$''* % JJqME rLc|jjdk(r+t|j|j|jS|j|j d}t |j||j|j|j\}}}}|rN|j|jjk(r+t|j|j|jSt||jj|j|j}t|jd|}t|dt!|dz |j"|j$|j&}t)||jj*}t||j|jS)Nr\rrRrZrr>r?r@rA)r=rSortValuesBlockwiserrrrYrTrrS_SetPartitionsPreSetIndexrq _constructorrr rrr?rfrArr])rI _divisions_byrXrr^ partitionsassignedrs rJrzSortValues._lower s| :: ! !Q && D..0I0I  4771:. %3 JJ   # #  % %]] & " 1a ))TZZ-C-CC& D..0I0I /     , ,Y 7 C C E// $**mZ@   NQ.**&&LL  h (:(:;" d(($*C*C  rLcddlm}m}m}t ||r7t |j |j|j|jSt ||r7t|j |j|j|jSt ||r#|j||r|j|St |trt||||j}t|}|j j Dcgc] }||vs| }}|j j |k(ryt#|t#||j |g|j$dd|j'dSt |t(rX|j'dFt#|t#||j g|j$ddg|j$ddSyycc}w)Nrryr|r~r\r]r)rr rzr{rQrr=r,rrZrr^r_rrr)r]r`rarbr&)rIrcrdr rzr{r]rgs rJrizSortValues._simplify_upFsEE fd # fhhDNN  fd # fhhDNN  ff %$*L*L J+ ..v6 6 fj )1fjTWWG'w/G&*jj&8&8KsC7NsKGKzz!!W,4<T 4::g.Cqr1BCy)  v{ +/0<4:V TZZ>&//!"*=>AEqrAR = ,Ls G*G*)returnrK)rxryrzr{r|r~rwrarrrrrrqrrrirrLrJrrs K  $ I( ##      $ L#rLrceZdZdZgdZdZy)ruaShuffles the DataFrame according to its new divisions. Simplifies the Expression to blockwise pre-processing, shuffle and blockwise post-processing expressions. Parameters ---------- frame: Expr Frame-like expression where the index is set. _other: Expr | Scalar Either a Series-like expression to use as Index or a scalar defining the column. drop: bool Whether to drop the old column. new_divisions: int Divisions of the resulting expression. ) r=rdrprrZrSrPrfrAc  |jjj|j}t |j|}t |j d|}t|jtrt |d|j}t|dt|jdz d|j|j}t||jDcgc] }|dk7s | c}}t|jtrd\}}n,|j |jjj"}}|jj$|j&|j(d|j*f}t,j/|} t1||jjj"|||j jjj2| |j4} t7| Scc}w)Nrr'r\Tr)Tr'rg)rWrqrrwrr r=rQrdr rrrfrArr]rprrGrTrZrS divisions_lrur _SetIndexPostrrPrt) rIrXrrrrrpset_namelru_keycomputed_divisionsrws rJrzSetPartition._lowersJJ$$11$//2CD .tzz9E $**mZ@ dkk4 (h$++>H   12Q6&&LL   ("2"2IQa=6HqI  dkk4 (+ND(!YY (8(8(=(=(D JJ    # # NN  MM  +..w7!  JJ   ! !   JJ   $ $ * *     "),,1Js % H 0H N)rxryrzrr{rrrLrJruruls" K'-rLrucVeZdZgdZdddZeeZdZe jdZ y)r)r= new_divisionsrZrTr)rZrc`t|jjjdgSr)r+r=rqrrHs rJrqz_SetPartitionsPreSetIndex._metas$))66s;<} tAjBtAjDd|d z |d ztAjDd|d z ||jGjG}nV|j>}tQ|jNd|d z jS|jN|d z djGz}| jU} | jU} tWjjt"jXr8jj}| jK|} | jK|} | j3j+s| j3j+rd}n| j>}|r| jNd|d z n| jNd dj[d}|r| jNd dn| jNd|d z j[d}| jG| j]|jGk(xrF| jG| j]|jGk(xr||kj5}|| jG| jG|fS#t $r} t"j$j&j)jjsVt+fdj,Drddj.dfnd\} } t1d | d j.d | d | | d} ~ wwxYw#t tHf$rQtAjDd|d z |d zjKtL}|jN|jG}YwxYw)Nr)RepartitionQuantilesnew_collectionrRc3VK|] }j|jk("ywrNrF)rrr=rWs rJrz'_calculate_divisions..Zs"Lu{{eAhnn4Ls&)rz`.dropna(subset=['z'])`)seriesz`.loc[series[~series.isna()]]`z-Divisions calculation failed for non-numeric z 'z}'. This is probably due to the presence of nulls, which Dask does not entirely support in the index. We suggest you try with .r\)rxpfpFTror)/dask.dataframe.dask_exprrrr9rqrr*rcat as_ordered_exprrmap_partitionsr5minmax TypeErrorrrrrr8r]rrkisnaallsumrrrsizer1interplinspacetolistrrHrSrrVuniquebfillrQrrrr)r=rWrrZrerSrrrXr\r]erBsuggested_methodsizesempty_dataframe_detectedtotalr,indexesrr^maxes2mins2s`` rJrr<s+NU[[!e$EKK--.u%))446<<!( /{XV W 5 ! 0 0 7 5 ! 0 0 7"  42 E!wwy1557E $))EN$:;Q? +u'8'89 NN 9 ++aQ a8;;q!a%+##%fh  NN !a%(//1 2Y^^AEG5L5S5S5U U  ::>E  vv||,,U[[->->?LemmLL/ |4@AA "C! &?uBuzzlS++;*rs("  #)'   ,R@C   f4$f4R28k28j6 6 rF&FR-D'3-'3T     FF\2 ?"D*iD*N!IT!IHf%fRZ'ZzE-8E-P= =,FI,F^!! ) " BB !      ,! X?X? X?  X?  X?rL