bioUddlmZddlZddlmZddlmZddlmZddl Z ddl m Z m Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZd ZddZdZdZdZdZdZ ddZ!dZ" d ddZ#dZ$ ddZ%y)) annotationsN)reduce)product)mul)TaskTaskRef)Array)meta_from_array)tokenize)flatten)HighLevelGraph)Ma Dask's reshape only supports operations that merge or split existing dimensions evenly. For example: >>> x = da.ones((6, 5, 4), chunks=(3, 2, 2)) >>> x.reshape((3, 2, 5, 4)) # supported, splits 6 into 3 & 2 >>> x.reshape((30, 4)) # supported, merges 6 & 5 into 30 >>> x.reshape((4, 5, 6)) # unsupported, existing dimensions split unevenly To work around this you may call reshape in multiple passes, or (if your data is small enough) call ``compute`` first and handle reshaping in ``numpy`` directly. c ltdDsJtdz }t|dz }ttDcgc]}d}}tt|Dcgc]}d}}ig} } |dk\s|dk\r|||k(r |||<|||<|| |<|dz}|dz}6|} ||} | dk(r d||<|dz}n| dk(rd||<| j||dz}n| | krw|dz } || |<| dk\rBt t | |dz| kr)|| | <| dz} | dk\rt t | |dz| kr)|| | <t t | |dz| k7rt ttfdt|DrKt|dzD] }|||< |tjtt| |z||<nt| dz|dzD] }|f||< t t tt| dz|dz}t| ||| <t| |}t| |||}t|| |||<|dz}| dz }n| | kDr|r t d|dz }|dk\r=t t |||dz| kr$|dz}|dk\rt t |||dz| kr$t t |||dz| k7rt tt t ||dz|dzt|||<t|dz|dzD] }||f||< t!fd||D||<t||}t||||}t|||||<|dz }|dz}|dk\r|dk\rt!|t!|| | fScc}wcc}w)Nc3<K|]}t|tywN) isinstancetuple.0cs M/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/dask/array/reshape.py z"reshape_rechunk.."s6z!U#6sr)rc3FK|]}t||k(ywr)len)riinchunksinshapes rrz"reshape_rechunk..Is#Ea3x{#wqz1Es!zWreshape_blockwise not implemented for expanding dimensions without passing chunk hints.c3(K|] }|z ywr)rrcss rrz"reshape_rechunk..ss+QAG+Qs)allrrangeappendrrNotImplementedError_not_implemented_messagemathprodmap expand_tuple_cal_max_chunk_size_smooth_chunks_calc_lower_dimension_chunkscontract_tupler)routshaperdisallow_dimension_expansioniioirresult_inchunksresult_outchunks mapper_inone_dimensionsdindoutileftchunk_reduction max_in_chunkoleftr!s` ` @rreshape_rechunkr=!sv 6X6 66 6 W B X B%*3w<%89t9O9&+CM&:;;; "B~I 'R1W 2;(2, &"*2,OB #+B< R IbM !GB !GB bk| !8"&OB  !GB QY#' R  ! !" % !GB 4ZFEIbM vc7526+BCdJ#% %   vc7526+BCdJ "Ie c752623t;)*BCCE59EErAv5A)1!OA&5'/|diiXeB/07( $uqy"q&17A*1!*OA&7#)c#x BQRF7S.T"U)5huo)W&28UBG "02|_#(D#UB( $ !GBB 4Z+)mFE1*Xeb1f-E!F!L 1*Xeb1f-E!F!Lc8EBF34;)*BCCXeai"q&9:B"0"r"BOB 519b1f- 5'/{n # 5',+Q_R=P+Q&Q U #.xR@L-eRGWX "> %#OB B !GBg 'R1Wj  !5)9#:I~ UUs:;s N,+ N1c Fttdt|||dzS)Nc"tt|Srrrxs rz._calc_lower_dimension_chunks..s fS!nr)rr)r)chunksstartstops rr-r-s/  $ VED1H- .  rDc|}t|||}||k(r|Std||Dr|dz }td||Dr||dzkrttj||z }||}t |dk(r|d}t ||}tj||z } | g|z} t | |z|z D]} | | xxdzcc<t| ||<td| Dr||krt||||S|S|t||z} g} |D]}|| z|kr| j|tj|| z|z }tj||z } | gtj|z}t | |z|z D]} || xxdzcc<| j|t| ||<|S)Nc3&K|] }|dk( ywrNr rrBs rrz!_smooth_chunks..s 5a1f 5rrc3&K|] }|dk( ywrJr rKs rrz!_smooth_chunks..s/a16/rL) r+r"r'ceilrminr#rr,maxr$extend)r9r1r;r3 ileft_origmax_result_in_chunkfactorresult_in_chunkelem ceil_elem new_inchunkrother_max_chunk result_inelem_in new_in_chunks rr,r,s4J-oubI** 5oe4 5 5    5oe4 5 5 rAv~.=>)%0  1 $ #1%D&F $-0I$+.K9v-45 $A!# $%*;%7OE "/;//EBJ%j"lOTT, !2S9O5PPOI* /_, <$$W-7_#<|#KL IIg&67 ){TYYv->> y61G;<)A Oq(O)  . /&+9%5OE " rDc tttt||dzDcgc]}t ||c}Scc}w)Nr)intrrr#rP)rErFrGaxiss rr+r+sB  +0q+A B4S  B   CsAc&|dk(r|Sg}|D]`}|}t||z d}|d|zk\r1|jt||t|z}|d|zk\r1|sP|j|bt|t|k(sJt |S)z >>> expand_tuple((2, 4), 2) (1, 1, 2, 2) >>> expand_tuple((2, 4), 3) (1, 1, 1, 1, 2) >>> expand_tuple((3, 4), 2) (1, 2, 2, 2) >>> expand_tuple((7, 4), 3) (2, 2, 3, 1, 1, 2) r)rPr$r^sumr)rErToutrrBparts rr*r*s{ C  1v:q!1t8m JJs4y ! TNA1t8m JJqM v;#c( "" " :rDct||zdk(sJg}d}|D]*}||z }||z}||z}||z}|s|j|,t|S)zReturn simple chunks tuple such that factor divides all elements Examples -------- >>> contract_tuple((2, 2, 8, 4), 4) (4, 8, 4) r)rbr$r)rErTrcresidualchunkdivgoods rr.r.st v; 1 $$ $ CH vo6>|  JJt   :rDc ddlm}tt||}|Dcgc] }|dk7s | }}t |t |kr|t |t |z dkDr t dt |dk(r|j dk(r|S||jtt|dz tfd|D}t|||j|k(r|St|t |}dt||z}|jdk(rtt!|j#} |fdt |zz} | t%| t&j(t+| |i} td |D} t-j.|| |g } t1| || | St |j}t |}|s6||kDr1|j3t4j7t9||z d}t;|j||j<\}}}}|j3|}t?tA|jBgg|Dcgc]}t9t |c}}t?tA|gg|Dcgc]}t9t |c}}t?tA|}tE|||Dcic],\}}}|t%|t&j(t+||.} }}}t-j.|| |g } t1| ||| Scc}wcc}wcc}wcc}}}w) abReshape array to new shape Parameters ---------- shape : int or tuple of ints The new shape should be compatible with the original shape. If an integer, then the result will be a 1-D array of that length. One shape dimension can be -1. In this case, the value is inferred from the length of the array and remaining dimensions. merge_chunks : bool, default True Whether to merge chunks using the logic in :meth:`dask.array.rechunk` when communication is necessary given the input array chunking and the output shape. With ``merge_chunks==False``, the input array will be rechunked to a chunksize of 1, which can create very many tasks. limit: int (optional) The maximum block size to target in bytes. If no limit is provided, it defaults to using the ``array.chunk-size`` Dask config value. Notes ----- This is a parallelized version of the ``np.reshape`` function with the following limitations: 1. It assumes that the array is stored in `row-major order`_ 2. It only allows for reshapings that collapse or merge dimensions like ``(1, 2, 3, 4) -> (1, 6, 4)`` or ``(64,) -> (4, 4, 4)`` .. _`row-major order`: https://en.wikipedia.org/wiki/Row-_and_column-major_order When communication is necessary this algorithm depends on the logic within rechunk. It endeavors to keep chunk sizes roughly the same when possible. See :ref:`array-chunks.reshaping` for a discussion the tradeoffs of ``merge_chunks``. See Also -------- dask.array.rechunk numpy.reshape r)sanitize_indexrz&can only specify one unknown dimensionc30K|] }|dk(rn|yw)rlNr )rs missing_sizes rrzreshape..QsDa2gl14Dszreshape-)rc3"K|]}|f ywrr )rds rrzreshape..`s+t+s  dependenciesmeta)#dask.array.slicingrkrr)r ValueErrorndimsizerr_sanity_checksshaper r npartitionsnextr __dask_keys__rrreshaperr from_collectionsr rechunkdictfromkeysr#r=rElistrnamezip)rBr{ merge_chunkslimitrkrn known_sizesrurkeynew_keydskrEgraphr7r8r outchunks_x2rin_keysout_keysshapesabros @rrrsT2 #ne, -E#/qBw1/K/ ;#e*$ u:K( (1 ,EF F u:?qvv{H%affvc;/J&JK DeDD1eww% 1c%j )D E* *D}}71??,-.'D3u:--WaiiuEF+U++//cLUD&t44 agg,C u:D C$J IIdmmE#*$5q9 : / IHiA 8 B7BGG9Ix'H!c!f 'HIJGGTFHY%GeCFm%GHIH '9% &Fx&9   Aq% 4199gaj% 00 C   + +D#RD IE id 33_0L(I%G s L.L.L3 L81L=ctjt|jr!t dt |jzt t|d|jk7r t dy)Nz_Array chunk size or shape is unknown. shape: %s Possible solution with x.compute_chunk_sizes()rz)total size of new array must be unchanged) npisnanrbr{rwstrrrry)rBr{s rrzrzzsa xxAGG  =?B177| L  c5!&DEE'rDc|dvrtt|jf}t|ts|f}t ||t ||jk(r<||jk(r-t|j|j|j|Sdt||z}ttdt|jD}t ||jkDr| t!dttd|D}ttd|D}ttd |jD}tt#||D cgc]8\}\} } t%j&| t%j&| k7r|| | f:} } }} | r:t)d | dd Dcgc]}|d  c}d | dd Dcgc]}|dd c}dt#|||D cic]S\} } }|ft | zt+|ft | zt,t/|jft | z|U}} } }t1j2|||g}t||||j4S| t)dt7|j||jd\}}}}ttd|jDDcgc]}t9|||}}t9t t;t |j||}ttd|D}t#|||D cic]S\} } }|ft | zt+|ft | zt,t/|jft | z|U}} } }g}d}tt=|D]_\}}g}t?|D]+}|jA|||zt ||z dz -|jAt |||z}at1j2|||g}t||t t=||j4Scc} } }wcc}wcc}wcc}} } wcc}wcc}} } w)a; Blockwise-reshape into a new shape. The regular reshape operation in Dask preserves C-ordering in the array which requires a rechunking for most reshaping operations, making the computation relatively expensive. Blockwise-reshape reshapes every block into the new shape and concatenates the results. This is a trivial blockwise computation but will return the result in a different order than NumPy. This is a good solution for subsequent operations that don't rely on the order. Parameters ---------- x: Array The input array to reshape. shape : int or tuple of ints The new shape should be compatible with the original shape. If an integer, then the result will be a 1-D array of that length. One shape dimension can be -1. In this case, the value is inferred from the length of the array and remaining dimensions. chunks: tuple of ints, default None The chunk sizes for every chunk in the output array. Dask will expand the chunks per dimension into the cross product of chunks for every chunk in the array. An error is raised if chunks is given and the number of dimensions decreases. .. note:: This information is required if the number of dimensions is increased. Dask cannot infer the output chunks in this case. The keyword is ignored if the number of dimensions is reduced. Notes ----- This is a parallelized version of the ``np.reshape`` function with the following limitations: 1. It does not return elements in the same order as NumPy would 2. It only allows for reshapings that collapse like ``(1, 2, 3, 4) -> (1, 6, 4)`` Examples -------- >>> import dask.array as da >>> import numpy as np >>> x = da.from_array(np.arange(0, 27).reshape(3, 3, 3), chunks=(3, 2, (2, 1))) >>> result = reshape_blockwise(x, (3, 9)) >>> result.chunks ((3,), (4, 2, 2, 1)) The resulting chunks are calculated automatically to match the new shape. >>> result.compute() array([[ 0, 1, 3, 4, 2, 5, 6, 7, 8], [ 9, 10, 12, 13, 11, 14, 15, 16, 17], [18, 19, 21, 22, 20, 23, 24, 25, 26]]) >>> result = reshape_blockwise(result, (3, 3, 3), chunks=x.chunks) >>> result.chunks ((3,), (2, 1), (2, 1)) The resulting chunks are taken from the input. Chaining the reshape operation together like this reverts the previous reshaping operation that reduces the number of dimensions. >>> result.compute() array([[[ 0, 1, 2], [ 3, 4, 5], [ 6, 7, 8]], [[ 9, 10, 11], [12, 13, 14], [15, 16, 17]], [[18, 19, 20], [21, 22, 23], [24, 25, 26]]]) )rl)rlrtzreshape-blockwise-c3JK|]\}}tt|ywrr#r)rrrs rrz$reshape_blockwise..s!PDAq%A-!Ps!#Nz/Need to specify chunks if expanding dimensions.c3 K|]}|ywrr rs rrz$reshape_blockwise..s#6!A#6 c3DK|]}tt|ywrrrs rrz$reshape_blockwise..s)HA%A-)Hs c3 K|]}|ywrr rs rrz$reshape_blockwise..s"71"7rz3Chunk sizes do not match for the following chunks: rz%. The corresponding chunksizes are: rz". (restricted to first 5 entries).rrzESetting chunks is not allowed when reducing the number of dimensions.T)r0c3 K|]}|ywrr rs rrz$reshape_blockwise..s4a4rc32K|]}t|ywr)r#rs rrz$reshape_blockwise..s%F1eAh%Fs)!rrr{rrrzrrxr daskrrEr rr enumerate TypeErrorrr'r(rwr_reshape_blockwiserr r_metar=_convert_to_shaper)reversedr#r$)rBr{rEoutname chunk_tuples out_shapesout_chunk_tuples in_shapesrin_cout_cnon_matching_chunkschunk_in chunk_outrrrr5r6r nr_out_chunks output_chunksctr nr_chunks_dimdimension_chunksrVs rreshape_blockwisersh QWW%' eU #1e 5zQVV 0QVVQVVQXXA66"Xa%77G!PIahhMN N'#6v#678 )H)H IJ"7ahh"789 %.c)Z.H$I   =D%yy$))E"22e   E"5bq"9:QAaD:;<5EXY[Z[E\5]ae5]4^_34 /2. /   +)UJI  $ U9--" E(O34 !     //A3OUGV!''::  S  '6 t'#Aq)^g4188456  !Y7J & c#qxx !9nMG%F %FGH+. *J+    'Hi     Jy) )  QVVIh/ 0     C  M C%h}&=>=-( XD  # #Jtcz$:3};MPQ;QTU;U$V W XU#345 }   + +GSs KE x '>!?agg NNY ;5]  4 s&=Q. Q!  Q&,AQ+Q2AQ7c$|j|Sr)r)arrr{s rrr<s ;;u rDc6ggtt|jt|zz}tt d|}|D]}dg||< |j D]\}}||j ||tt d|S)Nc"|jSr)copyrAs rrCz#_convert_to_shape..Is affhrDrc"tt|Srr@rAs rrCz#_convert_to_shape..Ps vc1~rD)rsetvaluesrr)itemsr$r)r{r5one_dims output_shaperkvs rrr@s &(D C   " #$s8}4%L. =>L # Q!)1QuQx() -|< ==rD)F)TNr)rBr r{zint | tuple[int, ...]rEz"tuple[tuple[int, ...], ...] | Nonereturnr )r{ztuple[int, ...]r5zdict[int, int]rz list[int])& __future__rr' functoolsr itertoolsroperatorrnumpyrdask._task_specrrdask.array.corer dask.array.utilsr dask.baser dask.corer dask.highlevelgraphr dask.utilsrr&r=r-r,r+r*r.rrzrrrr rDrrs" )!,. ]V@M`>.\4~F26tO tO tO /tO tOn> >'5>AJ>rD