biddlZddlmZmZddlmZddlmZmZeje Z GddZ edZ d Zd Zeej d efd Zy) N)PoolRLock)tqdm) experimentalloggingceZdZdZy)ParallelBackendConfigN)__name__ __module__ __qualname__ backend_nameU/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/datasets/parallel/parallel.pyr r sLrr c ntjt||||||||| St||||||||| S)a **Experimental.** Apply a function to iterable elements in parallel, where the implementation uses either multiprocessing.Pool or joblib for parallelization. Args: function (`Callable[[Any], Any]`): Function to be applied to `iterable`. iterable (`list`, `tuple` or `np.ndarray`): Iterable elements to apply function to. num_proc (`int`): Number of processes (if no backend specified) or jobs (using joblib). types (`tuple`): Additional types (besides `dict` values) to apply `function` recursively to their elements. disable_tqdm (`bool`): Whether to disable the tqdm progressbar. desc (`str`): Prefix for the tqdm progressbar. single_map_nested_func (`Callable`): Map function that applies `function` to an element from `iterable`. Takes a tuple of function, data_struct, types, rank, disable_tqdm, desc as input, where data_struct is an element of `iterable`, and `rank` is used for progress bar. )r r_map_with_multiprocessing_pool_map_with_joblib) functioniterablenum_procbatched batch_sizetypes disable_tqdmdescsingle_map_nested_funcs r parallel_maprsV"))1- h':ulTXZp   (Hgz5,PTVl rc |t|kr|n t|}g} t|D][} t||z} t||z} | | zt| | z} | | z| | krdndz}| j||| ||||| ||f]t|t d| Dk7r*t dt|dt d| Dt jd|dt|d | Dcgc]}t|dc}d \}}|stftj}}t||| 5}|j|| }dddt jd |d Dcgc] }|D]}| }}}t jdt|d|Scc}w#1swYcxYwcc}}w)Nrc38K|]}t|dywr Nlen.0is r z1_map_with_multiprocessing_pool..7s:!C!I:zHError dividing inputs iterable among processes. Total number of objects z , length: c38K|]}t|dywr"r#r%s rr(z1_map_with_multiprocessing_pool..;s93qt99r)z Spawning z processes for z objects in slices of )NN)initargs initializerz Finished z processesz Unpacked z objects) r$rangeminappendsum ValueErrorloggerinforrset_lockrmap)rrrrrrrrr split_kwdsindexdivmodstartendr'r+r,poolmappedproc_resobjs rrr+s$s8}4x#h-HJxr(mx'(mh&e c%o-ck%#+Q158XeC%8':uV[]ikopq r 8}:z:::''*8}o69j99: <   KK H:_S]O;QfpRqabSVWXYZW[S\RqQrs'Hk !& DMM+ h{ C>t0*=> KK)H:Z01"( =hH =Sc =c =F = KK)CK=12 MSr >>>s2F0F59G5F>c  ddl jtj|5 j  fd|DcdddS#1swYyxYw)Nr)n_jobsc 3\K|]#}j|dddf%yw)NT)delayed)r&r?rrrjoblibrrs rr(z#_map_with_joblib..Us?!  3FNN1 2Hc7JX]_ceiko3p q! s),)rDparallel_backendr rParallel) rrrrrrrrrrDs ` ``` `@rrrMsZ   !6!C!CH U  v ! ! !     s $AA#rc#K|t_|dk(r ddlm}| ddt_y#dt_wxYww)a **Experimental.** Configures the parallel backend for parallelized dataset loading, which uses the parallelization implemented by joblib. Args: backend_name (str): Name of backend for parallelization implementation, has to be supported by joblib. Example usage: ```py with parallel_backend('spark'): dataset = load_dataset(..., num_proc=2) ``` sparkr)register_sparkN)r r joblibsparkrI)rrIs rrErE[s< *6&w. 2 -1*T*sA1 A >A) contextlibmultiprocessingrr tqdm.autorutilsrr get_loggerr r2r rrrcontextmanagerstrrErrrrRsw')   H %4D   2322r