bi*BddlmZddlZddlZddlmZddlmZddlZ ddl m Z ddl m ZddlmZddlmZddlmZdd lmZdd lmZmZdd lmZ ddZdZej>ded ddddd ddde j@ddf dZ ddZ!dZ"dZ#y)) annotationsN)partial) zip_longest) open_files)compute) read_bytes)flatten)dataframe_creation_dispatch)insert_meta_param_description make_meta)delayedrecordsutf-8strictc n||dk(}|dk7r |r td|| d<|xr|dk(| d<t|df||| |j|d|xsi} t| |j D cgc]\} }t t || | }} }|r | t} tt|i| S|Scc}} w)aWrite dataframe into JSON text files This utilises ``pandas.DataFrame.to_json()``, and most parameters are passed through - see its docstring. Differences: orient is 'records' by default, with lines=True; this produces the kind of JSON output that is most common in big-data applications, and which can be chunked when reading (see ``read_json()``). Parameters ---------- df: dask.DataFrame Data to save url_path: str, list of str Location to write to. If a string, and there are more than one partitions in df, should include a glob character to expand into a set of file names, or provide a ``name_function=`` parameter. Supports protocol specifications such as ``"s3://"``. encoding, errors: The text encoding to implement, e.g., "utf-8" and how to respond to errors in the conversion (see ``str.encode()``). orient, lines, kwargs passed to pandas; if not specified, lines=True when orient='records', False otherwise. storage_options: dict Passed to backend file-system implementation compute: bool If true, immediately executes. If False, returns a set of delayed objects, which can be computed at a later time. compute_kwargs : dict, optional Options to be passed in to the compute method compression : string or None String like 'gzip' or 'xz'. name_function : callable, default None Function accepting an integer (partition index) and producing a string to replace the asterisk in the given filename globstring. Should preserve the lexicographic order of partitions. r:;;  s)!B1c|5}|j|fi|dddtjj|jS#1swY2xYwN)r-ospathnormpath)r#openfiler'fs r,rrdsH  Q 1 77  HMM **  s AApandasiinferFc   ||dk(}|dk7r |r td|r|dk7s|s td|xsi}| durd} d t| trttj | } |rt |d f|||| d |}| rN|\}}} |d }t j fd |D}t fd t||D}n |\}}d}d}d}t|}| t|||| | ||| } t| } t||Dcgc]#\}}tt|||| | ||| |  %}}}nht|df|||d|}t j fd|D}|Dcgc].}tt|||| |  |j || 0}}t#j$|| Scc}}wcc}w)a Create a dataframe from a set of JSON files This utilises ``pandas.read_json()``, and most parameters are passed through - see its docstring. Differences: orient is 'records' by default, with lines=True; this is appropriate for line-delimited "JSON-lines" data, the kind of JSON output that is most common in big-data scenarios, and which can be chunked when reading (see ``read_json()``). All other options require blocksize=None, i.e., one partition per input file. Parameters ---------- url_path: str, list of str Location to read from. If a string, can include a glob character to find a set of file names. Supports protocol specifications such as ``"s3://"``. encoding, errors: The text encoding to implement, e.g., "utf-8" and how to respond to errors in the conversion (see ``str.encode()``). orient, lines, kwargs passed to pandas; if not specified, lines=True when orient='records', False otherwise. storage_options: dict Passed to backend file-system implementation blocksize: None or int If None, files are not blocked, and you get one partition per input file. If int, which can only be used for line-delimited JSON files, each partition will be approximately this size in bytes, to the nearest newline character. sample: int Number of bytes to pre-load, to provide an empty dataframe structure to any blocks without data. Only relevant when using blocksize. encoding, errors: Text conversion, ``see bytes.decode()`` compression : string or None String like 'gzip' or 'xz'. engine : callable or str, default ``pd.read_json`` The underlying function that dask will use to read JSON files. By default, this will be the pandas JSON reader (``pd.read_json``). If a string is specified, this value will be passed under the ``engine`` key-word argument to ``pd.read_json`` (only supported for pandas>=2.0). include_path_column : bool or str, optional Include a column with the file path where each row in the dataframe originated. If ``True``, a new column is added to the dataframe called ``path``. If ``str``, sets new column name. Default is ``False``. path_converter : function or None, optional A function that takes one argument and returns a string. Used to convert paths in the ``path`` column, for instance, to strip a common prefix from all the paths. $META Returns ------- dask.DataFrame Examples -------- Load single file >>> dd.read_json('myfile.1.json') # doctest: +SKIP Load multiple files >>> dd.read_json('myfile.*.json') # doctest: +SKIP >>> dd.read_json(['myfile.1.json', 'myfile.2.json']) # doctest: +SKIP Load large line-delimited JSON files using partitions of approx 256MB size >> dd.read_json('data/file*.csv', blocksize=2**28) NrrzSJSON file chunking only allowed for JSON-linesinput (orient='records', lines=True).Tr1c|Sr/)xs r,zread_json..s1)engine ) blocksizesampler include_pathrc3.K|] }|ywr/r9).0ppath_converters r, zread_json..s,N1^A->,Nsc3NK|]\}}|gt|zywr/)len)rCrDchunkrEs r,rFzread_json..s+!5=Q"#c%j0!s"%r/)metart)rrrc3BK|]}|jywr/)r1)rCr4rEs r,rFzread_json..s(OA)?(Os)r isinstancestrrpd read_jsonrCategoricalDtyper rread_json_chunkr rr rread_json_filer1dd from_delayed)r$rrr%r?r@rrrrJr=include_path_columnrEr'b_outfirstchunkspaths first_path path_dtype flat_paths flat_chunksrIr1r+filesr4s ` r,rPrPjsev })# uWXXf ) 4  &+Od"$$&#f5    #,    #( E65'a1J,,,N,NNJ !ADUFAS!J"ME6J JJfo <"# D +; C  t %GO $#        #    (((O(OO    $GN ##qvv&      ??5t ,,M  0  s (F>23Gc tj|j||} | jd|| fddd|} || jr|S|rt | |||} | S)NrrTrr)ioStringIOdecodeseekemptyadd_path_column) rIrrr= column_namer1r\r'rJsr#s r,rRrR&sg ELL623AFF1I  :)4 :6 :B BHH  RdJ ? Ir<cl|5}||f||d|} ddd|rt |||}  S#1swYxYw)Nra)rg) r4rrr=rhr1r\r' open_filer#s r,rSrS5sN Ei I DfE DV DE RdJ ? I EEs *3c ||jvrtd|d|jdi|tj|gt |z|iS)Nz(Files already contain the column name: 'z^', so the path column cannot use this name. Please set `include_path_column` to a unique name.)dtyper9)columnsrassignrOSeriesrH)r#rhr1rms r,rgrg=s^bjj 6{mD   299 O RYYvB/?u%MN OOr<) rNNTrrNNNr/)$ __future__rrbr0 functoolsr itertoolsrr5rO fsspec.corerdask.dataframe dataframerT dask.baserr" dask.bytesr dask.corer dask.dataframe.backendsr dask.dataframe.utilsr r dask.delayedr r-rregister_inplacerPrRrSrgr9r<r,r~s" !"-!?I     M`+ .--h7      <<w-8w-vRV Pr<