bim ddlmZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZmZed Zdd ed dd dddf dZddZdZdZy)) annotationsN)partial) open_files)concat) from_delayed) read_bytes)delayed) parse_bytessystem_encodingT)pureinferstrictFc | | tdt|tr t|}||dvr|} d}nd} t |fd|||| d|xsi} |C| D cgc]6} t t t tt| || 8} } nBg} tdt| |D]T}| |||z}t tt ttt| ||}| j|Vnt|f||jnd |d || d |xsi}|d }t|Dcgc]}t t ||||} }| rut tt#|d Dcgc]\}}|gt||zc}}}t%| |Dcgc]\}}t t&||} }}| s td||r t)| } | Scc} wcc}wcc}}wcc}}w)a\ Read lines from text files Parameters ---------- urlpath : string or list Absolute or relative filepath(s). Prefix with a protocol like ``s3://`` to read from alternative filesystems. To read from multiple files you can pass a globstring or a list of paths, with the caveat that they must all have the same protocol. blocksize: None, int, or str Size (in bytes) to cut up larger files. Streams by default. Can be ``None`` for streaming, an integer number of bytes, or a string like "128MiB" compression: string Compression format like 'gzip' or 'xz'. Defaults to 'infer' encoding: string errors: string linedelimiter: string or None collection: bool, optional Return dask.bag if True, or list of delayed values if false storage_options: dict Extra options that make sense to a particular storage connection, e.g. host, port, username, password, etc. files_per_partition: None or int If set, group input files into partitions of the requested size, instead of one partition per file. Mutually exclusive with blocksize. include_path: bool Whether or not to include the path in the bag. If true, elements are tuples of (line, path). Default is False. Examples -------- >>> b = read_text('myfiles.1.txt') # doctest: +SKIP >>> b = read_text('myfiles.*.txt') # doctest: +SKIP >>> b = read_text('myfiles.*.txt.gz') # doctest: +SKIP >>> b = read_text('s3://bucket/myfiles.*.txt') # doctest: +SKIP >>> b = read_text('s3://key:secret@bucket/myfiles.*.txt') # doctest: +SKIP >>> b = read_text('hdfs://namenode.example.com/myfiles.*.txt') # doctest: +SKIP Parallelize a large file by providing the number of uncompressed bytes to load into each partition. >>> b = read_text('largefile.txt', blocksize='10MB') # doctest: +SKIP Get file paths of the bag by setting include_path=True >>> b = read_text('myfiles.*.txt', include_path=True) # doctest: +SKIP >>> b.take(1) # doctest: +SKIP (('first line of the first file', '/home/dask/myfiles.0.txt'),) Returns ------- dask.bag.Bag or list dask.bag.Bag if collection is True or list of Delayed lists otherwise. See Also -------- from_sequence: Build bag from Python sequence Nz7Only one of blocksize or files_per_partition can be setN  z rrt)modeencodingerrors compressionnewline) delimiterr F)r blocksizesampler include_pathzNo files found) ValueError isinstancestrr rr listrfile_to_blocksrangelenrmapappendrencodedecode enumeratezip attach_pathr)urlpathrrrr linedelimiter collectionstorage_optionsfiles_per_partitionrrfilesfilblocksstart block_files block_lineso raw_blocksbipathpathsentrys H/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/dask/bag/text.py read_textrBsyP!4!@RSS)S! *  : :#G MG  #  $"   &!    G VFFq#e*.AB +#EU5H-HJ -gfo GCL V#  k* +   0=0Im**,u#%  $" qT J'  GFOAx ?   9QqT?S4Z]!33STEFIQVEW6AeT$ $UD1F )733f% MgB Ts;G2=G7;G<2 Hc#@K5}|]|j}|s gcdddS|j|}fd|ddDcgc]}||z c}|ddzDEd{n|D]}r|jfn|dddycc}w7.#1swYyxYww)Nc3BK|]}r|jfn|ywN)r>).0liner lazy_files rA z!file_to_blocks..s)+7y~~&D@s)readsplitr>)rrHrftextpartsrGs`` rAr%r%s  Ga  668D G G JJy)E:?*E$TI-Ebc R    G0ps rAr.r.s $isc|j||}|dvr"tj||}t|S|sgS|j |}|ddDcgc]}||z c}|j |s |ddz}|Sgz}|Scc}w)Nr)rrJ)r+ioStringIOr$rLendswith) rRrrline_delimiterrNlinesrOtouts rAr+r+s <<& )D77 D.9E{I >*+0":6aq>!6"mmN;E"#J  BD  7s BrE) __future__rrU functoolsr fsspec.corertlzr dask.bag.corer dask.bytesr dask.delayedr dask.utilsr r rBr%r.r+rQrAres^" "&! 3 t    PfG" rd