bi,ddlZddlmZmZddlmZddlmZddlm Z m Z m Z m Z m Z mZddlZddlmZddlmZdd lmZdd lmZmZdd lmZmZe r ddlZd d lmZddde fdZ!eGddZ"ddde#fdZ$y)N) dataclassfield)BytesIO)Path) TYPE_CHECKINGAnyClassVarDictOptionalUnion)config)DownloadConfig) array_cast) is_local_pathxopen)no_op_if_value_is_nullstring_to_dict) FeatureTypepdfpdfplumber.pdf.PDFreturnct5}|jD]'}|j|jj)|j cdddS#1swYyxYw)z-Convert a pdfplumber.pdf.PDF object to bytes.N)rpageswriterstreamgetvalue)rbufferpages P/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/datasets/features/pdf.py pdf_to_bytesr"sO !fII *D LL ) * !!!s AAA$ceZdZUdZdZeed<eddZe e ed<dZ e e ed <e je je j d Ze eed <eddd Ze ed <dZdee eeedfdefdZddeddfdZdedee dfffdZdee j:e j<e j>fde j<fdZ dde j<de j<fdZ!y)Pdfa1 **Experimental.** Pdf [`Feature`] to read pdf documents from a pdf file. Input: The Pdf feature accepts as input: - A `str`: Absolute path to the pdf file (i.e. random access is allowed). - A `pathlib.Path`: path to the pdf file (i.e. random access is allowed). - A `dict` with the keys: - `path`: String with relative path of the pdf file in a dataset repository. - `bytes`: Bytes of the pdf file. This is useful for archived files with sequential access. - A `pdfplumber.pdf.PDF`: pdfplumber pdf object. Args: mode (`str`, *optional*): The mode to convert the pdf to. If `None`, the native mode of the pdf is used. decode (`bool`, defaults to `True`): Whether to decode the pdf data. If `False`, returns the underlying dictionary in the format `{"path": pdf_path, "bytes": pdf_bytes}`. Examples: ```py >>> from datasets import Dataset, Pdf >>> ds = Dataset.from_dict({"pdf": ["path/to/pdf/file.pdf"]}).cast_column("pdf", Pdf()) >>> ds.features["pdf"] Pdf(decode=True, id=None) >>> ds[0]["pdf"] >>> ds = ds.cast_column("pdf", Pdf(decode=False)) >>> ds[0]["pdf"] {'bytes': None, 'path': 'path/to/pdf/file.pdf'} ``` TdecodeNF)defaultrepridrdtypebytespathpa_type)r&initr'_typec|jSN)r-)selfs r!__call__z Pdf.__call__Ms ||valuercxtjrddl}nd}t|tr|ddSt|t rt |j ddSt|ttfrd|dS|+t||jjr t|S|jd6tjj|drd|jddS|jd|jd#|jd|jddSt!d|d) zEncode example into a format for Arrow. Args: value (`str`, `bytes`, `pdfplumber.pdf.PDF` or `dict`): Data passed as input to Pdf feature. Returns: `dict` with "path" and "bytes" fields rNr,r+r,r*r+zRA pdf sample should have one of 'path' or 'bytes' but they are missing or None in .)rPDFPLUMBER_AVAILABLE pdfplumber isinstancestrrabsoluter+ bytearrayrPDFencode_pdfplumber_pdfgetosr,isfile ValueError)r2r5r:s r!encode_examplezPdf.encode_examplePs  & & J eS !!D1 1 t $ 01DA A y1 2 51 1  # 5*..:L:L(M(/ / YYv  *rww~~eFm/L!599V+<= = YYw  +uyy/@/L"YYw/69JK Kdejdkklm r4c|js tdtjrddl}n t d|i}|d|d}}||t d|dt|r|j|}|S|jd d }|jtjrtjntj} t||d } |j| } t!| } t#|d | } |j| S|jt%|5} | }ddd|S#t $rd} YawxYw#1swYSxYw)aiDecode example pdf file into pdf data. Args: value (`str` or `dict`): A string with the absolute pdf file path, a dictionary with keys: - `path`: String with absolute or relative pdf file path. - `bytes`: The bytes of the pdf file. token_per_repo_id (`dict`, *optional*): To access and decode pdf files from private repositories on the Hub, you can pass a dictionary repo_id (`str`) -> token (`bool` or `str`). Returns: `pdfplumber.pdf.PDF` zKDecoding is disabled for this feature. Please use Pdf(decode=True) instead.rNz6To support decoding pdfs, please install 'pdfplumber'.r,r+z@A pdf should have one of 'path' or 'bytes' but both are None in r8::repo_idtokenrbdownload_config)r% RuntimeErrorrr9r: ImportErrorrDropensplit startswith HF_ENDPOINTHUB_DATASETS_URLHUB_DATASETS_HFFS_URLrrArrr)r2r5token_per_repo_idr:r,bytes_r source_urlpatternrIrKrNfps r!decode_examplezPdf.decode_examplesso&{{lm m  & & VW W  $ " V}eGnf >| #cdicjjk!lmm &)*//$/C( %"&D!1"!5J&001C1CD//#99 %"0W"Ei"P 1 5 5g >'55&AOdD/JA*:??1-- 1 Q  &% $%   s E3E E EErcLddlm}|jr|S|d|ddS)zfIf in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.r)Valuebinarystringr*)featuresr_r%)r2r_s r!flattenz Pdf.flattens2#{{  xh r4storagectjj|jrltjdgt |ztj }tjj||gddg|j}ntjj|jrltjdgt |ztj}tjj||gddg|j}n*tjj|jr|jjddk\r|jd}n6tjdgt |ztj }|jjddk\r|jd}n6tjdgt |ztj}tjj||gddg|j}t||j S)aCast an Arrow array to the Pdf arrow storage type. The Arrow types that can be converted to the Pdf pyarrow storage type are: - `pa.string()` - it must contain the "path" data - `pa.binary()` - it must contain the image bytes - `pa.struct({"bytes": pa.binary()})` - `pa.struct({"path": pa.string()})` - `pa.struct({"bytes": pa.binary(), "path": pa.string()})` - order doesn't matter - `pa.list(*)` - it must contain the pdf array data Args: storage (`Union[pa.StringArray, pa.StructArray, pa.ListArray]`): PyArrow array to cast. Returns: `pa.StructArray`: Array in the Pdf arrow storage type, that is `pa.struct({"bytes": pa.binary(), "path": pa.string()})`. Ntyper+r,maskr)patypes is_stringrgarraylenr` StructArray from_arraysis_null is_binaryra is_structget_field_indexrrr-)r2rd bytes_array path_arrays r! cast_storagezPdf.cast_storages& 88  gll +((D6CL#8ryy{KKnn00+w1G'SYIZahapapar0sG XX   -4&3w<"7biikJJnn00':1FRXHY`g`o`o`q0rG XX   -||++G49%mmG4  hhvG '<299;O ||++F3q8$]]62 XXtfs7|&;"))+N nn00+z1JWV\L]dkdsdsdu0vG'4<<00r4c ritfd}tj|jDcgc]}||d ||dn|dndc}tj}tj|j djDcgc]%}|t jj|nd'c}tj}tjj||gddg|j}t||jScc}wcc}w)a4Embed PDF files into the Arrow array. Args: storage (`pa.StructArray`): PyArrow array to embed. Returns: `pa.StructArray`: Array in the PDF arrow storage type, that is `pa.struct({"bytes": pa.binary(), "path": pa.string()})`. Ncr|jdd}|jtjrtjntj }t ||}|j|dnd}t|}t|d|5}|jcdddS#1swYyxYw)NrGrHrIrJrLrM) rRrSrrTrUrVrrArrread)r,rYrZsource_url_fieldsrKrNr[rWs r! path_to_bytesz(Pdf.embed_storage..path_to_bytessD)"-J+5+@+@ASAS+T''Z`ZvZv !/z7 C K\Kh%))*;I*FGnrE,59OtT?C qvvx   s B--B6r+r,rfrh)rrjrm to_pylistr`rrBr,basenamerarorprqrr-)r2rdrWr|xrur,rvs ` r! embed_storagezPdf.embed_storages%  $ "     hh!**, UVTaQwZ-?qy)QwZgkk   XXNUmm\bNcNmNmNo pdt'7RWW  d #T A p ..,,k:-FRXHY`k`s`s`u,v'4<<00  qs D/ *D4r1)"__name__ __module__ __qualname____doc__r%bool__annotations__rr(r r<r)r rjstructr`rar-rr/r3r r+r>dictrEr]r rc StringArrayro ListArrayrwrr4r!r$r$s+#JFDd7B 70E8C=/&RYYibiik'RSGXc]Su5u=E3=!E#uiG[*[$\!ae!F8D8EY8t  }d3 3E.FFG  #1E".."..",,*V$W#1\^\j\j#1J&1R^^&1PRP^P^&1r4r$ct|drEt|jdr/|jjr|jjddSdt|dS)aA Encode a pdfplumber.pdf.PDF object into a dictionary. If the PDF has an associated file path, returns the path. Otherwise, serializes the PDF content into bytes. Args: pdf (pdfplumber.pdf.PDF): A pdfplumber PDF object. Returns: dict: A dictionary with "path" or "bytes" field. rnameNr7)hasattrrrr")rs r!r@r@sJsH'#**f"=#**// $77|C'899r4)%rB dataclassesrriorpathlibrtypingrrr r r r pyarrowrjrdownload.download_configrtablerutils.file_utilsrrutils.py_utilsrrr:rbrr+r"r$rr@rr4r!rsy (FF53C%!*!u! f1f1 f1R:3::r4