biL d Z ddlZddlZddlZddlZddlZddlmZ ddlmZ ddl m
Z
m Z m Z ddl
Z
ddlmZ ddlmZmZ dd lmZ dd
lmZ dd lmZ dd lmZ dd
lm Z ddl m Z m Z e e Z!e G d d Z"e G d d Z# G d de$ Z% G d de$ Z&e G d d Z'e G d d Z( G d d e)e*e(f Z+y) aw DatasetInfo record information we know about a dataset.
This includes things that we know about the dataset statically, i.e.:
- description
- canonical location
- does it have validation and tests splits
- size
- etc.
This also includes the things that can and should be computed once we've
processed the dataset as well:
- number of examples (in each split)
- etc.
N) dataclass)Path)ClassVarOptionalUnion) url_to_fs) DatasetCardDatasetCardData )config)Features) SplitDict)Version)
get_logger)asdict
unique_valuesc * e Zd ZU dZeed< dZeed< y)SupervisedKeysData inputoutputN)__name__
__module__ __qualname__r str__annotations__r H/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/datasets/info.pyr r 7 s E3OFCr r c * e Zd ZU dZeed< dZeed< y)DownloadChecksumsEntryDatar keyvalueN)r r r r" r r r# r r r r! r! = s CME3Or r! c e Zd ZdZy) MissingCachedSizesConfigErrorz;The expected cached sizes of the download file are missing.Nr r r __doc__r r r r% r% C s Er r% c e Zd ZdZy)NonMatchingCachedSizesErrorz/The prepared split doesn't have expected sizes.Nr& r r r r) r) G s 9r r) c V e Zd ZU dZee ed< dZee ed< d Z e
dedd fd Z y)PostProcessedInfoNfeaturesresources_checksumsc | j @t | j t s%t j | j | _ y y y N)r,
isinstancer
from_dictselfs r
__post_init__z PostProcessedInfo.__post_init__P s8 == $Z
x-P $ . .t}} =DM .Q $r post_processed_info_dictreturnc
t j | D ch c] }|j }} | di |j D ci c]
\ }}||v s || c}}S c c}w c c}}w Nr dataclassesfieldsnameitems)clsr5 f field_nameskvs r r1 zPostProcessedInfo.from_dictU s` '2'9'9#'>?!qvv? ?]'?'E'E'G\tq!1P[K[ad\]] @\ A A!A!) r r r r, r r
r r- dictr4 classmethodr1 r r r r+ r+ K sL #'Hhx '*.$ .>
^ ^:M ^ ^r r+ c e Zd ZU dZ ej
e Zeed< ej
e Z eed< ej
e Z
eed< ej
e Z eed< dZ e
e ed< dZe
e ed < dZe
e ed
< dZe
e ed < dZe
e ed < dZe
e ed
< dZe
eeef ed< dZe
e ed< dZe
e ed< dZ e
e ed< dZ e
e ed< dZ e
e ed< dZ e
e ed< g dZ!e"e#e ed< d Z$d(de
e fdZ%d)dZ&d Z'e(d e#d fd Z)e(d*d ede
e d d fd Z*e(d!ed d fd" Z+d+d,d#Z,d-d$Z-d efd%Z.e(d&ed d fd' Z/y). DatasetInfoa Information about a dataset.
`DatasetInfo` documents datasets, including its name, version, and features.
See the constructor arguments and properties for a full list.
Not all fields are known on construction and may be updated later.
Attributes:
description (`str`):
A description of the dataset.
citation (`str`):
A BibTeX citation of the dataset.
homepage (`str`):
A URL to the official homepage for the dataset.
license (`str`):
The dataset's license. It can be the name of the license or a paragraph containing the terms of the license.
features ([`Features`], *optional*):
The features used to specify the dataset's column types.
post_processed (`PostProcessedInfo`, *optional*):
Information regarding the resources of a possible post-processing of a dataset. For example, it can contain the information of an index.
supervised_keys (`SupervisedKeysData`, *optional*):
Specifies the input feature and the label for supervised learning if applicable for the dataset (legacy from TFDS).
builder_name (`str`, *optional*):
The name of the `GeneratorBasedBuilder` subclass used to create the dataset. It is also the snake_case version of the dataset builder class name.
config_name (`str`, *optional*):
The name of the configuration derived from [`BuilderConfig`].
version (`str` or [`Version`], *optional*):
The version of the dataset.
splits (`dict`, *optional*):
The mapping between split name and metadata.
download_checksums (`dict`, *optional*):
The mapping between the URL to download the dataset's checksums and corresponding metadata.
download_size (`int`, *optional*):
The size of the files to download to generate the dataset, in bytes.
post_processing_size (`int`, *optional*):
Size of the dataset in bytes after post-processing, if any.
dataset_size (`int`, *optional*):
The combined size in bytes of the Arrow tables for all splits.
size_in_bytes (`int`, *optional*):
The combined size in bytes of all files associated with the dataset (downloaded files + Arrow files).
**config_kwargs (additional keyword arguments):
Keyword arguments to be passed to the [`BuilderConfig`] and used in the [`DatasetBuilder`].
)default_factory descriptioncitationhomepagelicenseNr, post_processedsupervised_keys builder_name dataset_name config_nameversionsplitsdownload_checksums
download_sizepost_processing_size dataset_size
size_in_bytes)rQ rU rW r, rS _INCLUDED_INFO_IN_YAMLc | j >t | j t s$t j | j | _ | j >t | j t
s$t
j | j | _ | j st | j t sYt | j t rt | j | _ n$t j | j | _ | j >t | j t s$t j | j | _ | j nt | j t sSt | j t t f rt | j | _ y t di | j | _ y y y r8 )r, r0 r
r1 rM r+ rR r r rS r from_split_dictrN r tuplelistr2 s r r4 zDatasetInfo.__post_init__ s> == $Z
x-P $ . .t}} =DM *:d>Q>QSd3e"3"="=d>Q>Q"RD << #Jt||W,M$,,, &t|| 4 & 0 0 > ;; ":dkk9+M#33DKK@DK +Jt?S?SUg4h$..
>'94;O;O'P$'9'QD
Example:
```py
>>> from datasets import load_dataset
>>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
>>> ds.info.write_to_directory("/path/to/directory/")
```
wb) pretty_printN)
r open posixpathjoinr DATASET_INFO_FILENAME
_dump_inforL LICENSE_FILENAME
_dump_license)r3 dataset_info_dirra r^ fs_r? s r write_to_directoryz DatasetInfo.write_to_directory s , +G0E2GQ
WWY^^$4f6R6RSUY
Z :^_ OOALO 9 : << (8&:Q:QRTXY
&]^""1%
&
& : :
&
&s B?#C ?C Cc |j t j t | |rdnd j d y)zQDump info in `file` file-like object open in bytes mode (to support remote files) Nindentutf-8)writejsondumpsr encode)r3 filera s r rf zDatasetInfo._dump_info s-
4::fTl 1$OVVW^_`r c X |j | j j d y)zTDump license in `file` file-like object open in bytes mode (to support remote files)rq N)rr rL ru )r3 rv s r rh zDatasetInfo._dump_license s
4<<&&w/0r
dataset_infosc : D cg c] }||j c}t dkD rt fdD rd S dj t d D j }dj t d D j }dj t d D j }dj t d D j }d }d } | |||||| S c c}w ) Nr c 3 . K | ] }d |k( yw)r Nr ).0 dset_inforx s r z)DatasetInfo.from_merge.. s )gI-*:i*G)gs z
c 3 4 K | ] }|j y wr/ )rI r{ infos r r} z)DatasetInfo.from_merge.. s /[T0@0@/[ c 3 4 K | ] }|j y wr/ )rJ r s r r} z)DatasetInfo.from_merge.. ,UtT]],Ur c 3 4 K | ] }|j y wr/ )rK r s r r} z)DatasetInfo.from_merge.. r r c 3 4 K | ] }|j y wr/ )rL r s r r} z)DatasetInfo.from_merge.. s +STDLL+Sr )rI rJ rK rL r, rN )copylenallrd r strip) r> rx r| rI rJ rK rL r, rN s ` r
from_mergezDatasetInfo.from_merge s ;HbiILa )b
} !c)gYf)g&g # # kk-/[]/["[\bbd ;;},U},U UV\\^;;},U},U UV\\^++m+S]+S STZZ\ # +
cs
DDri r6 c R t |fi |xs i ^}}t j d| |s t d |j t j |t j dd 5 }t j | }ddd | j S # 1 sw Y xY w)a Create [`DatasetInfo`] from the JSON file in `dataset_info_dir`.
This function updates all the dynamically generated fields (num_examples,
hash, time of creation,...) of the [`DatasetInfo`].
This will overwrite all previous metadata.
Args:
dataset_info_dir (`str`):
The directory containing the metadata file. This
should be the root directory of a specific dataset version.
storage_options (`dict`, *optional*):
Key/value pairs to be passed on to the file-system backend, if any.
Example:
```py
>>> from datasets import DatasetInfo
>>> ds_info = DatasetInfo.from_directory("/path/to/directory/")
```
zLoading Dataset info from zECalling DatasetInfo.from_directory() with undefined dataset_info_dir.rrq encodingN) r loggerdebug
ValueErrorrb rc rd r re rs loadr1 )r> ri r^ rj rk r? dataset_info_dicts r from_directoryzDatasetInfo.from_directory s 4 +G0E2GQ 12B1CDE de e
WWY^^$4f6R6RSUXcjW
k -op $ ! -}} .// - -s .B B&r c
t j | D ch c] }|j }} | di |j D ci c]
\ }}||v s || c}}S c c}w c c}}w r8 r9 )r> r r? r@ rA rB s r r1 zDatasetInfo.from_dict s_ '2'9'9#'>?!qvv? ?V'8'>'>'@Utq!ADTadUVV @UrC c | j } |j di |j j D ci c] \ }}||s|t j | ! c}} y c c}}w r8 )__dict__updater= r deepcopy)r3 other_dataset_info ignore_none self_dictrA rB s r r zDatasetInfo.update se MM
/ 7 7 = = ?AqM 4==##
s $A#c | j di | j j D ci c] \ }}|t j | c}}S c c}}w r8 ) __class__r r= r r )r3 rA rB s r r zDatasetInfo.copy( sC t~~X ATATAV WADMM!$4!4 WXX Ws Ac i }t | }|D ]b }|| j v st | | }t |d r|j ||< >t |d r|j ||< ^|||< d |S )N
_to_yaml_list_to_yaml_string)r rY getattrhasattrr r )r3 yaml_dictr r" r# s r
_to_yaml_dictzDatasetInfo._to_yaml_dict+ s "4L$ +Cd111 c*5/2%*%8%8%:IcN U$56%*%:%:% r r? r@ rA rB s r _from_yaml_dictzDatasetInfo._from_yaml_dict9 s MM), == $ 0$,$<$?!qvv? ?Ny'8Mtq!AC)FN)Fr/ )T)r rG )r6 rG )0r r r r' r: fieldr rI r rJ rK rL r, r r
rM r+ rN r rO rP rQ rR r r rS rD rT rU intrV rW rX rY r r] r4 rl rf rh rE r r r1 r r r r r r r rG rG [ s *Z ){((=K=%K%%c:Hc:%K%%c:Hc:$;$$S9GS9#'Hhx '26NH ./648OX 018 #'L(3-&"&L(3-&!%K# %-1GXeCL)
*1 !FHTN!)--#'M8C='*.(3-."&L(3-&#'M8C='3HT#Y / R&&X`aeXf &:a1
tM':
. 0c 0HTN 0^k 0 0B W$ W= W W
Y t O O O Or rG c R e Zd Zdd dZed
d Zededd fd ZdeddfdZy) DatasetInfosDictr6 Nc Z i }t j j |t j }t j j |t j
}|s| j
| }|j | t j j | r_t |dd 5 }|j D ci c] \ }} |t | }
}} t j |
||rdnd d d d t j j | r"t j | } | j } n d } t# } |rJ|j% | | t dt' | z dz n| } | j) t+ | y y c c} }w # 1 sw Y xY w)Nwrq r rn ro z---
z
---
)ospathrd r DATASETDICT_INFOS_FILENAMEREPOCARD_FILENAMEr r existsrb r= r rs dumpr r datar
to_dataset_card_datar saver )
r3 dataset_infos_dir overwritera total_dataset_infosdataset_infos_pathdataset_readme_pathr? rQ r| dataset_infos_dict dataset_carddataset_card_datas
r rl z#DatasetInfosDict.write_to_directoryE sz WW\\*;V=^=^ _ ggll+V>V W"&"5"56G"H ""4(
77>>, -(#@
UAQdQjQjQl&7M{IK !22&" & ,a\tT
U 77>>- .&++,?@L , 1 1 L / 1 4 45F GMYMa Gc*;&< r r r? rQ r s r r z DatasetInfosDict.from_directorya s; 23D2EFG
77>>"'',,'8&:R:RS T + 0 06G1H6KcKc1c d i i !22112CDD
77>>"'',,'8&:[:[\ ]bggll#4f6W6WXcjk
op ?Ciil>P>P>R :K): $[%:%:;L%MM
5L
s 6)F F ?F F F&r c t |j d t t f rt |d t r= | |d D ci c]( }|j dd t j | * c} S t j |d }|d j dd |_ | |j |i S | S c c}w )Nr rQ default)r0 r r] rD rG r rQ )r> r dataset_info_yaml_dictr s r r z'DatasetInfosDict.from_dataset_card_datau s '++N;dD\ J+N;TB
7H6W 3 /22=)LkNiNi 2O + : :;L^;\ ] +<^+L+P+PQ^`i+j (L 4 4lCDD5Ls -Cc | rcd|v r.t |d t r|d j dd |d i}n0d|v r*t |d t r|d D ci c] }|d |
}}ni }i || j D ci c] \ }}||j c}}}|j D ]
\ }}||d< t
| dk( rMt t |j |d< |d j dd }|dk7 r
d|i|d |d< y y g |d< t |j D ]1 \ }}|j dd d|i|}|d j | 3 y y c c}w c c}}w )Nr rQ r r )
r0 rD r r] r= r r nextitervaluespopsortedappend) r3 r dataset_metadata_infosconfig_metadatarQ r| r dset_info_yaml_dictr s r r z%DatasetInfosDict.to_dataset_card_data s !22zBSTbBcei7j%n599-SUfguUv*& #44DUVdDegk9l ,=^+L*' $M2OC*& *
*,&#(#X\XbXbXde>Tk9; 7 7 99e#
5H4M4M4O
A0 05@#M2
A&'1,48>Q>X>X>Z9[4\!.1 / ? C CMSW X )+ &{9+N;9%n5 , 57!.1;ABUB[B[B];^ U7K!7*..}dC.;[-cLb-c*%n5<<=ST UC * fs
E(<E-)FF)r6 N)r6 r ) r r r rl rE r r
r r r r r r r D sT 98 & K] $&Uo &U$ &Ur r ),r' r r: rs r rc r pathlibr typingr r r fsspec fsspec.corer huggingface_hubr r
r r r, r
rS r utilsr
utils.loggingr utils.py_utilsr r r r r r! Exceptionr% r) r+ rG rD r r r r r r s
! , ,
! 8 % 1
H
FI F:) : ^ ^ ^ eO eO eOPjUtC ,- jUr