bi=0dZddlZddlmZmZddlmZmZddlm Z ddl m Z ddl m Z dd lmZdd lmZmZmZdd lmZdd lmZeeZGd deZ ddedeeeeefdee deee efdeeeefdeeeeff dZ ddedeeeefdee deee efdeeeeeff dZ! ddedeeeefdee deee efdeeeeefdeef dZ" ddedeedeeeeeeeeeeefffdee deee efdeeeefdeeeefdefdZ# ddedeedeeeeeeeeeeefffdee deee efdeeeefdeeeeffdZ$y)zList and inspect datasets.N)MappingSequence)OptionalUnion)DownloadConfig) DownloadMode)StreamingDownloadManager) DatasetInfo)dataset_module_factoryget_dataset_builder_classload_dataset_builder) get_logger)Versionc eZdZy)SplitsNotFoundErrorN)__name__ __module__ __qualname__K/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/datasets/inspect.pyrr&srrpath data_filesdownload_config download_moderevisiontokenc rt||||||}|Dcic]}|td|||||||d|c}Scc}w)a6Get the meta information about a dataset, returned as a dict mapping config name to DatasetInfoDict. Args: path (`str`): path to the dataset repository. Can be either: - a local path to the dataset directory containing the data files, e.g. `'./dataset/squad'` - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]), e.g. `'rajpurkar/squad'`, `'nyu-mll/glue'` or``'openai/webtext'` revision (`Union[str, datasets.Version]`, *optional*): If specified, the dataset module will be loaded from the datasets repository at this version. By default: - it is set to the local version of the lib. - it will also try to load it from the main branch if it's not available at the local version of the lib. Specifying a version that is different from your local version of the lib might cause compatibility issues. download_config ([`DownloadConfig`], *optional*): Specific download configuration parameters. download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`): Download/generate mode. data_files (`Union[Dict, List, str]`, *optional*): Defining the data_files of the dataset configuration. token (`str` or `bool`, *optional*): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub. If `True`, or not specified, will get token from `"~/.huggingface"`. **config_kwargs (additional keyword arguments): Optional attributes for builder class which will override the attributes if supplied. Example: ```py >>> from datasets import get_dataset_infos >>> get_dataset_infos('cornell-movie-review-data/rotten_tomatoes') {'default': DatasetInfo(description="Movie Review Dataset. This is a dataset of containing 5,331 positive and 5,331 negative processed sentences from Rotten Tomatoes movie reviews...), ...} ``` )rrrrrr)r config_namerrrrrr)get_dataset_config_namesget_dataset_config_info) rrrrrr config_kwargs config_namesr s rget_dataset_infosr%*svX, '# L&(   ,  #!+'       s4c t|f||||d|}t|tjj |}t |j jxs+|jjd|jxsdgS)aGGet the list of available config names for a particular dataset. Args: path (`str`): path to the dataset repository. Can be either: - a local path to the dataset directory containing the data files, e.g. `'./dataset/squad'` - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]), e.g. `'rajpurkar/squad'`, `'nyu-mll/glue'` or``'openai/webtext'` revision (`Union[str, datasets.Version]`, *optional*): If specified, the dataset module will be loaded from the datasets repository at this version. By default: - it is set to the local version of the lib. - it will also try to load it from the main branch if it's not available at the local version of the lib. Specifying a version that is different from your local version of the lib might cause compatibility issues. download_config ([`DownloadConfig`], *optional*): Specific download configuration parameters. download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`): Download/generate mode. data_files (`Union[Dict, List, str]`, *optional*): Defining the data_files of the dataset configuration. **download_kwargs (additional keyword arguments): Optional attributes for [`DownloadConfig`] which will override the attributes in `download_config` if supplied, for example `token`. Example: ```py >>> from datasets import get_dataset_config_names >>> get_dataset_config_names("nyu-mll/glue") ['cola', 'sst2', 'mrpc', 'qqp', 'stsb', 'mnli', 'mnli_mismatched', 'mnli_matched', 'qnli', 'rte', 'wnli', 'ax'] ``` rrrr dataset_namer default) r r osrbasenamelistbuilder_configskeysbuilder_kwargsgetDEFAULT_CONFIG_NAME)rrrrrdownload_kwargsdataset_module builder_clss rr!r!msh, '#   N,NIYIYZ^I_`K  ++002 3 %%))-9X9X9e\ef8rreturnc t|f||||d|}t|tjj |}t |j j}|rt|dk(r|dnd} nd} |jxs| S)aWGet the default config name for a particular dataset. Can return None only if the dataset has multiple configurations and no default configuration. Args: path (`str`): path to the dataset repository. Can be either: - a local path to the dataset directory containing the data files, e.g. `'./dataset/squad'` - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]), e.g. `'rajpurkar/squad'`, `'nyu-mll/glue'` or``'openai/webtext'` revision (`Union[str, datasets.Version]`, *optional*): If specified, the dataset module will be loaded from the datasets repository at this version. By default: - it is set to the local version of the lib. - it will also try to load it from the main branch if it's not available at the local version of the lib. Specifying a version that is different from your local version of the lib might cause compatibility issues. download_config ([`DownloadConfig`], *optional*): Specific download configuration parameters. download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`): Download/generate mode. data_files (`Union[Dict, List, str]`, *optional*): Defining the data_files of the dataset configuration. **download_kwargs (additional keyword arguments): Optional attributes for [`DownloadConfig`] which will override the attributes in `download_config` if supplied, for example `token`. Returns: Optional[str]: the default config name if there is one Example: ```py >>> from datasets import get_dataset_default_config_name >>> get_dataset_default_config_name("openbookqa") 'main' ``` r'r(rrNr*) r r r+rr,r-r.r/lenr2) rrrrrr3r4r5r.default_config_names rget_dataset_default_config_namer:sZ, '#   N,NIYIYZ^I_`K;66;;=>O474HA4Moa0SW'  * * A.AArr c t|f||||||d|}|j} | j|r|jn t }|||_|j t|j| |jt|j|D cic]} | j| j|dc} | _| S| Scc} w#t$r} td| d} ~ wwxYw)aGet the meta information (DatasetInfo) about a dataset for a particular config Args: path (`str`): path to the dataset repository. Can be either: - a local path to the dataset directory containing the data files, e.g. `'./dataset/squad'` - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]), e.g. `'rajpurkar/squad'`, `'nyu-mll/glue'` or``'openai/webtext'` config_name (:obj:`str`, optional): Defining the name of the dataset configuration. data_files (:obj:`str` or :obj:`Sequence` or :obj:`Mapping`, optional): Path(s) to source data file(s). download_config (:class:`~download.DownloadConfig`, optional): Specific download configuration parameters. download_mode (:class:`DownloadMode` or :obj:`str`, default ``REUSE_DATASET_IF_EXISTS``): Download/generate mode. revision (:class:`~utils.Version` or :obj:`str`, optional): Version of the dataset to load. As datasets have their own git repository on the Datasets Hub, the default version "main" corresponds to their "main" branch. You can specify a different version than the default "main" by using a commit SHA or a git tag of the dataset repository. token (``str`` or :obj:`bool`, optional): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub. If True, or not specified, will get token from `"~/.huggingface"`. **config_kwargs (additional keyword arguments): optional attributes for builder class which will override the attributes if supplied. )namerrrrrN) base_pathr)r<r)zsplit_generatorerrs rr"r"s>#   '#   G <>> from datasets import get_dataset_split_names >>> get_dataset_split_names('cornell-movie-review-data/rotten_tomatoes') ['train', 'validation', 'test'] ``` )r rrrrr)r"r-r?r/) rr rrrrrr#r>s rget_dataset_split_namesrH*sMZ #  '#   D    " ##r)NNNNN)NNNN)NNNNNN)%__doc__r+collections.abcrrtypingrrdownload.download_configrdownload.download_managerr #download.streaming_download_managerr r>r loadr r r utils.loggingr utils.versionrrlogger ValueErrorrstrdictr-boolr%r!r:r"rHrrrrWsa ! -"43I &" H  * 48048<.2(, @ @tT3/0@n-@E,"345 @ uS'\*+ @ E$)$ % @J/3048<37 ? ?uS'\*+?n-?E,"345 ? tT3/0 ?H/3048<37 ;B ;BuS'\*+;Bn-;BE,"345 ;B tT3/0 ;Bc];B@"&_c048<.2(,: :#:sHSM73c8TW=FX@Y;Y3ZZ[\:n- : E,"345 : uS'\*+ : E$)$ %::~"&_c048<.2(,7$ 7$#7$sHSM73c8TW=FX@Y;Y3ZZ[\7$n- 7$ E,"345 7$ uS'\*+ 7$ E$)$ %7$r