biddlmZmZddlmZmZmZmZmZddl Z erddl m Z eGddZ eGdd Zy) ) dataclassfield) TYPE_CHECKINGAnyClassVarOptionalUnionN) FeatureTypeceZdZUdZeeed<eddZe eed<dZ e eed<dZ e e ed <eddd Zeed <d Zd edeedfffdZy) Translationa`Feature` for translations with fixed languages per example. Here for compatibility with tfds. Args: languages (`dict`): A dictionary for each example mapping string language codes to string translations. Example: ```python >>> # At construction time: >>> datasets.features.Translation(languages=['en', 'fr', 'de']) >>> # During data generation: >>> yield { ... 'en': 'the cat', ... 'fr': 'le chat', ... 'de': 'die katze' ... } ``` languagesNFdefaultrepriddictdtypepa_typerinitr_typectjt|jDcic]}|tjc}Scc}wN)pastructsortedrstring)selflangs X/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/datasets/features/translation.py__call__zTranslation.__call__)s3yyt~~8NO$ +OPPOsA returnr clddlm}t|jDcic] }||d c}Scc}w)z2Flatten the Translation feature into a dictionary.r )Valuer)featuresr%rr)rr%ks r!flattenzTranslation.flatten,s+#,24>>,BCq5?"CCCs1)__name__ __module__ __qualname____doc__liststr__annotations__rrrrrrrrr"r rr(r!r r s*Cyd7B 7!E8C=!!GXc]!}5uEE3EQD}d3 3E.FFGDr1r ceZdZUdZdZeeed<dZee ed<e ddZ ee ed<dZ ee ed <dZeeed <e ddd Ze ed <d ZdZdZdedee dfffdZy)TranslationVariableLanguagesa`Feature` for translations with variable languages per example. Here for compatibility with tfds. Args: languages (`dict`): A dictionary for each example mapping string language codes to one or more string translations. The languages present may vary from example to example. Returns: - `language` or `translation` (variable-length 1D `tf.Tensor` of `tf.string`): Language codes sorted in ascending order or plain text translations, sorted to align with language codes. Example: ```python >>> # At construction time: >>> datasets.features.TranslationVariableLanguages(languages=['en', 'fr', 'de']) >>> # During data generation: >>> yield { ... 'en': 'the cat', ... 'fr': ['le chat', 'la chatte,'] ... 'de': 'die katze' ... } >>> # Tensor returned : >>> { ... 'language': ['en', 'de', 'fr', 'fr'], ... 'translation': ['the cat', 'die katze', 'la chatte', 'le chat'], ... } ``` Nr num_languagesFrrrrrrrc|jrtt|jnd|_|jrt|j|_yd|_yr)rrsetlenr4rs r! __post_init__z*TranslationVariableLanguages.__post_init__\s=8<DNN 34D48NNS0r1ctjtjtjtjtjdS)Nlanguage translation)rrlist_rr8s r!r"z%TranslationVariableLanguages.__call__`s8yybhhryy{&;BHHUWU^U^U`Labccr1c t|j}t|ddhk(r|S|jrSt||z rEtddjt t||z ddj|dg}|j D]J\}}t |tr|j||f*|j|Dcgc]}||fc}Ltt |\}}||dScc}w)Nr<r=zSome languages in example (z, z) are not in valid set (z).r;) r6r ValueErrorjoinritems isinstancer.appendextendzip) rtranslation_dictlang_settranslation_tuplesr textelr translationss r!encode_examplez+TranslationVariableLanguages.encode_examplecs)t~~&  Z$? ?# # ^^$4 5 @-diisCS?TW_?_8`.a-bbz{|E|EFN|O{PPRS   *002 GJD$$$"))4,7"))*E"D":*EF  G#&v.@'A"B <%lCC +Fs C? r#r cLddlm}m}||d||ddS)zCFlatten the TranslationVariableLanguages feature into a dictionary.r )Listr%rr;)r&rOr%)rrOr%s r!r(z$TranslationVariableLanguages.flattenzs))U8_-h0  r1)r)r*r+r,rrr-r/r4intrrr.rrrrrr9r"rMr rr(r0r1r!r3r33s>!%Ix~$#'M8C='d7B 7!E8C=!!GXc]!=EPUVE3VMdD. }d3 3E.FFG r1r3) dataclassesrrtypingrrrrr pyarrowrr&r r r3r0r1r!rTsR(@@% $D$D $DN M M  M r1