ukietdZddlmZddlmZmZddlZddlZddlZddl Z ddl m Z ddl Z ddl Z ddlZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZm Z m!Z!m"Z"m#Z#ddl$m%Z&ddl'm(Z(e Z)ejTddGddZ+ejTddGddZ,d%dZ-d&dZ. d'dZ/ d(dZ0 d(dZ1 d)dZ2 d)dZ3GddZ4Gd d!Z5Gd"d#Z6e6dZ7 d*d$Z8y)+z-Colocated Python function API implementation.) annotations)CallableSequenceN)Any)api) tree_util)util)pxla) xla_client) api_boundary)wraps) func_backend) _deserialize_deserialize_specs _make_specs_for_serialized_specs _serialize_serialize_specs)register_backend_cache) ifrt_programsT)frozenslotsc0eZdZUdZded<ded<ded<y) FunctionInfoz*User function wrapped by colocated_python.Callable[..., Any]fun str | Nonefun_sourceinfoinspect.Signature | None fun_signatureN)__name__ __module__ __qualname____doc____annotations__a/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/jax/experimental/colocated_python/func.pyrr+s2))r&rceZdZUdZdZded<dZded<dZded<dZded <dZ ded <dZ d ed <ddddddd  ddZ y)Specializationz/Specialization for a colocated_python function.Ntree_util.PyTreeDef | Nonein_specs_treedef'tuple[api.ShapeDtypeStruct, ...] | Nonein_specs_leaves*Callable[..., ShapeDtypeStructTree] | None out_specs_fnout_specs_treedefout_specs_leavesxc.DeviceList | Nonedevices)r+r-r/r0r1r3cZ| |j}n|j td| |j}n|j td| |j}n|j td| |j}n|j td| |j }n|j td| |j }nO|j tdt|tjstjt|}t||||||S)z,Creates a new specialization with overrides.zin_specs already specifiedzout_specs_fn already specifiedzout_specs already specifiedzdevices already specified) r+ ValueErrorr-r/r0r1r3 isinstancexc DeviceListtupler))selfr+r-r/r0r1r3s r'updatezSpecialization.update?s7..   * 3 44,,o   ) 3 44&&l   & 7 88 00   + 4 55..   * 4 55 g  ! 2 33  / eGn-g   r&) r+r*r-r,r/r.r0r*r1r,r3z+Sequence[jax.Device] | xc.DeviceList | None) r r!r"r#r+r$r-r/r0r1r3r;r%r&r'r)r)4s715.5=A/:A=A,:A26/6>B;B"&' & 6:AEAE6:BF=A030? 0 ? 0 4 0@0;0r&r)ct|tjstdt |dt j |j|j|jS)z7Extracts a spec for a value, which must be a JAX Array.zFcolocated_python only supports jax.Array as input and output, but got .)shapedtypesharding) r6jaxArrayr5typerShapeDtypeStructr>r?r@)xs r' _get_specrFrsW Asyy !  G9A    AGG177QZZ PPr&ct}|D]7}t|dd}||j|jj9|syt |dk7rt d|d|jS)zBReturns a representative device list from function call arguments.r@NzLAll arguments must use the same device list, but got multiple device lists: r=)setgetattraddr@_internal_device_listlenr5pop)argsdevice_list_setrEr@s r'_infer_devices_from_argsrQ~s(+/ r?.0sdss r' z)_compile_to_executable..s.58chh399cii0s>Ac34K|]}|jywrT)r@rWs r'rZz)_compile_to_executable..sA AT) committedctj||f}j|d}|j}tj|S)NF) with_tokens)r tree_leavesexecute_shardedconsume_with_handlerstree_unflatten)rOkwargs args_leavesexecute_resultresultsloaded_executable out_handlersr0s r'callz$_compile_to_executable..callsW))4.9k(88 59n44\Bg  % %&7 AAr&z#PjRtCompiler requires an HloProgramN)rrmake_colocated_python_programclientr9%make_colocated_python_compile_optionscompile_ifrt_programr global_avals_to_results_handlerhandlersrAerrorsJaxRuntimeErrorstrr)namerr+r-r0r1r3fun_and_specializationpickled_functionprogram ifrt_clientout_sdss out_shardingscompile_optionsrjerhris ` @@r'_compile_to_executabler}s   67  7 7 g8H ' !!+ .lowered_fun..s@QIaL@)rr tree_flattenr9rSINGLETON_RESULT_STOREpushr) rOrdresult result_leaves out_treedefout_spec_leavesr3infouids r' lowered_funz;_make_output_specs_and_push_result_fun..lowered_funsb TXXt &v &F!*!7!7!?M;@-@@O'',,S-@ K' BBr&r unknown_output_specs_and_push_resultrtrr+r-r0r1r3)returnz jax.Array) r+r-r0r1r3rrrrJrr}r9)rspecializationrrr1r0rtr3s` ` @r'&_make_output_specs_and_push_result_funrs  ( ( 44 4  ' ' 33 3  ) ) 11 1  ( ( 00 0    ++ +  " "'C)2(>(>&~'='=>)%% :y 1$ 6. /$   %66$44)-.$$ r&c \|jJ|jJ|jJ|jfd}tjdif\}}t |j dd}|d}t|||t||j|j|jS)z9Makes a function that pops results from the result store.cntjj}tj|SrT)rrrNrrc)rr0rs r'rz)_make_pop_result_fun..lowered_funs- 77;;C@M  # #$5} EEr&r%r r _pop_resultr) r0r1r3rrrJrr}r9)rrrrr-r+rtr0s ` @r'_make_pop_result_funrs  ) ) 55 5  ( ( 44 4    ++ +$66F'0&<&< >'#/# :y 1$ 6 $   'O,&88%66$$ r&c X|jJ|jJ|jJ|jJ|jJt |j dd}t||j |j|j|j|j|jS)z;Makes a function that asynchronously executes the function.r rr)r+r-r0r1r3rJrr})rrrts r'_make_async_execution_funr s  ( ( 44 4  ' ' 33 3  ) ) 55 5  ( ( 44 4    ++ + :y 1$   ((%66$44&88%66$$ r&ctjdjJjJjJt j dtjdfd}|S)z.specialized_func2s 9Q  %  + + 3  ( ( 06.# - $D$$ 3E$3 / / ,22"3!13N$=n$ 1 ! AC'ncBDG9Q9QL(1'9'9D&>( $J 433ZP<PI3<2H2H3 / /,22"3!&'7!83N$=n$ ";4!P s9Q|"4 26 2F C>B;C MC9Q9Q~C MsA5D%A>D%D1%D.1D;) r test_eventr+r-r3random getrandbits threadingLock)rrrrrrs`` @@@r'_uncached_get_specialized_funcr#s //?@  ( ( 44 4  ' ' 33 3    ++ +2# .. %DDL r&ceZdZdZej ddGddZej ddGddZdZ d d Z y ) _SpecializedCollectiona Collection of specialized functions for a single unspecialized function. The `get()` method retrieves the specialized function for the provided input spec, either by looking up a cache or by compiling the specialized function. Looking up a cache with an input spec as a key can be slow, because `Sharding`'s equivalence comparison is slow. Instead, we maintain two caches for the same value: we use the ID of the sharding object (via `WeakSpec`) as the key in one cache, and the corresponding strong references to the sharding object (via `StrongSpec`) as the key in another cache. Looking up the `WeakSpec`-keyed cache is fast. Note that the ID integer in the `WeakSpec` cache will remain valid as long as a strong-ref exists in the `StrongSpec` cache. The `StrongSpec`-keyed cache is unbounded, while the `WeakSpec`-keyed cache is LRU(1): if there is a miss in the `WeakSpec` cache but a hit in the `StrongSpec` cache, the strong-ref is the `StrongSpec` cache and the ID integer in the `WeakSpec` cache are both updated. T)r unsafe_hashcJeZdZUdZded<ded<ded<ded < d d Zy ) _SpecializedCollection.WeakSpecz;WeakSpec stores just the `id()` of the input spec sharding.ztuple[jax.numpy.dtype, ...]dtypesztuple[tuple[int, ...], ...]shapesztuple[int, ...] sharding_idstree_util.PyTreeDeftreedefctd|D|_td|D|_td|D|_||_y)Nc34K|]}|jywrT)r?rs r'rZz;_SpecializedCollection.WeakSpec.__init__..7a!''7r\c34K|]}|jywrT)r>rs r'rZz;_SpecializedCollection.WeakSpec.__init__..rr\c3FK|]}t|jywrT)idr@rs r'rZz;_SpecializedCollection.WeakSpec.__init__..sD11::Ds!)r9rrrr)r:rers r'__init__z(_SpecializedCollection.WeakSpec.__init__sB7;77dk7;77dkD DDddlr&N)reSequence[jax.Array]rr)r r!r"r#r$rr%r&r'WeakSpecrs2E '' ''!!  .9Lr&rc>eZdZUdZdZded<dZded< ddZy) !_SpecializedCollection.StrongSpecz/StrongSpec stores the full input spec sharding.Nr*r+r,r-c@td|D|_||_y)Nc32K|]}t|ywrTrrs r'rZz=_SpecializedCollection.StrongSpec.__init__..s"EA9Q<"Er)r9r-r+)r:re pytreedefs r'rz*_SpecializedCollection.StrongSpec.__init__s#"E"EEd'dr&)rerrr)r r!r"r#r+r$r-rr%r&r' StrongSpecrs193707?CO<C(.(;N(r&rct}i|_i|_i|_i|_d|_t j|_y)Nr) int _weak_to_id _id_to_weak _strong_to_id_id_to_compiled_counterrr_mu)r: CompiledIds r'rz_SpecializedCollection.__init__s<JJLDJLDNPDACDDM~~DHr&c tj||}|jj|}||j|S|j 5tj ||}|jj|d}|m|jj|}|j|=||j|<||j|<||j|<|j|cdddS|j0  r&rcReZdZdZedZddZdZ d dZ d dZ y) _CachedColocatedFunctionMakerzFunction maker for colocated Python functions. Generated functions are stored (cached) indefinitely so that they can be reused, until the cache is dropped. 'colocated_python_specialized_func_cachec||ntjj|_ggfd}tj j |j||_|_y)NcFjjyrT)r)specialized_collectionsspecialized_functionssr' clear_cachesz<_CachedColocatedFunctionMaker.__init__..clear_cachess##%!!#r&) uuiduuid4rheld_byr JAX_CACHErrr)r:rrrrs @@r'rz&_CachedColocatedFunctionMaker.__init__s`%17tzz|7G7GDL<>68$"++AA $;D !6Dr&c|jj|jj tjj |j y#t$rYywxYwrT)rrrrrrrAttributeError)r:s r'__del__z%_CachedColocatedFunctionMaker.__del__)sU  &&($$& #--AA$,,O    s)A A,+A,c d dfd }ddtjtfd}tj|}||_|S)z)Internal implementation of make_callable.c |d\}}n#tj|\}}t|}jj ||||S)a7Returns a colocated Python callable with extra specialization. Args: in_specs: Optionally specifies the expected input specs. Input specs are expressed as a `PyTree[ShapeDtypeStruct]` for `(args, kwargs)` of a function call. out_specs_fn: Optionally specifies a function that computes the output specs from input specs. If unspecified, colocated Python will compute the output specs during the very first execution, and this execution will be synchronous. devices: Optionally specifies the devices to execute the function on. Must be provided if `in_specs` has no leaves because devices cannot be inferred from input specs or arguments. Returns: A colocated Python callable with extra specialization. )NN)r+r-r/r3)rrr9_make_callabler;) in_specsr/r3r-r+in_specs_leaves_listrr:rs r' specializez@_CachedColocatedFunctionMaker._make_callable..specialize9sr2  ,6))1:1G1G 2 .. 45     /-'  r&cyrTr%r%r&r'z>_CachedColocatedFunctionMaker._make_callable..gr&cyrTr%r%r&r'rz>_CachedColocatedFunctionMaker._make_callable..hrr&c Btj||f\}}t|dk(}|r j t d j duxr j du}|sm|sk5 }|:t} jj|tj| dddj|| |i|}~|S5 }||r j t }nO|r&t jt|}n'|r%t jt!|} j"j|tj| ddd|i|}~|S#1swYxYw#1swY xYw)aExecutes the given Python function on the same devices as the arguments or as specialized. If the callable has not been specialized with output shapes and shardings (see `specialize` above), the very first call will run synchronously to discover output shapes and shardings, and will run asynchronously after. If specialized with output shapes and shardings, every execution of the callable will be asynchronous. rNzkNo devices found. colocated_python function without input arguments must be first specialized with devices.)r3)r-r+)rrrMr3r5r+r-rrappendweakrefrefrrr;rQr9r)rOrdrer+no_inputfully_specified_in_spec collectionrfuncrr:rspecialized_collections_wrefspecialized_functions_wrefwref_mus r'__call__z>_CachedColocatedFunctionMaker._make_callable..__call__ks'0&<&= r&)rErrzapi.ShapeDtypeStruct)rOz Sequence[Any]rr2)rtrsrrr+rr- tuple[api.ShapeDtypeStruct, ...]r0rr1rr3z xc.DeviceListrr)rrrr)rrrr)rrrr)rrr)9r# __future__rcollections.abcrrrinspectrrtypingrrrrAjax._srcrrr jax._src.interpretersr jax._src.libr r7jax._src.traceback_utilr jax._src.utilr !jax.experimental.colocated_pythonr/jax.experimental.colocated_python.serializationrrrrrjax.extend.backendrrjax.extend.ifrt_programsrShapeDtypeStructTreerrr)rFrQr}rrrrrrrrrr%r&r'r!s4".   &)0:]]S2d$/**0*d$/::0:z Q"2 2 2 *2 6 2 + 2 7 2 2 2 j" """ " "J   "    F ".U U"UUpxxv  4kk\8= ,r&